No accessory building shall be erected or altered so as to encroach upon the front side or rear yards except that a detached garage used solely for garage purposes and erected in the rear of the plot may, if necessary for accessibility, encroach on a rear or side yard, but may not reduce the width of either the rear or side yards to be less than five feet.