When the majority of buildings built on one side of the street between two intersecting streets at the time of the passage of this chapter have been built with a minimum setback different from that required by this chapter, no building hereafter erected or altered shall project beyond the minimum resulting setback so established, provided that no single-family dwelling shall be required to maintain a setback more than 40 feet. No accessory building shall be located in a side yard, the side yard distance being mandatory and not including accessory buildings.