Except in Business B districts, in no case shall any building, except a dwelling house situated in a business, commercial or manufacturing district, be required by this bylaw to be set back from a public way or private way upon which it abuts more than the average of the set backs of the buildings, if any, on the adjacent lots on either side.