Such building or structure shall in no case be enlarged or expanded unless the use therein is changed to a conforming use. An exception, however, shall be permitted whenever a one-family dwelling required a grant of variance from the provisions of the then-existing Zoning Ordinance to permit its construction, expansion or enlargement. In such instances, the building may thereafter be enlarged without need of further application to the Zoning Board of Adjustment for any additional variances to the extent only of adding thereto a second-story dormer or dormers to permit the use of said second story to the full limits of the exterior walls of the first floor, if said exterior walls were extended upward to the height of the second story. Nothing herein contained, however, shall prohibit the enlargement of a building which meets the use requirements of this chapter and is nonconforming because of yard, height or lot area regulations, provided that said enlargement results in conformity with all yard, area and height requirements at the conclusion thereof.