No nonconforming building shall be enlarged, extended or increased unless such enlargement would tend to reduce the degree of nonconformance, provided, however, that any nonconforming residential and accessory buildings now existing can be enlarged, extended or increased in size as long as the enlargement does not violate front, side, rear yard, height and total lot coverage regulations for the zone in which the building(s) exist(s) and does not continue an existing violation of front, side or rear yard setback(s), or height limitation(s), further in the direction where the extension does not violate the setback regulation.