Within the districts established by these regulations or by amendments which may later be adopted, there exist lots, structures, buildings and uses which were lawful before these regulations were effective or amended, but which would be prohibited, regulated or restricted under these regulations or future amendment. It is the intent of this regulation to permit these nonconformities to continue until they are removed, but not to encourage their survival. It is further the intent of these regulations that nonconformities shall not be enlarged upon, expanded or extended, or be used as grounds for adding other structures or uses prohibited elsewhere in the same district. In any district in which dwellings are permitted, such a use and customary accessory buildings may be erected on any single lot of record at the effective date of adoption of these regulations, notwithstanding limitations imposed by other provisions contained herein. Variance of yard requirements shall be obtained only through action of the Board.