A structure, the use of which does not conform to the use regulations for the district in which it is situated, shall not be enlarged or altered unless such structure, including such enlargement or alteration, is made to conform to all regulations, including use, for the district in which it is situated or unless permission therefor shall be obtained by application to and approval by the Board of Appeals.