Except upon the granting of a variance by the Zoning Board of Appeals, no building containing a nonconforming use shall be enlarged, altered, extended, reconstructed or restored or placed on a different portion of the lot or parcel of land occupied by such use on the effective date of this chapter, nor shall any external evidence of such use be increased by any means whatsoever. In the case of any lot containing a nonconforming residential use in a nonresidential district or a nonconforming nonresidential use in a residential district, any proposed new use or increase in the size of said use on any portion of such lot, which new use or increase in the size of said use is not accompanied by the total elimination of the existing nonconforming use, shall be deemed to be an enlargement and extension of an existing nonconforming use.