No existing structure devoted to a use not permitted by this chapter in the zoning district in which it is located shall be enlarged, extended, constructed, reconstructed, moved or otherwise have any structural alterations made to it except in changing the use of the structure to an authorized use in the zoning district in which it is located, unless the Board shall interpret that the enlargement, extension, construction, reconstruction or structural alteration is necessitated by the natural expansion and growth of trade of the nonconforming use.