Any structure lawfully erected and existing and any use lawfully being made of land or buildings which do not conform to this chapter, as adopted or as amended, may be continued to the same extent and for the same purpose but shall not be expanded or altered, except in conformance with this chapter. This exemption shall include buildings, structures and uses authorized by a building or special permit issued prior to the publication of the first hearing notice for an amendment to this chapter which would make them nonconforming, provided that the construction or use under such a permit is commenced within six months after the permit is issued and, in case of construction, is continued to completion in a reasonably expeditious manner.