Applicability. This bylaw or any amendments thereto shall not apply to the use of any structure or land lawfully in existence or lawfully begun, or to a building or special permit issued before the first publication or notice of the public hearing on such bylaw required by MGL c. 40A, § 6, as amended, but shall apply to any change or substantial extension of such use, to a building or special permit issued after the first notice of said public hearing, to any reconstruction, extension or structural change of such structure and to any alteration of a structure begun after the first notice of said public hearing to provide for its use for a substantially different purpose or for the same purpose in a substantially different manner or to a substantially greater extent, except where alterations, reconstruction, extension or structural change to a single- or two-family residential structure does not increase the nonconforming nature of said structure.