Except as hereinafter provided, this Zoning Bylaw shall not apply to structures or uses lawfully in existence or lawfully begun, or to a building permit or special permit issued before the first publication of notice of the public hearing on this Zoning Bylaw or any amendments thereto, but shall apply to any change or substantial extension of such use, to a building permit or special permit issued after the first notice of said public hearing, to any reconstruction, extension or structural change of such structure and to any alteration of a structure begun after the first notice of said public hearing to provide for its use in a substantially different purpose or for the same purpose in a substantially different manner or to a substantially greater extent except where alteration, reconstruction, extension or a structural change to a one-family or two-family residential structure does not increase the nonconforming nature of said structure.