These bylaws shall not apply to structures or uses lawfully in existence or lawfully begun, or to a building or special permit issued before the first publication of notice of the public hearing required by MGL c. 40A, § 5, at, which these bylaws, or any relevant part hereof, were adopted. Such prior, lawfully existing nonconforming uses and structures may continue, provided that no modification of the use or structure is accomplished, unless authorized hereunder.