Nothing in this bylaw shall require a change in the plans, construction or designated use of any structure on land for which a special permit is in effect at the time of adoption of this bylaw, or on which a building permit has been issued; subject, however, to any expiration term of such a special permit or to MGL c. 40A, § 5. The special permit granting authority may require any such special permit to conform with some or all requirements of this bylaw, if it is reviewed, amended, modified or transferred.