Nothing in this chapter shall require any change in the plans, construction, size or designated use of any building, structure, or part thereof, for which any building permit has been granted before November 20, 1979, provided that construction from such plans shall have been started within 60 days after November 20, 1979, and shall be diligently pursued to completion.