Nothing herein contained shall require any change in the plans, construction or designated use of a building for which a building permit has been heretofore issued, and plans for which are on file with the City Clerk at the time of the passage of this Chapter (September 21, 1964), and the construction of which in either case shall have been diligently prosecuted within one (1) year of the date of such permit, and the ground story framework of which, including the second (2nd) tier or beams, shall have been completed, according to such plans as filed, within two (2) years from the date of the passage of this Chapter (September 21, 1964).