Nothing herein contained shall require any change in plans, construction or designated use of a building, structure or sign for which a building permit has heretofore been issued and the construction of which shall have been diligently prosecuted within three months of the date of such permit, and the ground story framework of which, including the second tier of beams, shall have been completed within six months of the date of the permit, and where the entire building shall have been completed according to such plans as filed within one year from the date of the adoption of this chapter.