Nothing herein contained shall require any change in plans, construction or designated use of a building for which a building permit has been heretofore issued and the construction of which shall have been diligently prosecuted within three months of the date of such permit and the ground story framework of which, including the second tier of beams, has been completed according to such plans as filed within one year from the date of this chapter. If such conditions are not met, any building permit shall be held to be in violation of this chapter, subject to the penalties therein.