Nothing in this chapter shall require any change in plans, construction or designated use of a building for which a building permit has been heretofore issued and construction diligently prosecuted within six months of the date of such permit.