Nothing in this chapter shall require any change in plans, construction or designated use of a structure for which a construction permit has been heretofore issued and construction diligently pursued within six months of the date of the permit.