Nothing in this chapter shall require any change in plans, construction or designated use of a structure for which a building permit has been issued when construction has been diligently prosecuted within six months following the date of such permit, and completed within the term of such permit.