Nothing herein contained shall require any change in the plans, construction or designated use of a building or structure for which a building permit has been heretofore issued and substantial construction has taken place prior to the date of the adoption of this chapter.