Nothing herein contained shall require any change in the plans, construction or designated use of a building for which a building permit has been heretofore issued and substantial construction has taken place prior to the date of the adoption of this chapter.