Nothing in this chapter shall require any change in the plans, construction, size or designated use of any building permit which was issued before adoption of this chapter provided that construction pursuant to such plans has been substantially started and cannot be reasonably adapted to the provisions of this chapter, and provided further that the construction shall be diligently pursued to completion.