To avoid undue hardship, nothing in this Part shall be deemed to require a change in the plans, construction, or designated use of any structure on which actual construction was lawfully begun prior to the effective date of this chapter and upon which actual construction has been carried on diligently. Additionally, where excavation, demolition, or removal of an existing structure has begun preparatory to rebuilding, such activities shall be deemed actual construction provided that work is carried on diligently.