To avoid undue hardship, nothing in this article shall be deemed to require a change in the plans, construction or designated use of any building on which actual construction was lawfully begun prior to the effective date of adoption or amendment of this article and upon which actual building construction has been carried out diligently. "Actual construction" is hereby defined to include the placing of construction materials in permanent position and fastened in a permanent manner. Where excavation or demolition or removal of an existing building has been substantially begun preparatory to rebuilding, such excavation, demolition, or removal shall be deemed to be "actual construction," provided that the work shall be carried on diligently.