A nonconforming use of land, or a nonconforming use of a structure and land shall not be extended or enlarged after the effective date of the ordinance codified in this chapter by attachment on a building or premises of additional signs intended to be seen from off the premises, or by the addition of other uses of a nature which would be prohibited generally in the district involved. To avoid undue hardship, nothing in this chapter shall be deemed to require a change in the plans, construction, or designated use of any building on which actual construction was lawfully begun prior to the effective date of adoption or amendment of the ordinance codified in this chapter and upon which actual building construction has been diligently carried on. Actual construction is hereby defined to include the placing of construction materials in permanent position and fastened in a permanent manner, except that where demolition or removal of an existing building has been substantially begun preparatory to rebuilding, such demolition or removal shall be deemed to be actual construction, provided that work shall be diligently carried on until completion of the building involved.