For purposes of this section, "actual construction" is defined to be the actual placing of construction materials in their permanent position, fastened in a permanent manner, except that where demolition or removal of an existing building or structure has been substantially begun preparatory to rebuilding, such demolition and removal shall be deemed to be actual construction. Similarly, whenever a district shall be changed hereafter, the provisions of this chapter with regard to building permits duly and legally issued prior to the effective date of this chapter shall apply to building permits duly and legally issued for construction in such changed district prior to the effective date of the amendment affecting such change.