Nothing in this chapter shall be deemed to require a change in the plans, construction, or designated use of any building on which actual construction was lawfully begun prior to the effective date of adoption or amendment of this chapter and upon which actual building construction has been continuously carried on. "Actual construction" includes the placing of construction materials in permanent position and fastened in a permanent manner, and demolition, elimination, and removal of an existing structure in connection with such constructure, provided that actual construction work shall be continuously carried on until the completion of the building involved.