Any variance or exception from the terms of this chapter hereafter approved permitting the erection or alteration of any structure or structures or permitting a specified use of any premises shall expire by limitation unless such construction or alteration shall have been actually commenced on each and every structure permitted by variance, or unless such permitted use has been commenced, within one (1) year from the date of publication of the notice of the judgment or determination. The running of the period of limitation herein provided shall be tolled from the date of filing an appeal from the decision of the board to a court of competent jurisdiction, until the termination of such appeal or proceeding. For purposes of this subsection, the mere receipt of a building permit, without more, shall not be deemed to constitute actual commencement of construction or alteration.