Except as hereinafter provided, no building or premises shall be used except in conformity with the provisions of this chapter which apply to the district in which it is located. However, any nonconforming use existing at the time of passage of this section[1] may be continued provided that the building or premises involved shall neither be altered nor enlarged unless the use thereof shall be changed to a use permitted in the district. No nonconforming use, if once changed to a use permitted in the district in which it is located shall ever be changed back to a nonconforming use. Nothing in this chapter shall prevent the restoration of a building destroyed less than 50% of its value, exclusive of the foundations, by fire, explosion, act of God, or act of public enemy subsequent to the passage of this section, or shall prevent the continuance of the use of such building or part thereof as such use existed at the time of such destruction of such building or part thereof. Nothing in this chapter shall prevent the restoration of a wall declared unsafe by the Construction Official.