Restoration. A building devoted to a nonconforming use destroyed or damaged by fire, wind, explosion, structural failure or other natural causes to the extent of 50% or more of its fair value at the time of such damage shall not be repaired or rebuilt except in conformity with the provisions of this chapter.