Destruction. If any building or structure in which a nonconforming use is conducted is hereafter removed, or destroyed by fire, wind, explosion, structural failure or other natural cause, to the extent of 50% or more of its true market value at the time of such damage, as adjusted from assessed value based upon State Board of Equalization rates, it shall not be repaired or rebuilt, unless the appropriate variance is granted by the Board of Appeals.