Destruction. If any building or structure in which a nonconforming use is conducted is hereafter removed or destroyed by fire, wind, explosion, structural failure or other natural cause to the extent of 50% or more of its true market value at the time of such damage, as adjusted from assessed value based upon State Board of Equalization rates, it shall not be repaired or rebuilt unless a special permit is granted by the Planning Board.