(E) Restoration of damaged nonconforming building. A building or structure, all or substantially all of which is designed or intended for a use which is not permitted in the district in which it is located, and which is destroyed or damaged by fire or other casualty or act of God, to the extent that the cost of restoration to the condition in which it was before the occurrence will exceed 50% of the cost of restoration of the entire building or structure new, shall not be restored unless the building or structure, and the use thereof, shall conform to all regulations of the district in which it is located. In the event that such damage or destruction is less than 50% of the cost of restoration of the entire building or structure new, no repairs or reconstruction shall be made unless such restoration is started within one year from the date of the partial destruction and is diligently prosecuted to completion.