Any nonconforming use or structure existing at the time of the passage of this chapter may be continued upon the lot or in the structure so occupied, and any such structure may be restored or repaired in the event of partial destruction thereof. Nothing in this chapter shall prevent the strengthening or restoring to a safe condition any wall, floor or roof which has been declared unsafe by the Building Inspector.