Whenever a building is nonconforming because it fails to comply with any height, area, yard, off-street parking or requirements of this chapter, other than use, and such building is partially destroyed, such building may be restored to its prior condition; provided, however, that such restoration shall not enlarge the previously existing nonconformance.