With the exception of residential structures containing less than five units, a conditional use permit and, if necessary, a variance, shall be secured prior to any structural alteration, enlargement, extension, expansion or substantial rehabilitation of a nonconforming structure.