No building shall be erected or altered or exceed the height; accommodate or house a greater number of families; occupy a lesser amount of floor space; have narrower or smaller rear yards, front yards, side yards, inner or outer courts, than as specified in this chapter for the district or zone in which the building is located. However, such a nonconforming structure may be altered or enlarged so long as the use of the premise is permitted in the zoning district and any proposed addition is in conformance with the schedule of yards, height and any other zoning district requirements.