A nonconforming building or structure may be altered, but not enlarged or extended, during its life provided no walls or partitions, no doors or other openings are removed or installed, and no other structural changes are made either within or without the building or structure, unless said building is changed to a building or structure conforming to the requirements of this chapter.