The lawful use of a building existing at the time of the adoption of this chapter may be continued, although such use does not conform with the provisions hereof, and such use may be extended throughout the building, provided no structural alterations, except these required by law or ordinance, are made therein. If no structural alterations are made, a nonconforming use of a building may be changed to another nonconforming use of the same or a more restricted classification. The foregoing provision shall also apply to nonconforming uses in districts hereafter changed. Whenever a nonconforming use of a building has been changed to a more restricted use or to a conforming use, such use shall not thereafter be changed to a less restricted use.