The lawful use of a building existing upon the date this Chapter is adopted may be continued, although such use does not conform with the provisions of this Chapter, and such use may be extended throughout the building; providing that no structural alterations, except those required by law or ordinance, are made therein. If no structural alterations are made, a non-conforming use of a building may be changed to another non-conforming use of the same or more restricted classification. The foregoing provisions shall also apply to non-conforming uses in districts hereafter changed. Whenever a non-conforming use of a building has been changed to a more restricted use or to a conforming use, such use shall not thereafter be changed to a less restricted use.