The lawful use of a building existing on November 2, 1965, may be continued even though such use does not conform with the provisions hereof. If no structural alterations are made, a non-conforming use of a building may be changed to another non-conforming use of the same or more restricted classification. The foregoing provisions shall also apply to non-conforming uses in districts as may be hereafter changed. Whenever a non-conforming use of a building has been changed to a more restricted use or to a conforming use, such use shall not thereafter be changed to a less restricted use.