The lawful use of a building existing at the time of adoption of this Chapter may be continued, although such use does not conform with the provisions hereof. At no time will trailers be brought into the City of Clarence with the purpose being used as storage units. If no structural alterations are made, a non-conforming use of a building may be changed to another non-conforming use of the same or more restricted classification. The foregoing provisions shall also apply to non-conforming uses in districts hereafter changed. Wherever a non-conforming use of a building has been changed to a more restricted use or to a conforming use, such use shall not thereafter be changed to a less restricted one.