Except as herein provided, no building or premises shall be used except in conformity with the provisions of this chapter which apply to the district in which it is located. However, any nonconforming use existing at the date of adoption of this chapter may be continued, provided that the building or premises involved shall be neither altered nor enlarged unless the use shall be changed to a use permitted in the district. No nonconforming use, if once changed to a use permitted in the district in which it is located, shall ever be changed back to a nonconforming use.