Except as hereinafter provided, no premises shall be used and no building shall be used, constructed, reconstructed, extended or structurally altered except in conformity with the provisions of this chapter and the New York State Uniform Fire Prevention and Building Code which apply to the district involved. However, any nonconforming use existing at the time of passage of this chapter may be continued. No nonconforming use, once changed to a use permitted in the district in which it is located, shall ever thereafter be changed back to a nonconforming use.