Within the districts established by this chapter or amendments that may later be adopted, there exist lots, structures, uses of land and structures, and characteristics of use which were lawful at the time this chapter was passed or amended, but which would be prohibited, regulated, or restricted under the terms of this chapter or future amendment. It is the intent of this chapter to permit these nonconformities to continue until they are removed, but not to encourage their survival. It is further the intent of this chapter that nonconformities shall only be enlarged upon, expanded or extended to a certain degree, nor be used as grounds for adding other structures or uses prohibited elsewhere in the same district.