Within the districts established by this chapter or amendments that may later be adopted there exist lots, structures, uses of land and structures, and characteristics of use which were lawful before this chapter was passed or amended, but which would be prohibited, regulated, or restricted under the terms of this chapter or future amendment. It is the intent of this chapter to permit these nonconformities to continue until they are removed, but not to encourage their survival. It is further the intent of this chapter that nonconformities shall not be enlarged upon, expanded or extended, nor be used as grounds for adding other structures of uses prohibited elsewhere in the same district. This provision shall not apply to the expansion, extension or enlargement of residential structures located in residential districts which expansion, extension or enlargement may be granted by way of special use permit.