Within the districts established by this chapter or amendments thereto, there exists lots, structures and uses of land and structures which were lawful before adoption of this chapter but which are prohibited, regulated or restricted under the terms of this chapter or amendment. It is the intent of this chapter to permit any nonconformities to continue until they are removed, but not to encourage their survival. Such uses are declared to be incompatible with permitted uses in the districts involved. It is further the intent that nonconformities shall not be enlarged, expanded or extended, nor be used as grounds for adding other structures or uses prohibited elsewhere in the same district.