Within the districts established by this chapter or amendments that may later be adopted, there exist lots, structures and uses of land and structures which were lawful before this chapter was passed or amended but which would be prohibited, regulated or restricted under the terms of this chapter or future amendment. It is the intent of this chapter to permit these nonconformities to continue until they are removed. Such uses are declared by this chapter to be incompatible with permitted uses in the districts involved. It is further the intent of this chapter that, except for single- and two-family residences, nonconformities shall not be enlarged upon, expanded or extended nor be used as grounds for adding other structures or uses prohibited elsewhere in the same district, unless to make such use or structure conform to the minimum requirements of this chapter.