It is not intended by this chapter to amend, repeal, abrogate, annul or to impair or interfere with covenants, easements or restrictions between persons or to rescind a building permit heretofore issued by the Building Inspector of the Village. However, no provision in any such covenant, deed restriction or contract shall be deemed to justify noncompliance with any provision of this chapter.