Nothing contained in this chapter or amendment thereof shall require any change in the plans, construction or designated use of the building for which a permit has been issued, provided construction or use thereof has been started prior to the adoption hereof, or to an amendment hereto or where valid contracts for such construction or use have been entered into prior to the adoption of this chapter or amendment made thereto.