All structures for which permits have been obtained, and the construction of which or a portion of which has been begun, or for which a contract or contracts have been let, pursuant to a permit issued prior to the passage of this chapter, may be completed and used in accordance with the plans for which said permit was granted.