This provision may be invoked only once during any five-year period. Any subsequent application to repair, alter, enlarge or extend a building shall be based upon the size and location of the building at the time of the first repair, alteration, enlargement or extension.