In the event that the Building Inspector determines that the nonconforming nature of such structure would be increased by the proposed reconstruction, extension, alteration, or change, the Board of Appeals may, by special permit, allow such reconstruction, extension, alteration, or change where it determines that the proposed modification will not be substantially more detrimental than the existing nonconforming structure to the neighborhood.