Preexisting single- and/or two-family residential structures may be reconstructed, changed, extended or altered without the necessity of a public hearing, provided that it is determined that no such reconstruction, change, extension or alteration shall increase the nonconforming nature of said structure. Such determination shall be made by the Board of Appeals or the Zoning Administrator. In the event that such reconstruction, change, extension or alteration increases the nonconformity of said structure, no such reconstruction, change, extension or alteration shall be permitted unless there is a finding by the Board of Appeals pursuant to a public hearing that such reconstruction, change, extension or alteration shall not be substantially more detrimental than the existing nonconformity to the neighborhood.