A nonconforming use of a structure or parcel of land may, upon issuance of a special permit by the Planning Board, be changed to another nonconforming use which is of the same or lesser impact, except that no use prohibited by § 97-10C shall be permitted under any circumstances. No structure in which a nonconforming use has been changed to a use of lesser impact shall again be devoted to a nonconforming use with greater impact. In determining whether a use is of greater or lesser impact, the Planning Board shall consider the impact criteria listed in § 97-73.