In all instances, including those cited above, a special use permit may be revoked by the Zoning Board of Appeals or the Planning Board, after public hearing, if it is found and determined that there has been a substantial failure to comply with any of the terms, conditions, limitations and requirements imposed by said special use permit. Whichever Board was given the original power to grant the special use permit is the Board that has the authority to revoke the permit, according to the conditions set forth in this chapter.