The filling, removal, excavation or mining of mineral, sand, gravel, clay, bedrock, peat, loam or topsoil (herein referred to as "earth material operations") is permitted in all zones, with the exception of the land designated as the Gateway Conservation Zone, after issuance of a special permit by the Planning and Zoning Commission.