The Board of Selectmen, upon such application, may issue a permit for such earth removal in accordance with such plan and written statement or outline, and such other restrictions or conditions as it may require to ensure protection from damage to the public or natural water supply, damage to adjoining lands, roads, or structures, hazards to children and strangers, detriment to a neighborhood, and inconvenience to the public. If the Board finds that no restrictions or conditions will adequately protect against the above hazards it may refuse to issue such permit.