Where the proposed use is for the expansion or structural alteration of an existing single-family dwelling or the erection of an accessory use or structure in an agricultural or residential district, the Zoning Officer shall carefully consider the application for compliance with this chapter and may either issue or deny the permit applied for. When the application is for any other permitted use in any zone, including the physical alteration of the land form without the construction of any buildings or structure, the Zoning Officer shall, prior to the issuance of an permit, refer copies for such plans, drawings and statements to the Planning Board and the Town Engineer for review and recommendation; however, the Zoning Officer, in his discretion, may grant a permit without referral to the Planning Board and the Town Engineer for any of the following:
(1) Any site work being done on less than five acres and requiring less than 100 cubic yards of material displaced.
(2) Ponds of less than 1/2 acre in size.
(3) Any fill of less than 100 cubic yards.