In exercising the above-mentioned powers, the Zoning Board of Appeals may, in conformity with the provisions of this chapter, reverse or affirm, wholly or partly, or may modify the order, requirement, decision or determination, including a stop order or orders to cease and desist, as ought to be made. Notice of such decision shall forthwith be given to all parties of interest.