Any detached accessory building shall conform to all front, rear and side yard requirements unless otherwise provided in this chapter, except that accessory buildings located on lots in waterfront zone districts (R-1, R-1A, WD, R-SP, R-SC, R-HH, R-PB, R-BH or R-BB) or directly abutting a lagoon or bay shall conform to all principal building setbacks.