Accessory buildings and accessory uses customarily incident to the above uses (not involving the conduct of business), when located on the same lot, including a private garage for one or more cars, bona fide servants’ quarters not for rent or used for commercial purposes, and private club swimming pools except, however, any application for a building permit submitted to the building inspector for the construction of a private club swimming pool shall be referred by the building inspector to the city secretary who shall call a special hearing before the city council in the same manner and procedure as hearings before the board of adjustment.