Associate members. The board of selectmen shall also appoint three persons, inhabitants of the Town, as associate members of said board of appeals, who shall hold office for a term of three years, except that when associate members are first appointed hereunder, one shall be appointed for a term of one year, one appointed for a term of two years, and one member shall be appointed for a term of three years. In case of vacancy or inability to act or interest on the part of any member of the board of appeals, his place shall be taken by an associate member who shall be designated by the chairman of the board of appeals.