Any new construction on site in association with the office use shall maintain side and rear yard setbacks twice that ordinarily required in the zone. Where conversion of an existing dwelling or farm structure is proposed, the conversion shall not cause the structure to infringe into the otherwise required front, side or rear setback.