It shall be unlawful to commence or to proceed with the erection, construction, reconstruction, conversion, alteration, enlargement, extension, razing or moving of any building or structure or of any portion thereof without first having applied in writing to the Director of Community Development for building permit to do so and a building permit has been granted therefor; provided, however, that no building permit shall be required to move any accessory building onto a residential lot where the gross area covered by the accessory building is one hundred (100) square feet or less and the accessory building is not permanently attached to the ground as long as there are no more than a total of two (2) accessory buildings on the residential lot allowed pursuant to this exception.