Any nonconforming lot or building existing at the time of adoption of this chapter may have a building permit issued for a permitted use without an appeal to the Board of Adjustment, provided that the building coverage, parking and yard requirements are not exceeded or diminished, as the case may be, by more than 20%.