How to fight petty day-to-day corruption is a question often debated by politicians, by the public and in the economic literature. Early studies have noted that a simple and well-known way to fight day-to-day corruption is to create competition among corrupt officials. This paper shows that even a benevolent government might not encourage competition among officials in a way that eliminates corruption. This is due to a tradeoff between corruption and compliance costs. More differentiated bureaucratic services decrease compliance costs but increase the leeway for extortion. The analysis further reveals that exogenous shocks, for example in the form of foreign aid that aims to improve anti-corruption capacities, may prompt a benevolent government to increase the differentiation of bureaucratic services, thereby leading to an increase in corruption.