Some time ago Picnic published a blog post about our cryptocurrency powered code review leaderboard. The general goal is simple: achieve code review harmony via gamification of code reviewing and add cryptocurrency payment as a fun and novel reward system.

In the article we stated that we would open source the project in the future. That day has finally arrived! Thanks to Arturs Drozdovs, Nathan Kooij and team.

https://github.com/PicnicSupermarket/pr-leaderboard

You can now get you and your dev team quickly setup and earning and trading Picnic Coin with your code reviewing contributions. Setup should be trivial, just follow the configuration guide in the README.md.

Analysis & Findings

In the original blog post we also promised some analysis on how the dashboard was perceived within the company. The primary motivation for creating the leaderboard was to raise awareness and to essentially encourage a fair usage policy for code reviewing within Picnic. We opened the leaderboard to the backend developers at Picnic at the end of April 2018. After some time we took a look at the data to see whether it had any tangible impact on our code review process. We inspected GitHub data from our repositories.

Since adopting an intense code review process towards the end of 2016 we have observed a steady upwards trend of how much we value reviewing code in relation to how much code we produce.

As we approach the time we introduced the leaderboard, we see that the average effort that a developer spends on reviewing code as opposed to producing it begins to plateau at around 1.0. The reasoning of “I should give as much as I take” is understandable, it’s the minimum requirement of the fair usage policy. While at the time we were happy that we had achieved such an increase in reviewing effort in around a year and a half in, our thirst for quality code reviews wasn’t fully quenched.

Our goal is to incentivise developers to push towards a ratio of 2. We want to offset the diminishing of knowledge sharing as the company grows at a rapid rate into 2020. Having at least two sets of eyes on each pull request is our standard model, so this is certainly attainable. Whilst the leaderboard isn’t used on a daily basis by developers, its introduction helped bring awareness to developer code review contributions.

Sustaining the Trend

The deviation between reviews and pull request trends is ever increasing. Unsurprisingly, as we expand our tech team year in year out we see an increase in the rate of open PRs. What is striking to notice is that since adopting code reviews we have seen an ever increasing rate of code reviews that surpasses that of the number of PRs being opened. The gap is drawing ever larger which is great news for code quality, knowledge sharing and bug catching. From the following graph of rolling PR and review numbers, this trend is clear.

It is also clear that the introduction of the leaderboard had no significant effect on our growth in this regard. It turns out we were already on the right path. What is promising for the future however is that there appears to be no slowing down in the rate at which we are reviewing pull requests. From the crossover point at December 2017 to February of 2018 we saw an increase of 1.74x in the rate at which we are increasing our code reviews over opening pull requests in this time period. Looking at the same window a year on from 2018 to 2019 and this rate has increased to 1.98x, and this rate continued throughout 2019.

Conclusion

At Picnic we have gamified code reviews. We have devised a novel solution to address the imbalance in code production versus code reviewal in pursuit of knowledge sharing and a fairer code review process. What we discovered is that our process is already rapidly driving us towards this goal. Such gamification has had an effect on prompting developers to be more aware of their review contributions, but with a distinct existing upwards trajectory it is hard for us to single out the leaderboard as being the sole reason for this reinvigoration of the existing trend. What it did however, was start a wider conversation within the tech team on the topic of fair code reviews.

We believe that the introduction of such a leaderboard would be more pronounced within organisations with a more sluggish code review process. If you want to try it out and see how it impacts your work then go to https://github.com/PicnicSupermarket/pr-leaderboard and let us know how it works for you!