When building solutions, the code we write can last many years. While casually browsing legacy code we might wonder; is this still used? The missing documentation or outdated tests do not help us answer this. When asking around, nobody really knows. Let’s try to delete it, shall we? Then, chaos ensues: it turns out it is still used to support some legacy users, in case of emergency, or by that one forgotten integration everyone still uses.

When it is actually not used anymore, it is oftentimes because of historical reasons that code is still around. Use-cases might have been replaced by new functionalities or have become obsolete and code is forgotten to be deleted. In cases where it is not trivial to investigate this dead code, how can we increase our certainty?

Code coverage

There’s an interesting relation between the problem of finding dead code, and another widespread practice: measuring code coverage for tests. Let’s find out how these two worlds may converge!

At Picnic we use one of the tools that almost every Java developer has, directly or indirectly, interacted with: the Java Code Coverage Library (JaCoCo). This tool is mostly used to report code coverage on automated test suites to gain more confidence that a test covers what it should. It collects coverage metrics by instrumenting the bytecode through a Java agent when the Java class loader loads the classes. In a future blogpost we will further dive into how it works under the hood.

In principle, a Java agent can run in any environment, not just during development — as is typically the case with JaCoCo. Why not run it also in production to see actual code coverage?

Carlos Becker¹ and Markus Harrer² also brought this up before. We wanted to follow in their footsteps but instead fetch the coverage at any time in an ephemeral context that is Kubernetes. So let’s get started!

Setting up JaCoCo in Kubernetes

First, we need to get the JaCoCo Java agent JAR. We can retrieve it from the latest release or from the central Maven repository. There are several ways to get this JAR in your Kubernetes pod, such as copying it into the container’s image, or making it available through a mounted volume. Once it’s available, it’s time to start configuring the JVM. We can configure the agent using the -javaagent JVM argument. There are plenty of configuration options. At Picnic, we run it as follows:

-javaagent:/path/to/jacocoagent.jar=includes=tech.picnic.*,output=tcpserver,address=*

This enables the JaCoCo Java agent and configures it to only instrument classes in our tech.picnic.* packages. The more specific we are here, the less performance overhead we will have, as fewer classes will be instrumented. We also configure JaCoCo to write to incoming TCP connections through tcpserver, which we will use to interact with the agent. Using the server, we can fetch the data at any time while the pod is alive.

Note: As we expose a server here, security is important. By default, the JaCoCo server listens on port 6300. By setting address=* we only allow connections from local addresses. We do not expose port 6300 in our containers and services. We will later show that we perform Kubernetes port-forwarding when interacting with the JaCoCo Java agent.

Extracting and aggregating reports

Now that we have JaCoCo running in production, it’s time to gather data! As Kubernetes pods are ephemeral, we need to dump data to a persistent volume on termination or periodically fetch our data and accept risk of data loss. As this data is not critical to us and is only applicable for that particular revision of the service anyway, we choose the latter.

To fetch the data, we need to use JaCoCo’s CLI. This is also available as a JAR from JaCoCo’s latest release on GitHub.

For gathering JaCoCo’s data, we use a script similar to the following (simplified) version, which is structured as follows:

• It loops over all pods matching the specified Kubernetes cluster, namespace and pod label.
• In a subshell, it then forwards connections to the local port 6300 to the pod’s port 6300 in a background process. This background process will be interrupted once the subshell exits.
• After the port is forwarded, we use JaCoCo’s CLI to dump the data to the local machine.
• Once we collected all the data from all the pods, we merge the dumped data into a single data file.

Note: This script assumes that selected pods run the same software source-code revision. This is necessary as later we will combine coverage data with source code to have the coverage visualized, and JaCoCo can otherwise not distinguish between revisions. As such, as long as the pods run the same revision, it is also possible to run this multiple times and combine the data as the script does.

Now that we have our binary format of the JaCoCo coverage data, we can generate an HTML or XML report out of it. For this, we need to pass JaCoCo the compiled .class files and source .java files matching the revision.

To get HTML reporting of source code line coverage, JaCoCo requires that the classes are compiled with debug information such that it can relate the bytecode coverage to the source code. At Picnic, we use the Apache Maven Compiler Plugin to compile our code, which does this by default.

To generate the report and pass the class and source code files, we also provide a script below. Path matchers for these files are defaulted to the Maven directory structure, but can be overwritten.

Time for some cleaning!

Now that we have generated a report, we can inspect the generated report/index.html and look for coverage on some suspected legacy code. We are looking for red lines, which means code is not covered. Let’s dive into some examples.

This bit of legacy code has been around for a few years already and was marked as deprecated for a few months already. When searching for usages of this method across organization repositories, we find that it is indeed unused apart from tests!

Time to drop it and leave our codebase cleaner. The planned migration for this service may have become easier now too, given that we have less functionality to take into account. Time to have our developers revisit that migration.

Another example: take this deserializer that has some logic for handling legacy data. We wondered whether we can already drop support for these legacy objects, but apparently it still needs to deserialize some of these instances! Good that we didn’t just ask around or did some code searches and just deleted it, it could’ve been quite the issue!

As you can see, doing this analysis can help you get a clearer picture of what code is still used in your projects. It’s important to note, however, that this does not mean we can now delete all code without coverage. Some logic might be used in seasonal cases, demos, or emergencies. A careful look is still desired, but this helps us get a little more confident in what we may delete.

To see whether this is the case, we usually perform code searches to look at the age and commits for surrounding code. We identify whether code is exceptionally new — maybe this is a feature in development, or old and forgotten. Finding connected API endpoints and their documentation might also help to get an understanding why this code is around. We also search our ticket and communication systems for any references and of course simply ask around.

What about performance?

Every time we introduce new tooling in production environments, we should understand its effect on application performance. This holds especially when instrumenting our code as this can add quite some overhead in instructions to execute. To understand its performance impact, we first ran it on staging environments to find any immediate problems with resource usage. This allowed us to tweak the settings accordingly.

To determine the performance in production, we kept an eye out on the average duration of the request. We selected two 24 hour periods, covering different loads for this application. One period with running JaCoCo, and one without.

From this, we observed an average overhead of 0.03%. As that is such a small overhead in the context of Picnic, we found this an acceptable price to pay for the insights we gain.

What is next?

With our codebase in a better place, we will continue to periodically scan it for code we can delete. However, this does not stop us from thinking what more we can do with JaCoCo.

In a future blog post we will walk you through how we’ve modified JaCoCo to not only tell whether code has been covered, but also count how often it has been executed! Keep an eye out on our feed, to check it out.

Lastly, I want to express my gratitude to Stephan Schroevers for driving this initiative and his guidance, allowing me pursue these forms of analyses.

¹ Carlos Alexandro Becker — Measuring production code coverage with JaCoCo, 20 March 2017, link.
² Markus Harrer — Feststelltaste — Visualizing Production Coverage with JaCoCo, Pandas and D3, 16 July 2017, link.