Before our recent improvements to the Maven build process, our developers had mixed experiences with slow build times. For some, it was an opportunity to engage in light-hearted distractions, like the famous sword fighting in the ‘my code’s compiling’ XKCD cartoon. However, for most hardworking developers, it led to unproductive context switching. This constant waiting disrupted their workflow, making it challenging to maintain focus and productivity. While a few enjoyed the downtime, the majority found it difficult to keep a steady workflow with these interruptions.
The solution to this problem was to speed up our Maven builds and reduce waiting time. We conducted an investigation to identify possible improvements to our setup. In the end, we managed to reduce the build time from 13 minutes to an astonishing 1 minute in the best-case scenario! Below, we will explain our journey in detail.
Finding the right solutions
We began by gathering all possible ways to speed up Maven builds. This included looking into possible Maven improvements and reviewing ideas and suggestions from other developers over time. By considering their experiences and insights, we made sure we explored every option. Then, Developer platform engineers worked in parallel to create proof-of-concept implementations for each solution to test their effectiveness, strengths, and weaknesses. We measured local and CI build times against the same set of repositories to ensure that the results were reliable and comparable. After a divide-and-conquer strategy of each team member evaluating a solution, we presented the findings. Here we weighed pros and cons and made a final judgement on whether to adopt the solution.
We looked into the following solutions:
- Maven Daemon
- Maven Partial Builds Extension
- Maven Build Cache extension
- Gradle’s Develocity Maven Build Cache
The two most promising solutions turned out to be Maven Daemon and Maven Build Cache Extension. These solutions showed significant potential in reducing build times and improving overall efficiency while keeping costs down. Let’s dive into these in more detail.
Maven Daemon 😈
Maven Daemon is a tool designed to improve the performance of Maven builds by running a long-lived background process (daemon). It reduces the startup time for Maven commands by keeping an instance of Maven running in the background, allowing the JVM to be reused, and ready to execute build commands without the need for repeated startup overhead.
How to use Maven Daemon
Any regular mvn invocation can be replaced directly with mvnd. Upon first execution, a daemon will be started, which will be used for subsequent (faster) builds.
There is an output of the first run of `mvnd clean install`, when there is no running daemon at the moment:
$ mvnd clean install
Starting new daemon 15f66a0c (subsequent builds will be faster)...
And this is the output of the subsequent run:
$ mvnd clean install
Connected to daemon 15f66a0c, scanning tor projects...
It is important to note that mvnd runs the modules in parallel, and this is taken into account in the benchmark below by also running the mvn command with parallelism.
Benchmarks
Benchmarks conducted on four different repositories revealed that a hot mvnd execution, where the daemon is already running, can accelerate builds by up to 50% of builds that do not run tests or static analysis as this is the most common use-case among developers in our organization.
Local vs CI server
Using Maven Daemon significantly sped up our local builds, making our engineers’ daily iterations faster and more efficient. However, this solution wasn’t suitable for our CI environment with TeamCity. Since we don’t reuse TeamCity, there was no benefit to starting a daemon process in this context.
In response to this limitation, we will be exploring the possibility of reusing agents and investigating potential improvements in a different initiative to better optimize our CI environment.
Adoption
To ensure every Java team at Picnic could benefit from the increased speed of local builds using Maven Daemon, we created a comprehensive knowledge article outlining the steps and best practices. By sharing this detailed knowledge article, we ensured that all developers in our organization could easily adopt and benefit from using Maven Daemon, leading to faster build times and improved productivity.
Currently, we don’t have visibility in the adoption rate here, but we are looking into using OpenTelemetry Maven extensions to get more insights into this to better track and advocate
Maven Build Cache
One of the best ways to speed up builds is to do less. And the best way to do less is to eliminate duplicate or unnecessary work. Why should we want to rebuild and retest all modules in our project if I change a single test, or a small if-statement in a module no other module depends on? We can use a build cache to only execute Maven goals for modules that changed and modules that depend on these changes.
How does it work
The Maven Build Cache extension enhances the efficiency of large Maven projects by implementing the following set of features:
- Incremental Builds: Only the modified parts of the project graph are rebuilt.
- Subtree Support: Allows isolated builds of specific parts in multi-module projects.
- Version Normalization: Facilitates version-agnostic caching.
- Project State Restoration: Avoids repeating expensive tasks like code generation by restoring previous states from the cache.
Local Build Cache
When working locally, the Maven Build Cache stores the build output artifacts in a designated local cache directory. Here’s a step-by-step breakdown of how it operates:
- Initial Build:
- During the first build, Maven compiles the entire project as usual.
- For each module, the build cache extension computes a unique hash based on the module’s inputs (source files, Maven configuration, dependencies, etc.).
- The build outputs (e.g., compiled classes, packaged artifacts) are stored in the local cache directory with the computed key. - Subsequent Builds:
- Before building a module, Maven checks the local cache directory to see if there is a cached result for the module’s hash.
- If a match is found, the cached outputs are restored, and Maven goals executed when the cache was constructed are now skipped.
- If no match is found, Maven rebuilds the module, and the new outputs are cached for future use. - Cache Management:
- The local cache can be configured to limit its size and manage the retention of cached artifacts.
- Policies can be defined to periodically clean up old or infrequently used cache entries, ensuring the cache directory remains manageable. - Key Generation:
- The unique key for each module is generated by hashing the module’s inputs. This includes source code files, project model (POM file), plugin configurations and parameters, and dependency versions and configurations.
- By including all relevant inputs in the key, the build cache ensures that any change in the inputs results in a new key, prompting a rebuild.
The build cache uses hashing to generate unique keys for storing and restoring build results. Correctness is maximized by including all relevant files and functional plugin parameters, while reuse is enhanced by filtering non-essential files and minimizing controlled plugin parameters. Configuration involves balancing correctness and performance through an XML file, buildinfo.xml.
Usage in CI
In our CI process, we also leverage the Maven Build Cache solution to accelerate our CI builds. However, for CI, we employ a remote cache stored in an artifactory instead of using a local directory. In our case, cached artifacts are stored in Nexus. This approach has markedly increased the speed of our Java builds. Consequently, our builds finish sooner and build queues are shorter.
However, in certain situations, users need to perform a full build of their changes. To accommodate this, we introduced a TeamCity parameter called “Use Maven build cache” with the following options:
true: the cache is usedfalse: the cache is not useddefault: the repository-level setting is applied
It is good to note that we currently employ the cache only for builds that do not create production artifacts, as we wish to have these as ‘pure’ as possible. As such, we disable the cache for builds on default branches supporting continuous deployments, and in release builds.
Benchmarks
In the best-case scenario, the execution time of CI builds dropped by ~ 90% when execution of every module was skipped, this is how it was shown in the build logs:
[INFO] Skipping plugin execution (cached): install:install
Instead of 10 minutes 4 seconds, the build that was fully cached took only 1 minute 22 seconds. 😲
Of course, this is quite a rare case to run a build that has every module skipped. The average build time improvement across all Java repositories in Picnic is shown in the picture below. It is noticeable that build time dropped by 62% in general.
For some repositories, the build time improvement was more noticeable than for others. Projects with many modules benefit more than projects with only a few modules. This can be seen in the chart below showing P75 build times. Highly modular projects see their build times drop from 10 to 5 minutes. Projects with a few modules only benefit little from this improvement.
Conclusion
- Dramatic Build Time Reduction:
- Build times dropped from 12 minutes to 8 minutes in general. This significant improvement eliminates long waits, allowing developers to maintain focus and productivity. - Enhanced Developer Experience:
- Faster build times, especially for local builds, lead to fewer interruptions, enabling developers to stay in the zone and work more efficiently. This improvement has contributed to higher job satisfaction and overall happiness among our teams. - Increased Efficiency in CI Pipelines:
- Using the Maven Build Cache in our CI environment has sped up CI builds, reduced queue times, and made TeamCity agents available more quickly.
These improvements have significantly boosted developer morale and productivity. Reduced build times mean less context switching, quicker feedback, and a more enjoyable workflow.
Developers feel more empowered and efficient, resulting in higher-quality code and faster feature implementation.
In summary, the integration of the Maven Build Cache extension and Maven Daemon has transformed our build process, making it faster and more efficient. We are excited to continue leveraging these tools to maintain and further improve our build processes, ensuring our developers have the best possible experience.
In collaboration with Pieter Dirk Soels.
