Picnic 10 years: 2024 — Building a modular warehouse

Picnic 10 years: 2024 — Building a modular warehouse

In this edition of the Picnic 10 years blog we go back to 2024. 2024 was by all accounts an exciting year: I became a father for the second time, and Picnic began seriously scaling up its warehousing proposition. In this post, we take a look at how we tried to scale our warehouse systems landscape. In a previous post we already dove into our journey of developing the software for our first automated warehouse in 2020. In there, we already hinted that we had the intention to split up the monolith we built to control that warehouse.

Fast-forward 4 years and this is not only an intention, but very much a reality. This post describes what our warehouse landscape was in 2024, what we mean with a modular warehouse and some of the challenges that come along with it.

The evolution of our warehouse landscape

In 2024, we are not just dipping our toes into automation anymore, but rather it has become a significant part of Picnic’s fulfillment process. The first automated warehouse in Utrecht is scaling up from 70% to 100% of its capacity and we’ve finished development for our second automated warehouse in Dordrecht, set to launch in March 2024. We’re hard at work to be able to launch our 3rd and largest automated warehouse in Oberhausen in 2025. I got used to referring to these automated warehouses by their city names, so I’ll do the same in this post. Next to that, we also operate around 20 manual warehouses.

Picnic is a company that is very eager to learn from previous experiences, experiment with new concepts and do better on every iteration, and when it comes to warehouses this is no exception. Our Utrecht warehouse was a great learning experience from which we took a few important lessons:

1. It is capable of reaching much higher productivity levels than a manual warehouse if we have enough volume going through it.
2. It is expensive to build, not only from the tech side but also in terms of hardware.

These learnings together made us essentially conclude that this concept is great in very densely populated areas such as the Randstad region in the Netherlands or the Rhine-Ruhr region in Germany where there is enough demand to justify its costs and it can show its true potential. However, not all regions are as densely populated, so we wanted to explore other concepts that use automation to outperform manual operations. With that in mind, we designed and built Dordrecht in 2023 that we internally called a ‘hybrid warehouse’. In this warehouse, we do some picking in the ‘traditional’, manual way and some picking in an automated way and exclude the automation that made Utrecht so expensive. To be specific, the Goods To Person (GTP) picking, where stock comes from an Automated Storage and Retrieval System (ASRS) was not included.

In 2024 our focus was mainly on our third automated warehouse, to be launched in Oberhausen. This is in the Rhine-Ruhr region, so we decided to go for the fully automated variant again, like we have in Utrecht. However, next to high-level learnings about which concepts work where, of course we also had many smaller learnings from our site in Utrecht that we tried to adapt in Oberhausen, such as:

• Our order completion process, which is the process of adding ice packs to crates and putting a lid on top, did not work well in Utrecht and was causing bottlenecks. For Oberhausen, this area was redesigned completely.
• The location where we store crates after they are completely picked was a shared storage with other types of crates, which turns out to be hard to control. For Oberhausen, we designed a storage system dedicated to these types of crates.
• Both sites support GTP picking and zone picking and for Utrecht there are 2 buffers for when crates need to travel from 1 type of picking to the other: 1 for crates that need to go from the GTP area to the zone picking area and 1 for crates that need to do the opposite. In Oberhausen, there is just a single, combined buffer for both directions.
• And many more small differences

Next to the automated warehouses, we also have manual warehouses which come in 2 flavours:

1. A Fulfilment Center (FC), in which we pick orders placed by our customers into crates.
2. A Distribution Center (DC), in which we pick pallets and boxes of products for our FCs.

All in all, Picnic has a quite diverse set of warehouses and a very realistic outlook that this diversity will only grow with the business.

Modularisation

When developing the software for the first automated warehouse we made the conscious decision to develop a monolith to control this site. This decision has roughly led to the state that for automated warehouses we had 2 systems controlling it:

1. A Warehouse Control System (WCS) to control everything that happens on the conveyor system, such as storing crates in our storage system and sending them to a picking station.
2. A Warehouse Management System (WMS) that controls everything that happens outside of the conveyor system, such as receiving products on the dock and doing manual picking.

For our second automated warehouse in Dordrecht we were now faced with an important decision to make: do we continue expanding these systems along these boundaries or do we take a different approach? I guess it won’t be a surprise that we chose to take a different approach. The reasons to do so were both technical and business driven. Technically it was very challenging to expand these 2 systems and keep them stable. Both systems had multiple teams and 10+ developers working on it, and a very broad scope. From a business perspective there was also some merit to taking a different approach. When thinking about what Dordrecht should look like, we thought in terms of capabilities (or modules) that it should have, for example:

• The concept of GTP picking combined with an ASRS is very expensive, so we don’t want to have that here.
• The concept of conveyor-based zone-picking that we have in Utrecht works really well and is less expensive, so we want that to be the main type of picking.
• Our frozen assortment is relatively small compared to our chilled and ambient assortment, so those we want to pick like we do in our manual warehouses and simply put the picked crates on the conveyor system afterwards.
• We partnered with a new supplier of frame-loading robots that seemed to outperform the ones we have in Utrecht, so we want to try that out.

Essentially we were designing a warehouse as a set of (interconnected) modules, where we take concepts that we have seen working in other warehouses, connect them together and profit. If we were able to configure a warehouse as a set of re-usable modules then the only things we would still need to develop are the modules that are truly new, such as the new frame-loading robots. This was a very compelling promise and we decided that it was a better approach to split off and build these modules rather than grow our monoliths further. And so the idea of modular architecture was born.

Modular architecture

Our modular architecture from a high level consists of 2 types of systems: execution systems that act as modules and supervisory systems to connect modules together.

Execution systems

The main components of our modular architecture are, well, the modules. We quite quickly settled on the idea that a module should map to a specific (conveyor) area of a warehouse and only knows how to do the process of that area. Taking the example of a zone picking module, this module knows about a zone pick loop, the stations attached to this loop and the entrance and exit of this loop, like shown below:

A warehouse is of course more than 1 process and in reality would look something like this:

All processes are nicely encapsulated in a module and map to a physical area in the warehouse where this process happens. Add in some supporting modules that don’t reflect a warehouse process but are needed to support the overall flow, such as the loop in the example above, and you have the first part of our modular architecture. These set of modules we call execution systems, since they are responsible for executing something.

Supervisory systems

One key aspect of this architecture is that the execution systems only need to know about their own area and how to execute the process of that area. In the zone-pick example, it means that the execution system is responsible for sending totes into stations, instructing the operator at a station what to do, sending totes from stations back onto the loop, etc.

However, given only the local knowledge there are a lot of things the execution layer is not responsible for, but is needed to make a warehouse run, such as:

• Which crates should be picked first?
• When all picks for a certain crate are done, to which area does the crate go next?
• In case the same product is present in multiple zone pick areas, what do we pick where?
• If there are multiple routes to go from one area to another, which route do we take?

This is where the supervisory systems come into play. Roughly the role of these systems is to ensure the execution systems play nicely together. These systems are developed not with the responsibility for a certain area, but to solve a certain problem. We currently have supervisory systems such as:

• A ‘releasing’ service, responsible for deciding what area’s a crate needs to visit, in what sequence and which crate should be picked first.
• A routing service, responsible for finding the best path for a crate when it needs to go from module A to module B, instructing all modules the crate needs to travel through, to send that crate to a specific exit of that module.
• An empties management service, responsible for (fairly) distributing empty crates among execution systems that need them, such as picking systems.

Communication

These supervisory systems work much closer together than the execution systems. For example, the releasing service may decide that a crate needs to now go from a zone pick area to the dispatching area, which triggers routing to then find the best path to take. Therefore, we currently adhere to the following communication principles:

• Systems from the execution layer communicate with the supervisory layer and vice versa. This is the main form of communication and consists mostly of instructions given by the supervisory layer and reporting on progress by the execution layer.
• Systems from the supervisory layer can communicate with each other. We don’t put many restrictions on this type of communication, as we’re still finding out what good boundaries are within this layer.
• Systems from the execution layer can not communicate with each other.

We are very strict on the last point. For example, the zone-pick system does not know that its exit connects to a loop, it just knows that it is the exit. When a crate needs to leave, our routing service instructs the zone-pick system to bring it to the exit, and only the routing service knows that that exit actually connects to a loop and instructs the loop that a crate will arrive at one of its entrances at some point in the future.

New warehouse = new architecture

Back to 2024. We just completed development of our second automated warehouse and trialled this idea of a modular architecture. It did its first orders in March 2024 and officially opened in May. We proved that we could operate a warehouse in this new architecture and a new automated warehouse in Oberhausen is expected to launch in 2025. This warehouse is our biggest site to date and, inspired by the success of our second automated warehouse, we wanted to continue our modularisation journey.

However, this was a very different beast than the warehouse where we experimented with modularisation. Our trial looked a bit like my example above and we really learned a lot there, but there were a lot of new challenges with the warehouse in Oberhausen:

• There are about 2.5x as many execution systems, and a lot more connections between them.
• All these connections mean that there are actually quite a few options when crates need to go from A to B. We built routing with the idea that it would do path optimisation, but for the warehouse in Dordrecht the reality was that there is almost always just 1 choice.
• Oberhausen has GTP picking and stores crates with stock in an ASRS. This means that, among other complexities, we also need to set up a supervisory layer for managing crates with stock. In Dordrecht, we only handle crates for customers.

On the plus side, Oberhausen is similar to our warehouse in Utrecht and the challenges above have been solved there. However, Utrecht is controlled by our monolith, so the main thing to figure out is how to solve these challenges in our modular architecture.

We were on a tight timeline, our hardware supplier was well on track to deliver the Oberhausen site to us at the end of 2024, whether our software was ready for it or not. Given the challenges and the timeline, we decided to develop the software for Oberhausen in our modular architecture, but not migrate our warehouse in Utrecht to the modular architecture (yet).

Challenges

Our biggest tech project in 2024 was building the software for Oberhausen. There were a lot of challenges during the development process, from under-specified requirements (‘it should work in the same way as Utrecht’ turns out not to be a great specification) to underestimating how much effort carving smaller modules out of a monolith actually is, to dealing with a burst sprinkler system that short-circuited a large part of the conveyor network, preventing us from testing anything on-site for a while.

With a great team of really smart and dedicated people we were able to overcome all of these and I’m very proud to now see this site alive and kicking, showcasing that modularisation really works!

While being focused on Oberhausen we largely left our other warehouses untouched where we could. Utrecht still mostly runs on the monolithic system with a large set of optimisations made along the way. This means that we have incurred some tech debt along the way, where we now have different architectures powering different sites. As we move forward, having this modular architecture for all our warehouses will be our next big challenge.

Conclusion

To an extent, this story is to advocate for a microservice architecture, though in practice some of these services are definitely not ‘micro’. We’ve found that being too strict on how small you want to keep these services is counterproductive and finding the right boundaries is the hardest part of the journey.

I truly believe that the setup of execution and supervisory systems allows us to scale to tens or hundreds of (automated) warehouses. The promise of configuring a few modules and a new site starts to work is very real and with the great talent that we have at Picnic, the possibilities for new and exciting concepts are endless.

Interested in Joining Picnic?

Check out our Tech vacancies here!

Picnic 10 years: 2024 — Building a modular warehouse was originally published in Picnic Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.