Picnic is an online grocery store that delivers with its own fleet. Every driver we schedule and every pallet we order starts with one forecast: how many customer deliveries we expect for a future delivery date. We launch exciting new initiatives constantly, which our forecast accuracy used to not be able to keep up with: testing a single idea to adjust to new initiatives or improve the model used to take us weeks. Specifically, our previous model was so complex that it made backtesting very hard, resulting in us having to rely on long shadow deployments just to evaluate a single change. In this blog post, we describe how we rebuilt the forecast so that we could test new ideas within hours instead.
The forecast we are discussing here is called the delivery forecast, and is critical to Picnicâs operations. It predicts the number of customer deliveries for future delivery dates. Our hubs, where our delivery vehicles depart from, use it to schedule drivers. Our fulfilment centers, the warehouses where we pick customer orders, use it to schedule employees. The forecast is produced by a Temporal Fusion Transformer (TFT), an attention-based time-series model that combines static variables (which hub), future variables (weekday, holidays, hub sizes) and past observed values (demand in the past). The forecast is made up to 14 days ahead.
In the past, if we wanted to add a new feature to the model, we would create a shadow deployment: a second model running alongside production, which we could monitor without affecting operations. If it beat production over the following weeks, we would start using it in production. Of course, a few weeks is very little test data, and every idea we had would cost us weeks to test. In order to be able to fully backtest and better understand our model, we first had to simplify it. We did so by deliberately removing our single most complex, but also most predictive, feature, and rebuilding its signal ourselves.
Modelling the Shape of Demand
That feature we removed was the confirmed deliveries: the orders customers have already placed for a future delivery date. As described in a previous blog post, this feature turned our model into a black box. It made it harder to implement backtesting, made reasoning about the forecast complicated, and responded too strongly to operational constraints. What do we mean by operational constraints? Well, whenever we reach operational capacity for a future delivery date, we close that delivery moment to customers. The confirmed deliveries then stall, and the old forecast would drop as well. This was problematic, as the business needs a forecast of unconstrained demand: what our demand would look like if we did not have any operational constraints.
Instead of using confirmed deliveries as a feature, we set out to separately model how deliveries build up over time. To do so, we inspected what this build up generally looks like: we plotted the cumulative confirmed deliveries against the hours remaining until the ordering cutoff (the moment at which customers can no longer place orders for a given delivery moment). Figure 1 shows such a curve. Then, we divide every point by the final delivery count. The result is the share of deliveries confirmed so far, which we will call the α curve. If 40% of deliveries have been confirmed, the α curve is at 0.4.
We observed that the shape of α curves for the same weekday, daypart (morning or afternoon) and hub was quite predictable. For example, the α curves for the Monday afternoon dayparts in hub Haarlem had a very consistent shape throughout the year. If we can predict the shape of the α curve for future dayparts, then we can use it to estimate at each point in time what fraction of total demand the confirmed deliveries represent. We can then use this estimation to correct the prediction of our TFT model, which means we no longer need the confirmed deliveries feature.
We call our prediction of the shape of the α curve the modelled α curve. To model the α curve, we look at historical α curves for the same hub, weekday and daypart. We then take the median of these historical curves. The median represents what a typical α curve looks like, and is not affected by a few outlier curves. When we close early, for example, the α curve goes flat. This would drag down an average, but not a median. Figure 2 shows an example of a median curve.
As mentioned before, the business is looking for a forecast of unconstrained demand (what the demand would have been if we did not have any operational constraints). However, their definition of an unconstrained forecast is slightly different than what you may expect. The business wants the historical confirmed deliveries to stay as-is. If we currently have 100 confirmed deliveries for a future delivery date, and we were also closed for a part of the order window for that same delivery date, they want the confirmed deliveries to stay as they are. Only the remaining deliveries should be predicted as unconstrained.
As a solution, we define what we call a remainder model: We take our TFT prediction, strip out the share already confirmed according to the modelled α curve, and replace that share with whatâs actually confirmed. This results in the following equation:
This way we correct our forecast toward reality as orders come in, but only apply that correction to the âobservedâ part of our forecast (we are not simply extrapolating the modelled α curve). The remainder model correction for the example above is visualized in Figure 3. The confirmed deliveries can be inferred from the actual α. When the actual α is below the modelled α curve, the forecast is corrected downwards. When the actual α is above the modelled α curve, the forecast is corrected upwards.
Backtested over a full year, the remainder model performs on par with the old TFT with the confirmed demand feature (we will go more in depth on these backtesting results at the end of this blog). This shows that the modelled α curve captures the same predictive signal that confirmed deliveries did, but in a form we fully control, understand and can explain.
Two clear opportunities for improvement remained, and both come from the same source: our history is a record of what happened under operational constraints, not of what customers actually wanted. The first opportunity: the modelled α curves were still medians of constrained historical data, rather than true unconstrained curves. The second opportunity: we were training our TFT model on actuals rather than unconstrained targets.
Reconstructing Reality: The Unconstrainer
To get unconstrained curves and unconstrained targets, we need to know what the customer orders would have looked like without operational constraints. We never directly observe that, so we have to reconstruct/estimate it.
The problem has two parts. First: when we run out of capacity for a specific daypart and close it, we donât know what customers would have ordered during the time it was closed. In other words, we donât know what our lost demand is due to customers not being able to place orders. Second: when one daypart fills up, some customers spill over into adjacent dayparts, inflating demand there. For example, when our capacity is full for the Monday morning, customers might place an order for Monday afternoon instead. Ideally, we find a way to correct our observed demand for both closures and spillover. You can see an example of how closures and spillover affect the actual curve (grey line) in Figure 4.
This is where the modelled α curve becomes really powerful. Because it describes the expected shape of orders over time, we can use it to reason about what would have happened during closures and spillover windows. You can see the modelled α curve as the blue line in Figure 4.
For a daypart with closures and spillover from adjacent dayparts, we can model the observed demand (the constrained demand) at any point in time as a combination of two things: our daypartâs own modelled α curve during the time it was not closed, and the modelled α curves of any neighboring dayparts that closed and redirected their demand to our daypart. For example, the observed demand for Monday morning is a combination of the modelled α curve for Monday morning (excluding the periods in which Monday morning was closed) and the modelled α curves of Sunday afternoon (including only periods in which the Sunday afternoon was closed). We visualize both curves in Figure 5.
We let each neighboring daypart contribute its own modelled α curve, so that we can account for several neighbors at once. This model of the observed deliveries can be expressed using the following equation:
As you can see, we introduce two components we have not explained so far: the unconstrained demand and the spillover demand. We are interested in finding out that first component: what would the unconstrained demand have been? The spillover demand is also an unknown component, the rest of the equation is all known.
Since neither the unconstrained demand nor the size of each spillover contribution is directly observable, we have to solve for both if we want to uncover the unconstrained demand. We do so using Non-Negative Least Squares (NNLS). NNLS is a curve-fitting method: it picks the unconstrained demand and spillover demand numbers that best explain the observed orders, and never allows a negative number. The resulting unconstrained curve is shown in Figure 6Â below.
There are two ways in which we use our reconstructed unconstrained demand. The first way is that we will use this number as the training target for our TFT model, so it learns to predict what demand would have been without our operational constraints. The second way uses the set of unconstrained α curves we have uncovered. Taking the median of these unconstrained α curves gives us an unconstrained modelled α curve, which can be used by the remainder model to improve our short-term forecasts.
You may have noticed that the main input for the unconstrainer, the modelled α curve, is itself a product of the observed α curves. Therefore, it could still suffer from a bias due to operational constraints. In the future, we therefore might generate the modelled α curves from the unconstrained estimates iteratively, hoping that the curves eventually converge.
What Backtesting Revealed
We built our new TFT and remainder models in such a way that all their components could be historically reconstructed. Finally, backtesting became something we can do at scale. We can now evaluate every change against history, and we can even run multiple months of evaluation in parallel. A backtest replays history as if we were running in production, so every step only sees data that was available at that moment. For each month we want to backtest, we train a model on only the data available on the first day of that month. For each day in that month, we use that model to forecast with the data available as of that day. For each day we also rerun the remainder model at several simulated times of the day, using only the data available at that time, which mimics how the forecast updates as orders come in.
In the evaluation of our backtests we use two metrics, both scored against the unconstrained target. The WAPE (weighted absolute percentage error) is the sum of absolute errors divided by the sum of unconstrained deliveries. This describes the accuracy of our model. The WPE (weighted percentage error) calculates the sum of errors divided by the sum of unconstrained deliveries, and tells us whether we are structurally over- or under-forecasting.
We collect all evaluations of our backtests in extensive dashboards. In addition to backtest results, these dashboards also include predictions made by the production systems, including business overrides. This helps us build and maintain trust and alignment between business teams and our engineering team.
The first backtesting results surfaced something interesting: training the exact same model twice, with nothing changed, could produce noticeably different predictions. Across five identical retrains, WAPE varied significantly. Some of the âimprovementsâ we had been shipping based on shadow testing were inside that variation. A single backtest was not reliable enough to determine whether a model had improved.
Another interesting finding was that the more simple Time-Series Dense Encoder (TiDE) performed as well as the more complex (and expensive) TFT we had been using so far. Because of these two findings, we decided to train an ensemble of five TiDE models in parallel and average their forecasts. As TiDE is much cheaper than TFT, this resulted in significantly reduced training times of our models, and even faster backtesting.
The first results are shown in Figure 7, and show that the ensemble was not just more stable but also more accurate: it significantly reduced the WAPE on horizons beyond a week. This resulted in us equaling the accuracy of our old model with our much simpler TiDE and remainder model combination. As a next step, we tried training on the unconstrained rather than observed delivery counts. This took another significant percentage off the WAPE beyond a week.
Figure 8 below shows us what the WPE looked like for our experiments. The old model tends to increasingly underforecast as the horizon grows. The remainder model overshoots slightly instead, and the unconstrained targets pull it back down towards zero at long horizons.
Together, these results show that our new model is not only simpler and easier to control than the old one, but also outperforms it when trained on unconstrained targets on both accuracy and bias. In addition, being able to demonstrate the accuracy of our model with backtest greatly helps us to evaluate new ideas and convince our stakeholders of the value of our work.
What We Gained
We replaced one complicated feature with a small set of explainable components: the remainder model and its (unconstrained) curves, which we can compute cheaply, inspect individually, and improve piece by piece. This modular forecast allowed us to finally backtest our ideas and get from idea to evaluation within hours.
Along the way, we learned a number of lessons. First, when you build models and choose features, ask whether you can backtest them. A feature that is only available in a form you cannot rebuild historically can make a model more accurate but harder to understand and improve. Second, measure the noise of your own training before you trust an improvement. Lastly, investing in robust backtesting pipelines and evaluation dashboards can accelerate development and help you spot issues faster.
None of these improvements would have been realized without the business teams trusting us engineers. The case for the remainder model was control and backtestability, but it was not expected to result in accuracy improvements on its own. The trust paid off fast: redesigning the system made it possible to test ideas faster and see which ones actually helped.
Weâre not done! Next up is a backlog we can finally work through at pace: better unconstrainer models, testing our new unconstrained α curves, and many new features to explore. If you also like to get from idea to execution at high pace, check out our open positions.
Building a Modular Demand Forecast: How We Made Testing New Ideas a Breeze was originally published in Picnic Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.
Parinith Iyer
