Authored by Thijs Sluijter and Majid Hajiehidari.
At Picnic, we really care about what our customers eat. We believe that everyone should enjoy delicious and healthy meals with their family every day of the week. That said, we understand that juggling the demands of a modern career and family life can leave little time or energy for planning meals, let alone cooking them. Therefore, one of our core missions is to make it as easy as possible for our customers to find the meals they like, with the ingredients, tastes and cooking methods that suit their day-to-day needs. To do so, our culinary colleagues have created a large catalogue of thousands of recipes. However, with great assortment comes great responsibility (to show the right items to the right customers), and that’s where our recipe recommender comes to the rescue.
In a previous blog post, we introduced our first attempt at a recipe recommender. We started simple with a solid collaborative filtering setup; however, in the “Looking Ahead” section of that very blog, we already mentioned the Two-Tower model as the next step. We built it shortly after and used it in production for the last two years.
But life goes on — and Picnic with it. In those 2 and a half years, we continuously developed our dinner buying proposition and redesigned our “Meals Page”, the section in our app that showcases our recipes and fresh meals, earlier this year.
New UI, New Recommendation Paradigm
The fundamental change between the old Meals Page (Meal Planner) and the new Meals Page (Meals 2.0) is an intent shift — instead of focusing on the “Weekly Menu,” we moved to a more inspirational approach, showcasing a wide view of our recipe assortment. Different sections cater to different needs, including a customer’s previously bought recipes, highlights for new recipes, and marketing campaigns such as our collaboration with Ottolenghi. The new page also offers a much better experience for new and onboarding customers, as they immediately see the wide range that we offer. This last point is an important business focus, and catering to new customers was an important goal in developing our new model.
Of course, a complete UI redesign also forced us to rethink the way we do recipe recommendations at Picnic.
Previously, the goal was to filter our assortment of thousands down to a small set of recommendations (three for each day of the week). The new page required us to do recommendations on many different sections, each containing just a small set of candidates. This fundamentally changed our recommendation problem from retrieval to ranking, and that change warranted a rethink of our modelling approach. The most important insight is this: with fewer items to score in each set, we can apply a more powerful model that requires more computation per item.
However, before developing a complex new model, we had to evaluate the current setup. We believe in starting simple, and that meant finding out how well our Two-Tower model performed in the new Meals Page. To do so, we introduced a simple popularity baseline and ran an A/B test against the Two-Tower model in all markets. We found that in the Netherlands (our largest meals market), the Two-Tower model and the baseline performed on par, while in Germany and France the baseline even outperformed the Two-Tower model! These results might not be too surprising to many recommender systems veterans, as popularity baselines can be notoriously hard to beat. Nevertheless, we gained two clear conclusions:
- As expected, the Two-Tower model was not the right approach for this new UI.
- The popularity baseline could serve as an efficient benchmark during offline development.
So to summarise: (1) our UI changed from a narrow selection to a broader, more exploration-focused design, (2) our modelling approach changed from retrieval to ranking with a special focus on new customers, and (3) our Two-Tower model was not good enough, so we had to build a new model to beat the baseline. Now let’s dive into how we did just that.
Retrieval and Ranking Models
In the context of recommendation models, there are two main types of models:
- Retrieval models: Optimised to efficiently retrieve potentially relevant candidates among a big set of candidates, for instance to narrow down 10 million items to 10 thousand potentially relevant ones.
- Ranking models: Designed to operate on top of retrieval models (or smaller candidate sets). After the retrieval model narrows down the candidates, the ranking model takes the retrieved candidates and ranks them for the customer. These models are focused on recommendation performance over a smaller set of candidates, and they’re not as scalable as retrieval models.
The move to Meals 2.0 made our Two-Tower retrieval model insufficient, as it struggled to outperform a simple popularity baseline. To resolve this, we made a key change: we replaced the Two-Tower model with a Deep & Cross Network (DCN) ranking architecture, enabling more direct modelling of customer-recipe interactions.
We also redesigned the training data and objective. The Two-Tower model could train from purchases alone because its retrieval loss supplied the comparison examples implicitly. For a batch of purchased customer-recipe pairs, it scores every customer against every recipe in the batch: softmax cross-entropy rewards the observed pair and treats recipes from the other pairs as in-batch negatives.
Although this teaches the model to rank purchased recipes above other recipes within a training batch, it is more precisely a contrastive retrieval loss than a conventional ranking loss. This setup needs no explicit negative labels, but its negatives are determined by batch composition and are not necessarily recipes the customer chose not to purchase.
The DCN model produces one score for each customer-recipe example and optimises binary cross-entropy (BCE), so every example needs a positive or negative label. We therefore made negative sampling an explicit data-design choice: an ordered recipe is positive. In contrast, a viewed-but-not-ordered recipe is negative only on a day when the same customer ordered at least one other recipe. This remains implicit feedback rather than proof that the customer disliked the recipe. Still, it ties negatives to observed exposure and purchase intent instead of deriving them from unrelated items in a training batch.
Feature Engineering with Representation Learning
One practical advantage of deep learning is that it can shift some feature engineering from manual design into the model itself, though the extent of this varies by use case. For example, a classical regression model might need high-order polynomials to capture non-linearities, or feature crosses to capture interactions between features. In this example, if you replace your linear model with a deep-enough Multi-Layer Perceptron (MLP) model, you don’t need high-order polynomial features or explicit feature crosses.
Looking at this promise of deep learning from another perspective, there’s the famous essay from Richard Sutton, the father of Reinforcement Learning, called The Bitter Lesson. The essay states that scaling up the data and model size can outperform expert knowledge in a field of expertise, and frames this finding as the bitter lesson. Inspired by Sutton’s thoughts, we decided to experiment and figure out what this means in our context.
For the recipe recommendation model, we’re mostly interested in customer features such that we can tailor our recommendations to every customer’s personal preferences. It can be expensive to compute engineered features and store them. On top of that, the marginal effort of adding another customer feature to the model stays constant over time, so it’s hard to scale the approach of hand-engineering them. Instead of precomputing features like “number of meat products bought in the last 8 weeks,” we fed sequences of historical purchases to the model: We let the embedding/pooling layer produce a trainable customer representation based on purchase histories (Representation Learning). By using this representation learning method, we were able to replace all previous hand-engineered customer features with just two embedding features: recipe purchase history and article purchase history. Not only did this significantly reduce the feature-engineering effort, but it also produced much better model performance.
To create the purchase history embeddings, we first embed each historical purchase of a customer into a trainable embedding space, then pool all the purchase vectors into one history vector. For the pooling strategy, we tried both mean pooling and attention pooling; in our case, attention pooling didn’t improve performance. As mean pooling is simpler and doesn’t add parameters to the model, we settled on that strategy. After mean pooling, the vector can be fed to the DCN model as a feature, creating our final input.
As a result, we simplified our training pipeline significantly, especially in terms of data pre-processing. This simplification helped us increase model and dataset size with great ease, and shifted our focus to other important aspects of recommendation modelling, such as iterating on negative mining strategies. For the feature engineering itself, our thinking moved from crafting manual aggregations to iterating on representation design, sampling, pooling, and regularisation.
Autoresearch
Over the development phase, we utilised agents quite a lot. In some specific use-cases, we saw 10x engineering productivity. One area where agents accelerated our development was using “Autoresearch” for model tuning. Autoresearch was popularised by the prominent ML researcher Andrej Karpathy. We took inspiration from Karpathy’s approach and tailored it to our requirements. Here’s the setup: we put an agent in a loop where, in each iteration, the agent runs the whole training pipeline with a set of hyperparameters it decides on. It then observes the results, logs the experiment, and adds ideas for the next iteration to the backlog.
There are a couple of key considerations to be aware of when using this process:
- What the agent could change: learning rate, hidden sizes, dropout, L2-regularisation, history length, batch size, embedding dimensions.
- What it could not change: objective, data split, evaluation protocol, production constraints.
- What made it safe: fixed validation metrics, experiment logging, reproducible configs.
Picnic operates in 3 countries, each with different tastes and palates. For instance, this includes specific culinary habits like the German preference for Abendbrot compared to the French tradition of warm, multi-course lunches (déjeuner). When we develop something that works for the German market, there’s no guarantee that the same thing works in France. So we A/B test and roll out separately for each market. But it’s also important to tune the model hyperparameters (network size, embedding dimensions, etc.) based on market-specific data. For example, Picnic started in France much later than in the Netherlands, and there’s less data to train on; hence, it is natural to have a smaller model in France. Autoresearch helped us efficiently and quickly tune the model for each market, meaning we were able to iterate offline rapidly before going online. This enabled us to launch 3 successful A/B-tests for 3 markets in less than 2 months.
Not only did autoresearch speed up per-market tuning, but it also solved real feature engineering challenges. For example, purchase history as a feature was doing very well for train loss but not so great for validation loss. Meaning we had an overfitting problem after ditching hand-engineered features and adopting purchase history. Autoresearch helped us iterate on a set of hyperparameters quickly and come up with better regularisation that makes purchase history work for the validation split.
After successfully incorporating Autoresearch into our development cycle, we realised that the dynamics of ML model development had changed for us. Agents made tasks such as hyperparameter tuning and minor architectural decisions much easier, allowing us to run far more structured experiments than before. The bottleneck shifted from manually choosing experiments to designing a reliable and performant experiment loop.
Sweet Results
Utilising this new setup to run many automated experiments offline, we quickly saw promising results. Our DCN model was outperforming the popularity baseline by as much as 3x on hitrate@15. However, we still needed to verify the real-world performance of the DCN model. So we did an A/B test measuring two main metrics:
- customer conversion rate: out of all the customers who visited the page, how many bought at least one recipe?
- recipe buying share: out of all customers that placed an order, how many included a recipe?
Buying share is the main KPI we are trying to influence, but it’s usually quite hard to move. Conversion therefore serves as a good proxy and additionally tells you something about page quality and customer convenience.
After two weeks of testing, we saw some pretty amazing results. Not only did we improve overall customer conversion by more than 8% (relative), we also improved our difficult-to-move north star KPI by 1.5% (absolute)! That’s much more than a marginal improvement; in terms of conversion, the new model was as large an improvement as the launch of the new page by itself! Moreover, when breaking down the performance to different customer segments, we saw that the DCN is especially effective for new and onboarding customers. This shows that including article purchase history as a feature allows the model to recommend recipes to customers that have never bought recipes with us before! As mentioned earlier, this is especially important to help onboard more customers to buy meals with Picnic.
Now that we’d confirmed online performance in NL, we were confident that offline performance would correlate with online improvements, and thus rolled out A/B tests in Germany and France. Those tests both showed similar results, proving the power of our setup for international rollouts.
One final result is that, as we rebuilt our recommendation system, the net code addition was only a few hundred LoC. That does not seem too bad, especially when considering we implemented DCN fully custom in PyTorch!
Looking Back, Looking Ahead
At Picnic, we definitely like to move fast, and we are not afraid of taking risks. The way we completely redesigned our recipe recommendation setup shows that. In general, we believe that it is a very healthy practice to take a step back and take a critical look at your projects every so often. A complete redesign might be less costly than endlessly maintaining old systems, and opportunities only tend to present themselves when you go look for them.
However, moving fast does not always mean you have to break things. We didn’t hesitate to redesign our pipeline, but we used good engineering principles to inform the decision. Our design had a solid theoretical foundation: ranking smaller sets allows for more complex modelling. Benchmarking against a popularity baseline helped us evaluate our new model. Our experimentation approach shows how we can move fast with a solid process: Applying autoresearch and the bitter lesson allowed us to iterate with incredible speed. At the same time, a rigorous offline-online evaluation pipeline meant we were getting meaningful results.
So what’s next, you ask? We are currently rolling out the model all over our app. At Picnic, we believe that when you build something good, you should enjoy it and so should our customers. But of course, Picnic being Picnic, we already have the next projects lined up, and we have many ideas on how to make our DCN model even better.
The most obvious thing would be to add more features. In its current state, the model knows a lot about the customer and their purchase history, but it doesn’t know anything about the context: what’s the weather like? Is it summer, winter, Easter or Christmas? So adding some contextual features is likely low-hanging fruit.
After those simple additions, we can look at more advanced methods of enriching the input data. For example, the current embeddings for articles and recipes are learned by the model from scratch, capturing only the interaction patterns the model sees. However, we have much more data on the articles and recipes we sell: brands, sizes, prices, descriptions, cooking instructions, images, you name it. This data can be fed to an embedding model to obtain rich semantic embeddings capturing many aspects of our article and recipe assortment. We already use this type of semantic embedding in our search pipelines. So a logical next step would be to integrate them into our recommender model.
Additionally, we can iterate on how we present the data to the model. Our current negative mining strategy is quite naive, so that warrants further exploration. Another route to explore is the pooling mechanism in our purchase histories; it seems obvious that a purchase from 6 months ago is less relevant now than what the customer bought last week, but our current approach does not consider that. Replacing the mean-pooling with an attention mechanism and adding positional embeddings could be an interesting way to capture that.
Then there is real-time data in our app. We might want to utilise live interaction signals as the customer is interacting with our Picnic app. For example, adding a pasta recipe to your favourites might mean we should recommend more pasta, while adding a soup to your basket might mean you’ve seen enough soup.
And the list goes on. At Picnic, we tend to have the luxury of having more potential projects than people. If that sounds like you and you found this blog interesting, please consider joining us to help build some of those projects. Check out our open positions below!
The Bitter Lesson, Sweet Results: How We Rebuilt Picnic’s Recipe Recommender was originally published in Picnic Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.
Eric Smith
