15 / Machine learning / Prediction
Apprentice Chef
Two prediction studies on a meal-kit business, judged by how far each model's test score falls below its training score, and by whether the result survives a re-run.
01 / The question
Which customers spend more, and who signs up?
Apprentice Chef is a meal-kit subscription business. The dataset has 1,946 customers and 28 raw features. Part 1 predicts revenue (a regression), and Part 2 predicts who subscribes to the Halfway There cross-sell promotion (a classification).
02 / What the models do
Many learners, one weighted blend each.
Revenue. Revenue is log-transformed and the features standardised before a 75/25 split. KNN, random forest, gradient boosting, ridge, lasso, elastic net and a neural network are compared. The final model is a weighted blend of a tuned GBM (0.6), KNN (0.2) and Lasso (0.2).
Cross-sell. Email addresses are split into domains and bucketed as professional, personal or junk. The 75/25 split is stratified on the 68/32 class balance, and models are scored on AUC because predicting that everyone subscribes already gets about 68% accuracy. The final blend is logistic regression (0.6), GBM (0.3) and KNN (0.1) on predicted probabilities.
03 / What the data says
The email address is the funnel.
The promotion is sent by email, and conversion depends on which kind of address the customer gave.
| Domain group | Customers | Subscribed |
|---|---|---|
| Professional | 696 | 80.0% |
| Personal | 861 | 69.9% |
| Junk | 389 | 41.6% |
Two behavioural signals point the same way. Subscribers cancel before noon more often than non-subscribers (a mean of about 1.58 versus 1.04), which is a penalty-free action used by customers who actively manage their orders. Customers registered with a mobile number subscribe at about 70%, against 55% for those without one.
On the revenue side, median revenue peaks at a median meal rating of 4 rather than 5, a central-tendency bias in self-reported ratings, and it rises with customer-service contacts up to about 9 to 10 before falling.
04 / Out-of-sample fit
The flexible models memorise. The simple one doesn't.
In the cross-sell study the logistic regression scores 0.729 on training data and 0.726 on test data. The flexible models lose 0.14 to 0.41 AUC out of sample, and the neural network reaches a perfect 1.000 training AUC and 0.591 on test.
In the revenue study the tuned GBM alone has the highest test R² (0.784), but also a gap of 0.173. Blending it with Lasso and KNN gives 0.777 with a gap of 0.140. On a re-run the blend has a MAPE of 15.6%, and 75.6% of predictions land within 20% of actual revenue.
05 / Does it reproduce?
Some results hold. Some depend on the tuner.
Both notebooks were re-run end to end with the same data and split.
| Result | Committed | Re-run |
|---|---|---|
| Revenue blend, train / test R² | 0.916 / 0.777 | 0.916 / 0.777 |
| Revenue tuned GBM, train / test R² | 0.957 / 0.784 | 0.957 / 0.784 |
| Revenue neural network, train / test R² | 0.967 / 0.430 | 0.880 / 0.403 |
| Revenue “final” R² / MAPE | 0.763 / 16.3% | cell fails |
| Cross-sell logistic, test AUC / gap | 0.7258 / 0.0035 | 0.7266 / 0.0016 |
| Cross-sell tuned GBM, train / test AUC | 0.817 / 0.712 | 0.994 / 0.674 |
| Cross-sell final blend, test AUC | 0.721 | 0.712 |
| Cross-sell final blend, gap | 0.047 | 0.179 |
The Optuna studies are created without a seed, and the cross-sell GBM is fit with study.best_params. Its random_state is fixed, but the search is not, so the tuned GBM, and the blend that gives it 30% of the weight, changes from run to run. The revenue GBM, whose parameters are hard-coded, reproduces exactly. The logistic regression also has its regularisation strength searched, but it lands in nearly the same place (0.7258 vs 0.7266 test AUC).
final_test_pred).06 / What to change next
Seed the search, benchmark the blend.
Seed the Optuna studies, or hard-code the chosen parameters, so every reported number can be regenerated. Choose blend weights by cross-validation instead of by comparing test scores, and report cross-validated AUC and R² with a spread rather than a single split. Keep the logistic regression as the benchmark any blend has to beat, and restore or remove the missing revenue cell so the final figures can be traced.