The problem
I wanted to understand the path from messy shopping events to useful recommendations. That meant ingestion, quality checks, deduplication, features, retrieval, ranking and evaluation, rather than starting with a tidy model-ready table.
Why I split the aggregation
The first global Gold aggregation over roughly 410 million Silver events exhausted temporary storage after a ~30.6 GiB DuckDB spill. I redesigned the work around monthly partitions so a failure would have a smaller recovery boundary.
Ordered session aggregation needed another step. I split monthly sessions into 64 deterministic hash buckets on user_session and aggregated one bucket at a time. Keeping a session in one bucket preserved the ordering contract while bounding the aggregate state.
Completed outputs are written atomically and reused on restart. The aim was to make the computation fit the machine and make retries predictable.
Bronze, Silver, and Gold outputs
The Bronze layer processed 411,709,736 events from October 2019 to April 2020. Removing 1,384,422 exact duplicates left 410,325,314 canonical Silver events. Monthly Gold outputs include user-item interactions, item and user statistics, category affinity and session sequences.
The modeling path compares popularity, co-visitation, collaborative retrieval, hybrid retrieval and purchase ranking. MLflow records experiments. A local FastAPI service exposes the chosen bundle with an explicit popularity fallback for unknown users.
The separate offline benchmark
Data-processing scale and model-evaluation scale are different. The ranking experiment used 1,499,021 training interactions, 50,000 training sessions, 5,000 validation users and 5,000 held-out test users.
Training covered October–February, validation used March, and final testing used April 2020.
| Metric | Popularity | Purchase ranker |
|---|---|---|
| MRR@20 | 0.02443 | 0.03556 |
| Recall@10 | 0.05685 | 0.07087 |
| NDCG@20 | 0.03593 | 0.04429 |
Absolute ranking quality remains modest. These results describe the defined offline test; they do not establish online engagement, revenue, or users served by a live production service.
What I would work on next
I would investigate candidate recall, cold-start behavior, and realistic latency and memory benchmarks before treating this as a deployed recommendation service. Online experiments, continuous retraining and freshness guarantees remain future work.
Source & evidence
Code, experiment results, and implementation notes.