Making a 411.7M-event pipeline restartable
My first aggregation ran out of temporary storage. I split the work by month and session bucket so I could reuse finished work after a retry.
- Python
- DuckDB
- Parquet
- FastAPI
BERLIN, GERMANY / MSc DATA SCIENCE
Hi, I’m
DATA & ML ENGINEERING
I build data pipelines and compare ML models against simple baselines. I’ve also used a GKE lab to learn how a service recovers from a bad release.
Open to junior Data & ML Engineering roles
01 / SELECTED WORK
A pipeline that ran out of disk, a GKE release that couldn’t pull its image, and a weekly baseline that beat a Transformer.
My first aggregation ran out of temporary storage. I split the work by month and session bucket so I could reuse finished work after a retry.
I built a pipeline on synthetic fulfillment data, then set up a separate GKE API lab. I released a deployment pointing to a nonexistent image, then verified recovery.
I compared eight forecasting approaches on four weeks of hourly cart and order counts. Repeating last week’s pattern gave the lowest mean WAPE on the development folds.
02 / HOW I WORK
What changed after I ran out of disk, compared forecast errors, and tested a bad release.
In REES46, splitting the work only helped if every event in a session still went to the same bucket.
Pipelines & data qualityFor OTTO, I chose the model on chronological development folds and kept the last 96 hours for the final test.
Evaluation & reproducibilityIn FulfillAI, I checked how Kubernetes behaved when it couldn’t pull the new image and how to get back to a healthy release.
Delivery & operations labsOPEN TO JUNIOR OPPORTUNITIES
I’m looking for junior Data and ML Engineering roles. If you have a role in mind or a question about the work, I’d like to hear from you.
Contact me