When people ask me what I do for a living, “data engineer” usually earns a polite nod and a quick subject change. So let me try a better answer.
The plumbing analogy
Imagine a city. Somewhere there’s a reservoir full of water, somewhere else there’s a tap in someone’s kitchen, and in between there are kilometres of pipes, valves, pumps, and pressure gauges keeping the water clean, drinkable, and arriving at the right time.
A data engineer builds the same thing, but for data instead of water. The reservoir might be a credit card processor, a website’s clickstream, an ERP system, or a CSV someone in Finance keeps emailing around. The kitchen tap is a dashboard, a machine learning model, or a report on someone’s desk. My job is everything in between.
That “everything in between” is usually what people don’t see, and it’s most of the work.
A normal week
In a typical week I’m doing some mix of:
- Moving data: pulling it from where it lives (a database, an API, a file drop) and landing it somewhere we can query it.
- Cleaning it: dates that arrive as strings, currencies in three different formats, the customer who’s spelled four different ways. Most of the value I add is here.
- Modelling it: turning raw events into the tables analysts and data scientists actually want to work with. This is where SQL stops being a query language and starts being a design tool.
- Watching it: writing the alerts that wake someone up at 3 a.m. when a pipeline breaks, then trying very hard to make sure that someone isn’t me.
- Talking to humans: figuring out what the business actually needs, which is almost never what the first email asked for.
Why I like it
I like data engineering because the feedback loop is honest. Either the numbers add up at the end of the pipeline or they don’t. Either the dashboard loads in two seconds or it doesn’t. There’s a lot of room for craft inside that, but the truth shows up quickly.
It’s also a job where being curious pays off more than being clever. The best data engineers I’ve worked with aren’t the ones who know the most exotic tools — they’re the ones who, when something looks weird, can’t stop themselves from digging until they understand why.
That’s the part I’m trying to write about on this site. Not “10 SQL tips you didn’t know” — more like the small moments where I learned something and wanted to remember it.