Data EngineeringDelivered

Automated Distributor ETL for Unilever

Hands-off SFTP-to-analytics for daily sales & stock data

An automated ELT pipeline that detects distributor files landing on SFTP, validates and homologates them, and lands analytics-ready data in S3 for real-time querying via StarRocks.

DagsterdbtPostgreSQLAWS S3StarRocksSFTPPython

Problem Statement

Distributors of corporate clients like Unilever upload sales and stock data to SFTP every day. The data had to be detected, validated, homologated and made query-ready in real time, with no manual handling and corrections reflected immediately.

Files arrive continuously on SFTP and must be auto-detected and processed.
Inconsistent product codes and units of measure across distributors.
Customers need a complete, up-to-date view of sales and stock in real time.

Headline Outcomes

Zero

Manual intervention

Real-time

Data freshness

Homologated

Cross-distributor consistency

The Solution

A Dagster-orchestrated ELT flow. It extracts on file arrival, validates format and fields, loads RAW to PostgreSQL, transforms and homologates against history in dbt, stages, then publishes to S3 for StarRocks analytics.

Dagster detects SFTP arrivals and triggers extraction automatically.

Two-tier validation: initial format/field checks, then consistency checks against historical data.

dbt transforms and homologates product codes and units of measure for cross-distributor comparability.

Approved data lands in S3 and is queryable in real time through StarRocks.

System Architecture

How the data flows

SFTP Arrival

CSV from distributors

Validate

Format + field checks

RAW Load

PostgreSQL

Transform

dbt homologation

S3 + StarRocks

Real-time analytics

Result 01

Eliminated manual handling of daily distributor uploads end-to-end.

Result 02

Unified product codes and units so all distributor data is comparable.

Result 03

Gave clients an always-current view of sales and stock operations.

Build Something Like This Services & Pricing More Case Studies

From the blog

Data EngineeringJun 5, 2026

Dagster vs Airflow vs Prefect for ETL in 2026

Dagster vs Airflow vs Prefect for ETL in 2026, compared by the one question they disagree on: what a pipeline is actually made of. Field notes from a 50M-records/day build.

DagsterAirflowPrefectETL

Read Article12 min read

Taking on new projects · Outside IR35

Have a data pipeline or warehouse problem worth solving?

From messy source data to analytics-ready warehouses that cut cost. Let's scope it. I reply within one business day.

Start a Project Connect on LinkedIn