Skip to content
All Projects
Data EngineeringDelivered

Cloud Data Warehouse Migration on AWS

Re-architecting on-prem ETL into AWS for 80% better resource utilisation

Led a data-warehouse migration to AWS for a pan-African financial-services group, moving 20–30 ETL scripts and stored procedures onto Glue, Redshift and CloudFormation to improve resource utilisation by 80%.

AWSAWS GluePySparkSQLPL-SQLAmazon RedshiftAWS S3CloudFormationRDSSSIS
Problem Statement

An on-premises data warehouse was capping the business: limited scalability, high operational costs, and slow access for analytics teams. Together these slowed decision-making and held back growth.

  • On-prem warehouse offered limited scalability and high fixed operational cost.
  • Analytics teams had restricted, slow access to the data they needed.
  • Infrastructure constraints throttled data processing and decision-making.
Headline Outcomes
+80%job re-architecting

Resource utilisation

Reducedelastic compute

Operational cost

Minimalphased cutover

Migration disruption

The Solution

A sequenced, low-disruption migration to AWS. It re-architected 20–30 ETL streams and stored procedures with Glue, Redshift, S3 and CloudFormation, then added data-governance policies and a catalogue so quality and access improved alongside cost.

Migrated 20–30 ETL scripts and stored procedures with Python, PySpark and SQL.

Re-architected jobs on AWS Glue, Redshift, S3 and CloudFormation for elastic scale.

Optimised resource utilisation by 80% through job re-architecting and right-sizing.

Defined data-governance policies and a data catalogue with business stakeholders.

System Architecture

How the data flows

01

On-Prem Warehouse

Legacy source

02

ETL Re-architect

Python · PySpark · SQL

03

AWS Glue

Managed ETL

04

Redshift + S3

Cloud warehouse

05

Governance & Catalog

Quality + access

Result 01

Delivered an 80% gain in resource utilisation with lower running costs.

Result 02

Improved data quality and accessibility through governance and cataloguing.

Result 03

Ran the cutover with minimal disruption to critical analytics.

Build Something Like ThisServices & PricingMore Case Studies
Further reading

From the blog

Data Engineering

AWS DMS Limitations: What It Won't Migrate for You

AWS DMS copies your table data, not your stored procedures, indexes, triggers or validation. A practical guide to what Database Migration Service leaves you to do.

AWS DMSDatabase MigrationAWSData Engineering
Read Article12 min read
Data Engineering

Cloud Data Warehouse Migration: Snowflake vs Redshift vs BigQuery

A cloud data warehouse migration guide to Snowflake vs Redshift vs BigQuery vs Databricks: how to choose on cost, lock-in and performance, and how to de-risk the move.

Data WarehouseSnowflakeRedshiftBigQuery
Read Article14 min read
Taking on new projects · Outside IR35

Have a data pipeline or warehouse problem worth solving?

From messy source data to analytics-ready warehouses that cut cost. Let's scope it. I reply within one business day.

Start a ProjectConnect on LinkedIn