British Retail Consortium: Enterprise Data Warehouse and Retail Intelligence Platform

Retail intelligence, brought together.

Automated ELT pipelines, a shared data warehouse and reporting for a major retail association.

Retail / Industry Intelligence
Data engineering
Project

200+retail members

11platforms unified

The Challenge

What stood in the way.

The existing reporting process relied heavily on spreadsheets, manual formatting and static outputs. That made it harder to preserve history, prove data quality, reuse data and grow the reporting portfolio efficiently.

Manual shaping

Analysts repeatedly reformatted incoming data to match master sheets, adding effort and exposure to errors.

Fragmented history

Historical data was not consistently centralised, which limited retrieval and longitudinal analysis.

Quality uncertainty

Manual checks made it difficult to evidence reliability across transformations and outputs.

Static, Excel-led reporting

Dependence on PDF outputs and spreadsheet workflows limited interactive exploration, reuse and governance.

Our Solution

How we answered it.

We designed an Azure-native platform that runs from ingestion to insight. Source acquisition, lake storage, Synapse processing, Azure SQL and Power BI are connected, with governance and operational controls applied across the stack.

ProblemChoosing a data platform that fits BRC's Azure services and governance model.
Our answerWe assessed Microsoft Synapse, Databricks and Snowflake. Databricks needed additional Azure network and security layers and Snowflake carried a higher estimated platform cost
ProblemSource files arrive by email, from web endpoints and through APIs, each in its own format.
Our answerLogic Apps, APIs and Azure Functions bring files into ADLS landing storage, and staging notebooks standardise them into structured outputs before modelling.
ProblemData quality was hard to prove across transformations.
Our answerValidation checks and table reconciliation detect discrepancies, and every table carries an insertion timestamp, update timestamp and job ID for diagnosis and traceability.
ProblemBusiness logic was repeated inside individual report files.
Our answerWe moved it into reusable fact, dimension and semantic KPI structures that serve every monitor.

How It Works

The parts that made the difference.

Automated acquisition

Logic Apps retrieve email attachments and web files, while APIs and Azure Functions support additional source patterns. Analysts no longer move files by hand.

Layered data architecture

Files land in ADLS before any shaping, then move through staging, raw Delta tables, a curated semantic layer and the warehouse, so source data stays traceable.

Reusable data models

Dimensions, facts, lookup structures and a semantic KPI layer hold business logic once and serve multiple monitors.

Warehouse and Power BI

Fact, dimension and metadata tables load into Azure SQL for analyst access and interactive Power BI reporting with secure sharing.

Results

What changed for the client.

The strongest value is structural. BRC now has a repeatable data and analytics capability that reduces dependence on manual preparation, makes more historical and current data available for analysis and lets it add