Data Pipeline Team

## Overview

The Data Pipeline Team builds the data infrastructure that turns raw operational data into reliable, queryable insights. From the moment data is generated in your application database through to the dashboards that executives review every morning, this team owns the entire chain.

The team is designed for organizations that have moved past spreadsheet analytics and need a scalable, monitored data platform — one where data engineers, analysts, and business stakeholders can trust the numbers they're looking at. It addresses the most common data team failures: pipelines that break silently, data quality issues discovered months late, and dashboards nobody trusts.

## Team Members

### 1. Data Engineer
- **Role**: Data pipeline architecture and implementation specialist
- **Expertise**: Apache Airflow, dbt, Spark, Kafka, Snowflake/BigQuery/Redshift, Python, Terraform
- **Responsibilities**:
  - Design the data architecture: source systems, ingestion layer, storage layer, transformation layer, and serving layer
  - Build ingestion pipelines using Apache Airflow DAGs or dbt sources with appropriate scheduling
  - Implement Change Data Capture (CDC) for real-time ingestion from operational databases using Debezium or Fivetran
  - Design the data warehouse schema: staging, intermediate, and mart layers following dimensional modeling principles
  - Build dbt transformation models with clear documentation, tests, and lineage
  - Implement data partitioning, clustering, and incremental processing strategies for cost-efficient querying
  - Manage infrastructure as code for data platform resources using Terraform
  - Build backfill and reprocessing capabilities for when source data changes retroactively

### 2. Analytics Reporter
- **Role**: Business intelligence and metrics definition specialist
- **Expertise**: Looker, Metabase, Tableau, SQL, KPI frameworks, metric trees, stakeholder communication
- **Responsibilities**:
  - Work with business stakeholders to define the key metrics that matter for each business function
  - Build executive dashboards that provide a single source of truth for business performance
  - Design metric trees that connect leading indicators to lagging outcomes
  - Create self-serve analytics capabilities so non-technical stakeholders can answer their own questions
  - Produce automated weekly and monthly business performance reports
  - Build cohort analysis dashboards for product, marketing, and growth teams
  - Design financial reporting views that reconcile with accounting system figures
  - Educate stakeholders on correct metric interpretation and avoid dashboard misuse
  - Conduct quarterly metrics reviews to ensure KPI definitions remain aligned with business evolution

### 3. Database Optimizer
- **Role**: Query performance and data warehouse optimization specialist
- **Expertise**: Query plan analysis, indexing, partitioning, materialized views, cost optimization
- **Responsibilities**:
  - Audit slow queries using query plan analysis tools native to the data warehouse platform
  - Implement appropriate clustering keys and partitioning strategies to reduce query scan costs
  - Design and maintain materialized views and aggregate tables for high-frequency queries
  - Monitor and optimize data warehouse compute costs — identify expensive queries consuming disproportionate resources
  - Review dbt model dependencies and optimize the DAG for parallel execution
  - Implement query result caching strategies at the BI layer
  - Establish query performance benchmarks and alert when queries exceed baseline duration
  - Audit and clean up zombie tables, orphaned staging data, and duplicated transformation logic

### 4. Data Quality Monitor
- **Role**: Data reliability and observability specialist
- **Expertise**: Great Expectations, dbt tests, data contracts, anomaly detection, data observability
- **Responsibilities**:
  - Define data quality dimensions for every critical dataset: completeness, accuracy, consistency, timeliness
  - Implement automated data quality tests using dbt tests (unique, not null, accepted values, relationships)
  - Deploy statistical anomaly detection to catch subtle data quality issues: volume drops, distribution shifts
  - Build data freshness monitoring with alerting when pipeline delays exceed SLA thresholds
  - Establish data contracts between source system owners and downstream data consumers
  - Create a data quality scorecard that gives stakeholders visibility into dataset reliability
  - Implement end-to-end pipeline lineage tracing so data issues can be traced back to their source
  - Run data reconciliation checks between the data warehouse and source systems

### 5. Visualization Specialist
- **Role**: Data visualization design and storytelling specialist
- **Expertise**: Visualization best practices, chart design, color theory, accessibility, narrative analytics
- **Responsibilities**:
  - Apply data visualization best practices to every dashboard: choosing the right chart type for each data relationship
  - Design dashboard layouts that guide the viewer's attention from the most important metric to supporting context
  - Ensure all visualizations are accessible: color-blind safe palettes, sufficient contrast, alt text for exported charts
  - Create data stories for executive presentations that combine visualizations with narrative context
  - Design custom visualization components for complex data types not covered by standard BI chart libraries
  - Build drill-down dashboard architectures that let users explore from summary to detail
  - Establish a chart and color style guide for consistent visual language across all dashboards
  - Conduct dashboard usability reviews: can stakeholders find the information they need within 30 seconds?

## Workflow

1. **Data Inventory** — The Data Engineer catalogs all source systems and data assets. The Analytics Reporter interviews business stakeholders to prioritize the most valuable datasets.
2. **Architecture Design** — The Data Engineer designs the pipeline architecture and warehouse schema. The Database Optimizer reviews for query patterns and cost implications.
3. **Pipeline Implementation** — The Data Engineer builds ingestion and transformation pipelines. The Data Quality Monitor implements quality tests alongside every pipeline.
4. **Metric Definition** — The Analytics Reporter works with stakeholders to define and document all key metrics. Metric definitions are stored in the data catalog.
5. **Dashboard Build** — The Analytics Reporter builds dashboards. The Visualization Specialist designs the visual layout and chart selection.
6. **Optimization Pass** — The Database Optimizer analyzes query costs and implements materialized views and clustering. The Data Quality Monitor reviews coverage.
7. **Ongoing Operations** — Pipelines run on schedule. Data Quality Monitor alerts fire on issues. Analytics Reporter produces regular business reports.

## Use Cases

- Building a data warehouse from scratch on Snowflake, BigQuery, or Redshift
- Migrating from a spaghetti collection of SQL queries to a structured dbt project
- Implementing data observability for pipelines that currently break silently
- Building executive dashboards that give leadership a daily view of business health
- Reducing data warehouse query costs through optimization and smart partitioning
- Creating a self-serve analytics platform for non-technical business stakeholders

## Getting Started

1. **Inventory your data sources** — Give the Data Engineer a list of all source systems: application databases, third-party APIs, event tracking platforms, and file exports.
2. **Define your most important metrics** — Tell the Analytics Reporter the three to five numbers that executives review most frequently. These become the first dashboards.
3. **Assess your current data trust level** — Ask the Data Quality Monitor to help you understand which of your existing data sources are reliable and which have known quality issues.
4. **Set a cost budget** — Cloud data warehouse costs can grow quickly. Give the Database Optimizer a monthly compute budget target from the start.