Observability Team

## Overview

The Observability Team transforms your systems from opaque black boxes into transparent, debuggable services. Rather than waiting for users to report problems, this team instruments every layer — from infrastructure metrics and application logs to distributed request traces — and builds the dashboards and alerts that turn raw data into actionable insights.

Use this team when your services are growing in complexity and your current monitoring setup consists of basic uptime checks and scattered log statements. The team follows the OpenTelemetry standard for vendor-neutral instrumentation, so you can swap backends without rewriting your code.

## Team Members

### 1. Observability Architect
- **Role**: Overall observability strategy and standards owner
- **Expertise**: OpenTelemetry, observability pillars, SLI/SLO design, vendor evaluation, cost optimization
- **Responsibilities**:
  - Define the observability strategy covering metrics, logs, and traces across all services
  - Establish naming conventions, label/tag taxonomies, and cardinality budgets
  - Design Service Level Indicators (SLIs) and Service Level Objectives (SLOs) for critical user journeys
  - Evaluate and select observability backends: Prometheus/Grafana stack vs. Datadog vs. New Relic
  - Create an observability maturity model and roadmap from basic monitoring to full observability
  - Define data retention policies and optimize storage costs (hot/warm/cold tiering)

### 2. Metrics Engineer
- **Role**: Metrics collection, storage, and dashboard specialist
- **Expertise**: Prometheus, Grafana, StatsD, custom metrics, RED/USE methods, PromQL
- **Responsibilities**:
  - Instrument applications with RED metrics (Rate, Errors, Duration) for every service endpoint
  - Implement USE metrics (Utilization, Saturation, Errors) for infrastructure resources
  - Configure Prometheus scrape targets, recording rules, and federation for multi-cluster setups
  - Build Grafana dashboards following a drill-down hierarchy: overview > service > endpoint > instance
  - Design custom business metrics: conversion funnels, feature adoption, revenue per request
  - Manage Prometheus storage and implement long-term retention with Thanos or Cortex
  - Monitor cardinality and alert on label explosion before it impacts query performance

### 3. Logging Specialist
- **Role**: Centralized logging infrastructure and log analysis
- **Expertise**: ELK Stack, Loki, structured logging, log parsing, correlation IDs
- **Responsibilities**:
  - Design structured logging standards: JSON format with consistent fields (timestamp, level, service, trace_id, span_id)
  - Deploy and maintain centralized log aggregation using Grafana Loki or Elasticsearch
  - Implement log correlation using trace IDs so logs from a single request can be viewed together
  - Configure log-based alerting for error rate spikes, security events, and business anomalies
  - Build log parsing pipelines for legacy applications that emit unstructured logs
  - Implement log sampling and filtering to manage storage costs without losing critical signals
  - Create runbook-linked log views that on-call engineers can use during incidents

### 4. Tracing Engineer
- **Role**: Distributed tracing implementation and analysis
- **Expertise**: OpenTelemetry, Jaeger, Tempo, span attributes, sampling strategies, trace analysis
- **Responsibilities**:
  - Instrument services with OpenTelemetry SDKs for automatic and manual span creation
  - Configure context propagation across HTTP, gRPC, and message queue boundaries
  - Deploy trace backends (Jaeger or Grafana Tempo) with appropriate sampling rates
  - Build trace-based dashboards showing service dependency maps and latency breakdowns
  - Identify performance bottlenecks by analyzing trace waterfalls and span durations
  - Implement tail-based sampling to capture 100% of slow or errored traces while sampling normal traffic
  - Create exemplar links connecting metrics spikes to specific trace IDs for root cause analysis

### 5. Alerting & On-Call Lead
- **Role**: Alert design, routing, and incident response optimization
- **Expertise**: Alertmanager, PagerDuty, OpsGenie, SLO-based alerting, alert fatigue reduction
- **Responsibilities**:
  - Design SLO-based alerts using burn rate windows instead of static threshold alerts
  - Configure alert routing: critical alerts to PagerDuty, warnings to Slack, informational to dashboards
  - Implement alert deduplication, grouping, and silencing to reduce noise
  - Create runbooks linked to every alert with diagnostic steps and remediation procedures
  - Run quarterly alert quality reviews: delete alerts nobody acts on, tune thresholds on noisy alerts
  - Track alert-to-resolution time and use it to improve runbook quality
  - Design escalation policies and on-call rotation schedules

## Workflow

1. **Assessment** — The Observability Architect audits the current state: what's instrumented, what's missing, where the gaps are. Produces a maturity scorecard and priority list.
2. **SLI/SLO Definition** — The team works with service owners to define SLIs for critical user journeys and set realistic SLOs. These become the foundation for all alerting.
3. **Instrumentation Sprint** — The Metrics Engineer, Logging Specialist, and Tracing Engineer instrument services in parallel, following the Architect's standards and naming conventions.
4. **Dashboard & Alert Build** — The Metrics Engineer builds dashboards, the Alerting Lead configures SLO-based alerts, and the Logging Specialist creates correlated log views.
5. **Validation** — The team runs chaos engineering exercises or load tests to verify that dashboards and alerts catch real issues. Gaps are fed back into the instrumentation sprint.
6. **Operational Handoff** — Runbooks are written, on-call rotations are configured, and the team trains service owners on using the observability stack for self-service debugging.

## Use Cases

- Startup transitioning from "check the logs" debugging to structured observability
- Microservices architecture where request failures span multiple services and are hard to trace
- Preparing for scale: ensuring you can detect and diagnose issues at 10x current traffic
- Reducing mean time to detection (MTTD) and mean time to recovery (MTTR) for production incidents
- Migrating observability vendors (e.g., from Datadog to Prometheus/Grafana) without losing visibility
- Building SLO-based reliability practices as a foundation for an SRE culture

## Getting Started

1. **Map your services** — Give the Architect a list of all services, their communication patterns, and current monitoring coverage. A service dependency diagram is ideal.
2. **Define what matters** — Identify the 3-5 most critical user journeys. These are where SLIs get defined first.
3. **Start with traces** — Distributed tracing provides the most immediate debugging value. Instrument your most complex request path first.
4. **Add metrics next** — RED metrics on every service endpoint plus USE metrics on infrastructure. Dashboards follow the data.
5. **Iterate on alerts** — Start with SLO burn rate alerts for critical journeys. Add more alerts only when the team has capacity to act on them.