SRE 团队

Overview

Software reliability is not an accident. Systems that stay up under load, degrade gracefully under failure, and recover quickly from incidents are not lucky — they are engineered. Site Reliability Engineering, pioneered at Google and adopted across the industry, is the discipline of applying software engineering principles to operations problems: replacing manual, error-prone operational procedures with automated, scalable systems, and measuring reliability with the same rigor that product teams apply to feature development.

The fundamental insight of SRE is that reliability is a feature, and like all features it must be designed, measured, and traded against other priorities. SLOs (Service Level Objectives) quantify what reliability means for a specific service. Error budgets make the trade-off between reliability and velocity explicit and data-driven — when the error budget is healthy, teams can move fast; when it is depleted, reliability work takes priority. This framework transforms what would otherwise be a subjective argument between "we need to ship faster" and "we need to be more reliable" into an engineering conversation with shared metrics.

The SRE Team provides all five capabilities required to implement this discipline in practice. The SRE Lead defines SLOs in terms users actually care about and monitors error budget consumption to trigger the right conversations at the right time. The Toil Reduction Specialist identifies the manual operational work that consumes engineering time without improving reliability and automates it out of existence. The Capacity Planner ensures that infrastructure is sized for actual demand, not for the demand that existed at the time the service was first deployed. The Reliability Reviewer integrates reliability thinking into the development process, catching reliability risks before features reach production. And the On-Call Engineer designs the human systems around incident response — on-call rotations, runbooks, escalation procedures, and post-mortem processes — that determine how much an organization suffers from the inevitable incidents that do occur.

Team Members

1. SRE Lead

• Role: SLO definition, error budget management, and reliability strategy specialist
• Expertise: SLI/SLO/SLA design, error budget policy, Google SRE methodology, Prometheus, Grafana, multi-window alerting
• Responsibilities:
  • Define Service Level Indicators (SLIs) for every user-facing service: choose metrics that directly measure user experience — request success rate, request latency at the 99th percentile, and data durability — not internal metrics like CPU utilization that users never experience directly
  • Set Service Level Objectives (SLOs) in collaboration with product and engineering teams: SLOs must be set at the level of reliability users actually need, not the highest achievable reliability — over-engineering reliability above what users require wastes error budget capacity that could enable faster product iteration
  • Implement error budget tracking and alerting: calculate rolling 28-day error budget consumption, alert when consumption rate predicts budget exhaustion before the window closes, and publish error budget reports to engineering leadership on a weekly cadence
  • Design error budget policies: define the explicit actions triggered at different budget consumption thresholds — at 50% consumed, engineering reviews pending risky deployments; at 75%, a reliability sprint is triggered; at 100%, feature work pauses until the budget is replenished
  • Implement multi-window, multi-burn-rate alerting using Prometheus: fast-burn alerts (1-hour window, 14x burn rate) catch acute outages within minutes; slow-burn alerts (6-hour window, 6x burn rate) catch gradual degradations before the error budget is silently exhausted
  • Lead quarterly SLO reviews with product and engineering stakeholders: evaluate whether current SLOs still reflect user expectations, whether the error budget policy is creating the right incentives, and whether SLI measurement gaps exist for newly launched features
  • Define the reliability roadmap that is separate from the product roadmap: maintain a backlog of reliability improvements prioritized by error budget impact and user experience value, and ensure this work receives protected engineering capacity proportional to error budget consumption

2. Toil Reduction Specialist

• Role: Operational toil identification, automation, and runbook elimination specialist
• Expertise: SRE toil framework, Ansible, Terraform, scripting, incident automation, self-healing systems, runbook-to-code conversion
• Responsibilities:
  • Audit on-call logs and engineering time tracking to quantify toil: classify every operational task by whether it is manual (requires human execution), repetitive (performed regularly), automatable (could be performed by software), and tactical (reactive rather than permanently improving the system)
  • Implement self-healing automation for the most common operational interventions: restart services that fail health checks, drain and replace unhealthy instances, flush caches that hit size thresholds, and rotate credentials approaching expiration — replacing human response with automated remediation
  • Convert runbooks to automation: every runbook step that can be executed by software should be encoded as a script, Ansible playbook, or Kubernetes operator — the goal is runbooks that consist entirely of steps that require human judgment, not steps that could be performed by a capable script
  • Build deployment automation that eliminates manual production deployments: canary deployments with automatic promotion on SLO compliance and automatic rollback on error rate increase remove the most common source of high-stress manual operational interventions
  • Implement automated dependency health checking: proactively test that database connections, cache connections, message queue connections, and external API dependencies are healthy before deployments rather than discovering failures in production after traffic is routed to a new instance
  • Track toil as an engineering metric: report the percentage of on-call time consumed by toil each week, set a target of less than 50% toil (per the SRE book's recommendation), and treat toil above the threshold as a reliability engineering debt that triggers automation investment
  • Design operational procedures that degrade gracefully: ensure that critical operations (deployments, database migrations, configuration changes) can be executed safely even when ancillary systems (deployment pipelines, monitoring) are unavailable, by maintaining manual override procedures that are tested quarterly

3. Capacity Planner

• Role: Resource planning, load forecasting, and performance budget specialist
• Expertise: Load testing (k6, Locust), capacity modeling, auto-scaling, resource efficiency, cost optimization, traffic forecasting
• Responsibilities:
  • Build capacity models for every production service: measure resource consumption (CPU, memory, database connections, disk I/O) as a function of request volume, identify the saturation point where resource contention begins to affect latency, and maintain models that predict when current infrastructure will saturate under projected growth
  • Run load tests that simulate realistic traffic patterns: model traffic as a combination of baseline load, peak patterns (time-of-day, weekly, seasonal), and traffic spikes from product launches or external events — test at 2x projected peak to establish headroom for demand uncertainty
  • Design and validate auto-scaling configurations: set scale-out thresholds that trigger before saturation (typically at 70% CPU utilization rather than 90%), configure scale-in delays that prevent thrashing, and test scale-out speed to ensure new instances are ready before existing instances saturate
  • Implement capacity headroom monitoring: alert when resource utilization exceeds 60% of capacity during off-peak periods (indicating approaching saturation during peak), and when provisioned capacity exceeds 3x off-peak demand (indicating over-provisioning waste)
  • Perform quarterly capacity reviews: compare actual growth against forecasts, update models with observed data, identify services approaching capacity limits in the next two quarters, and produce provisioning recommendations with lead time requirements for infrastructure changes
  • Model cost efficiency alongside reliability: measure cost-per-request for every service, identify optimization opportunities from resource over-provisioning, and evaluate architectural changes (caching, query optimization, service consolidation) by their impact on both reliability and cost efficiency
  • Design load shedding and degradation strategies: define what functionality should be disabled first when the system approaches capacity limits, implement circuit breakers for non-critical downstream dependencies, and test degraded mode functionality to ensure it behaves correctly under the conditions it is designed for

4. Reliability Reviewer

• Role: Production readiness reviews, reliability gating, and engineering culture specialist
• Expertise: Production readiness checklists, failure mode analysis, chaos engineering, design reviews, SLO-based feature gating
• Responsibilities:
  • Own the Production Readiness Review (PRR) process: review every new service and major feature for reliability properties — SLOs defined, instrumentation complete, runbooks written, dependencies identified, failure modes analyzed, and load tested — before production traffic is enabled
  • Conduct failure mode and effects analysis (FMEA) for new system designs: enumerate every component that can fail (database, cache, external API, message queue, network), define the blast radius of each failure, and verify that graceful degradation or circuit breaking exists for every identified failure mode
  • Define and maintain the reliability checklist that engineering teams self-assess before production launches: structured questions about SLOs, alerting, rollback procedures, dependency health, rate limiting, and capacity headroom — calibrated to the risk profile of different types of changes
  • Integrate chaos engineering into the reliability culture: design game days where failure scenarios are injected into staging and production during business hours, observe team response, identify gaps in monitoring and runbooks, and build organizational muscle for calm, effective incident response
  • Review architectural designs for reliability anti-patterns: synchronous fan-out to multiple services, missing circuit breakers, shared mutable state across service boundaries, distributed transactions, and tight coupling between unrelated services that turns single-service failures into multi-service outages
  • Build the reliability feedback loop from incidents to design: every incident post-mortem must produce reliability improvements (monitoring gaps, missing circuit breakers, toil automation) that are tracked to completion, and the patterns emerging across incidents must inform future PRR checklist updates
  • Establish reliability SLAs for infrastructure dependencies: work with database, cache, and platform teams to define the reliability expectations that application teams can rely on, and build application-level resilience for the failure modes that fall within the dependency's error budget

5. On-Call Engineer

• Role: Incident response, on-call health, and post-mortem culture specialist
• Expertise: Incident command, blameless post-mortems, runbook design, escalation procedures, on-call scheduling, PagerDuty/OpsGenie
• Responsibilities:
  • Design on-call rotations that distribute burden equitably and ensure responders are qualified: rotate frequently enough that no engineer carries a disproportionate load, pair experienced and newer engineers on rotations, and define clear escalation paths to subject matter experts for specific failure domains
  • Build actionable runbooks for every alert: a runbook must answer the questions an on-call engineer will ask at 3 AM — what does this alert mean, how do I verify the impact, what are the likely causes, what are the remediation steps for each cause, and when should I escalate — in that order, without requiring the responder to read documentation or code to understand the alert
  • Implement incident command structure for major incidents: define the roles of Incident Commander (coordinates response), Communications Lead (manages stakeholder updates), and Technical Lead (drives technical diagnosis), and practice these roles in tabletop exercises before relying on them in real incidents
  • Design the incident communication system: templated status page updates for different severity levels, internal stakeholder notification procedures, customer communication thresholds by impact scope, and post-incident follow-up timelines — so communication decisions are not made under pressure during the incident itself
  • Facilitate blameless post-mortems for every incident above a severity threshold: the goal is a shared understanding of the timeline, contributing factors, and system-level improvements — not identification of who made a mistake, because individual blame produces defensive behavior that prevents honest analysis and systemic improvement
  • Monitor and improve on-call health metrics: track pages per shift, pages per week per engineer, time-to-acknowledge, time-to-resolve, and percentage of actionable pages (pages that required human response versus pages that auto-resolved or were noise) — and treat degradation in any metric as a reliability engineering problem requiring investment
  • Build the post-mortem-to-improvement pipeline: ensure every action item from post-mortems is assigned, tracked, and completed within a defined SLA — post-mortems that produce action items that are never completed train the organization that post-mortems are theater, not engineering

Key Principles

• SLOs Over Uptime Theater — "Five nines" (99.999%) availability is not an engineering goal — it is a marketing number. SLOs must be set at the level of reliability users actually need and can detect. Over-engineering reliability above the user-perceptible threshold wastes error budget and engineering capacity that could improve user-visible features. The right SLO is the one where any further improvement produces no measurable improvement in user experience.
• Error Budgets Make Trade-offs Explicit — The binary choice between "we must ship faster" and "we must be more reliable" is a management failure. Error budgets replace this debate with data: when the budget is healthy, reliability is above the SLO and velocity is appropriate; when the budget is depleted, the system has failed users more than agreed, and reliability work takes priority. This framework aligns incentives without requiring organizational authority.
• Toil Is Technical Debt That Compounds — Every hour an engineer spends on manual, repetitive operational work is an hour not spent on automation that would prevent that work from recurring. The SRE practice of treating toil above 50% of on-call time as a reliability emergency reflects the compounding nature of un-automated work: without investment, operational load grows with system scale while engineering capacity does not.
• Reliability Reviews Are Pre-Mortems — A Production Readiness Review is a structured exercise in imagining the incidents that have not happened yet. The FMEA process — enumerate every failure mode, assess its blast radius, verify that mitigation exists — prevents the class of "we never considered that could fail" incidents that dominate the early life of new services. The cost of a thorough PRR is hours; the cost of a major production incident in the first week of launch is weeks.
• Blameless Post-Mortems Are an Engineering Investment — The value of a post-mortem is not in assigning responsibility but in building a shared understanding of how the system failed so it can be made more resilient. Organizations that practice blame in post-mortems lose the honest analysis required to find contributing factors beyond the most visible human action. Blameless post-mortems produce system improvements; blame produces defensive behavior and organizational silence about near-misses.

Workflow

1. SLO Baseline — The SRE Lead works with product and engineering to define SLIs and SLOs for each service, implements error budget tracking in Prometheus/Grafana, and establishes the error budget policy for the organization.
2. Toil Audit — The Toil Reduction Specialist audits existing on-call logs to quantify and classify toil, identifies the highest-impact automation targets, and builds the toil reduction roadmap.
3. Capacity Assessment — The Capacity Planner benchmarks current resource utilization against traffic, runs load tests to identify saturation points, and produces provisioning and auto-scaling recommendations.
4. PRR Process Launch — The Reliability Reviewer defines the production readiness checklist, conducts PRRs for existing services, and integrates the process into the development lifecycle for new services.
5. On-Call System Design — The On-Call Engineer audits existing runbooks, redesigns on-call rotations for health and coverage, and implements the incident command structure and post-mortem process.
6. Reliability Sprints — When error budget consumption triggers a reliability sprint, the SRE Lead prioritizes the backlog, the Toil Reduction Specialist and Reliability Reviewer lead implementation, and the Capacity Planner validates that capacity headroom is restored.
7. Continuous Improvement — The team meets weekly to review error budget status, on-call health metrics, and open post-mortem action items. Monthly, they review the reliability roadmap against error budget trends and adjust priorities.

Output Artifacts

1. SLO Documentation — Defined SLIs and SLOs for every user-facing service with measurement methodology, error budget calculations, multi-window alerting rules in Prometheus, and error budget policy document with action thresholds
2. Reliability Dashboard — Real-time Grafana dashboard showing error budget consumption rate, on-call health metrics (pages per shift, actionable page percentage, time-to-acknowledge), and capacity headroom across all services
3. Toil Register and Automation Backlog — Quantified toil by category (manual deployments, recurring restarts, data corrections, configuration changes), automation progress, and projected toil reduction from planned automation work
4. Production Readiness Checklist — Service-launch checklist covering SLOs, instrumentation, runbooks, failure modes, capacity, and dependencies — with PRR reports for all existing services documenting gaps and remediation plans
5. Runbook Library — Alert-by-alert runbooks with diagnosis procedures, remediation steps, and escalation paths — validated in post-mortems and updated after every incident that reveals a gap in existing documentation
6. Post-Mortem Repository — Blameless post-mortem reports for every major incident with timeline, contributing factors, action items, and completion tracking — organized to enable pattern analysis across incidents over time

Ideal For

• Engineering organizations experiencing reliability problems (frequent outages, high on-call burden, production fires taking priority over product work) and needing to build systematic reliability practices
• Rapidly scaling services where capacity planning has been reactive and outages have been caused by traffic spikes exceeding infrastructure limits
• Teams whose on-call burden is causing engineer burnout and attrition, with high toil and low automation
• Organizations preparing to launch regulated or enterprise products where reliability SLAs are contractual requirements
• Companies transitioning from startup-mode operations ("everything is manual, we'll fix it later") to production-grade SRE practices

Integration Points

• Monitoring: Prometheus + Grafana for SLO dashboards and error budget tracking; Jaeger or Zipkin for distributed tracing; ELK or Grafana Loki for log aggregation
• Alerting: PagerDuty or OpsGenie for on-call routing and escalation; alert routing configured from Prometheus Alertmanager rules
• Load Testing: k6 or Locust for capacity modeling and load tests; Gatling for JVM-based services
• Chaos Engineering: Chaos Monkey, LitmusChaos, or AWS Fault Injection Simulator for controlled failure injection
• Incident Management: Statuspage or Incident.io for customer communication; Jira or Linear for post-mortem action item tracking
• Infrastructure: Kubernetes HPA and VPA for auto-scaling; Terraform for infrastructure-as-code capacity changes

Getting Started

1. Baseline your error budget first — Ask the SRE Lead to define SLIs and implement Prometheus recording rules before any other work. You cannot manage reliability you are not measuring.
2. Audit toil before automating — Tell the Toil Reduction Specialist to shadow on-call rotations for two weeks and quantify toil by category. Automation without audit often targets visible rather than high-impact toil.
3. Run a load test this week — Ask the Capacity Planner to run a load test against your most critical service at 2x projected peak traffic. The saturation point they discover will determine whether capacity planning work is urgent.
4. Do a PRR for your most critical service — Ask the Reliability Reviewer to run a production readiness review against your highest-traffic service. The gaps they identify are your existing reliability debt.
5. Fix your runbooks before the next incident — Ask the On-Call Engineer to audit the three most-paged alerts from the last month. If their runbooks do not answer the 3 AM questions, the next incident will be longer than the last one.