Deep Dive 13-The AI-for-SRE Bridge: Your Unique Lane¶

A self-contained reference chapter on the rarest and most defensible identity in 2026 applied AI: the engineer who has actually been on-call.

0. Why this chapter exists¶

The rest of the curriculum tells you, correctly, to stop introducing yourself with "I'm a Bamboo plugin engineer." That sentence is too narrow, too vendor-locked, and too disconnected from where the money and the interesting problems are. The roadmap is right.

But the roadmap is also, quietly, leaving an asset on the table.

You are not "an SRE who is now learning AI." You are a backend / SRE engineer with production-incident scar tissue, telemetry literacy, and distributed-systems instincts, who is now learning to build LLM systems. That combination is not a transitional embarrassment to be hidden in your bio. It is the rarest and most under-supplied combination in the 2026 applied-AI labor market.

This chapter exists to:

Name your existing assets explicitly so you stop discounting them.
Show the market segments that specifically pay for this combination.
Lay out the recurring AI-for-SRE problem patterns with enough rigor that you can build them.
Re-frame your Bamboo + Datadog plugin work from "old job" to "case study with telemetry I already understand."
Hand you a 90-day project plan and a year-2 roadmap that compounds the bridge identity.

The thesis: AI applied to SRE / observability / incident management is underserved in 2026, and you are unusually well positioned to serve it. Treat the rest of this document as evidence and execution detail for that thesis.

1. The thesis, stated bluntly¶

There are roughly four populations in this market right now:

Population A-AI engineers who never operated production. They can fine-tune a model and ship a Streamlit demo, but they cannot tell you what an SLO is, have never been paged at 3am, and have never had to write a postmortem with their VP reading it. When their LLM service starts misbehaving in production, they reach for "let's just retry" or "let's just bump the timeout."
Population B-SREs who have not learned LLMs. They know how to run distributed systems but treat anything generative as black magic. Their org has an "AI initiative" they are not part of. They will be increasingly squeezed as ops work consolidates around AIOps platforms.
Population C-generalist software engineers drifting toward either side opportunistically.
Population D-engineers who are fluent in both halves. They can design an LLM agent, and they can tell you how its failure modes will interact with the existing on-call rotation, the alerting stack, the deploy gates, and the incident review process.

Population D is small. The supply pipeline is slow because each half is a multi-year apprenticeship and the two cultures (academic ML and production ops) historically barely talked. The demand is rising on every front: AIOps vendors, LLM observability vendors, frontier labs, internal AI platform teams, and a fresh wave of AI-for-DevOps startups all need Population D engineers who understand both halves and can translate between them.

You are halfway there already. The other half is what the rest of this curriculum is for.

The corollary that nobody tells you: in 2026 it is faster and cheaper to teach an experienced SRE the LLM stack than to teach a fresh AI engineer how production actually works. The latter requires real outages, real on-call rotations, and real customers complaining-none of which scale with a course.

2. Your existing assets, named explicitly¶

You should be able to recite this list cold. Each item is something most applied-AI engineers do not have, and each one matters.

2.1 Production-incident intuition¶

You have been paged. You have been the IC on an incident bridge. You know what it feels like when graphs go red and you are not sure if it is your service, the dependency, or the dashboard itself.

This shows up in concrete instincts:

You suspect correlated, not coincident, failures by default.
You ask "when did this start?" before you ask "what is broken?"
You separate symptoms from causes in your head without thinking about it.
You know that "we just restarted it and it's fine now" is not a root cause.
You know that the on-call who finds a regression at 2am is too tired to write a quality postmortem at 9am, and that this is a process problem, not a personnel problem.

These instincts are the single hardest thing to teach an LLM-systems engineer. You already have them.

2.2 Telemetry literacy¶

You know what metrics, logs, and traces should contain. You know the difference between a counter and a gauge, a histogram and a summary. You know that p50 lies and p99 with low sample size lies harder. You know what cardinality is and why it bankrupts your bill.

In LLM systems, this maps directly:

Token counters are counters; they need rate, not absolute.
Latency to first token and latency to completion are separate distributions; treating them as one will mislead you.
Tool-call success rate is an SLI shaped exactly like a downstream-dependency success-rate SLI.
Eval score over time is a gauge that needs a baseline, not a threshold.
Cost per request is a histogram, not an average.

Almost every LLM observability product in 2026 is rebuilding the same primitives Datadog APM has had for a decade. You already think in those primitives.

2.3 Distributed-systems instincts¶

You think in terms of partial failure, retries, idempotency, backpressure, fan-out / fan-in, and timeout budgets. You know that a 30-second timeout in a service called from a 10-second-timeout service is a bug, not a config.

LLM systems amplify all of these:

Retries on a non-idempotent generation will produce duplicate side effects.
Streaming responses change the failure model-a 200 OK can still die mid-token.
Tool-calling agents fan out into N sub-requests; you need a budget across the whole tree, not per call.
Long-context requests have queueing and head-of-line blocking that a generalist will miss.

You will catch these design errors in code review without needing a textbook.

2.4 CI/CD discipline¶

You have built and operated build pipelines, deploy gates, canaries, and rollback procedures. You know that "the deploy passed CI" is not the same as "the deploy is safe." You know what a feature flag is for.

LLM-system deploys have more moving parts than code deploys, not fewer:

Prompt templates change.
Model versions change.
RAG indices change.
Fine-tuned weights change.
Tool schemas change.

Each is independently rollable and each has its own canary discipline. The teams that ship these without discipline are the ones rebuilding incident response from scratch every month. You already know what discipline looks like.

2.5 Customer-of-AI experience¶

You have used Copilot in your IDE, ChatGPT for queries, and Claude for code. You have opinions about what works, what hallucinates, and what is worth paying for. You have personal experience of the failure modes from the user side.

This is non-trivial. A surprising number of "AI engineers" have built products they themselves would not use, because they have never been the on-call user of a tool that gave them confidently wrong information at 3am. You have. You will design defaults that respect that.

2.6 A working production codebase you can point at¶

The Bamboo plugin and the Datadog metrics work give you a concrete artifact to anchor your portfolio in. We will return to how to frame this in section 11. The short version: you are not starting from a blank repo when you build an incident-RCA agent-you are starting with a real telemetry source and real (sanitized) historical incidents. Most applied-AI candidates would pay for that.

3. The 2026 job market for the bridge skillset¶

This is not exhaustive; it is a map of the lanes where Population D sells at a premium.

3.1 AIOps vendors¶

The category-AI for IT operations-was coined years ago by the analyst firms and predates the LLM wave. The 2024–2026 wave is bringing LLMs into it.

Representative vendors and what they ship:

Datadog-AI-driven anomaly detection, watchdog-style automated insights, and a growing set of LLM-augmented features for triage and summarization.
New Relic-applied AI for observability and incident response.
Dynatrace-Davis AI for causal analysis and root-cause assist on top of their topology graph.
Splunk-AI assistants over their query language and investigation workflow.
PagerDuty-AIOps for alert grouping, suppression, and incident summarization.

What these vendors actually need is engineers who can (a) build LLM features and (b) credibly evaluate them against the messy reality of customer telemetry. Your Datadog plugin time is direct, citable evidence that you can read customer telemetry. Treat it that way in interviews.

3.2 LLM observability vendors¶

This is a younger, hotter category that emerged in 2023–2024 and has matured fast.

Representative vendors:

Langfuse, LangSmith-tracing and evals for LLM apps.
Arize, Helicone, Braintrust-observability, monitoring, eval, and experimentation for LLM systems.

Every one of these is rebuilding APM concepts (traces, spans, percentiles, alerts, dashboards, SLOs) for LLM-shaped workloads. They are explicitly hiring engineers who have shipped real observability tooling. "I shipped a Datadog plugin" is the kind of line that gets you a screen.

3.3 Frontier-lab platform / SRE roles¶

Anthropic, OpenAI, Cohere, and similar labs run production inference at a scale that breaks generalist intuition. Their SRE and platform teams have been hiring continuously and increasingly screen for LLM-shaped failure-mode literacy on top of standard SRE skills: head-of-line blocking under long-context load, KV-cache memory pressure, model-version rollouts, eval regression detection, prompt-injection-as-incident.

You are an SRE who is deliberately developing LLM-shaped failure-mode literacy. That is the bullseye.

3.4 Internal AI platform teams at Fortune-500s¶

By 2026, essentially every large enterprise has an internal team building "the AI platform"-the shared infrastructure that lets product teams build LLM features without each one re-inventing prompt management, eval, observability, and access control.

These teams routinely struggle to hire because the role requires both platform-engineering chops and enough ML literacy to make sane defaults. The bridge skillset is exactly what they want, and the comp tends to be very competitive without the volatility of pure-play AI startups.

3.5 AI-for-DevOps and incident-response startups¶

A cohort of newer companies sits explicitly at the AI-for-SRE intersection. incident.io is the clearest example, and there are several others doing AI-augmented incident response, on-call assistants, runbook agents, postmortem drafting, and alert triage.

These companies hire for the bridge identity by name. They will read "SRE turning into applied-AI engineer" as a feature, not a bug.

3.6 Where this market is heading¶

A reasonable, non-fabricated read of the trend lines:

The LLM-observability category is consolidating; the survivors will look more like APM vendors with LLM-aware semantics.
AIOps vendors will absorb LLM-augmented features as a standard feature set, not a differentiator.
Internal AI platform teams will become the dominant buyer of bridge-skillset talent over the next few years simply because there are more of them than there are vendors.
"Have you ever been on-call?" will increasingly appear on screening loops at frontier labs.

The plain implication: build the bridge identity now, while the supply gap is wide.

4. The recurring problem patterns where AI helps SRE¶

You should be able to name and sketch each of these from memory. The next chapters of this document treat the high-leverage ones in depth.

Incident triage-given an incoming alert, classify severity, deduplicate, and propose a first action.
Root-cause analysis (RCA)-given an active incident, correlate metrics, logs, and traces; produce ranked hypotheses with evidence.
Runbook execution-given a known scenario, run the prescribed playbook with human-in-the-loop gates.
Postmortem drafting-given an incident timeline, produce a first-draft postmortem in the team's template.
Anomaly detection-classical statistical detection augmented by LLM context filtering.
Natural-language observability-translate "show me errors in checkout in the last 30 minutes" into the right query DSL.
Code-change risk classification-given a PR, predict deploy risk; surface concerns; gate with HIL.
Customer-impact correlation-given an incident, answer "which customers were affected, and how badly?"

The patterns are not mutually exclusive; a real production AIOps surface is several of them stitched together. They share a small set of architectural primitives, which is what makes the skillset coherent.

5. Pattern 1-AI-augmented incident triage¶

5.1 The problem¶

A monitoring system fires alerts at the on-call. Most alerts are noise. Some are real but well-understood (the runbook handles them). A small fraction are real and novel. Human triage is slow and inconsistent at 3am. The cost of mistakes is asymmetric: missing a real high-severity alert is much worse than over-paging on a low one.

5.2 The interface¶

Input: alert payload-metric or log signal, threshold, current value, recent context (last N minutes of related signals), service ownership, runbook link, recent deploys for the implicated service.
LLM task: classify severity, identify whether the symptoms match a known runbook entry, propose a first action.
Output: a structured triage record (severity, runbook match or null, first-action proposal, confidence) plus a Slack message to the on-call channel.

5.3 Architecture sketch¶

+----------------+       +-----------------+       +----------------+
|  Alert source  | --->  |  Triage worker  | --->  |  Slack / pager |
|  (Datadog, PD) |       |  (this service) |       |  channels      |
+----------------+       +-----------------+       +----------------+
                              |     ^
                              |     |
                              v     |
                         +-----------------+
                         |  Context fetch  |
                         |  - metrics tool |
                         |  - logs tool    |
                         |  - deploys tool |
                         |  - runbook RAG  |
                         +-----------------+
                                |
                                v
                         +-----------------+
                         |   LLM call      |
                         |  (constrained   |
                         |   JSON output)  |
                         +-----------------+
                                |
                                v
                         +-----------------+
                         |  Eval & audit   |
                         |   logger        |
                         +-----------------+

The triage worker is a small stateless service. It receives the alert, fetches a bounded amount of context (cardinality-limited; do not stuff a million log lines into a prompt), retrieves relevant runbook chunks via RAG, calls the LLM with a strict JSON schema for the output, logs the full input and output for later eval, and posts a structured Slack message.

5.4 Eval¶

Build a labelled set of historical alerts. For each, a human labels:

True severity in retrospect.
Whether a known runbook applied.
What the correct first action was.

Evaluate the LLM triage on this set. The two metrics that matter most:

Recall on high-severity-never miss a real fire.
False-positive rate on high-severity-never cry wolf often enough that the on-call mutes the channel.

A simple precision/recall trade-off curve over a confidence threshold is your friend. Choose the operating point with eyes open, not by accident.

5.5 Production discipline¶

Fail-safe to human. If the LLM fails, errors out, or returns low confidence, escalate the alert exactly as the existing system would have.
Never auto-page or auto-resolve. The system suggests; humans decide. This is a hard rule for the first year.
Log everything. Every input, every output, every action taken downstream. You will need this for incident review when the AI is wrong.
Eval on every model change. Treat model upgrades as deploys. Run the labelled set as a regression test.

5.6 What the bridge engineer brings here¶

A pure-AI engineer will build this without the fail-safe rails and without the eval set, because their cultural reflex is "ship the demo." A pure-SRE engineer will not build it at all, because their cultural reflex is "automation that touches the alerting stack is forbidden." You will ship the safer, more useful version because you understand both reflexes.

6. Pattern 2-AI-augmented root-cause analysis¶

6.1 The problem¶

An incident is open. Symptoms are visible. The on-call has a hypothesis space that is too large to manually walk through under stress. You want a system that, given the incident context, can propose ranked hypotheses with evidence.

6.2 The interface¶

Input: incident record-symptom description, time window, suspected services, recent deploys, links to symptom dashboards.
LLM task: query metric, log, and trace tools; retrieve relevant runbook context; generate ranked hypotheses with citations to evidence.
Output: a structured RCA report-hypotheses ranked by likelihood, each with the evidence that supports and contradicts it, plus suggested next checks.

6.3 Architecture sketch¶

            +------------------+
            |  Incident record |
            +------------------+
                     |
                     v
            +------------------+
            |   RCA agent      |
            |   (LLM + tools)  |
            +------------------+
              |   |   |    |
              v   v   v    v
        metrics logs traces runbook
         tool   tool tool    RAG
              \  |  /        |
               \ | /         |
                vvv          v
            +------------------+
            |   LLM reasoning  |
            |   loop with tool |
            |   calls          |
            +------------------+
                     |
                     v
            +------------------+
            |  Structured RCA  |
            |  report          |
            +------------------+

The agent runs a bounded reasoning loop: propose a hypothesis, query a tool to test it, fold the evidence back in, repeat up to N steps. Each tool call is logged. Each hypothesis must cite specific evidence by reference (a log query result, a metric value at a timestamp, a trace ID).

6.4 The harder problem-hallucinated correlations¶

The single most dangerous failure mode is the LLM confidently asserting a correlation that does not exist. "Service X latency spiked at 14:02 and the deploy happened at 14:01, so the deploy caused it." That sentence might be right, or it might be that the deploy was a config-only change to a different region.

Defenses:

Evidence-must-cite. The output schema requires every hypothesis to cite specific tool outputs. No citation means the hypothesis is dropped, not displayed.
Adversarial eval. Include in the eval set incidents where the LLM is given misleading context (a deploy that did not cause the incident); measure how often it falsely accuses the deploy.
Time alignment hygiene. The LLM is bad at minute-level reasoning across many services. Pre-compute time-aligned views before handing them to it; do not ask it to reason from raw timestamps.
Reasoning trace surfaced to human. The on-call sees the chain of tool calls and evidence, not just the conclusion.

6.5 Realistic walk-through¶

Consider an incident: checkout p99 latency is 5x baseline starting 14:02 UTC.

The agent fetches metrics for the checkout service over the last hour. Confirms the spike at 14:02.
It queries deploy events. Finds a deploy of the cart service at 13:58.
It queries traces from checkout that show 90% of the latency is now spent in a downstream call to cart.
It queries cart metrics-error rate is unchanged but latency is up.
It queries cart logs filtered to the deploy time and finds new log lines about a cache miss path.
It generates a ranked hypothesis: "Cart deploy at 13:58 introduced a cache miss path that increased downstream latency, propagating to checkout p99." Evidence: deploy event, trace breakdown, cart latency metric, cart log lines.
Suggested next check: roll back the cart deploy in canary; observe checkout p99.

The on-call still makes the rollback call. The agent saved them ten minutes of dashboard chasing.

6.6 Eval¶

Use historical incidents with known root causes. Replay the symptom snapshot through the agent. Score:

Top-1 correctness-was the actual root cause the highest-ranked hypothesis?
Top-3 correctness-was it in the top three?
False-confidence rate-did the agent rank a wrong hypothesis as high confidence?

This is the place where your existing telemetry is gold. You already have months or years of real incidents in your previous orgs and your plugin user base. The eval set is sitting there waiting.

7. Pattern 3-Postmortem agent¶

7.1 The problem¶

Postmortems are time-consuming, important, and consistently late. The on-call who handled the incident is exhausted; the org wants the doc within 48 hours; the doc is the primary input to the org's learning loop.

7.2 The interface¶

Input: incident timeline (Slack thread, alert log, deploy events, code changes, dashboards linked during the incident).
LLM task: draft a postmortem in the team's template-timeline, root cause, contributing factors, customer impact, action items.
Output: a structured doc, ready for human edit. Never a final doc. Always a draft.

7.3 Why this is high-leverage¶

It is bounded: the LLM is summarizing material that already exists, not generating novel claims. The failure mode is "boring draft" rather than "dangerous wrong action." The human always edits before publishing. And it saves the on-call hours of post-incident drudgery, which directly improves the quality of the rest of the postmortem (a fresh human is a better contributor than a depleted one).

7.4 Architecture¶

The architecture is simpler than RCA. A scheduled or manually-triggered job pulls the incident artifacts, normalizes them into a structured timeline, retrieves the team's postmortem template, and prompts the LLM to produce the draft. The output is written to a draft doc and shared with the incident commander.

Important details:

Template-conformant output. The team has a template; the draft must conform to it. Use structured generation, not freeform.
No invented facts. The prompt must instruct the model to mark unknowns as [TODO: confirm] rather than guess. Eval for this; it is the most common failure mode.
Customer-impact section requires data. Hook the customer-impact section to a real query against your customer-impact correlation pipeline, not a guess.

7.5 Eval¶

This one resists fully-automated eval. The honest answer is human-rated quality on three axes:

Factual accuracy. No invented facts.
Completeness. All template sections present.
Clarity. Prose readable to a non-incident-attendee.

Periodically sample drafts; have the original incident commander rate them. Track the trend.

7.6 What you bring¶

You have written postmortems. You know the difference between a postmortem that produces real action items and one that is a CYA document. You will design the prompt and the template integration so the output produces the former, because you have suffered through the latter.

8. Pattern 4-Natural-language observability¶

8.1 The problem¶

Engineers spend significant time translating "what they want to know" into "what their observability tool's query DSL accepts." This is a tax on every investigation. An LLM that translates English to Datadog / Splunk / Loki / PromQL well enough is genuinely useful.

8.2 The interface¶

Input: a natural-language question. "Show me error responses from the checkout service in the last 30 minutes that aren't on our known-issues list, grouped by status code."
LLM task: emit a valid DSL query for the chosen backend.
Output: the query string, plus a one-sentence explanation of what the query does.

8.3 Architecture¶

The high-leverage move is constrained decoding plus few-shot grounding, not freeform generation.

Provide the DSL grammar (or a curated subset) in the system prompt.
Provide a curated set of natural-language → DSL examples chosen to cover the dimensions of variation: time ranges, filters, groupings, joins, aggregations.
Constrain the output to be valid DSL via either a structured-output schema or a post-generation parse-and-retry loop. Reject obviously wrong queries before showing them.
For a Datadog or Splunk integration, surface the query in their UI, do not execute it directly without a click. The user is one keystroke from running it; that is enough.

8.4 Eval¶

Build a curated set of 100+ natural-language queries with expected DSL outputs, stratified by complexity:

Simple filter queries (~30%).
Aggregations and groupings (~30%).
Time-series with deltas / rates (~20%).
Joins / multi-source (~10%).
Edge cases (negation, regex, exclude lists) (~10%).

Score by execution-equivalence: does the generated query return the same result as the expected one over a fixed snapshot of data? This is more useful than string-match because there are usually multiple correct queries.

8.5 Why your background matters¶

You know the DSLs. You know which queries are commonly written wrong by humans (regex on high-cardinality fields, p99 over too-small windows). You can curate the eval set with judgment that a fresh LLM engineer cannot.

9. Pattern 5-Code-change risk classification¶

9.1 The problem¶

Most production incidents are caused by deploys. Most deploys are safe. The on-call wants to know which of today's twelve deploys is the one to worry about.

9.2 The interface¶

Input: PR diff, files touched, test changes, author tenure, recent deploy history of the touched services.
LLM task: classify deploy risk as low / medium / high; surface specific concerns.
Output: structured risk record + a comment on the PR or a flag in the deploy gate.

9.3 Architecture¶

A webhook on PR open or merge fires a worker that pulls the diff and metadata, calls the LLM with a strict schema, posts the result back. The model output goes into a risk record table that feeds the deploy gate.

9.4 Eval¶

Use historical PRs that were rolled back as positives. Use a sampled set of normal merged PRs as negatives. Score precision/recall on rollback prediction.

The honest truth: most LLM risk classifiers do not beat a strong heuristic baseline (touching files known to be hotspots, modifying production-config files, large diffs by recent hires). Always evaluate against the heuristic baseline. The LLM earns its keep by surfacing concerns in natural language that the heuristic cannot articulate.

9.5 Production discipline¶

Use as a deploy gate with HIL, not auto-block. A high-risk classification adds a required reviewer, not a hard stop.
Track override rate. If humans override "high" 80% of the time, the model is wrong, not the humans.
Calibration matters. A model that says "high" on 50% of PRs is useless. Track distribution.

10. Pattern 6-Anomaly detection augmented by LLM context¶

Classical anomaly detection (statistical, often unsupervised) generates a stream of candidate anomalies. Most are not actionable: deploys, marketing pushes, scheduled jobs, holiday traffic patterns, known-issue residuals.

The LLM's job is not to find anomalies. It is to filter them: given the candidate anomaly and the recent change context, is this anomaly actionable or expected?

This is a classic "small-LLM-as-filter" pattern and it works well because:

The LLM is given a tight context window of structured data, not asked to reason about raw timeseries.
The decision is binary-ish (forward or suppress) with a confidence.
False suppression is the failure mode that matters; eval for it specifically.

The architectural primitive is the same as Pattern 1 but the failure-cost asymmetry is different (suppressing real signal is the dangerous direction; in Pattern 1 it was over-paging).

11. Reusing the Bamboo + Datadog plugin work¶

11.1 The reframe¶

The plugin is not your identity. It is a case study with real production telemetry that you have legal and contextual access to.

In your portfolio, in your résumé, in your interviews, the plugin should appear in this shape:

"I built an LLM-powered incident-summarization layer on real production telemetry from a Bamboo CI/CD environment with Datadog metrics, with an eval set of 50 historical incidents showing X% reduction in time-to-first-hypothesis."

Notice what changed:

The headline noun is the LLM layer, not the plugin.
The plugin is the substrate, not the product.
The eval set, not the code, is the load-bearing artifact.
The metric is operationally meaningful (time-to-first-hypothesis) rather than vanity (lines of code).

11.2 Why the eval set is the asset¶

LLM engineering interviewers in 2026 have seen a thousand "I built a chatbot over my data" projects. They are trained to ignore them. What they have not seen is "I built an eval set on real incident data and used it to gate model and prompt changes." That sentence is rare because the data is rare. You have it.

A good eval set takes weeks to build, requires real domain context, and is the thing that lets a project ship with confidence. It is the artifact a senior interviewer asks about. Make it the centerpiece.

11.3 Q2 anchor narrative¶

Slot this work as your Q2 capstone in the curriculum. Concretely:

Week 1: scrub a set of 50 historical incidents from Bamboo + Datadog, anonymize, label with severity and root cause.
Weeks 2–3: build an incident-RCA agent using one of the patterns in this document. Start with Pattern 1 (triage); upgrade to Pattern 2 (RCA) if time allows.
Weeks 4–6: run the eval; iterate prompt and architecture; document the iteration in a public notebook.
Weeks 7–8: write the public artifact (blog post, GitHub repo, optional talk submission).

This single project, well-executed, is more credibility than five generic LLM courses.

12. The unique observability questions LLM systems raise¶

This is the most interesting territory in the bridge: new SRE questions that the field does not yet have settled answers to. You are unusually well-positioned to contribute answers, not just consume them.

12.1 What is an SLI for an LLM service?¶

The standard SRE doctrine says an SLI is a quantitative measure of service quality from the user's perspective. For a non-LLM API, this is usually:

Availability (fraction of requests that succeeded).
Latency (some percentile of response time).
Correctness (fraction of responses that were not erroneous).

LLM services break the third axis. "Did the response succeed" is no longer binary, because a 200 OK can return a confidently wrong answer that does more harm than a 500.

A reasonable SLI set for an LLM service:

Availability-request-completed-successfully rate. Same as before.
Latency to first token, latency to final token-distributions, not means.
Eval-passing rate-fraction of responses that pass an automated eval (rubric, regex, structured-output schema, model-graded judgment) in shadow mode.
Tool-call success rate-for agentic services, the fraction of tool calls that returned successfully.
Hallucination rate-for RAG-backed services, fraction of responses with claims not supported by retrieved context, measured by an eval.

Each is measured continuously, not at deploy. Each has a target.

12.2 What is an error budget for a generative system?¶

The classical formulation: SLO = 99.9% availability ⇒ error budget = 0.1% of requests can fail per window. Burn the budget faster than expected ⇒ slow down deploys.

For an LLM service, "error" is gradient-valued. A response can be 90% right. A hallucination might or might not have caused customer harm. So we need budgets per SLI:

A latency budget (p99 over target) is straightforward and identical to non-LLM SRE.
An eval-passing-rate budget-"fewer than X% of responses can fail eval per week."
A hallucination budget-"fewer than X% of responses can fail the grounding check."
A safety budget-"zero responses can violate the safety policy" (often zero-tolerance, with a hard escalation when the budget is touched).

The novelty: budget exhaustion does not always mean "freeze deploys." It might mean "rollback the most recent prompt change" or "rollback the model version"-which leads to the next question.

12.3 Canary deploys for prompt changes¶

Same shape as code canaries; specifics differ.

The "canary" is a fraction of traffic routed to the new prompt template.
The "metrics" being compared are eval scores, latency, cost-per-request, and tool-call success rate.
The "rollback" is reverting the prompt-template version pointer.

Crucial difference from code canaries: the eval signal is often noisier than latency metrics, so the canary needs more traffic or a longer window before deciding. Plan for it.

12.4 The rollback unit for an LLM service¶

This is where most teams trip. An LLM service has at least four orthogonal rollable axes:

Prompt template-the text and structure of the prompt.
Model version-provider model identifier.
RAG index-the retrieval corpus and its embeddings.
Fine-tuned weights-if applicable.

Each axis has its own change-management cadence, its own canary discipline, and its own rollback procedure. Treating them as one ("we deployed v2.3") is the equivalent of deploying code, config, schema migrations, and infra changes in a single commit.

The bridge engineer instinct: each axis is its own deploy with its own gate.

12.5 Change management for production prompts¶

A prompt-change procedure that respects the SRE-doctrine analogy:

Source-controlled. Prompts live in a git repo with PR review. No "edited in the UI."
Versioned and immutable. Each prompt template has a version ID; the running service references the version, not the latest pointer.
Eval-gated. Merging a prompt change requires the eval suite to pass with at most a configured regression tolerance.
Canary-rolled out. Ten percent of traffic for a defined window with health metrics watched.
Rolled back via pointer flip. No code deploy needed to revert.

This is exactly the discipline you applied to Bamboo plans and Datadog dashboards. You can carry it directly across.

13. Bridging Datadog instincts to the LLM-observability stack¶

13.1 The mental map¶

Most LLM-observability tools are rebuilding APM-shaped primitives with LLM-aware semantics. The shape carries; the labels change.

Datadog instinct	LLM-observability translation
Metrics (statsd / DogStatsD)	Prometheus / Grafana with OTel-native pipelines
Logs	Loki, OTel logs, or vendor-native LLM trace stores
Traces (APM)	OTel traces with the GenAI semantic conventions
APM service map	Agent / tool-call graph in Langfuse / Arize
Datadog APM detail view	LangSmith / Langfuse trace detail, with prompt + tool-call breakdown
Watchdog / anomaly detection	Eval drift detection, often custom
Synthetic monitoring	Eval suites run on a schedule against the live service
Dashboards	Same dashboards; new metric semantics on top
Monitors	Eval-score monitors, hallucination-rate monitors

The OpenTelemetry GenAI semantic conventions are the most important emerging standard here. They define how to instrument LLM calls, tool calls, and embeddings across vendors. Reading them once will make the rest of the stack legible.

13.2 The migration playbook¶

For a working Datadog shop adopting LLM observability:

Keep Datadog as the system-of-record for infra metrics. Latency, error rate, host metrics, container metrics-no reason to move them.
Add an LLM-trace store (Langfuse, LangSmith, Arize, Helicone, Braintrust-pick one based on team workflow, not features). LLM traces are expensive to send to general APM.
Connect them via OTel. Span context flows across; you can pivot from a high-latency LLM call in Langfuse to the underlying infra trace in Datadog.
Build a unified incident view. When an alert fires, the on-call needs to see infra metrics, LLM eval scores, and tool-call success in one dashboard. Build it; vendors do not give it for free yet.
Treat eval suites as first-class monitors. Schedule them; alert on regressions; budget for them.

What stays from the Datadog way of life: SLO discipline, runbooks, on-call rotations, change-management. What changes: the specific telemetry primitives in some sub-spans. The cultural infrastructure ports unchanged. Lean on it.

14. The positioning narrative¶

The interview-grade version of your story should sound roughly like this. You should not memorize this verbatim-you should internalize it until it is just true.

"I spent several years as a backend / SRE engineer running production systems with Bamboo and Datadog. Two things kept happening. One: my team's incidents were increasingly LLM-related-hallucinations, prompt regressions, retrieval drift-and our existing observability tools did not have shape for them. Two: I watched ML teams ship LLM features without SLOs, without canaries, without runbooks, and pay for it.

So I started building the bridge. I built an incident-RCA agent on top of real Bamboo + Datadog telemetry, with a labelled eval set of 50 historical incidents. I built an eval framework for prompt changes that gated deploys the same way unit tests gate code. I'm building, as a capstone, an LLM-observability dashboard that unifies infra metrics with eval scores and tool-call success.

The pitch is simple: I bring SRE rigor-SLOs, error budgets, canaries, runbooks, postmortems-into LLM systems. Most AI teams do not have that rigor; most SRE teams do not have the LLM literacy. I'm trying to be one of the small number of engineers who has both."

Specific concrete projects to point at:

Q2 anchor: incident-RCA agent on Bamboo + Datadog telemetry with labelled eval set.
Q3 anchor: eval framework for prompt changes; deploy-gating integration.
Capstone: unified LLM-observability dashboard.

14.1 Conferences and venues¶

This skillset has natural homes:

SREcon (USENIX)-the premier SRE conference; AI-for-SRE talks land well there.
KubeCon / CloudNativeCon-observability tracks; OTel GenAI conventions are increasingly featured.
AI Engineer Summit / AI Engineer World's Fair-applied-AI-engineer audience; the "we brought SRE rigor to LLMs" framing is rare and valued.
Observability conferences-including vendor-hosted ones; the bridge angle is novel.
Local meetups-most cities have an SRE meetup and an applied-AI meetup; speaking at both establishes you in both communities.

Submit talks. The proposal alone, even when rejected, sharpens the narrative. The accepted talks are portfolio gold.

15. Non-obvious advice¶

A handful of judgment calls you will not find in the standard curriculum.

15.1 Do not downplay the SRE half¶

The instinct, when pivoting, is to bury the previous identity. Resist it for this pivot. Frontier-lab SRE-platform interviews increasingly probe "have you ever been on-call?" because they have learned, painfully, that engineers without operational scar tissue make systems that page humans needlessly.

When asked about your background, lead with: "I was an SRE-I have been the on-call for production systems with real customers." Then bridge: "And I am applying that discipline to LLM systems." Both halves are load-bearing.

15.2 Bring SRE rigor into AI engineering¶

This is the single highest-leverage move you can make in any AI-engineering team you join. Most teams do not have:

SLOs on their LLM service.
Error budgets that gate deploys.
Runbooks for known LLM failure modes.
Canary discipline on prompt changes.
Postmortems with action items that actually get tracked.

You will arrive in your first AI-engineering role with all of this in your bones. Use it. The team will thank you within the first incident.

15.3 The AI-for-SRE direction will likely outpace SRE-uses-AI¶

Two related lanes:

AI-for-SRE: building AI tools for SRE problems (the patterns in this document).
SRE-uses-AI: using off-the-shelf AI tools to do SRE work better.

The second is valuable but commoditizing fast as AIOps vendors ship features. The first is where the engineering depth and the comp premium live, and where the hiring signal is rare. Bias toward the first.

15.4 The eval set is the moat¶

In any project on this bridge, the eval set is more valuable than the model code. Models change. Prompts change. Vendors change. The eval set-the labelled, domain-specific, hard-won corpus of "what good looks like"-is what lets you operate any of those models with confidence.

Treat building the eval set as the project, not as preparation for the project.

15.5 Open-source one thing¶

Pick one of the patterns in this document, build it well, and open-source it after scrubbing customer data. The repo plus the blog post is more credible than a résumé bullet, because it is verifiable.

The one to pick first is the one whose eval set you can build cleanly from your existing data. For you, that is probably incident triage or RCA over Bamboo + Datadog historical incidents.

16. The 90-day side project that demonstrates the bridge¶

This is the concrete plan. Execute it and you have, at the end, a project that is more valuable than any course on the curriculum for the bridge identity.

16.1 Goal¶

Ship a public, evaluated, documented LLM-augmented incident-triage system on real Datadog-shaped alert data, with an open-source repo and a blog post.

16.2 Pattern¶

Pattern 1 from section 5-incident triage. It is the pattern with the cleanest eval shape, the most defensible failure mode, and the easiest data to scrub.

16.3 Week-by-week¶

Weeks 1–2: data and eval set. Pull 50 historical alerts from your existing telemetry. Anonymize aggressively (replace customer names, hostnames, IPs). For each, label by hand: severity in retrospect (low/medium/high), root cause category, correct first action. This is the load-bearing step. Do not skip or rush it.

Weeks 3–4: baseline pipeline. Build the simplest version: alert in, LLM call with a strict JSON schema for triage output, structured response out. No tools, no RAG yet. Run it against the eval set. Record precision, recall, false-positive rate, and false-negative rate by severity class.

Weeks 5–6: context and runbook RAG. Add a context-fetching step: recent metrics for the implicated service, recent deploys, runbook chunks via RAG. Re-run the eval. Compare to baseline. The point is not just to improve numbers-it is to measure what each architectural addition buys.

Weeks 7–8: ablations and prompt iteration. Run ablations: which context sources matter? Which prompt structure works best? Which model is worth the price? Document each finding with eval numbers, not vibes.

Weeks 9–10: production discipline. Add the operational rails: fail-safe paths, audit logging, calibration check, model-change regression test. Even if you never run this in real production, having the rails is the difference between a demo and a system.

Weeks 11–12: artifact production. Open-source the repo with eval numbers in the README. Write the blog post: "Applying eval discipline to LLM-augmented incident triage." Submit a talk proposal somewhere-SREcon, AI Engineer Summit, a local meetup. Even a rejected proposal sharpens the artifact.

16.4 What to measure and report¶

In the README and the blog post:

Eval set size and composition.
Baseline precision / recall by severity.
Improvement from each architectural addition (context fetching, RAG, etc.).
False-positive and false-negative analysis with examples.
Cost per triage and latency distribution.
Honest discussion of what did not work.

Honesty about failure modes is the credibility marker that distinguishes an engineer from a marketer.

16.5 What this project credentials you for¶

AIOps vendor screens.
LLM-observability vendor screens.
Internal-AI-platform team screens.
AI-for-DevOps startup screens.
Frontier-lab SRE-platform screens.

It is one project. It is enough to credential the bridge identity at every door listed in section 3.

17. Practical exercises¶

Six exercises that match the chapter. Do at least three before moving on.

Exercise 17.1-SLIs and SLOs for an LLM-powered incident-triage service¶

Define, in writing:

Three to five SLIs for the service.
An SLO target for each.
The measurement window.
The error-budget policy when the SLO is missed.

Constraints: at least one SLI must be eval-derived (not pure availability/latency). At least one must be operationally meaningful to the on-call user, not just to the engineer running the service.

Exercise 17.2-Canary playbook for a prompt-template change¶

Author a playbook for rolling out a prompt-template change to a production LLM service. It should include:

Pre-deploy gate (eval suite with regression tolerance).
Canary traffic fraction and duration.
Monitored metrics during canary.
Automatic rollback criteria.
Manual rollback procedure with the exact pointer flip.
Postmortem trigger criteria if the canary fails.

Exercise 17.3-Eval set for natural-language-to-Datadog-query¶

Design (do not build yet) a 100-pair eval set, stratified by:

Query type (filter, aggregation, time-series, join).
Time range (last N minutes, today vs yesterday, week-over-week).
Cardinality risk (low-cardinality groupings vs high-cardinality).
Operator complexity (negation, regex, exclude lists).

Document the stratification and the rationale. The exercise is in the design, not the labelling. Most LLM evals fail because their composition was thoughtless.

Exercise 17.4-RCA-agent architecture diagram¶

Draw the architecture for an RCA agent. It should have, explicitly:

Alert source ingestion.
Tool-call layer for metrics, logs, traces.
Runbook RAG.
LLM reasoning loop with bounded steps.
Output channels (Slack, incident doc, PagerDuty annotation).
Audit log.
Eval harness path (offline replay against historical incidents).

Annotate each component with its failure mode and the rail that catches it.

Exercise 17.5-Conference talk abstract¶

Write a 200-word talk abstract for "How we applied SRE rigor to LLM observability." Submit it (yes, actually submit) to one venue from section 14.1. The abstract should:

Name the gap (most AI teams do not have SLOs, canaries, runbooks).
Name your contribution (what you built and what it measured).
Name the takeaway (one or two practices an audience member can apply on Monday).

If it is rejected, revise and submit elsewhere. The abstract is the artifact regardless of acceptance.

Exercise 17.6-Year-2 roadmap¶

Sketch a 12-month roadmap that doubles down on the AI-for-SRE bridge. It should have:

Four quarter-anchor projects, each shippable and evaluable.
Four specialty deepening points (one per quarter): areas you will go from "competent" to "credibly expert" in (suggested candidates: eval methodology, OTel GenAI conventions, agentic-system reliability, LLM cost optimization).
A target external artifact per quarter (blog post, talk, OSS repo, internal whitepaper).
A target community engagement per quarter (talk submission, meetup organization, OSS contribution to a relevant project).

Constraint: every quarter must produce one artifact a hiring manager can read in five minutes and two artifacts a deep interviewer can probe for an hour.

18. Summary and what to do next¶

The thesis again, compressed:

The bridge between AI engineering and SRE is rare, valuable, and underserved.
You are halfway across it already, and the curriculum will get you the rest of the way.
Your existing assets-incident intuition, telemetry literacy, distributed-systems instincts, CI/CD discipline, customer-of-AI experience, real telemetry to point at-are not legacy baggage. They are the moat.
The market-AIOps vendors, LLM observability vendors, frontier-lab platform teams, internal AI platforms, AI-for-DevOps startups-pays a premium for this combination.
The technical patterns are coherent and learnable: triage, RCA, postmortem, NL-to-query, change-risk, anomaly filtering. Each shares architectural primitives.
Your Bamboo + Datadog work, reframed, is a case-study substrate, not an identity.
One well-executed 90-day project on real telemetry credentials you across the entire market.
The eval set is the moat within the moat.

What to do this week:

Re-read sections 2 and 11. Make them part of how you talk about yourself.
Pick one pattern from sections 5–10 and commit to building it as the 90-day project.
Start the eval set today. 50 historical, anonymized incidents with severity and root-cause labels. The data is the load-bearing artifact. Everything else compounds on top of it.
Submit one talk abstract this quarter, even (especially) if you are not sure you are ready. The deadline is what produces the artifact.

The shortest version of the chapter, if you forget the rest: you are not an SRE who is late to AI. You are an applied-AI engineer who has already operated production. There are not many of you. Act accordingly.