Datadog PagerDuty Integration: Automate Incident Triage
Connect Datadog alerts with PagerDuty incidents, then give the on-call engineer a concise, evidence-linked triage brief instead of another dashboard hunt.

TL;DR: what you'll build. When a Datadog monitor pages PagerDuty, the on-call engineer still spends the first few minutes hunting for context: which service, which recent alerts, what changed. This article shows an IT Operations or SRE lead how to build a Major app and workflow that watches new PagerDuty incidents, pulls the related Datadog monitor and service details, and has an agent draft a source-linked triage brief for a human to review. The native Datadog-PagerDuty integration still does the paging. Major adds the governed layer that gathers context automatically.
Why incident triage starts with context gathering
The Datadog-to-PagerDuty pipeline is well worn: a monitor breaches, PagerDuty pages the right person, and the clock starts. The part that still eats minutes is what happens next.
The on-call engineer opens the PagerDuty incident, then tabs over to Datadog to find the monitor, check its recent history, and figure out which service it belongs to. On a bad night with several related alerts firing close together, that context-gathering can take longer than the actual fix.
This is where automation on Major fits. Not by replacing the page or the responder's judgment, but by running the repeatable retrieval work (find the incident, find the monitor, find the service, pull recent related alerts) the same way every time, and handing the responder a brief instead of a blank search. That pattern of a governed AI workflow with durable state is the same one behind Major's Snowflake board metrics automation, just applied to incidents instead of board decks.
The owning function here is IT Operations or SRE; the process itself is cross-industry, since any team running production infrastructure on Datadog and PagerDuty hits the same first-ten-minutes problem. It's the same reasoning behind governed workflows and connector automation generally: deterministic steps for the repeatable retrieval, judgment reserved for the parts that need it. And because this build touches incident records that may later matter for a retrospective or an audit, the same audit evidence for AI-generated code thinking applies to audit evidence for AI-drafted incident briefs: every read and every approval needs a trail.
What the Datadog and PagerDuty integration already handles
Before building anything new, it helps to be precise about what the native integration already does, because the triage layer should not duplicate it.
According to Datadog's PagerDuty integration docs, connecting the two tools lets you:
- Add a pagerduty-serviceName handle to a monitor's notification configuration, so a breached threshold pages the right PagerDuty service.
- Declare a Datadog incident directly from a PagerDuty alert, with severity mapped automatically from the monitor's status (an alert or no-data state maps to PagerDuty's error severity, a warning state maps to warning, and an OK or other state maps to info). Datadog says this mapping is automatic and cannot be modified.
- Auto-resolve the PagerDuty incident when the monitor recovers, as long as the handle is included in the monitor's recovery message (not scoped only to the alert condition).
- Page responders or teams directly from a Datadog incident, and set up automated notification rules that trigger a PagerDuty handle when incident conditions are met.
One documented limit matters for anyone building around this: Datadog caps monitor notification length at 1,024 characters when sending to PagerDuty, which can truncate the alert description before it reaches the page. A triage brief that reads the full monitor state from the API, rather than the truncated page text, avoids losing detail here.
None of this native wiring builds a persistent record of what happened during an incident, links the PagerDuty incident back to the specific Datadog monitors involved over time, or drafts a plain-language summary for the next person who opens the ticket. That's the gap this build fills.
Systems, APIs, and the app-agent-workflow split
Datadog: Monitors and Organizations
Datadog's REST API authenticates with an API key plus an application key, both sent as headers, verified through a dedicated authentication validation endpoint (see Datadog API authentication). The relevant object for this build is the Monitors API, which exposes endpoints to get a monitor's details, search monitors, and get all monitors, plus the Organizations object for tenant-level context.
Datadog also offers a Webhooks Integration for outbound event notifications, documented separately from the core API reference. The overview page does not publish a specific numeric rate limit; Datadog states that per-org quotas are documented separately, so confirm your account's limit before setting a polling interval.
Gotcha worth planning around: the 1,024-character truncation on PagerDuty-bound notifications means the API, not the page text, is the dependable source for monitor state and tags.
PagerDuty: Incidents and Services
PagerDuty's REST API runs over a single host, api.pagerduty.com, and uses an Authorization header carrying an account API token (PagerDuty's own format is Token token=<value>; see PagerDuty authentication). Account API tokens can be read-only or full-access and are tied to the account, not a user; only account administrators can generate them. A malformed or missing header returns an unauthorized error.
The incidents endpoint and the services resource are what this build reads. PagerDuty's REST API overview also notes the API is for managing configuration and incident data, not for sending monitoring events (that's the separate Events API, which Datadog already uses under the hood for paging).
PagerDuty documents a REST API rate limit of 960 requests per minute per account API key, user key, or app OAuth token, with rate-limit response headers (remaining count and reset time) and a 429 status when you exceed it (see PagerDuty rate limits). For event delivery, PagerDuty's current webhook model is v3 webhook subscriptions, created through the Webhook Subscriptions API, which can filter by service, team, or account and deliver a defined set of incident event types (an incident being triggered, acknowledged, resolved, and others) to an HTTP endpoint you register (see PagerDuty webhooks overview). Use a v3 subscription if your account supports it; otherwise fall back to a scheduled poll against the incidents endpoint, sized conservatively against the 960-request ceiling.
No regulation in Major's IT vertical research applies specifically to this process, so this build leans on general governance rather than a named compliance rule. If your organization layers SOC 2 or similar controls on top, the audit and access-control points below still apply.
Architecture: app, agent, workflow
The build splits cleanly into three layers, and keeping that split explicit is what keeps the system predictable.
- Workflow (deterministic): triggers on a new PagerDuty incident, checks for duplicates, calls the Datadog and PagerDuty APIs to pull monitor and service context, writes the raw facts to the app's tables, and routes the draft to a reviewer. None of this requires judgment, so none of it runs through the model twice.
- Agent (judgment only): reads the retrieved facts and writes a short, labeled summary. Its job is to group related alerts, separate what the data actually shows ("observed") from what might explain it ("possible"), and flag anything missing. It does not decide root cause, and it does not touch the incident's status.
- App (state and UI): holds the incident history, the linked Datadog context, the generated briefs, and the review trail, so a responder (or an auditor) can see exactly what was pulled, what the agent said, and who approved what.
A simple analogy: the workflow is the stock clerk who finds every box on the shelf and lines them up. The agent is the person who reads the labels and writes a one-paragraph note about what's there. Neither one decides what to do about a shortage. That's still the on-call engineer's call.

Datadog PagerDuty incident triage architecture on Major
How to build Datadog PagerDuty triage on Major
- Connect Datadog and PagerDuty as custom API connectors. Major's connector catalog does not list native Datadog or PagerDuty connectors today, so configure both as custom REST connectors: Datadog with the API key and application key held in the credential proxy, PagerDuty with an account API token sent as an
Authorizationheader. You should see both connectors listed as active with scoped, read-only credentials (PagerDuty's read-only account tokens are sufficient for this build). - Build the app and its tables. Create tables for incidents (PagerDuty incident ID, service, status, opened and resolved timestamps), monitor observations (Datadog monitor ID, status, tags, timestamp pulled), source links (URLs back to the Datadog monitor and PagerDuty incident), briefs (the generated summary text and version), and review events (reviewer, decision, timestamp). You should see a working table view where a new row in the incidents table appears the moment a PagerDuty incident opens.
- Add the agent with a narrow, bounded prompt. Give it only the retrieved facts, never raw API credentials or unrelated incidents, and instruct it to separate observed data from hypotheses and flag gaps explicitly.
"Using only the monitor observations and incident fields provided, write a 4-6 sentence triage brief. List the affected service, the monitor's current and recent status, and any other monitors tagged to the same service that changed in the last hour. Label facts from the data as 'observed'. If you suggest a likely cause, label it clearly as 'possible, not confirmed'. If data is missing, say so instead of guessing."
- Set the workflow trigger. Prefer a PagerDuty v3 webhook subscription filtered to incident-triggered events for the services you're covering; if your account doesn't support v3 subscriptions yet, schedule a poll against the incidents endpoint every 1-2 minutes, sized well under PagerDuty's 960-requests-per-minute limit. You should see the workflow fire within moments of a new incident during a test page.
- Test with a synthetic incident. Trigger a low-severity Datadog monitor in a test environment, confirm it opens a PagerDuty incident, and watch the workflow populate the incidents and monitor observations tables, then generate a brief. You should see a complete brief with linked source URLs before the real on-call rotation ever sees it.
- Roll out with scoped permissions. Grant the on-call team read access to the app and limit who can edit connector credentials or approve a brief that would write back to the incident, using SSO and role-based access. You should see only IT Operations and SRE leads able to change the Datadog or PagerDuty credentials.
A second useful prompt, for the app itself, when first describing the build to Major:
"Build an app with tables for PagerDuty incidents, Datadog monitor observations, source links, generated briefs, and review events. Show a table view per incident with its linked monitors and the latest brief, and a review button that records an approval or rejection."
Governance and IT control
Treat this build like any production integration touching incident data, because it is one.
- Scope the PagerDuty token to read-only account access wherever possible, since this workflow only needs to read incidents and services, not create or resolve them.
- Hold the Datadog API key and application key, and the PagerDuty token, in Major's credential proxy rather than in the workflow code or the agent's prompt, so no secret is ever visible to the model.
- Use SSO and role-based access to separate who can view briefs (the on-call rotation and their managers) from who can edit connector credentials or the agent's prompt (a smaller IT admin group).
- Log every API read, every generated brief, and every reviewer decision to an audit trail, so a later retrospective can see exactly what the agent saw and what a human approved.
- Require explicit human approval before any action writes back to the PagerDuty incident itself (a note, a status change, anything). The workflow can draft; only a person commits.
Failure handling and edge cases
Build for the ways this pipeline can go wrong, not just the happy path.
- API errors and rate limits. PagerDuty returns a 429 status with rate-limit-remaining and rate-limit-reset headers when you exceed 960 requests per minute; back off and retry using that reset value rather than a fixed delay. Datadog does not publish a fixed limit on its overview page, so monitor your own account's response headers and back off on repeated errors.
- Partial syncs. If the Datadog call succeeds but the PagerDuty service lookup fails (or vice versa), save what you have and mark the brief as partial rather than guessing at the missing half.
- Duplicate events. A PagerDuty webhook payload can resend messages if actions fire in quick succession; use the incident ID as a dedupe key before creating a new row or a new brief.
- Stale or truncated data. Because Datadog truncates PagerDuty-bound notifications at 1,024 characters, always pull monitor state from the API rather than trusting the page text alone.
- What the agent must never do. It must not declare a root cause, close or reassign an incident, or silently drop a missing-data warning. If it can't find monitor or service data, the brief should say so plainly instead of filling the gap with a guess.
- Fallback to a person. If the workflow or the agent errors out, or a reviewer rejects a brief, the incident still exists in PagerDuty exactly as before; nothing about this build changes what the responder sees if the automation fails.
What to measure
Track metrics the process owner already watches, without inventing new ones:
- Percentage of incidents that get a brief before the responder opens the ticket (coverage).
- Percentage of briefs with complete source links to both the Datadog monitor and the PagerDuty incident (linked-source completeness).
- How often a responder edits or rejects the generated brief versus accepting it as-is (acceptance rate).
- Workflow error rate: failed API calls, timeouts, and partial syncs per week.
This build does not claim a time-saved number. Measure coverage and acceptance first, and let the team's own before-and-after judgment decide whether it's working.

Incident context data flow and approval gate
Before you go live
- [ ] Datadog API key and application key scoped and stored in the credential proxy
- [ ] PagerDuty account token set to read-only access where possible
- [ ] Webhook subscription (or polling schedule) tested against a synthetic incident
- [ ] Dedupe logic confirmed against repeated or out-of-order webhook messages
- [ ] Agent prompt reviewed to confirm it separates "observed" from "possible" and never proposes an incident action
- [ ] Reviewer approval step required before any writeback to the PagerDuty incident
- [ ] RBAC confirmed: who can view briefs versus who can edit connector credentials
- [ ] Audit log checked for API reads, generated briefs, and reviewer decisions
Start building on Major
Build a governed incident-context app that turns repeatable Datadog-to-PagerDuty gathering into deterministic steps, while the agent drafts an evidence-linked brief for an on-call engineer to review.
Related articles
- Automate Ramp-to-NetSuite Expense Coding with AI Review
- Automate Snowflake Board Metrics Briefs with Slack
- Greenhouse BambooHR Integration: Automate New-Hire Onboarding
About the author
Ritvij Gautam is Head of Growth at Major.
Last reviewed November 7, 2026 by Ritvij Gautam.
Frequently asked questions
- How does Datadog integrate with PagerDuty?
- Datadog's native PagerDuty integration routes monitor alerts to a PagerDuty service using a notification handle, and can declare or auto-resolve a Datadog incident based on PagerDuty paging activity. That covers paging and severity mapping. A separate triage-enrichment layer, built on Major, pulls monitor and service context after an incident opens and drafts a brief for the responder; it does not replace the native routing.
- Can Datadog alerts create or resolve PagerDuty incidents?
- Datadog can declare an incident from a PagerDuty alert with severity mapped automatically from the monitor's status, and can auto-resolve the PagerDuty incident when the monitor recovers, provided the PagerDuty handle is included in the monitor's recovery message rather than scoped only to the alert condition. See Datadog's PagerDuty integration documentation for the exact configuration.
- What should an AI incident triage brief include?
- A useful brief names the incident and affected service, lists the relevant Datadog monitor states and timestamps, links back to the source records in both systems, and clearly separates observed facts from possible explanations. It should flag missing data rather than guessing, and it should end with the brief awaiting human review, not an automatic action.
- Should an AI agent resolve PagerDuty incidents automatically?
- No. Resolution should stay with the responder or with an explicitly authorized deterministic rule, such as Datadog's documented auto-resolve behavior tied to a monitor recovering. A generated summary is not evidence that an incident is resolved, and an agent should never close, reassign, or reprioritize an incident based on its own interpretation of the data.
- Do I need a native Major connector for Datadog or PagerDuty?
- No. Neither tool has a listed native connector in Major's catalog today, so both are configured as custom REST API connectors: Datadog using an API key plus an application key sent as headers, and PagerDuty using an account API token sent in the Authorization header. Credentials are held in Major's credential proxy rather than in workflow code or agent prompts.
- How often should the workflow check for new PagerDuty incidents?
- A PagerDuty v3 webhook subscription filtered to incident-triggered events is preferable, since it delivers new incidents immediately without polling. If webhook subscriptions aren't available on your account, a scheduled poll every one to two minutes against the incidents endpoint works, as long as the request volume stays well under PagerDuty's documented 960-requests-per-minute rate limit.