Triggering Firetiger from Datadog Alerts

When a Datadog monitor alerts, you can have Firetiger automatically start working on it. Datadog posts the alert to a Firetiger HTTP endpoint, and Firetiger turns the alert into either an investigation or a custom agent run — depending on which endpoint you point the Datadog webhook at.

Both use Datadog’s built-in Webhooks integration, so they work with any monitor type (metric, log, APM, anomaly, etc.) and require no code on your side.

Investigation vs. agent: which one do you want?

These are two different things and they work differently. Most teams connecting Datadog alerts want investigations.

  Fire an investigation Fire a custom agent
What it creates A first-class Investigation — a scoped diagnostic run that turns your telemetry into an explanation and, when warranted, opens a durable Issue A plain agent session for an agent you configured
Best for “Something alerted — figure out what happened and why.” Generic, no per-agent setup. A purpose-built agent you’ve already set up with specific instructions, connections, or a runbook
Endpoint POST …/InvestigationService/CreateInvestigation (the API) POST https://api.cloud.firetiger.com/webhooks/{token} (the agent’s webhook)
Auth API key (HTTP Basic) The opaque token in the URL — no API key
Request body Structured JSON: an investigation plus an initial_activities message that carries the prompt Any payload — passed through to the agent verbatim
Shows up in The Investigations surface in the UI; can escalate to an Issue The agent’s page (Sessions)

If you’re not sure, start with an investigation — it’s the diagnostic primitive and needs no agent setup. The two paths are documented separately below; you can wire up either or both.

An investigation is created through the Firetiger API. You pass the alert’s details as an initial activity — a seed message — which is what the agent reads as its opening prompt, so it immediately starts diagnosing: pulling the relevant logs, correlating traces and metrics, and identifying what changed. (The description field is investigation metadata only — it does not reach the agent, so the prompt must go in initial_activities.)

1. Create an API key

The investigation endpoint authenticates with an API key (HTTP Basic auth). In Firetiger, go to Settings → API keys, create a Read-write key (creating an investigation is a write), and copy the credentials. See API Keys for details.

The dialog gives you a username, a password, and a pre-computed Authorization: Basic <base64> header — you’ll paste that header into Datadog.

2. Create the webhook in Datadog

  1. In Datadog, open the Webhooks integration tile (Integrations → Webhooks) and click New.

  2. Fill in:

    Field Value
    Name firetiger-investigation (becomes the @webhook-firetiger-investigation handle)
    URL https://api.cloud.firetiger.com/firetiger.investigations.v1.InvestigationService/CreateInvestigation

  3. Under Custom Headers, add your API-key Basic auth header (Datadog takes headers as JSON):

    {
  "Authorization": "Basic <your-base64-credentials>",
  "Content-Type": "application/json"
}

  4. Enable Custom Payload and provide the investigation body. Datadog substitutes its $-prefixed template variables before sending. The agent’s prompt is the content string inside initial_activities[].user.text. You don’t need to set display_name — when the seed message is present the server generates the investigation’s title from it.

    {
  "investigation": {},
  "initial_activities": [
    {
      "user": {
        "text": {
          "content": "A Datadog monitor alerted ($ALERT_TYPE). Investigate the root cause and affected scope.\n\nMonitor ID: $ALERT_ID\nStatus: $ALERT_STATUS\nHost: $HOSTNAME\nTags: $TAGS\nDatadog link: $LINK\n\nDetails:\n$EVENT_MSG",
          "role": "USER"
        }
      }
    }
  ]
}
  5. Click Save.

The richer the seed message, the faster the agent orients itself. Embedding $LINK lets it (and you) jump back to the Datadog event.

3. Reference the webhook in your monitor

See Wiring the webhook into a monitor below — add @webhook-firetiger-investigation to the monitor message.

4. What you get back

CreateInvestigation returns the created investigation, including its name and status:

{
  "investigation": {
    "name": "investigations/inv_abc123",
    "displayName": "Elevated error rate in payments service",
    "status": "INVESTIGATION_STATUS_EXECUTING",
    "createTime": "2024-06-15T14:30:00Z"
  }
}

The investigation shows up in the Investigations surface in the UI, where you can watch it run. You can also poll it programmatically with GetInvestigation — see the Investigations API reference.

Path B: Fire a custom agent

Use this when you’ve already built an agent for a specific job (with its own instructions, connections, or runbook) and want a Datadog alert to kick it off. Every Firetiger agent has a manual trigger with a stable webhook URL; POSTing to it starts a session with the alert payload as the opening message. See Agent Webhooks for the full reference.

1. Get the agent’s webhook URL

Open the agent in Firetiger, look at the Triggers section, and copy the Webhook URL:

https://api.cloud.firetiger.com/webhooks/dGhpc0lzQVRlc3RUb2tlbjEyMzQ1Njc4OTA

The opaque token in the URL is the credential — no API key or Authorization header is required. Treat the URL as a secret.

2. Create the webhook in Datadog

  1. In Datadog, open Integrations → Webhooks → New.

  2. Fill in:

    Field Value
    Name firetiger-agent (becomes the @webhook-firetiger-agent handle)
    URL The agent webhook URL you copied above

  3. Enable Custom Payload. The body is passed to the agent as-is, so include whatever context is useful:

    {
  "alert_title": "$EVENT_TITLE",
  "alert_type": "$ALERT_TYPE",
  "alert_status": "$ALERT_STATUS",
  "monitor_id": "$ALERT_ID",
  "host": "$HOSTNAME",
  "tags": "$TAGS",
  "link": "$LINK",
  "body": "$EVENT_MSG",
  "message": "A Datadog monitor has alerted. Investigate the cause: check the relevant logs, traces, and metrics around the time of this alert and identify what changed."
}
  4. Click Save.

No Authorization header is needed — the URL token authenticates the request.

3. Reference the webhook in your monitor

Add @webhook-firetiger-agent to the monitor message (see below).

4. What you get back

A successful invocation returns the created session name:

{
  "session": "agents/uvxtwa1yoyrx/sessions/sess456"
}

The agent starts working immediately. It receives a structured message containing the timestamp, the request headers (a few, like X-Forwarded-For, are omitted), and the payload body (truncated to 50 kB if larger). You can watch the session on the agent’s page in the UI.

Wiring the webhook into a monitor

This step is the same for both paths. A webhook only fires when a monitor’s notification message mentions its handle. Edit the monitor (or create one) and add the handle to the notification message:

{{#is_alert}}
@webhook-firetiger-investigation
Error rate on checkout exceeded threshold.
{{/is_alert}}

The handle (@webhook-firetiger-investigation or @webhook-firetiger-agent) matches the Name you gave the webhook. Wrapping it in a matched {{#is_alert}} … {{/is_alert}} block (and optionally a separate {{#is_warning}} … {{/is_warning}} block) means Firetiger is only invoked on the transitions you care about — not on recovery or no-data, unless you want that too. Datadog requires these conditional section tags to be opened and closed in matching pairs.

Save the monitor. Use Test Notifications in the monitor editor to fire a sample alert and confirm an investigation (or session) shows up in Firetiger.

Tips

  • One webhook, many monitors. A single handle can be referenced from any number of monitors. Include $ALERT_ID and $TAGS in the payload so the run can tell which monitor and service fired.
  • Route alert classes differently. Create separate webhooks (e.g. firetiger-db pointed at a DB-specialist agent) and reference whichever handle fits the monitor.
  • Scope the work by writing an explicit seed message — the initial_activities text (Path A) or the message field (Path B) — naming the service, the suspected subsystem, or a runbook to follow.
  • Use $TAGS[key] to pull a single tag value, e.g. $TAGS[service].
This site uses Just the Docs, a documentation theme for Jekyll.