How to Build an "On-Call Agent" using PagerDuty & GPT-4o

1. The Architecture

We will build a simple middleware service using Python (Flask) that connects three APIs. Here is the data flow:

• Trigger: PagerDuty sends a webhook when an incident is triggered.
• Context: Our Python script uses the PagerDuty API to fetch the latest log_entries for that incident.
• Brain: We send those logs to OpenAI's GPT-4o API with a specialized system prompt.
• Action: The script posts the AI's diagnosis to Slack using the chat.postMessage method.

2. Prerequisites

Before writing code, ensure you have the following API keys:

• PagerDuty: An API Key (read-only is fine for fetching logs) and a Generic Webhook V3 URL (we will generate this in step 4).
• OpenAI: An API Key with access to the gpt-4o model.
• Slack: A Bot User OAuth Token with chat:write permissions.

3. The Code (Python)

Create a file named app.py. We will use Flask to accept the webhook and the OpenAI client to process the data.

import os
from flask import Flask, request, jsonify
from openai import OpenAI
from slack_sdk import WebClient
import pdpyras

app = Flask(__name__)

# Configuration
PD_API_KEY = os.getenv("PD_API_KEY")
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
SLACK_BOT_TOKEN = os.getenv("SLACK_BOT_TOKEN")
SLACK_CHANNEL = "#incidents-ai"

# Initialize Clients
pd_session = pdpyras.APISession(PD_API_KEY)
ai_client = OpenAI(api_key=OPENAI_API_KEY)
slack_client = WebClient(token=SLACK_BOT_TOKEN)

@app.route('/webhook', methods=['POST'])
def pagerduty_webhook():
    data = request.json
    messages = data.get('messages', [])
    
    for message in messages:
        if message['event'] == 'incident.trigger':
            incident_id = message['incident']['id']
            handle_trigger(incident_id)
            
    return jsonify({"status": "received"}), 200

def handle_trigger(incident_id):
    # 1. Fetch Logs from PagerDuty
    # Note: detailed logic omitted for brevity, but you would use
    # /incidents/{id}/log_entries to get the error details.
    logs = pd_session.rget(f"/incidents/{incident_id}/log_entries")
    error_snippet = extract_error_from_logs(logs)

    # 2. Ask GPT-4o for Analysis
    response = ai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a Senior SRE. Analyze the following error logs and provide 3 probable root causes and a recommended fix."},
            {"role": "user", "content": f"Incident {incident_id} Logs: {error_snippet}"}
        ]
    )
    analysis = response.choices[0].message.content

    # 3. Post to Slack
    slack_client.chat_postMessage(
        channel=SLACK_CHANNEL,
        text=f"*Auto-Analysis for Incident {incident_id}*\n\n{analysis}"
    )

def extract_error_from_logs(logs):
    # Simple helper to grab the first 1000 chars of the trigger log
    return str(logs[0]['channel']['details'])[:1000]

if __name__ == '__main__':
    app.run(port=5000)

This script sets up a listener on /webhook. When PagerDuty hits it, we extract the incident ID, query PagerDuty for the full log context, and pass that to GPT-4o.

4. The GPT-4o System Prompt

The magic is in the prompt. You don't want a generic answer; you want an SRE's opinion. Use this prompt structure:

5. Next Steps: From "Read Only" to "Action"

Once you trust the agent, you can upgrade it. Instead of just posting to Slack, give the agent tools (using OpenAI Function Calling) to execute safe commands:

• Restart Pod: kubectl delete pod {pod_name}
• Clear Cache: redis-cli flushall
• Scale ASG: Increase desired capacity by 1.

This moves you from "Observability" into the true "Self-Healing Enterprise" we discussed in our pillar guide.

Frequently Asked Questions (FAQ)

Sources & Recommended Reading

• PagerDuty Developer Documentation: Webhooks v3
• OpenAI API: GPT-4o Model Capabilities
• Slack API: chat.postMessage Method
• Google SRE Book: Eliminating Toil