Skip to content

About

Clients for triggering incidents, e.g., Pagerduty

Resources

Stars

0 stars

Watchers

1 watching

Forks

Repository files navigation

Incident Clients Gradle Check

Implementation of incident clients for various notification systems.

Supported Clients

Client Module Java Module
PagerDuty V2 Events API incident-pagerduty software.sava.incident_pagerduty
incident.io Incidents V2 incident-io software.sava.incident_io
incident.io HTTP Alert Sources incident-io software.sava.incident_io
Generic Webhook / Slack / Telegram incident-webhook software.sava.incident_webhook

All depend on incident-core (software.sava.incident_core), which provides the provider-neutral IncidentClient API described below.

Usage

PagerDuty

Full example

final var client = PagerDutyEventClient.clientBuilder()
    .defaultClientName("CLIENT_NAME")
    .defaultRoutingKey("INTEGRATION_KEY")
    .createClient();

final var payload = PagerDutyEventPayload.build()
    .summary("ex-summary")
    .source("ex-source")
    .severity(PagerDutySeverity.critical)
    .timestamp(ZonedDateTime.now(UTC))
    .component("ex-component")
    .group("ex-group")
    .eventClass("ex-class")
    .customDetails("ex-num-metric", 1)
    .customDetails("ex-boolean", true)
    .customDetails("ex-string", "val")
    .link(PagerDutyLinkRef.build()
        .href("https://github.com/sava-software/incident-client")
        .text("Sava Incident PagerDuty Event Client")
        .create())
    .image(PagerDutyImageRef.build()
        .src("https://www.pagerduty.com/wp-content/uploads/2016/05/pagerduty-logo-green.png")
        .href("https://www.pagerduty.com/")
        .alt("pagerduty")
        .create())
    .create();

final var triggerResponseFuture = client.triggerDefaultRouteEvent(payload);

final var changeEventPayload = PagerDutyChangeEventPayload.build(payload).create();
final var changeEventResponseFuture = client.defaultRouteChangeEvent(changeEventPayload);

final var triggerResponse = triggerResponseFuture.join();

final var ackResponse = client.acknowledgeEvent(triggerResponse.dedupKey()).join();

final var resolveResponse = client.resolveEvent(triggerResponse.dedupKey()).join();

final var changeEventResponse = changeEventResponseFuture.join();

PagerDuty limits Events API payloads to 512 KB, so the client refuses any request body over PagerDutyEventClient.MAX_REQUEST_BODY_BYTES (524,288 UTF-8 bytes) with an IllegalArgumentException before sending it.

The Events API authenticates with the routing key alone, so the client sends no Authorization header; a custom endpoint that needs one sets it with extendRequest.

incident.io

Full example

idempotencyKey and visibility are required by the API; a request without them is rejected. Severities, incident types, statuses, roles, and custom fields are all workspace-specific ids — look them up in your incident.io workspace.

try (final var httpClient = HttpClient.newHttpClient()) {
  final var client = IncidentIoClient.clientBuilder()
      .bearerToken("BEARER_TOKEN")
      .httpClient(httpClient)
      .createClient();

  final var request = CreateIncidentRequest.requestBuilder()
      .idempotencyKey(UUID.randomUUID().toString())
      .name("Test Incident")
      .summary("Test Java client")
      .visibility(CreateIncidentRequest.Visibility.PRIVATE)
      .mode(CreateIncidentRequest.Mode.test)
      .severityId("SEVERITY_ID")
      .incidentTypeId("INCIDENT_TYPE_ID")
      .incidentRoleAssignments(List.of(new CreateIncidentRequest.IncidentRoleAssignment(
          "INCIDENT_ROLE_ID",
          CreateIncidentRequest.UserReference.byEmail("responder@example.com"))))
      .customFieldValues(Map.of("CUSTOM_FIELD_ID", "value"))
      .build();

  final var response = client.createIncident(request).join();

  System.out.println(response.reference());
  System.out.println(response.permalink());
}

customFieldValues is sugar for text custom fields; select, catalog, link, and numeric fields use customFieldEntries with a matching CustomFieldValue — see the module README. slackChannelNameOverride, incidentTimestampValues, and retrospectiveIncidentOptions round out IncidentsCreatePayloadV2.

incident.io HTTP alert sources

Full example

An HTTP alert source deduplicates on a key and resolves with the same key; the workspace's alert routes decide whether the alert becomes an incident. The client authenticates with the alert source's own secret, not an API key.

final var client = IncidentIoAlertEventClient.clientBuilder()
    .alertSourceConfigId("ALERT_SOURCE_CONFIG_ID")
    .token("ALERT_SOURCE_SECRET")
    .createClient();

client.sendAlertEvent(AlertEvent.build()
    .title("Disk full on db-1")
    .status(AlertEvent.Status.firing)
    .deduplicationKey("db-1-disk")
    .metadata("service", "db")
    .create()).join();

client.sendAlertEvent(AlertEvent.build()
    .title("Disk space recovered on db-1")
    .status(AlertEvent.Status.resolved)
    .deduplicationKey("db-1-disk")
    .create()).join();

See the module README for the metadata value model and the provider-neutral adapter.

Generic Webhook / Slack / Telegram

Full example

incident-webhook POSTs a structured message to a single configured webhook endpoint — fire-and-forget notification, not incident management. WebhookFormats.GENERIC_JSON (provider id webhook) sends a canonical JSON document of the alert for receivers that do their own mapping; WebhookFormats.SLACK_TEXT (provider id slack) sends a plain {"text":"..."} Slack incoming-webhook message; TelegramTextFormat (provider id telegram) sends a Bot API sendMessage body to https://api.telegram.org/bot<TOKEN>/sendMessage with a configured chatId and, optionally, a forum topic (messageThreadId) and severities sent silently (silentSeverities); its link previews are always disabled, and an alert to a basic group that was upgraded to a supergroup is sent once more to the supergroup, with a WARNING logged (see the module README).

final var client = WebhookClient.clientBuilder()
    .endpoint("https://hooks.slack.com/services/T000/B000/XXXX")
    .createClient();

final IncidentClient incidentClient = client.incidentClient(WebhookFormats.SLACK_TEXT);

Provider-Neutral Usage

Service-level code can be written against IncidentClient and switch providers via configuration. IncidentClients creates a client from configuration alone — provider modules on the module or class path register themselves via ServiceLoader, and the provider config value selects one:

incident.provider=pagerduty
incident.routingKey=INTEGRATION_KEY
incident.provider=incident.io
incident.bearerToken=BEARER_TOKEN
incident.visibility=private
incident.severityIds.CRITICAL=SEVERITY_ID
incident.provider=incident.io-alerts
incident.token=ALERT_SOURCE_SECRET
incident.alertSourceConfigId=ALERT_SOURCE_CONFIG_ID
incident.resolveTitle=Recovered
final IncidentClient incidentClient = IncidentClients.createClient(properties, "incident");

The JSON equivalent wraps the provider's config object, with provider first:

{"provider": "pagerduty", "config": {"routingKey": "INTEGRATION_KEY"}}

Adapters can also be created in code from a native client:

final IncidentClient incidentClient = usePagerDuty
    ? pagerDutyEventClient.asIncidentClient()
    : incidentIoClient.incidentClientBuilder()
        .visibility(CreateIncidentRequest.Visibility.PRIVATE)
        .severityId(IncidentSeverity.CRITICAL, "SEVERITY_ID")
        .createClient();

final var alert = IncidentAlert.build()
    .summary("Validator missed its leader slot")
    .details("No block produced for slot 350000000.")
    .severity(IncidentSeverity.CRITICAL)
    .source("validator-01.example.com")
    .timestamp(ZonedDateTime.now(UTC))
    .customDetail("slot", 350000000L)
    .create();

final var response = incidentClient.reportIncident(alert).join();

if (incidentClient.supportsResolve()) {
  incidentClient.resolveIncident(response.key()).join();
}

A correlation key longer than the client's declared limit (IncidentClient#maxKeyLength(): 255 characters for PagerDuty, none for incident.io and the webhook formats) reaches the provider shortened by IncidentClient.boundKey, to the key's first characters, a colon and the SHA-256 of the whole key, the same way on every report and resolve, so an alert may be keyed by whatever identifies its subject and resolved by that key or by response.key().

IncidentService.createService(incidentClient) wraps a client with retries driven by a retry-delay function. When a provider asks for a wait before the next attempt, such as a 429's Retry-After (in seconds, or as a date measured from the response's own Date header), the service waits it out before retrying, up to a limit of one minute by default (IncidentService.DEFAULT_MAX_RETRY_AFTER); a longer wait is not served: the call ends with the provider's failure after the failed attempt's own retry delay, whatever the retry count, so an alert is never held for an hour in silence and the caller can fall back. IncidentService.createService(incidentClient, maxRetryAfter) sets another limit, and Duration.ZERO refuses every such wait. A client that throws where it should return a failed future, as the PagerDuty adapter does for a request body over its size limit, fails the future the service returns, so a refusal by the client arrives the way a failed request does.

The incident.io adapter maps IncidentSeverity values onto workspace severity ids supplied at build time, and has no programmatic resolve: supportsResolve() returns false and resolveIncident(String) fails the returned future with an UnsupportedOperationException. It also needs an incidentTimestampId before it can send IncidentAlert#timestamp() — incident.io timestamps are workspace ids — and carries source() and customDetails() as text appended to the incident summary, since incident.io custom fields are workspace schema objects rather than an arbitrary key/value map. Provider-specific features — PagerDuty links, images, and change events; incident.io custom fields and role assignments — remain on the provider clients.

The incident.io alert-source adapter (incident.io-alerts) sends a firing event keyed by the alert's key, with severity, source, timestamp and custom details in its metadata, and resolves by sending a resolved event with the same key. That resolve addresses the alert's lifecycle: whether an incident is created from the alert, and whether it closes when the alert resolves, depends on the workspace's alert routes, so supportsResolve() does not promise that an incident closes. Every event needs a title, and nothing upstream says whether a resolve event's title replaces the alert's, so a resolve sends the configured resolveTitle, or the key when none is set.

About

Clients for triggering incidents, e.g., Pagerduty

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages