Alerts

Alerts are triggered when a query language expression is matched. Your expression will be evaluated over the time interval specified and you will be notified via email, text message, or an HTTP callback.

Integrating with PagerDuty, OpsGenie, and other services

Email is the best method of integrating Instrumental alerts into PagerDuty and other incident response tools. Just create a new email integration within your preferred service and add that email address to your Instrumental alert configuration (alert notifications can be sent to multiple email addresses).

If your service supports regex email parsing, it's easy to open and close issues as Instrumental alerts open and close. An example PagerDuty configuration would be:

  • Trigger an Alert if...
    • Subject contains "Alert Opened"
    • Incident Key matches this regular expression: "Opened: (.*) at \d\d?:\d\d"
  • Resolve an Alert if...
    • Subject contain "Alert Closed"
    • Incident Key matches this regular expression: "Closed: (.*) at \d\d?:\d\d"
  • Ignore emails that don't match the above conditions. We send a test email each time you create a new alert. If you don't ignore unmatched emails, these test emails will open an issue in PagerDuty every time you create a new alert in Instrumental. Alternatively, you can create the alert and add the PagerDuty email address to the alert configuration after we send the initial test email.

For reference, our alert notification subject line format is:
[Project Name] Alert Opened: [Alert Name] at 8:31 AM Wed Dec 7

HTTP Callback Specification

You can specify an endpoint URL when you create an alert, and when an alert occurs a HTTP POST request will be issued to the URL.

The parameters will be sent as application/x-www-form-urlencoded string value pairs. The cause field will contain JSON encoded data you may decode as necessary. The following requests will be made to your URL in cases where an OPEN, CLOSE or TEST alert event occurs.

Open

alert_id=ALERT_ID&
alert_config_id=ALERT_CONFIG_ID&
project_id=PROJECT_ID&
project_name=PROJECT_NAME&
name=ALERT_CONFIG_NAME&
opened_at=WHEN_ALERT_OPENED_UNIX_TIMESTAMP&
cause=JSON_DICTIONARY_OF_METRICS_CAUSING_ALERT&
state=open

Close

alert_id=ALERT_ID&
alert_config_id=ALERT_CONFIG_ID&
project_id=PROJECT_ID&
project_name=PROJECT_NAME&
name=ALERT_CONFIG_NAME&
oepned_at=WHEN_ALERT_OPENED_UNIX_TIMESTAMP&
closed_at=WHEN_ALERT_CLOSED_UNIX_TIMESTAMP&
cause=JSON_DICTIONARY_OF_METRICS_CAUSING_ALERT&
state=closed

Test

alert_id=ALERT_ID&
alert_config_id=ALERT_CONFIG_ID&
project_id=PROJECT_ID&
project_name=PROJECT_NAME&
name=ALERT_CONFIG_NAME&
opened_at=WHEN_ALERT_OPENED_UNIX_TIMESTAMP&
closed_at=WHEN_ALERT_CLOSED_UNIX_TIMESTAMP&
cause=EMPTY_JSON_DICTIONARY&
state=test

NOTE: If your server cannot be reached in 10 seconds, the request will time out and be abandoned.