Instrumental Query Language

When creating graphs or submitting API requests, query language expressions are used to perform complex queries on your data such as:

  • Retrieve a set of metrics matching a particular pattern
  • Find the deviation of a metric versus its average value
  • Get the average value of the top 25 metrics matching a pattern

These expressions can be simple or very complex. Complex expressions are often the most useful and they'll usually make use of these features:

Basic Expression

The most basic expression is simply the name of a single metric, which results in a line graph of that metric's aggregated datapoints:

production.db001.cpu

Advanced Expression

But what if you wanted to monitor the average database CPU across all of your servers and compare it to your historical average? This single expression demonstrates the power of combining wildcard groups, aliases, time shifting, and functions. You'd add the average of all the CPU metrics:

ts_average(db-*.cpu.in_use)

And then add the average again, but with a 24 hour lookback:

ts_average(db-*.cpu.in_use @ 24 hours) as cpu_24_hours_ago

A graph with this expression looks like:

Query language