Functions
Multi-Metric Arguments
Every function supports a variable number of arguments. Both of the following expressions are valid:
ts_average(test.gauge_metric.*) ts_average(test.gauge_metric.a, test.gauge_metric.b, test.gauge_metric.c)
Returned Values
Functions return either a list of metrics or a single time series metric. Functions that return a list of metrics are prefixed with either series_, gauge_ or increment_. Functions that will only ever return a single time series are prefixed with ts_.
Function Types
Aggregate functions group values together, similar to SQL:
- Average ts_average()
- Sum ts_sum()
- Count gauge_count()
- Total gauge_total()
- Integral series_integral()
- Increment Average increment_average()
- Moving Average series_moving_average()
Filter functions provide focus by removing (and sometimes adding) data points:
- Max ts_max()
- Min ts_min()
- Clamp series_clamp()
- Top N / Bottom N series_top_n(), series_bottom_n()
- Continuous series_continuous()
Transformative functions aggregate values, but then perform further calculations:
- Normalize series_normalize()
- Derivative series_derivative()
- Deviation series_deviation()
- Present series_present()
- Growth Rate growth_rate()
Miscellaneous functions:
Average
ts_average(metric_pattern, ...)
Returns a single time series. The value at each point of the returned time series is the average of the value at the identical point amongst all the provided time series.
Showing the single average of multiple time series can be useful when measuring the general trend of a value among many metrics.
ts_average(webserver.*.request_time) as average_request_time
Sum
ts_sum(metric_pattern, ...)
Returns a single time series. The value at each point of the returned time series is the sum of all values at the identical point amongst the provided time series.
The sum of points from many time series can be useful when rolling up many metrics into a single value.
ts_sum(web.*.request_count) as all_web_requests
Count
gauge_count(metric_pattern, ...)
Returns a list of time series whose size is equal to the number of metrics matching the arguments provided. The value of each point in those time series corresponds to the number of times that metric was reported to Instrumental.
Typically this function would be used to graph the number of times a gauge metric was reported to Instrumental, like:
gauge_count(web.request_time) as num_requests
Total
gauge_total(metric_pattern, ...)
Returns the a list of time series whose size is equal to the number of metrics matching the arguments provided. The value of each point in those time series corresponds to the sum of values recorded for that metric in Instrumental at a given point in time.
This function can be useful when attempting to find the maximum value for a gauge metric at a given point in time.
gauge_total(database.query_time) as total_query_time
Integral
series_integral(metric_pattern, ...)
Returns a list of time series whose values begin at 0 and then cumulatively add the current points value to the previous value at each point.
The integral is useful in visualizing the accumulated total of a given metric over a period of time.
series_integral(web.ads.interstitial.clickthrough)
Increment Average
increment_average(metric_pattern, ...)
Returns a list of time series whose size is equal to the number of metrics matching the arguments provided. The value of each point in those time series corresponds to the average of the total values received by Instrumental for that metric at a given point.
Beneficial when trying to find the average of an increment metric, you can use this when trying to measure the rate of a metric that is always increasing.
increment_average(emails.delivered) as avg_emails_delivered
Moving Average
series_moving_average(window, metric_pattern, ...)
Returns the moving average for each of the provided time series, where the number of previous steps examined is equal to the window argument.
The moving average is a useful function to filter extremely noisy datasets, letting you more easily visualize changes in values.
series_moving_average(10, server.db.num_active_threads)
Clamp
series_clamp(min, max, metric_pattern, ...)
Returns a list of time series where the value of each data point is clamped between the provided minimum and maximum values.
This can be used to remove unwanted spikes or dips in data that negatively impact data display or other calculations.
series_clamp(0, 1000, worker.*.jobs_completed)
Max
ts_max(metric_pattern, ...)
Returns a single time series. The value at each point of the returned time series is the highest value at the identical point of all the provided time series.
The largest value among many metrics is useful when establishing an upper bound of a given set of metrics.
ts_max(external_service.*.latency) as max_connection_latency
Min
ts_min(metric_pattern, ...)
Returns a single time series. The value at each point of the returned time series is the lowest value at the identical point of all the provided time series.
The minimum value of many metrics is useful when determining the lower bound of a given set of metrics.
ts_min(worker.*.jobs_processed) as slow_job_processors
Top N / Bottom N
series_top_n(amount, metric_pattern, ...)
series_bottom_n(amount, metric_pattern, ...)
Returns a list of up to amount time series. The list will contain the series with largest average values for series_top_n() or smallest average values in the case of series_bottom_n().
series_top_n(5, user.*.comments_posted) as chattiest_users series_bottom_n(3, client.*.posts_submitted) as least_submitting_clients
Continuous
series_continuous(initial_default, metric_pattern, ...)
Makes the provided set of metrics continuous. Any point in a metric which is not present is filled in with the value of the previous point. If the initial point in the metric is not present, the initial default value is used.
This can be used to remove unwanted segments of missing data that would otherwise negatively impact data display or other calculations
series_continuous(100, worker.*.disk.free_percent)
Normalize
series_normalize(metric_pattern, ...)
Returns a list of time series where the value of each data point normalized between 0 and 1
This can be used to scale disparate time series to the same range, for easier comparison.
series_normalize(worker.*.jobs_completed)
Derivative
series_derivative(metric_pattern, ...)
Returns a list of time series whose values represent the amount of change between each point and its previous point.
The derivative can be useful when visualizing how active a particular dataset can be, as well as using it to compute further values based on the change taking place.
series_derivative(mobile.apple.iap.revenue)
Deviation
series_deviation(metric_pattern, ...)
Returns a list of time series whose size is equal to the number of metrics matching the arguments provided. The value of each point in those time series corresponds to the difference of that given point from the average of the time series over that time series' duration.
The deviation from the average of the time series can be useful when visualizing how erratic a given metric may be.
series_deviation(web.*.cpu.in_use) as cpu_use_deviation
Present
series_present(metric_pattern, ...)
Transforms the provided list of time series into a list of time series where the value at each point is 0 if there is no data at that point, and 1 otherwise.
This can be useful if you need to alert on if a metric is being reported or not - for instance, if a system goes down and stops reporting a given metric.
series_present(worker.*.cpu.in_use) as is_worker_alive
Growth Rate
growth_rate(metric_pattern, ...)
Returns the rate of growth in a cumulative (always growing) metric. Often called "counters", these metrics are frequently used to report server and system data. To prevent a sudden drop in the metric from dramatically skewing output, growth_rate() will return 0 when gaps or decreases in the metric are detected.
This contrasts with series_derivative(), which returns extreme values when confronted with drops in metric value (such as during a server restart). These extremes are likely to obscure useful information by skewing graph ranges. growth_rate() filters out these extreme value changes by simply returning 0, resulting in data that is much more likely to be useful when graphed. See the example graphic for an illustration of the difference between the two functions.
growth_rate(network.received.bytes)
Constant
constant(value)
Returns a metric that has the same value at all times.
Typically this function would be used show a threshold value, like:
constant(5) as alert_threshold
Log
log(base_num, metric_pattern, ...)
log10(metric_pattern, ...)
Returns a list of time series whose values are the base_num logarithm of their original value, or in the case of log10(), the base 10 logarithm.