When designing a Grafana dashboard, one of the most insightful metrics is the percentile of request latencies, such as the 95th percentile (P95). P95 provides a clear picture of the worst-case scenario that most users experience, making it an essential tool for performance monitoring. However, an issue can arise where the P95 metric consistently shows an unexpected value, such as 4.75 seconds across all routes, which may not accurately reflect the actual performance of the application.
The Problem: 4.75s Across the Board
In some cases, when setting up a Grafana dashboard with Prometheus as the data source, all routes might display a consistent P95 value of 4.75 seconds, regardless of the actual load or traffic patterns. This anomaly can be puzzling, especially when requests to the service are generally quick, well under 5 seconds. The root cause of this issue often lies in how histogram buckets are configured in Prometheus.
Prometheus allows users to define histogram buckets with specific boundaries. These buckets collect counts of observations that fall within each boundary. For instance, if buckets are defined at 1s, 2s, 3s, and so on up to 5s, Prometheus will count how many requests took less than 1s, how many took between 1s and 2s, and so forth.
If a request takes 2 seconds, it will increment the counters for all buckets with an upper boundary greater than 2 seconds (e.g., <3s, <4s, etc.).
Understanding Percentiles
The 95th percentile (P95) is a value below which 95% of the data points in a dataset fall. In the context of request latencies, P95 represents a threshold where 95% of the requests have a latency equal to or lower than this value.
How Prometheus Calculates P95 Using Histograms
Prometheus uses histograms to calculate percentiles like P95. This is the step-by-step process of finding P95,
Histogram Setup:
A histogram is a collection of buckets, each representing a range of values.
For example, if you have buckets at 1s, 2s, 5s, and 10s, they represent the intervals:
<1s1s ≤ latency < 2s2s ≤ latency < 5s5s ≤ latency < 10s≥10s
Data Collection:
Prometheus counts how many observations fall into each bucket. These counts are cumulative, meaning each bucket also includes the counts of all lower buckets.
Calculating the P95:
To calculate the P95, Prometheus needs to determine the latency value at which 95% of the requests fall below.
Assume we have a set of request latencies distributed across buckets as follows:
- Bucket
<1s: 100 requests - Bucket
<2s: 150 requests (cumulative: 100 + 150 = 250) - Bucket
<5s: 300 requests (cumulative: 250 + 300 = 550) - Bucket
<10s: 200 requests (cumulative: 550 + 200 = 750) - Bucket
>=10s: 50 requests (cumulative: 750 + 50 = 800)
Let’s say we have 1000 total requests.
- Determine the Rank:
The rank for the P95 is calculated as
Rank= 95/100×Total Observations=0.95×1000=950So, we’re looking for the latency value at the 950th request.
2. Identify the Bucket:
Starting from the lowest bucket, we sum the cumulative counts until we exceed the rank.
In this case:
- The first 100 requests are under
<1s. - The next 150 requests are under
<2s>, making a cumulative count of 250. - The next 300 requests are under
<5s>, making a cumulative count of 550. - The next 200 requests are under
<10s>, making a cumulative count of 750. - Finally, the last 50 requests are under
>=10s, making a cumulative count of 800
To reach the 950th request, we must go beyond the <10s bucket. Since 750 requests are in buckets less than 10s, the next bucket >=10s contains the 950th request.
3. Interpolate (If Needed):
Since we are within a bucket with boundaries, you could interpolate to find a more precise value, but often the upper boundary of the bucket is used as the P95.
In this case, since we’re in the last bucket, the exact latency of the 950th request is at the boundary or within the >=10s bucket.
Why 4.75s Might Be Observed as P95
If the histogram buckets have upper boundaries like 5s, 10s, etc., and most requests fall under the 5s bucket, the cumulative count might reach 95% within this bucket. As a result, Prometheus may approximate the P95 value around 4.75s, even if the actual requests are much faster. The value of 4.75 is 95% of 5, which corresponds to the starting bucket.
4.75 here is 95 Percentage of 5, which is the starting bucket.
How to Fix It
To resolve this issue, the histogram buckets should be redefined to better capture the range of latencies that the service is experiencing. This can be done by adding more granular buckets below 5s, such as 100ms, 200ms, 500ms, and 1s.
With finer-grained buckets below the 5s mark, Prometheus can better capture the distribution, leading to a more accurate P95 calculation.
