Spike Protection

A spike is a significant, temporary increase in your event volume. Because Sentry bills based on the number of events sent monthly, spikes can quickly consume your quota for an entire month. Spike Protection guards against this by establishing a spike threshold for the average number of events each of your projects sends and then dropping them after that threshold is reached.

Spike Protection currently applies to errors, transactions, and attachments.

How Spike Protection Works

Spike Protection can be enabled on a per-project basis for your organization by any team member with Manager, Billing or Owner-level permissions. To select which project to set it up for, go to Settings > Spike Protection. You'll be able to toggle it on for individual projects or click “Enable All” to set it up for all your projects at once.

How Sentry Detects Spikes

Sentry uses an algorithm to establish a spike threshold for each project. When this threshold is reached, a spike is flagged. Sentry then applies a dynamic rate limit to your project and begins discarding events to keep your quota from being consumed too quickly. This protects you from short-term spikes. But because of the algorithm's dynamic nature, if the event volume continues to be high for a sustained period of time, it'll become your new baseline and you may quickly run through your quota.

Spike thresholds are recalculated every hour for the duration of the spike. This is done to gradually increase thresholds over time to account for new traffic patterns.

Spike Protection will automatically deactivate once the spike event passes, until the next time a spike is detected.

A Dynamic Algorithm

In order to adjust for certain patterns in the number of events your application sends, Sentry's algorithm evaluates your per-project hourly spike threshold by accounting for daily and hourly seasonality. It's based on the higher of two calculations: One, determined by the minimum number of events that would be considered a spike based on your quota. The second, a weighted projection based on your usage over the last 168 hours (7 days).

It's done this way so that new projects without 7 days' worth of data can benefit from spike protection. Additionally, the minimum event calculation can be used to minimize false positives in smaller or newer projects so that spikes aren't flagged incorrectly. Read on to learn about our Spike Protection algorithm in detail.

Notifications

While Spike Protection notifications are turned off by default, you can turn them on for key projects you want to monitor in two ways:

• By adding the email “Notification Actions”. This will notify users with Owner, Manager, and Billing roles who are connected with that project via email.

• By adding Slack or Pagerduty “Notification Actions” (if you have these integrations installed). This will notify users who are connected with the project via Slack or Pagerduty.

Managing a Spike

We recommend taking the following steps to manage your spikes:

• Check to see which issues are consuming your quota.
• Set rate limits on the DSN keys for the projects related to the spike.
• Set up metric alerts for the number of events in a project.
• If a specific release version has caused the spike, add the version identifier to the project's inbound filters to avoid accepting events from that release.
• Set up a pay-as-you-go budget to make sure you have time to adjust your volume in the event of a future spike.

To review the events that were dropped to save your quota with Spike Protection, go to the "Usage Stats" tab of the Stats page for your Sentry org and select the event of interest from the "Category" dropdown. You'll be able to see project-level details of your stats by using the project selector.

To ensure accuracy, your time window on the Stats page must be between a minimum of 6 hours, to a maximum of 30 days for spikes and a spike threshold to appear. In addition, only spikes that are over an hour long will be viewable.

To see spikes that are less than an hour long, visit Sentry's Audit Log by going to the Settings page and selecting "Audit Log" under the "ORGANIZATION" section. You'll be able to see spikes by filtering for spike-protection.activated, spike-protection.deactivated, spike-protection.disabled, or spike-protection.enabled in the "Select Action" dropdown.

How the Spike Protection Algorithm Works

As mentioned above, our Spike Protection Algorithm is based on the higher of two calculations: a minimum event calculation and a usage-based calculation. Read on to learn the details of how it works.

Minimum Event Calculation

This calculation, which is the first step of our algorithm, identifies a minimum number of events, using your quota as a guide. This number takes the maximum of either 500 events or the result of the following formula (3 \* your quota)/(720 \* number of projects). The number of projects is capped at 5 and is used to adjust the floor. The equation represents your project using up three times your overall quota in 30 days if events are continually ingested at this hourly rate, thus flagging the project for a potential spike.

Usage-Based Calculation

This calculation, which is the second step of our algorithm, calculates hourly data from the past seven days to determine spike limit projections for the next seven days. This data is used to calculate weighted averages, which takes into account weekly and hourly seasonality. For example, the weighted average calculated for Monday at 3 pm is more heavily influenced by data points on Monday or the hours around 3 pm. This weighted average is then multiplied by a multiplier that is 5 times the overall standard deviation of the past week — this multiplier is bounded between 3 and 6.

Decay Factor for Dropped Events

Accepted events and dropped events are processed separately. An exponential decay is added to the contribution of dropped events to the overall threshold. The contribution of dropped events decays to ~10% weight 24 hours after a spike, and is essentially zero two days after a spike. This provides a mechanism for spike protection to still adjust (slowly) to increases in volume, but will provide better protection in the days following a significant spike.

Example

In this example, the project usually ingests 100-200 events per hour. There's been a spike that's reached 50,000 events, as shown in the graph below:

In the following graph, we can see a zoomed in perspective of the 12-hour period of the spike, along with a line indicating the spike limit as it's being recalculated over the course of the spike:

Throughout the spike, the recalulating limit has the following effect:

• 1st hour: 6k events ingested, limit is recalculated to 2083, 3917 events dropped
• 2nd hour: 34k events ingested, limit is recalculated to 2873, 31217 events dropped
• 3rd hour: 55k events ingested, limit is recalculated to 5452, ~49k events dropped
• 4th hour: 49k events ingested, limit is recalculated to 7628, ~41k events dropped
• 5th hour: 41k events ingested, limit is recalculated to 9371, ~31k events dropped

For this particular example:

• Org quota: 500k
• Events ingested during the spike: ~478k
• Events accepted overall: ~157k

Here's an example of spike limit projections for a week, taking into account seasonality:

These regular differences in event ingestion don't cause a spike to occur.

Bursty Projects

There may be instances where a project routinely accepts a high volume of events in a very short period of time by design — for example projects that orchestrate cron/Airflow jobs or task runners. The screenshot below shows an example of this kind of behavior:

If this is expected behavior for a given project in your organization, you may want to consider turning off spike protection in the project settings to ensure necessary events aren't dropped.