Apply Circuit Breaker Assertion

The Apply Circuit Breaker assertion defines thresholds for failure conditions which, when exceeded, prevent blocks of your policy from executing for a configurable period. This is ideal for avoiding bottlenecks that arise due to request processing slowdowns that are caused by sluggish or malfunctioning back-end systems.
gateway93
The Apply Circuit Breaker assertion defines thresholds for failure conditions which, when exceeded, prevent blocks of your policy from executing for a configurable period. This is ideal for avoiding bottlenecks that arise due to request processing slowdowns that are caused by sluggish or malfunctioning back-end systems.
This is a composite assertion that behaves similarly to the All Assertions Must Evaluate to True Assertion. The main difference: internal Event Trackers that record the history of failures (policy failure or exceeding a maximum latency). Once the Apply Circuit Breaker assertion detects a circuit has exceeded a threshold, the assertion fails and none of its child assertions execute. Depending on the surrounding policy logic, a failed Apply Circuit Breaker assertion could cause further branching or it could fail the entire policy.
You can enable one or both of these circuits in the assertion:
  • Policy Failure Circuit:
    The circuit breaks after a certain number of child policy failures, within a given sampling window (for example, "over the last 5 seconds"). When the circuit is broken, the Apply Circuit Breaker assertion returns "Falsified". No child assertions are processed for subsequent requests, for the duration of the recovery period.
  • Latency Circuit:
    The circuit breaks after a specified number of latency failures. A latency failure occurs when the length of time to execute the child assertions exceed a threshold value.
The policy failure circuit and the latency circuit are independent. Tripping either circuit causes the assertion to fail.
You may configure multiple instances of this assertion in a policy. Each circuit maintains its own internal Event Tracker of failures (unless you define custom Event Trackers to be shared).
Contents:
Comparing this assertion against...
  • All Assertions Must Evaluate to True Assertion: Both assertions process each child assertion until one fails. "All Assertions..." continues to process each request the same way. If a child assertion consistently fails due to timeouts on the back end (for example), then all prior child assertions are processed needlessly. The Apply Circuit Breaker assertion tracks the failures. When a threshold is exceeded, the assertion returns "falsified" and processing of child assertions is blocked for a predefined recovery period.
  • Apply Rate Limit Assertion: This assertion can throttle requests once predefined rates are exceeded, but it cannot skip blocks of policy.
  • Branching logic: You can configure branching logic using other composite assertions; for example, Run All Assertions Concurrently Assertion. Branching logic can also avoid executing blocks of policy under certain conditions. The Apply Circuit Breaker assertion extends this functionality by defining policy failure and latency circuits.
Context Variables
None
Cluster Properties
Property
Description
circuitBreaker.eventTrackerCleanupInterval
The Event Tracker cleanup interval. In every interval, failure records older than the interval are removed from Event Trackers (for example, using the default value: every 10 minutes, failures older than 10 minutes are removed).
Default:
600000
ms (10 minutes)
The cleanup interval should be at least twice the size of the largest sampling window. Otherwise, it could lead to an unpredictable number of failures before the circuit trips due to events being cleaned up within the sampling window.
circuitBreaker.forceCircuitOpenEventTrackerIdList
A list of Event Tracker IDs that immediately forces open any circuit that uses the ID. The child assertions are not executed.
This cluster property behaves like a "short circuit" list.
Default: the list is empty
Auditing
Audit ID
Level
Message
Example
11500
WARNING
{0} Circuit tripped; open until {1}
Policy Failure Circuit tripped; open until 12:01:12.1432
11501
INFO
This {0} Circuit is forced open based on its Event Tracker ID: {1}
This Latency Circuit is forced open based on its Event Tracker ID: MyBackendService-Latency
11502
WARNING
Value for {0} Circuit {1} is invalid: {2}
Value for Policy Failures Circuit Sampling Window is invalid: -200
11503
INFO
{0} Circuit open until {1}
Policy Failure Circuit open until 12:01:12.1432
Latency Circuit open until 12:01:12.1432 for Event Tracker ID 'MyBackendService-Latency'
 
Properties
Setting
What you should know...
Policy Failure Circuit:
Enable this option to have the assertion respond to child policy failures.
Max Failures
The circuit "trips" once the assertion has counted this many failures, within the Sampling Window size. Once tripped, the assertion fails (returns "Falsified") for the duration of the Recovery Period.
Sampling Window
Failures are counted only for this recent timeframe. Default is
5000
ms.
Recovery Period
Once the circuit "trips", the assertion fails for this recovery period. During this time, failures are not counted because no child assertions are processed. When the Recovery Period has elapsed, the assertion once again allows processing of the child assertions. Default is
10000
ms.
To prevent old policy failures from being counted again for future circuit failure calculations, set the Recovery Period to greater than the Sampling Window.
Event Tracker ID
Enable this option to assign your own Event Tracker identifier for this circuit. You may specify context variables, to create dynamic circuits.
For more information, see "Understanding Custom Event Trackers" below.
Latency Circuit:
Enable this option to have the assertion respond to latency failures. A 'latency failure' occurs when the child assertions take longer to process than the defined latency threshold. 
Max Failures
Sampling Window
Recovery Period
These settings work the same as under "Policy Failure Circuit".
The same tip applies about setting the Recovery Period greater than the Sampling Window.
Max Latency
This is the latency threshold. The Gateway tracks the amount of time that it takes for all the child assertions to process. If it exceeds the Max Latency, then that is counted as a latency failure.
Event Tracker ID
This setting works the same as under "Policy Failure Circuit".
Understanding Custom Event Trackers
In its default configuration, the Apply Circuit Breaker assertion sets no Custom Event Trackers. Each circuit uses its own internal Event Tracker to track failure events. You can enable advanced circuit-based behavior by creating Custom Event Trackers.
Sharing Event Trackers
You can share Event Trackers across circuits by configuring them to use the same Custom Event Tracker ID. By referring to the same Event Tracker, two or more circuits may share a failure event history, while maintaining independent thresholds and recovery periods. In this scenario, if the circuits use the same thresholds and recovery periods, they can also share a circuit state. Sharing Event Trackers allows services on a Gateway to have multiple points of detection of similar errors. For example, latency when routing to multiple services in a particular data center can be aggregated to a common threshold for tripping circuits.
Dynamic Circuits
You can create dynamic circuits at runtime by configuring the Apply Circuit Breaker assertion to use context variables in a Custom Event Tracker ID. These dynamic circuits apply to the same blocks of policy, but have different failure event histories and circuit states since they have different Event Trackers. With this functionality, different Apply Circuit Breaker assertions can allow their nested policy to execute for some requests, but not others. This gives you more fine-grained and context-aware error handling.
Example:
A service published on a Gateway uses a Route via HTTP(S) Assertion to send requests to an API. This assertion is nested within an Apply Circuit Breaker assertion with a Policy Failure Circuit enabled. Requests to the service require an API Key to be provided as a header. Every developer consuming the service has their own API key.
Without Custom Event Trackers, if any developer makes a change that causes their Gateway HTTP requests to the API to fail, this is counted as a policy failure. When the Policy Failure Circuit trips, routing for all the developers cease, even if their requests were succeeding.
To prevent this undesired consequence, set a Custom Event Tracker ID based on the API Key sent in the header (for example,
${request.http.header.apikey}
). Now there are separate circuits created at runtime for each API Key. Only the developer sending failing requests is blocked, because only the circuit associated with that developer's API Key is tripped.
Frequently Asked Questions
Question
Answer
Can I add multiple circuit breaker assertions to my policy?
Absolutely. You can define one or more circuits per instance of the Apply Circuit Breaker assertion. For example, one assertion may define a child policy execution failure threshold, a second may define a latency limit threshold, while a third may define both.
Does the circuit breaker apply across the entire cluster?
No, the Apply Circuit Breaker assertion only applies per node.
What happens if I enable both circuits?
When both Policy Failure and Latency circuits in the assertion are enabled, either circuit tripping will short-circuit the assertion. Note that if both types of failures occur in the same execution of the assertion, both failures are recorded.
What happens if I disable both circuits?
You can disable both circuits if you want to "turn off" the assertion temporarily. In this case, all child assertions are run and the Apply Circuit Breaker assertion does not count policy or latency failures.
What response is returned when the policy failure circuit "trips"?
The service returns the status that is received from the last executed child assertion. During the recovery period, the assertion returns 'Falsified'.
I added a Raise Error Assertion as a child assertion. Why didn't it count as a policy error?
When a Raise Error assertion is processed, it is not a failure; the assertion succeeds. The Apply Circuit Breaker assertion treats a Raise Error the same way as the All Assertions Must Evaluate to True Assertion.