Concurrency Considerations in HTTP Load Balancing

Be aware of potential failure cases when you plan for load balancing. It is possible that the entire inbound concurrency limit on a Gateway is consumed by requests.
gateway83
Be aware of potential failure cases when you plan for load balancing. It is possible that the entire inbound concurrency limit on a Gateway is consumed by requests.
When there are a large number of requests executing concurrently, it is possible to overrun the limit of back-end connections. When this happens, additional requests are queued within the Gateway, occupying threads. Eventually, the inbound connection limit is reached and the load balancer health check fails to obtain a new connection. This causes the load balancer to assume that the Gateway itself is failing. The normal remedy is to remove the Gateway node from the pool. This puts additional concurrency on the other Gateway nodes, using up their inbound connection pools more quickly, causing the same event to each node in sequence. This is called a
cascaded failure
.
To avoid this situation, consider your concurrency requirements and ensure that there are larger values further inside the network.
Consider the following diagram:
  • The limit at the back-end load balancer (6) might be set to 1200 concurrent, to handle all eventualities.
  • The connections per host (4) is expected to be about 1100.
  • The Gateway inbound (3) may vary, but in general will not exceed 1000.
This ensures that there is no case where the previous link in the system is unable to obtain a connection to the next link.
loadbalancer_callouts.png
  1. Client Software
  2. External Firewall
  3. Load balancer
  4. Gateway Cluster
  5. Firewall between DMZ in High Security Zone
  6. Back-End Load Balancer
  7. Back-End Systems
Every point has possible bottlenecks, where the concurrency load is larger than what is allowed by the tools in the pipeline:
  • Firewalls sometimes (not often) limit concurrency.
  • Load Balancers can sometimes limit per-node concurrency (consult the documentation for your Load Balancer for details)
  • The Gateways may limit HTTP Listener Inbound concurrency (through the
    io.httpCoreConcurrency
    and
    io.httpMaxConcurrency
     cluster properties)
  • The Gateways may limit Outbound HTTP Routing concurrency using these Gateway System Properties:
    • Single target maximum back end:
      com.l7tech.common.http.prov.apache.CommonsHttpClient.maxConnectionsPerHost
    • All targets maximum back-end concurrency:
      com.l7tech.common.http.prov.apache.CommonsHttpClient.maxTotalConnections