HTTP Load Balancer Best Practices

This topic provides an overview of load balancing and scaling in gateway environments. API Gateways differ slightly from Web servers in their behavior.
gateway83
This topic provides an overview of load balancing and scaling in
CA API Gateway
environments. API Gateways differ slightly from Web servers in their behavior.
Most environments use load balancers as standard practice. These load balancers reside between the clients and the gateway, and between the gateway and the back-end servers. The following diagram shows a standard environment used during performance tests and in production. Note the firewalls at the front and back, with load balancers front and back.
loadbalancer.png
The environment shown above presents unique challenges due to the number of network components.
HTTP is a stateless protocol in theory. In practice, managing the state and the low level management detail can have a significant effect on performance of your API Gateway.
Desired Outcomes of Load Balancing
These are the most important outcomes of load balancing:
  • Failover, High Availability: 
    The most basic and reasonable outcome: load balancers prevent requests from being sent to non-operable servers. How effective this is depends on failure detection.
  • Balancing Load: 
    Despite the obvious name, this is actually the least understood and many side effects are counter intuitive. For example, shifting the load to different nodes may actually 
    increase
     the CPU load on the servers due to SSL session negotiation overhead. Similarly, round-robin can sometimes cause additional load on all systems from lack of HTTP keep-alive or from SSL session overhead.
  • Reducing Overload of Individual Nodes: 
    This is a very important effect of good load balancing, but not a necessary outcome. For example, strict round-robin balancing could result in requests being sent to an overloaded system.
    Similarly, when a given group of client systems is proxied via NAT, then IP address-based affinity will not balance the load as well. This configuration is common in large corporations. Even some communication service providers proxy entire global regions behind such proxies. Another scenario to consider is in-front cache engines that affect in-bound IPs and SSL session uniqueness.