Avoid Asymmetric Routing in Load Balancing (pfSense example)

Introduction

My previous blogs Use pfSense to Load Balance Web Servers (1) and Use pfSense to Load Balance Web Servers (2) introduced the deployment of pfSense as load balancer to distribute web traffic to backend server nodes (i.e. Clst1-S1 and Clst2-S2; Clst2-S1 and Clst2-S2). pfSense hosts Server Cluster 1’s virtual IP 10.10.20.20 and Server Cluster 2’s virtual IP 10.10.20.30.

In the previous lab, when we accessed http://10.10.20.20 from internal Mgmt PC (10.10.10.10/24), the traffic was successfully load balanced to either Clst1-S1 (10.10.20.21) or Clst1-S2(10.10.20.22).
pfsense_lab_topo

Failed Scenario

However, I received a question that when access http://10.10.20.20 from Mgmt2 (diagram below) which is in the same subnet as the backend nodes, Mgmt2 cannot reach the web service.

Mgmt IP: 10.10.10.10 (successfully accessed http://10.10.20.20)
Mgmt2 IP: 10.10.20.10 (failed to access http://10.10.20.20)
Cluster 1 VIP: 10.10.20.20
Cluster 1 Node 1 IP: 10.10.20.21
Cluster 1 Node 2 IP: 10.10.20.22
pfsense_snat_topo_issue

I replicated the failed scenario and observe the following:pfSense_mgmt2_failed.png

Asymmetric Routing

What is the difference between access the web service from Mgmt and Mgmt2?

Mgmt PC is external to the web service subnet. When the user requests to access http://10.10.20.20, the traffic reaches pfSense load balancer, and then forwarded to either Clst1-S1(10.10.20.21) or Clst1-S2(10.10.20.22). Let’s assume Clst1-S1 responses to the request this time. Since Mgmt PC is in a different subnet (10.10.10.0/24), the return traffic reaches its default gateway on pfSense (10.10.20.1) first, and then routed to Mgmt PC.
pfSense_SNAT_topo_extmgmt.png

However, Mgmt2 PC is internal to the web service subnet. When the user requests to access http://10.10.20.20, the traffic reaches pfSense load balancer, and then forwarded to Clst1-S1 (10.10.20.21). Since Mgmt2 PC is in the same subnet as the web servers (10.10.20.0/24), the return traffic goest to Mgmt2 PC directly via SW1, without transiting through the default gateway on the load balancer.

Asymmetric routing occurs. Although some devices have tolerance on asymmetric routing,  these days, we still try to avoid whenever we can. For example, F5 load balancer allows asymmetric routing but it will limit the features. Asymmetric routing also adds network complexity and security concerns.
pfSense_SNAT_topo_intmgmt.png

SNAT as Solution

If the business requirement says the user must have access to the web service from the same subnet, then SNAT can be a solution to avoid the asymmetric routing problem.

pfSenseLB translates the source IP of the traffic initiated from Mgmt2 from 10.10.20.10 to 10.10.10.11. In this case, when Clst1-S1 receives the traffic from Mgmt2, it will response to 10.10.10.11, which forces the return traffic through pfSenseLB. pfSenseLB then translates 10.10.10.11 back to 10.10.20.10 and sends to Mgmt2.
pfSense_SNAT_topo_SNAT.png

pfSense_SNAT_topo_SNAT2.png

The following screenshot demonstrates SNAT configuration details on pfSense.
pfSense_snat.png

After the SNAT, we can successfully access http://10.10.20.20 from Mgmt2 now.
pfSense_mgmt2_success.png

NAT hit can be checked using shell command ‘pfctl -vvs nat’.
psSense_nat_hit.png

End

Be careful of asymmetric routing in load balancing design. For example, one-arm and multi-path (nPath) design may involve asymmetric routing. The selection of design models depends on business requirements. SNAT is a potential solution to asymmetric routing problem.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s