My previous blogs Use pfSense to Load Balance Web Servers (1) and Use pfSense to Load Balance Web Servers (2) introduced the deployment of pfSense as load balancer to distribute web traffic to backend server nodes (i.e. Clst1-S1 and Clst2-S2; Clst2-S1 and Clst2-S2). pfSense hosts Server Cluster 1’s virtual IP 10.10.20.20 and Server Cluster 2’s virtual IP 10.10.20.30.
In the previous lab, when we accessed http://10.10.20.20 from internal Mgmt PC (10.10.10.10/24), the traffic was successfully load balanced to either Clst1-S1 (10.10.20.21) or Clst1-S2(10.10.20.22).
However, I received a question that when access http://10.10.20.20 from Mgmt2 (diagram below) which is in the same subnet as the backend nodes, Mgmt2 cannot reach the web service.
Mgmt IP: 10.10.10.10 (successfully accessed http://10.10.20.20)
Mgmt2 IP: 10.10.20.10 (failed to access http://10.10.20.20)
Cluster 1 VIP: 10.10.20.20
Cluster 1 Node 1 IP: 10.10.20.21
Cluster 1 Node 2 IP: 10.10.20.22
I replicated the failed scenario and observe the following:
What is the difference between access the web service from Mgmt and Mgmt2?
Mgmt PC is external to the web service subnet. When the user requests to access http://10.10.20.20, the traffic reaches pfSense load balancer, and then forwarded to either Clst1-S1(10.10.20.21) or Clst1-S2(10.10.20.22). Let’s assume Clst1-S1 responses to the request this time. Since Mgmt PC is in a different subnet (10.10.10.0/24), the return traffic reaches its default gateway on pfSense (10.10.20.1) first, and then routed to Mgmt PC.
However, Mgmt2 PC is internal to the web service subnet. When the user requests to access http://10.10.20.20, the traffic reaches pfSense load balancer, and then forwarded to Clst1-S1 (10.10.20.21). Since Mgmt2 PC is in the same subnet as the web servers (10.10.20.0/24), the return traffic goest to Mgmt2 PC directly via SW1, without transiting through the default gateway on the load balancer.
Asymmetric routing occurs. Although some devices have tolerance on asymmetric routing, these days, we still try to avoid whenever we can. For example, F5 load balancer allows asymmetric routing but it will limit the features. Asymmetric routing also adds network complexity and security concerns.
SNAT as Solution
If the business requirement says the user must have access to the web service from the same subnet, then SNAT can be a solution to avoid the asymmetric routing problem.
pfSenseLB translates the source IP of the traffic initiated from Mgmt2 from 10.10.20.10 to 10.10.10.11. In this case, when Clst1-S1 receives the traffic from Mgmt2, it will response to 10.10.10.11, which forces the return traffic through pfSenseLB. pfSenseLB then translates 10.10.10.11 back to 10.10.20.10 and sends to Mgmt2.
The following screenshot demonstrates SNAT configuration details on pfSense.
After the SNAT, we can successfully access http://10.10.20.20 from Mgmt2 now.
NAT hit can be checked using shell command ‘pfctl -vvs nat’.
Be careful of asymmetric routing in load balancing design. For example, one-arm and multi-path (nPath) design may involve asymmetric routing. The selection of design models depends on business requirements. SNAT is a potential solution to asymmetric routing problem.