Avoid Asymmetric Routing in Load Balancing (pfSense example)

Introduction

My previous blogs Use pfSense to Load Balance Web Servers (1) and Use pfSense to Load Balance Web Servers (2) introduced the deployment of pfSense as load balancer to distribute web traffic to backend server nodes (i.e. Clst1-S1 and Clst2-S2; Clst2-S1 and Clst2-S2). pfSense hosts Server Cluster 1’s virtual IP 10.10.20.20 and Server Cluster 2’s virtual IP 10.10.20.30.

In the previous lab, when we accessed http://10.10.20.20 from internal Mgmt PC (10.10.10.10/24), the traffic was successfully load balanced to either Clst1-S1 (10.10.20.21) or Clst1-S2(10.10.20.22).
pfsense_lab_topo

Failed Scenario

However, I received a question that when access http://10.10.20.20 from Mgmt2 (diagram below) which is in the same subnet as the backend nodes, Mgmt2 cannot reach the web service.

Mgmt IP: 10.10.10.10 (successfully accessed http://10.10.20.20)
Mgmt2 IP: 10.10.20.10 (failed to access http://10.10.20.20)
Cluster 1 VIP: 10.10.20.20
Cluster 1 Node 1 IP: 10.10.20.21
Cluster 1 Node 2 IP: 10.10.20.22
pfsense_snat_topo_issue

I replicated the failed scenario and observe the following:pfSense_mgmt2_failed.png

Asymmetric Routing

What is the difference between access the web service from Mgmt and Mgmt2?

Mgmt PC is external to the web service subnet. When the user requests to access http://10.10.20.20, the traffic reaches pfSense load balancer, and then forwarded to either Clst1-S1(10.10.20.21) or Clst1-S2(10.10.20.22). Let’s assume Clst1-S1 responses to the request this time. Since Mgmt PC is in a different subnet (10.10.10.0/24), the return traffic reaches its default gateway on pfSense (10.10.20.1) first, and then routed to Mgmt PC.
pfSense_SNAT_topo_extmgmt.png

However, Mgmt2 PC is internal to the web service subnet. When the user requests to access http://10.10.20.20, the traffic reaches pfSense load balancer, and then forwarded to Clst1-S1 (10.10.20.21). Since Mgmt2 PC is in the same subnet as the web servers (10.10.20.0/24), the return traffic goest to Mgmt2 PC directly via SW1, without transiting through the default gateway on the load balancer.

Asymmetric routing occurs. Although some devices have tolerance on asymmetric routing,  these days, we still try to avoid whenever we can. For example, F5 load balancer allows asymmetric routing but it will limit the features. Asymmetric routing also adds network complexity and security concerns.
pfSense_SNAT_topo_intmgmt.png

SNAT as Solution

If the business requirement says the user must have access to the web service from the same subnet, then SNAT can be a solution to avoid the asymmetric routing problem.

pfSenseLB translates the source IP of the traffic initiated from Mgmt2 from 10.10.20.10 to 10.10.10.11. In this case, when Clst1-S1 receives the traffic from Mgmt2, it will response to 10.10.10.11, which forces the return traffic through pfSenseLB. pfSenseLB then translates 10.10.10.11 back to 10.10.20.10 and sends to Mgmt2.
pfSense_SNAT_topo_SNAT.png

pfSense_SNAT_topo_SNAT2.png

The following screenshot demonstrates SNAT configuration details on pfSense.
pfSense_snat.png

After the SNAT, we can successfully access http://10.10.20.20 from Mgmt2 now.
pfSense_mgmt2_success.png

NAT hit can be checked using shell command ‘pfctl -vvs nat’.
psSense_nat_hit.png

End

Be careful of asymmetric routing in load balancing design. For example, one-arm and multi-path (nPath) design may involve asymmetric routing. The selection of design models depends on business requirements. SNAT is a potential solution to asymmetric routing problem.

Advertisements

Set up NGINX as Reverse Proxy with Caching

Introduction

This lab reuses the server infrastructure built in Deploy Scalable and Reliable WordPress Site on LEMP(1), but add another Nginx server as load balancer/reverse proxy (LB01) in front of the web servers (WEB01 and WEB02). Caching will be enabled on LB01 and tested as well.

Boxes highlighted in RED below are deployed in the lab. Although WEB02 is not deployed in the current lab, it can be deployed in the same way as WEB01, described in Deploy Scalable and Reliable WordPress Site on LEMP(2); and proxied by LB01 as shown in the later configuration section.
nginx_reverseproxy_cache.png

Key Concepts

Forward Proxy vs. Reverse Proxy

Forward proxy can be used when servers/clients from a company’s internal network to reach internet resources. It helps keep user IP anonymous, filter URLs and may speed up internet surfing by caching web content.

Reverse proxy can be used when internet users try to access a company’s internal resource. The user request arrives at the reverse proxy server, which forward the request to a backend server that can fulfill it, and returns the server’s response to the client. It hides the company’s actual server IP from attackers, and reduces the load on the actual server by providing cached content itself.

Load Balancing vs. Reverse Proxy

Nginx site provides a good explanation on this topic: https://www.nginx.com/resources/glossary/reverse-proxy-vs-load-balancer/

In this lab, Nginx is set up as load balancer and reverse proxy.

Deployment Steps

Step 1 – Install Nginx on Ubuntu 16.04

Select $5/month Ubuntu 16.04 droplet on DigitalOcean. DigitalOcean calls its Virtual Private Server (VPS) ‘droplet’. Refer to Deploy Scalable and Reliable WordPress Site on LEMP(1) for the details about DigitalOcean and the droplets used in my labs.

Install Nginx on the newly created droplet LB01, by executing the following command:

sudo apt-get update
sudo apt-get -y install nginx

Step 2 – Configure Reverse Proxy

Edit Nginx site configuration on LB01 to pass on web requests to backend web servers.

sudo nano  /etc/nginx/sites-enabled/default

Use Nginx HTTP ‘upstream’ module to realise load balacing and reverse proxy to multiple backend servers. Refer to the official module documentation for details. Update the content of ‘/etc/nginx/sites-enabled/default’ as below:

#define a upstream server group called 'webserver'. 'ip_hash' enables session persistence is required. '10.132.84.104' is WEB01's private IP; '10.132.84.105' is WEB02's private IP.
upstream webserver {
                ip_hash;
                server 10.132.84.104;
                server 10.132.84.105;
}

server {
	listen 80 default_server;
	listen [::]:80 default_server ipv6only=on;

	root /var/www/html;

	# Add index.php to the list if you are using PHP
	index index.php index.html index.htm;

	location / {
		# Call the upstream server group 'webserver', which we defined earlier. We can add additional proxy parameters in '/etc/nginx/proxy_params' if required.
                proxy_pass http://webserver;
                include /etc/nginx/proxy_params;
	}
      }  

Restart Nginx service to make our change work.

sudo service nginx restart

Test access to our WordPress site (created in Deploy Scalable and Reliable WordPress Site on LEMP(2)) via LB01’s public IP. We should see the same page as we directly access to WEB01’s public IP.
nginx_LB1.png
If things don’t work, check the error log ‘/var/log/nginx/error.log’ on LB01. We can use ‘cat’ to display file content, but we use ‘tail’ this time to list the final ‘n’ records.

# -n 6 means to display the final 6 records in the given file.
tail -n 6 /var/log/nginx/error.log

Step 3 – Configure Cache Server

Configure cache in Nginx site configuration file ‘/etc/nginx/sites-enabled/default’. The current file content can be viewed using ‘cat’, ‘less’ or ‘more’.

cat = can be used to join multiple files together and print the result on screen (it will not show page by page)

more = to view a text file one page at a time, press spacebar to go to the next page

less = is much the same as more command except it also supports page up/down and string search. Less is the enhanced version of ‘more’.

Further details refer to ‘Linux Command 7 – more, less, head, tail, cat‘.

Update ‘/etc/nginx/sites-enabled/default’ as following. Refer details in ‘Nginx Caching‘, but additional include ‘proxy_cache_valid’ directive. In my lab, if ‘proxy_cache_valid’ is unset, cached status always shows ‘MISS’. Please refer to Nginx Content Caching for proxy_cache_valid directive details.

#cache files will be saved in subdirectories (1:2) under '/tmp/nginx'. 
#cache zone called 'my_zone' is created with 10MB in size to store cache keys and other metadata
#'inactive=60m' means asset will be cleared from cache if not accessed within 60 mins 
 '200 10m' means response with the code 200 are considered valid for 10 mins.
proxy_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:10m inactive=60m;

#proxy_cache_key defines the key (identifier) for a request.If the request has the same key as a cached response, then cached response is sent to the client.
proxy_cache_key "$scheme$request_method$host$request_uri";

#'proxy_cache_valid' is to set how long cached responses are considered valid.
proxy_cache_valid 200 10m;

upstream webserver {
ip_hash;
server 10.132.84.104;
}

server {
	listen 80 default_server;
	listen [::]:80 default_server ipv6only=on;
        root /var/www/html;

	# Add index.php to the list if you are using PHP
	index index.php index.html index.htm;

	server_name _;

	location / {
                #use the cache zone 'my_zone', which we defined earlier.
                proxy_cache my_zone;
                
                #add an informative header 'X-Proxy-Cache' in response to tell us whether we hit cached content or miss.
                add_header X-Proxy-Cache $upstream_cache_status;
                proxy_pass http://webserver;
                include /etc/nginx/proxy_params;
                }
	}

‘/tmp/nginx’ is the cache file path we defined earlier.
cache_path.png

Finally, let’s test the content cache. When first time visit, ‘X-Proxy-Cache’ shows ‘MISS’; but shows ‘HIT’ when re-visit.’X-Proxy-Cache: EXPIRED’ shows, when the asset is not accessed in 60 mins and cleared from cache.
nginx_cache_hit.png

Use pfSense to Load Balance Web Servers (2)

Use pfSense to Load Balance Web Servers (1) introduces pfSense, the lab setup, VM specs and download links. This blog will demonstrate pfSense configuration, test and troubleshooting details.

Configuration

pfSense Configuration

An overview of pfSense configuration steps are as below along with key information for each step, testing and troubleshooting approach.
pfSense_config_LB.png

Step 1: Initial Configuration

Boot up pfSense VM and wait till installation is completed. Remove pfSense.iso image from the VM and reboot the VM. The following screen will show and guide you through the initial setup.
pfsense_intial_setup.png

Select 2) to configure interface IPs. Please note LAN interface is the default management interface. In our case, we can access pfSense web GUI from https://10.10.10.1.

WAN interface requires default gateway address, ‘192.168.10.1’ in our case. Routing can also be modified after accessing pfSense webconfig GUI.

Step 2: Access pfSense Web GUI

Access pfSense Web GUI from https://10.10.10.1 from the management PC 10.10.10.10. The default username is ‘admin‘ and password ‘pfsense‘. User password can be changed under ‘System/User Management’ as below. Radius and LDAP authentication is also supported.
pfsense_user.png

The default web GUI (HTTPS) port is 443. It can be changed to user-defined port number under’System/Advanced/Admin Access’, as below:
pfsense_https_port.png

Step 3: Create Virtual IP

We need to create a virtual IP under ‘Firewall/Virtual IPs’, which will be used as load balancer’s virtual server IP later in Step 5. The virtual server IP will further forward traffic to the web servers in the load balancing pool. Please refer to the load balanced data flow diagram in Use pfSense to Load Balance Web Servers (1).

Create ‘IP Alias’ type virtual IP if there is single pfSense. Create ‘CARP’ type virtual IP if there are two pfSense in a cluster.CARP stands for ‘Common Address Redundancy Protocol’, functioning similar to VRRP and HSRP.
pfsense_VIP.png
As part of testing/troubleshooting, please make sure the virtual IP is reachable from required subnet. Ping may be temporarily allowed for test purpose.Please note ‘ping’ is ICMP, neither TCP nor UDP.

Step 4: Create Load Balancer Pool

We then create load balancer pool where we can define member servers, under ‘Services/Load Balancer/Pools’. Default monitoring protocol includes ICMP, TCP, HTTP, HTTPS and SMTP. If additional protocol is required, it can be added under ‘Monitors’.
pfSense_Pools.png

Step 5: Create Load Balancer Virtual Servers

Virtual server is created to host the load balancer’s shared IP. It uses the virtual IP we created before in Step 3. We also assign load balancer pool created in Step 4 to virtual server as below:
pfsense_VS.png

As part of testing/troubleshooting, please make sure no error under ‘Status/Load Balancer’ and ‘Status/System Logs/Load Balancer’. For HTTP and HTTPS traffic, if the load balancer members and/or the virtual server are not configured appropriate, the access may fallback to the pfSense web GUI.

Step 6: Tailor Firewall Rules

Since pfSense also functions as firewall, we will need to tailor the firewall rules to allow required traffic and block unwanted traffic. Firewall rules are configured under ‘Firewall/Rules’, as below:
pfsense_firewall_rules.png

Please note, pfSense firewall rules allow us to define traffic direction as well as application to the specified interface. For example, if we have traffic initiated from LAN to SVR; then we allow traffic from LAN net (all LAN subnet IPs) to SVR net (all SVR subnet IPs) and apply the rule to LAN interface on the pfSense. pfSense is stateful firewall by default, we don’t have to set up rules for the return traffic.

Another easy way to figure out what firewall rules are required is to block all uncertain traffic and check what traffic is blocked under ‘Status/System/Logs/Firewall’. Then pass the required traffic directly from the blocked list by clicking ‘+’, as blow:
pfsense_firewall_log.png

Test Access to Load Balanced IP

We then test access to the load balanced IP. The network topology is in Use pfSense to Load Balance Web Servers (1).
pfsense_data_flow

User access the load balanced IPs from a computer over the Internet. When s/he access http://10.10.20.20, the following shows:
pfsense_clst1_LAN.png

The user access is load balanced between Server 1 and Server 2 in Cluster 1 as above screenshot.

Similarly, when the user access http://10.10.20.30 or http://192.168.10.30, the following shows:
pfsense_clst2.png

The user access is load balanced between Server 1 and Server 2 in Cluster 2 as above screenshot.

10.10.20.20 and 10.10.20.30 are examples of using internal IP as load balanced IP; while 192.168.10.30 is example of using external IP as load balanced IP.

You may need to clear cache if browser is not working as expected.

Use pfSense as Layer2 Firewall/Bridged Interface

pfSense does support Layer 2 firewall mode (also called transparent mode) by bridging the required interfaces, under ‘Interfaces/(assign)/Bridges’ as below:
pfsense_bridge

Layer 2 mode will allow the load balanced IP using external IP, while member servers also use external IP subnet. Use case example is as below:
pfsense_layer2_usecase.png

pfSense firewall bridge configuration reference is available here.

Site-R1 Cisco 7200 Router Configuration

Site-Site-R1#show run
Current configuration : 1258 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Site-Site-R1
!
boot-start-marker
boot-end-marker
!
no aaa new-model
no ip icmp rate-limit unreachable
ip cef
ip tcp synwait-time 5
!
no ip domain lookup
!         
multilink bundle-name authenticated
!
interface FastEthernet0/0
 ip address 200.10.10.10 255.255.255.0
 duplex full
!
interface Ethernet1/0
 ip address 192.168.10.1 255.255.255.0
 duplex full
!
interface Ethernet1/1
 no ip address
 shutdown
 duplex half
!
interface Ethernet1/2
 no ip address
 shutdown
 duplex half
!
interface Ethernet1/3
 no ip address
 shutdown
 duplex half
!
ip route 0.0.0.0 0.0.0.0 200.10.10.1
ip route 10.10.10.0 255.255.255.0 192.168.10.10
ip route 10.10.20.0 255.255.255.0 192.168.10.10
ip route 192.168.20.0 255.255.255.0 192.168.10.10
no ip http server
no ip http secure-server
!
logging alarm informational
no cdp log mismatch duplex
!
control-plane
!
gatekeeper
 shutdown
!
line con 0
 exec-timeout 0 0
 privilege level 15
 logging synchronous
 stopbits 1
line aux 0
 exec-timeout 0 0
 privilege level 15
 logging synchronous
 stopbits 1
line vty 0 4
 login
!
end

Last But Not Least

  • Make sure routing, IP schema  and etc. are well planned.
  • Make sure only open minimum required ports on firewall.
  • Make sure proper zone segmentation using firewall to enforce security.
  • Use centrally managed authentication and authorisation, using remote user data source.

Use pfSense to Load Balance Web Servers (1)

What is pfSense?

pfSense is a FreeBSD-based distribution to be installed as physical or virtual machine. It can function as below:

  • Router
  • Firewall
  • Switch
  • Web proxy
  • Load balancer
  • supporting high availability
  • supporting Radius and LDAP authentication
  • Click here for pfSense function list

pfSense is managed via web GUI. HTTPS is enabled by default, while HTTP can be enabled if required.

pfSense provides incident-based support with cost. Their support hours are 7am-7pm CST. Should out-of-hour support be required, pfSense requires advance notice. Please refer here for pfSense support FAQ. This is to be considered in production as it may not meet SLA requirements in some organisations.

Also lab is not production. It is critical to take non-functional requirements such as supportability, scalability, availability/reliability, performance and etc. into consideration in production.

In addition, although pfSense is even more multi-functional than  Juniper SRX, security may require dedicated function per device and therefore multi-layer protection. A similar example in server infrastructure world is…we normally don’t put Active Directory (AD) and Certificate Authority (CA) on a same server.

Lab Topology

The lab is created using GNS3 with VMs hosted in VirtualBox. Please refer to my previous blog Install CSR1000v on GNS3 regarding how to import VMs into GNS3. I used dummy switch in this lab; however, proper layer 3 switch can be set up as described in my log GNS3 Lab: Connect to Physical LAN and Use Layer3 Switch.

Ubuntu Mate are used to simulate all servers and PCs. Apache is installed on Clst1-S1, Clst1-S2, Clst2-S1 and Clst2-S2 to simulate web servers.

pfSense is installed as VM on Virtual Box with 3 physical interfaces: WAN (e0), SVR(e2) and LAN(e1).pfSense webgui is accessible from Mgmt PC.

VM specs and software download links will be provided in next section.

The lab topology is as below:

pfsense_lab_topo.png

Load balancing data flow is as below:

  1. User access load balancing virtual IP 10.10.20.20, which relay HTTP traffic to the two web servers in Cluster 1. pfSenseLB SVR interface (e2) IP is 10.10.20.1.
  2. User access load balancing virtual IP 10.10.20.30, which relay HTTP traffic to the two web servers in Cluster 2.
  3. User access load balancing virtual IP 192.168.10.30, which relay HTTP traffic to the two web servers in Cluster 2. pfSenseLB WAN interface (e0) IP is 192.168.10.10.

pfsense_data_flow.png

VM Specs and Software Download

Ubuntu Mate

Ubuntu Mate is available from here. VM specs are as below:

Attribute Value
Operating System Ubuntu (64-bit)
RAM 1 GB
Storage 12 GB
Adapter 1 Not attached

Apache 2 is installed by running ‘sudo apt-get install apache2’.

pfSense

pfSense is available from here. VM specs are as below:

Attribute Value
Operating System FreeBSD (64-bit)
RAM 500 MB
Storage 2 GB
Adapter 1,2,3 Not attached

To be Continued…

psSense configuration will be introduced in ‘Use pfSense to Load Balance Web Servers (2)’, which I will post over this week.

Next: Use pfSense to Load Balance Web Servers (2)

SIPp on Linux to Generate Voice Load

Introduction

This lab introduces the installation and basic use of SIPp, a free voice performance testing tool. It can emulate thousands of user agents making SIP calls and generate both SIP and RTP traffic. Although it is not as powerful as Cyara, contact centre testing tool (http://cyara.com/), it is FREE, so good for lab or limited budget.

Please visit SIPp website for product details: http://sipp.sourceforge.net/.

Continue reading

PowerShell to Automate Cisco UCSM (4): PS with .Net to develop UI and provision service profile

Introduction

PowerShell to Automate UCS (1): Lab Setup introduced the UCSM and PowerTool lab setup.

PowerShell to Automate UCS (2): PowerGUI Editor introduced how to use PowerGUI as PowerShell script editor to simplify script development test; and also script example.

PowerShell to Automate UCS (3): Convert UCSM GUI to Script introduced how to start from 0 and write PowerShell script to manage UCSM by converting UCSM GUI to script.

This blog demonstrates how to integrate PowerShell and .Net to develop a user interface to check whether a server is in use, then select unused server and provision service profile from the established service profile template.

Continue reading