Deploy Scalable and Reliable WordPress Site on LEMP(3)


In previous post Deploy Scalable and Reliable WordPress Site on LEMP(2), we successful set up Linux+Nginx+PHP+MySQL (LEMP) stack to hold WordPress site. However, Nginx and PHP services were enabled on the same server WEBo1.

In this lab, we will separate PHP to an external server PHP01 and leave WEB01 as Nginx web server only. It adds some flexibility scalability strategy. For performance enhancement details, please refer to Scaling PHP apps via dedicated PHP-FPM nodes, a test post I found online.

Deployment Steps

This lab involves PHP01 deployment and configuration change on WEB01 to forward PHP requests to PHP01. The topology is as below:

Step 1 – Configure PHP01 as php-fpm node

Boot another $5 ubuntu server from DigitalOcean, details available in Deploy Scalable and Reliable WordPress Site on LEMP(1).

#log onto PHP01
#update and install glusterfs client, php-fpm and php-mysql services on PHP01
sudo apt-get update
sudo apt-get -y install glusterfs-client php-fpm php-mysql
#make a folder called 'gluster' under root
sudo mkdir /gluster
#mount the 'file_store' volume on FS01 to '/gluster'on PHP01. '' is FS01's private IP. 'glusterfs' is the filesystem type.
sudo mount -t glusterfs /gluster

# add the partition to fstab so it mounts automatically at boot time.
sudo echo " /gluster glusterfs defaults,_netdev 0 0" >> /etc/fstab

Verify PHP01 can access the same WordPress folder we created on FS01 before, by executing the following command on PHP01.

sudo ls -la /gluster/www

Step 2 – WEB01 forward PHP requests to PHP01

Log onto WEB01, edit the nginx site default file by executing the following command:

sudo nano /etc/nginx/sites-enabled/default

Update the file as below; where ‘’ is PHP01’s private IP, and 9000 is the port used by fastcgi. Comment the local fastcgi socket ‘unix:/run/php/php7.0-fpm.sock’.

	# pass the PHP scripts to FastCGI server listening on the php-fpm socket
        location ~ \.php$ {
                try_files $uri =404;
                #fastcgi_pass unix:/run/php/php7.0-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;

Restart Nginx service by executing the following command:

service nginx restart

Let’s now log onto the WordPress site, see whether it is still working as expected in Deploy Scalable and Reliable WordPress Site on LEMP(2).

Unfortunately, we get ‘502 Bad Gateway’ message this time.
Fortunately, Nginx provides us error log. Execute the following command on WEB01 to view the log.

cat /var/log/nginx/error.log

Nginx error log reveals the following:

2016/11/04 10:25:21 [error] 18867#18867: *1 connect() failed (111: Connection refused) while connecting to upstream, client:x.x.x.x, server:y.y.y.y, request: "GET / HTTP/1.1", upstream: "fastcgi://", host: "y.y.y.y"

OK…it appears WEB01 passed on the request to PHP01, but PHP01 refused the connection. Step 3 will help resolve the issue.

Step 3 – Allow PHP01 to Listen WEB01

On PHP01, edit PHP ‘www.conf’ file to allow listen on WEB01.

sudo nano /etc/php/7.0/fpm/pool.d/www.conf

Perform the following changes in ‘www.conf’.

#Add the following line to allow WEB01's private IP
listen.allowed_clients =

#Comment the following line by adding ";" in the front.
;listen = /run/php/php7.0-fpm.sock

#Add the following line to have php-fpm listen on port 9000
listen = 9000

Restart PHP on PHP01 and Nginx on WEB01.

#On PHP01 restart php service
service php7.0-fpm restart
#On WEB01 restart nginx service
service nginx restart

Let’s now access the WordPress site again. ‘Hello world!’ – it’s working!


The End


Deploy Scalable and Reliable WordPress Site on LEMP(2)

Deploy Scalable and Reliable WordPress Site on LEMP(1) introduced the LEMP design, lab setup and MySQL configuration. This post will further introduce how to deploy Gluster distributed file system and Nginx web server. PHP will be initially enabled on the Nginx web server to prove WordPress site is working. The next post will introduce hosting PHP on a separate server.

Step 2 – Gluster Distributed File System

Gluster is a scale-out network attached file system. It aggregates storage servers, known as ‘storage bricks’ into one large parallel network file system. Virtual volumes are created across the member bricks. Servers with GlusterFS client service installed can have the remote virtual volume mounted.

There are 3 types of virtual volumes. Please refer to ‘GlusterFS Current Features & Roadmap‘ for details.

  • Distributed Volume: similar to RAID 0 without replica; files are evenly spread across bricks.
  • Replicated Volume: similar to RAID 1, which copies files to multiple bricks.
  • Distributed Replicated Volume: Distributes files across replicated bricks.

In this lab, we will deploy 1 node in the GlusterFS cluster, which means ‘Distributed Volume’ mode is used. Additional bricks can be added later.

Configuration is as below. Refer Gluster installation guide for details:

#update and install gluster on FS01. '-y' means automatically answering yes to all prompts.
sudo apt-get update

#ubuntu Personal Package Archive(PPA) requires 'software-properties-common' to be installed first.
sudo apt-get install -y software-properties-common

#add the community GlusterFS PPA
sudo add-apt-repository -y ppa:gluster/glusterfs-3.8

#update again
sudo apt-get update

#install GlusterFS server
sudo apt-get install -y glusterfs-server

#create a volume called 'file_store'. If replica is required, add 'replica n' after the volume name. 
#'' is the brick's private IP. If multiple bricks exit, all member IPs are required in the command.
gluster volume create file_store transport tcp force

#start the 'file_store' volume.The volume is ready-to-use now.
gluster volume start file_store

Step 3 – Nginx Web Server

In this step, we will create a Nginx web server with PHP integrated initially; mount the virtual volume created in Step 2  to the web server; download WordPress files to the mounted folder; and then update WordPress config file to point to the mounted folder and connect to the database created in Step 1.

The following configuration shows which services are to be installed and how to mount external volume as partition.

#update and install nginx, glusterfs client, php-fpm and php-mysql services on WEB01
sudo apt-get update
sudo apt-get -y install nginx glusterfs-client php-fpm php-mysql

#Nginx used to require set php-fpm pathinfo to false 'cgi.fix_pathinfo=0' in 'php.ini'.Default is true '1'. It was a security issue related with Nginx and older version (5.0) of php-fpm.
#the change is not required as we are using PHP7.0

#make a folder called 'gluster' under root
sudo mkdir /gluster
#mount the 'file_store' volume on FS01 to '/gluster'on WEB01. '' is FS01's private IP. 'glusterfs' is the filesystem type.
sudo mount -t glusterfs /gluster

# add the partition to fstab so it mounts automatically at boot time.
sudo echo " /gluster glusterfs defaults,_netdev 0 0" >> /etc/fstab

# create a folder 'www' under '/gluster'. '/gluster/www' will be the web root.
sudo mkdir /gluster/www

We now need to modify the default Nginx server block to point to our new web root, ‘/gluster/www’.

sudo nano /etc/nginx/sites-enabled/default

Modify ‘/etc/nginx/sites-enabled/default’ file content as below. Changes are highlight in BLUE.

server {
	listen 80 default_server;
	listen [::]:80 default_server ipv6only=on;

	root /gluster/www;
	index index.php index.html index.htm;

	# Make site accessible from http://localhost/

	location / {
		# First attempt to serve request as file, then
		# as directory, then fall back to displaying a 404.
		#try_files $uri $uri/ =404; 
                #the following config sends everything through to index.php and keeps the appended query intact.
                try_files $uri $uri/ /index.php?q=$uri&$args;
	# pass the PHP scripts to FastCGI server listening on the php-fpm socket
        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_pass unix:/run/php/php7.0-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;              

Download and configure WordPress.

#download WordPress files and unzip
wget -O /root/wp.tar.gz
tar -zxf /root/wp.tar.gz -C /root/

#copy the WordPress files to our new web root on WEB01. After the copy, the files are also shown on FS01, as it is the storage destination mounted to WEB01.
cp -Rf /root/wordpress/* /gluster/www/.

#copy sample WordPress config file to 'wp-config.php', where we can define the database connection.
cp /gluster/www/wp-config-sample.php /gluster/www/wp-config.php

We now update ‘/gluster/www/wp-config.php’ with the ‘wordpress1’ database and ‘wpuser1’ information, which was created in Step 1.

// ** MySQL settings - You can get this info from your web host ** //
/** The name of the database for WordPress */
define('DB_NAME', 'wordpress1');

/** MySQL database username */
define('DB_USER', 'wpuser1');

/** MySQL database password */
define('DB_PASSWORD', 'password');

/** MySQL hostname */
define('DB_HOST', '');

/** Database Charset to use in creating database tables. */
define('DB_CHARSET', 'utf8');

/** The Database Collate type. Don't change this if in doubt. */
define('DB_COLLATE', '');

We now make the web root ‘gluster/www’ owned by the user ‘www-data’ and the group ‘www-data’, which Nginx process will use. Then finally restart the Nginx and php-fpm services to make our previous config changes working.

chown -Rf www-data:www-data /gluster/www
service nginx restart
service php7.0-fpm restart

Step 4 – WordPress Site

We now can access the WordPress site by WEB01’s public IP or DNS name if it’s been setup. The following should show and lead to initial setup wizard.

Self-hosted WordPress provides much more flexibility and features than hosted by

To be Continued

Deploy Scalable and Reliable WordPress Site on LEMP(1)


In this lab, we will develop and deploy a scalable and reliable WordPress site on LEMP stack. Different from LAMP stack, LEMP uses nginx [engine x], instead of Apache as web server. Compared with Apache, nginx can handle much more concurrent hits. I found an article on the Internet, demonstrating the performance difference ‘Web server performance comparison‘.

Please refer to nginx official site for further details.

Design Rationale

A design proposed for production environment is as below. Firewall is not illustrated in the digram, but required in production environment.

Design Key Points:

  • A pair of load balancers in active/passive HA cluster, sharing single virtual IP (VIP). DNS name will be associated to the VIP.
  • The separation of web, application and backend layers in different subnets, allows flexibility to enforce security.
  • Web cluster, PHP cluster, database cluster and file server cluster are created to achieve high availability and horizontal scalability
  • PHP servers, file serves and database servers are external to web serves, which helps scalability. File server cluster will store WordPress files and mounted to web nodes and PHP nodes.

Lab Description

Lab Scope

Due to the limit of my time and computing resource (= ‘money’), the boxes highlighted in RED in the above diagram are in the current lab scope. If time allows, I will deploy LVS load balancers as well. If you are interested in pfSense as load balancer, please refer to my posts Use pfSense to Load Balance Web Servers (1) and Use pfSense to Load Balance Web Servers (2).

Lab Servers and Software

I used DigitalOcean for cloud servers. It is cheaper than AWS. The cheapest instance is only US$5/month.

DigitalOcean $5 deal provides: 1 CPU, 512MB RAM, 20GB SSD, and 1000GB transfer. All servers used in this lab are $5 servers.

If you decide to use DigitalOcean, please use my referral link You will get US$10 in credit immediately; and I may get some referral benefits as well, so win-win.

The downside is DigitalOcean provides less features. It allows private IP… but the private IP is automatically generated based on the selected datacentre (DC). For example, all my servers are in New York 3 DC, and their private IPs are all in the same subnet. It means I cannot manipulate IP allocation and routing as I did in AWS. In addition, it provides little ready-to-use security mechanism, though we can use Linux native firewall and install additional security services.

It is good and bad: good for quick DevOp testing, no hustle with network; bad for production environment or if you particular like network…myself for example 🙂

Web Service Layer Application/Service Platform
Load Balancer Layer Linux Virtual Server(LVS) Not deployed yet
Web Server Layer Nginx Ubuntu16.04.1×64
Application Layer php-fpm php-mysql Ubuntu16.04.1×64
Backend Layer-Database MySQL Ubuntu14.04.5×64
Backend Layer-File System Gluster Ubuntu16.04.1×64

Deployment Steps

Step 1 – MySQL Database

#update and install mysql server on DB01. A GUI window will appear to assign mysql root password
sudo apt-get update
sudo apt-get -y install mysql-server

#create 'wordpress1' database. '-u' followed by username.'-p' means password, it will require password in a separate line.
sudo mysqladmin -u root -p create wordpress1

#change root password if required.
sudo mysqladmin -u root -p password 

#enter mysql shell and enter password in separate line.
sudo mysql -u root -p

#create a user (CREATE USER 'wpuser1'), who can access from any host (@'%'), and with a password ('password'). Remember to add ';' at the end of each command under mysql shell to complete a command.
CREATE USER 'wpuser1'@'%' IDENTIFIED BY 'password';

#grant user 'wpuser1' full permission in particular to 'wordpress1'database, but not global database.
GRANT ALL PRIVILEGES ON wordpress1.* TO 'wpuser1'@'%';

#verify the existence of 'wordpress1' database and 'wpuser1'
show databases;
select User from mysql.user;

#Update database permissions and exit mysql shell
flush privilege;

#Edit mysql config file to update the bind address from to the actual private address. 
#Refer 'Note' section for details.
sudo nano /etc/mysql/my.cnf

#restart mysql service
sudo service mysql restart

Note – MySQL Bind Address: 

If the bind address in ‘my.cnf’ remains loopback, the database will not allow remote database access. When accessing the website, the following will show:
We will need to edit ‘/etc/mysql/my.cnf‘ as following, where ‘’ is the DB server’s private IP.

Then we execute ‘service mysql restart‘ to restart mysql service. The following screenshot shows the DB server was listening on localhost port 3306 before restarting mysql service; and listening on DB01’s IP after restarting the service.

I also checked firewall status to make no traffic is accidentally blocked.


To be Continued

Next Deploy Scalable and Reliable WordPress Site on LEMP(2)

Use pfSense to Load Balance Web Servers (2)

Use pfSense to Load Balance Web Servers (1) introduces pfSense, the lab setup, VM specs and download links. This blog will demonstrate pfSense configuration, test and troubleshooting details.


pfSense Configuration

An overview of pfSense configuration steps are as below along with key information for each step, testing and troubleshooting approach.

Step 1: Initial Configuration

Boot up pfSense VM and wait till installation is completed. Remove pfSense.iso image from the VM and reboot the VM. The following screen will show and guide you through the initial setup.

Select 2) to configure interface IPs. Please note LAN interface is the default management interface. In our case, we can access pfSense web GUI from

WAN interface requires default gateway address, ‘’ in our case. Routing can also be modified after accessing pfSense webconfig GUI.

Step 2: Access pfSense Web GUI

Access pfSense Web GUI from from the management PC The default username is ‘admin‘ and password ‘pfsense‘. User password can be changed under ‘System/User Management’ as below. Radius and LDAP authentication is also supported.

The default web GUI (HTTPS) port is 443. It can be changed to user-defined port number under’System/Advanced/Admin Access’, as below:

Step 3: Create Virtual IP

We need to create a virtual IP under ‘Firewall/Virtual IPs’, which will be used as load balancer’s virtual server IP later in Step 5. The virtual server IP will further forward traffic to the web servers in the load balancing pool. Please refer to the load balanced data flow diagram in Use pfSense to Load Balance Web Servers (1).

Create ‘IP Alias’ type virtual IP if there is single pfSense. Create ‘CARP’ type virtual IP if there are two pfSense in a cluster.CARP stands for ‘Common Address Redundancy Protocol’, functioning similar to VRRP and HSRP.
As part of testing/troubleshooting, please make sure the virtual IP is reachable from required subnet. Ping may be temporarily allowed for test purpose.Please note ‘ping’ is ICMP, neither TCP nor UDP.

Step 4: Create Load Balancer Pool

We then create load balancer pool where we can define member servers, under ‘Services/Load Balancer/Pools’. Default monitoring protocol includes ICMP, TCP, HTTP, HTTPS and SMTP. If additional protocol is required, it can be added under ‘Monitors’.

Step 5: Create Load Balancer Virtual Servers

Virtual server is created to host the load balancer’s shared IP. It uses the virtual IP we created before in Step 3. We also assign load balancer pool created in Step 4 to virtual server as below:

As part of testing/troubleshooting, please make sure no error under ‘Status/Load Balancer’ and ‘Status/System Logs/Load Balancer’. For HTTP and HTTPS traffic, if the load balancer members and/or the virtual server are not configured appropriate, the access may fallback to the pfSense web GUI.

Step 6: Tailor Firewall Rules

Since pfSense also functions as firewall, we will need to tailor the firewall rules to allow required traffic and block unwanted traffic. Firewall rules are configured under ‘Firewall/Rules’, as below:

Please note, pfSense firewall rules allow us to define traffic direction as well as application to the specified interface. For example, if we have traffic initiated from LAN to SVR; then we allow traffic from LAN net (all LAN subnet IPs) to SVR net (all SVR subnet IPs) and apply the rule to LAN interface on the pfSense. pfSense is stateful firewall by default, we don’t have to set up rules for the return traffic.

Another easy way to figure out what firewall rules are required is to block all uncertain traffic and check what traffic is blocked under ‘Status/System/Logs/Firewall’. Then pass the required traffic directly from the blocked list by clicking ‘+’, as blow:

Test Access to Load Balanced IP

We then test access to the load balanced IP. The network topology is in Use pfSense to Load Balance Web Servers (1).

User access the load balanced IPs from a computer over the Internet. When s/he access, the following shows:

The user access is load balanced between Server 1 and Server 2 in Cluster 1 as above screenshot.

Similarly, when the user access or, the following shows:

The user access is load balanced between Server 1 and Server 2 in Cluster 2 as above screenshot. and are examples of using internal IP as load balanced IP; while is example of using external IP as load balanced IP.

You may need to clear cache if browser is not working as expected.

Use pfSense as Layer2 Firewall/Bridged Interface

pfSense does support Layer 2 firewall mode (also called transparent mode) by bridging the required interfaces, under ‘Interfaces/(assign)/Bridges’ as below:

Layer 2 mode will allow the load balanced IP using external IP, while member servers also use external IP subnet. Use case example is as below:

pfSense firewall bridge configuration reference is available here.

Site-R1 Cisco 7200 Router Configuration

Site-Site-R1#show run
Current configuration : 1258 bytes
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
hostname Site-Site-R1
no aaa new-model
no ip icmp rate-limit unreachable
ip cef
ip tcp synwait-time 5
no ip domain lookup
multilink bundle-name authenticated
interface FastEthernet0/0
 ip address
 duplex full
interface Ethernet1/0
 ip address
 duplex full
interface Ethernet1/1
 no ip address
 duplex half
interface Ethernet1/2
 no ip address
 duplex half
interface Ethernet1/3
 no ip address
 duplex half
ip route
ip route
ip route
ip route
no ip http server
no ip http secure-server
logging alarm informational
no cdp log mismatch duplex
line con 0
 exec-timeout 0 0
 privilege level 15
 logging synchronous
 stopbits 1
line aux 0
 exec-timeout 0 0
 privilege level 15
 logging synchronous
 stopbits 1
line vty 0 4

Last But Not Least

  • Make sure routing, IP schema  and etc. are well planned.
  • Make sure only open minimum required ports on firewall.
  • Make sure proper zone segmentation using firewall to enforce security.
  • Use centrally managed authentication and authorisation, using remote user data source.

Use pfSense to Load Balance Web Servers (1)

What is pfSense?

pfSense is a FreeBSD-based distribution to be installed as physical or virtual machine. It can function as below:

  • Router
  • Firewall
  • Switch
  • Web proxy
  • Load balancer
  • supporting high availability
  • supporting Radius and LDAP authentication
  • Click here for pfSense function list

pfSense is managed via web GUI. HTTPS is enabled by default, while HTTP can be enabled if required.

pfSense provides incident-based support with cost. Their support hours are 7am-7pm CST. Should out-of-hour support be required, pfSense requires advance notice. Please refer here for pfSense support FAQ. This is to be considered in production as it may not meet SLA requirements in some organisations.

Also lab is not production. It is critical to take non-functional requirements such as supportability, scalability, availability/reliability, performance and etc. into consideration in production.

In addition, although pfSense is even more multi-functional than  Juniper SRX, security may require dedicated function per device and therefore multi-layer protection. A similar example in server infrastructure world is…we normally don’t put Active Directory (AD) and Certificate Authority (CA) on a same server.

Lab Topology

The lab is created using GNS3 with VMs hosted in VirtualBox. Please refer to my previous blog Install CSR1000v on GNS3 regarding how to import VMs into GNS3. I used dummy switch in this lab; however, proper layer 3 switch can be set up as described in my log GNS3 Lab: Connect to Physical LAN and Use Layer3 Switch.

Ubuntu Mate are used to simulate all servers and PCs. Apache is installed on Clst1-S1, Clst1-S2, Clst2-S1 and Clst2-S2 to simulate web servers.

pfSense is installed as VM on Virtual Box with 3 physical interfaces: WAN (e0), SVR(e2) and LAN(e1).pfSense webgui is accessible from Mgmt PC.

VM specs and software download links will be provided in next section.

The lab topology is as below:


Load balancing data flow is as below:

  1. User access load balancing virtual IP, which relay HTTP traffic to the two web servers in Cluster 1. pfSenseLB SVR interface (e2) IP is
  2. User access load balancing virtual IP, which relay HTTP traffic to the two web servers in Cluster 2.
  3. User access load balancing virtual IP, which relay HTTP traffic to the two web servers in Cluster 2. pfSenseLB WAN interface (e0) IP is


VM Specs and Software Download

Ubuntu Mate

Ubuntu Mate is available from here. VM specs are as below:

Attribute Value
Operating System Ubuntu (64-bit)
Storage 12 GB
Adapter 1 Not attached

Apache 2 is installed by running ‘sudo apt-get install apache2’.


pfSense is available from here. VM specs are as below:

Attribute Value
Operating System FreeBSD (64-bit)
RAM 500 MB
Storage 2 GB
Adapter 1,2,3 Not attached

To be Continued…

psSense configuration will be introduced in ‘Use pfSense to Load Balance Web Servers (2)’, which I will post over this week.

Next: Use pfSense to Load Balance Web Servers (2)

AWS Exam Preparation: Product Mindmap

Finally I got a chance to update the blog before October. It was a busy month. I got a new job hopefully to be more focused on cloud, preparing AWS exam, starting Linux/OpenStack architecture training and travel 🙂

As a fan of sharing, I put my exam prep in the following mind map, which not only helps exam, but also outline AWS products, if you are interested in what they do.