A dump of all the important information about MVP’s Digital Ocean servers and security setup.
apt-get updateapt-get dist-upgraderebootDifferent apt-get command options
Path: /etc/fail2ban
wordpress-hard.conf, wordpress-soft.conf files have been addedfail2ban-client.php WP Plugin interacts with wordpress-hard.conf to protect wp-login page from spamming failed logins.Some dummy on a company network screws up their login 5 times in a row and blocks everyone in the company from the website for X amount of time. (Here is syntax to fix it: fail2ban-client set <jail name> unbanip <ip>).
You can adjust failed login threshold by editing /etc/fail2ban/jail.d/wordpress.conf.
Sometimes websites behind Sucuri/Cloudflare firewalls will get blocked by fail2ban because all incoming IPs are the same.
The old way to fix this was to update the log format for that site to make use of the “X-Forwarded-For” (Sucuri) or “CF-Connecting-IP” (Cloudflare) headers. The firewall passes on the user ip in these headers so we could use that for fail2ban security instead. Examples of this exist on both apache and nginx servers already. Look into nginx.conf/apache.conf and look for the log format titled “sucuri” or “cloudflare”, the details are already setup and ready to implement.
A better way for NGINX servers is to use the ngx_http_realip_module. We can include the firewall IP addresses and where the real IP should be pulled from in those specific sitituations. See these articles from Cloudflare and Sucuri for more information.
As of Sep 17, 2020 these are the IPs we need to include:
Sucuri
# Define trusted Firewall IPs
set_real_ip_from 192.88.134.0/23;
set_real_ip_from 185.93.228.0/22;
set_real_ip_from 66.248.200.0/22;
set_real_ip_from 208.109.0.0/22;
set_real_ip_from 2a02:fe80::/29; # this line can be removed if IPv6 is disabled
# Define header with original client IP
real_ip_header X-Forwarded-For;
CloudFlare
# Define trusted Firewall IPs
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/12;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2c0f:f248::/32;
set_real_ip_from 2a06:98c0::/29;
# Define header with original client IP
real_ip_header CF-Connecting-IP;
Double check that no new IP address have been added since this guide was written.
fail2ban-client statusfail2ban-client status <jail name>Path: /etc/ufw
Servers: Prod1, legacy1, legacy2
Notes: Apache, PHP, no server-side caching
Path: /etc/apache2/
Test changes: apachectl -t
Servers: Prod2, etc
Notes: Nginx, PHP-FPM, PHP FastCGI cache
Path: /etc/nginx (/etc/nginx/global is where most of the config is located)
Example workflow setting up a new website on an MVP Nginx server.
Create a new folder in /var/www/<example.com> and populate it with the necessary directories. More information about these directories and what they do can be found at Nginx Website Folder Structure.
The following directories need to be created in the /var/www/example.com files/folder.
Note: Additional steps are required to get daily backups running and placed into the backups/ folder. See the description in Nginx Website Folder Structure for backups/.
In /etc/nginx/sites-available create a configuration file for the website.
Sample configuration files have been created and can be copied for easy setup. It is also easy to copy an existing website config file and find/replace.
These are the files that are left for reference in /etc/nginx/sites-available
Once your configuration file is setup symlink it to the /etc/nginx/sites-enabled/ folder with the following command:
ln -s \
/etc/nginx/sites-available/example.com \
/etc/nginx/sites-enabled/
Test changes: nginx -t
Restart or reload Nginx.
# (Recommended) Reload Nginx
service nginx reload
# Restart Nginx if you have to
service nginx restart
For MacOS users edit your /etc/hosts file. See example below using vim:
sudo vim /etc/hosts
# /etc/hosts
# Add this to the bottom of the file
12.34.56.78 example.com www.example.com
Note: Remember to remove this after testing is complete.
Generate TLS certificates post launch for both www and non-www.
certbot certonly --nginx -d example.com www.example.com
Test changes: nginx -t
Restart/reload Nginx.
Path: /etc/nginx/global
fpm package is installed along with it./etc/nginx/upstreams duplicate one of the upstream conf files and point the socket at your newly installed php. Update the variable name. It should be super straigtforward, basically just update the version number./etc/nginx/global/php-pool.conf update the default upstream to your new php using the variable name from step 2. Test nginx and reload.Thought: It should be possible to define unique PHP upstreams in the site-enabled/<site name> file.
Path: /etc/nginx/sites-enabled/<site name> and website folder
Pretty straight forwarded. Major caveat you need to make sure not to duplicate variable names. Redirect variable names are global. Using the site name in the variable is the easiest way to make it unique.
redirect-map.conf formats redirects as follows:
/old path/ /new path/;
Caches and reduces SQL queries. It is not really needed for generic PE websites since they hit the cache 99% of the time. Websites with either heavy backends or where users are logged in can benefit from Redis. Example: Namsa.com needs it, Roadtrek could use it.
WP Redis plugin is installed, it handles interaction and command running on the server. Pretty hands off.
The first time you enable Redis it can be a bit funky. I usually get logged out, and sometimes have to re-enable the plugin. Just do that a few times and you are golden.
redis-cli FLUSHALL
Path: /var/www/<site>
Website Folders:
backups/cache/logs/Deploybot Files/Folders:
deploy-cache/shared/releases/currentbackups/A backup script takes daily, weekly, monthly backups of uploads and database. The folder and it’s contents need to be owned by mvp:mvp because the backup script uses wp-cli which hates using root to run commands.
All the backups scripts are stored and run from /var/www/.tasks. You can update websites the script runs on by editing sites.sh and adding the new site to the array.
cache/Nginx handles folder permissions and file contents. This is the FastCGI cache location for this website. The WP Nginx-Cache plugin needs to reference this folder.
logs/Website error.log and access.log files go here. These paths are defined in the /etc/nginx/sites-enabled/<site conf> config file.
PHP-FPM Config Settings contains helpful maths for setting starting points for pm.max_children, pm.start_servers, pm.min/max_spare_servers.
MYSQL Tuner
curl -L http://mysqltuner.pl/ | perl
Apache2Buddy
curl -sL https://raw.githubusercontent.com/richardforth/apache2buddy/master/apache2buddy.pl | perl
See the SFTP page for more information on a possible SFTP workflow.
In principle it isn’t much harder than using LetsEncrypt. Instead of letting LE automate the generation of the certificates, the Commercial Certificate Agency (CA) does so and you manually drop them onto the server. The last step, pointing to the certificates from the website configuration file, remains the same.
Client’s usually ask us to generate the CSR which they will send to the CA to request the SSL certificate. It is possible to install a client provided CSR if needed, but we will need the private key too. So it is better/safer to handle it ourselves.
Generate the CSR and Private key using OpenSSL. (Replacing example.com).
openssl req -newkey rsa:2048 -keyout example.com.key -out example.com.csr
The client will take the CSR and return the certificate package (usually a zip). The package should provide an example.com.crt file and an intermediate.crt file.
Note on file names
Every CA formats names these files differently, which can be annoying. It may require some parsing to figure out which file is which. I suggest renaming the files once you figure it out.
The intermediate certificate is sometimes chained to the SSL certificate. Differences are partially due to CA’s trying to make it “easier” for people to install the certificates. They will ask the client for what environment the cert will be installed on, and they will preemptively do some things.
In the case of GoDaddy, the SSL certificate had a random name and the intermediate certificate bundle was titled gd_bundle-g2-1.crt (or at least it was that way once).
We need the private key, SSL certificate, and the intermediate certificate. For security reasons the private key should be kept somewhere only accessible by the root user. Besides that it doesn’t really matter where the certificates are stored. In the past I have kept them in the Apache/Nginx folders just for ease of access. (i.e. /etc/nginx/ssl/example.com/*).
Hopefully, the private key is called example.com.key the certificate is called example.com.crt and the intermediate cert is called intermediate.crt
Part of the reason the intermediate is sometimes chained to the SSL certificate is because for Nginx you need to combine the two.
cat example.com.crt intermediate.crt > example.com.chained.crt
Or simply copy paste into one file and rename.
Similar to adding certs from Lets Encrypt, in sites-enabled/example.com.conf point to where the SSL chained certificate and key are installed.
ssl_certificate /path/to/certificate/example.com.chained.crt;
ssl_certificate_key /path/to/certificate/example.com.key;
Test with nginx -t and restart if everything is good (service nginx restart)
Similar to adding certs from Lets Encrypt, in sites-available/example.com.conf point to where the SSL chained certificate and key are installed.
SSLCertificateFile /path/to/certificate/example.com.crt
SSLCertificateKeyFile /path/to/certificate/example.com.key
SSLCACertificateFile /path/to/certificate/intermediate.crt
You may need to enable the Apache SSL mod, a2enmod ssl.
Test apachectl -t and restart if everything is good (service apache2 restart)
If the client has an external firewall, things are more complicated. For Mason Wells I opted just to have Sucuri support handle everything. They can generate the CSR and install the certs.
If you want to install on the server and make it worth with an external firewall like Sucuri/Cloudflare… good luck.
You can fine tune your backup strategy per backup script. At the end of each daily/weekly/monthly there is a commented section called Cleanup old backups. Change the ages change how long backups are held.
I have been reducing backup durations due to storage concerns with some of the bigger websites. Holding onto the uploads folder takes up a lot of space. Even gzipped like it is.
Currently this is the configuration (numbers are in days):
| Script | Uploads | Database |
|---|---|---|
| Daily | 4 | 7 |
| Weekly | 42 | 63 |
| Monthly | 128 | 365 |
Start off by copying the entire backup configuration folder from a working setup. This can probably be found at /var/www/.tasks/. Included in this folder are the following files:
sites.shdaily_backups.shweekly_backups.shmonthly_backups.shIn order to generate the database backups the use of WP-CLI is required. See here for installation instructions. The WP-CLI will take care of the hard work of connected to each website DB and generating the backup.
The sites.sh file is the only file that needs to be updated when adding/removing websites. It includes references for the web root folder, and a list of all the websites that need backups. The daily/weekly/monthly backup scripts reference the website list found here.
The primary quirk of this arrangement is that WP-CLI HATES running as the root user. For that reason I have had to create a new user, usually named mvp.
# Add new user, create home directory, specify shell access
useradd mvp -m -s /bin/bash
# Add a group
addgroup mvp
# Add new user to new group
usermod -aG mvp mvp
After setting up the user I will copy and paste the ~/.ssh/ folder from the root user. Since I am the only person accessing the new mvp user it is fine to reuse my key.
cp -r ~/.ssh/ /home/mvp/
Open a new terminal window and try connecting to the server using the new user.
Using the new server user (mvp) set the cron jobs. These will run daily, weekly, and monthly at like 3 in the morning.
Open crontab using the following command.
crontab -e
This is how the cron works now. Change timing or script locations to meet your needs.
# Daily Backups
0 3 * * * /bin/bash /var/www/.tasks/daily_backups.sh > /dev/null 2>&1
# Weekly Backups
15 3 * * 1 /bin/bash /var/www/.tasks/weekly_backups.sh > /dev/null 2>&1
# Monthly Backups
30 3 1 * * /bin/bash /var/www/.tasks/monthly_backups.sh > /dev/null 2>&1
Lastly, make sure that the new user (mvp) has permission to write to the backups folder.
chown -R mvp:mvp /var/www/example.com/backups/