Emulate remote servers for web applications with VirtualBox - Part 2

In Part 1 of this guide, we showed how to configure VirtualBox to emulate remote servers for web applications. Now, let's see how to configure a Django application using NGINX on this “remote” server.

In this guide, we will create and configure a PostgreSQL database to serve a Django application. Let's configure NGINX as a reverse proxy to interact with our application. And we will also configure Gunicorn as an adapter between the entry point of the NGINX server and the Django application.

The following instructions have been tested on an Ubuntu 20.04.1 LTS x86_64. In the configuration presented, the Host is our physical machine, and the Guest (our virtual machine) represents the remote server.

Create a web application

Create a database with Postgres

Connected to the Guest (that means, to the server), update the package manager:

myGuestUser@guest:~$ sudo apt-get update && sudo apt-get upgrade

And install the necessary dependencies:

$ sudo apt-get install python3-dev python3-pip python3-virtualenv libpq-dev postgresql postgresql-contrib
...
$ python3 --version
Python 3.8.5

Let's start by creating a database and a bank user for our application. Open an interactive Postgres session:

$ sudo -u postgres psql

Create a database for the project, together with a user to access it:

postgres=# CREATE DATABASE myproject;
postgres=# CREATE USER luke WITH PASSWORD 'password';

Take the opportunity to configure the bank according to Django's requirements.

postgres=# ALTER ROLE luke SET client_encoding TO 'utf8';
postgres=# ALTER ROLE luke SET default_transaction_isolation TO 'read committed';
postgres=# ALTER ROLE luke SET timezone TO 'UTC';

Provide the new user with access to manage the created bank. And, when you're done, exit the Postgres prompt.

postgres=# GRANT ALL PRIVILEGES ON DATABASE myproject TO myGuestUser;
postgres=# \q

Postgres is now configured so that Django can connect to your database and manage your information.

Create a virtual environment for Python

To make it easier to manage the application and its dependencies, create an isolated Python environment. Start by installing virtualenv:

$ sudo -H pip3 install --upgrade pip
$ sudo -H pip3 install virtualenv

Then, create a directory to hold the project files:

$ mkdir ~/myprojectdir
$ cd ~/myprojectdir

And create a virtual Python environment, and activate it:

$ virtualenv myprojectenv
$ source myprojectenv/bin/activate

Your prompt should change to indicate that you are now operating in a Python virtual environment. You should see something like:

(myprojectenv) myGuestUser@guest:~/myprojectdir$

With your virtual environment active, install the necessary dependencies to start a Django project:

(myprojectenv)$ pip install django gunicorn psycopg2-binary

Create a Django project

Good! We can now create a new Django project:

(myprojectenv)$ django-admin.py startproject myproject ~/myprojectdir

The first thing we should do with our newly created project files is to adjust the settings in settings.py. Make sure to include the address assigned to the Guest in ALLOWED_HOSTS.

// settings.py
...
ALLOWED_HOSTS = ['0.0.0.0', '192.168.11.89']

Then find the section that sets up access to the database. Tell Django to use the psycopg2 adapter:

// settings.py
...
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'myproject',
        'USER': 'myGuestUser',
        'PASSWORD': 'password',
        'HOST': 'localhost',
        'PORT': '',
    }
}

Finally, add a configuration indicating where the static files should be stored.

// settings.py
...
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

Remember to save the file when you're done. Now, we can migrate the initial database schema to our PostgreSQL database:

(myprojectenv) myGuestUser@guest:~/myprojectdir$ ./manage.py makemigrations
(myprojectenv) myGuestUser@guest:~/myprojectdir$ ./manage.py migrate

Create an administrative user for the project:

(myprojectenv) myGuestUser@guest:~/myprojectdir$ ./manage.py createsuperuser

You will have to select a username, provide an email address, and choose a password. When everything is done, collect the static files generated by the application. They will be stored in the directory indicated in the configuration file.

(myprojectenv) myGuestUser@guest:~/myprojectdir$ ./manage.py collectstatic

Now we can test the application by starting the server:

(myprojectenv) myGuestUser@guest:~/myprojectdir$ ./manage.py runserver 0.0.0.0:8000

In the Host's web browser, visit the Guest's IP address over port 8000:

http://192.168.11.89:8000

You should see the default Django home page:

Attention: To access the server from the Guest, you can use a text browser:

$ sudo apt-get install links

In this case, remember to include the address of the local server in the ALLOWED_HOSTS list of the settings.py file:

// settings.py
...
ALLOWED_HOSTS = ['0.0.0.0', '192.168.11.89']

When accessing the application's address, you should see something like:

$ links 0.0.0.0:8000
django

   View release notes for Django 3.1

The install worked successfully! Congratulations!
...

Connect the application with the server

Connect the Django app with Gunicorn

We can end the application by pressing CTRL+C, to test if we can access it using Gunicorn as an entry point. Start the Gunicorn interface with the command:

(myprojectenv) myGuestUser@guest:~/myprojectdir$ gunicorn --bind 0.0.0.0:8000 myproject.wsgi

And access the application again through the browser. If everything goes well, stop the application again with CTRL+C, and exit the virtual environment with the command:

(myprojectenv) $ deactivate

With that, we can implement a more robust way to execute this interface, using an operating system socket. With super user permission, create a socket file for Gunicorn:

$ sudo vim /etc/systemd/system/gunicorn.socket

And add the following content to the created file to describe the socket:

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock

[Install]
WantedBy=sockets.target

Then, create a service file for Gunicorn, also with super user permission:

$ sudo vim /etc/systemd/system/gunicorn.service

In the [Unit] section of this file, connect the service with the created socket. In the [Service] section, specify the user and the group that will have control over the process performed, the path to the application directory, and the command to be executed to start the service. Finally, in the [Install] section, indicate that the service should be started with the boot.

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=luke
Group=www-data
WorkingDirectory=/home/myGuestUser/myprojectdir
ExecStart=/home/myGuestUser/myprojectdir/myprojectenv/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          myproject.wsgi:application

[Install]
WantedBy=multi-user.target

Now, start and enable the Gunicorn socket.

$ sudo systemctl start gunicorn.socket
$ sudo systemctl enable gunicorn.socket
Created symlink /etc/systemd/system/sockets.target.wants/gunicorn.socket → /etc/systemd/system/gunicorn.socket.

This will create the socket file in /run/gunicorn.sock now and at boot. We can confirm that the operation was successful by verifying that the socket file was actually created.

$ file /run/gunicorn.sock
/run/gunicorn.sock: socket

Check the status of the process to find out if it was able to start:

$ sudo systemctl status gunicorn.socket
● gunicorn.socket - unicorn socket
     Loaded: loaded (/etc/systemd/system/gunicorn.socket; enabled; vendor preset: enabled)
     Active: active (listening) since Sat 2021-01-16 05:25:06 UTC; 10min ago
   Triggers: ● gunicorn.service
     Listen: /run/gunicorn.sock (Stream)
      Tasks: 0 (limit: 1074)
     Memory: 0B
     CGroup: /system.slice/gunicorn.socket

Jan 16 05:25:06 venus systemd[1]: Listening on unicorn socket.

If the systemctl status command indicated that an error occurred, or if the gunicorn.sock file was not found, it is likely that the Gunicorn socket was not created correctly. Check the socket records with the command:

$ sudo journalctl -u gunicorn.socket

Check the /etc/systemd/system/gunicorn.socket file again to correct any problems before proceeding.

We can now test the socket activation mechanism. The gunicorn.service is not yet active, as the socket has not yet received any connections.

$ sudo systemctl status gunicorn
● gunicorn.service - gunicorn daemon
     Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● gunicorn.socket

Send a connection to the socket:

$ curl --unix-socket /run/gunicorn.sock localhost

The HTML output of the application must be seen in the terminal. This indicates that Gunicorn was started and was able to serve the Django application. Check that the Gunicorn service is now active:

$ sudo systemctl status gunicorn
● gunicorn.service - gunicorn daemon
     Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-01-16 06:44:11 UTC; 2min 1s ago
TriggeredBy: ● gunicorn.socket
   Main PID: 1049 (gunicorn)
      Tasks: 4 (limit: 1074)
...

If the result of curl or the result of the systemctl status command indicates that a problem has occurred, check the logs for more details:

$ sudo systemctl status gunicorn

Check the /etc/systemd/gunicorn.service file for problems. And, if you make changes, restart the process with the commands:

$ sudo systemctl daemon-reload
$ sudo systemctl restart gunicorn

Connect the Gunicorn adapter with NGINX

Now that Gunicorn is configured, we can install NGINX:

$ sudo apt-get install nginx

Next, we need to configure NGINX to drive traffic to the Django application. With super user permission, create the following file:

$ sudo vim /etc/nginx/sites-available/myproject

In this file, open a new block to configure the server, specifying its gateway and its access IP address. At this moment, tell NGINX where to find the application's static files, and forward all requests directly to the Gunicorn socket.

# /etc/nginx/sites-available/myproject

server {
    listen 80;
    server_name 192.168.11.89;

    location /static/ {
        root /home/myGuestUser/myprojectdir;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:/run/gunicorn.sock;
    }
}

When you're done, create a symbolic link to this file in the sites-enabled directory:

$ sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled

Test if the configuration has any syntax errors:

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

If no errors are reported, restart NGINX:

$ sudo systemctl restart nginx

Et voilá! In the Host's web browser, visit the Guest's IP address, this time over port 80:

http://192.168.11.89:80

You should see the default Django homepage again!

Attention: If you encounter problems, consult the following logs for indicative messages:

NGINX process logs: sudo journalctl -u nginx

NGINX access logs: sudo less /var/log/nginx/access.log

NGINX error logs: sudo less /var/log/nginx/error.log

Gunicorn service records: sudo journalctl -u gunicorn

Gunicorn socket logs: sudo journalctl -u gunicorn.socket

If you update any settings, remember to restart the processes to load the changes: - If you update the Django application, restart the Gunicorn process:

$ sudo systemctl restart gunicorn

If you change the socket or the Gunicorn service, reload the daemon and restart these processes:

$ sudo systemctl daemon-reload
$ sudo systemctl restart gunicorn.socket gunicorn.service

If you change the NGINX configuration, test the configuration and restart NGINX:

$ sudo nginx -t
$ sudo systemctl restart nginx

Conclusions

That’s it! We now have a Django application running on a virtual “remote” server. Our application is connected to a PostgreSQL database, and to the Gunicorn request adapter. And the connections between the adapter and customer requests are served by NGINX.

From here, if you want to make your application more robust, try to protect traffic to your server using SSL/TLS using Let’s Encrypt, for example. Or maybe try to connect NGINX to a process controller, such as the supervisor.

Did you like this post? Did you have any questions? Do you have any suggestions? Leave a comment!