abidibo.net

search

Deploy django applications with nginx, uwsgi, virtualenv, south, git and fabric, part 4

deploy django fabric git programming virtualenv

This is the fourth part of the django deploy environment, you may find the third part here.

In this part we'll see how to freeze the project requirements, use versioning and prepare all in order to automate the deployment in production with fabric (which will be the argument of the last post, part 5).

Create a requirements.txt

As said in all previous posts, all the packages used by our project have been installed in a virtualenv which is strictly connected to our project. And if you were asking yourself why this is a convenient way to approach the django developement issue here comes the answer.

Activate the virtualenv, go inside the container folder of your django project and then perform this simple command

$ pip freeze > requirements.txt

This command simply creates a requirements.txt files which contains all the project dependencies, look at the image illustrating the project folder structure in the part 2, you'll see the same file there.

The nice part here is that you may setup an equivalent environment, with all the dependencies needed by your project, in a virtualenv created in a total different system (machine) only by running the command

$ pip install -r requirements.txt

So pip will install all the dependencies found in the requirements.txt sequentially for you. NIce isn't it?

Ok now we have our requierments file, it's time to think about versionig our project.

Versioning with git

You may prefer to use other versioning software like mercurius or subversion, but here we'll use git.

This is not the place to learn git commands, but basically cd into the project container folder and then perform the following ones:

$ git init
$ git add .
$ git commit -a -m "first commit"

Ok, now it's time to prepare the production environment!

Setting up the remote

In the part 1 we've installed the needed packages, now we have to set up the environment. Personally I follow these steps.

Create a new user

May be convenient to create a new user for each new web application, I'll create the user myproject tied to the web application myproject, so as root

# mkdir /home/myproject
# useradd -s /bin/bash -d /home/myproject myproject
# passwd myproject
# usermod -a -G sshlogin myproject

Such user will need some root privileges as a sudoer, you'd rather to set only the needed commands to perform as root (for example he have to be able to restart the server, we'll see this in the next post), here for simplicity we add him to the sudoers giving all privileges. To do so open the file /etc/sudoers and add the user through the following line:

myproject ALL=(ALL) ALL

Create a bare git repository

This repository will be added as a remote repository to our git locale repository. The idea is that when we make significant changes to our project, developing in locale, we commit the changes, push them to the remote and then update the production web application cloning the bare repository. So (as myproject user)

$ cd /home/myproject
$ mkdir git
$ mkdir git/repositories
$ cd git/repositories
$ mkdir myproject.git
$ cd myproject.git
$ git init --bare

Create a virtual host to serve our webapp

Create a new file under /usr/local/nginx/sites-available named myproject with the following content:

# file: /usr/local/nginx/sites-available/myproject
# nginx configuration for myproject
server {
  listen 80;
  server_name mydomain.com
  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;
  location /static {
    root /home/myproject/sites/myproject/shared;
  }
  location /media {
    root /home/myproject/sites/myproject;
  }
  location / {
    uwsgi_pass unix:/tmp/uwsgi_myproject.sock;
   include /usr/local/nginx/conf/uwsgi_params;
  }
}

Why the shared directory? We'll see it when speaking about fabric, in the next post.
Then enable the virtualhost and reload or restart the web server

$ ln -s /usr/local/nginx/sites-available/myproject /usr/local/nginx/sites-enabled/myproject
$ sudo /etc/init.d/nginx/restart

Let's create our database

Now it's time to create the project database and maybe create a specific user which can perform operations over it, I use mysql so

$ mysql -u root -p
mysql> CREATE DATABASE db_myproject;
mysql> USE mysql; mysql> INSERT INTO user (Host, User, Password) VALUES ('localhost', 'db_user', PASSWORD('db_password'));
mysql> INSERT INTO db (Host, Db, User, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Create_tmp_table_priv, Lock_tables_priv, Create_view_priv, Show_view_priv, Create_routine_priv, Alter_routine_priv, Execute_priv, Event_priv, Trigger_priv) VALUES ('localhost', 'db_myproject', 'db_user', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y');
mysql> FLUSH PRIVILEGES;

Connect local git repository to the remote one

Come back to the locale machine.

We've to connect the remote and locale repositories so cd into the project container folder and then

$ git remote add origin myproject@mydomain.com:git/repositories/myproject.git
$ git push origin master

You will be asked for a password, insert the user password chosen before and if all works as expected all your versioned code will be "copied" into the bare repository.

Need for some production settings? I think so

Since your django settings differs from locale to remote, create a settings_production.py file in the project container folder. Basically this file will be the same as the locale one, but with some changes:

change the DATABASES dictionary in order to fit your situation
change the DEBUG setting
change the MEDIA_ROOT value since it is an absolute path
The django way to treat MEDIA and STATIC paths has changed through years. But basically the MEDIA ROOT contains the file uploaded by the user and so can't be versioned! Static files instead are tied to the application itself and can be versioned, so in remote I use to set the MEDIA ROOT path outside the project folder, the STATIC ROOT inside. In locale I have the MEDIA ROOT inside the project folder but I use .gitignore to avoid versioning it.

I'll post here my two settings file, locale and production since when starting with django I think these are concepts which may confuse if you have a base php background for example.

Locale settings.py

# coding=utf-8
import os
PROJECT_ROOT = os.path.join(os.path.realpath(os.path.dirname(__file__)), '..')
DEBUG = True
TEMPLATE_DEBUG = DEBUG
ADMINS = (
('abidibo', 'abidibo@gmail.com'),
)
MANAGERS = ADMINS
DATABASES = {
  'default': {
    'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
    'NAME': 'db_myproject', # Or path to database file if using sqlite3.
    'USER': 'root', # Not used with sqlite3.
    'PASSWORD': '', # Not used with sqlite3.
    'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
    'PORT': '', # Set to empty string for default. Not used with sqlite3.
  }
}
# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'Europe/Rome'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en_en'

SITE_ID = 1

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True
# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale.
USE_L10N = True
# If you set this to False, Django will not use timezone-aware datetimes.
USE_TZ = True
# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/home/media/media.lawrence.com/media/"
MEDIA_ROOT = PROJECT_ROOT + '/media'
# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash.
# Examples: "http://media.lawrence.com/media/", "http://example.com/media/"
MEDIA_URL = '/media/'
# Absolute path to the directory static files should be collected to.
# Don't put anything in this directory yourself; store your static files
# in apps' "static/" subdirectories and in STATICFILES_DIRS.
# Example: "/home/media/media.lawrence.com/static/"
STATIC_ROOT = PROJECT_ROOT + '/static/'
# URL prefix for static files.
# Example: "http://media.lawrence.com/static/"
STATIC_URL = '/static/'
# Additional locations of static files
STATICFILES_DIRS = (
  # Put strings here, like "/home/html/static" or "C:/www/django/static".
  # Always use forward slashes, even on Windows.
  # Don't forget to use absolute paths, not relative paths.
  PROJECT_ROOT + '/myproject/static',
)

# List of finder classes that know how to find static files in
# various locations.
STATICFILES_FINDERS = (
  'django.contrib.staticfiles.finders.FileSystemFinder',
  'django.contrib.staticfiles.finders.AppDirectoriesFinder',
  # 'django.contrib.staticfiles.finders.DefaultStorageFinder',
)

# Make this unique, and don't share it with anybody.
SECRET_KEY = '99999999!q9jv_5xrhl5!)%9==5syki79y)ua*62sji6x$n9x('

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
  'django.template.loaders.filesystem.Loader',
  'django.template.loaders.app_directories.Loader',
  # 'django.template.loaders.eggs.Loader',
)

MIDDLEWARE_CLASSES = (
  'django.middleware.common.CommonMiddleware',
  'django.contrib.sessions.middleware.SessionMiddleware',
  'django.middleware.csrf.CsrfViewMiddleware',
  'django.contrib.auth.middleware.AuthenticationMiddleware',
  'django.contrib.messages.middleware.MessageMiddleware',
  # Uncomment the next line for simple clickjacking protection:
  # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

SESSION_ENGINE = 'django.contrib.sessions.backends.file'

ROOT_URLCONF = 'myproject.urls'

# Python dotted path to the WSGI application used by Django's runserver.
WSGI_APPLICATION = 'myproject.wsgi.application'

TEMPLATE_DIRS = (
  # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
  # Always use forward slashes, even on Windows.
  # Don't forget to use absolute paths, not relative paths.
  PROJECT_ROOT + '/myproject/templates',
)

INSTALLED_APPS = (
  'django.contrib.auth',
  'django.contrib.contenttypes',
  'django.contrib.sessions',
  'django.contrib.sites',
  'django.contrib.messages',
  'django.contrib.staticfiles',
  # Uncomment the next line to enable the admin:
  'django.contrib.admin',
  # Uncomment the next line to enable admin documentation:
  # 'django.contrib.admindocs',
  'django.contrib.flatpages',
  'south',
  'registration',
  'sorl.thumbnail',
  'myapp',
)

AUTH_PROFILE_MODULE = 'myapp.UserProfile'
ACCOUNT_ACTIVATION_DAYS = 3

""" email """
EMAIL_HOST = 'localhost'
EMAIL_PORT = 25

# A sample logging configuration. The only tangible logging
# performed by this configuration is to send an email to
# the site admins on every HTTP 500 error when DEBUG=False.
# See http://docs.djangoproject.com/en/dev/topics/logging for
# more details on how to customize your logging configuration.
LOGGING = {
  'version': 1,
  'disable_existing_loggers': False,
  'filters': {
    'require_debug_false': {
      '()': 'django.utils.log.RequireDebugFalse'
    }
  },
  'handlers': {
    'mail_admins': {
      'level': 'ERROR',
      'filters': ['require_debug_false'],
      'class': 'django.utils.log.AdminEmailHandler'
    }
  },
  'loggers': {
    'django.request': {
      'handlers': ['mail_admins'],
      'level': 'ERROR',
      'propagate': True,
    },
  }
}
 

settings_production.py for the same project, notice the MEDIA ROOT, DEBUG and DATABASES changes

# coding=utf-8

import os

PROJECT_ROOT = os.path.join(os.path.realpath(os.path.dirname(__file__)), '..')

DEBUG = False
TEMPLATE_DEBUG = DEBUG

ADMINS = (
('abidibo', 'abidibo@gmail.com'),
)

MANAGERS = ADMINS

DATABASES = {
  'default': {
    'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
    'NAME': 'db_myproject', # Or path to database file if using sqlite3.
    'USER': 'db_user', # Not used with sqlite3.
    'PASSWORD': 'db_password', # Not used with sqlite3.
    'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
    'PORT': '', # Set to empty string for default. Not used with sqlite3.
  }
}

# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'Europe/Rome'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en_en'

SITE_ID = 1

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale.
USE_L10N = True

# If you set this to False, Django will not use timezone-aware datetimes.
USE_TZ = True

# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/home/media/media.lawrence.com/media/"
MEDIA_ROOT = "/home/myproject/sites/myproject/media/"

# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash.
# Examples: "http://media.lawrence.com/media/", "http://example.com/media/"
MEDIA_URL = '/media/'

# Absolute path to the directory static files should be collected to.
# Don't put anything in this directory yourself; store your static files
# in apps' "static/" subdirectories and in STATICFILES_DIRS.
# Example: "/home/media/media.lawrence.com/static/"
STATIC_ROOT = PROJECT_ROOT + '/static/'

# URL prefix for static files.
# Example: "http://media.lawrence.com/static/"
STATIC_URL = '/static/'

# Additional locations of static files
STATICFILES_DIRS = (
  # Put strings here, like "/home/html/static" or "C:/www/django/static".
  # Always use forward slashes, even on Windows.
  # Don't forget to use absolute paths, not relative paths.
  PROJECT_ROOT + '/myproject/static',
)

# List of finder classes that know how to find static files in
# various locations.
STATICFILES_FINDERS = (
  'django.contrib.staticfiles.finders.FileSystemFinder',
  'django.contrib.staticfiles.finders.AppDirectoriesFinder',
  # 'django.contrib.staticfiles.finders.DefaultStorageFinder',
)

# Make this unique, and don't share it with anybody.
SECRET_KEY = 'kk4r7s_0!q9jv_5xrhl5!)%9==5syki79y)ua*62sji6x$n9x('

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
  'django.template.loaders.filesystem.Loader',
  'django.template.loaders.app_directories.Loader',
  # 'django.template.loaders.eggs.Loader',
)

MIDDLEWARE_CLASSES = (
  'django.middleware.common.CommonMiddleware',
  'django.contrib.sessions.middleware.SessionMiddleware',
  'django.middleware.csrf.CsrfViewMiddleware',
  'django.contrib.auth.middleware.AuthenticationMiddleware',
  'django.contrib.messages.middleware.MessageMiddleware',
  # Uncomment the next line for simple clickjacking protection:
  # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

SESSION_ENGINE = 'django.contrib.sessions.backends.file'

ROOT_URLCONF = 'myproject.urls'

# Python dotted path to the WSGI application used by Django's runserver.
WSGI_APPLICATION = 'myproject.wsgi.application'

TEMPLATE_DIRS = (
  # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
  # Always use forward slashes, even on Windows.
  # Don't forget to use absolute paths, not relative paths.
  PROJECT_ROOT + '/myproject/templates',
)

INSTALLED_APPS = (
  'django.contrib.auth',
  'django.contrib.contenttypes',
  'django.contrib.sessions',
  'django.contrib.sites',
  'django.contrib.messages',
  'django.contrib.staticfiles',
  # Uncomment the next line to enable the admin:
  'django.contrib.admin',
  # Uncomment the next line to enable admin documentation:
  # 'django.contrib.admindocs',
  'django.contrib.flatpages',
  'south',
  'registration',
  'sorl.thumbnail',
  'myapp',
)

AUTH_PROFILE_MODULE = 'myapp.UserProfile'
ACCOUNT_ACTIVATION_DAYS = 3

""" email """
EMAIL_HOST = 'localhost'
EMAIL_PORT = 25

# A sample logging configuration. The only tangible logging
# performed by this configuration is to send an email to
# the site admins on every HTTP 500 error when DEBUG=False.
# See http://docs.djangoproject.com/en/dev/topics/logging for
# more details on how to customize your logging configuration.
LOGGING = {
  'version': 1,
  'disable_existing_loggers': False,
  'filters': {
    'require_debug_false': {
      '()': 'django.utils.log.RequireDebugFalse'
    }
  },
  'handlers': {
    'mail_admins': {
      'level': 'ERROR',
      'filters': ['require_debug_false'],
      'class': 'django.utils.log.AdminEmailHandler'
    }
  },
  'loggers': {
    'django.request': {
      'handlers': ['mail_admins'],
      'level': 'ERROR',
      'propagate': True,
    },
  }
}

Ok the MEDIA ROOT in production is

MEDIA_ROOT = "/home/myproject/sites/myproject/media/"

What is sites/... ?

Simple, I put all the web applications of the user myproject under the directory /home/myproject/sites , indeed such user may have more applications or not? Why I've not created such directory when working over the remote? The answer is fabric, he'll do the work for me.

We'll use this file with fabric.

Here comes fabric

The goal of all this environment and deploy architecture is that I'd like to be able to do all the dirty stuffs in locale and when my project is stable enough I'd like to setup it in production this way:

$ fab production setup

Stop.

And when some changes are needed, or some bugs have to be corrected, I'd like to work in locale, reach a new stable version, commit it, update the remote git repository and update the production project this way:

$ fab production deploy

Stop.

Fabric will do all he work for me, update the code accordingly to the git repository, migrate the db, install new requirements, keep track of the previous release so that it's possible to undo if something went wrong, restart the services...

Next

Continue reading part 5, fabric rulez.

Subscribe to abidibo.net!

If you want to stay up to date with new contents published on this blog, then just enter your email address, and you will receive blog updates! You can set you preferences and decide to receive emails only when articles are posted regarding a precise topic.

I promise, you'll never receive spam or advertising of any kind from this subscription, just content updates.

Subscribe to this blog

Comments are welcome!

blog comments powered by Disqus

Your Smartwatch Loves Tasker!

Your Smartwatch Loves Tasker!

Now available for purchase!

Featured

Django admin and bootstrap 5

Bootstrap 5 has come, currently in beta release, and seems already very stable.

So the question is: are you looking for ...

About code optimization, learn from exercises

Let's see an example of exercise you can face during a job interview, and let's use it to understand some ...

Notes on the Pearson correlation coefficient

The Pearson correlation coefficient is a measure of the linear correlation between two variables X and Y. It has a ...