Django model migrations

At the start of this chapter you learned how Django models are closely tied to the migrations. Recapping, the migrations consists of registering the evolution of Django models in an app's models.py file into 'migration files', with the purpose of later reviewing and applying these models.py changes to a database.

In essence, migration files serve as a buffer between a database and the changes made to a Django project's models defined in models.py files. With data being such a delicate piece of a project, migration files provide highly-desirable functionalities, such as the ability to preview database change before they're committed (e.g. sqlmigrate in listing 7-4) and also the ability to go back to a certain point in time in a models.py file state, reverting what are generally complex DDL structure changes.

Migration file creation

The manage.py makemigrations command is the entry point to create migration files. If you execute this command without arguments, Django inspects all the models.py files for apps declared in the INSTALLED_APPS variable and creates a migration file for apps whose models.py contents has changed from prior migrations. Table 7-4 describes the most common makemigrations arguments and their purpose.

Table 7-4. Django most used makemigrations arguments

As you can see in table 7-4, the makemigrations command offers multiple ways to create migration files. You can create empty migration files, you can simulate the creation of migration files to inspect changes first, and you can also create migration files with a specific name, among other things.

Migration file renaming

Migration files are not set in stone, so it's possible to rename migration files. What steps you need to take to rename a migration file, depend on whether a migration has been applied to a database o not. To determine the state of a migration file with respect to a database, execute the python manage.py showmigrations, if a migration file has an X beside it, it means it has been applied to the database.

For migrations that haven't been applied to the database, you can rename a migration file directly in the migrations folder to a more descriptive name. At this point, the migration file is just a representation of model changes that no one else knows about, so you can even delete the migration file if needed.

For migrations that have been applied to a database you have two alternatives. The first option is to rename the migration file, alter the database table that holds the migration activity to reflect this new name and update other migration file dependencies (if any) to also reflect this new name. Once you rename the migration file inside the migrations folder, access the django_migrations database table and look for the record with the old migration file name and update it to the reflect the new migration name. Next, if there's a newer migration file than the one you're renaming, the newer migration file will have a dependencies statement -- described in the migration file structure section -- that must be updated with the new name.

The second alternative is to rollback to a migration prior to the migration file you want to rename, at which point you can simply rename the migration file -- as an un-applied database migration file -- and then re-apply the migration process back to the most recent migration file. An upcoming section describes migration file rollback in greater detail.

Migration file squashing

A models.py files that undergoes many changes can generate dozens or even hundreds of migrations files. In these circumstances, it's possible to squash multiple migration files into a single migration file to simplify file migration management. Note the term 'squash' is used vs. the more technically accurate term 'merge', because migration file merging refers to conflicting migration files, see the --merge option in table 7-4.

The manage.py squashmigrations command is designed to squash multiple migration files, its syntax is the following:

manage.py squashmigrations <app_name> <squash_up_to_migration_file_serial_number>

As you can see, the squashmigrations command requires you specify both the app on which you want to squash migration files, as well as the migration serial number up to which you want to squash (e.g. squashmigrations stores 0004, generates a single migration file for the stores app from the migration files 0001, 0002, 0003 and 0004).

The squashmigrations command also supports an additional positional argument to change the start of the squashing process from the default 0001 (e.g. squashmigrations stores 0002 0004, generates a single migration file from the migration files 0002, 0003 and 0004)

Like all file merging mechanisms, there's always a possibility squashmigrations may not be able to produce automatic results, in which case it generates the message 'Manual porting required', where it's necessary to manually edit the squashed migration file (e.g. just like it can happen with other file merging conflict operations in platforms like git).

Squashed migration files follow the naming convention:

<initial_serial_number>_squashed_<up_to_serial_number>_<date>.py 

You can rename squashed migration file just like regular migration files, just follow the same steps described in the previous section, depending on whether the squashed migration file has been applied to a database or not.

Squashed migration files take over the duties of un-squashed migration files. You can keep the old (un-squashed) migration files as long as you want, but they only continue to serve a purpose until the squashed migration file is applied to a database. Behind the scenes, squashed migration files use the replaces migration field -- described in the next section -- to indicate which migration files it replaces. Therefore once you apply a squashed migration file to a database, the migration files in replaces are ignored.

Migration file structure

Although migration files are automatically created based on the changes made to models.py files vs. the prior migration files belonging to the same models.py files, this doesn't mean you can't or won't have to change the internal structure of migration files. Listing 7-30 illustrates the basic structure of a Django migration file.

Listing 7-30 Django migration file basic structure

from django.db import migrations, models

class Migration(migrations.Migration):

    initial = True
    
    replaces = [
    ]
    
    dependencies = [
    ]
    
    operations = [
    ]

First, notice in listing 7-30 all migration files include a class named Migration that inherits its behavior from django.db.migrations.Migration. This allows migrations to automatically receive a series of default behaviors, similar to how Django model classes inherit their behavior from the django.db.models.Model class.

Inside each Migration class are a series of fields which determine the actions of the migration file. The initial field is a boolean value present on the initial migration file for every app (i.e. migration files with the 0001 serial number). The replaces field is a list field used by squashed migration files to declare which migration files it replaces, a value which is automatically populated when you create a squashing migration file.

The dependencies and operations fields are by far the two most common fields in migrations files. Although they're automatically populated once a migration file is created -- just like other migration file fields -- these two fields are the ones you're most likely to change if you require adjusting the logic executed by a migration file.

The dependencies field is a list of tuples with the ('<app_name>','<migration_file>') syntax, where each tuple represents a migration dependency. For example, by default the second migration file for an app named about contains the following dependencies value:

    dependencies = [
        ('about', '0001_initial'),
    ]

This tells Django the migration file depends on the execution of the migration file 0001_initial in the about app, ensuring this last migration file is run first.

The most common scenario for editing the dependencies field is to add inter-app migration file dependencies. For example, if the online app depends on data from the stores app created by its 0002_data_population migration file, you can add a dependency tuple to the online app's first migration file to ensure it's run after the stores migration files (e.g.('stores', '0002_data_population')).

The operations field declares a list of migration operations^[7]. Migration operations include all database related tasks performed by migrations. If you were wondering how Django generates the DDL to create, delete, alter or rename the database table behind a model, it's all based on migration operations.

For most cases, Django generates the migration operations based on the changes made to models in a models.py file. For example, if you add a new model, the next migration file includes a django.db.migrations.operations.CreateModel() migration operation; if you rename a model in the models.py, the next migration file includes a RenameModel() operation from the same django.db.migrations.operations package; this same mechanism occurs when you change a model field (AlterModel()), add an index (AddIndex()) and perform all the other modifications possible to models in a models.py file.

The most common scenario for editing the operations field in a migration file is to add non-DDL operations (e.g. SQL DML- Data Manipulation Language) which can't be reflected as part of model changes. For example, you can insert SQL queries as part of a migration file through the RunSQL migration operation and you can also run Python logic as part of a migration file through the RunPython migration operation. The upcoming section 'Django model initial data set up' describes how to use the RunSQL and RunPython migration operations.

Migration file rollback

Reverting a database to a previous state of a Django model can be done by rolling back migration files. Reverting a database to a previous migration file is as simple as passing an additional argument to the same migrate command that applies migration files. For example, the migrate stores 0001 statement tells Django to migrate the stores app to the the 0001 migration file, if the app's database state is in a more recent migration file (e.g. 0004), Django rollsback migration files until the database reflects the 0001 migration file.

But as simple as the migration file rollback command is, the actual rollback process is anything but simple. Since migration files can contain multiple DDL and DML operations -- as described in the previous section -- there are certain migration operations that are considered irreversible. This means that once a migration is applied, Django can't determine with certainty how to undo it.

When a rollback is attempted on a migration with an irreversible operation, Django throws the error django.db.migrations.exceptions.IrreversibleError. Of course, irreversible does not mean impossible, but it does mean additional work to make a migration file reversible.

Most irreversible migration operations happen on DDL migration operations (e.g. RunSQL, RunPython) where you execute certain logic as part of the migration file. To make these type of migration operations reversible, you must equally provide the logic to revert the logic applied as part of the migration file. The upcoming section 'Django model initial data set up' describes how to create reverse operations for the RunSQL and RunPython migration operations.

1. https://docs.djangoproject.com/en/1.11/ref/migration-operations/     ↑