Why does Django randomly break when I use a deleted model in a migration?

I inherited a messy project, so as I've been cleaning up, I've deleted a lot of models. Sometimes, I have to copy date out of the model into another one before deletion. I've been writing migrations that do both. Everything is working fine - sometimes. And seemingly randomly, sometimes it doesn't work.

Process:

  • Make the necessary changes to the codebase, including deleting the model and references to it.
  • Create a migration which:
    • copies data out of model
    • deletes the model

Here's an example of what such a migration would look like:

from django.db import migrations, models

def doit(apps, schema_editor):
    OldModel = apps.get_model("myapp", "OldModel")
    NewModel = apps.get_model("myapp", "NewModel")
    for obj in OldModel.objects.all():
        ...transform and save data in NewModel...

class Migration(migrations.Migration):
    dependencies = [("myapp", "0001_initial")]
    operations = [
        migrations.RunPython(doit, reverse_code=migrations.RunPython.noop),
        migrations.DeleteModel(name="OldModel"),
    ]

Sometimes the migration runs as expected. Other times the migration fails with:

LookupError: App 'myapp' doesn't have a 'OldModel' model.

I cannot find any notable difference between the migrations that work and the migrations that blow up this way. I've had multiple examples of each, and have wracked my brain trying to spot the difference.

My understanding is that the apps object passed to your function by the RunPython() method provides a simulation of all models as they exist at that point in the migration chain, regardless of the state of your codebase, so it shouldn't matter that the model no longer exists.

Yet, randomly, sometimes it does?

Just had a minor breakthrough. Here's a snippet of my current migration for context:

def migrate__fundmanager(apps, schema_editor):
    Company = apps.get_model("everest", "Company")
    Fund = apps.get_model("everest", "Fund")
    FundManager = apps.get_model("everest", "FundManager")

    for fm in FundManager.objects.all():
        ...

At that point in the migration, I got a LookupError for FundManager - even though it literally was findable two lines up.

Examining the tracebook yielded several clues:

Traceback (most recent call last):
  File "/code/manage.py", line 20, in <module>
    main()
  File "/code/manage.py", line 16, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/code/everest/migrations/0028_company.py", line 10, in migrate__fundmanager
    for fm in FundManager.objects.all():
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 280, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 69, in __iter__
    obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 515, in from_db
    new = cls(*values)
  File "/usr/local/lib/python3.10/site-packages/dirtyfields/dirtyfields.py", line 37, in __init__
    reset_state(sender=self.__class__, instance=self)
  File "/usr/local/lib/python3.10/site-packages/dirtyfields/dirtyfields.py", line 163, in reset_state
    new_state = instance._as_dict(check_relationship=True)
  File "/usr/local/lib/python3.10/site-packages/dirtyfields/dirtyfields.py", line 99, in _as_dict
    all_field[field.name] = deepcopy(field_value)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 265, in _reconstruct
    y = func(*args)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 2154, in model_unpickle
    model = apps.get_model(*model_id)
  File "/usr/local/lib/python3.10/site-packages/django/apps/registry.py", line 211, in get_model
    return app_config.get_model(model_name, require_ready=require_ready)
  File "/usr/local/lib/python3.10/site-packages/django/apps/config.py", line 270, in get_model
    raise LookupError(
LookupError: App 'everest' doesn't have a 'FundManager' model.
  • the dirtyfields package is being invoked, which is a reminder that the "simulated model" is only skin-deep - the real base classes below that are still referenced
  • the thread of execution going through dirtyfields is leading to another call to apps.get_model() for "FundManager", but this time it's the real django.apps, not the simulated one used in the migration itself

So that's what I think's causing it. I have no idea what the right solution is. All that comes to mind is:

  1. avoid using cool base classes like the dirtyfields package (not a good option, as it's so useful)
  2. use RunSQL to do data migration instead of RunPython (often very difficult compared to a Python solution)

UPDATE: Just thought of a third option - editing my "0001_initial.py" migration and removing the dirtyfields base class from FundManager's CreateModel statement:

bases=(dirtyfields.dirtyfields.DirtyFieldsMixin, models.Model),

That worked. I still have absolutely no clue why this was necessary for this model but not the other nearly identical one that I'm removing in the same migration, and which was removable without LookupError despite still having it's dirtyfields base class attached...

Back to Top