Django Mongodb Backend not creating collections and indexes

Summary Running Django migrations against our MongoDB database does not create MongoDB collections or indexes as defined in our app. The command completes without errors, but no collections or indexes are provisioned in MongoDB. Environment Django: 5.2.5 django-mongodb-backend: 5.2.2 Python: 3.11.14 Database setup: PostgreSQL as default, MongoDB as secondary via django-mongodb-backend Steps to Reproduce

  • Configure DATABASES with a mongodb alias (see snippet below).
  • Implement models that should live in MongoDB and include indexes/constraints.
  • Implement a database router that routes models with use_db = "mongodb" to the mongodb DB. Run:
  • python manage.py makemigrations mailbot_search_agent
  • python manage.py migrate mailbot_search_agent --database=mongodb

Expected

  • MongoDB collections are created for the models that declare use_db = "mongodb".
  • Declared indexes and unique constraints are created.
  • If supported by backend, custom Atlas Search/Vector index definitions are applied. Actual migrate --database=mongodb completes, but:
  • Collections are not created (or get created only after first write).
  • Indexes defined in migrations (0002) and in model Meta/indexes are not present in MongoDB.
  • Atlas Search/Vector indexes (declared via backend-provided Index classes) are not created.

DATABASES Configuration (snippets)

MONGO_CONNECTION_STRING = os.environ.get("MONGO_CONNECTION_STRING")
MONGO_DB_NAME = os.environ.get("MONGO_DB_NAME", "execfn")

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "execfn",
        "USER": "execfn_user",
        "PASSWORD": os.environ.get("DJANGO_DB_PASSWORD"),
        "HOST": "localhost",
        "PORT": "5432",
    },
    "mongodb": {
        "ENGINE": "django_mongodb_backend",
        "NAME": "execfn",
        "HOST": os.environ.get("MONGO_CONNECTION_STRING"),
    },
}

Router (snippets)

class DatabaseRouter:
    MONGODB = "mongodb"
    DEFAULT = "default"

    def _get_model_db(self, model):
        # Class-level attribute `use_db` drives routing
        use_db = getattr(model, "use_db", None)
        return use_db or self.DEFAULT

    def db_for_read(self, model, **hints):
        return self._get_model_db(model)

    def db_for_write(self, model, **hints):
        return self._get_model_db(model)

    def allow_relation(self, obj1, obj2, **hints):
        db1 = self._get_model_db(obj1._meta.model if hasattr(obj1, "_meta") else type(obj1))
        db2 = self._get_model_db(obj2._meta.model if hasattr(obj2, "_meta") else type(obj2))
        return db1 == db2

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        if "model" in hints:
            model = hints["model"]
            return db == self._get_model_db(model)

        if model_name is not None:
            try:
                from django.apps import apps
                model = apps.get_model(app_label, model_name)
                return db == self._get_model_db(model)
            except Exception:
                pass

        if db == self.DEFAULT:
            return True
        if db == self.MONGODB:
            return False

        return False

Model Definitions (snippets)

# mailbot_search_agent/models.py
from django.db import models
from django_mongodb_backend.fields import ArrayField, ObjectIdAutoField
from django_mongodb_backend.indexes import SearchIndex, VectorSearchIndex
from execfn.common.models import MongoDBModel  # abstract, sets use_db="mongodb"

class Email(MongoDBModel):
    use_db = "mongodb"
    _id = ObjectIdAutoField(primary_key=True)
    message_id = models.CharField(max_length=255)
    thread_id = models.CharField(max_length=255)
    user_id = models.IntegerField()
    subject = models.CharField(max_length=512)
    snippet = models.TextField(null=True)
    received_at = models.DateTimeField()
    labels = models.JSONField(default=list)
    content = models.TextField()
    from_email = models.CharField(max_length=320)
    from_name = models.CharField(max_length=255)
    to_emails = ArrayField(models.CharField(max_length=320), default=list)
    to_names  = ArrayField(models.CharField(max_length=255), default=list)
    # cc_/bcc_ omitted for brevity

    class Meta:
        db_table = "emails"
        constraints = [
            models.UniqueConstraint(fields=["message_id"], name="message_id_unique"),
        ]
        indexes = [
            SearchIndex(
                field_mappings={
                    "subject": {"type": "string"},
                    "content": {"type": "string"},
                    "from_email": {"type": "string"},
                    "from_name": {"type": "string"},
                },
                name="emails_text_search_index",
            ),
            models.Index(fields=["user_id"]),
        ]

class EmailEmbedding(MongoDBModel):
    use_db = "mongodb"
    _id = ObjectIdAutoField(primary_key=True)
    gmail_message_id = models.CharField(max_length=255)
    user_id = models.IntegerField()
    subject = models.CharField(max_length=512)
    content = models.TextField()
    received_at = models.DateTimeField()
    labels = ArrayField(models.CharField(max_length=255), default=list)
    chunk_start = models.IntegerField()
    chunk_end = models.IntegerField()
    embedding = ArrayField(models.FloatField(), size=1536)
    from_email = models.CharField(max_length=320)
    from_name = models.CharField(max_length=255)

    class Meta:
        db_table = "email_embeddings"
        constraints = [
            models.UniqueConstraint(
                fields=["gmail_message_id", "chunk_start", "chunk_end"],
                name="embedding_chunk_unique",
            )
        ]
        indexes = [
            models.Index(fields=["user_id"]),
            SearchIndex(
                field_mappings={
                    "subject": {"type": "string"},
                    "content": {"type": "string"},
                    "from_email": {"type": "string"},
                    "from_name": {"type": "string"},
                },
                name="embeddings_text_search_index",
            ),
            VectorSearchIndex(
                fields=["embedding", "user_id", "gmail_message_id", "received_at", "from_email", "from_name"],
                similarities="cosine",
                name="embeddings_vector_search_index",
            ),
        ]

Migrations 0

  • 001: legacy relational models (no Mongo collections)
  • 0002: contains index/collection creation for Mongo (our expectation) Command used: python manage.py migrate mailbot_search_agent --database=mongodb

Problem Details

  • Despite the router routing these models to mongodb, applying 0002 on --database=mongodb does not create the declared collections or indexes.
  • Unique constraints and Django models.Index definitions appear not to materialize in Mongo.
  • Custom indexes declared via django_mongodb_backend.indexes.SearchIndex and VectorSearchIndex also do not materialize.

Questions for the MongoDB backend team

  • Are models.Index, UniqueConstraint, and the backend’s SearchIndex/VectorSearchIndex expected to be emitted into actual MongoDB indices via Django migrations with ENGINE = "django_mongodb_backend"?
  • If yes, what migration operations are supported and how should we structure migrations to ensure collections and indexes are created on migrate --database=mongodb?
  • If no, what is the recommended approach for provisioning:
  • Collections (with validators if supported)
  • Unique indexes
  • Atlas Search and Vector Search indexes
  • via Django migration operations (e.g., custom operations, RunPython) using this backend?
  • Is there a known limitation where collections are only created upon first write, and indexes must be created programmatically instead of via migrations?
Вернуться на верх