Optimizing Django ORM for Hybrid Queries (PostgreSQL + Vector Similarity Search)

I'm implementing a RAG (Retrieval-Augmented Generation) system that requires combining traditional Django ORM filtering with vector similarity searches. The specific workflow needs to:

First filter products by standard relational fields (e.g., category="books")

Then perform vector similarity search on just that filtered subset of product descriptions

Current Implementation and Challenges:

# Current inefficient approach (two separate operations)
books = Product.objects.filter(category="books")  # Initial DB query
vectors = get_embeddings([b.description for b in books])  # Expensive embedding generation
results = faiss_search(vectors, query_embedding)  # Vector search

Key problems with this approach:

Requires loading all filtered records into memory
Makes two separate passes over the data
Doesn't leverage PostgreSQL's native capabilities when using pgvector

What I've Tried:

Raw SQL with pgvector:

query = """
SELECT id FROM products 
WHERE category = 'books'
ORDER BY description_embedding <=> %s
LIMIT 10
"""
results = Product.objects.raw(query, [query_embedding])

Problem: Loses Django ORM benefits like chaining, model methods.

django-pgvector extension:

from pgvector.django import L2Distance

books = Product.objects.annotate(
    distance=L2Distance('description_embedding', query_embedding)
).filter(category="books").order_by('distance')[:10]

Problem: Doesn't scale well with complex filter conditions

Expected Solution:

Looking for a way to: