Django filter on a large list- giving incorrect count or syntax error on ms sql- filtering in chunks is very slow

I am trying to filter records from a model using a list of values: Sample:

items= ['a@xyz.com','b@xyz.com','c@abc.com'.......]

data= TestModel.objects.filter(email__in=items)

items contains more than 2000 entries and the database being used in SQL. To avoid count incorrect error: I tried:

data= TestModel.objects.all()
chunks = [items[x:x+1000] for x in range(0, len(items), 1000)]
for chunk in chunks:                    
    chunk_data=Testmodel.objects.filter(email__in=set(chunk))
    data= data | chunk_data
return data

I am not sure if this is the right approach and also the results are very slow. Any suggestions if there is a proper work around to filter data using a large list in django?

Back to Top