How to use cache system for large and dynamic data

I want to use a cache system for my django web application, I have been using django-cachepos for caching my data but now I have some issues like having big data for reports which includes a lot of filters and should be online when data upate .

Now my response time is too long (like 12 13 s ) and I want some guides from you to implant a cache sytem or a cache strategy that can helps to improve my code

cosider this that appliacation is going to be a large scale project and right now have 3 4 lead from all interner and cant stop being online

Ps. I have this problem on some other endpoints with requests with lots of data

I try to use redis functions to set and get data but I have no idea how to use it with filters and even cache response data

It looks like django-cacheops uses Redis under the hood, which is just a simple key/value store. This means for each query it will be creating a key with the encoded filters and the results as the value (read code under "def _cache_key(...)" here). There is additional logic for determining all keys that need to be invalidated on changes (read invalidation.py here). You won't have control over this behavior (for example updating the cached results instead of evicting them), so considering other strategies will likely mean ditching django-cacheops.

You say you are still getting long wait times (12-13s requests), you will need to do more digging to understand why. django-cacheops has statistics for analyzing this. You should be checking what % of cache hits vs misses. I am guessing you are getting more misses than hits with the nature of the dynamic data (keys being invalidated before they produce hits) as well as the number of filters (if subsequent queries are unique in filter combinations, you will never get a cache hit and always query the DB instead of the cache). A restricted cache size will amplify this behavior (less space means more evictions to make room for new queries).

After this analysis there are three things you should consider:

  1. keep django-cacheops and experiment with applying some or most of the filters in code (filter down a more complete/common set of results after they are returned from the cache). A good example is if you always query for the last 30 days of data, only send this filter to django-cacheops and apply other filters in code, this would look something like:
qs = Article.objects.filter(created_at__gte=thirty_days_ago).cache()
results = [row for row in qs if apply_filters(row, other_filters)]
  1. ditch django-cacheops. Consider using more optimized Redis datatypes for your retrieval needs. You can learn how to set/get by using the python redis client or django cache configured for your project.
  2. ditch key/value caching altogether and consider another secondary storage optimized for faster retrieval of dynamic data. Note this depends on the data (structured or unstructured, time series, how "big data" are you talking?). I suspect elasticsearch/opensearch will be a good option worth considering, but you did not provide enough context.
Back to Top