Memory quota exceeded when running Django Celery On Heroku
I have a Django project for running periodic tasks using Celery that is deployed to Heroku from GitHub. The project is a web scraping project that scrapes through other websites and stores the result in backend. The website is deployed on heroku well and the other parts are working fine. The issue is that whenever I start the periodic tasks, I get the error
Process running mem=760M(148.5%) Error R14 (Memory quota exceeded) in the logs. Also no results is saved in the database. The website is working fine in my localhost without any errors.
Here is my Procfile configuration;
web: gunicorn jobWebsite.wsgi --log-file - --log-level debug worker: celery -A jobWebsite worker -l info -B
In the second line, I combined the worker and beat processes to use one dyno. Also, here is a part of the logs that I am getting;
2023-01-24T21:58:41.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-flat-79602 sample#active-connections=6 sample#load-avg-1m=0.42 sample#load-avg-5m=0.405 sample#load-avg-15m=0.395 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16084924kB sample#memory-free=12314188kB sample#memory-cached=2049948kB sample#memory-redis=443248bytes sample#hit-rate=0.29167 sample#evicted-keys=0 2023-01-24T21:59:11.683479+00:00 heroku[worker.1]: Process running mem=760M(148.5%) 2023-01-24T21:59:11.685197+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded) 2023-01-24T21:59:32.045205+00:00 heroku[worker.1]: Process running mem=760M(148.5%) 2023-01-24T21:59:32.046763+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
I have used the Heroku postgres database and also the Heroku Redis database in this project. Any assistance will be highly appreciated.