Pymemcache OSError: [Errno 99] Cannot assign requested address

Context:

We have a django application running inside a container on our cloud instance. We recently started seeing errors when we try to access value from django cache in an api end point.

cache.get('key')

This api endpoint is very frequently accessed by our users.

The full error that we are seeing is attached below.

Error trace

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/www/iridize/tipcms/views.py", line 2141, in cross_app_new
    cache_value = cache.get(cache_key, {})
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/backends/memcached.py", line 75, in get
    return self._cache.get(key, default)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347, in get
    return self._run_cmd("get", key, default, default=default, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322, in _run_cmd
    return self._safely_run_func(client, func, default_val, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymemcache/client/hash.py", line 211, in _safely_run_func
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymemcache/client/base.py", line 1494, in get
    return client.get(key, default)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymemcache/client/base.py", line 687, in get
    return self._fetch_cmd(b"get", [key], False, key_prefix=self.key_prefix).get(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133, in _fetch_cmd
    self._connect()
  File "/usr/local/lib/python3.11/site-packages/pymemcache/client/base.py", line 424, in _connect
    sock.connect(sockaddr)
OSError: [Errno 99] Cannot assign requested address

We are using memcached for caching purposes and our cache config in django looks like this.

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': os.environ.get('MEMCACHE_SERVICE', '127.0.0.1:11211'),
        'OPTIONS': {
            "use_pooling": True
        }
    }
}

My colleague suspected that we may not be having open ports on our pod and he did some investigation

It can be tested easily by counting the busy ports in the pod with:

cat /proc/net/tcp|wc -l In pods we observed 25k busy ports or even more, when the max port number is around 30k.

Also a more specific command searching for the memcache port number (11211 = hex 2BCB) can be done with this, Also added a 03: filter, that is counting the non-released ports only (these are not busy anymore, but the WAIT_TIME is still in progress, so they are not released yet)

cat /proc/net/tcp|grep 2BCB|grep " 03:"|wc -l

Some context about above block: If we try to hit our server with several hundreds of requests the number of ports linked with memcached keeps on increasing.

Question:

Why does pymemcached open new connection for every cache.get request, since our server is hit with huge number of requests, we are running out of ports as the connections are opened very frequently. We thought pymemcached opens some connections and reuse them instead of opening new connections every time. We tried to control this with max_pool_size, pool_idle_timeout parameters where the timeout parameter which has value 0 should not discard any connections and we assumed that some reuse of connections will happen but that is not happening doc link: https://pymemcache.readthedocs.io/en/latest/apidoc/pymemcache.client.hash.html but still not able to prevent many connections opened by the pymemcached.

Software versions: Django==4.2.15 pymemcache==4.0.0

Вернуться на верх