We are getting intermittent internal server errors on the UK RSP that seem to be linked to the choice of Nublado image.
This may be a red herring, but there appears to be an issue with Docker credentials (traceback below from the nublado-controller logs). Is it possible some images have different permissions than others? Or, is it possible the credentials aren’t loading sometimes?
Traceback (most recent call last):
File "/opt/venv/lib/python3.13/site-packages/controller/storage/docker.py", line 147, in list_tags
headers = await self._authenticate(config.registry, r)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.13/site-packages/controller/storage/docker.py", line 243, in _authenticate
raise DockerRegistryError(msg)
controller.exceptions.DockerRegistryError: Authentication credentials for registry.hub.docker.com rejected
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/venv/lib/python3.13/site-packages/controller/background.py", line 170, in _loop
await call()
File "/opt/venv/lib/python3.13/site-packages/controller/services/image.py", line 319, in refresh
to_prepull = await self._source.update_images(self._config, cached)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.13/site-packages/controller/services/source/docker.py", line 292, in update_images
tags = await self._docker.list_tags(self._config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.13/site-packages/controller/storage/docker.py", line 156, in list_tags
raise DockerRegistryError(msg, method="GET", url=url) from e
controller.exceptions.DockerRegistryError: Cannot parse response from Docker registry: DockerRegistryError: Authentication credentials for registry.hub.docker.com rejected
Thanks,
Dave (UK IDAC Team)