Permissions problem when installing phalanx

Support Rubin Science Platform
, ,
#1

Hello,

I am having difficulties with permissions when attempting a deployment of the RSP (ukidacdev). I am trying to figure out which part of my (I suspect) Vault configuration is wrong.

The following two commands look fine:
phalanx secrets sync --secrets secrets.yaml ${ENVIRONMENT}:

Created Vault secret for argocd
Created Vault secret for cert-manager
Created Vault secret for gafaelfawr
Created Vault secret for mobu
Created Vault secret for nublado
Created Vault secret for portal
Created Vault secret for postgres
Created Vault secret for squareone
Created Vault secret for ssotap
Created Vault secret for tap
Created Vault secret for pull-secret

phalanx vault audit ${ENVIRONMENT}:

<no output>

However, when I do phalanx environment install $ENVIRONMENT, then hit ‘y’, I get:

Traceback (most recent call last):
  File "/home/ubuntu/.local/share/mamba/bin/phalanx", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/phalanx/cli.py", line 112, in wrapper
    f(*args, **kwargs)
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/phalanx/cli.py", line 545, in environment_install
    environment_service.install(environment, vault_credentials, git_branch)
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/phalanx/services/environment.py", line 95, in install
    argocd_password = self._get_argocd_password(vault)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/phalanx/services/environment.py", line 180, in _get_argocd_password
    argocd_secret = vault.get_application_secret("argocd")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/phalanx/storage/vault.py", line 207, in get_application_secret
    r = self._vault.secrets.kv.read_secret(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/hvac/api/secrets_engines/kv_v2.py", line 98, in read_secret
    return self.read_secret_version(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/hvac/api/secrets_engines/kv_v2.py", line 153, in read_secret_version
    return self._adapter.get(
           ^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/hvac/adapters.py", line 146, in get
    return self.request("get", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/hvac/adapters.py", line 408, in request
    response = super().request(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/hvac/adapters.py", line 376, in request
    self._raise_for_error(method, url, response)
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/hvac/adapters.py", line 294, in _raise_for_error
    utils.raise_for_error(
  File "/home/ubuntu/.local/share/mamba/lib/python3.12/site-packages/hvac/utils.py", line 41, in raise_for_error
    raise exceptions.VaultError.from_status(
hvac.exceptions.Forbidden: 1 error occurred:
        * permission denied

, on get # URL removed, contains /v1/secret/data/<vault environment path>/argocd

vault kv get on the path works, so maybe it’s a policy issue? I am assuming if my VAULT_TOKEN, VAULT_SECRET_ID or VAULT_ROLE_ID were wrong, then the audit or secrets sync would tell me.

Any ideas greatly appreciated.

Dave

#2

Secrets sync is unrelated to the application role used for installation. (Secrets sync uses a Vault token with read/write access, and installation uses a Vault app role with read-only access.) Try running:

phalanx vault audit

to check that the permissions on the role are correct. You can also try recreating the Vault read app role used for installation (with phalanx vault create-read-approle in case you cut and pasted the app role information incorrectly.

1 Like