Error when creating butler

Support Data Preview 0
#1

Hi,

Something seems to have gone wrong and I’m either no longer able to create a butler or to connect to the dp0.2 data? The error below occurs in any code I try to run, including tutorial 4a.

When it try to run the code in a notebook on RSP:

config = ‘dp02’
collections = ‘2.2i/runs/DP0.2’
butler = dafButler.Butler(config, collections=collections)

I get the message:

lsst.daf.butler._butler ERROR: Failed to instantiate Butler from config s3://butler-us-central1-dp02-user/butler.yaml.

This is followed by a long jupyter traceback error message ending with:

OperationalError: (psycopg2.OperationalError) connection to server at “10.163.0.3”, port 5432 failed: fe_sendauth: no password supplied

(Background on this error at: Error Messages — SQLAlchemy 2.0 Documentation)

Any ideas what might be causing this? It was working fine about a week ago. I then started trying to run tutorial 9 with a previous weekly build, but not sure this error is related. I restarted my server several times with the recommended build.

#2

Dear Brian,

Hmmm… I am not sure. I was able to run those commands –

config = ‘dp02’
collections = ‘2.2i/runs/DP0.2’
butler = dafButler.Butler(config, collections=collections)

– on data.lsst.cloud using the weekly_2023_21 build.

It does sounds like some permissions error, though.
Have you completely logged out and re-logged in and tried it again?

Thanks!

Best regards,
Douglas

–

#3

Hi,

Yes, I’ve tried logging out and back in and still get the same issue. Any other ideas?

Thanks,

Brian

#4

That is very strange.

The RSP notebook troubleshooting guide ( link ) suggests possibly removing or renaming the .local file in your home directory on the RSP. Then you might try logging out and back in to the RSP again. If that doesn’t work, I’ll need to consult with others. I hope it works! Let me know! Thanks!

#5

Hi,

Thanks for the suggestion. I tried renaming the .local directory then logging back in but got the same result, though it did create a new .local.

But this might be on the right track. I was messing around with installing RAPID (unsuccessfully) and definitely might have broken some dependency somewhere.

Thanks!

#6

Please check that ls -l $PGPASSFILE in a Terminal or !ls -l $PGPASSFILE in a notebook cell gives a result like this:

-rw------- 1 ktl ktl 228 Aug  9 13:05 /home/ktl/.lsst/postgres-credentials.txt
#7

Hi,

Yes, that gives the right response.

#8

Do you happen to have a file named ~/.lsst/db-auth.yaml? If so, you might try moving it aside or removing it altogether.

#9

Hi,

No, all I have in the .lsst directory is aws-credentials.ini and postgres-credentials.txt

#10

Very strange. Can you do this in a Terminal?

$ env | grep '^PG'
PGPASSFILE=/home/ktl/.lsst/postgres-credentials.txt
$ keyring --disable
$ pgcli -h 10.163.0.3 -d idfdp02 -u postgres
Server: PostgreSQL 13.10
Version: 3.5.0
Home: http://pgcli.com
postgres@10:idfdp02>
#11

Can you paste what is in the bottom of your Jupyterlab window frame? The bit that looks like this:

Mem: 619.40 / 16384.00 MB

Weekly 2023_30 [d6e9aa29…] (sciplat-lab:w_2023_30) Large (4.0 CPU, 16Gi RAM) data.lsst.cloud

#12

Hi,

I get some interesting responses:

$ env | grep ‘^PG’
PGPASSFILE=/home/bmorsony/.lsst/postgres-credentials.txt
$ keyring --disable
Traceback (most recent call last):
File “/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/bin/keyring”, line 11, in
sys.exit(main())
File “/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/keyring/cli.py”, line 134, in main
return cli.run(argv)
File “/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/keyring/cli.py”, line 64, in run
core.disable()
File “/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/keyring/core.py”, line 48, in disable
raise RuntimeError(msg)
RuntimeError: Refusing to overwrite /home/bmorsony/.config/python_keyring/keyringrc.cfg
$ pgcli -h 10.163.0.3 -d idfdp02 -u postgres
Password for postgres:
connection failed: FATAL: password authentication failed for user “postgres”
connection to server at “10.163.0.3”, port 5432 failed: FATAL: password authentication failed for user “postgres”

The pgcli command ask for a password. This looks similar to the error I get when starting a butler, so maybe there’s a password missing somewhere for me?

#13

Hi,

I just have this at the bottom, it doesn’t seem to have the memory stuff:

LSST | Idle

Weekly 2023_21 [a7536e6d…] (sciplat-lab:w_2023_21) data.lsst.cloud

DP02_04a_Introduction_to_the_Butler.ipynb

Ln 1, Col 1

Mode: Command

#14

Two more things to check:

$ cat ~/.config/python_keyring/keyringrc.cfg
[backend]
default-keyring=keyring.backends.null.Keyring
$ md5sum ~/.lsst/postgres-credentials.txt
0dee2cd76230c4e0c3e9c78b61b61d58
#15

Hi,

I get:

$ cat ~/.config/python_keyring/keyringrc.cfg
[backend]
default-keyring=keyring.backends.null.Keyring
$ md5sum ~/.lsst/postgres-credentials.txt
60893e87165af3f2e802dfceae40cf98 /home/bmorsony/.lsst/postgres-credentials.txt

Thanks.

#16

OK, so somehow your postgres-credentials.txt has been modified. Try moving it aside (mv ~/.lsst/postgres-credentials.txt ~/.lsst/postgres-credentials-old.txt) and restarting your server.

#17

Hi,

I did that, but still not working. The new postgres-credentials.txt seems to have the same contents. Here’s what the file contains:

10.255.0.3:5432:idfdp01:postgres:XXX
104.155.150.220:5432:hsc:postgres:XXX
104.155.150.220:5432:dp02:postgres:XXX

Thanks.

#18

(Please don’t post credentials in public places without redaction.)

That’s definitely incomplete, and it’s also definitely not your fault. The RSP admins will have to look into why there’s no entry for 10.163.0.3:5432:idfdp02:postgres which is needed to access the dp02 Butler repo.

#19

We think this should be fixed now. Thanks for reporting the problem!

#20

Hi,

Yes, it seems to be working now. Thanks so much for the help!