Ap_pipe does not obey `--reuse-outputs-from differencer`

Support
#1

Hi, I am trying to use ap_pipe task from the v21_0_0 pipeline on products already processed and differenced with hscPipe 8.4. I tried to use --reuse-outputs-from differencer option to skip the Ccdprocessor and differencer. However, I see that it still runs the CcdProcessor, and the differencer.

I have narrowed down the issue to the following condition in ap_pipe which has a value False:

calexpRef.datasetExists("calexp", write=True)

If instead, I run calexpRef.datasetExists("calexp"), I have checked that the condition is True.

Does ap_pipe require the calexpRef to be writable and can someone please explain if there is a reason for that?

#2

Just to be sure I’ve understood correctly, are you running ap_pipe.py on a single repository that has raws, calexps, and differenced images all together?

IIRC, the write=True was mostly an optimization, on the assumption that calexps would always be outputs from ap_pipe. I don’t think it occurred to us that it might be used in combination with a different pipeline.

#3

@kfindeisen yes, I have a repository which already has raws calexps and differenced images from HSCpipe 8.4 and I wanted to use the ap_pipe to populate the database with associations.

#4

Unfortunately, I don’t have a good workaround, since association itself was never designed to be run from the command line. I’ve filed DM-29121 to remove the write=True keyword.

#5

Thank you for your patience. I have a fix on ap_pipe/tickets/DM-29121 that seems to work for a mock repository; can you confirm that downloading it fixes your problem? (It may not be compatible with a 21.0 stack, but should work with the w_2021_11 weekly.)

#6

Thanks a lot @kfindeisen, I realized that hscPipe is not writing out WARPEDEXP files by default. So I had to anyway rerun the difference imaging with the LSST pipeline. I will test if this fix is able to help with the difference images produced by the LSST pipeline itself.

Apologies for a noob question: Can you please tell me how do I set this particular version up? Do I have to install the whole pipeline w_2021_11 and then run a setup inside a git cloned directory?

#7

That is how I would do it, yes.

If you’re managing the LSST pipeline with lsstsw, it has an option to build from branches directly instead. However, lsstsw is very difficult to set up in the first place, so I don’t recommend that approach if you’re already using a different method like newinstall.sh.

#8

I’ve gone ahead and merged the changes; they should appear in the next weekly (w_2021_13, I think). If you’re still having problems with that version, please let me know.