I am running into a very strange issue around processCcd.py. I am trying to launch a series of batch jobs to the canfar cloud (uses condor) where each job would process a single HSC frame.
During setup, I created a VM that is interacted with through SSH. On that machine, the lsst_distrib was installed. A script was also created, that copies onto the VM an image, ingests that image into the butler, runs processCcd, and copies the image back. When logged into the VM directly, that script is fully successful, and the resultant processed CCD are beautiful. Everything looks great.
So that machine is dupicated as a VM for use in batch mode.
When exactly the same job is launched through condor, things trip up. Oddly, the terminal output of processCcd.py is exactly the same as it is when run interactively, and the data extension is processed correctly. But the second extension that contains the pixel flags for sources, background, bad pixels etc., is blank, containing only zeros.
The batch job raises no flags, logs no errors, and the processCcd script finishes as normal.
So there is something strange about the process that fills the second extension that doesn’t work correctly, but doesn’t take the script down or through any errors (or even any warnings).
Anyone have any ideas what to look for?
A few details:
-ubuntu 18.04 OS
-running on intel broadwell hardware
-the source and setup calls for the lsst distribution are all good
-PATH, PYTHONPATH, and LD_LIBRARY_PATH variables are identical between the interactive VM and that run in batch mode.