As mentioned in previous posts, we’re writing our own obs_ package for a wide-field survey telescope. It’s working really nicely through to difference imaging (we’ve not experimented much with forced photometry), and we’re getting some sweet difference images out
However, we find that the imageDifference step takes quite a long time compared to other steps. We assume that part of this is because, as a pipe_task, imageDifference is a serial process, rather than a parallelised batch process. Although do correct me if I’m wrong on this.
Are there any plans to release a driver capable of parallel batch-processing for imageDifference in the near future? If not, does anyone have an idea of how challenging it would be to write our own, using the other pipe_drivers as a guide?
I’m so glad to hear that the pipeline is working so well for you! I’m sure you must be anxious to move forward, but when you have a chance would you mind posting details about your experience (the good, the bad and the ugly) that we could use to improve our docs so that everyone can benefit from your hard work?
It should be easy to wrap image differencing in a parallelised batch process: BatchParallelTask was made for this very purpose (its docstring says “Use this subclass of BatchCmdLineTask if you don’t need to use the Pool directly, but just want to iterate over many objects (like a multi-node version of the ‘-j’ command-line argument)”). Have a look at SingleFrameDriverTask, which uses this approach. Let me know if you run into any problems, and I’ll be happy to help.
Here’s a gist of an imageDifferenceDriver.py I wrote a while ago. As Paul said, there isn’t much to it and it should just work. I don’t think anything should have changed to make it outdated. Let me know if that does the trick for you.
Thanks for your quick responses, and apologies for my slow response due to travel.
I’m waiting to hear back from my student, Lydia, as to whether that code works with our version of the stack (v14), Colin, we’ll keep you posted. Even if it doesn’t, your code’s a huge help, so thank you very much!
Lydia is now at a good point to collate our notes and put a post together outlining our experiences up to difference imaging. We should have something ready to post within the next couple of weeks.
