Generating custom catalogs from a small set of visit images and their templates

Support
#1

Dears,

First, I want to thank everyone who participated in the work and writing of the DP1 paper. I found it extremely well written and informative.

We are working on a pipeline for searching optical counterparts to high-energy transients. For this purpose, we have prepared a set of visit images injected with fake sources, initially using data from DP02 and now from DP1. We are using these images to test custom detection pipelines that will be fine-tuned for the sources of interest in our research.

Our approach so far has been to take bits and pieces from the LSST pipeline (e.g., routines from the diffim package) and use them to produce custom catalogues, tables, and data structures to be later searched for interesting transients. This has some unfortunate consequences:

  1. We are writing a lot of code that is probably reinventing the wheel and is neither as efficient nor as battle-tested as what is already in place.
  2. Our data products are not conformant those produced by LSST, which means we need to develop separate analysis pipelines and glue code to handle both our custom data products and the official LSST data products

Now my question: is it feasible to run the whole DIA pipeline on a (small-sized) set of user-provided visit and template images, using custom configurations, and receive as output fresh (and small-ish) versions of, at least, the DIASources, DiaObjects, and ForcedSourceOnDiaObject catalogs?

If this is achievable, does anyone have a pointer on where to start looking for a hook?
I have checked the docs, the official DP02/DP1 tutorials and the community forum search but could not find anything answering this question.

Thank you so much,
Peppe

#2

Yes, although you need to start using butler. There are other people investigating using their own data in the Rubin pipelines. See for example the discussion here:

It’s a bit tricky to set up a butler where you haven’t started with raw exposures and want to go straight to processed visits, but it is possible, albeit we don’t really have any tooling for that to calculate the detector regions for you from the FITS WCS, for example.

If you look at the software paper (PSTN-019) you will see a discussion of a lot of the issues, including fake source injection.

3 Likes
#3

Thank you Tim. I will check out on creating custom datasets and collections with the butler, and the problem you mention with WCSs. Will probably come back for a few questions more.

Ciao,
Peppe

#4

The process to do all this starting from raws does have some documentation. The problem with starting from a visit is you have to define the visit dimension records manually to allow your files to be ingested. You also need to worry a little bit about how compatible your FITS files are with what the downstream pipelines expect in terms of calibration extensions that you likely don’t have in your files.

1 Like