I’ve just merged DM-51176, which gives the general-purpose quantum graph builder used by pipetask nearly complete support for fixed HEALPix and HTM dimensions.
PipelineTask authors are now free to use dimensions like healpix8 or htm9 in their tasks and connections, and everything should Just Work.
At present, this means the current HiPS tasks (HighResolutionHipsTask, GenerateHipsTask, and GenerateColorHipsTask) can now be run in a single QG, or even the same QG as the pipeline that builds the coadds they start from. However, they still require their output directory/bucket to be provided via configuration, so I have not added them to pipelines in drp_pipe, since we don’t want those to require any configuration overrides to be usable. And @natelust is nearing complete on new HiPS-generation tasks that will leverage the new QG builder functionality to make that process significantly more parallel.
Note that support for healpixN and arbitrary htmN dimensions was added to the QG builder, not the butler query system, which has two important implications:
- these dimensions are still not fully supported in butler queries; you can query for existing datasets with those dimensions, but they still can’t participate in spatial joins to other dimensions like
patchorvisit; - when building QGs, constraining these dimensions in the
--data-querydoes not work, and you can’t use a dataset with these dimensions as a--dataset-query-constraint(even an implicit one) if there is another spatial dimension in the pipeline connections.
You can work around both of these limitations by generating a table of overlaps between the other relevant butler dimension(s) (e.g. patch) and the healpixN or htmN dimension for the area of interest and passing it via the --data-id-table pipetask argument or the join_data_coordinate_table method in the butler query system.