PhotoCalib has replaced Calib: welcoming our nanojansky overlords

Data Management
, ,
#1

With the merger of DM-17029 and DM-10156, the LSST Science Pipelines now provides calibrated fluxes in units of nanojansky (nJy), internally uses reference catalogs in nJy, and uses afw.image.PhotoCalib to manage the photometric calibration of images, replacing afw.image.Calib. This post summarizes the changes to reference catalogs as well as the API changes involved, including where we did and did not provide backwards compatibility, and provides some suggestions how how to transition code to use the new objects.

Reference Catalogs

New refcats will now be in nJy

IngestIndexedReferenceTask now generates reference catalogs with flux units of nJy; similarly the schema returned from LoadReferenceObjectsTask.makeMinimalSchema has nJy flux units. Indexed reference catalogs now have format_version=1, and unversioned catalogs (with jansky flux units) are assumed to be format_version=0. In addition, new reference catalogs will have their units explicitly specified as units=nJy, so that you can check the units on the catalog fluxes directly.

Custom code that previously assumed the fluxes in reference catalogs were Jy will now return values that are off by 1e9. Everything tested by lsst_distrib, lsst_dm_stack_demo, and ci_hsc has been converted to treat fluxes correctly.

As we are phasing out Astrometry.net reference catalogs, we do not provide a way to generate new a.net-style catalogs with nJy fluxes.

Converting old reference catalogs

We provide backwards compatibility with old reference catalogs (both HTM Indexed and Astrometry.net) by converting them in memory when they are loaded from disk. If you use version 0 reference catalogs, you will see warning messages in the logs about this conversion, including a description of how to convert your catalogs on disk to have nJy fluxes. You can convert your on-disk HTM Indexed catalogs via the command-line tool meas_algorithms/bin/convert_refcat_to_nJy.py: see its help description (via running with -h) for more details.

In the next few weeks, we will provide converted versions of the existing gaia, ps1, and sdss catalogs on lsst-dev. Watch for an announcement.

PhotoCalib replaces Calib

Overview of PhotoCalib

The PhotoCalib object can contain a position-aware afw.math.BoundedField, which allows for spatially-varying photometric calibrations (as produced by Jointcal), in addition to non-spatially-varying “mean” calibrations (as produced by PhotoCalTask). PhotoCalib is mathematically defined as:

instFlux [counts] * calibration(x,y) = flux [nJy]

PhotoCalib provides methods for:

  • converting instrumental fluxes and errors (instFlux, instFluxErr) to AB magnitude (instFluxToMagnitude) and flux (instFluxToNanojansky). These methods take inputs as either scalars or SourceRecords and SourceCatalogs with a flux field name. If you pass both instFlux and instFluxErr to the conversion functions, you will get back a Measurement object containing the value and error (think of it as a namedtuple).
  • efficiently converting an entire image to calibrated fluxes (calibrateImage).
  • converting an AB magnitude to instrumental flux (magnitudeToInstFlux).
  • other methods for extracting various components of the calibration (getCalibrationMean, getCalibrationErr, getInstFluxAtZeroMagnitude, computeScaledCalibration).

In general it is strongly preferred to use the provided conversion functions rather than extracting numerical parameters and doing conversions by hand.

PhotoCalib is a C++ object, so you can find the detailed documentation on the LSST doxygen page.

Deprecated interfaces and behavior changes

To ease the transition from Calib to PhotoCalib, I’ve added deprecated interfaces that behave like the old Calib methods, including:

  • getMagnitude to convert instFlux to AB magnitude, and getFlux AB magnitude to instFlux. Instead use instFluxToMagnitude and magnitudeToInstFlux for these conversions. These new methods can take a wider range of arguments, and have clearer names.
  • getFluxMag0 now always returns NaN for the second element (previously fluxMag0Err). Instead use getInstFluxAtZeroMagnitude and/or getCalibrationMean/getCalibrationErr, depending on your exact needs.
  • PhotoCalib is immutable, which means that the setFluxMag0 method that was previously used to create a new Calib object is no longer valid: calling it will raise a RuntimeError with a description of how to create a PhotoCalib.
  • PhotoCalib will never raise an exception when computing magnitudes from instrumental fluxes: zero or negative instFlux values transform to NaN magnitudes. I have added deprecated setThrowOnNegativeFlux and getThrowOnNegativeFlux methods to PhotoCalib as no-ops. In addition, because PhotoCalib does not special-case negative instrumental fluxes, calling e.g. photoCalib.instFluxToMagnitude(instFlux=-5.0, instFluxErr=1.0) will result in NaN for magnitude, but a real, positive value for magnitude error, instead of returning NaN for both.
  • getCalib on Exposure and ExposureInfo is deprecated in favour of getPhotoCalib; both will return the PhotoCalib associated with that exposure.
  • Use butler.get("calexp_photoCalib", dataId) to get the PhotoCalib of an exposure if you do not need to load the entire exposure from disk.

Due to the timing of merging DM-10156 and adding the deprecated package to our conda environment, the deprecated interfaces described above do not yet emit warnings at the python level (they should emit C++ compiler warnings). These warnings should start appearing next week, once @ktl and I sort out exactly how to implement deprecation warnings for pybind11 objects.

Using astropy for unit conversion

To convert between nJy and AB magnitudes (or other flux units), we recommend using astropy.units. For example, you can see that 575 nJy is approximately the 5-sigma 30s imaging depth of LSST, and the AB magnitude zero is nearly, but not exactly, 3631 Jy as it is often hard coded to be:

>>> import astropy.units as u
>>> (575*u.nJy).to(u.ABmag)
<Magnitude 24.50083039 mag(AB)>
>>> (0*u.ABmag).to_value(u.nJy)
3630780547701.003

The existing afw.image functions abMagFromFlux and fluxFromABMag will be deprecated in favour of the above at some point in the future. Be aware that they assume fluxes in janskys, not nanojanskys, and use the imprecise 3631 Jy reference flux value. The error conversion methods abMagErrFromFluxErr and fluxErrFromABMagErr do not yet have obvious replacements from astropy. See DM-16903 for details.

5 Likes
Units of calexps?
Data Management Science Pipelines Design Document
Question about FLUXMAG0 header value in older versions
#2

Note: if you experience failures to read data with the following error:

TypeError: 'Cannot read Exposure FITS version >= 0'

it is because you are trying to use an old (pre-PhotoCalib, Exposure version=0) stack to read data generated by a new (post-PhotoCalib, Exposure version=1) stack.

#3

The converted refcats are available on lsst machines at /datasets/refcats/htm/v1/ Please update your symlink if you use one.

The symlink at /datasets/hsc/repo/ref_cats has been updated to point there.

2 Likes
Error in single frame processing of HSC data
#4

Hi,
My colleague @rfahed and I have followed these steps to convert the fluxes of our original refcats (in Jy) to nJy. For our LSST sims, the mag zeropoint values are around ~28.3 in g-band.
However, the magnitude zeropoint we get after processing is

>>> mag_zp=calexp.getPhotoCalib().instFluxToMagnitude(1)
>>> mag_zp
13.21043669448315

Do you have any idea what’s going wrong with the conversion to AB mag?

#5

@pollack : please start a new thread for debugging your calibration question.