How should I introspect config values?

Data Management
#1

Running LSST tasks from a notebook, I wanted to set the thresholdType:

config = SourceDetectionTask.ConfigClass()
config.thresholdType = "???"

Rather than looking at the source code to find the valid options, the best I could do was look at

SourceDetectionTask.ConfigClass()._fields["thresholdType"].allowed

(note the _). Is there a better way? I’d like to be able to ask, `what are my options’, and get all the config entries without knowing what sort of config field I was looking at – ideally, I’d like to use jupyter’s mouse-over help to get it…

#2

There is a better way:

>>> print SourceDetectionTask.ConfigClass.thresholdType.__doc__
specifies the desired flavor of Threshold
Allowed values:
    pixel_stdev	threshold applied to per-pixel std deviation
    variance	threshold applied to image variance
    value	threshold applied to image value
    stdev	threshold applied to image std deviation

Obviously, that’s not at all easy to guess, and now that I think about it, it could be really hard to get this displayed more automatically. There are two problems.

The first is that config fields are descriptors, and I think that makes getting the right help from a config instance impossible using the idiomatic Python approach (it’s a Python language limitation): once we’ve said config.thresholdType, all we have is the string value, and there’s no way for Python’s help() function to do anything useful with that. Maybe we could have a string subclass that holds the documentation? But even if we did, we’d run into the next problem…

Getting the right help from a config class is broken by a choice made by the Python help() function. Doing SourceDetectionTask.ConfigClass.thresholdType returns an instance of pex.config.ChoiceField that has the __doc__ attribute that we want - but that __doc__ is an instance attribute, not a class attribute. A class __doc__ attribute also exists (this is the documentation for pex.config.ChoiceField itself), and that’s what help() displays (and by extension, IPython’s ? operator). We’re not the only ones who have this problem: compare print numpy.sin.__doc__ with help(numpy.sin). There might be some hook to customize what help() does, but if there is, I’m surprised NumPy hasn’t utilized it yet.

1 Like
#3

Numpy has their own info function to work around this issue:

For some objects, np.info(obj) may provide additional help. This is particularly true if you see the line “Help on ufunc object:” at the top of the help() page. Ufuncs are implemented in C, not Python, for speed. The native Python help() does not know how to view their help, but our np.info() function does.

from https://github.com/numpy/numpy/blob/master/numpy/init.py#L35 Indeed, Numpy has an entire module with docstrings for their C-defined unfuncs.

#4

In fact,

>>> np.info(lsst.meas.algorithms.SourceDetectionTask.ConfigClass.thresholdType)
specifies the desired flavor of Threshold
Allowed values:
        pixel_stdev     threshold applied to per-pixel std deviation
        variance        threshold applied to image variance
        value   threshold applied to image value
        stdev   threshold applied to image std deviation
1 Like
#5

In addition, Jim’s message above could be misinterpreted to say that the Config class’s documentation is not easily accessible via help(). This is not the case. Both

help(lsst.meas.algorithms.SourceDetectionTask.ConfigClass)

and

cfg = lsst.meas.algorithms.SourceDetectionTask.ConfigClass() help(cfg)

appear to give quite useful results for “Data descriptors defined here”, although as he says looking at an individual Field's documentation requires a different method.

#6

This np.info() seems like a hack to me, working around a python problem. Is it something that we ought to embrace (i.e. should I update the notebook?)