Problems using version w_2021_48 and w_2021_33

Yuanyu · November 29, 2021, 8:08am

Hello, I am now learning the lsst pipeline, I have installed two verions: w_2021_48 and w_2021_33 cause the tutorial series were developed using the w_2021_33 version.
At the very begining of the tutorial,Getting started tutorial part 2: calibrating single frames with Pipeline Tasks, when I execute the command

butler query-data-ids SMALL_HSC --collections HSC/RC2/defaults --datasets 'raw'

If I source the version w_2021_48, There are something wrong, the result is:

 (lsst-scipipe-0.7.0) [yu@localhost rc2_subset]$ butler query-data-ids SMALL_HSC --collections HSC/RC2/defaults --datasets 'raw'
No results. Try --help for more information.

So I open a new shell and this time I setup the w_2021_33 version. This time the result seems to nomal ：

(lsst-scipipe-0.6.0) [yu@localhost rc2_subset]$ butler query-data-ids SMALL_HSC --collections HSC/RC2/defaults --datasets 'raw'
band instrument detector physical_filter exposure
---- ---------- -------- --------------- --------
   y        HSC       41           HSC-Y      322
   y        HSC       42           HSC-Y      322
   y        HSC       47           HSC-Y      322
   y        HSC       49           HSC-Y      322
   y        HSC       50           HSC-Y      322
   y        HSC       58           HSC-Y      322
   y        HSC       41           HSC-Y      346

I want to know what cause it, thank you!

Yuanyu · November 29, 2021, 9:36am

But when I step forward using the version w_2021_33,executing the command:

pipetask run -b $RC2_SUBSET_DIR/SMALL_HSC/butler.yaml \
             -p $RC2_SUBSET_DIR/pipelines/DRP.yaml#singleFrame \
             -i HSC/RC2/defaults \
             -o u/$USER/single_frame \
             --register-dataset-types

It can’t work, and the log is:

Traceback (most recent call last):
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/22.0.1-14-g3316c66+9f7b2bc49a/bin/pipetask", line 29, in <module>
    sys.exit(main())
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/22.0.1-14-g3316c66+9f7b2bc49a/python/lsst/ctrl/mpexec/cli/pipetask.py", line 45, in main
    return cli()
  File "/home/yu/w_2021_33/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/yu/w_2021_33/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/yu/w_2021_33/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/yu/w_2021_33/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/yu/w_2021_33/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/yu/w_2021_33/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/22.0.1-14-g3316c66+9f7b2bc49a/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 101, in run
    pipeline = _doBuild(ctx, **kwargs)
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/22.0.1-14-g3316c66+9f7b2bc49a/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 64, in _doBuild
    return script.build(**kwargs)
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/22.0.1-14-g3316c66+9f7b2bc49a/python/lsst/ctrl/mpexec/cli/script/build.py", line 83, in build
    pipeline = f.makePipeline(args)
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/22.0.1-14-g3316c66+9f7b2bc49a/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 482, in makePipeline
    pipeline = Pipeline.from_uri(args.pipeline)
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/22.0.1-20-g5ce7af2+ff3b2f8649/python/lsst/pipe/base/pipeline.py", line 244, in from_uri
    pipeline: Pipeline = cls.fromIR(pipelineIR.PipelineIR.from_uri(uri))
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/22.0.1-20-g5ce7af2+ff3b2f8649/python/lsst/pipe/base/pipelineIR.py", line 849, in from_uri
    return cls(loaded_yaml)
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/22.0.1-20-g5ce7af2+ff3b2f8649/python/lsst/pipe/base/pipelineIR.py", line 511, in __init__
    self._verify_labeled_subsets()
  File "/home/yu/w_2021_33/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/22.0.1-20-g5ce7af2+ff3b2f8649/python/lsst/pipe/base/pipelineIR.py", line 570, in _verify_labeled_subsets
    raise ValueError(f"Labels {labeled_subset.subset - self.tasks.keys()} were not found in the "
ValueError: Labels {'TE4', 'nsrcMeasVisit', 'TE3'} were not found in the declared pipeline

And I search the ValueError on the community, no one has report that problem before, what should I do to handle that?
And I try to execute the above comand in version w_2021_48, It raise error,too, here are part of the log:

lsst.skyCorr.maskObjects.detection INFO: Detected 2654 positive peaks in 817 foo                                                                                                             tprints to 2.5 sigma
py.warnings WARNING: /home/yu/lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux                                                                                                             64/meas_algorithms/22.0.1-31-g9833a5f6+831340e22d/python/lsst/meas/algorithms/de                                                                                                             tection.py:447: FutureWarning: Default position argument overload is deprecated                                                                                                              and will be removed in version 24.0.  Please explicitly specify a position.
  sigma = psf.computeShape().getDeterminantRadius()

lsst.skyCorr.maskObjects.detection INFO: Detected 2662 positive peaks in 805 foo                                                                                                             tprints to 2.5 sigma
lsst.skyCorr INFO: Background 0: 4858408 pixels
lsst.skyCorr INFO: Background 1: 4813900 pixels
lsst.skyCorr INFO: Background 2: 4916797 pixels
lsst.skyCorr INFO: Background 3: 4976760 pixels
lsst.skyCorr INFO: Background 4: 4807877 pixels
lsst.skyCorr INFO: Background 5: 4699401 pixels
lsst.skyCorr INFO: Sky frame scale: [11147.98713122]
lsst.skyCorr INFO: Background 0: 4888947 pixels
lsst.skyCorr INFO: Background 1: 4848228 pixels
lsst.skyCorr INFO: Background 2: 4955204 pixels
lsst.skyCorr INFO: Background 3: 5016086 pixels
lsst.skyCorr INFO: Background 4: 4857070 pixels
lsst.skyCorr INFO: Background 5: 4734631 pixels
py.warnings WARNING: /home/yu/lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux                                                                                                             64/pipe_drivers/22.0.0+26c05adf09/python/lsst/pipe/drivers/background.py:872: Ru                                                                                                             ntimeWarning: invalid value encountered in true_divide
  return convolved/denominator

lsst.ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'skyCorr' on quan                                                                                                             tum {instrument: 'HSC', visit: 30482, ...} took 42.167 seconds
lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 1320 quanta successfully, 0 fail                                                                                                             ed and 0 remain out of total 1320 quanta.

Yuanyu · November 29, 2021, 10:49am

I also have some questions on Building a package with the installed Science Pipelines stack . when I excute the command

git clone https://github.com/lsst/daf_butler

What version of the daf_butler it belongs to? What should I add if I want to download the daf_butler that belong to w_2021_48 or w_2021_33?
What’s the difference between I dierctly cp and git clone?

 cp lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/ ~/

or

git clone https://github.com/lsst/daf_butler

Can I diectly cp the directory and modify on it instead of git clone?
And also, When I

scons -Q -j 6 opt=3 tests

in the directory, error like:

======================= 6 warnings, 216 errors in 20.46s =======================
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /home/yu/butler/daf_butler/tests/.tests/pytest-daf_butler.xml.failed
The following tests failed:
/home/yu/butler/daf_butler/tests/.tests/pytest-daf_butler.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1

What’s wrong with that? Thank you!

jbosch · November 29, 2021, 3:26pm

For the butler query-data-ids problem, it is possible that the w_2021_33 results, and the w_2021_48 lack of results is correct, if something went wrong setting up the repository (it may even be a problem with the tutorial, which I’m not very familiar with, if that was developed with w_2021_33). There have been bugs fixed between those releases that could explain this behavior. One way to check that might be to run:

butler query-datasets SMALL_HSC raw --collections HSC/RC2/defaults

The bugs I am thinking of did not affect that similar call.

Your pipetask run with w_2021_48 looks fine to me; the warnings there can be ignored. The pipetask run with w_2021_33 problem looks like a mismatch between the pipeline at $RC2_SUBSET_DIR/pipelines/DRP.yaml#singleFrame and the release, but I’m not sure, and I don’t know enough about the tutorial to speculate on how possible such a mismatch is.

If you want to check out a source version of daf_butler consistent with a particular weekly release, do e.g.

git checkout w.2021.48

in other words, the git tag for a weekly release replaces the underscores (_) with periods (.).

timj · November 29, 2021, 3:49pm

You have to look in there to see the failure.

If you are running scons locally you have to make sure you use setup -k -r . or something similar so that EUPS knows you are wanting to run the tests on the local checkout.

Your problems though are mainly related to w33 being very old and the pipeline is not compatible with it. Updating daf_butler won’t help because the pipelines are defined elsewhere.

Yuanyu · November 30, 2021, 6:45am

Thank you!
The comannd work!

butler query-datasets SMALL_HSC raw --collections HSC/RC2/defaults

Here are part of the result:

(lsst-scipipe-0.7.0) [yu@localhost rc2_subset]$ butler query-datasets SMALL_HSC raw --collections HSC/RC2/defaults

type     run                      id                  band instrument detector physical_filter exposure
---- ----------- ------------------------------------ ---- ---------- -------- --------------- --------
 raw HSC/raw/all d0998707-a8f4-5825-a13a-ab04be5f3439    y        HSC       41           HSC-Y      322
 raw HSC/raw/all 5fd3efcf-81ad-57a5-b205-463c1521ee60    y        HSC       42           HSC-Y      322
 raw HSC/raw/all bb7f3ec8-481f-5933-a1c3-22fbf2a06544    y        HSC       47           HSC-Y      322
 raw HSC/raw/all 8f71b3e0-2ba6-50cf-b8c5-bfe821e1cdc1    y        HSC       49           HSC-Y      322

jbosch · November 30, 2021, 4:36pm

It was pointed out to me that the change in the query-data-ids results was a known issue, and I just didn’t recognize that it was what you had run into (despite being the person responsible for the change).

In short, the old behavior with w_2021_33 was incorrect, because you didn’t pass the dimensions of the data ID to the command - note that butler query-data-ids accepts positional arguments - and it seemed to work because it (incorrectly) automatically included the dimensions of the raw dataset (query-datasets is supposed to do this, but query-data-ids is more subtle). The behavior with w_2021_48 was perhaps a bit closer to correct, but the error message is very misleading, and that’s something we’ve opened a ticket to fix.

The right way to use query-data-ids to get similar results (the tutorial needs to be updated, too) is:

butler query-data-ids SMALL_HSC --collections=HSC/RC2/defaults --datasets='raw' exposure detector

Yuanyu · December 6, 2021, 2:20pm

I have run the whole LSST and HSC pipeline now, since my teacher gives me a mission to develop a pipeline for a new observatory(not the whole pipeline, only the very beginning include removing instrumental signatures with dark, bias and flat field and then do photometric and astrometric ). like HSC’s pipeline based on LSST’s.
To achieve these, where should I focus on? Where should I change? (For example, we use different CCDs, we have different FOV,and we may need to have some changes on the code to calibrate the raw image?) What should I do to realize it? And what documents should I read?
Thank you very much!

timj · December 6, 2021, 3:59pm

You need to create an “obs” package describing your instrument. This is a good place to start:

https://pipelines.lsst.io/v/weekly/modules/lsst.obs.base/creating-an-obs-package.html

Yuanyu · December 20, 2021, 6:48am

I have a question, it may be kinda stupid, I want to know how LSST’s pipeline do astrometry,and I want to read and modify these codes, and finally use them to do astrometry for other observatory’s images, and I follow the Command editting

(lsst-scipipe-0.7.0) [yu@localhost astrom]$ which singleFrameDriver.py          ~/lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_drivers/22.0.0+26c05adf09/bin/singleFrameDriver.py
(lsst-scipipe-0.7.0) [yu@localhost astrom]$ cat ~/lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_drivers/22.0.0+26c05adf09/bin/singleFrameDriver.py
#!/usr/bin/env python
from lsst.pipe.drivers.singleFrameDriver import SingleFrameDriverTask
SingleFrameDriverTask.parseAndSubmit()

And I opened the lsst.pipe.drivers.singleFrameDriver, find it inherits from BatchParrelTask,which inherits from BatchCmdLineTask, then to CmdLineTask, and finally to Task. I find no footprint that use the package in

(lsst-scipipe-0.7.0) [yu@localhost astrom]$ pwd
/home/yu/lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/meas_astrom/22.0.1-12-g44dc1dc+b59be3614c/python/lsst/meas/astrom
(lsst-scipipe-0.7.0) [yu@localhost astrom]$ ls
approximateWcs.py                    pessimisticPatternMatcherUtils.cc
astrometry.py                        pessimisticPatternMatcherUtils.so
denormalizeMatches.py                polynomialTransform.cc
directMatch.py                       polynomialTransform.so
display.py                           __pycache__
fitAffineWcs.py                      ref_match.py
fitSipDistortion.py                  scaledPolynomialTransformFitter.cc
fitTanSipWcs.py                      scaledPolynomialTransformFitter.so
__init__.py                          SConscript
makeMatchStatistics.cc               setMatchDistance.py
makeMatchStatistics.so               sip
matchOptimisticB                     sipTransform.cc
matchOptimisticBTask.py              sipTransform.so
matchPessimisticB.py                 verifyWcs.py
pessimistic_pattern_matcher_b_3D.py  version.py

Parts of these must be applied to the images in some sequences while doing the astrometry.
So, how can I know how LSST’s do astrometry? What packages did it call? And the order of them?
Thank you!

ktl · December 20, 2021, 7:03pm

(Answer posted in original topic.)

Yuanyu · January 13, 2022, 12:27pm

I am now reading the website you give me: How to create a new LSST obs package — LSST Science Pipelines.
But I didn’t know how to Creating a MetadataTranslator named ExampleTranslator which should be imported later on

from astro_metadata_translator import ExampleTranslator

by reading the MetadataTranslator and the links it gives.
So how can I realize it?
(I guess HSC may have experience this step and finally make a new package and it may named obs_HSC, it may be helpful to read the ExampleTranslator(HSCTranslator) HSC create)
Thank you!

timj · January 13, 2022, 5:26pm

ExampleTranslator is a placeholder for whatever translator you write – it is not a translator that is available to use as an example.

For your instrument metadata translator you can either write a new package, send a pull request to GitHub - lsst/astro_metadata_translator: Observation metadata handling infrastructure, or include the translator in your obs package.

You can see the CFHT MegaPrime, DECam, and HSC translators are directly in astro_metadata_translator at astro_metadata_translator/python/astro_metadata_translator/translators at main · lsst/astro_metadata_translator · GitHub

The DECam one might be a good template for you: astro_metadata_translator/decam.py at main · lsst/astro_metadata_translator · GitHub

The LSSTCam translators are currently direclty in the obs_lsst package because it’s easier to quickly iterate on them when they are all in one place: obs_lsst/python/lsst/obs/lsst/translators at main · lsst/obs_lsst · GitHub

@raphaelshirley is developing a metadata translator for a VISTA instrument.

What is the instrument you are working on?

Yuanyu · January 15, 2022, 8:49am

Thank you!
You and other people really help me a lot! And your detailed documents about pipeline are pretty useful!
I am a postgraduate student new to astronomy, and I am working on WFST in China.

siangyuhuang · February 11, 2022, 5:18pm

How do you solve this ValueError problem “Labels {‘TE4’, ‘nsrcMeasVisit’, ‘TE3’} were not found in the declared pipeline” ? I faced this, too.

jeffcarlin · February 11, 2022, 11:33pm

That ValueError problem likely relates to using an older version of the pipeline. Try using a recent weekly pipeline version (as recent as possible), and make sure that you checkout (and set up) the same tagged versions of the Science Pipelines and rc2_subset (i.e., if you’re using weekly w_2022_05, then checkout and setup versions with tags w.2022.05, following the instructions in Jim’s reply above).

fsklich · February 18, 2022, 1:04pm

Hello Jeff.
I’m migrating to the latest V23 of the pipeline. Looking forward to it, as I have learned a lot from the version I installed Q3 last year.
I have a tech question for you, as it relates to doing some self-debugging.
I’m fond of using DEBUG level logging, with pipetasks parameters "–log-level DEBUG --long-log --log-tty ".
In the log files I enjoy seeing the actual “funcName” and “lineno” and whatever the log statement provides.
My reading on python logging suggests that we should be able to ALSO have the logging module include the actual ARGUMENTS that are passed to the subject “funcName”.
Are you familiar how I can do this? Do the LSST developers use this feature to test/verify that expected ARGUMENTS are being passed to a function? Realizing also that arguments can take several forms.
I enjoy doing a lot of debugging on my own and hope you can help me affirm how to do this.
Enjoy your posts…usually very clear with good suggestions.
Fred, Dallas

timj · February 18, 2022, 4:03pm

We do not set stack_info=True in our logging – it’s something that each log user has to decide would be important to include in the log and for production code I doubt we would ever want to include it.

Calculating just the caller arguments for the function (taking stacklevel into account) is possible but I’m not sure how we would decide when such a thing should be calculated – the inspect module has a pretty big overhead and we’d want to avoid it in general. If we find that there is a use case we could possibly write a special log helper that calculates it and logs a message at TRACE level. We would not want every debug log message inside a function to have to recalculate the stack.

siangyuhuang · February 19, 2022, 5:25am

I see. How do I checkout the version of rc2_subset. Where can I find that version of it ? Or How do I create the new version of rc2_subset ? I use w_2022_08 version of pipeline currently.

fsklich · February 19, 2022, 1:25pm

Timj
I never suggested anything about production code. Geeze.
The relevant extracts from my post are:
My reading on python logging suggests that we should be able to ALSO have the logging module include the actual ARGUMENTS
and
I enjoy doing a lot of debugging on my own …
I want this for my own analysis and debugging…not for production lsst.
I never suggested we would calculate “just” the caller arguments…How do you calculate an argument? I only wish to include them.
Never mind Tim…I can see other options on the public domain…just thought that LSST developers may have a common/quick/easy trick to include the args passed in function calls.
I’ll figure it out.
Fred