Hello, I am a postgraduate student new to astronomy, and I am working on WFST, trying to use your well designed LSST pipeline to process our instrument’s data. At the very begining, my aim is just about the part the astrometry, but then I find the pipeline is well designed and can apply to other instruments once we set the package documents well. So, finally i decide to create a package obs_wfst. But I meet a lot of problems and I want to post the problems about creating a new package concentratly in this topic.
First, about How to create a new LSST obs package, I am following the four steps, about the instrument-test:
import unittest
import lsst.utils.tests
import lsst.obs.example
from lsst.obs.base.instrument_tests import InstrumentTests, InstrumentTestData
class TestExampleCam(InstrumentTests, lsst.utils.tests.TestCase):
def setUp(self):
physical_filters = {"example g filter",
"example z filter"}
self.data = InstrumentTestData(name="Example",
nDetectors=4,
firstDetectorName="1_1",
physical_filters=physical_filters)
self.instrument = lsst.obs.example.ExampleCam()
if __name__ == '__main__':
lsst.utils.tests.init()
unittest.main()
and the 4 tests passed 3, the test about sqlite falled.
self = <sqlalchemy.dialects.sqlite.pysqlite.SQLiteDialect_pysqlite object at 0x7f1442060c70>, cursor = <sqlite3.Cursor object at 0x7f14336727a0>
statement = 'INSERT INTO instrument (name, visit_max, exposure_max, detector_max, class_name) VALUES (?, ?, ?, ?, ?)'
parameters = ('WFST', 1073741824, 1073741824, 10, '_instrument.WFSTCamera'), context = <sqlalchemy.dialects.sqlite.base.SQLiteExecutionContext object at 0x7f14336816d0>
def do_execute(self, cursor, statement, parameters, context=None):
> cursor.execute(statement, parameters)
E sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: instrument.name
E [SQL: INSERT INTO instrument (name, visit_max, exposure_max, detector_max, class_name) VALUES (?, ?, ?, ?, ?)]
E [parameters: ('WFST', 1073741824, 1073741824, 10, '_instrument.WFSTCamera')]
E (Background on this error at: https://sqlalche.me/e/14/gkpj)
../../../../../conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.7.0/lib/python3.8/site-packages/sqlalchemy/engine/default.py:719: IntegrityError
================================================================================== short test summary info ==================================================================================
FAILED test_instrument.py::TestInstrument::test_register - sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: instrument.name
================================================================================ 1 failed, 3 passed in 1.79s ================================================================================
And I inspect the sqlite document of HSC, find that in the instrument table, it’s class_name is lsst.obs.subaru.HyperSuprimeCam, I guess, it should have a similar format. I think this question can be solved if i can import my package like
from lsst.obs.wfst import WFSTCamera
not
import sys
sys.path.append("/home/yu/lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_wfst/python/obs/wfst")
from _instrument import WFSTCamera
And this make the function get_full_type_name(self) can’t work well.
So, how can I let my package can interacte with your interface and can import by lsst.obs.wfst?
I also wondering the the meaning of 22.0.1-44-g74bdbb4e+988f982fce in the url, looks like the dataset id in sqlite. I guess the package I created should also have folder named similar to commuicate with the interface?
/home/yu/lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_subaru/22.0.1-44-g74bdbb4e+988f982fce/hsc/
Second, is about the method makeDataIdTranslatorFactory
def makeDataIdTranslatorFactory(self) -> TranslatorFactory:
# Docstring inherited from lsst.obs.base.Instrument.
factory = TranslatorFactory()
factory.addGenericInstrumentRules(self.getName(), calibFilterType="band",
detectorKey="ccdnum")
# DECam calibRegistry entries are bands or aliases, but we need
# physical_filter in the gen3 registry.
factory.addRule(_DecamBandToPhysicalFilterKeyHandler(self.filterDefinitions),
instrument=self.getName(),
gen2keys=("filter",),
consume=("filter",),
datasetTypeName="cpFlat")
return factory
I guess it’s a package that communicate from gen2 to gen3, if I want to make a completely gen3 package, should I need that? without this won’t pass the test.
Third, it’s about the transmission curves, they are many documents about transmissions curve for hsc:
[yu@localhost transmission]$ pwd
/home/yu/lsst_stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_subaru/22.0.1-44-g74bdbb4e+988f982fce/hsc/transmission
[yu@localhost transmission]$ ls
filterTraces.py wHSC-IB945.txt wHSC-NB656.txt
M1-2010s.txt wHSC-N1010.txt wHSC-NB718.txt
modtran_maunakea_am12_pwv15_binned10ang.dat wHSC-N400.txt wHSC-NB816.txt
qe_ccd_HSC.txt wHSC-NB387.txt wHSC-NB921.txt
README.txt wHSC-NB468.txt wHSC-NB926.txt
throughput_popt2.txt wHSC-NB515.txt wHSC-NB973.txt
throughput_win.txt wHSC-NB527.txt
And in hsc’s _instrument.py, there are method about how to make use of these documents to create transmission curve, for example:
# Write transmission sensor
sensorTransmissions = getSensorTransmission()
datasetType = DatasetType("transmission_sensor",
("instrument", "detector",),
"TransmissionCurve",
universe=butler.registry.dimensions,
isCalibration=True)
butler.registry.registerDatasetType(datasetType)
for entry in sensorTransmissions.values():
if entry is None:
continue
for sensor, curve in entry.items():
dataId = DataCoordinate.standardize(baseDataId, detector=sensor)
refs.append(butler.put(curve, datasetType, dataId, run=run))
# Write filter transmissions
filterTransmissions = getFilterTransmission()
datasetType = DatasetType("transmission_filter",
("instrument", "physical_filter",),
"TransmissionCurve",
universe=butler.registry.dimensions,
isCalibration=True)
butler.registry.registerDatasetType(datasetType)
for entry in filterTransmissions.values():
if entry is None:
continue
for band, curve in entry.items():
dataId = DataCoordinate.standardize(baseDataId, physical_filter=band)
refs.append(butler.put(curve, datasetType, dataId, run=run))
# Write atmospheric transmissions
atmosphericTransmissions = getAtmosphereTransmission()
datasetType = DatasetType("transmission_atmosphere", ("instrument",),
"TransmissionCurve",
universe=butler.registry.dimensions,
isCalibration=True)
butler.registry.registerDatasetType(datasetType)
for entry in atmosphericTransmissions.values():
if entry is None:
continue
refs.append(butler.put(entry, datasetType, {"instrument": self.getName()}, run=run))
# Associate all datasets with the unbounded validity range.
butler.registry.certify(collection, refs, Timespan(begin=None, end=None))
But I didn’t find similar method in DEcam’s _instrument.py, but in the quantum graph about the single_frame_process, it actually make use of the transmission curve, so where is the transmission of DEcam? What make this difference happen? And I also noticed that there are something like bfkernel, stary night in the hsc’ _instrument.py.
single_frame.pdf (23.0 KB)
There are also other questions, and I will post here later. I think i’m just a recruit new to astronomy and still in bootcamp, the problems may seems naive for you. Any answers, sugesstions or documents you point out for me will be pretty helpful.
Thank you!
