Instructions
These are some step-by-step instructions on how to use the stack to ingest raft test-stand data and process it with eotest.
-
Install a weekly build, with a version greater than or equal to w_2017_50
- installation instructions can be found here:
- https://pipelines.lsst.io/install/newinstall.html
- alternatively, use a shared stack on, e.g. lsst-dev (all the weekly builds are on there already, it’s really easy and potentially saves a lot of installation pain!)
-
Setup your stack. This example should work on lsst-dev:
. /ssd/lsstsw/stack3/loadLSST.bash
setup lsst_distrib -t w_2017_50 -
Setup eotest (as it’s not part of lsst_distrib). Note that this is a fork of eotest, to be used until the changes made here are merged upstream (and perhaps even then, e.g. if you want to make pdfs from the outputs)
NB: you must be within that directory, and in future you will need to do something likesetup -j eotest -r ~/path/to/eotesteach time you setup the stack.git clone git@github.com:lsst-dm/eotest.git
cd eotest
git checkout tickets/DM-11348
setup -j -r .
scons -
Make some TS8 data available. Let’s pick one raft, I’m going to use RTM-005 (aka RTM2). This would go something along the lines of:
mkdir /path/to/ts8_data
rsync -r <username>@rhel6-64.slac.stanford.edu:/nfs/farm/g/lsst/u1/mirror/BNL-prod/prod/LCA-11021_RTM/LCA-11021_RTM-005/* /path/to/ts8_data --progress
If you’re working on lsst-dev you can find this data already copied over, in /datasets/comCam/raw/BNL/prod/LCA-11021_RTM/LCA-11021_RTM-005.
-
Now we enter stack-land. Make a repository for this data, put a mapper in it, and ingest the data. We want to only ingest the .fits files which are part of the actual acquisitions, so we use the following pattern to select them. Hopefully this is robust. Note that we are using the
ComCamMapperfromobs_comCamas this is a single raft. If you want to process data from more than one raft, make a different repo for each raft. (Note that you can have multiple runs in each repo, but you can’t have multiple rafts in a repo).mkdir /path/to/ts8_repo
echo "lsst.obs.comCam.ComCamMapper" > /path/to/ts8_repo/_mapper
ingestImages.py /path/to/ts8_repo /path/to/ts8_data/*/*_acq/v0/*/S*/*.fits -
Assuming all the above worked, in that repo you should now have the _mapper, a directory called “raw” containing symlinks to all the ingested files, and a sqlite3 registry file called “registry.sqlite3”. We can now open a python session/jupyter notebook and do things. Let’s start by making a butler for that repo.
.
# import the relevant stuff
import lsst.daf.persistence as dafPersist
from lsst.cp.pipe import RunEotestTask
# make ourselves a butler
repo_path = '/path/to/ts8_repo/'
butler = dafPersist.Butler(repo_path)
Now, if you used the path on the SLAC servers in this example, you will have copied and ingested data from many different runs, which obviously need to be processed separately. So, we ask the butler which runs are available like so:
runs = butler.queryMetadata('raw', ['run'])
print(runs)
which should return something like
['3849', '4080', '4390', '4418', '5597', '5704', '5819', '4068', '4389', '4417', '5105', '5703', '5811']
So, pick a run number to use. If you’re very brave you can try to deduce which the good runs are using things like eTravellers and the Camera Data Portal. I’m just going to pick 4389 because word on the grape-vine is that that’s a good one.
# instantiate a config for our task, and set the mandatory output config parameter
runEoConfig = RunEotestTask.ConfigClass()
runEoConfig.eotestOutputPath='/path/to/ts8_repo/eotest_output'
# examples of how you turn the eotest sub-tasks on and off. All on by default
# runEoConfig.doFe55 = False
# runEoConfig.doReadNoise = False
# runEoConfig.doBrightPixels = False
# runEoConfig.doDarkPixels = False
# runEoConfig.doTraps = False
# runEoConfig.doCTE = False
# runEoConfig.doPTC = False
# runEoConfig.doFlatPair = False
# instantiate our task use the config
eotestTask = RunEotestTask(config=runEoConfig)
# run the task, specifying our chosen run if there is more than one in the repo
eotestTask.runEotestDirect(butler, run=4389)
# once the task has run, make a pdf of the results
eotestTask.makeEotestReport(butler)
If all went well, eotest should have run, and produced all its results and written them in the usual eotest output format. If things went really well and you have a TeX distro installed with pdflatex on your path, then pdfs for each sensor in the raft should have been produced in /path/to/ts8_repo/eotest_output/plots.