Making a new camera.yaml, and what to do afterwards! I.e., where do I begin??
(Below are conversations edited for clarity)
Jacqueline McCleary [4:33 pm]
Hello! Please forgive the stupid questions to come, but I am definitely a newbie – though long-time DESC lurker. I am trying to write a new obs_ package for SuperBIT and am all kinds of lost. Based on suggestion, I started by forking obs_auxTel (hi @merlin) and am now editing camera.yaml file to suit some mock data. I am a shade confused on the usage of pointer-looking things ("&", “*”) in said camera.yaml file, as well as ">>"
Also, in e.g. RawSerialPrescanBBox, “serial” is perpendicular to readout direction and “parallel” perpendicular to the direction of readout? At least the BIT only has one amplifier and one CCD (one raft/one sensor/one array!), so this is the easiest case possible!
John Parejko [4:35 PM]
Camera definitions are YAML format. Here’s a quick reference to it: http://yaml.org/refcard.html
Merlin Fisher-Levine [4:35 PM]
Oh boy… this is supposed to get easier in the future, but that day is not yet upon us I’m afraid. The good news: you won’t need to worry about * or & if you have only one amp, I think, they’re used for inheritance type things where you want to tile something
Jacqueline McCleary [4:36 PM]
sorry, I should have been more clear. It seems like you define a generic CCD and AMP key and then specific instances of CCD and AMP inherit those?
Simon Krughoff [4:39 PM]
The offset and reference pixel are for making the focal plane model.
Merlin Fisher-Levine [4:39 PM]
They’re for, if I recall, the nominal offset of some fiducial on your CCD from the boresight/center of the focalplane/some nominal WCS.
Jacqueline McCleary [4:42 PM]
Here’s another newbie question: after developing a reasonable camera.py, what happens next? It seems like there are 100 things that all need to happen at the same time, and I’m having trouble prioritizing. I made some mock calibration files (so I will ignore the making of master calibs for now) but do I need to write a new ingest.py? do I NEED a defects file? do I make a new mapper?
I have looked at obs_decam, obs_subaru (hsc and suprimecam) and now obs_auxTel so there are a lot of worked examples, but they are all wonderfully different
Simon Krughoff [4:46 PM]
I would address ingest.py next.
Merlin came in later to address my questions!
Merlin Fisher-Levine [3:55 AM]
I made some mock calibration files (so I will ignore the making of master calibs for now)
If they’re very high signal mocks (i.e. with the shot noise already beaten down) then these should indeed serve well as master calibs. However, you will still need to ingest them. More on that in a bit…
do I NEED a defects file?
No, you don’t need a defects file for things to run. There are config options to will allow you to run without them. obs_auxTel and obs_comCam have no defect files yet, and they run OKish, take a look at the default config overrides in their /config/*.py files, you’ll see defects switched off in there.
do I make a new mapper?
Yes, but it sounds like you’re already on to this 
I have looked at obs_decam, obs_subaru (hsc and suprimecam) and now obs_auxtel so there are a lot of worked examples, but they are all wonderfully different
Yes. Sorry about that 
. obs_subaru is by far the most matrure, but is also therefore complex, plus it supports two different cameras, so breaks the mould there a bit, but that is the best example in the above. obs_decam is mostly made of evil and should not be copied unless you really have to. Much of its evil stems from trying to simultaneously support “community pipeline” inputs as well as the raw data, though there is other evil in there too… beware when copying this package. obs_auxTel (and it’s slightly-more-complex brother obs_comCam) are the newest of the bunch, and should be a little cleaner. However, and this is a big one, these “cameras” have never been on sky, and the packages have only just been born, so it’s likely they will fail in all sorts of interesting ways as they have never been used to run processCcd.py! They will, in time (and actually worryingly soon), but they haven’t yet, so use them with caution.
John Parejko [4:46 PM]
I’d suggest filling in the tests that are implemented in obs_base, but some of the more useful ones haven’t been merged to master yet. You could look at obs_test/tests/test_obs_test.py for a start. Basically, you’d copy that file and update the various values in it so they are correct for your camera, and then write code to make the tests pass.
See tests.py in obs_base for the aggregator class for all of the tests, and various test classes (e.g. camera_tests.py).
Jacqueline McCleary [4:50 PM]
Thank you @parejkoj I will use that to guide workflow.
So if I understand correctly, the next steps would be to build/modify an ingest.py, see if mapper needs to change, and then run through the assorted tests you linked until the code works?
John Parejko [4:53 PM]
Those steps sound reasonable. Unfortunately, we don’t really have a “how to build an obs package” document. [now you do 
–Ed.]
John Parejko [5:10 PM]
I wrote that test framework, and implemented it for a couple of existing obs packages. But that doesn’t mean I know the best way to go about fleshing out those tests in a new obs package.
I have some ideas about that, but they’re informed by my knowledge of the internals of the tests, not necessarily how a “user” would see them. @merlin is a good example of such a “user”.
Jacqueline McCleary [5:13 PM]
OK, I see. I have a dumb question: obs_decam had a separate ingestDecam.py (or something) and an ingestCalibs.py. AuxTel seems to have both merged into one. I take it that obs_auxTel's practice is OK?
Merlin Fisher-Levine [3:55 AM]
As I said above, obs_decam is evil. Do not follow the ingestMyCamera.py pattern, this only exists in obs_decam because of the community pipeline, you should just run ingestImages.py for your ingest. However, obs_auxTel has not merged ingestImage.py and ingestCalibs.py into one, and nor should you, this is just a misunderstanding (of which you have amazingly few given how bizarre the world you’re entering into is!). You will need both; ingestImages.py ingests your raw data, both the on-sky frames and the raw, input calibration frames (the individual biases and darks and flats etc). One then runs some nasty scripts in pipe_drivers called constructBias/Dark/Flat.py to gang these together to make your master calibs. Then these processed calibs are ingested into the calibration database using ingestCalibs.py, which you will need to do even if you’re not using these scripts to produce master calibs, as you will need to have entries in the calibration database in order for them to be found when running things like processCcd.py.
Brian Stalder [8:14 AM]
Thank you Merlin for these introductions. BTW, Simon and I have successfully adapted the obs_auxtel template for a new camera and got it to run processCCD.py through ISR [instrument signature removal – Ed.] (albeit with everything turned off) with real data. Slight issue with a missing bypass_ccdExposureId which needed to be added. Now working on setting up calibs.
Merlin Fisher-Levine [8:15 AM]
Ooooh, congratulations! That is no mean feat!
@bstalder if that’s something that is public/you can share, then @Jacqueline McCleary I suggest you start from that, as it’s known to work to a later processing stage!
Brian Stalder [8:17 AM]
https://github.com/bstalder/obs_atlas
LSST stack obs package for ATLAS telescopes
Website: http://fallingstar.com/home.php
it’s 1 CCD, 16 amps, in a flat (no MEF) raw fits file.
John Parejko [10:15 AM]
I’m still shocked that so many groups are currently trying to write obs packages, when we have no instructions on how to do so, each of our current ones is idiosyncratic, gen3 butler is going to change everything, and the obs package redesign is going to change everything again! And we never finished the test suite that might have made the process easier.
JEM’s takeaway: none of this is for the faint of heart, you are wandering into a construction project BYO hard hat, you get what you pay for, count your blessings, etc., etc.