Folks at UW have requested “Butlerized access to logs” (on this page), I’m looking for a little more information; Is the idea simply to keep this information in a repository so that it can be retrieved via Butler? In this case it might be as simple as defining the dataset type for that kind of information (right?)
Or are you wanting to be able to manage configurations and write logs via the Butler (@ktl didn’t very much like this idea, but we can discuss it)
@KSK, can you or someone up there provide more details?
Depends what we mean by logs. Do we mean tabular information generated by other systems that can be queried based on time/filter/airmass. Do we mean requesting actual text files in the classical definition of log. Or the parsed version of log files stored in the QA system? “log” can mean a surprising number of things to different people.
I may be asking for the wrong things, so it’s possible that the butler is not the answer here, but we frequently find ourselves in the position of wanting to look at the logs and configs forensically. Currently, there is no easy way to get these.
There are multiple problems. The output from the logger is not automatically persisted anywhere unless a switch is used. The configs are saved, but there is no way to explore the persisted configs programmatically. The configuration of the stack at the time of the run is not persisted and can’t be automatically.
My first thought is that these are all items related to provenance tracking (the configuration definitely is) and should be handled by that system. Log handling is something that SQuaRE are looking into in a serious way. Associating a log with a particular run which associates it with particular output data products and input files and configuration is something that I think has to be solved outside the butler.