Hi, I would like to create a unit test using Butler which involves creating some things in a temporary collection and then removing the collection once the test is done. But I have trouble removing a collection. What would be a proper procedure for cleaning it up and removing? Can someone provide some sample code?
Thanks
Are you testing with a Butler that has existing content in it? Most of our testing is with smaller Butler repositories that are either constructed from a testdata package with the data and an exported YAML file (see pipelines_check package), or have a small Butler SQLite registry committed to git. Regardless we clean up everything after the testing and nothing needs to see the new run collection.
If you have a large shared butler that you want to run unit tests from then your best bet is for the unit test to create a local butler and use Butler.transfer_from() to copy over just the datasets you need (if your main repo is using a POSIX datastore then transfer with symlinks will be fine). Then you run your tests in that local SQLite registry using a temp directory as datastore and delete it when you are done.
OK thanks, I’ll read up on transfer_from()
You will need to use w.2022.30 (or a recent daily) to get the usable version of Butler.transfer_from() that also copies dimension records across. That should be published tonight.
I will add that it is of course possible to call removeRuns() as part of a test cleanup but it’s entirely possible you will end up with something failing where call shutil.rmtree() on a temp dir is going to be much more reliable. If you do end up using collections in your main repo, you might want to adopt a standardized naming scheme for your collections so you can run a clean up script outside of the test environment to pick up any leftovers.
I see, thanks