In DM-3373 and a few other tickets, we ported over from HSC scripts for constructing calibs. These reside in pipe_drivers, and use the MPI process pool in ctrl_pool. Example command-lines for using these scripts are on DM-5124, but here’s a rough guide:
- Bin scripts are
- These are like a regular
CmdLineTask, but with additional arguments that configure how to do the parallelism:
smp(Symmetric Multi-Processing, i.e., run on the local machine with multiple cores),
pbs(submit the batch request to your PBS cluster) or
slurm(submit the batch request to your Slurm cluster).
--jobis the job name (used by batch systems to identify the job, and in naming output log files).
--timeis the walltime limit for batch systems, per element (in this case, the element is individual CCDs).
slurmmodes) specifies the number of cores to use.
slurmmodes) specifies the number of nodes and CPUs per node to use.
--batch-submitcan be used to provide additional options for batch submission (i.e., qsub or sbatch, as appropriate).
--mpiexeccan be used to provide additional options for
mpiexec. On our Slurm clusters at Princeton, we’ve found we need
-bind-to socketto prevent trouble.
--dry-runcan be used to test the command-line before submitting.
- You will probably need some configuration additions in your obs_* package.
- The individual scripts write
fringe_configdatasets, so your Mapper needs to support those.
- Overrides can be added to your
@merlin is trying to get this working with DECam at the moment. Hopefully we can report success soon! Users of other cameras are welcome to request help here.