Currently, I’m trying to pre-process DECam images using the -j tag to parallelize the process.
I specifically try running:
pipetask run -b /mnt/flock/jahumada/data_hits -i DECam/raw/all,DECam/calib -o DECam/raw/crosstalk-sources -p $CP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml -j 40 --register-dataset-types
but in reality, it utilizes only 12 cores. I ran the command again but with
-j 12 but then it utilized around 8. Tried to run it then with 8 cores, but then again the process effectively uses fewer cores. I was wondering if the tag
-j has any limits/restrictions? My reference for parallelization is Parallel processing with command-line tasks — LSST Science Pipelines
Any other reference that you think could help clear this problem, I will appreciate a lot
Thanks a lot!