S18 Reprocessing of HSC PDR1

dm-dev
Tags: #<Tag:0x00007ff62bc223f0>

(Hsin-Fang Chiang) #1

We have finished reprocessing the entire HSC PDR1 data, using the DM Software Stack w_2018_15.

For team members with access to lsst-dev, the output repositories are available at:

  • /datasets/hsc/repo/rerun/DM-13666/UDEEP/
  • /datasets/hsc/repo/rerun/DM-13666/DEEP/
  • /datasets/hsc/repo/rerun/DM-13666/WIDE/

More details of this reprocessing can be found at:
https://confluence.lsstcorp.org/display/DM/S18+HSC+PDR1+reprocessing


(Paul Price) #2

Congratulations, @hsinfang!

When we processed the Wide dataset at Princeton a few years ago, our utilisation estimate was about 1k core-weeks. Your value of 6718 node-hours looks similar, but interpretation requires knowing the number of cores per node; what did you use?


(Hsin-Fang Chiang) #3

For wide, it was

  • 45 cores per node in singleFrame
  • 48 cores per node in skyCorrection
  • 12 cores (for read) in mosaic
  • 12 cores per node in coadd and multiBand
    (overly conservative, I know…)

That sums up to ~720 core-weeks.


(Paul Price) #4

What’s the number if you include the cores you weren’t actively using but scheduled to steal their memory (because you couldn’t use them to do anything else)? More like 2k core weeks?


(Hsin-Fang Chiang) #5

Each node of Verification Cluster has 24 physical cores or 48 virtual cores (reference here). So indeed it’s ~2k core weeks, or half if you count the physical cores.