Jenkins `stack-os-matrix` job may now run on OSX 10.14/Mojave

Data Management DM Notifications
, , , ,
#1

A new agent/node named mojave-1 was added to the production DM jenkins environment yesterday evening (DM-18107). It is a bare metal 2018 updated Mac mini, which is a departure from our existing ESXi VMs running on “Mac pro” hardware. As the “current” generation Mac pro hardware is ~5 years old, we would prefer to invest in more modern hardware. If no major issues arise with the new mini, we are planning to add enough additional units to support a smooth transition between osx releases in the future. Also worth noting that a more recent version of Xcode is installed. The clang compiler string is clang-1000.10.44.4. Small sample size benchmarking suggests the mini has a modest performance increase with builds of lsst_distrib lsst_ci from a clean state taking ~2:45.

The only jenkins jobs currently allowed to run on mojave-1 are stack-os-matrix and scipipe/lsst_distrib. As there are four existing sierra-X nodes, this means there is a ~1 in 5 chance (job/node affinity makes the distribution not completely random) of running on mojave. The nightly/weekly/official eups tarball builds are still running on sierra. The loose plan is to RFC moving the tarball releases to mojave once there is enough capacity for every stack-os-matrix build to comfortably run on it. A related issue is if science pipeline developers would like to change the value of MACOSX_DEPLOYMENT_TARGET at the same time, which has long been set to 10.9.

1 Like
#2

I have been wondering about that. Everything does build fine using it and I’ve had to force it in a couple of builds for mojave to work (10.8 no longer works and was a default in some places). I won’t object to us tweaking that value.

#3

As there are four existing sierra-X nodes, this means there is a ~1 in 5 chance (job/node affinity makes the distribution not completely random) of running on mojave.

It seems odd to me that the Jenkins system doesn’t distinguish between these nodes. Should we be worried about a stack-os-matrix job having intermittent build failures depending on which OSX version it got run on?

#4

In a perfect world we’d run multiple macOS versions along with multiple Linux versions. At the moment we are severely resource constrained on macOS so I’m fine with pooling all macOS build machines.

#5

This is only currently true for the stack-os-matrix and scipipe/lsst_distrib jobs as a short term measure to try out building with the refreshed mac mini.

1 Like