Jenkins stack-os-matrix changes, new "blueocean" UI, pytest "junit" reporting

jenkins
pytest
junit
blueocean
Tags: #<Tag:0x00007f7f784a79d8> #<Tag:0x00007f7f784a7898> #<Tag:0x00007f7f784a7758> #<Tag:0x00007f7f784a7618>

(josh) #1

stack-os-matrix changes

Yesterday afternoon, the jenkins stack-os-matrix job was converted from a matrix job into a pipeline job. There were several factors motivating this change:

  • A number of irritating problems with the matrix job type that upstream is not interested in fixing as they consider pipeline to be a replacement
  • The desire to add aggregated pytest test reporting instead of having test reports for each os/python version recorded independently for each matrix sub-configuration
  • Compatibility with the “blueocean” dashboard

This has resulted in several end-user visible changes, including:

  • Sadly, the old build history is gone. The build ID sequence was preserved but HTML links to builds prior to 26452 are broken.
  • The matrix status on the build page is gone as are links to the “sub-configurations” (eg., centos-7,py2).
  • The console output from all build configurations are now merged into a single console under the “classic” jenkins UI (this is not true under “bluecoean”). Using blueocean to look at the console logs is recommended.
  • The python version multiple choice parameter has been removed. The plugin that provided this parameter type has been deprecated and is incompatible with blueocean. All stack-os-matrix builds now run on both py2 and py3.

Blueocean dashboard

Blueocean is a new jenkins UI/dashboard that is intended to eventually replace the classic Jenkins UI. It provides a modern web-app interface based on react.js. This is a new project but it is evolving extremely quickly. There are still some rough spots but it is currently the best option for viewing the status of pipeline jobs with “parallel branches” (eg, stack-os-matrix).

Opening blueocean

Users are able to move between the “classic” UI and blueocean as desired. Most pages in the classic UI have a link in the left hand menu to open blueocean. Eg.:

Returning to “classic” UI

All blueocean pages have an exit icon in the top right hand corner than may be used to return to the classic UI.

Running a build

Starting from the stack-os-matrix job page in blueocean:

Click on the Run button:

Input any desired build parameters and then click on the Run button at the bottom of the dialog:

Console

Pipeline status / console output can be accessed a couple of ways.

For a few seconds after starting a new build, there will be a status pop up in the lower left hand corner with an Open link.

Or at any time from the job page by clicking on the build description:

Console output from the various build configurations can be accessed by clicking on nodes in the pipeline status display. A running or queued node will be a hollow blue color, green if it finished successfully and red if there was a failure.

When a node / build configuration has stopped running, all of the console output section will be collapsed. The section of stack-os-matrix of most (probably only) interest to end-users is ./buildbot-scripts/jenkins_wrapper.sh -- Shell Script.

Clicking on the section description will toggle it open/closed.

pytest / junit reports

As part of @timj’s recent work to switch to pytest as a test runner, test results are being recorded in the junit/xunit XML format which jenkins is able to parse and display. junit test reporting has been added to stack-os-matrix job.

Please be aware that the test execution times reported in jenkins are currently wrong and should be ignored until a resolution is found.

blueocean test report

From the status page for a build, click on the Tests tab in the upper right hand corner:


classic UI test report

From the build page, click on the Test Result link:



(Scott Daniel) #2

This seems to have done away with the ability to get a list of all of the builds you started. I used to be able to go to

https://ci.lsst.codes/user/danielsf/builds

and see a record of every build I had started. That no longer works. Was this intentional? Is there anyway to get this feature back?


(josh) #3

@danielsf I’ve poked at jenkins’ internal api and the method for listing a user’s builds does not return any pipeline jobs. I’ll make a bug report upstream.


(josh) #4

Upstream bug report: https://issues.jenkins-ci.org/browse/JENKINS-46529
Our tracking issue: https://jira.lsstcorp.org/browse/DM-11696


(Krzysztof Findeisen) #5

Is there any way in the Blue Ocean UI to stop console updates from forcing the browser to the bottom of the page? It’s really annoying when I’m trying to access something else (typically the header, the list of subruns, or something else at the top of the page).


New Jenkins (ci.lsst.codes) documentation
(josh) #6

Not currently. It also really needs a link to return to the top of the page for jobs with a large amount of console output.


(josh) #7

Upstream issue for the console output problem: https://issues.jenkins-ci.org/browse/JENKINS-45182