ORAN Developer's Guide to CI Resources and Processes at the LF

This is a guide for software developers and testers about the continuous integration (CI) resources and processes used by the project and supported by the Linux Foundation.

Continuous Integration Network Resources

Most of these resources authenticate users via a Linux Foundation identity.

Development Procedures and Policies

This section guides the developer in submitting code, reviewing code, tracking the status of builds and requesting creation of release versions. These recommended practices come from the Linux Foundation.

Always create a branch

When working with a git repository cloned from Gerrit you can create as many local branches as you like.  None of the branches will be pushed to the remote Gerrit server, the branches remain forever private.  Creating branches for each task will allow you to work on multiple independent tasks in parallel, let you recover gracefully from various situations, and generally save aggravation and time. Also see these instructions on tagging and branching for releases:

Tagging and Branching Steps Process

Write good git commit messages

The Linux Foundation strongly recommends (and eventually may enforce) Gerrit commit message content. A git commit message must meet these requirements:

1. The first line should have a short change summary, up to 50 characters
2. The second line must be blank
3. The body of the commit message should have a detailed change description
4. The line before the footer must be blank
5. The footer (last block of lines following a blank line) must consist of these lines:
   1. Issue-ID: line with a valid Jira issue number, composed and inserted manually by the committer
   2. Change-Id: line, which is automatically generated and inserted by git review
   3. Signed-off-by line, which is automatically generated and inserted by git commit

An example is shown below:

Null check for ClientResponse

NullPointerException might be thrown as cres is nullable here

Issue-Id: PROJECT-999
Change-Id: I14dc792fb67198ebcbabfe80d90c48389af6cc91
Signed-off-by: First Last <fl1234567890@my-company.com>

Please note that the Jira system's pattern matcher that finds issue IDs in Gerrit reviews is extremely limited.  The "Issue-Id" line must be EXACTLY as shown above! The keyword must be written in mixed case as shown, then a colon, then a space, then the appropriate project name in ALL CAPS, then a hyphen, then the number.

Working with Docker images

The docker image name is set by the Jenkins job, as configured by the JJB. The method for choosing the docker image tag is also configured in the Jenkins job, one of the following:

• "latest": This constant string is always used
• "git-describe": The output of the "git describe" command is used. This requires that the repository has at least one git  tag.
  
• "yaml-file": The file container-tag.yaml in the project repository is queried for the entry with key "tag"

For the non-constant options, the development team must manage the tag or the version string in the YAML file.

The Jenkins verify job creates a docker image to test the creation process, then discards the image at completion.

The Jenkins merge job creates a docker image and pushes the image to the staging registry.Jenkins job sets the PUSH registry to STAGING.

Creation of a final release image requires opening a ticket with LF release-engineering team.  They sign the image found in the staging registry and move it manually to the release registry.

Using Gerrit

The project uses Gerrit to automate the process of reviewing all code changes before they are committed to the Git repository. For a tutorial on git you might start here:

https://git-scm.com/doc

Also see this guide published by the LF Release Engineering team about using Gerrit:

https://docs.releng.linuxfoundation.org/en/latest/gerrit.html

Prohibited Content

Gerrit is designed to host text files – source code.  It enforces a size threshold on every commit, the default limit is 5MB.  Further the Linux Foundation prohibits committing binary files such as compiled executables, jar files, zip archives and so on. An exception is made for binary picture (image) files in GIF, JPG and PNG formats, but the size limit must still be honored.

Quickstart Guides

The quickstart guides below describe the command-line procedures for performing common tasks on Gerrit.  The command-line tool "git review" is the most reliable and can be used on any platform.  Windows users should install "Git Bash" to gain support for command-line git.

Quickstart: Create and submit a change for review

git checkout -b my-local-branch
# work work work
git commit -as
    # Write a commit message including a line "Issue-ID:" with Jira ticket nr
git review master

Quickstart: Revise your open gerrit review

git checkout my-local-branch
# revise revise revise
git commit -a --amend
    # Revise your message but do NOT alter the Change-Id: line
git review master

Quickstart: Revise any open gerrit review

# Look up the change number shown top-left in Gerrit web, here "999".
git review -d 999
# revise revise revise
git commit -a --amend
    # Check your message -- do NOT alter the Change-Id: line
git review --no-rebase master

Quickstart: Resolve a conflicted review

git checkout master
git pull origin master
# Look up the change number in Gerrit, it's shown top-left, here 999
git review -d 999
git rebase master
# fix conflicts
git add .
git rebase --continue
git review master

Quickstart: Squash commits after you forgot the "--amend" flag

# count number of commits to be squashed, the 2 below is for two
git reset --soft HEAD~2
git commit

Reviewing and merging on Gerrit web site

Submitted reviews with changes should appear in the Gerrit web interface. The Jenkins job builder will publish test results, giving "Verified +1" if the test succeeds and -1 if the test fails.  A contributor or committer can review it with a -1/0/+1. A committer then must approve it with a +2 rating and merge the code to the master branch.

The committer may take any of several actions, such as clicking on the "Reply" button, adding reviewers, adding a review comment, and moving the flags to +2 and +1

Once a committer/contributor approves it, the code can be merged to the master branch.

Triggering Jenkins jobs from Gerrit

Jobs can be triggered at Jenkins by posting a comment in the Gerrit system on a review.

Post "recheck" to re-run the verify job.  This may be necessary due to intermittent network failures at Jenkins, because a dependency has become available, etc.

Post "remerge" to re-run the merge job.  This may be necessary due to intermittent network failures at Jenkins, to recreate a version of an artifact, etc.

Post "run-sonar" to run the Sonar analysis job (if applicable).

Releasing your project

When the project determines that an artifact in a staging repository is ready for release, a release can then be performed as follows:

1. Find the Jenkins job that created the release candidate.  Look among its output logs for the file with the name of the staging repository, it will have a URL like this:
   1. https://logs.o-ran-sc.org/production/vex-yul-o-ran-sc-jenkins-1/common-dataservice-maven-dl-stage-master/4/staging-repo.txt.gz
2. Open a ticket with the LF helpdesk to sign the artifact(s) in the staging repo and promote them to the release repo. Example request is below.
   
   1. Send email to helpdesk@o-ran-sc.org with CC to affected people, only To and Cc people can see the ticket status
   2. Supply the Jenkins job URL and Jenkins log URL from above.
   3. Supply the name of the autorelease repository.
3. Add a tag for the release on the Gerrit web site.

Example release request:

Please release the PROJECT-NAME jar and docker image at version VERSION
Stage job:
    https://jenkins.o-ran-sc.org/job/PROJECT-stage-master/201/
Logs:
    https://logs.o-ran-sc.org/production/vex-yul-o-ran-sc-ra-jenkins-1/PROJECT-stage-master/201/
Autorelease repo name:
    autorelease-1150
Thanks in advance

Once complete the artifact(s) will appear in the Nexus2 release repository and/or Nexus3 release registry.

Jenkins Job Builder Templates

All jobs in the Jenkins server are generated from Jenkins Job Builder (JJB) templates. The templates are maintained in the "ci-management" git repository.  A project template is defined for each project in a YAML file (e.g., "com-log.yaml").  The project templates use job templates defined by the Linux Foundation Global JJB as well as job templates custom to this project.