This is a guide for software developers and testers about the continuous integration (CI) resources and processes used by the project and supported by the Linux Foundation.
Continuous Integration Network Resources
Most of these resources authenticate users via a Linux Foundation identity.
URL | Description |
---|
https://identity.linuxfoundation.org | The Linux Foundation identity management web site. |
https://gerrit.o-ran-sc.org | The Gerrit code review server hosts the Git repositories and supports the review and merge workflow. The review process basically requires all users to follow something like a git pull-request process, but restricts the publication (push) of private branches. |
https://jenkins.o-ran-sc.org | Jenkins is the continuous-integration server aka the build server. All users can see the contents of the Jenkins; no users can directly modify the configuration nor start a job in this Jenkins. Jobs are configured by Jenkins Job Builder (JJB) files in the ci-management gerrit repository. |
https://jenkins.o-ran-sc.org/sandbox | The Jenkins sandbox is a place for testing Jenkins job configurations. After requesting access, users can create jobs, reconfigure jobs, trigger builds etc. New JJB templates should be tested in the sandbox before submitting for review and use on the main Jenkins server. |
https://jira.o-ran-sc.org | Jira is the web site that supports agile development with epics, user stories, and especially issues (bug reports). |
https://nexus.o-ran-sc.org | Not heavily used by this project because Java is not a key technology, this Nexus 2 repository holds Maven (jar and pom) artifacts produced by builds. Snapshot and staging builds are all deployed to this repository. Release artifacts are created by promoting artifacts manually from the staging repository after suitable approval. Publicly accessible to users without credentials. All users should be able to access and browse artifacts through this URL. |
https://nexus3.o-ran-sc.org/ | This Nexus 3 server stores project Docker images and also mirrors external registries. Supports the following: These registries are open for anonymous read access. The Jenkins server has credentials to push images to the snapshot and staging registries, and to promote images to the release registry. Manual push of images is not supported. |
https://packagecloud.io | This site hosts binary artifacts such as Debian (.deb) packages, Red Hat Package Manager (.rpm) packages and more in these repositories: - o-ran-sc/master
- o-ran-sc/staging
- o-ran-sc/release
Also see Binary Repositories at PackageCloud.io |
https://sonarcloud.io | Sonar tools analyze Java and Python code for quality issues and code coverage achieved by unit tests (JUnit, tox) and publish results to this cloud-based server. |
Development Procedures and Policies
This section guides the developer in submitting code, reviewing code, tracking the status of builds and requesting creation of release versions. These recommended practices come from the Linux Foundation.
Always create a branch
When working with a git repository cloned from Gerrit you can create as many local branches as you like. None of the branches will be pushed to the remote Gerrit server, the branches remain forever private. Creating branches for each task will allow you to work on multiple independent tasks in parallel, let you recover gracefully from various situations, and generally save aggravation and time. Also see these instructions on tagging and branching for releases:
Tagging and Branching Steps Process
Write good git commit messages
The Linux Foundation strongly recommends (and eventually may enforce) Gerrit commit message content. A git commit message must meet these requirements:
- The first line should have a short change summary, up to 50 characters
- The second line must be blank
- The body of the commit message should have a detailed change description, wrap lines at 72 characters max
- The line before the footer must be blank
- The footer (last block of lines following a blank line) must consist of these lines:
- Issue-ID: line with a valid Jira issue number, composed and inserted manually by the committer
- Change-Id: line, which is automatically generated and inserted by git review
- Signed-off-by line, which is automatically generated and inserted by git commit
An example is shown below:
Null check for ClientResponse
NullPointerException might be thrown as cres is nullable here
Issue-Id: PROJECT-999
Change-Id: I14dc792fb67198ebcbabfe80d90c48389af6cc91
Signed-off-by: First Last <fl1234567890@my-company.com> |
Please note that the Jira system's pattern matcher that finds issue IDs in Gerrit reviews is extremely limited. The "Issue-Id" line must be EXACTLY as shown above! The keyword must be written in mixed case as shown, then a colon, then a space, then the appropriate project name in ALL CAPS, then a hyphen, then the number.
Working with Docker images
The docker image name is set by the Jenkins job, as configured by the JJB. The method for choosing the docker image tag is also configured in the Jenkins job, one of the following:
- "latest": This constant string is always used
- "git-describe": The output of the "git describe" command is used. This requires that the repository has at least one git tag.
- "yaml-file": The file container-tag.yaml in the project repository is queried for the entry with key "tag"
For the non-constant options, the development team must manage the tag or the version string in the YAML file.
The Jenkins verify job creates a docker image to test the creation process, then discards the image at completion.
The Jenkins merge job creates a docker image and pushes the image to the staging registry.Jenkins job sets the PUSH registry to STAGING.
Creation of a final release image requires opening a ticket with LF release-engineering team. They sign the image found in the staging registry and move it manually to the release registry.
Using Gerrit
The project uses Gerrit to automate the process of reviewing all code changes before they are committed to the Git repository. For a tutorial on git you might start here:
https://git-scm.com/doc
Also see this guide published by the LF Release Engineering team about using Gerrit:
https://docs.releng.linuxfoundation.org/en/latest/gerrit.html
Prohibited Content
Gerrit is designed to host text files – source code. It enforces a size threshold on every commit, the default limit is 5MB. Further the Linux Foundation prohibits committing binary files such as compiled executables, jar files, zip archives and so on. An exception is made for binary picture (image) files in GIF, JPG and PNG formats, but the size limit must still be honored.
Quickstart Guides
The quickstart guides below describe the command-line procedures for performing common tasks on Gerrit. The command-line tool "git review" is the most reliable and can be used on any platform. Windows users should install "Git Bash" to gain support for command-line git.
Quickstart: Create and submit a change for review
git checkout -b my-local-branch # work work work git commit -as # Write a commit message including a line "Issue-ID:" with Jira ticket nr git review master |
Quickstart: Submit a change to a branch other than "master"
Quickstart: Revise your open gerrit review
git checkout my-local-branch # revise revise revise git commit -a --amend # Revise your message but do NOT alter the Change-Id: line git review master |
Quickstart: Revise any open gerrit review
# Look up the change number shown top-left in Gerrit web, here "999". git review -d 999 # revise revise revise git commit -a --amend # Check your message -- do NOT alter the Change-Id: line git review --no-rebase master |
Quickstart: Resolve a conflicted review
git checkout master git pull origin master # Look up the change number in Gerrit, it's shown top-left, here 999 git review -d 999 git rebase master # fix conflicts git add . git rebase --continue git review master |
Quickstart: Squash commits after you forgot the "--amend" flag
# count number of commits to be squashed, the 2 below is for two git reset --soft HEAD~2
git commit
|
Reviewing and merging changes
Submitted reviews with changes should appear in the Gerrit web interface. The Jenkins job builder will publish test results, giving "Verified +1" if the test succeeds and -1 if the test fails. A contributor or committer can review it with a -1/0/+1. A committer then must approve it with a +2 rating and merge the code to the master branch.
The committer may take any of several actions, such as clicking on the "Reply" button, adding reviewers, adding a review comment, and moving the flags to +2 and +1
Once a committer/contributor approves it, the code can be merged to the master branch.
Create a Git branch or tag
A project committer can create a branch at any time using the Gerrit web site. Similarly a project committer can add a tag to the repository using the Gerrit web site. To reach the administration screen for a repository, start with the "Browse" menu at the top, pick Repositories, find the repository in the list and click on its link. To create a branch, pick "Branches" on the left side, then click the Create New button, as shown in the screen shot below. Similarly to create a tag, from the administration screen for the repository pick "Tags" on the left side, then click the Create New button (screen not shown).
Triggering Jenkins jobs from Gerrit
Jobs can be triggered at Jenkins by posting a comment in the Gerrit system on a review.
Post "recheck" to re-run the verify job. This may be necessary due to intermittent network failures at Jenkins, because a dependency has become available, etc.
Post "remerge" to re-run the merge job. This may be necessary due to intermittent network failures at Jenkins, to recreate a version of an artifact, etc.
Post "run-sonar" to run the Sonar analysis job (if applicable).
Releasing Binary Artifacts
In the LF environment, "to release" means to promote an artifact from a staging area that is ephemeral a release area that is permanent. For example, move a Docker image from a Nexus staging registry where images are deleted after two weeks to a Nexus release registry where images are kept "forever." This process is implemented by Jenkins jobs that react to files committed via Gerrit. Guarding the process with the Gerrit merge task means only committers can release artifacts.
Once testing against the staging repo version has been completed (see above ) and the project has determined that the artifact in the staged repository is ready for release, the project team can use the new-for-2019 self-release process. For details on the JJB templates please see https://docs.releng.linuxfoundation.org/projects/global-jjb/en/latest/jjb/lf-release-jobs.html
Releasing a Java/maven artifact
- Find the Jenkins stage job that created the release candidate. Look among its output logs for the file with the name: staging-repo.txt.gz, it will have a URL like this:
- https://logs.acumos.org/production/vex-yul-acumos-jenkins-1/common-dataservice-maven-dl-stage-master/4/staging-repo.txt.gz
- Create a new release yaml file in the directory "releases/" at the repo root.
- The file name should be anything, but most projects use a pattern like "release-maven.yaml". An example of the content appears below.
- The file content has the project name, the version to release, and the log directory you found above, altho in abbreviated form.
- Create a new change set with just the new file, commit to git locally and submit the change set to Gerrit.
- After the verify job succeeds, a project committer should merge the change set. This will tag the repository with the version string AND release the artifact.
Example release yaml file content:
--- distribution_type: maven version: 1.0.0 project: example-project log_dir: example-project-maven-stage-master/17/ |
After the release merge job runs to completion, the jar files should appear in the Nexus2 release repository.
Releasing a Docker artifact
For a Docker image the release yaml file must list the desired release tag and the existing container tags. Example release yaml file content:
--- distribution_type: container
container_release_tag: 1.0.0
container_pull_registry: nexus.o-ran-sc.org:10003
container_push_registry: nexus.o-ran-sc.org:10002
project: test ref: b95b07641ead78b5082484aa8a82c900f79c9706
containers:
- name: test-backend
version: 1.0.0-20190806T184921Z
- name: test-frontend
version: 1.0.0-20190806T184921Z |
After the release merge job runs to completion, the container images should appear in the Nexus3 release registry.
Releasing a Python package
For a Python package the release yaml file must list the log directory, python version and more. Example release yaml file content:
--- distribution_type: pypi log_dir: ric-plt-lib-rmr-python-pypi-merge-master/1 pypi_project: rmr python_version: '3.7' version: 1.0.0 project: myproject |
If you use a valid decimal value anywhere (like 3.7 above), put it in single quotes so it can be parsed as a string, not a number.
After the release merge job runs to completion, the packages should appear in the https://pypi.org index.
Releasing a PackageCloud DEB/RPM package
This is not yet supported by a self-release process. Write a ticket at https://jira.linuxfoundation.org/servicedesk
Jenkins Job Builder Templates
All jobs in the Jenkins server are generated from Jenkins Job Builder (JJB) templates. The templates are maintained in the "ci-management" git repository. A project template is defined for each project in a YAML file (e.g., "com-log.yaml"). The project templates use job templates defined by the Linux Foundation Global JJB as well as job templates custom to this project.