This is a guide for software developers and testers about the continuous integration (CI) resources and processes used by the project and supported by the Linux Foundation.
Most of these resources authenticate users via a Linux Foundation identity.
|https://identity.linuxfoundation.org||The Linux Foundation identity management web site.|
|https://gerrit.o-ran-sc.org||The Gerrit code review server hosts the Git repositories and supports the review and merge workflow. The review process basically requires all users to follow something like a git pull-request process, but restricts the publication (push) of private branches.|
|https://jenkins.o-ran-sc.org||Jenkins is the continuous-integration server aka the build server. All users can see the contents of the Jenkins; no users can directly modify the configuration nor start a job in this Jenkins. Jobs are configured by Jenkins Job Builder (JJB) files in the ci-management gerrit repository.|
|https://jenkins.o-ran-sc.org/sandbox||The Jenkins sandbox is a place for testing Jenkins job configurations. Users must request login credentials on the sandbox, it does not use the LF identity or single sign-on feature. After receiving access, users can create jobs, reconfigure jobs, trigger builds etc. New JJB templates should be tested in the sandbox before submitting for review and use on the main Jenkins server.|
|https://jira.o-ran-sc.org||Jira is the web site that supports agile development with epics, user stories, and especially issues (bug reports).|
Not heavily used by this project because Java is not a key technology, this Nexus 2 repository holds Maven (jar and pom) artifacts produced by builds. Snapshot and staging builds are all deployed to this repository. Release artifacts are created by promoting artifacts manually from the staging repository after suitable approval. Publicly accessible to users without credentials. All users should be able to access and browse artifacts through this URL.
This Nexus 3 server stores project Docker images and also mirrors external registries. Supports the following:
These registries are open for anonymous read access. The Jenkins server has credentials to push images to the snapshot and staging registries, and to promote images to the release registry. Manual push of images is not supported.
This site hosts binary artifacts such as Debian (.deb) packages, Red Hat Package Manager (.rpm) packages and more in these repositories:
|https://sonarcloud.io||Sonar tools analyze Java and Python code for quality issues and code coverage achieved by unit tests (JUnit, tox) and publish results to this cloud-based server.|
This section guides the developer in submitting code, reviewing code, tracking the status of builds and requesting creation of release versions. These recommended practices come from the Linux Foundation.
When working with a git repository cloned from Gerrit you can create as many local branches as you like. None of the branches will be pushed to the remote Gerrit server, the branches remain forever private. Creating branches for each task will allow you to work on multiple independent tasks in parallel, let you recover gracefully from various situations, and generally save aggravation and time. Also see these instructions on tagging and branching for releases:
Tagging and Branching Steps Process
The Linux Foundation strongly recommends (and eventually may enforce) Gerrit commit message content. A git commit message must meet these requirements:
An example is shown below:
Null check for ClientResponse NullPointerException might be thrown as cres is nullable here Issue-Id: PROJECT-999 Change-Id: I14dc792fb67198ebcbabfe80d90c48389af6cc91 Signed-off-by: First Last <email@example.com>
The project uses Gerrit to automate the process of reviewing all code changes before they are committed to the Git repository. For a tutorial on git you might start here:
Also see this guide published by the LF Release Engineering team about using Gerrit:
The command-line tool "git review" is the most reliable and can be used on any platform. As of this writing, version 1.28.0 or later is required; version 1.26.0 will not work. Windows users should install "Git Bash" to gain support for command-line git. The most common way of installing is to use pip:
pip install git-review
Gerrit is designed to host text files – source code. It enforces a size threshold on every commit, the default limit is 5MB. Further the Linux Foundation prohibits committing binary files such as compiled executables, jar files, zip archives and so on. An exception is made for binary picture (image) files in GIF, JPG and PNG formats, but the size limit must still be honored.
The quickstart guides below describe the command-line procedures for performing common tasks on Gerrit.
git checkout -b my-local-branch
git review other-branch
git checkout my-local-branch
# Look up the change number shown top-left in Gerrit web, here "999".
git checkout master
Submitted reviews with changes should appear in the Gerrit web interface. The Jenkins job builder will publish test results, giving "Verified +1" if the test succeeds and -1 if the test fails. A contributor or committer can review it with a -1/0/+1. A committer then must approve it with a +2 rating and merge the code to the master branch.
The committer may take any of several actions, such as clicking on the "Reply" button, adding reviewers, adding a review comment, and moving the flags to +2 and +1
Once a committer/contributor approves it, the code can be merged to the master branch.
https://nexus3.o-ran-sc.org/#browse/browse:docker.staging (make sure not to log in. Otherwise the link does not work). Go to correct image in the tree (note that the latest o-ran-sc components are below the top-level folder "o-ran-sc" and the same component under the same name directly on top level is some outdated version not relevant anymore). and then open the subtree "tags", e.g., o-ran-sc → ric-plt-submgr → tags → ...). There should be at least one version under the tag subtree (e.g. this link for the near-RT RIC subscription manager image: https://nexus3.o-ran-sc.org/#browse/browse:docker.staging:v2%2Fo-ran-sc%2Fric-plt-submgr%2Ftags). If not, then there's no staging image in the staging repo for this component. Alternatively use this curl command: "curl -X GET https://nexus3.o-ran-sc.org:10004/v2/o-ran-sc/ric-plt-submgr/tags/list" (replace the part in bold with the correct component obtained via "curl -X GET https://nexus3.o-ran-sc.org:10004/v2/_catalog" (not sure if this shows components that do not currently have a tag)).
If there's no tag version in the staging repository, we will need to re-run a "merge job" as per "remerge" in the previous section "Triggering Jenkins jobs from Gerrit". This was done e.g. in this review https://gerrit.o-ran-sc.org/r/c/ric-plt/submgr/+/4526 for the subscription manager.
Example release yaml file content:
After the release merge job runs to completion, the jar files should appear in the Nexus2 release repository.
For a Docker image the release yaml file must list the desired release tag and the existing container tags. Example release yaml file content:
distribution_type: container container_release_tag: 1.0.0 container_pull_registry: nexus.o-ran-sc.org:10003 container_push_registry: nexus.o-ran-sc.org:10002 project: test
After the release merge job runs to completion, the container images should appear in the Nexus3 release registry.
For a Python package the release yaml file must list the log directory, python version and more. Example release yaml file content:
If you use a valid decimal value anywhere (like 3.7 above), put it in single quotes so it can be parsed as a string, not a number.
After the release merge job runs to completion, the packages should appear in the https://pypi.org index.
2020-Dec-14: There is a process that involves the keyword "stage-release" (see section "Triggering Jenkins jobs from Gerrit" above) to publish packages to packagecloud. This works with two merges. First one with updates to ci/package-tag.yaml and ci/control. Merge that change and after merging add a comment with only the keyword "stage-release". After this create a second review for an updated "releases/*.yaml" with correct version, log_dir and ref with commit hash. Submit that change for merging as well. This will do the actual moving of the package from the staging directory in packagecloud to the release directory. A bit of information on this is also available in section "PackageCloud Release Files" from this LF guide: link.
OLD notes: The self-release process for PackageCloud is in active development as of December 2019. Until it is ready, write a ticket at https://jira.linuxfoundation.org/servicedesk
Every sonar analysis job consists of these steps:
build-wrapper-dump.json, which can be consumed by the Sonar scanner.
Examples from the O-RAN-SC project RMR:
More details about configuration, building for scanning, and specific file and/or directory exclusion in C/C++ repositories is provided on the Configure Sonar for C/C++ page.
go-acc $(go list ./... | grep -vE '(/tests|/enums)' )
Examples from the O-RAN-SC project Alarm:
target/jacoco.exec, which can be consumed by the Sonar scanner.
Examples from the O-RAN-SC project Dashboard:
Execution of the build via tox (really just tests) should create the output file coverage.xml, which can be consumed by the Sonar scanner.
Examples from the O-RAN-SC project A1:
[testenv:clm] # use pip to report dependencies with versions whitelist_externals = sh commands = sh -c 'pip freeze > requirements.txt'
Then the Jenkins job needs to be updated, see the following commit for an example: https://gerrit.o-ran-sc.org/r/c/ci-management/+/2446.
Now Sonar issues should show up in your code.
Note! At the moment there is a bug that show up if Lombok is used in the code with a version below 1.18.12. If you have this problem, download Lombok version 1.18.12 or higher and repeat the installation procedure described here, https://howtodoinjava.com/automation/lombok-eclipse-installation-examples/.
All jobs in the Jenkins server are generated from Jenkins Job Builder (JJB) templates. The templates are maintained in this project's ci-management git repository. The templates use features from the Linux Foundation Global JJB as well as features custom to this project.
After creating or modifying a YAML file, submit the change to Gerrit where it will be verified and can be reviewed. Only the LF Release Engineering team can merge changes to the ci-management repository.
The argument is a simple shell-style globbing pattern to limit the scope. This example should create all jobs that start with the prefix "your-job-name".
[job_builder] ignore_cache=True keep_descriptions=False recursive=True update=jobs [jenkins] query_plugins_info=False url=https://jenkins.o-ran-sc.org/sandbox user=your-sandbox-user-name password=your-sandbox-api-token
After pushing a job to the sandbox, either via the Jenkins `deploy-jjb` job or directly, you can run the job on the code in your repository, usually the master branch, to test the job. Follow this procedure:
You can also test a job on a Gerrit change set that has not yet been merged. Follow this procedure:
Jenkins build minions are OpenStack virtual machines. The software, the number of cores, amount of virtual memory and amount of disk memory are determined by files in directory
These jobs are NOT triggered automatically if a change is made to a supporting file such as an Ansible role definition. Regular project members cannot launch these jobs manually, only the LF #RelEng team has the privilege. One way to avoid writing a ticket is to find a previously merged commit to a file in the packer area and post the the usual Jenkins comment "remerge" on it, but this is not foolproof.
Upon completion of a job the job page shows the image name that was built, a long string starting "ZZCI" and ending with a timestamp, see the screenshot below. Copy that string (very carefully!) into the appropriate config file, submit as a gerrit change set, and wait for merge. Then the new image will be available for use with the specified build node label.