Introduction
The helm manager is a service for managing helm charts. The service provides a REST API for onboarding of charts as well as installation and uninstallation of applications based on these charts.
...
The Helm manager (and chart repo) can be executed in one of the following deployments:
- As docker container on a local machine with kubernbetesAs kubernetes service and pod on a local machine with kubernetes or in minikube
- As kubernetes service and pod on a kubernetes cluster
- As docker container on a local machine with kubernbetes (see Draft - Release E - Run)
Prerequisites
The following need to be installed, latest version, on the machine
- Kubernetes or minikube including kubectl
- curl
- git (if buidling image)
- maven (if buidling image)
- git (for downloading of the repo
- docker ( docker (for image build and image registry)
- helm
Preparation
Clone the nonrtric repo from gerrit and change dir to helm-manager. Make sure to use the correct branch when cloning, use "master" until the branch for D Release is available.
...
If the Helm Manager shall be installed in a kubernetes cluster the actions below shall be made on a node in cluster. FIXME: branch name
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ git clone "https://gerrit.o-ran-sc.org/r/nonrtric" -b <branch>
$ cd helm-manager |
Build image
There is no need to build the image for the helm manager unless changes are made to the Dockerfile in the current dir.
If no changes are needed, skip this section.
Instruction to build the image, run the command blelow:
Create helm chart for test
Create a helm chart for this test. Package the chart into an archive. Run these two commands. The result should be a file named simple-app-0.1.0.tgz.
Code Block |
---|
|
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ mvnhelm clean package |
The built image should be named: o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT. This can be be cah
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ docker images | grep helm
|
create simple-app
$ helm package simple-app |
Run in kubernetes
This instruction is valid for running both in a local kubernetes or in a kubernetes ccluster
If running in a local kubernetes, set the env KUBE_HOST to the host of the kubernetes control planeExpected output:
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
o-ran-sc/nonrtric-helm-manager 1.0.0-SNAPSHOT 56e50ade8c37 3 minutes ago 498MB
|
$ kubectl cluster-info
Kubernetes control plane is running at https://kubernetes.docker.internal:6443
$ KUBE_HOST="kubernetes.docker.internal" |
If running in a kubernetes cluster or in minikube, set the env KUBE_HOST to the ip of the kubernetes control plane.Note: Replace the image name in the docker-hm.sh and helm-manager if this image shall be used. For the helm-manager.ymal the parameters image and imagePullPolicy shall be configured like this:
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
title | Example |
---|
|
$ kubectl cluster-info
Kubernetes master is running image: o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT
imagePullPolicy: Never |
Note: Locally built images are not available to a kubernetes cluster unless the image is made available in an image repo accessible from within the cluster.
Create helm chart for test
...
at https://10.2.0.103:6443
$ KUBE_HOST=10.2.0.103 |
Check if the nonrtric names space exists. If not, create the namespace
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl get ns nonrtric
$ kubectl create ns nonrtric |
Start the chartmuseum service and pod
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl apply -f kube-cm.yaml |
Add the chart, created in the section 'Create helm for test', to the chartmuseum repo. The node port of the chartmuseum service is obtained and env var CM_PORT is assigned that port number.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ helm create simple-app
$ helm package simple-app |
Run in kubernetes
This instruction is valid for running both in a local kubernetes and in a kubernetes ccluster
If running in a local kubernetes, set the env KUBE_HOST to the host of the kubernetes control plane
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl cluster-info
Kubernetes control plane is running at https://kubernetes.docker.internal:6443
$ KUBE_HOST="kubernetes.docker.internal" |
...
CM_PORT=$(kubectl get svc chartrepo -n nonrtric -o jsonpath='{...ports[?(@.name=="'http'")].nodePort}')
$ curl --data-binary @simple-app-0.1.0.tgz -X POST http://$KUBE_HOST:$CM_PORT/api/charts
{"saved":true} |
Create a service account for the helm manager. This example service account bind to the "cluster-admin" role which normally has full permissions to the add/change/read/delete any kubernetes object. It is advisable to bind the service account to a ClusterRole with less permissions if desired.
Code Block |
---|
language | bash |
---|
theme | Midnight | title | Example |
---|
|
$ kubectl cluster-info
Kubernetes master is running at https://10.2.0.103:6443
$ KUBE_HOST=10.2.0.103apply -f helm-manager-sa.yaml
serviceaccount/helm-manager-sa created
clusterrolebinding.rbac.authorization.k8s.io/helm-manager-sa-clusterrolebinding created |
Start the helm manager. Four objects will be created. Note that the service is defined as a NodePort. This enables access from outside the cluster and is also a precondition for the test script to work. Change 'type' to 'ClusterIP' in the 'helmmanagerservice' service definition in the file helm-manager.yaml. Check if the nonrtric names space exists. If not, create the namespace
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl getapply ns nonrtric
$ kubectl create ns nonrtric |
...
-f helm-manager.yaml
service/helmmanagerservice created
pod/helmmanagerservice created
persistentvolume/helm-manager-service-pv created
persistentvolumeclaim/helm-manager-service-pvc created |
The chartmuseum repo need to added to helm. This operation must be called with a url accessible from the helm manager pod.
Go into the helm manager container and add the repo.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl applyexec -fit helmmanagerservice kube-cm.yaml |
Add the chart, created in the section 'Create helm for test', to the chartmuseum repo. The node port of the chartmuseum service is obtained and env var CM_PORT is assigned that port number.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ CM_PORT=$(kubectl get svc chartrepo -n nonrtric -o jsonpath='{...ports[?(@.name=="'http'")].nodePort}')
$ curl --data-binary @simple-app-0.1.0.tgz -X POST http://$KUBE_HOST:$CM_PORT/api/charts
{"saved":true} |
Create a service account for the helm manager. This example service account bind to the "cluster-admin" role which normally has full permissions to the add/change/read/delete any kubernetes object. It is advisable to bind the service account to a ClusterRole with less permissions if desired.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl apply -f helm-manager-sa.yaml
serviceaccount/helm-manager-sa created
clusterrolebinding.rbac.authorization.k8s.io/helm-manager-sa-clusterrolebinding created |
Start the helm manager. Four objects will be created. Note that the service is defined as a NodePort. This enables access from outside the cluster and is also a precondition for the test script to work. Change 'type' to 'ClusterIP' in the 'helmmanagerservice' service definition in the file helm-manager.yaml.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl apply -f helm-manager.yaml
service/helmmanagerservice created
pod/helmmanagerservice created
persistentvolume/helm-manager-service-pv created
persistentvolumeclaim/helm-manager-service-pvc created |
The chartmuseum repo need to added to helm. This operation must be called with a url accessible from the helm manager pod.
Go into the helm manager container and add the repo.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl exec -it helmmanagerservice -n nonrtric -- sh
# helm repo add cm http://chartrepo.nonrtric:8080
"cm" has been added to your repositories
$ exit |
The helm manager is now running and configured with a chart repo.
Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':
- Namespace 'ckhm' is created in kubernetes if not existing
- Onboard chart
- Install chart
- Uninstall chart
- Remove (the onboarded) chart
All operations should report "OK".
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ ./test.sh kube $KUBE_HOST |
Example output of the script
-n nonrtric -- sh
# helm repo add cm http://chartrepo.nonrtric:8080
"cm" has been added to your repositories
$ exit |
The helm manager is now running and configured with a chart repo.
Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':
- Namespace 'ckhm' is created in kubernetes if not existing
- Onboard chart
- Install chart
- Uninstall chart
- Remove (the onboarded) chart
All operations should report "OK".
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ ./test.sh kube $KUBE_HOST |
Example output of the script
Code Block |
---|
|
Start test
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:32743/helm/charts
Curl OK
Response: 200
Body: {"charts":[]}
============
Onboard app
===========
curl -sw %{http_code} http://localhost:32743/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json
Curl OK
Response: 200
Body:
=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
Curl OK
Response: 200
Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
===========
Install app |
Code Block |
---|
|
Start test
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:32743/helm/charts
install Curl-X OK
POST -H Content-Type:application/json -d @simple-app-installation.json
Curl OK
Response: 200201
Body: {"charts":[]}
============
Onboard =========
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
-X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json
Curl OK
Curl OK
Response: 200
Body:
============={"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
========
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
Curl OK
Response: 200
Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
helm ls to list installed app
=============================
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
simpleapp ckhm 1 2021-06-01 22:24:12.797624073 +0000 UTC deployed simple-app-0.1.0 1.16.0
===============================
Install app
===========
curlsleep 30 -sw %{http_code} http://localhost:32743/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.json
Curl OK
Response: 201
Body:
give the app some time to start
=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
Curl OK
Response: 200
Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
============================
List svc and pod of the app
=============================
helm ls to list installed app
=============================
NAME
NAME TYPE NAMESPACE REVISION UPDATED CLUSTER-IP EXTERNAL-IP PORT(S) AGE
simpleapp-simple-app ClusterIP 10.105.1.129 <none> 80/TCP STATUS CHART 30s
NAME APP VERSION
simpleapp ckhm 1 READY STATUS 2021-06-01 22:24:12.797624073 +0000 UTC deployed RESTARTS AGE
simpleapp-simple-app-0.1.0 1.16.0-858c798f97-k6vsl 1/1
==========================================
sleep 30 - give the app some time to startRunning 0 30s
=========================
Uninstall app simple-app
=================
============================
List svc and pod of the app
==
curl -sw %{http_code} http://localhost:32743/helm/uninstall/simple-app/0.1.0 -X DELETE
Curl OK
Response: 204
Body:
sleep 10 - give the app some time to remove
============================
NAME TYPE=================
List svc and pod of the app - should be gone
=============================================
No resources found in ckhm namespace.
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
simpleapp-simple-app ClusterIP 10.105.1.129 <none> READY 80/TCPSTATUS 30s
NAME READY STATUS RESTARTS AGE
simpleapp-simple-app-858c798f97-k6vsl 10/1 RunningTerminating 0 30s41s
========================
Uninstall app
Get apps - simple-app
========================
curl -sw %{http_code} http://localhost:32743/helm/uninstall/simple-app/0.1.0 -X DELETEcharts
Curl OK
Response: 204200
Body:
sleep 10 - give the app some time to remove
=============================================
List svc and pod of the app - should be gone
=============================================
No resources found in ckhm namespace.
NAME READY STATUS RESTARTS AGE
simpleapp-simple-app-858c798f97-k6vsl 0/1 Terminating 0 41s
=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
Curl OK
Response: 200
Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
============
Delete chart
===========
curl -sw %{http_code} http://localhost:32743/helm/charts/simple-app/0.1.0 -X DELETE
Curl OK
Response: 204
Body:
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:32743/helm/charts
Curl OK
Response: 200
Body: {"charts":[]}
Test result All tests ok
End of test |
Cleanup of all created kubernetes object
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl delete -f helm-manager.yaml
service "helmmanagerservice" deleted
pod "helmmanagerservice" deleted
persistentvolume "helm-manager-service-pv" deleted
persistentvolumeclaim "helm-manager-service-pvc" deleted
$ kubectl delete -f kube-cm.yaml
service "chartrepo" deleted
pod "chartrepo" deleted
persistentvolume "chartrepo-pv" deleted
persistentvolumeclaim "chartrepo-pvc" deleted
$ kubectl delete -f helm-manager-sa.yaml
serviceaccount "helm-manager-sa" deleted
clusterrolebinding.rbac.authorization.k8s.io "helm-manager-sa-clusterrolebinding" deleted |
Run in docker with local kubernetes
The helm-manger is possible to run as a docker container. However, a local kubernetes must be running where the application can be installed.
Create a private docker network, unless it already exists, for the containers to run in.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ docker network create nonrtric-docker-net |
Start the chartmuseum container in a separate window. This will be used as a chart repository. Make note of the port, 8222, which will be available on local host. Charts uploaded to the chartmuseum container will be availed in the mounted dir 'charts' on your host.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ ./docker-cm.sh |
Code Block |
---|
language | bash |
---|
title | docker-cm.sh |
---|
linenumbers | true |
---|
collapse | true |
---|
|
docker run \
--rm \
-it \
-p 8222:8080 \
--name chartmuseum \
--network nonrtric-docker-net \
-e DEBUG=1 \
-e STORAGE=local \
-e STORAGE_LOCAL_ROOTDIR=/charts \
-v $(pwd)/charts:/charts \
ghcr.io/helm/chartmuseum:v0.13.1 |
Add the chart, created in the section 'Create helm for test', to the repo
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ curl --data-binary "@simple-app-0.1.0.tgz" -X POST http://localhost:8222/api/charts
{"saved":true} |
Start the helm manager in a separate window. Make note of the port, 8112, which will be available on local host. This is the port to the Helm manager REST API.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ ./docker-hm.sh |
Code Block |
---|
language | bash |
---|
title | docker-hm.sh |
---|
collapse | true |
---|
|
docker run \
--rm \
-it \
-p 8112:8083 \
--name helmmanagerservice \
--network nonrtric-docker-net \
-v $(pwd)/mnt/database:/var/helm-manager/database \
-v ~/.kube:/root/.kube \
-v ~/.helm:/root/.helm \
-v ~/.config/helm:/root/.config/helm \
-v ~/.cache/helm:/root/.cache/helm \
-v $(pwd)/config/KubernetesParticipantConfig.json:/opt/app/helm-manager/src/main/resources/config/KubernetesParticipantConfig.json \
-v $(pwd)/config/application.yaml:/opt/app/helm-manager/src/main/resources/config/application.yaml \
nexus3.o-ran-sc.org:10004/o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT |
The chartmusem repo need to added to helm. This operation must be called with a url accessible from the helm manager container.
Go into the helm manager container and add the repo.
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ docker exec -it helmmanagerservice sh
# helm repo add cm http://chartmuseum:8080
"cm" has been added to your repositories
$ exit |
The helm manager is now running and configured with a chart repo.
Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':
- Namespace 'ckhm' is created in kubernetes if not existing
- Onboard chart
- Install chart
- Uninstall chart
- Remove (the onboarded) chart
All operations should report "OK".
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ ./test.sh docker |
Expected oputput:
Code Block |
---|
|
Start test
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:8112/helm/charts
Curl OK
Response: 200
Body: {"charts":[]}
============
Onboard app
===========
curl -sw %{http_code} http://localhost:8112/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json
Curl OK
Response: 200
Body:
=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
Curl OK
Response: 200
Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
===========
Install app
===========
curl -sw %{http_code} http://localhost:8112/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.json
Curl OK
Response: 201
Body:
=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
Curl OK
Response: 200
Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
=============================
helm ls to list installed app
=============================
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSIO
simpleapp ckhm 1 2021-06-01 16:31:30.255849815 +0000 UTC deployed simple-app-0.1.0 1.16.0
==========================================
sleep 30 - give the app some time to start
==========================================
============================
List svc and pod of the app
============================
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
simpleapp-simple-app ClusterIP 10.102.209.44 <none> 80/TCP 31s
NAME READY STATUS RESTARTS AGE
simpleapp-simple-app-858c798f97-vm6hd 1/1 Running 0 31s
========================
Uninstall app simple-app
========================
curl -sw %{http_code} http://localhost:8112/helm/uninstall/simple-app/0.1.0 -X DELETE
Curl OK
Response: 204
Body:
sleep 10 - give the app some time to remove
=============================={"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
===============
List svc and pod of the app - should be goneDelete chart
=============================================
No resources found in ckhm namespace.
No resources found in ckhm namespace.
curl -sw %{http_code} http://localhost:32743/helm/charts/simple-app/0.1.0 -X DELETE
Curl OK
Response: 204
Body:
=====================
Get apps - simple-app
=====empty
================
curl -sw %{http_code} http://localhost:811232743/helm/charts
Curl OK
Response: 200
Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}
============
Delete chart
===========
curl -sw %{http_code} http://localhost:8112/helm/charts/simple-app/0.1.0 -X DELETE
Curl OK
Response: 204
Body:
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:8112/helm/charts
Curl OK
Response: 200
Body: {"charts":[]}
Test result All tests ok
End of test]}
Test result All tests ok
End of test |
Cleanup of all created kubernetes object
Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
|
$ kubectl delete -f helm-manager.yaml
service "helmmanagerservice" deleted
pod "helmmanagerservice" deleted
persistentvolume "helm-manager-service-pv" deleted
persistentvolumeclaim "helm-manager-service-pvc" deleted
$ kubectl delete -f kube-cm.yaml
service "chartrepo" deleted
pod "chartrepo" deleted
persistentvolume "chartrepo-pv" deleted
persistentvolumeclaim "chartrepo-pvc" deleted
$ kubectl delete -f helm-manager-sa.yaml
serviceaccount "helm-manager-sa" deleted
clusterrolebinding.rbac.authorization.k8s.io "helm-manager-sa-clusterrolebinding" deleted |