Page History

This is work in progress

Introduction

The helm manager is a service for managing helm charts. The service provides a REST API for onboarding of charts as well as installation and uninstallation of applications based on these charts.

In addition to the helm manager, a chart repository is used to store the available . charts for onboarding and installation.

...

As docker container on a local machine with kubernbetes
As kubernetes services/pods service and pod on a local machine with kubernetes or in minikube
As kubernetes services/pods service and pod on a kubernetes cluster

Preparation

Clone the nonrtric repo from gerrit and change dir. Make sure to use the correct branch when cloning.

All needed scripts for start and run the helm manager, in docker or kubernetes are in this directory.

Prerequisites

The following need to be installed, latest version, on the machine

Kubernetes or minikube including kubectl
curl
git (if buidling image)
maven (if buidling image)
docker (for image build and image registry)
helm

Preparation

Clone the nonrtric repo from gerrit and change dir to helm-manager. Make sure to use the correct branch when cloning, use "master" until the branch for D Release is available.

All needed scripts for starting and running the helm manager, in docker or kubernetes are in this directory.

If the Helm Manager If the Helm Manager shall be installed in a kubernetes cluster the actions below shall be made on a node in cluster.

Code Block

language	bash
theme	Midnight

$ git clone "https://gerrit.o-ran-sc.org/r/nonrtric" -b <branch>

$ cd helm-manager

Build image

There is no need to build the image for the helm manager unless changes are made to the Dockerfile in the current dir.

If no changes are needed, skip this section.

Instruction to build the image, run the command blelow:

Code Block

language	bash
theme	Midnight

$ mvn clean package

The built image should be named: o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT. This can be be cah

Code Block

language	bash
theme	Midnight

$ docker images | grep helm

Expected output:

Code Block

language	bash
theme	Midnight

o-ran-sc/nonrtric-helm-manager  1.0.0-SNAPSHOT  56e50ade8c37  3 minutes ago                1.0.0-SNAPSHOT  498MB

Note: Note, replace Replace the image name in the docker-hm.sh and kubehelm-hm.sh manager if this image shall be used. For the helm-manager.ymal the parameters image and imagePullPolicy shall be configured like this:

Code Block

language	bash

    image: o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT
    imagePullPolicy: Never

Note: Note: Locally built images are not available to a kubernetes cluster unless the image is made available in an image repo accessible from within the cluster.

Create helm chart for test

Create a helm chart for this test. Package the chart into an archive. Run these two commands. The result should be a file named simple-app-0.1.0.tgz.

Code Blockcode

language	bash
theme	Midnight

$ helm create simple-app

$ helm package simple-app

Run in

...

kubernetes

The helm-manger is possible to run as a docker container. However, a local kubernetes must be running where the application can be installed.

This instruction is valid for running both in a local kubernetes and in a kubernetes ccluster

If running in a local kubernetes, set the env KUBE_HOST to the host of the kubernetes control planeCreate a private docker network for the containers to run in

Code Block

language	bash
theme	Midnight

$ docker network create nonrtric-docker-netkubectl cluster-info
Kubernetes control plane is running at https://kubernetes.docker.internal:6443
$ KUBE_HOST="kubernetes.docker.internal"

If running in a kubernetes cluster or in minikube, set the env KUBE_HOST to the ip of the kubernetes control plane.Start the chartmuseum container. Make note of the port, 8222, which will be available on local host. Charts uploaded to the chartmuseum container will be availed in the dir 'charts' on your host.ls

Code Block

language	bash

$ ./docker-cm.sh

Code Block

language	bash
title	docker-cm.sh
linenumbers	true
collapse	true

docker run --rm -it \
    -p 8222:8080 \
    --name chartmuseum \
    --network nonrtric-docker-net \
    -e DEBUG=1 \
    -e STORAGE=local \
    -e STORAGE_LOCAL_ROOTDIR=/charts \
    -v $(pwd)/charts:/charts \
    ghcr.io/helm/chartmuseum:v0.13.1

Add the chart, created in the section 'Create helm for test', to the repo

theme	Midnight
title	Example

$ kubectl cluster-info
Kubernetes master is running at https://10.2.0.103:6443
$ KUBE_HOST=10.2.0.103

Check if the nonrtric names space exists. If not, create the namespace

Code Block

language	bash
theme	Midnight

$ kubectl get ns nonrtric
$ kubectl create ns nonrtric

Start the chartmuseum service and pod

Code Block

language	bash
theme	Midnight

$ kubectl apply -f kube-cm.yaml

Add the chart, created in the section 'Create helm for test', to the chartmuseum repo. The node port of the chartmuseum service is obtained and env var CM_PORT is assigned that port number.

Code Block

language	bash
theme	Midnight

$ CM_PORT=$(kubectl get svc chartrepo -n nonrtric -o jsonpath='{...ports[?(@.name=="'http'")].nodePort}')

Code Block

language	bash

$ curl --data-binary "@simple-app-0.1.0.tgz" -X POST http://localhost:8222$KUBE_HOST:$CM_PORT/api/charts
{"saved":true}

Create a service account for Start the helm manager. Make note of the port, 8112, which will be available on local host. This is the port to the Helm manager REST API.

...

language	bash

...

This example service account bind to the "cluster-admin" role which normally has full permissions to the add/change/read/delete any kubernetes object. It is advisable to bind the service account to a ClusterRole with less permissions if desired.

Code Block

language	bash
title	docker-hm.sh
collapse	true

docker run \
    --rm  \
    -it \
    -p 8112:8083  \
    --name helmmanagerservice \
    --network nonrtric-docker-net \
    -v $(pwd)/mnt/database:/var/helm-manager/database \
    -v ~/.kube:/root/.kube \
    -v ~/.helm:/root/.helm \
    -v ~/.config/helm:/root/.config/helm \
    -v ~/.cache/helm:/root/.cache/helm \
    -v $(pwd)/config/KubernetesParticipantConfig.json:/opt/app/helm-manager/src/main/resources/config/KubernetesParticipantConfig.json \
    -v $(pwd)/config/application.yaml:/opt/app/helm-manager/src/main/resources/config/application.yaml \
    o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT

theme	Midnight

$ kubectl apply -f helm-manager-sa.yaml
serviceaccount/helm-manager-sa created
clusterrolebinding.rbac.authorization.k8s.io/helm-manager-sa-clusterrolebinding created

Start the helm manager. Four objects will be created. Note that the service is defined as a NodePort. This enables access from outside the cluster and is also a precondition for the test script to work. Change 'type' to 'ClusterIP' in the 'helmmanagerservice' service definition in the file helm-manager.yaml.

Code Block

language	bash
theme	Midnight

$ kubectl apply -f helm-manager.yaml
service/helmmanagerservice created
pod/helmmanagerservice created
persistentvolume/helm-manager-service-pv created
persistentvolumeclaim/helm-manager-service-pvc created

The chartmuseum The chartmusem repo need to added to helm. This operation must be called with an a url accessable accessible from the containerhelm manager pod.

Go into the helm manager container and add the reprepo.

Code Block

language	bash
theme	Midnight

$ dockerkubectl exec -it helmmanagerservice sh
$ -n nonrtric -- sh
# helm repo add cm http://chartmuseumchartrepo.nonrtric:8080
"cm" has been added to your repositories
$ exit

The helm manager is now running and configured with a chart repo.

Run the script test.sh to execute a the sequence for installing the application 'simpleapp' namespace 'ckhm':

Namespace 'ckhm' is created in kubernetes if not existing
Onboard chart
Install chart
Uninstall chart
Remove (the onboarded) chart

All operations should report "OK".

Code Block

language	bash
theme	Midnight

$ ./test.sh docker

Run in kubernetes

This instruction is valid for running both in a local kubernetes and in a kubernetes ccluster

Check if the nonrtric names space exists

kubectl get ns nonrtric

If the namespace does not exist, create it.

kubectl create ns nonrtric

Start the chartmuseum service and pod

kubectl apply ~~f cm/svc~~ app.yaml

Find the nodeport of the service

kubectl get svc chartrepo -n nonrtric -o jsonpath='{...ports[?(@.name=="'http'")].nodePort}'

Add the chart to the repo

curl --data-binary "@simple-app-0.1.0.tgz" -X POST http://localhost:<port-to-chartrepo>/api/charts

If running in kubernetes cluster

Add the service account for the helm manager. Note that this uses the 'cluster-admin' role which has wide permission. This should only be used for test.

kubectrl apply ~~f helm-manager-sa.yam~~

Edit the svc-app.yaml, set 'service-acccount-name' to helm-manager-sa

Start the helm manager

kubectl apply ~~f svc~~ app.yaml

Go into the helm manger pod and add the chartrepo

kubectl exec -it helmmanagerservice ~~n nonrtric -~~ sh

cmd prompt: helm repo add cm http://chartrepo.nonrtric:8080

cmd prompt: exit

Run the test script.

This script will onboard chart, install an application based on the chart, unstall the application and finally remove the chart.

...

kube $KUBE_HOST

Example output of the script

Code Block

language	bash
collapse	true

Start test
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[]}

============
Onboard app
===========
curl -sw %{http_code} http://localhost:32743/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json
 Curl OK
  Response: 200
  Body: 

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

===========
Install app
===========
curl -sw %{http_code} http://localhost:32743/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.json
 Curl OK
  Response: 201
  Body: 

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

=============================
helm ls to list installed app
=============================
NAME     	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART           	APP VERSION 
simpleapp	ckhm     	1       	2021-06-01 22:24:12.797624073 +0000 UTC	deployed	simple-app-0.1.0	1.16.0     

==========================================
sleep 30 - give the app some time to start
==========================================
============================
List svc and  pod of the app
============================
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
simpleapp-simple-app   ClusterIP   10.105.1.129   <none>        80/TCP    30s
NAME                                    READY   STATUS    RESTARTS   AGE
simpleapp-simple-app-858c798f97-k6vsl   1/1     Running   0          30s

========================
Uninstall app simple-app
========================
curl -sw %{http_code} http://localhost:32743/helm/uninstall/simple-app/0.1.0 -X DELETE
 Curl OK
  Response: 204
  Body: 

sleep 10 - give the app some time to remove
=============================================
List svc and  pod of the app - should be gone
=============================================
No resources found in ckhm namespace.
NAME                                    READY   STATUS        RESTARTS   AGE
simpleapp-simple-app-858c798f97-k6vsl   0/1     Terminating   0          41s

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

============
Delete chart
===========
curl -sw %{http_code} http://localhost:32743/helm/charts/simple-app/0.1.0 -X DELETE
 Curl OK
  Response: 204
  Body: 

================
Get apps - empty
================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[]}

Test result  All tests ok 
End of test

Cleanup of all created kubernetes object

Code Block

language	bash
theme	Midnight

$ kubectl delete -f helm-manager.yaml
service "helmmanagerservice" deleted
pod "helmmanagerservice" deleted
persistentvolume "helm-manager-service-pv" deleted
persistentvolumeclaim "helm-manager-service-pvc" deleted

$ kubectl delete -f kube-cm.yaml
service "chartrepo" deleted
pod "chartrepo" deleted
persistentvolume "chartrepo-pv" deleted
persistentvolumeclaim "chartrepo-pvc" deleted

$ kubectl delete -f helm-manager-sa.yaml
serviceaccount "helm-manager-sa" deleted
clusterrolebinding.rbac.authorization.k8s.io "helm-manager-sa-clusterrolebinding" deleted

Run in docker with local kubernetes

The helm-manger is possible to run as a docker container. However, a local kubernetes must be running where the application can be installed.

Create a private docker network, unless it already exists, for the containers to run in.

Code Block

language	bash
theme	Midnight

$ docker network create nonrtric-docker-net

Start the chartmuseum container in a separate window. This will be used as a chart repository. Make note of the port, 8222, which will be available on local host. Charts uploaded to the chartmuseum container will be availed in the mounted dir 'charts' on your host.

Code Block

language	bash
theme	Midnight

$ ./docker-cm.sh

Code Block

language	bash
title	docker-cm.sh
linenumbers	true
collapse	true

docker run \
	--rm \
	-it \
    -p 8222:8080 \
    --name chartmuseum \
    --network nonrtric-docker-net \
    -e DEBUG=1 \
    -e STORAGE=local \
    -e STORAGE_LOCAL_ROOTDIR=/charts \
    -v $(pwd)/charts:/charts \
    ghcr.io/helm/chartmuseum:v0.13.1

Add the chart, created in the section 'Create helm for test', to the repo

Code Block

language	bash
theme	Midnight

$ curl --data-binary "@simple-app-0.1.0.tgz" -X POST http://localhost:8222/api/charts
{"saved":true}

Start the helm manager in a separate window. Make note of the port, 8112, which will be available on local host. This is the port to the Helm manager REST API.

Code Block

language	bash
theme	Midnight

$ ./docker-hm.sh

Code Block

language	bash
title	docker-hm.sh
collapse	true

docker run \
    --rm  \
    -it \
    -p 8112:8083  \
    --name helmmanagerservice \
    --network nonrtric-docker-net \
    -v $(pwd)/mnt/database:/var/helm-manager/database \
    -v ~/.kube:/root/.kube \
    -v ~/.helm:/root/.helm \
    -v ~/.config/helm:/root/.config/helm \
    -v ~/.cache/helm:/root/.cache/helm \
    -v $(pwd)/config/KubernetesParticipantConfig.json:/opt/app/helm-manager/src/main/resources/config/KubernetesParticipantConfig.json \
    -v $(pwd)/config/application.yaml:/opt/app/helm-manager/src/main/resources/config/application.yaml \
    nexus3.o-ran-sc.org:10004/o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT

The chartmusem repo need to added to helm. This operation must be called with a url accessible from the helm manager container.

Go into the helm manager container and add the repo.

Code Block

language	bash
theme	Midnight

$ docker exec -it helmmanagerservice sh
# helm repo add cm http://chartmuseum:8080
"cm" has been added to your repositories
$ exit

The helm manager is now running and configured with a chart repo.

Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':

Namespace 'ckhm' is created in kubernetes if not existing
Onboard chart
Install chart
Uninstall chart
Remove (the onboarded) chart

All operations should report "OK".

Code Block

language	bash
theme	Midnight

$ ./test.sh docker

Expected oputput:

Code Block

language	bash
collapse	true

Start test
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[]}

============
Onboard app
===========
curl -sw %{http_code} http://localhost:8112/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json
 Curl OK
  Response: 200
  Body: 

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

===========
Install app
===========
curl -sw %{http_code} http://localhost:8112/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.json
 Curl OK
  Response: 201
  Body: 

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

=============================
helm ls to list installed app
=============================
NAME     	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART           	APP VERSIO 
simpleapp	ckhm     	1       	2021-06-01 16:31:30.255849815 +0000 UTC	deployed	simple-app-0.1.0	1.16.0     

==========================================
sleep 30 - give the app some time to start
==========================================
============================
List svc and  pod of the app
============================
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
simpleapp-simple-app   ClusterIP   10.102.209.44   <none>        80/TCP    31s
NAME                                    READY   STATUS    RESTARTS   AGE
simpleapp-simple-app-858c798f97-vm6hd   1/1     Running   0          31s

========================
Uninstall app simple-app
========================
curl -sw %{http_code} http://localhost:8112/helm/uninstall/simple-app/0.1.0 -X DELETE
 Curl OK
  Response: 204
  Body: 

sleep 10 - give the app some time to remove
=============================================
List svc and  pod of the app - should be gone
=============================================
No resources found in ckhm namespace.
No resources found in ckhm namespace.

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

============
Delete chart
===========
curl -sw %{http_code} http://localhost:8112/helm/charts/simple-app/0.1.0 -X DELETE
 Curl OK
  Response: 204
  Body: 

================
Get apps - empty
================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[]}

Test result  All tests ok 
End of test

Show all results

Space shortcuts

Page tree

Versions Compared

Old Version 3

New Version Current

Key

Introduction

Preparation

Prerequisites

Preparation

Build image

Create helm chart for test

Run in

kubernetes

Run in kubernetes

Run in docker with local kubernetes