This is work in progress
Introduction
The helm manager is a service for managing helm charts. The service provides a REST API for onboarding of charts as well as installation and uninstallation of applications based on these charts.
In addition to the helm manager, a chart repository is used to store the available . charts for onboarding and installation.
...
- As docker container on a local machine with kubernbetes
- As kubernetes services/pods service and pod on a local machine with kubernetes or in minikube
- As kubernetes services/pods service and pod on a kubernetes cluster
Prerequisites
The following need to be installed, latest version, on the machine
- Kubernetes or minikube including kubectl
- curlgit
- git (if buidling image)
- maven (if buidling image)
- docker (for builing image build and image registry)
- helm
Preparation
Clone the nonrtric repo from gerrit and change dir to helm-manager. Make sure to use the correct branch when cloning, use "master" until the branch for D Release is available.
All needed scripts for start starting and run running the helm manager, in docker or kubernetes are in this directory.
...
Code Block | ||||
---|---|---|---|---|
| ||||
$ git clone "https://gerrit.o-ran-sc.org/r/nonrtric" -b <branch> $ cd helm-manager |
Build image
There is no need to build the image for the helm manager unless changes are made to the Dockerfile in the current dir.
...
Instruction to build the image, run the command blelow:
Code Block | ||||
---|---|---|---|---|
| ||||
$ mvn clean package |
The built image should be named: o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT. This can be be cah
Code Block | ||||
---|---|---|---|---|
| ||||
$ docker images | grep helm |
Expected output:
Code Block | ||||
---|---|---|---|---|
| ||||
o-ran-sc/nonrtric-helm-manager 1.0.0-SNAPSHOT 56e50ade8c37 3 minutes ago 498MB |
Note: Note, replace Replace the image name in the docker-hm.sh and kubehelm-hm.sh manager if this image shall be used. For the helm-manager.ymal the parameters image and imagePullPolicy shall be configured like this:
Code Block | ||
---|---|---|
| ||
image: o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT
imagePullPolicy: Never |
Note: Locally built Note: Locally built images are not available to a kubernetes cluster unless the image is made available in an image repo accessible from within the cluster.
Create helm chart for test
Create a helm chart for this test. Package the chart into an archive
...
language | bash |
---|---|
theme | Midnight |
...
. Run these two commands. The result should be a file named simple-app-0.1.0.tgz.
Code Block | ||||
---|---|---|---|---|
| ||||
$ helm create simple-app
$ helm package simple-app |
Run in kubernetes
This instruction is valid for running both in a local kubernetes and in a kubernetes ccluster
...
Code Block | ||||
---|---|---|---|---|
| ||||
$ kubectl apply -f kube-cm.yaml |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
apiVersion: v1
kind: Service
metadata:
name: chartrepo
namespace: nonrtric
labels:
run: chartrepo
spec:
type: NodePort
ports:
- port: 8080
targetPort: 8080
protocol: TCP
name: http
selector:
run: chartrepo
---
apiVersion: v1
kind: Pod
metadata:
name: chartrepo
namespace: nonrtric
labels:
run: chartrepo
spec:
securityContext:
runAsUser: 0
containers:
- name: chartrepo
image: ghcr.io/helm/chartmuseum:v0.13.1
imagePullPolicy: Always
ports:
- name: http
containerPort: 8080
env:
- name: DEBUG
value: "1"
- name: STORAGE
value: "local"
- name: STORAGE_LOCAL_ROOTDIR
value: "/var/chartrepo/charts"
- name: DISABLE_API
value: "false"
volumeMounts:
- mountPath: /var/chartrepo/charts
name: chartrepo-pv
volumes:
- name: chartrepo-pv
persistentVolumeClaim:
claimName: chartrepo-pvc
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: chartrepo-pv
annotations:
pv.beta.kubernetes.io/gid: "999"
labels:
run: chartrepo
spec:
storageClassName: chartrepo-standard
capacity:
storage: 10Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
hostPath:
path: "/tmp/chartrepo"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: chartrepo-pvc
namespace: nonrtric
labels:
run: chartrepo
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Mi
storageClassName: chartrepo-standard
volumeMode: Filesystem |
Add the chart, created in the section 'Create helm for test', to the chartmuseum repo. The node port of the chartmuseum service is obtained and env var CM_PORT is assigned that port number.
Code Block | ||||
---|---|---|---|---|
| ||||
$ CM_PORT=$(kubectl get svc chartrepo -n nonrtric -o jsonpath='{...ports[?(@.name=="'http'")].nodePort}')
$ curl --data-binary @simple-app-0.1.0.tgz -X POST http://$KUBE_HOST:$CM_PORT/api/charts
{"saved":true} |
Create a service account for the helm manager. This example service account bind to the "cluster-admin" role which normally has full permissions to the add/change/read/delete any kubernetes object. It is advisable to bind the service account to a ClusterRole with less permissions if desired.
Code Block | ||||
---|---|---|---|---|
| ||||
$ kubectl apply -f helm-manager-sa.yaml
serviceaccount/helm-manager-sa created
clusterrolebinding.rbac.authorization.k8s.io/helm-manager-sa-clusterrolebinding created |
Start the helm manager. Four objects will be created. Note that the service is defined as a NodePort. This enables access from outside the cluster and is also a precondition for the test script to work. Change 'type' to 'ClusterIP' in the 'helmmanagerservice' service definition in the file helm-manager.yaml.
Code Block | ||||
---|---|---|---|---|
| ||||
$ kubectl apply -f helm-manager.yaml
service/helmmanagerservice created
pod/helmmanagerservice created
persistentvolume/helm-manager-service-pv created
persistentvolumeclaim/helm-manager-service-pvc created |
The chartmuseum repo need to added to helm. This operation must be called with a url accessible from the helm manager pod.
Go into the helm manager container and add the repo.
Code Block | ||||
---|---|---|---|---|
| ||||
$ kubectl exec -it helmmanagerservice -n nonrtric -- sh
# helm repo add cm http://chartrepo.nonrtric:8080
"cm" has been added to your repositories
$ exit |
The helm manager is now running and configured with a chart repo.
Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':
- Namespace 'ckhm' is created in kubernetes if not existing
- Onboard chart
- Install chart
- Uninstall chart
- Remove (the onboarded) chart
All operations should report "OK".
Code Block | ||||
---|---|---|---|---|
| ||||
$ ./test.sh kube $KUBE_HOST |
Example output of the script
Add the chart, created in the section 'Create helm for test', to the chartmuseum repo. The node port of the chartmuseum service is obtained and env var CM_PORT is assigned that port number.
Code Block | ||||
---|---|---|---|---|
| ||||
$ CM_PORT=$(kubectl get svc chartrepo -n nonrtric -o jsonpath='{...ports[?(@.name=="'http'")].nodePort}')
$ curl --data-binary @simple-app-0.1.0.tgz -X POST http://$KUBE_HOST:$CM_PORT/api/charts
{"saved":true} |
Create a service account for the helm manager. This example service account bind to the "cluster-admin" role which normally has full permissions to the add/change/read/delete any kubernetes object. It is advisable to bind the service account to a ClusterRole with less permissions if desired.
Code Block | ||||
---|---|---|---|---|
| ||||
$ kubectl apply -f helm-manager-sa.yaml
serviceaccount/helm-manager-sa created
clusterrolebinding.rbac.authorization.k8s.io/helm-manager-sa-clusterrolebinding created |
Start the helm manager. Four objects will be created. Note that the service is defined as a NodePort. This enables access from outside the cluster and is also a precondition for the test script to work. Change 'type' to 'ClusterIP' in the 'helmmanagerservice' service definition in the file helm-manager.yaml.
Code Block | ||||
---|---|---|---|---|
| ||||
$ kubectl apply -f helm-manager.yaml
service/helmmanagerservice created
pod/helmmanagerservice created
persistentvolume/helm-manager-service-pv created
persistentvolumeclaim/helm-manager-service-pvc created |
The chartmuseum repo need to added to helm. This operation must be called with a url accessible from the helm manager pod.
Go into the helm manager container and add the repo.
Code Block | ||||
---|---|---|---|---|
| ||||
$ kubectl exec -it helmmanagerservice -n nonrtric -- sh
# helm repo add cm http://chartrepo.nonrtric:8080
"cm" has been added to your repositories
$ exit |
The helm manager is now running and configured with a chart repo.
Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':
- Namespace 'ckhm' is created in kubernetes if not existing
- Onboard chart
- Install chart
- Uninstall chart
- Remove (the onboarded) chart
All operations should report "OK".
Code Block | ||||
---|---|---|---|---|
| ||||
$ ./test.sh kube $KUBE_HOST |
Example output of the script
Code Block | ||||
---|---|---|---|---|
| ||||
Start test
===== | ||||
Code Block | ||||
| ||||
Start test ================ Get apps - empty ================ curl -sw %{http_code} http://localhost:32743/helm/charts Curl OK Response: 200 Body: {"charts":[]} ============ Onboard app =========== curl -sw %{http_code} http://localhost:32743/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json Curl OK Response: 200 Body: ===================== Get apps - simple-app ===================== curl -sw %{http_code} http://localhost:32743/helm/charts Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} =========== Install appGet apps - empty ================ curl -sw %{http_code} http://localhost:32743/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.jsoncharts Curl OK Response: 201200 Body: Body: {"charts":[]} ===================== Get apps - simple-app == Onboard app =================== curl -sw %{http_code} http://localhost:32743/helm/charts} http://localhost:32743/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} ============================= helmGet lsapps to list installed app ============- simple-app ================= NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION hm-app hm-app-ns 1 2021-05-19 17:15:26.068017436 +0000 UTC deployed hm-app-0.1.0 1.16.0 kall1 default 2 2021-05-16 21:32:46.710349 +0200 CEST deployed app1-0.1.0 1.16.0 kall11 default 1 2021-05-16 21:40:57.073819 +0200 CEST deployed app1-0.1.0 1.16.0 kalle3 default 1 2021-05-17 01:17:03.671637 +0200 CEST deployed app3-0.1.0 1.16.0 kalle3 app3-ns 1 2021-05-17 01:22:29.514917 +0200 CEST deployed app3-0.1.0 1.16.0 ==== curl -sw %{http_code} http://localhost:32743/helm/charts Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} =========== Install app =========== curl -sw %{http_code} http://localhost:32743/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.json Curl OK Response: 201 Body: ===================== Get apps - simple-app ===================== curl -sw %{http_code} http://localhost:32743/helm/charts Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} ============================= helm ls to list installed app ============================= NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION simpleapp ckhm 1 2021-06-01 22:24:12.797624073 +0000 UTC deployed simple-app-0.1.0 1.16.0 ========================================== sleep 30 - give the app some time to start ========================================== ============================ List svc and pod of the app ============================ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE simpleapp-simple-app ClusterIP 10.105.1.129 <none> 80/TCP 30s NAME READY STATUS RESTARTS AGE simpleapp-simple-app-858c798f97-k6vsl 1/1 Running 0 30s ======================== Uninstall app simple-app ======================== curl -sw %{http_code} http://localhost:32743/helm/uninstall/simple-app/0.1.0 -X DELETE Curl OK Response: 204 Body: sleep 10 - give the app some time to remove ============================================= List svc and pod of the app - should be gone ============================================= No resources found in ckhm namespace. NAME READY STATUS RESTARTS AGE simpleapp-simple-app-858c798f97-k6vsl 0/1 Terminating 0 41s ===================== Get apps - simple-app ===================== curl -sw %{http_code} http://localhost:32743/helm/charts Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} ============ Delete chart =========== curl -sw %{http_code} http://localhost:32743/helm/charts/simple-app/0.1.0 -X DELETE Curl OK Response: 204 Body: ================ Get apps - empty ================ curl -sw %{http_code} http://localhost:32743/helm/charts Curl OK Response: 200 Body: {"charts":[]} Test result All tests ok End of test |
...
Code Block | ||||
---|---|---|---|---|
| ||||
$ kubectl delete -f kubehelm-hmmanager.yaml service "helmmanagerservice" deleted pod "helmmanagerservice" deleted persistentvolume "helm-manager-service-pv" deleted persistentvolumeclaim "helm-manager-service-pvc" deleted $ kubectl delete -f kube-cm.yaml service "chartrepo" deleted pod "chartrepo" deleted persistentvolume "chartrepo-pv" deleted persistentvolumeclaim "chartrepo-pvc" deleted $ kubectl delete -f helm-manager-sa.yaml serviceaccount "helm-manager-sa" deleted clusterrolebinding.rbac.authorization.k8s.io "helm-manager-sa-clusterrolebinding" deleted |
Run in docker with local kubernetes
The helm-manger is possible to run as a docker container. However, a local kubernetes must be running where the application can be installed.
...
Start the helm manager in a separate window. Make note of the port, 8112, which will be available on local host. This is the port to the Helm manager REST API.
...
language | bash |
---|---|
theme | Midnight |
.
...
Code Block | |||||||||
---|---|---|---|---|---|---|---|---|---|
| |||||||||
$ ./docker-hm.sh |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
docker rundocker run \ --rm \ -it \ -p 8112:8083 \ --name helmmanagerservice \ --networkrm nonrtric-docker-net \ -v $(pwd)/mnt/database:/var/helm-manager/database \ -v ~/.kube:/root/.kubeit \ -v ~/.helm:/root/.helmp 8112:8083 \ -v ~/.config/helm:/root/.config/helm-name helmmanagerservice \ -v ~/.cache/helm:/root/.cache/helm-network nonrtric-docker-net \ -v $(pwd)/configmnt/KubernetesParticipantConfig.jsondatabase:/optvar/app/helm-manager/src/main/resources/config/KubernetesParticipantConfig.jsondatabase \ -v ~/.kube:/root/.kube \ -v $(pwd)/config/application.yaml~/.helm:/optroot/app/helm-manager/src/main/resources/config/application.yaml.helm \ -v ~/.config/helm:/root/.config/helm \ o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT |
The chartmusem repo need to added to helm. This operation must be called with a url accessible from the helm manager container.
Go into the helm manager container and add the repo.
Code Block | ||||
---|---|---|---|---|
| ||||
$ docker exec -it helmmanagerservice sh
# helm repo add cm http://chartmuseum:8080
"cm" has been added to your repositories
$ exit |
The helm manager is now running and configured with a chart repo.
Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':
- Namespace 'ckhm' is created in kubernetes if not existing
- Onboard chart
- Install chart
- Uninstall chart
- Remove (the onboarded) chart
...
-v ~/.cache/helm:/root/.cache/helm \
-v $(pwd)/config/KubernetesParticipantConfig.json:/opt/app/helm-manager/src/main/resources/config/KubernetesParticipantConfig.json \
-v $(pwd)/config/application.yaml:/opt/app/helm-manager/src/main/resources/config/application.yaml \
nexus3.o-ran-sc.org:10004/o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT |
The chartmusem repo need to added to helm. This operation must be called with a url accessible from the helm manager container.
Go into the helm manager container and add the repo.
Code Block | ||||
---|---|---|---|---|
| ||||
$ ./test.sh docker docker exec -it helmmanagerservice sh # helm repo add cm http://chartmuseum:8080 "cm" has been added to your repositories $ exit |
The helm manager is now running and configured with a chart repo.
Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':
- Namespace 'ckhm' is created in kubernetes if not existing
- Onboard chart
- Install chart
- Uninstall chart
- Remove (the onboarded) chart
All operations should report "OK".
Code Block | ||||
---|---|---|---|---|
| ||||
$ ./test.sh docker |
Expected oputput:
Code Block | ||||
---|---|---|---|---|
| ||||
Start test
| ||||
Code Block | ||||
| ||||
Start test ================ Get apps - empty ================ curl -sw %{http_code} http://localhost:8112/helm/charts Curl OK Response: 200 Body: {"charts":[]} ============ Onboard app =========== curl -sw %{http_code} http://localhost:8112/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json Curl OK Response: 200 Body: ===================== Get apps - simple-appempty ===================== curl -sw %{http_code} http://localhost:8112/helm/charts Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} ============ InstallOnboard app =========== curl -sw %{http_code} http://localhost:8112/helm/installcharts -X POST -H Content-Type:application/json -d F chart=@simple-app-0.1.0.tgz -F values=@simple-app-installationvalues.yaml -F info=<simple-app.json Curl OK Response: 201200 Body: ===================== Get apps - simple-app ===================== curl -sw %{http_code} http://localhost:8112/helm/charts Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} ============================= helm ls to list installed app ============================= NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION hm-app hm-app-ns 1 2021-05-19 17:15:26.068017436 +0000 UTC deployed hm-app-0.1.0 1.16.0 kall1 default 2 2021-05-16 21:32:46.710349 +0200 CEST deployed app1-0.1.0 1.16.0 kall11 default 1 2021-05-16 21:40:57.073819 +0200 CEST deployed app1-0.1.0 1.16.0 kalle3 app3-ns 1 2021-05-17 01:22:29.514917 +0200 CEST deployed app3-0.1.0 1.16.0 kalle3 default 1 2021-05-17 01:17:03.671637 +0200 CEST deployed app3-0.1.0 1.16.0 == Install app =========== curl -sw %{http_code} http://localhost:8112/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.json Curl OK Response: 201 Body: ===================== Get apps - simple-app ===================== curl -sw %{http_code} http://localhost:8112/helm/charts Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} ============================= helm ls to list installed app ============================= NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSIO simpleapp ckhm 1 2021-06-01 16:31:30.255849815 +0000 UTC deployed simple-app-0.1.0 1.16.0 ========================================== sleep 30 - give the app some time to start ========================================== ============================ List svc and pod of the app ============================ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE simpleapp-simple-app ClusterIP 10.102.209.44 <none> 80/TCP 31s NAME READY STATUS RESTARTS AGE simpleapp-simple-app-858c798f97-vm6hd 1/1 Running 0 31s ======================== Uninstall app simple-app ======================== curl -sw %{http_code} http://localhost:8112/helm/uninstall/simple-app/0.1.0 -X DELETE Curl OK Response: 204 Body: sleep 10 - give the app some time to remove ============================================= List svc and pod of the app - should be gone ============================================= No resources found in ckhm namespace. No resources found in ckhm namespace. ===================== Get apps - simple-app ===================== curl -sw %{http_code} http://localhost:8112/helm/charts Curl OK Response: 200 Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]} ============ Delete chart =========== curl -sw %{http_code} http://localhost:8112/helm/charts/simple-app/0.1.0 -X DELETE Curl OK Response: 204 Body: ================ Get apps - empty ================ curl -sw %{http_code} http://localhost:8112/helm/charts Curl OK Response: 200 Body: {"charts":[]} Test result All tests ok End of test |
...