Versions Compared

Old Version 12

changes.mady.by.user Björn Magnusson

Saved on

compared with

New Version 13

changes.mady.by.user Björn Magnusson

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In addition to the helm manager, a chart repository is used to store the available . charts for onboarding and installation.

...

  • As docker container on a local machine with kubernbetes
  • As kubernetes services/pods service and pod on a local machine with kubernetes or in minikube
  • As kubernetes services/pods service and pod on a kubernetes cluster

Prerequisites

...

  • Kubernetes or minikube including kubectl
  • curlgit
  • git (if buidling image)
  • maven (if buidling image)
  • docker (for builing image build and image registry) 
  • helm

Preparation

Clone the nonrtric repo from gerrit and change dir to helm-manager. Make sure to use the correct branch when cloning, use "master" until the branch for D Release is available.

All needed scripts for start starting and run running the helm manager, in docker or kubernetes are in this directory.

...

Instruction to build the image, run the command blelow:

Code Block
languagebash
themeMidnight
$ mvn clean package

The  built image should be named: o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT. This can be be cah

Code Block
languagebash
themeMidnight
$ docker images | grep helm

Expected output:

Code Block
languagebash
themeMidnight
o-ran-sc/nonrtric-helm-manager  1.0.0-SNAPSHOT  56e50ade8c37  3 minutes ago   498MB

Note: Note, replace Replace the image name in the  docker-hm.sh and kubehelm-hm.sh  manager  if this image shall be used. For the helm-manager.ymal the parameters image and imagePullPolicy shall be configured like this:

Code Block
languagebash
    image: o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT
    imagePullPolicy: Never

Note: Locally built images are not available to a kubernetes cluster unless the image Note: Locally built images are not available to a kubernetes cluster unless the image is made available in an image repo accessible from within the cluster.

Create helm chart for test

Create a helm chart for this test. Package the chart into an archive. Run these two commands. The result should be a file named simple-app-0.1.0.tgz.

Code Block
language
Code Block
languagebash
themeMidnight
$ helm create simple-app
Successfully packaged chart and saved it to: /nonrtric/helm-manager/simple-app-0.1.0.tgz

$ helm package simple-app

...

Code Block
languagebash
themeMidnight
$ kubectl apply -f kube-cm.yaml
Code Block
languagebash
titlekube-cm.yaml
collapsetrue
apiVersion: v1
kind: Service
metadata:
  name: chartrepo
  namespace: nonrtric
  labels:
    run: chartrepo
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    run: chartrepo

---

apiVersion: v1
kind: Pod
metadata:
  name: chartrepo
  namespace: nonrtric
  labels:
    run: chartrepo
spec:
  securityContext:
    runAsUser: 0
  containers:
  - name: chartrepo
    image: ghcr.io/helm/chartmuseum:v0.13.1
    imagePullPolicy: Always
    ports:
    - name: http
      containerPort: 8080
    env:
    - name: DEBUG
      value: "1"
    - name: STORAGE
      value: "local"
    - name: STORAGE_LOCAL_ROOTDIR
      value: "/var/chartrepo/charts"
    - name: DISABLE_API
      value: "false"
    volumeMounts:
    - mountPath: /var/chartrepo/charts
      name: chartrepo-pv
  volumes:
  - name: chartrepo-pv
    persistentVolumeClaim:
      claimName: chartrepo-pvc


---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: chartrepo-pv
  annotations:
    pv.beta.kubernetes.io/gid: "999"
  labels:
    run: chartrepo
spec:
  storageClassName: chartrepo-standard
  capacity:
    storage: 10Mi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  hostPath:
    path: "/tmp/chartrepo"

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: chartrepo-pvc
  namespace: nonrtric
  labels:
    run: chartrepo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Mi
  storageClassName: chartrepo-standard
  volumeMode: Filesystem

Add the chart, created in the section 'Create helm for test', to the chartmuseum repo. The node port of the chartmuseum service is obtained and env var CM_PORT is assigned that port number.

Code Block
languagebash
themeMidnight
$ CM_PORT=$(kubectl get svc chartrepo -n nonrtric -o jsonpath='{...ports[?(@.name=="'http'")].nodePort}')
$ curl --data-binary @simple-app-0.1.0.tgz -X POST http://$KUBE_HOST:$CM_PORT/api/charts
{"saved":true}

Create a service account for the helm manager. This example service account bind to the "cluster-admin" role which normally has full permissions to the add/change/read/delete any kubernetes object. It is advisable to bind the service account to a ClusterRole with less permissions if desired.

Code Block
languagebash
themeMidnight
$ kubectl apply -f helm-manager-sa.yaml
serviceaccount/helm-manager-sa created
clusterrolebinding.rbac.authorization.k8s.io/helm-manager-sa-clusterrolebinding created

Start the helm manager. Four objects will be created. Note that the service is defined as a NodePort. This enables access from outside the cluster and is also a precondition for the test script to work. Change 'type' to 'ClusterIP' in the 'helmmanagerservice' service definition in the file helm-manager.yaml. 

Code Block
languagebash
themeMidnight
$ kubectl apply -f helm-manager.yaml
service/helmmanagerservice created
pod/helmmanagerservice created
persistentvolume/helm-manager-service-pv created
persistentvolumeclaim/helm-manager-service-pvc created

The chartmuseum repo need to added to helm. This operation must be called with a url accessible from the helm manager pod.

Go into the helm manager container and add the repo.

Code Block
languagebash
themeMidnight
$ kubectl exec -it helmmanagerservice -n nonrtric -- sh
# helm repo add cm http://chartrepo.nonrtric:8080
"cm" has been added to your repositories
$ exit

The helm manager is now running and configured with a chart repo.

Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':

  • Namespace 'ckhm' is created in kubernetes if not existing
  • Onboard chart
  • Install chart
  • Uninstall chart
  • Remove (the onboarded) chart

All operations should report "OK".

Code Block
languagebash
themeMidnight
$ ./test.sh kube $KUBE_HOST

Example output of the script


Add the chart, created in the section 'Create helm for test', to the chartmuseum repo. The node port of the chartmuseum service is obtained and env var CM_PORT is assigned that port number.

Code Block
languagebash
themeMidnight
$ CM_PORT=$(kubectl get svc chartrepo -n nonrtric -o jsonpath='{...ports[?(@.name=="'http'")].nodePort}')
$ curl --data-binary @simple-app-0.1.0.tgz -X POST http://$KUBE_HOST:$CM_PORT/api/charts
{"saved":true}


Create a service account for the helm manager. This example service account bind to the "cluster-admin" role which normally has full permissions to the add/change/read/delete any kubernetes object. It is advisable to bind the service account to a ClusterRole with less permissions if desired.

Code Block
languagebash
themeMidnight
$ kubectl apply -f helm-manager-sa.yaml
serviceaccount/helm-manager-sa created
clusterrolebinding.rbac.authorization.k8s.io/helm-manager-sa-clusterrolebinding created


Start the helm manager. Four objects will be created. Note that the service is defined as a NodePort. This enables access from outside the cluster and is also a precondition for the test script to work. Change 'type' to 'ClusterIP' in the 'helmmanagerservice' service definition in the file helm-manager.yaml. 

Code Block
languagebash
themeMidnight
$ kubectl apply -f helm-manager.yaml
service/helmmanagerservice created
pod/helmmanagerservice created
persistentvolume/helm-manager-service-pv created
persistentvolumeclaim/helm-manager-service-pvc created

The chartmuseum repo need to added to helm. This operation must be called with a url accessible from the helm manager pod.

Go into the helm manager container and add the repo.

Code Block
languagebash
themeMidnight
$ kubectl exec -it helmmanagerservice -n nonrtric -- sh
# helm repo add cm http://chartrepo.nonrtric:8080
"cm" has been added to your repositories
$ exit

The helm manager is now running and configured with a chart repo.

Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':

  • Namespace 'ckhm' is created in kubernetes if not existing
  • Onboard chart
  • Install chart
  • Uninstall chart
  • Remove (the onboarded) chart

All operations should report "OK".


Code Block
languagebash
themeMidnight
$ ./test.sh kube $KUBE_HOST

Example output of the script

Code Block
languagebash
collapsetrue
Start test
=====
Code Block
languagebash
collapsetrue
Start test
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[]}

============
Onboard app
===========
curl -sw %{http_code} http://localhost:32743/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json
 Curl OK
  Response: 200
  Body: 

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

===========
Install appGet apps - empty
================
curl -sw %{http_code} http://localhost:32743/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.jsoncharts
 Curl OK
  Response: 201200
   Body: Body: {"charts":[]}

=====================
Get apps - simple-app
==
Onboard app
===================
curl -sw %{http_code} http://localhost:32743/helm/charts} http://localhost:32743/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

=============================
helmGet lsapps to list installed app
=============- simple-app
================
NAME     	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART           	APP VERSION
hm-app   	hm-app-ns	1       	2021-05-19 17:15:26.068017436 +0000 UTC	deployed	hm-app-0.1.0    	1.16.0     
kall1    	default  	2       	2021-05-16 21:32:46.710349 +0200 CEST  	deployed	app1-0.1.0      	1.16.0     
kall11   	default  	1       	2021-05-16 21:40:57.073819 +0200 CEST  	deployed	app1-0.1.0      	1.16.0     
kalle3   	default  	1       	2021-05-17 01:17:03.671637 +0200 CEST  	deployed	app3-0.1.0      	1.16.0     
kalle3   	app3-ns  	1       	2021-05-17 01:22:29.514917 +0200 CEST  	deployed	app3-0.1.0      	1.16.0     =====
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

===========
Install app
===========
curl -sw %{http_code} http://localhost:32743/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.json
 Curl OK
  Response: 201
  Body: 

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

=============================
helm ls to list installed app
=============================
NAME     	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART           	APP VERSION 
simpleapp	ckhm     	1       	2021-06-01 22:24:12.797624073 +0000 UTC	deployed	simple-app-0.1.0	1.16.0     

==========================================
sleep 30 - give the app some time to start
==========================================
============================
List svc and  pod of the app
============================
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
simpleapp-simple-app   ClusterIP   10.105.1.129   <none>        80/TCP    30s
NAME                                    READY   STATUS    RESTARTS   AGE
simpleapp-simple-app-858c798f97-k6vsl   1/1     Running   0          30s

========================
Uninstall app simple-app
========================
curl -sw %{http_code} http://localhost:32743/helm/uninstall/simple-app/0.1.0 -X DELETE
 Curl OK
  Response: 204
  Body: 

sleep 10 - give the app some time to remove
=============================================
List svc and  pod of the app - should be gone
=============================================
No resources found in ckhm namespace.
NAME                                    READY   STATUS        RESTARTS   AGE
simpleapp-simple-app-858c798f97-k6vsl   0/1     Terminating   0          41s

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

============
Delete chart
===========
curl -sw %{http_code} http://localhost:32743/helm/charts/simple-app/0.1.0 -X DELETE
 Curl OK
  Response: 204
  Body: 

================
Get apps - empty
================
curl -sw %{http_code} http://localhost:32743/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[]}

Test result  All tests ok 
End of test

...

Start the helm manager in a separate window. Make note of the port, 8112, which will be available on local host. This is the port to the Helm manager REST API.

...

languagebash
themeMidnight

.

...

Code Block
languagebash
titledocker-hm.sh
collapsetrue
themeMidnight
$ ./docker-hm.sh


Code Block
languagebash
titledocker-hm.sh
collapsetrue
docker rundocker run \
    --rm  \
    -it \
    -p 8112:8083  \
    --name helmmanagerservice \
    --networkrm nonrtric-docker-net \
    -v $(pwd)/mnt/database:/var/helm-manager/database \
    -v ~/.kube:/root/.kube  -it \
    -v ~/.helm:/root/.helmp 8112:8083  \
    --v ~/.config/helm:/root/.config/helmname helmmanagerservice \
    -v ~/.cache/helm:/root/.cache/helm-network nonrtric-docker-net \
    -v $(pwd)/configmnt/KubernetesParticipantConfig.jsondatabase:/opt/appvar/helm-manager/src/main/resources/config/KubernetesParticipantConfig.json/database \
    -v ~/.kube:/root/.kube \
    -v $(pwd)/config/application.yaml~/.helm:/optroot/app/helm-manager/src/main/resources/config/application.yaml.helm \
    -v ~/.config/helm:/root/.config/helm \
    o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT

The chartmusem repo need to added to helm. This operation must be called with a url accessible from the helm manager container.

Go into the helm manager container and add the repo.

Code Block
languagebash
themeMidnight
$ docker exec -it helmmanagerservice sh
# helm repo add cm http://chartmuseum:8080
"cm" has been added to your repositories
$ exit

The helm manager is now running and configured with a chart repo.

Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':

  • Namespace 'ckhm' is created in kubernetes if not existing
  • Onboard chart
  • Install chart
  • Uninstall chart
  • Remove (the onboarded) chart

...

-v ~/.cache/helm:/root/.cache/helm \
    -v $(pwd)/config/KubernetesParticipantConfig.json:/opt/app/helm-manager/src/main/resources/config/KubernetesParticipantConfig.json \
    -v $(pwd)/config/application.yaml:/opt/app/helm-manager/src/main/resources/config/application.yaml \
    o-ran-sc/nonrtric-helm-manager:1.0.0-SNAPSHOT

The chartmusem repo need to added to helm. This operation must be called with a url accessible from the helm manager container.

Go into the helm manager container and add the repo.

Code Block
languagebash
themeMidnight
$ ./test.sh docker docker exec -it helmmanagerservice sh
# helm repo add cm http://chartmuseum:8080
"cm" has been added to your repositories
$ exit


The helm manager is now running and configured with a chart repo.

Run the script test.sh to execute the sequence for installing the application 'simpleapp' namespace 'ckhm':

  • Namespace 'ckhm' is created in kubernetes if not existing
  • Onboard chart
  • Install chart
  • Uninstall chart
  • Remove (the onboarded) chart

All operations should report "OK".

Code Block
languagebash
themeMidnight
$ ./test.sh docker

Expected oputput:

Code Block
languagebash
collapsetrue
Start test
Code Block
languagebash
collapsetrue
Start test
================
Get apps - empty
================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[]}

============
Onboard app
===========
curl -sw %{http_code} http://localhost:8112/helm/charts -X POST -F chart=@simple-app-0.1.0.tgz -F values=@simple-app-values.yaml -F info=<simple-app.json
 Curl OK
  Response: 200
  Body: 

=====================
Get apps - simple-appempty
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

============
InstallOnboard app
===========
curl -sw %{http_code} http://localhost:8112/helm/installcharts -X POST -H Content-Type:application/json -d F chart=@simple-app-0.1.0.tgz -F values=@simple-app-installationvalues.yaml -F info=<simple-app.json
 Curl OK
  Response: 201200
  Body: 

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

=============================
helm ls to list installed app
=============================
NAME     	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART           	APP VERSION
hm-app   	hm-app-ns	1       	2021-05-19 17:15:26.068017436 +0000 UTC	deployed	hm-app-0.1.0    	1.16.0     
kall1    	default  	2       	2021-05-16 21:32:46.710349 +0200 CEST  	deployed	app1-0.1.0      	1.16.0     
kall11   	default  	1       	2021-05-16 21:40:57.073819 +0200 CEST  	deployed	app1-0.1.0      	1.16.0     
kalle3   	app3-ns  	1       	2021-05-17 01:22:29.514917 +0200 CEST  	deployed	app3-0.1.0      	1.16.0     
kalle3   	default  	1       	2021-05-17 01:17:03.671637 +0200 CEST  	deployed	app3-0.1.0      	1.16.0    ==
Install app
===========
curl -sw %{http_code} http://localhost:8112/helm/install -X POST -H Content-Type:application/json -d @simple-app-installation.json
 Curl OK
  Response: 201
  Body: 

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

=============================
helm ls to list installed app
=============================
NAME     	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART           	APP VERSIO 
simpleapp	ckhm     	1       	2021-06-01 16:31:30.255849815 +0000 UTC	deployed	simple-app-0.1.0	1.16.0     

==========================================
sleep 30 - give the app some time to start
==========================================
============================
List svc and  pod of the app
============================
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
simpleapp-simple-app   ClusterIP   10.102.209.44   <none>        80/TCP    31s
NAME                                    READY   STATUS    RESTARTS   AGE
simpleapp-simple-app-858c798f97-vm6hd   1/1     Running   0          31s

========================
Uninstall app simple-app
========================
curl -sw %{http_code} http://localhost:8112/helm/uninstall/simple-app/0.1.0 -X DELETE
 Curl OK
  Response: 204
  Body: 

sleep 10 - give the app some time to remove
=============================================
List svc and  pod of the app - should be gone
=============================================
No resources found in ckhm namespace.
No resources found in ckhm namespace.

=====================
Get apps - simple-app
=====================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[{"releaseName":"simpleapp","chartName":"simple-app","version":"0.1.0","namespace":"ckhm","repository":"cm"}]}

============
Delete chart
===========
curl -sw %{http_code} http://localhost:8112/helm/charts/simple-app/0.1.0 -X DELETE
 Curl OK
  Response: 204
  Body: 

================
Get apps - empty
================
curl -sw %{http_code} http://localhost:8112/helm/charts
 Curl OK
  Response: 200
  Body: {"charts":[]}

Test result  All tests ok 
End of test

...