DRAFT - Solution:

Problem - no local/common image repository is available when running in a k8s multi node cluster or when running in single/multi node cluster using other container runtime than docker (e.g. containerd)

This causes problem for images, that need to be build locally, when running tests but also creates an uncertainty when running on snapshot or staging images. Potentially, two pods that are supposed to use the image may download two different snapshot images if for example the the images is updated in nexus between the first pod download and the second pod download. For locally built images - these are simply not available to kube in multi node cluster or in clusters using other runtimes than docker.

The easy solution is to use a docker hub repo as a common image repo. Images needed for test are retagged with the docker hub prefix and then pushed to the docker hub repo. Pods are then configured to pull images form the docker hub.

The nonrtric function test env can perform this operation automatically (a docker hub repo need to be created in beforehand and the user need to logged in to docker in the shell where the test  script shall be executed).

Use the flags when running a testscript :

--repo-policy   - set to 'local' or 'remote' to control if only local images shall be pushed or if both local and remote images shall be pushed - the 'remote' setting is preferred 

--image-repo   - set to the name of the docker hub repo


Original problem description:

=====================

Ref:

When new images are built for testing they are only available in the node where they are built dynamically by the test script, so how to make the newly built image available on all nodes in the cluster. 

(Only for running test in a K8s environment with multiple worker nodes)

  • Alternative 1: Create a custom private docker registry for the newly built images, and deploy it along with Non-RT-RIC, then they are available for all nodes (not pushed to main nexus):
    • docker registry image: https://hub.docker.com/_/registry
    • Local means that there is clash with other test that might also be using
    • By default requires TLS - and needs a CA signed certificate(commonly trusted by all clients). But this needs a domain name - which isn't available.
    • Possible to configure self-signed certificates. But this needs extra configuration on all cluster nodes to trust this self-signed certificates, which may not be allowed in some cloud platform.
      • Can be controlled with a Pull Policy? (Don't think so .... TBC)
  • Alternative 2: Use Docker-Hub
    • (Need to create an account & log in to docker hub before running test. Then images will then be pushed to that private docker hub, and its URL will be used while test is running)) 
    • Only locally built images are pushed to Docker-Hub - otherwise use nexus. Currently we have three simulator images(simulations for: Message Router/Callback Receiver/Producer Stub) that are build on the fly when running the test scripts.
    • Possible race condition when generating snapshot versions - but previous version may be picked up if nexus version (same tag) is newer than local docker hub - but this is unlikely (Need to be aware of it though)
    • Released images will always come from nexus - this is ONLY for testing staging & snapshot images.


... But everyone who tests in a k8s multi-node cluster (where a fresh build is part of the test) must also hit this issue ... Surely there is a known solution? Possible something simple we missed?

When running test in a single node k8s cluster, above problem does not occur.

Recording:





  • No labels