Introduction:

Anomaly Detection (AD), QoE Predictor (QP) and Traffic Steering xApp's are three components in Anomaly Detection use case in Near RT RIC platform. AD xApp aims to detect anomalous UE's present in network. It communicates with Traffic Steering xApp via RMR and send Anomalous UE's information to TS. TS request QP xApp to predict throughput on given anomalous UE list. QP xApp use Machine learning algorithm to predict throughput for a UE in given cell and send that to TS. TS xApp collaborate with AD and QP to gain knowledge on current and neighboring cells and performs a handover to a cell having maximum throughput(if needed).


Features:

AD:

  • AD xApp is a component in near RIC platform that communicates with Traffic Steering xApp
  • It create a connection to InfluxDB and fetch continuously latest stream of UE data. Also perform write data method for predicted outcomes into InfluxDB 
  • Use machine learning algorithm to detect anomalous users and find out the degradation type
  • Send anomalous UE's along with degradation type to Traffic Steering xApp via RMR

QP:

  • It creates a connection to InfluxDB to fetch and write UE and Cell metrics
  • QP process a UE ID and fetch latest relevant UE and cell metrics from InfluxDB as and when it gets a prediction request from TS
  • Use machine learning algorithm to predict throughput in neighboring cells for a given anomalous users
  • Send these predictions to TS xApp

TS:

  • TS xApp expect anomalous users list from AD as and when AD detects
  • Send prediction request to QP for these anomalous users 
  • Find cell to handover(if needed) and perform the handover


Architecture:


Prerequisites:

Near-RT RIC platform setup is required. Link for Near-RT RIC installation.

InfluxDB component needs to be deployed (part of Near-RT RIC installation)

Traffic Steering (TS) xApp needs to be build, deployed and onboarded on the Near RT RIC platform 

AD xApp needs to be build, deployed and onboarded on the Near RT RIC platform

QP xApp needs to be build, deployed and onboarded on the Near RT RIC platform


InfluxDB :

    **Onetime setup for InfluxDB**

    Once Kubernetes setup is done, we have to create PersistentVolume through the storage class for the InfluxDB database. Mentioned below is one time process that  should be followed before deploying the InfluxDB in ricplt namespace.

    # User has to check the following namespace exist or not using
    % kubectl get ns ricinfra

    # If the namespace doesn’t exist, then create it using: 
    % kubectl create ns ricinfra

   % helm install stable/nfs-server-provisioner --namespace ricinfra --name nfs-release-1
   % kubectl patch storageclass nfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
   % sudo apt install nfs-common


Onboarding and Deployment of AD, QP and TS xApp:

a) xApp's can be cloned from Gerrit using the following command:

      AD : git clone "https://gerrit.o-ran-sc.org/r/ric-app/ad"

      TS : git clone "https://gerrit.o-ran-sc.org/r/ric-app/ts"

      QP : git clone "https://gerrit.o-ran-sc.org/r/ric-app/qp

build the images locally after cloning it.

OR  
b) build and push the docker image  from nexus repository  :

     AD:

            docker build -t nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ad:0.0.2 .
            docker push  nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ad:0.0.2

    TS:
            docker build -t nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ts:1.0.1 .
            docker push  nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ts:1.0.1

   QP:

           docker build -t nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-qp:0.0.3 .
           docker push  nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-qp:0.0.3


Onboarding xApp's:

Step:1

Each of these xApp's have a descriptor in their Gerrit repo under the xapp-descriptor/ directory. None of them have xapp specific controls and therefore no individual json schema.
Here are the URLs which can be included in HTTP POST call to onboard tool.

AD : https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ad.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD
TS : https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD

QP : https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/qp.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD 


Step:2
Here we are preparing for API calls into the xApp On-Boarder by providing the locations of the xApp descriptors.

AD : echo '{"config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ad.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD" }' > onboard.ad.url 
TS : echo '{"config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD" }' > onboard.ts.url

QP : echo '{"config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/qp.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD" }' > onboard.qp.url 


Step:3
Invoke the API calls into the xApp On-boarder, providing it the locations of the xApp descriptors.

AD : curl --location --request POST "http://$(hostname):32080/onboard/api/v1/onboard/download" --header 'Content-Type: application/json' --data-binary "@./onboard.ad.url"
TS : curl --location --request POST "http://$(hostname):32080/onboard/api/v1/onboard/download" --header 'Content-Type: application/json' --data-binary "@./onboard.ts.url"

QP : curl --location --request POST "http://$(hostname):32080/onboard/api/v1/onboard/download" --header 'Content-Type: application/json' --data-binary "@./onboard.qp.url"


Checking the on-boarded charts:
curl --location --request GET "http://$(hostname):32080/onboard/api/v1/charts"


Deploying xApp:
Deploy the xApp's by invoking the xApp Manager's API. Once receiving the deploy API call, the xApp Manager will make API call into Helm/Kubernetes to deploy the xApp's Helm chart.
The Routing Manager is also involved if the xApp needs to process RMR messages, – it will complete the routes and send out route updates.

AD : curl --location --request POST "http://$(hostname):32080/appmgr/ric/v1/xapps" --header 'Content-Type: application/json' --data-raw '{"xappName": "ad"}'

TS : curl --location --request POST "http://$(hostname):32080/appmgr/ric/v1/xapps" --header 'Content-Type: application/json' --data-raw '{"xappName": "trafficxapp"}'

QP : curl --location --request POST "http://$(hostname):32080/appmgr/ric/v1/xapps" --header 'Content-Type: application/json' --data-raw '{"xappName": "qp"}'


Verification:

AD : Check for logs in pods where AD is running, It should have following logs(It might take some time for AD to detect an anomalous user since ideally majority of users will be normal in network and to get motioned logs below):


TS : Check for logs in pods where TS is running, It should have following logs:




QP : Check for logs in pods where QP is running, It should have following logs:


Deleting xApp:

AD : curl --location --request DELETE "http://$(hostname):32080/appmgr/ric/v1/xapps/ad" --header 'Content-Type: application/json'

TS : curl --location --request DELETE "$(hostname):32080/appmgr/ric/v1/xapps/trafficxapp" --header 'Content-Type: application/json'

QP : curl --location --request DELETE "$(hostname):32080/appmgr/ric/v1/xapps/qp" --header 'Content-Type: application/json'


Notes:

  1. All three xApp's needs to be deployed in following sequence AD→ TS→ QP for end to end testing of AD use case
  2. Names of the xApp to be deployed must match with what the on-boarder has
  3. AD uses message types TS_ANOMALY_UPDATE to send anomalous UE's and TS_ANOMALY_ACK for acknowledgment from TS on receiving same message successfully
  4. QP send UEPredicitonSet message to TS and expects predicted throughput message callback 


  • No labels

79 Comments

  1. dear all,
    if after deployment AD xapp you see problems wih database connection in xapp logs, you need to change database address:
    (please note that i don't know if this is best solution, but resolves that problem)

    • go to: ric-app-ad/ad
    • in files insert.py, database.py change host for ip address of your influxdb database (you can get it from kubectl get services -n ricplt)
      go to ric-app-ad folder
    • docker build .
    • tag build image as nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ad:0.0.2
    • kubectl rollout restart deployment ricxapp-ad
    1. Thanks Kamil for pointing this out. You don not need to update host ip for influxdb database. I have mentioned in AD repo docs that we need to change host as <svc-name>.<namespace>. for me it was r4-influxdb.ricplt. (maybe for you it is ricplt-influxdb.ricplt). Please let me know if it helps.


      1. requests.exceptions.ConnectionError: HTTPConnectionPool(host='ricplt-influxdb.ricplt', port=8086)

        kubectl get services -n ricplt
        ricplt-influxdb ClusterIP 10.107.175.83 <none> 8086/TCP,8088/TCP 25d

        in my case, that method not working (sad)

        1. Hi Kamil Kociszewski


          I am getting this error in AD log

          requests.exceptions.ConnectionError: HTTPConnectionPool(host='r4-influxdb.ricplt', port=8086): Max retries exceeded with url: /query?q=DROP+DATABASE+%22UEData%22 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fef6e69a6d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

          when I run

          kubectl get services -n ricplt

          I do not see any "ricplt-influxdb"

          Am I missing any step. I followed the instructions from this page.

          Also there is a typo in Step 2:

          QP : echo '{"config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD" }' > onboard.qp.url 

          ts.git should be corrected to qp.git



          1. Hi Obaidz8 which version of RIC platform you have deployed? influxdb is part of dawn release.

            1. Hi Deepanshu Karnwal,


              Thank you.


              I did 

              git clone http://gerrit.o-ran-sc.org/r/it/dep

              then used the

              RECIPE_EXAMPLE/example_recipe_oran_dawn_release.yaml

              Now I can see:

              ricplt-influxdb ClusterIP 10.109.206.25 <none> 8086/TCP,8088/TCP 62m


              but when I run:

              curl --location --request POST "http://$(hostname):32080/appmgr/ric/v1/xapps" --header 'Content-Type: application/json' --data-raw '{"xappName": "ad"}'


              I get the message below:
              "operation XappDeployXapp has not yet been implemented"



                1. Hi Javier Renart Espi

                  Thank you.

                  I have tried the above instructions for the xApps.

                  I am still getting the same error in AD.


                  requests.exceptions.ConnectionError: HTTPConnectionPool(host='r4-influxdb.ricplt', port=8086): Max retries exceeded with url: /query?q=DROP+DATABASE+%22UEData%22 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd6fed35690>: Failed to establish a new connection: [Errno -2] Name or service not known'))


                  kubectl get services -n ricplt show:


                  ricplt-influxdb ClusterIP 10.107.31.253 <none> 8086/TCP,8088/TCP


                  Do I need to change host from host='r4-influxdb.ricplt' to ricplt-influxdb ?


                  How can I do it ?


                  Thank you for your help.






                  1. There are 4 files where you need to change the host:


                    qp/qp/database.py

                    qp/qp/insert.py

                    ad/ad/database.py

                    ad/ad/insert.py

                  2. Obaidz8you can install either way (dms_cli tool or with approach mentioned on this page). And you will have to update influxdb host in four places as Javier has mentioned above(qp/qp/database.py, qp/qp/insert.py, ad/ad/database.py, ad/ad/insert.py).


                    in the host part you can either provide ip of your influx pod or you can provide <svc-name>.<namespace>.

                    In your case (ricplt-influxdb ClusterIP 10.107.31.253 <none> 8086/TCP,8088/TCP) it should be : 

                    host = 10.107.31.253 or

                    ricplt-influxdb.plt

                    1. Hi Deepanshu Karnwal Javier Renart Espi

                      I have tried both with IP address and ricplt-influxdb.ricplt, still does not work.

                      When I run

                      kubectl get pods --all-namespaces -o wide


                      I can see

                      ricplt ricplt-influxdb-meta-0 0/1 Pending 0 3h38m <none> <none> <none> <none>


                      Could this be a problem ?


                      Thank you


                      1. yes, db is not correctly deployed. 
                        please find instructions here
                        Onetime setup for Influxdb
                        Installation Guides — it-dep master documentation (o-ran-sc.org)

                        1. Thank you very much Kamil Kociszewski. This fixed the issue. I was doing the Influxdb setup after deploying the RIC platform. 


                          Now I am geeting another error.

                          AD output: ("version": "0.0.2")

                          [INFO] Sending Anomalous UE to TS
                          [INFO] Message to TS: message sent Successfully
                          [INFO] Received acknowldgement from TS (TS_ANOMALY_ACK): {'payload': b'[{"du-id": 1002, "ue-id": "Car-1", "measTimeStampRf": 1636521455171, "Degradation": "Throughput RSRP RSRQ RSSINR"}]', 'payload length': 115, 'message type': 30004, 'subscription id': -1, 'transaction id': b'05ef31ae41ef11ec9fae3647da2fc8ca', 'message state': 0, 'message status': 'RMR_OK', 'payload max size': 3136, 'meid': b'', 'message source': 'service-ricxapp-trafficxapp-rmr.ricxapp:4560', 'errno': 0}
                          /opt/conda/lib/python3.7/site-packages/sklearn/base.py:442: UserWarning: X does not have valid feature names, but IsolationForest was fitted with feature names
                          "X does not have valid feature names, but"
                          /opt/conda/lib/python3.7/site-packages/sklearn/base.py:442: UserWarning: X does not have valid feature names, but IsolationForest was fitted with feature names
                          "X does not have valid feature names, but"


                          TS Output: (  "version": "1.0.0")

                          [INFO] AD Callback got a message, type=30003, length=115
                          [INFO] Payload is [{"du-id": 1002, "ue-id": "Car-1", "measTimeStampRf": 1636521455171, "Degradation": "Throughput RSRP RSRQ RSSINR"}]
                          [INFO] Prediction Request length=30, payload={"UEPredictionSet": ["Car-1"]}
                          sigenaddr: error from getaddrinfo: target=service-ricxapp-qpdriver-rmr.ricxapp.svc.cluster.local:4560 host=service-ricxapp-qpdriver-rmr.ricxapp.svc.cluster.local port=4560(port): error=(-2) Name or service not known
                          [ERROR] send failed: 2


                          QP Output: ("version": "0.0.4")

                          1636628390 1/RMR [INFO] sends: ts=1636628390 src=service-ricxapp-qp-rmr.ricxapp:4560 target=service-ricxapp-trafficxapp-rmr.ricxapp.svc.cluster.local:4560 open=0 succ=0 fail=0 (hard=0 soft=0)


                          Do I need to install QP driver app ?

                          It seems that the service below is not running:

                          target=service-ricxapp-qpdriver-rmr.ricxapp.svc.cluster.local:4560


                          1. You need to change /ric-app-ts/assets/bootstrap.rt and build dockerfile again.

                            # bootstrap route table

                            newrt | start
                            mse | 20011 | -1 | service-ricplt-a1mediator-rmr.ricplt:4562
                            mse | 30000 | -1 | service-ricxapp-qp-rmr.ricxapp.svc.cluster.local:4560
                            mse | 30004 | -1 | service-ricxapp-ad-rmr:4560
                            newrt | end

                            1. Hi Kamil Kociszewski,


                              Thank you.


                              I have updated the route table now I have another error in TS xApp.


                              [INFO] Prediction Callback got a message, type=30002, length=125
                              [INFO] Payload is {"Pedestrian-1": {"c1/B2": [2, 2], "c1/N77": [0, 0], "c3/B13": [1063, 1063], "c4/B13": [3421, 3421], "c2/B13": [4118, 4118]}}
                              [INFO] Sending a HandOff CONTROL message to "http://localhost:5000/api/echo"
                              [INFO] HandOff request is {
                              "command": "HandOff",
                              "seqNo": 40369,
                              "ue": "Pedestrian-1",
                              "fromCell": "c1/B2",
                              "toCell": "c2/B13",
                              "timestamp": "Mon Nov 15 16:31:07 2021",
                              "reason": "HandOff Control Request from TS xApp",
                              "ttl": 10
                              }
                              [ERROR] curl_easy_perform() failed: Couldn't connect to server


                              Do I need to set some IP address ?


                              Thanks




                              1. its correct at this stage, there is no further route through E2 and message is seneded to some echo server that can be defined by you.
                                AD use case is finish here.

      2. Hi, this worked for me, with: host="ricplt-influxdb.ricplt"

      3. Deepanshu Karnwalcan you kindly mention the document please ? Thank you

    2. Thanks for the fix.

      I had to repeat this process for qp pod too - in both database.py and insert.py

  2. Hi Deepanshu Karnwal

    After deploying the xapps, I'm not getting any HandOff message, but everything else looks OK. What can be wrong?

    Also, in the Use Case you only deploy QP, AD and TS; but if I don't deploy QP-Driver xApp, QP xapp throws an error because can't connect to QPdriver. Is this normal? Is now QPDriver also needed?

    These are the versions I'm deploying, on Dawn RIC. 

    AD 0.0.2; QP 0.0.2, QPDriver 1.1.0; Trafficxapp 1.0.0


    1. Hello Javier,

      In dawn release we have removed QP driver functionality for TS use case. You will need to install QP 0.0.4 for proper communication. Find details here:
      https://docs.o-ran-sc.org/projects/o-ran-sc-ric-app-qp/en/latest/release-notes.html#id1

      Thanks

      1. Thanks, I'm using version 0.0.4 now, but I have the same problem. TS_UE_LIST (30000) messages from TS are not received by QPdriver.

        1. Javier, you don't need QP driver xApp for TS use case now. You only need TS, AD and QP. TS_UE_LIST message is between TS->QP

          1. In which versions? 

            TS on v1.1.1 sends TS_UE_LIST messages to the QPdriver xapp. If I uninstall the QPDriver xapp I get an error that it can't find the qpdriver xapp.

            TS has on its bootstrap route table qpdriver xapp, not qp.  

            1. I changed the route table to send the messages to the QP, and now it works. The problem now is that after HandOff request, I get this error: "curl_easy_perform() failed: Couldn't connect to server"


              1. It is expected behavior Javier Renart Espi. Since we don't have complete functionality of closed loop as of now but in TS xApp we have implemented an API call to take action. 

                1. Ok, thanks. I see that the TS xapp has a callback for A1 policies, but I don't think it's receiving any policy from the A1 interface. Is there any way to trigger this messages? Is the SMO supposed to send the policy automatically when the TS xapp is deployed?

  3. Is the AD Use Case an evolution of the Cherry's Traffic Steering Use Case? I see they're similar but the TS UC taked into account the available PRBs of the cells, while the AD doesn't. How do both use cases differ?

    1. Javier  you are right TS UC(consists AD, QP, TS xApp's) in dawn is evolution of Cherry TS UC. AD deals specific to UE parameters.

      Hope that helps.

      1. But the TS in Dawn doesn't use for example the available PRB of the cells, something that previously the TS UC did, right?

        1. Can you share (link) where TS UC had used PRB of cells in past? 

          1. This is what I meant: Traffic Steering Flows But I'm not sure how this worked as I didn't manage to make it work. 


            I've seen that QP xapp currently uses the last 11 pdcpBytesDl and pdcpBytesUl from the Cell to predict the UE's pdcpBytesDl and pdcpBytesUl. Could you kindly explain why you're using these parameters and the VAR model? I see that we also have signal quality values like RSRP that could be used to predict the throughput right? I'm not familiar with the VAR model used and stationary time series, so maybe I'm missing something. 


            Another question, how has the UE and Cell data in this UC been generated? 

            1. Link you shared has TS flow before Dawn release and AD/QP/TS xApp's have not used available PRB of the cells so far. As per my understanding KPIMON is supposed to push data into database from where different xApp's can fetch as per their need. 

              QP xApp currently is working on (time series problem) forecasting throughput for next few timestamps in advance for a given user and its current neighbor cells. So TS can hand it over to best performing cell if required. VAR (Vector auto regression) is such time series algorithm for forecasting multiple parameters. One can fine tune lag parameter(how far one has to go in past to predict next outcomes). I am not sure if we can predict throughput based on RSRP (Please share more information on this if you have). 


              Hope this helps!



              1. Thanks for the info! 

                Our idea was to use RSRP, distance and cell load to predict the UE throughput. I'll get back to you if I find more info about it. 

  4. I have built it successfully but when I do a push it has an error so anyone knows how to resolve it???


    root@leo-VirtualBox:/home/leo/ric-app-hw/ad#docker build -t nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ad:0.0.2 .
    Sending build context to Docker daemon 20.89MB
    Step 1/15 : FROM frolvlad/alpine-miniconda3
    ---> 9ca2d6f3bd4a
    Step 2/15 : RUN mkdir -p /opt/route/
    ---> Using cache
    ---> fe695a352526
    Step 3/15 : COPY --from=nexus3.o-ran-sc.org:10002/o-ran-sc/bldr-alpine3-rmr:4.0.5 /usr/local/lib64/librmr* /usr/local/lib64/
    ---> Using cache
    ---> 4d63fdd9fd1a
    Step 4/15 : COPY --from=nexus3.o-ran-sc.org:10002/o-ran-sc/bldr-alpine3-rmr:4.0.5 /usr/local/bin/rmr* /usr/local/bin/
    ---> Using cache
    ---> 8d3c38093076
    Step 5/15 : ENV LD_LIBRARY_PATH /usr/local/lib/:/usr/local/lib64
    ---> Using cache
    ---> be9abe16b846
    Step 6/15 : COPY local.rt /opt/route/local.rt
    ---> Using cache
    ---> 015ce0bdbc64
    Step 7/15 : ENV RMR_SEED_RT /opt/route/local.rt
    ---> Using cache
    ---> dd40800d0c06
    Step 8/15 : RUN apk update && apk add gcc musl-dev
    ---> Using cache
    ---> 8656cf7e6c15
    Step 9/15 : RUN pip install ricxappframe
    ---> Using cache
    ---> f84e4921c88b
    Step 10/15 : COPY setup.py /tmp
    ---> Using cache
    ---> c9d8fb04ebaa
    Step 11/15 : COPY LICENSE.txt /tmp/
    ---> Using cache
    ---> 4f37167802ec
    Step 12/15 : COPY ad/ /ad
    ---> Using cache
    ---> 31ff7f1275f4
    Step 13/15 : RUN pip install /tmp
    ---> Using cache
    ---> b1a93104cc46
    Step 14/15 : ENV PYTHONUNBUFFERED 1
    ---> Using cache
    ---> 7f8bc7470d22
    Step 15/15 : CMD PYTHONPATH=/ad:/usr/lib/python3.7/site-packages/:$PYTHONPATH run-ad.py
    ---> Using cache
    ---> d6ad2450a2bc
    Successfully built d6ad2450a2bc
    Successfully tagged nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ad:0.0.2


    root@leo-VirtualBox:/home/leo/ric-app-hw/ad# sudo docker push nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ad:0.0.1
    The push refers to repository [nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ad]
    5c54dab035e1: Preparing
    5d3e835823db: Preparing
    d4894aca3542: Preparing
    79e07d1323a3: Preparing
    52ae9c81f8d2: Preparing
    fd989337ff63: Waiting
    3d04cf930870: Waiting
    e10e73199034: Waiting
    376e7a9179e7: Waiting
    eaea412865da: Waiting
    b04280651494: Waiting
    c7ddd411f0cc: Waiting
    e2eb06d8af82: Waiting
    unauthorized: access to the requested resource is not authorized
    root@leo-VirtualBox:/home/leo/ric-app-hw/ad#

    1. i am getting the same error now

  5. in case, you meet problem after deploying QP xApp (problem with redis.py, problem with import from redis._compat), just add 
    RUN pip install --force-reinstall redis==3.0.1
    after 
    RUN pip install ricxappframe
    in Dockerfile and rebuild image using
    docker build --network host --no-cache .

  6. Hello All,

    I encounter an error in trafficxapp as CrashLoopBackOff.

    Except for the ts xapp, All the pods are the latest and working normally with QP version is 0.0.4, ad version is 0.0.2, trafficxapp version is 1.2.0.

    There is errors in logs:

    terminate called after throwing an instance of 'std::__ios_failure'
    what(): basic_filebuf::underflow error reading the file: Is a directory


    Does anyone also face this issue, or have the solution to this?

    Thanks a lot!

    1. Hi Wayne,

      Actually, the problem is related to xapp-frame-cpp. This error is related to XAPP_DESCRIPTOR_PATH env variable.
      Please set this variable to /opt/ric/config/config-file.json.

      I am already working on this issue in xapp-frame-cpp (RIC-883).
      Thanks for reporting.

      1. Hi Alexandre Huff,

        It works! Thank you.

        I have updated the env variable and now I have another error in TS xApp.

        There is an error in logs:

        [ERROR] failed to send a RIC Control Request message to RC xApp, error_code=14, error_msg=Connect Failed

        The version of RC xApp is the latest, 1.0.0, and it's running.

        Do I need to set the ip & port of grpc-server in config-file.json or schema.json?

        Thx.

  7. Dear all,


    I got this error when I tried to onboard the TS xApp (version 1.2.0):

    {
    "error_source": "config-file.json",
    "error_message": "'__empty_control_section__' is a required property",
    "status": "Input payload validation failed"
    }

    When I revert the version of config-file.json to 1.1.1 by using this command, the onboarding was OK and the xApp run well:

    echo '{"config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/config.json;h=10d10fc9809b6eac9bd335ba8f255cd08612a66c;hb=c6e27b05b6fdf906f2b81e3cb30cad123c40bcb8" }' > onboard.ts.url


    Could anyone enlighten me about this problem? Perhaps I should upgrade the platform?


    Thanks,

    Baud Prananto


    1. With TS xApp version 1.2.0, you need to provide the additional schema file for this controls section since it's not empty.

      You can use the command below for onboarding:

      echo '{ "config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/config.json;hb=addc582ffeccd02f502f2122d4dff0fa19b0a38f","controls-schema.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/schema.json;hb=addc582ffeccd02f502f2122d4dff0fa19b0a38f" }' > onboard.ts.url

      1. Great, the onboarding works. Thank you very much.

        However, I still get CrashLoopBackOff error after deploying the xApp. I already added the environment variables:

        export XAPP_DESCRIPTOR_PATH=/opt/ric/config/config-file.json

        Is this the correct command?

        Thanks for any help.

        1. Hi Baud Prananto

           I am also facing same issue and not able to resolve so far. could you please share the solution if you have solved it? or can someone in community please help ?
          Thanks a lot

          1. Hi Chandra and Baud,

            XAPP_DESCRIPTOR_PATH needs to be passed via helm charts. This requires some additional steps using dms_cli and deploy TS manually for now.
            Please try these following steps:
            1 - onboard TS via dms_cli
            2 - download TS helm charts from dms_cli (charts are generated while onboarding)
            3 - change XAPP_DESCRIPTOR_PATH in helm charts just downloaded
            4 - deploy TS using helm3: $helm install <xapp_chart_name> <chart_dir> --namespace ricxapp [-f overridefile]

            This CrashLoopBackOff is related to RIC-883.

            I will keep you updated.

            Thanks,

            1. Hi,

              TS xApp has been updated to version 1.2.1. This will fix the CrashLoopBackOff issue.

              Thanks,

              1. Hi Alexandre Huff Thanks for updating the version.

                I am currently using below json_url path for 1.2.0 

                echo '{ "config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/config.json;hb=addc582ffeccd02f502f2122d4dff0fa19b0a38f","controls-schema.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/schema.json;hb=addc582ffeccd02f502f2122d4dff0fa19b0a38f" }' > onboard.ts.url

                AND to run TS xAPP 1.2.1 version do I need to modify anything in here, Could you please suggest?

                Thanks in Advance

                1. Dear Alexandre Huff

                  Thank you for version 1.2.1, it works well now using standard onboarding and deployment method. I didn't use dms_cli.


                  Dear Chandra Shekar Telagamaneni

                  I use the following command to create the script:

                  echo '{"config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD","controls-schema.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/ts.git;a=blob_plain;f=xapp-descriptor/schema.json;hb=HEAD"}' > onboard.ts.url


                  Thanks,

                  Baud



                  1. Hi Chandra Shekar Telagamaneni,

                    Please, use those updated links provided by Baud Prananto. They will always point to the most updated files at the HEAD of the repo.

                    Baud PranantoThanks for suggesting the link you are using.

                    Regards,

                    1. Thanks a lot It is working Fine Now.

                      Regards
                      Chandra Telagamaneni

                  2. Thank you for the above command Baud Prananto

                    Regards
                    chandra

  8. Is there any demo or setup we can follow to see how all these work? What I see is all setting it up but there is no guide on how they work by editing the parameters in the codes or simulating an alert in the AD Xapp and etc.


     AD→ TS → QP→ E2→ Kpimon 

  9. Hi All,
    while checking logs of AD pod, got below error

    requests.exceptions.ConnectionError: HTTPConnectionPool(host='r4-influxdb.ricplt', port=8086): Max retries exceeded with url: /query?q=DROP+DATABASE+%22UEData%22 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4e94c0f890>: Failed to establish a new connection: [Errno -2] Name or service not known'))

    after installing InfluxDB using steps from https://docs.o-ran-sc.org/projects/o-ran-sc-it-dep/en/latest/installation-guides.html#ric-applications 

    kubectl get services -n ricinfra 

    nfs-release-1-nfs-server-provisioner ClusterIP 10.105.234.32 <none> 

    I have modified host for 2 files in AD pod based on current page comment suggestion-

    database.py
    ----------
    From host = str (default='r4-influxdb.ricplt.svc.cluster.local') to host = 10.105.234.32
    & insert.py
    --------
    from host= 'r4-influxdb.ricplt.svc.cluster.local' to host = 10.105.234.32

    but still getting same error in the logs.

    could you please help me with next steps to resolve this error or new method to resolve it, Would be great.

    Thanks & Regards
    Chandra Shekar Telagamaneni

    1. I also have this issue but the forum very slow in responding.. I think i have this months back

  10. @Chandra Sekhar 


    Please check kubectl get SVC -n riplt and provide influxdb ip as host in database.py and insert.py


    Example: influxdb ip: xx:xx:xx:xx 

    in database.py and insert.py give host = influxdb ip


    Or

     You can get service name of influxdb by "kubectl get SVC -n ricplt" and put host = <influxdb_svc_name>.ricplt instead of r4-influxdb.ricplt.svc.cluster.local


    Example: influxdb svc-name = ricplt-influxdb then give host= ricplt-influxdb.ricplt

    1. Hi Deepanshu Karnwal

      after doing kubectl get svc -n ricplt got below result
      ricplt-influxdb ClusterIP 10.101.146.32 <none> 8086/TCP,8088/TCP

      used IP as this host= 10.101.146.32 for database.py and insert.py saved the files.  exited from pod
      kubectl logs [AD_Pod_name] -n ricxapp
      in the logs still I see same error.

      is their anything which I need to run commands to reflect changes made in files of the pod?

      could you please suggest?

      Thanks in advance

      1. Hi Chandra,

        You should not change the files in the running pod.

        You should:

        1. Change source files in your VM
        2. Build docker image again
        3. Onboard and deploy the xapp again
  11. Is there any other document or demo available to explain this use case? I didn't found more details about KPIMON xAPP and how to use E2 sim for this use case. can we use A1 policy management for anomaly detection and TS?

    1. Hi Rajesh,

      Regarding using A1 policy in TS.
      TS receives A1 policy and only logs it. A1 policy enforcement will be enabled in the next days.

      Thanks,

  12. Hi all,


    I am trying to use dms_cli command to onboard the AD xApp and the QP xApp but where is the schema file for these xApp.


    I use the below command to clone the files from github but cannot find the schema file in the xapp-descriptor.


        AD : git clone "https://gerrit.o-ran-sc.org/r/ric-app/ad"

          TS : git clone "https://gerrit.o-ran-sc.org/r/ric-app/ts"

          QP : git clone "https://gerrit.o-ran-sc.org/r/ric-app/qp

    1. Hi Leo,

      Using schema file to onboard an xApp is not always required (currently).
      What dms_cli does is using a default schema when it is not available. You can just skip the schema file.
      Other xApps like TS, requires additional configuration in config.json (i.e., controls section) that is not defined in the default schema. So, it is required to specify the schema file that validades that additional configuration.

      Thanks,

      1. But when i use the dms_cli command they will required the schema file else will prompt an error...


        leo@leo-VirtualBox:~/oran/ad/xapp-descriptor$ dms_cli onboard config.json
        [Errno 2] No such file or directory: '../../../docs/xapp_onboarder/guide/embedded-schema.json'
        leo@leo-VirtualBox:~/oran/ad/xapp-descriptor$


        1. You can provide the path to the embedded-schema.json that should be located where you installed dms_cli.

          For example, I checked out appmgr repo in my home directory, so my location to the embedded-schema.json is:

          ~/appmgr/xapp_orchestrater/dev/docs/xapp_onboarder/guide/embedded-schema.json

        2. In this case, for some reason dms_cli is not able to fetch that file.
          What you can do in this case is copying that embedded-schema.json to the directory where config.json file is, and provide it via dms_cli.

          Thanks,

  13. is there any use cases on how to modify the codes in the TS xapp to trigger some abnormal traffic and the AD xapp will send an alarm or etc... 


    there are really little to no guides on how to use these xapps...

  14. Hi All,

    I am Curious to know how influx DB data is populated? and
    I was trying to insert a record with random Data same as Anomaly detected record like [ue-id=Car-2] into Influxdb through below command to find whether reflect in logs of AD.

    INSERT AD,Anomaly=1 Degradation="RSRP RSSINR" du-id=1001 nbCellIdentity_0="c1/B2" nbCellIdentity_1="c1/N77" nbCellIdentity_2="c4/B13" nbCellIdentity_3="c5/B13" nbCellIdentity_4="c6/B13" nrCellIdentity="c1/B13" prb_usage=45 rsrp=-126.16104463859146 rsrp_nb_0=-125.0355543946282 rsrp_nb_1=-127.1563464008487 rsrp_nb_2=-128.18422423590131 rsrp_nb_3=-129.21877319624657 rsrp_nb_4=-130.02026359693942 rsrq=-50.792886341911839 rsrq_nb_0=-50.795964422674814 rsrq_nb_1=-50.792489332184743 rsrq_nb_2=-50.79722072030917 rsrq_nb_3=-50.797263888541963 rsrq_nb_4=-50.804882540836275 rssinr=16.06774281421192 rssinr_nb_0=17.19323305817518 rssinr_nb_1=15.07244105195467 rssinr_nb_2=12.044563216902066 rssinr_nb_3=13.010014256556804 rssinr_nb_4=14.208523855863962 targetTput=0.75 throughput=0.19410518515959901 ue-id="Car-8" x=348 y=-1 1439938800000000000

    Here I am not Sure which attributes are tags and where I need to mention (,) for tags. Got below error for above command.

    ERR: {"error":"unable to parse 'AD,Anomaly=1 Degradation=\"RSRP RSSINR\" du-id=1001 nbCellIdentity_0=\"c1/B2\" nbCellIdentity_1=\"c1/N77\" nbCellIdentity_2=\"c4/B13\" nbCellIdentity_3=\"c5/B13\" nbCellIdentity_4=\"c6/B13\" nrCellIdentity=\"c1/B13\" prb_usage=45 rsrp=-126.16104463859146 rsrp_nb_0=-125.0355543946282 rsrp_nb_1=-127.1563464008487 rsrp_nb_2=-128.18422423590131 rsrp_nb_3=-129.21877319624657 rsrp_nb_4=-130.02026359693942 rsrq=-50.792886341911839 rsrq_nb_0=-50.795964422674814 rsrq_nb_1=-50.792489332184743 rsrq_nb_2=-50.79722072030917 rsrq_nb_3=-50.797263888541963 rsrq_nb_4=-50.804882540836275 rssinr=16.06774281421192 rssinr_nb_0=17.19323305817518 rssinr_nb_1=15.07244105195467 rssinr_nb_2=12.044563216902066 rssinr_nb_3=13.010014256556804 rssinr_nb_4=14.208523855863962 targetTput=0.75 throughput=0.19410518515959901 ue-id=\"Car-8\" x=348 y=-1 1439938800000000000': bad timestamp"}

    Could you please help me out?

    Thanks a lot 

  15. Dear all,


    When I run and check the log of AD xApp, I got an error in the beginning:

    ModuleNotFoundError: No module named 'redis._compat'

    Does anyone have the solution?

    I'm running version  0.0.2.


    Thanks,

    Baud

    1. Hi Baud Prananto
      clone AD xapp repo locally and do below steps:
      IN Dockerfile  after this line - RUN pip install ricxappframe 
      add below line
      RUN pip install --force-reinstall redis==3.0.1
      and run below command
      docker build --network host --no-cache

      I hope this will resolve you're issue

      Thanks

      1. Dear Chandra Shekar Telagamaneni

        I tried to re-clone, re-build using that additional line, re-onboard, and re-deploy but still the same result.

        If I check the redis version using pip3 list and pip list, it shows 3.0.1. 


        Here is the complete log of the AD xApp:

        1646199310 1/RMR [INFO] ric message routing library on SI95/g mv=3 flg=02 (f25dc7f 4.0.5 built: May 7 2020)
        Traceback (most recent call last):
        File "/opt/conda/bin/run-ad.py", line 8, in <module>
        sys.exit(start())
        File "/ad/main.py", line 126, in start
        xapp = Xapp(entrypoint=entry, rmr_port=4560, use_fake_sdl=False)
        File "/opt/conda/lib/python3.7/site-packages/ricxappframe/xapp_frame.py", line 540, in __init__
        super().__init__(rmr_port=rmr_port, rmr_wait_for_ready=rmr_wait_for_ready, use_fake_sdl=use_fake_sdl)
        File "/opt/conda/lib/python3.7/site-packages/ricxappframe/xapp_frame.py", line 86, in __init__
        self.sdl = SDLWrapper(use_fake_sdl)
        File "/opt/conda/lib/python3.7/site-packages/ricxappframe/xapp_sdl.py", line 53, in __init__
        self._sdl = SyncStorage()
        File "/opt/conda/lib/python3.7/site-packages/ricsdl/syncstorage.py", line 124, in __init__
        self.__dbbackend = ricsdl.backend.get_backend_instance(self.__configuration)
        File "/opt/conda/lib/python3.7/site-packages/ricsdl/backend/__init__.py", line 40, in get_backend_instance
        backend_module = import_module('.' + backend_module_name, package=package)
        File "/opt/conda/lib/python3.7/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
        File "<frozen importlib._bootstrap>", line 983, in _find_and_load
        File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 728, in exec_module
        File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
        File "/opt/conda/lib/python3.7/site-packages/ricsdl/backend/redis.py", line 31, in <module>
        from redis._compat import nativestr
        ModuleNotFoundError: No module named 'redis._compat'
        1646199311 1/RMR [INFO] sends: ts=1646199311 src=service-ricxapp-ad-rmr.ricxapp:4560 target=service-ricxapp-trafficxapp-rmr:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
        1646199342 1/RMR [INFO] sends: ts=1646199342 src=service-ricxapp-ad-rmr.ricxapp:4560 target=service-ricplt-a1mediator-rmr.ricplt:4562 open=0 succ=0 fail=0 (hard=0 soft=0)


        Any help is appreciated.


        Thanks,

        Baud

        1. HI Baud Prananto
          I also had the same issue. I added RUN pip install --force-reinstall redis==3.0.1 into the Dockerfile.
          built the image locally with same tag but with different version just for reference purpose.
          then I just changed already present image with my locally built image in deployment file of already running AD xapp pod.

          In this way, I was able to rectify the issue. I hope this will help.

          Thanks

          1. Dear Chandra Shekar Telagamaneni

            Thank you, it works now.

            Sorry I'm little bit confused with the image management in docker.  I think I need to clean up the existing images to make the newly built images to take effect.

            I used docker image prune command to clean up.

            BTW could you (or anyone) please enlighten me about building the image locally? 

            1. Is tagging really necessary? Is docker build -t nexus3.o-ran-sc.org:10002/o-ran-sc/ric-app-ad:0.0.2 . mandatory, or docker build . enough? 
            2. How can I choose the docker image that will be deployed?
            3. Is it possible to deploy the xApp without building locally, i.e. using online image?

            Thanks,

            Baud

  16. Hi All,
    I am trying to get anomaly detected for manual insertion of data into the measurements of UEData database of INFLUXDB.

    I have added similar data into the all measurements present in the UEData database as similar to already anomaly detected records like ue-id=Car-2 & ue-id=Car-4. But I see no anomaly detected for the records which I have added manually by seeing  the logs in AD pod.

    Is there any specific attribute in Measurements/tables which helps anomaly detector to detect anomaly? or
    the Values needs to be in particular range or specific data needs to be added? to detect anomaly?

    any reference record to insert manually and can see anomaly detected, that will be a great help.

    Thanks a lot

  17. Hi All, Is there community weekly meeting going on for this project? for future development or the bug fixing
    if there is, could you Please share the details?

    Thanks a lot

  18. Hi all,


    When I run the xApps AD, it does not show me data from the rest of the UEs in the database. It only shows me this:


    [INFO] Sending Anomalous UE to TS

    [INFO] Message to TS: message sent Successfully

    /opt/conda/lib/python3.7/site-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but IsolationForest was fitted with feature names

             "X does not have valid feature names, but"

    [INFO] Sending Anomalous UE to TS

    [INFO] Message to TS: message sent Successfully

    /opt/conda/lib/python3.7/site-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but IsolationForest was fitted with feature names

            "X does not have valid feature names, but"

    /opt/conda/lib/python3.7/site-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but IsolationForest was fitted with feature names

             "X does not have valid feature names, but"



    That is received by the xApp TS as "Car-4":


    [ERROR] failed to send a RIC Control Request message to RC xApp, error_code=14, error_msg=Connect Failed

    [INFO] Prediction Callback got a message, type=30002, length=142

    [INFO] Payload is {"Car-4": {"c1/B13": [2656, 2656], "c1/B2": [2, 2], "c1/N77": [0, 0], "c4/B13": [3421, 3421], "c2/B13": [4118, 4118], "c3/B13": [1063, 1063]}}

    [ERROR] failed to send a RIC Control Request message to RC xApp, error_code=14, error_msg=Connect Failed

    [INFO] AD Callback got a message, type=30003, length=99

    [INFO] Payload is [{"du-id": 1001, "ue-id": "Car-4", "measTimeStampRf": 1647339405414, "Degradation": "RSRP RSSINR"}]

    [INFO] Prediction Request length=30, payload={"UEPredictionSet": ["Car-4"]}

    [INFO] Prediction Callback got a message, type=30002, length=142

    [INFO] Payload is {"Car-4": {"c1/B13": [2656, 2656], "c1/B2": [2, 2], "c1/N77": [0, 0], "c4/B13": [3421, 3421], "c2/B13": [4118, 4118], "c3/B13": [1063, 1063]}}


    Any idea what could be happening?

    Thanks a lot,

  19. Hi Team, In TS logs I have observed error about RC Xapp

    [INFO] Payload is {"Car-4": {"c1/B13": [2656, 2656], "c1/B2": [2, 2], "c1/N77": [0, 0], "c4/B13": [3421, 3421], "c2/B13": [4118, 4118], "c3/B13": [1063, 1063]}}
    [ERROR] failed to send a RIC Control Request message to RC xApp, error_code=14, error_msg=Connect Failed
    [INFO] Prediction Callback got a message, type=30002, length=142

    I have tried Onboarding RC Xapp using below steps:

    Steps to onboard the xapp
    1)echo '{ "config-file.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/rc.git;a=blob_plain;f=xapp-descriptor/config.json;hb=HEAD","controls-schema.json_url": "https://gerrit.o-ran-sc.org/r/gitweb?p=ric-app/rc.git;a=blob_plain;f=xapp-descriptor/schema.json;hb=HEAD" }' > onboard.rc.url

    2)Onboard the rc xapp using the below command :
    $curl --location --request POST "http://localhost:32088/onboard/api/v1/onboard/download" --header 'Content-Type: application/json' --data-binary "@./onboard.rc.url"

    After 2nd step getting error as below. I assume there is issue in onboard.rc.url file
    {
    "error_source": "config-file.json",
    "error_message": "'xapp_name' is a required property",
    "status": "Input payload validation failed"
    }

    could you please suggest here?
    Thanks a lot

    1. Hi Chandra,

      Please, try to onboard RC using dms_cli.
      You can check out how to install dms_cli here: https://docs.o-ran-sc.org/projects/o-ran-sc-it-dep/en/latest/installation-guides.html#ric-applications

      Please let me know if that worked for you.
      Thanks,

      1. Dear Alexandre HuffChandra Shekar Telagamaneni


        Could you please explain why suddenly RC xApp comes into the picture?

        I see the error in my TS xApp as well but it seems the prediction works well. 

        What is the function of this RC xApp in this use case?

        The wiki page of RC xApp doesn't explain its position in this use case.


        Thanks,

        Baud

        1. Hi Baud Prananto

          RC xApp does not comes suddenly into the picture of this use case. It has been discussed for a while for this use case.

          The RC xApp is responsible for sending control messages to the RAN using E2AP SM. We note that the RC xApp is not really mandatory to run this use case.

          Currenlty, TS has two interfaces to send control messages (Rest and gRPC):

          1. gRPC: TS issues a hand-off message to the RC xApp requesting it (RC) to send the corresponding control message to the RAN using E2AP SM.
          2. Rest: We keep this interface for that cases that do not need to use the RC xApp. If you are simulating the RAN and want to send control messages from TS to the simulator, you can still use the rest interface.

          Please check out the "controls" section in the xApp descriptor file to choose the interface to send control messages. Also, check out the schema file in the same directory so that you can find examples of configuration.

          Thanks,

          Alexandre.

          1. Dear Alexandre Huff,


            Thank you very much for the explanation. 

            So it is not mandatory for this use case because we still use dummy communication to E2AP SM. 


            Thanks,

            Baud