The RIC Message Router (RMR) is a thin library which allows applications to send messages to other applications. RMR provides insulation from the actual message transport system (e.g. Nanomsg or NNG), as well as providing endpoint selection based on message type. This page serves as an anchor point for RMR related pages.

Announcements

The following are announcements related to RMR during the past 30 days; newest announcements first.


2021 3 May

Version 4.7.4 has been pushed to the release area of the package cloud repository.  See the revision history on the RTD site for the changes.


Current Version

Cherry development begins with version 4.7.4 on the master branch; 4.7.4 current in the package cloud staging repo.

Cherry Release:  Version 4.4.5 (November 2020)

Bronze Release :  version tag 4.0.5 (6 June 2020)  -- Code is currently frozen; only bug fixes and documentation updates are being applied.

Release A frozen: 1.11.1 (Amber branch) 6 November 2019

Trial branch (r2-temp): 1.10.2 Fall 2019


For a detailed description of API changes, and their related commits, please consult the CHANGES file at the top level of the RMR repo or on the RMR read the docs page.

Changes to the API will be announced via this wiki page, so it is assumed that anybody interested in being notified about API changes will place a watch on this page.


Supported Transport Mechanisms

A transport mechanism is a "mid-layer" protocol which makes use of TCP/UDP and provides features such as automatic connection reestablishment and connection acceptance. RMR was originally built using the Nanomsg transport library, and was extended to use NNG (Nanomsg next generation) as NNG provides several enhancements, and Nanomsg has been all but abandoned from a maintenance perspective.  While the underlying "wire" protocol of both NNG and Nanomsg are the same, the APIs are different; RMR has supported both mechanisms by supplying two sets of libraries (librmr and librmr_nng).   At this point in time, NNG is stable, and as the support for Nanomsg is reduced, thus RMR will drop support  for Nanomsg. In other words, beginning with version 1.0.45 (initally announced as happening in 1.0.44, but has been pushed by one) only librmr_nng will be included.  This change should not have any impact on applications using RMR as the RMR API is not changing. The only change would be that applications using librmr will need to alter their build process to change the underlying RMR library to librmr_nng. 

Developer Information

The following links are generally useful for developers writing applications (xAPPs) which make use of RMR.

External Systems

The following links are likely useful for developers which must communicate with RMR based applications (e.g. route manager applications).

RMR Development

The following pages hold information that should be useful to anybody modifying and/or testing RMR itself.

RMR Packages

Several frameworks (wrappers/bindings) are being developed which provide a more simplistic interface to the xAPP developer. Installation of the framework will likely require that the RMR runtime package be installed.  RMR packages (debian and RPM) are currently published on the html site https://packagecloud.io.   

The generic URL below can be used to obtain a list of available packages (current version listed at the top of this page).
https://packagecloud.io/app/o-ran-sc/staging/search?q=rmr&filter=all&filter=all&dist=

Links on these pages redirect to a generalised information HTML page; the actual wget command needed to pull the package is on the right side of these pages near the bottom.

As an example, the following can be added to a Docker file to install the RMR runtime package:


docker example
# Install RMr (runtime and dev) from debian package cached on packagecloud.io
ARG RMR_VER=3.3.0

RUN wget -nv --content-disposition https://packagecloud.io/o-ran-sc/staging/packages/debian/stretch/rmr_${RMR_VER}_amd64.deb/download.deb
RUN dpkg -i rmr_${RMR_VER}_amd64.deb


Performance

The following pages discuss RMR and NNG performance.

Support Software

When the RMR runtime package is installed, there may be one or more support binaries which are included. These are installed (by default) into /usr/local/bin. The links below describe each.

Tutorials

The following are links to short tutorials covering some aspects of RMR usage


  • No labels

8 Comments

  1. Bengi Mizrahi

    Hi Scott,

    I get segmentation fault at tools_static.c:330 in function if_addrs_t* mk_ip_list( char* port ). I know it is very unlikely that an RMR Client runs on a machine with an active Cisco VPN tunnel but ele->ifa_addr turns out to be 0x0 when a tunnel is on:

    The value of ele is:

    ele struct ifaddrs * 0x555555ad80b0
    - ifa_next struct ifaddrs * 0x555555ad8168
    - ifa_name char * 0x555555ad8154 "tun0"
    - ifa_flags unsigned int 69841
    - ifa_addr struct sockaddr * 0x0
    - ifa_netmask struct sockaddr * 0x0
    - ifa_ifu union {...} {...}
    - ifa_data void * 0x555555ad87f0


  2. Kiel Friedt

    I am in the process of porting the near realtime RIC from x86 to ARM and I'm getting sysmalloc asserts when running the E2 service on the RIC. Valgrind is pointing to RMR:

    1614193709 300/RMR [INFO] ric message routing library on SI95/g mv=3 flg=00 (84423e6 4.4.6 built: Feb 11 2021)
==300== Invalid write of size 8
==300==    at 0x4AFAFCC: pthread_mutex_init (pthread_mutex_init.c:101)
==300==    by 0x4879717: uta_ring_config (ring_static.c:134)
==300==    by 0x4882EC7: init (rmr_si.c:629)
==300==    by 0x488357F: rmr_init (rmr_si.c:795)
==300==    by 0x1C1343: getRmrContext(sctp_params&) (sctpThread.cpp:1967)
==300==    by 0x1ABD27: main (sctpThread.cpp:374)
==300==  Address 0x6fb5798 is 0 bytes after a block of size 40 alloc'd
==300==    at 0x4845BFC: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==300==
==300== Invalid write of size 8
==300==    at 0x4AFAFCC: pthread_mutex_init (pthread_mutex_init.c:101)
==300==    by 0x48796BB: uta_ring_config (ring_static.c:123)
==300==    by 0x4882ED7: init (rmr_si.c:630)
==300==    by 0x488357F: rmr_init (rmr_si.c:795)
==300==    by 0x1C1343: getRmrContext(sctp_params&) (sctpThread.cpp:1967)
==300==    by 0x1ABD27: main (sctpThread.cpp:374)
==300==  Address 0x6fb5808 is 0 bytes after a block of size 40 alloc'd
==300==    at 0x4845BFC: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==300==
==300== Invalid write of size 8
==300==    at 0x4AFAFCC: pthread_mutex_init (pthread_mutex_init.c:101)
==300==    by 0x487D873: fd2ep_init (rtable_si_static.c:533)
==300==    by 0x4882EFB: init (rmr_si.c:635)
==300==    by 0x488357F: rmr_init (rmr_si.c:795)
==300==    by 0x1C1343: getRmrContext(sctp_params&) (sctpThread.cpp:1967)
==300==    by 0x1ABD27: main (sctpThread.cpp:374)
==300==  Address 0x6fba568 is 0 bytes after a block of size 40 alloc'd
==300==    at 0x4845BFC: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==300==
==300== Invalid write of size 8
==300==    at 0x4AFAFCC: pthread_mutex_init (pthread_mutex_init.c:101)
==300==    by 0x488335B: init (rmr_si.c:736)
==300==    by 0x488357F: rmr_init (rmr_si.c:795)
==300==    by 0x1C1343: getRmrContext(sctp_params&) (sctpThread.cpp:1967)
==300==    by 0x1ABD27: main (sctpThread.cpp:374)
==300==  Address 0x6fc27c8 is 0 bytes after a block of size 40 alloc'd
==300==    at 0x4845BFC: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)


    the code that seems to be causing the assert:

    ctx->nrivers = MAX_RIVERS;                                              // the array allows for fast index mapping for fd values < max
ctx->rivers = (river_t *) malloc( sizeof( river_t ) * ctx->nrivers );
ctx->river_hash = rmr_sym_alloc( 129 );                         // connections with fd values > FD_MAX have to e hashed
memset( ctx->rivers, 0, sizeof( river_t ) * ctx->nrivers );
for( i = 0; i < ctx->nrivers; i++ ) {
      ctx->rivers[i].state = RS_NEW;                          // force allocation of accumulator on first received packet
}


    Any suggestion on how I can debug this? When running without valgrind i get these message:

    {"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = nano=38000 "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = nano value = 38000"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = loglevel=error "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = loglevel value = error"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = volume=log "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = volume value = log"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = local-ip=127.0.0.1 "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = local-ip value = 127.0.0.1"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = prometheusMode=pull "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = prometheusMode value = pull"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = prometheusPushTimeOut=10 "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = prometheusPushTimeOut value = 10"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = prometheusPushAddr=127.0.0.1:7676 "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = prometheusPushAddr value = 127.0.0.1:7676"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = prometheusPort=8088 "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = prometheusPort value = 8088"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = trace=stop "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = trace value = stop"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = external-fqdn=e2t.com "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = external-fqdn value = e2t.com"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = pod_name=E2TERM_POD_NAME "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = pod_name value = E2TERM_POD_NAME"}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = sctp-port=36422 "}
{"ts":1614192983341,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = sctp-port value = 36422"}
[New Thread 0xffff9dbfc1d0 (LWP 289)]
[New Thread 0xffff9d3fb1d0 (LWP 290)]
1614192983 286/RMR [INFO] ric message routing library on SI95/g mv=3 flg=00 (84423e6 4.4.6 built: Feb 11 2021)
e2: malloc.c:2401: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.


    with valgrind I'm able to bypass the memory issues and it appears that the RMR service is running:

    1614193710 300/RMR [INFO] sends: ts=1614193710 src=6bcb30f9450b:38000 target=10.0.2.15:38010 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193741 300/RMR [INFO] sends: ts=1614193741 src=6bcb30f9450b:38000 target=10.0.2.15:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193741 300/RMR [INFO] sends: ts=1614193741 src=6bcb30f9450b:38000 target=10.0.2.15:3801 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193741 300/RMR [INFO] sends: ts=1614193741 src=6bcb30f9450b:38000 target=10.0.2.15:4801 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193741 300/RMR [INFO] sends: ts=1614193741 src=6bcb30f9450b:38000 target=10.0.2.15:38010 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193772 300/RMR [INFO] sends: ts=1614193772 src=6bcb30f9450b:38000 target=10.0.2.15:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193772 300/RMR [INFO] sends: ts=1614193772 src=6bcb30f9450b:38000 target=10.0.2.15:3801 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193772 300/RMR [INFO] sends: ts=1614193772 src=6bcb30f9450b:38000 target=10.0.2.15:4801 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193772 300/RMR [INFO] sends: ts=1614193772 src=6bcb30f9450b:38000 target=10.0.2.15:38010 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193803 300/RMR [INFO] sends: ts=1614193803 src=6bcb30f9450b:38000 target=10.0.2.15:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
1614193803 300/RMR [INFO] sends: ts=1614193803 src=6bcb30f9450b:38000 target=10.0.2.15:3801 open=0 succ=0 fail=0 (hard=0 soft=0)

  3. Scott Daniels

    @Kiel   Thank you.  

    There are two bug fixes which were applied to the cherry branch and bring RMR version to 4.5.2.   My first suggestion is to pull that code as these bugs caused potential memory corruption issues which could manifest themselves into something that you're seeing undere valgrind.   The memory corruption issues were related to message rings, and so there is a good chance that this is already fixed. 


    If you are still encountering a problem, please open a jira ticket (http://jira.o-ran-sc.org)  and assign it to me.  Add any new details; no need to copy what you have captured here.  


    In the mean time I will attempt to pull RMR onto an ARM device and run it's integration tests under valgrind to see if I can reproduce.  I've not seen any valgrind complaints on x86 based devices, but that's not to say there isn't a problem on ARM.   

    Let me know if you have any questions. 

    1. Kiel Friedt

      As far as x86, i do not see this issue. Ive only encountered this on ARM thus far. I am not sure if this related to some of the C types being different sizes on x86 vs intel. I did pull the latest changes and see the same issue. I will create a ticket.


  4. Scott Daniels

    Great -- thanks.  Agree, it's probably an assumption about size of something that has gone wrong.  Appreciate the leg work!   

  5. Pranesh Kulkarni

    Hi , 

    Intermittently seeing KEEP_ALIVE_RESP not being sent to e2mgr  due to RMR route issue : 

    {"log":"{\"ts\":1641371580695,\"crit\":\"ERROR\",\"id\":\"E2Terminator\",\"mdc\":{\"PID\":\"\"},\"msg\":\"Failed to send E2_TERM_KEEP_ALIVE_RESP, on RMR state = 2 ( RMR_ERR_NOENDPT - send//call could not find an endpoint based on msg type)\"}\r\n","stream":"stdout","time":"2022-01-05T08:33:00.69549698Z"}


    Since the KEEP ALIVE RESPs are not reaching e2mgr, keepAlive expiry is hit in e2mgr  - causing it to remove e2tinstance

    {"log":"{\"crit\":\"WARN\",\"ts\":1641371940695,\"id\":\"E2Manager\",\"msg\":\"#E2TKeepAliveWorker.E2TKeepAliveExpired - e2t address: 10.110.239.160:38000 time expired, shutdown e2 instance\",\"mdc\":{\"time\":\"2022-01-05 08:39:00.695\"}}\r\n","stream":"stdout","time":"2022-01-05T08:39:00.695817141Z"}


    There is a supposedly a similar issue already reported  RIC-837 - RMR is not forwarding messages from E2T to E2M TO DO

    Any update/fix available for the same , let me know. 

    Thanks.

  6. Pranesh Kulkarni

    Hi,

    Between two pods, if there is no RMR message activity say for more than 15mins, we see that the very first message after the inactivity time is lost.

    This issue is particularly observed between "e2term" and "xAPP". The first message sent from e2term after a idle time of > 15min (approx) is getting dropped. However the 2nd successive message is received at xAPP - and that the RMR connection apparently seems to be recovered, since the following messages, there after, are all received. 

    Is there some idle(inactivity) connection timer or some configuration at RMR that is causing this ?


    Regards,

    Pranesh.


    1. Pranesh Kulkarni

      This issue is observed with messages receiving at xAPP