RMR executes a separate thread which listens for route updates from a routing manager application, and manages the internal route table with the received changes. In most cases there are no outward signs that RMR has received and updated route information, and no means to request that the current route table be dumped to some external device. To assist with debugging, the route collection code can be placed into verbose mode which will cause log messages to be written to stderr as the route table is updated. This page explains how to do this.
Verbosity Control File
During initialisation the RMR code will check environment variables for a reference to the verbosity control file. The variable, RMR_VCTL_FILE, should reference an existing file which RMR will open and read the verbosity level from the first line. During the operation of an RMR based application, the route collection code will check the file's contents with each route update, and thus the verbosity level may be dynamically altered during execution. However, if the environment variable, or the actual file, does not exist when the application starts, it cannot be added later; in other words it is not possible to enable verbosity for an application if the variable and file did not exist at the application's start.
The following example shows how the file can be created, and the file name supplied to an RMR based application.
export RMR_VCTL_FILE=/tmp/rmr.vctl echo 1 >$RMR_VCTL_FILE RMR_SEED_RT=/tmp/seed.rt ./msg_sender 10
The example above starts the msg_sender application which will find the verbose control file in /tmp. The initial verbose setting is 1.
Verbose Level 1 Output
Given this simple seed table
newrt|start mse | 1 | -1 | app03:4560 mse | 2 | -1 | app02:4560;app03:4560 mse | 0 | -1 | app02:4560,app03:4560 newrt|end
the following debugging messages would be written as it is read during initialisation:
[DBUG] old route table: [DBUG] rtstats: nil table [DBUG] new route table: [DBUG] RMR route table stats: [DBUG] RMR route table endpoints: [DBUG] RMR rt endpoint: target=app02:4560 open=0 [DBUG] RMR rt endpoint: target=app03:4560 open=0 [DBUG] RMR rtable: 2 known endpoints [DBUG] RMR route table entries: [DBUG] RMR rte: key=ffffffff00000000 mtype= 0 sid= -1 nrrg= 1 refs=1 [DBUG] RMR rte: key=ffffffff00000001 mtype= 1 sid= -1 nrrg= 1 refs=1 [DBUG] RMR rte: key=ffffffff00000002 mtype= 2 sid= -1 nrrg= 2 refs=1 [DBUG] RMR rtable: 3 mt entries in table [DBUG] RMR route table meid map: [DBUG] RMR rtable: 0 meids in map
The basic output in verbose level 1 is to show the "old table" (prior to the updates), followed by the new route table. In the case of the seed table, there is no previous table, so it shows as a "nil table" in the output. Each endpoint (application:port) in the table should be listed with a line in the output which shows the target name:port or IP-address:port, along with a count of the total number of endpoints. The seed table has three route entries, which reference just two endpoints, so the expected number listed is 2. In addition to listing the endpoint, the connection state is reflected with the open= value. 0 indicates that the connection has not been established; 1 indicates that RMR has attempted to connect. The actual state of the connection is hidden by NNG, and thus this does NOT reflect the true connection state.
Each route table entry is also listed in the output. This should show the message type (mtype), subscription ID (sid) and the number of round robin groups (nrrg) which were found. The refs= is an internal diagnostic indicating the number of internal references to the entry. The key is the hashing key used to map a message type/subscription ID to the route table entry in the in-core hash.
MEID output in verbose level 1
If the seed table contains MEID entries, or MEID updates are sent by the route manager, additional information is written to the standard error. The MEID debugging is a bit more verbose and is illustrated below which shows the previous seed table as the old table.
[DBUG] meid_parse: meid map start found [DBUG] parse_meid_ar: add/replace meid: meid000 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid000 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid001 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid001 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid002 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid002 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid003 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid003 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid004 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid004 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid005 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid005 owned by: control1 state=1 [DBUG] parse_meid_ar: add/replace meid: meid100 owned by: control2 state=1 [DBUG] parse_meid_ar: add/replace meid: meid100 owned by: control2 state=1 [DBUG] parse_meid_ar: add/replace meid: meid101 owned by: control2 state=1 [DBUG] parse_meid_ar: add/replace meid: meid101 owned by: control2 state=1 [DBUG] parse_meid_ar: add/replace meid: meid102 owned by: control2 state=1 [DBUG] parse_meid_ar: add/replace meid: meid102 owned by: control2 state=1 [DBUG] parse_meid_ar: add/replace meid: meid103 owned by: control2 state=1 [DBUG] parse_meid_ar: add/replace meid: meid103 owned by: control2 state=1 [DBUG] end of meid map noticed [DBUG] old route table: [DBUG] RMR route table stats: [DBUG] RMR route table endpoints: [DBUG] RMR rt endpoint: target=app02:4560 open=0 [DBUG] RMR rt endpoint: target=app03:4560 open=0 [DBUG] RMR rtable: 2 known endpoints [DBUG] RMR route table entries: [DBUG] RMR rte: key=ffffffff00000000 mtype= 0 sid= -1 nrrg= 1 refs=2 [DBUG] RMR rte: key=ffffffff00000001 mtype= 1 sid= -1 nrrg= 1 refs=2 [DBUG] RMR rte: key=ffffffff00000002 mtype= 2 sid= -1 nrrg= 2 refs=2 [DBUG] RMR rtable: 3 mt entries in table [DBUG] RMR route table meid map: [DBUG] RMR rtable: 0 meids in map [DBUG] new route table: [DBUG] RMR route table stats: [DBUG] RMR route table endpoints: [DBUG] RMR rt endpoint: target=control1 open=0 [DBUG] RMR rt endpoint: target=control2 open=0 [DBUG] RMR rt endpoint: target=app02:4560 open=0 [DBUG] RMR rt endpoint: target=app03:4560 open=0 [DBUG] RMR rtable: 4 known endpoints [DBUG] RMR route table entries: [DBUG] RMR rte: key=ffffffff00000000 mtype= 0 sid= -1 nrrg= 1 refs=2 [DBUG] RMR rte: key=ffffffff00000001 mtype= 1 sid= -1 nrrg= 1 refs=2 [DBUG] RMR rte: key=ffffffff00000002 mtype= 2 sid= -1 nrrg= 2 refs=2 [DBUG] RMR rtable: 3 mt entries in table [DBUG] RMR route table meid map: [DBUG] RMR meid=meid000 owner=control1 open=0 [DBUG] RMR meid=meid001 owner=control1 open=0 [DBUG] RMR meid=meid002 owner=control1 open=0 [DBUG] RMR meid=meid003 owner=control1 open=0 [DBUG] RMR meid=meid004 owner=control1 open=0 [DBUG] RMR meid=meid005 owner=control1 open=0 [DBUG] RMR meid=meid100 owner=control2 open=0 [DBUG] RMR meid=meid101 owner=control2 open=0 [DBUG] RMR meid=meid102 owner=control2 open=0 [DBUG] RMR meid=meid103 owner=control2 open=0 [DBUG] RMR rtable: 10 meids in map
The output when the MEID updates are parsed show each entry as it is parsed, and then the old and new tables. The new table shows information for each meid along with it's "owner" and the open state which is the same as described previously.
Verbose Level 2 Output
When '2' is placed into the verbose control file, the output is a bit more lengthy. Specifically, when parsing the mse entries the following output is also generated which just describes each record encountered:
[DBUG] start of route table noticed [DBUG] create rte for mtype=1 subid=-1 key=ffffffff00000001 [DBUG] add endpoint ts=1 app03:4560 [DBUG] create rte for mtype=2 subid=-1 key=ffffffff00000002 [DBUG] add endpoint ts=2 app02:4560 [DBUG] add endpoint ts=2 app03:4560 [DBUG] create rte for mtype=0 subid=-1 key=ffffffff00000000 [DBUG] add endpoint ts=0 app02:4560 [DBUG] add endpoint ts=0 app03:4560 [DBUG] end of route table noticed
For the MEID parsing, there is no significant additional information generated (it looks like there is a bug and the parse_meid_ar output in verbose level 1 should really only be generated in level 2).
4 Comments
Chris Lott
Please consider move this valuable material to the doc/ section of your gerrit repo so it's pushed into RTD. Wikis are a fine place to hide material.
Scott Daniels
I've no problem with putting all of the doc into the repo, but....
I want to maintain doc in one place, and I _thought_ the wikis were that place. Maybe that was the case before RTD came along.
Chris Lott
Most teams shirk their documentation responsibilities. You are the rare exception. I think you get to choose your venue. I just happen to dislike wikis, in part because creating anything more complex then a text paragraph is usually a UI nightmare.
Scott Daniels
Thank you for the complement! I hate wikis -- The UI sucks, it's impossible to "reuse" doc without cut/paste which (as you've pointed out) leaves one copy to rot. Now that there is a way to publish, almost automatically, from doc in the repo, I'll start to move the content from here and do it right the second, or is this the third, time :)