Document status | DRAFT |
---|---|
Document owner | |
Owner Details | +972-54-2891166, ab600d@intl.att.com |
Table of Contents | ||
---|---|---|
|
Abbreviation
RM | Routing Manager |
E2T | E2 Terminator |
KA | Keep Alive |
General
In R3 the The RIC contains several E2T Instances, each one of them manages several RANs. E2M is the one which manages them. It means:
When E2T Instance starts running and finished its Initialization, it sends RMR message (E2 Initialization) to E2M. E2M Inform to RM and Registers it in the E2M DB.
When a new RAN is requested to sends the RIC Setup Request, E2M chooses the E2M with the least # of RANs, Inform to RM the association and associate this RAN to this this E2T Instance in DB
When RAN is stopped, E2M dissociation it from the E2T InstanceLost connection, E2M Inform to RM the disassociation and disassociate this RAN to this E2T Instance in DB
E2M Handle Keep Alive (KA) RMR handshake with the E2M Instances. When an Instance doesn’t respond for period of time to KA Request from E2M, E2M divides its Associated RANs to other living E2T Instances and kills (Through K8S) this instance. Probably E2M will receive a new E2 Initialization to E2M. E2M Registers it in the E2M DB, signs all the RANs associated with it as Disconnected, delete the E2T address from DB and inform RM the disassociation.
For any new or dead E2M Instance, and for any RAN Association / dissociationdisassociation, E2M informs Routing ManagerRM. Routing Manager RM updates the Routing tables and publish an updated RMR to all RIC.When there are no living E2T instance, the RAN which should connected are treated as Orphans. E2M should associate them to new living E2T instance when available
- When existing E2T Instance restarts and sends RMR message (E2 Initialization) to E2M. E2M Inform to RM the disassociation of all its RANs and disassociate them in DB
- If the RM isn't accessible, "positive" scenario (New E2T Address, new RAN Setup) are not executed, waiting for additional E2T will sends its Address or the RAN renew its Setup. But "negative" scenario (Like Lost connection, KA kills E2T, Existing E2T restarts) are executed. When RM will restarts and query E2M, it will sync with the E2T and RANs association and publish RMR to all RIC.
Keep Alive
Each Configurable time (100 500 Milliseconds currently) E2M sends broadcast KA Request to all living E2T Instances through the RMR.
Every E2T Instance which gets it, responds with KA Response.
If any E2T Instance doesn’t respond for a configurable time (400 1500 Milliseconds currently) - E2M “kills” it:
E2M asks the K8S to kill the E2T Instance.
E2M divides disassociate the RANs associated with this “dying” instance to other living E2T Instances. If not found - they are treated as Orphans.E2M Updates the Routing Managers about the association of the RANs to other E2T Instances, about the Orphans (if any) , signs them as disconnected and Updates the RM about the disassociation of the RANs and about the deletion of the dead E2T Instance.
If success - E2M stores in its DB the new associations and sends Setup for this RANs through the updated RMR
There is some logic how to sync between parallel flows - Killing an E2T Instance parallel to E2 Initialize arrives. See below.
...
As described, when E2T Instance starts running and finished its Initialization, it sends RMR message (E2 Initialization) to E2M.
If this is a new address (IP) then -
E2M updates the Routing Manager there is new E2T Instance
If success - E2M Registers it in the E2M DB. If fail - E2M does nothing. E2T will sends another time E2 Initialize to E2M if no KA Request arrives during the 500 Milliseconds after the previous E2 Initialize.If there are orphans - E2M will handle them here (See below)
If this is an existing E2T Instance, then
E2M re-sends Setup for this RANs through the RMR. The RMR still contains the association of the RANs to this E2T Instancedisassociate the RANs associated with this "restart" instance, signs them as disconnected and Updates the RM about the disassociation of the RANs
There is some logic how to sync between parallel flows - Killing an E2T Instance parallel to E2 Initialize arrives.
...
E2M manages States that E2T Instance can be in the DB:
Active (normal state). Selector mechanism can choose it.
To be deleted - KA moves it to this state in DB the moment is understand it is dead
. Selector mechanism can’t choose it. It is
stay instayed in this state until
Routing manager success (in this case the entire entry is deleted) or failure, which is moved to the next stepRouting Manager Failure
In this state Selector mechanism can’t choose it. In this state, either KA will try, in the next 100 milliseconds interval to update Routing manager or E2T initialize move it to Active.E2T shutdown process completed.
In case E2T initialize arrives and this E2T is in “To be Deleted” state - Don’t do anything. E2T will send another E2T initialize in case no KA during 300 milliseconds after E2 Initialize.
current State / trigger | State = Active | State =To be Deleted |
---|
KA decide to kill E2T instance |
|
---|
Divide RANs in memory to others instances;
Update Routing Manager:
If Success - store in DB the new Mapping RANs to Instances; Sends Setup(s);
If Fail - Move to Routing Manager Failure State;
| (Transient State, should not occurs)
|
Do all the steps Just like in Active
E2 Initialize |
|
---|
|
|
Don’t update Routing Manager
|
Send setup to all RANs related to this E2T
KA Response |
|
|
---|
E2M Restart |
|
|
---|
Association RAN to Instance
There are 2 places where E2M search for a living E2T Instance to associate it with RAN:
When a Setup Request from Dashboard arrives. This is for New RAN or for existing RAN which isn’t associated with any E2T Instance
When KA kills an E2T Instance and divides its RANs to other living E2T Instances
E2M chooses the E2T Instance with the least # of RANs.
Anyhow, until the Routing Manager isn’t synced with this association, it isn’t stored in E2M DB
Setup Request from Dashboard which fails either in choosing any E2T Instance or updating the Routing Manager, either for New RAN or existing which isn’t associate with any E2T Instance , returns special rejection error is telling the Dashboard to try later.
In case KA failure in choosing the E2T to the RAN, the RANs are inserted to Orphan List. See below how E2M handle with Orphan List. It is usually when the last E2T Instance fails.
Dissociation RAN
The only place where RAN is associated with E2T address is when Setup Request arrives.
It is stored in DB - in 2 places
- This RAN is added to the list of RANs served in the E2T Object
- The E2T Address is added to the RAN Object
- The RM is informed on this Association between RAN and the E2T address
- But also if RM isn't accessible, E2M doesn't rollback nor sending Setup Failure to remote RAN - since the simple way to ask the E2T to send Setup Failure not through the normal MEID flow is quite complicated.
Dissociation RAN
These are the
RAN which is already associated with E2T Instance, stays with this association, even if it lost connection. E2M will try to re-new the Setup until Max Retries (Configurable) reached.
There are 2 places where E2M dissociate RAN:
When the Max Retries of sending Setup arrives, as a result of Lost Connection. The RAN is disassociate.
When a request to Red Button arrives (See below)
Lost Connection arrives from E2T
When existing E2T Instance restart and E2M receives E2T Initialize message
- When KA decides to kill an E2T Instance
- When Red Button tears down all RANs.
- In all these places - E2T address is cleared from the RAN(s) object(s), the RAN(s) removed from the list of RANs served in the E2T Object
- The RM is informed
Interface to Routing Manager
Routing Manager RM updates the Routing Table in the following ways:
Add or Delete E2T Instance - There are 2 RMR messages that are “Broadcast”:
KA Request
Clear all Connection (“Red Button”)
Associate RAN(s) to E2T Instance (it can overrides previous association)
Delete RAN association
E2M updates the Routing Manager RM in the following flows:
When a new E2T Instance Initialized.
When a a new RAN is added Setup Request arrived from the RAN and associated to existing E2T Instance
When KA kills existing E2T Instance - in this update it also associates disassociates list of RANs to other existing E2T Instance (Or dissociate list of RANs since it was the last E2T Instance) and also delete E2T Instance
When RAN lost connection and it exceed the Max Retry.connection
When a Red Button is requested - E2M dissociate all RANs
When existing E2T Instance initialized, it is the first, and list of Orphans “are waiting” for it - E2M associates list of RANs to this E2T Instance it disassociates list of RANs served by this Instance
- Also, when RM initialized, it queries E2M on the mapping between E2T and RANs and update the RIC with the new RMR. See E2 Manager R3 Interface Design#GetAllE2TRequest
There are 4 REST APIs Routing Manager published. U can read their Swagger at Routing Manager Swagger
...
When Red Button request from Dashboard arrives, E2M inform Routing Manager to disassociate all RANs
E2M sends “broadcast” the Close all connections to all E2T Instances.
Orphans
As described above, when last E2T Instance fails, its RAN are entering the Orphan List.
The KA Job, after sending the KA request and after checking there is no any E2T which doesn’t respond and need to be killed. examines the Orphan List is not empty and there is a E2T Instance to associate them. If yes, it is updating the Routing Manager and if OK - sends Setup on each RAN
Also, on Red Button, we empty the Orphan List.
RAN State, Association and Orphan
To summarize, RAN can be in the following:
Associated with E2 Instance
RAN can be in the following state:
Connected
Connected Failure
Disconnect, but with Count of Retry less then Max, Usually this is in case RMR Failure
Associated with the Orphan List
RAN should be in a Disconnect State, while Count of Retry is 0.
Disassociated
RAN can be in the following state:
Shut Down
Disconnect, but with Count of Retry equal to Max