Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Document status
DRAFT
Document owner
Owner Details

+972-54-2891166,   ab600d@intl.att.com


Table of Contents
absoluteUrltrue

Abbreviation



RMRouting Manager
E2TE2 Terminator
KAKeep Alive

General

The RIC contains several E2T Instances, each one of them manages several RANs. E2M is the one which manages them. It means:

...

Keep Alive

  • Each Configurable time (100 500 Milliseconds currently) E2M sends broadcast KA Request to all living E2T Instances through the RMR.

  • Every E2T Instance which gets it, responds with KA Response.

  • If any E2T Instance doesn’t respond for a configurable time (400 1500 Milliseconds currently) - E2M “kills” it:

    • E2M asks the K8S to kill the E2T Instance.

    • E2M divides disassociate the RANs associated with this “dying” instance to other living E2T Instances. If not found - they are treated as Orphans.E2M Updates the Routing Managers about the association of the RANs to other E2T Instances, about the Orphans (if any) , signs them as disconnected and Updates the RM about the disassociation of the RANs and about the deletion of the dead E2T Instance.

    • If success - E2M stores in its DB the new associations and sends Setup for this RANs through the updated RMR

  • There is some logic how to sync between parallel flows - Killing an E2T Instance parallel to E2 Initialize arrives. See below.

...

  • As described, when E2T Instance starts running and finished its Initialization, it sends RMR message (E2 Initialization) to E2M.

  • If this is a new address (IP) then -

    • E2M updates the Routing Manager there is new E2T Instance

    • If success - E2M Registers it in the E2M DB. If fail - E2M does nothing. E2T will sends another time E2 Initialize to E2M if no KA Request arrives during the 500 Milliseconds after the previous E2 Initialize.

  • If

    there are orphans - E2M will handle them here (See below)

    If this is an existing E2T Instance, then

    • E2M re-sends Setup for this RANs through the RMR. The RMR still contains the association of the RANs to this E2T Instancedisassociate the RANs associated with this "restart" instance, signs them as disconnected and Updates the RM about the disassociation of the RANs

  • There is some logic how to sync between parallel flows - Killing an E2T Instance parallel to E2 Initialize arrives.

...

E2M manages States that E2T Instance can be in the DB:

  • Active (normal state). Selector mechanism can choose it.

  • To be deleted KA moves it to this state in DB the moment is understand it is dead

    . Selector mechanism can’t choose it

    . It is

    stay in

    stayed in this state until

    Routing manager success (in this case the entire entry is deleted) or failure, which is moved to the next step

    Routing Manager Failure

    In this state Selector mechanism can’t choose it. In this state, either KA will try, in the next 100 milliseconds interval to update Routing manager or E2T initialize move it to Active.

    E2T shutdown process completed. 

In case E2T initialize arrives and this E2T is in “To be Deleted” state - Don’t do anything. E2T will send another E2T initialize in case no KA during 300 milliseconds after E2 Initialize.


current State / trigger

State = Active

State =To be Deleted

State = Routing Manager Failure

KA decide to kill E2T instance

  • Move to “To be deleted” State;
  • Try to find POD in K8S and if found kill it; Even if fails - continue.
  • Divide RANs in memory to others instances;

  • Update Routing Manager:

  • If Success - store in DB the new Mapping RANs to Instances; Sends Setup(s);

  • If Fail - Move to Routing Manager Failure State;

    • Disassociate all RANs, Delete the E2T Address at all.
    • Update RM, if fail – error but continue
    • In order to distinguish between 2 sequences KA cycle, KA updates the Timestamp when it moves the RAN to  this state, and sees it wasn't stayed in this state too long. It is called E2 Instance Deletion Timeout MS (Micro seconds) 

    (Transient State, should not occurs)

    • Do all the steps Just like in Active (Previous Cell), except updating the state which is already “To be Deleted”.

    Do all the steps Just like in Active

    E2 Initialize

    • Don’t update

    Routing Manager
    • RM

    • Disassociate all RANs and inform RM

    Send setup to all RANs related to this E2T
    • Ignore: E2T will send another E2 Initialize if no KA Request after previous E2 Initialize for some times.

  • Don’t update Routing Manager

  • Move to Active State, update the

    KA

    Timestamp to NOW.
  • Send setup to all RANs related to this E2T

  • KA

    Response

    • Update Timestamp

    • Ignore. This Instance is about to deleted

    E2M Restart

    • Update Timestamp
    • ?????
    Ignore. This Instance is about to deleted

    Association RAN to Instance

    • There are 2 places where E2M search for a living E2T Instance to associate it with RAN:

      1. When a Setup Request from Dashboard arrives. This is for New RAN or for existing RAN which isn’t associated with any E2T Instance

      2. When KA kills an E2T Instance and divides its RANs to other living E2T Instances

    • E2M chooses the E2T Instance with the least # of RANs.

    • Anyhow, until the Routing Manager isn’t synced with this association, it isn’t stored in E2M DB

    • Setup Request from Dashboard which fails either in choosing any E2T Instance or updating the Routing Manager, either for New RAN or existing which isn’t associate with any E2T Instance , returns special rejection error is telling the Dashboard to try later.

    • In case KA failure in choosing the E2T to the RAN, the RANs are inserted to Orphan List. See below how E2M handle with Orphan List. It is usually when the last E2T Instance fails.

    Dissociation RAN

    • The only place where RAN is associated with E2T address is when Setup Request arrives.

    • It is stored in DB -  in 2 places

    1. This RAN is added to the list of RANs served in the E2T Object
    2. The E2T Address is added to the RAN Object
    • The RM is informed on this Association between RAN and the E2T address
    • But also if RM isn't accessible, E2M doesn't rollback nor sending Setup Failure to remote RAN - since the simple way to ask the E2T to send Setup Failure not through the normal MEID flow is quite complicated.

    Dissociation RAN

    • These are the

    • RAN which is already associated with E2T Instance, stays with this association, even if it lost connection. E2M will try to re-new the Setup until Max Retries (Configurable) reached.

    • There are 2 places where E2M dissociate RAN:

      1. When the Max Retries of sending Setup arrives, as a result of Lost Connection. The RAN is disassociate.

      2. When a request to Red Button arrives (See below)

      3. Lost Connection arrives from E2T

      4. When existing E2T Instance restart and E2M receives E2T Initialize message

      5. When KA decides to kill an E2T Instance
      6. When Red Button tears down all RANs.
    • In all these places - E2T address is cleared from the RAN(s) object(s), the RAN(s) removed from the list of RANs served in the E2T Object
    • The RM is informed

    Interface to Routing Manager

    • Routing Manager RM updates the Routing Table in the following ways:

      1. Add or Delete E2T Instance - There are 2 RMR messages that are “Broadcast”:

        1. KA Request

        2. Clear all Connection (“Red Button”)

      2. Associate RAN(s) to E2T Instance (it can overrides previous association)

      3. Delete RAN association

    • E2M updates the Routing Manager RM in the following flows:

      1. When a new E2T Instance Initialized.

      2. When a new RAN is added a Setup Request arrived from the RAN and associated to existing E2T Instance

      3. When KA kills existing E2T Instance - in this update it also associates list of RANs to other existing E2T Instance (Or dissociate disassociates list of RANs since it was the last E2T Instance) and also delete E2T Instance

      4. When RAN lost connection and it exceed the Max Retry.connection  

      5. When a Red Button is requested - E2M dissociate all RANs

      6. When existing E2T Instance initialized, it is the first, and list of Orphans “are waiting” for it - E2M associates list of RANs to this E2T Instance it disassociates list of RANs served by this Instance

    • Also, when RM initialized, it queries E2M on the mapping between E2T and RANs and update the RIC with the new RMR.  See E2 Manager R3 Interface Design#GetAllE2TRequest
    • There are 4 REST APIs Routing Manager published. U can read their Swagger at Routing Manager Swagger

    ...

    • When Red Button request from Dashboard arrives, E2M inform Routing Manager to disassociate all RANs

    • E2M sends “broadcast” the Close all connections to all E2T Instances.

    Orphans

    • As described above, when last E2T Instance fails, its RAN are entering the Orphan List.

    • The KA Job, after sending the KA request and after checking there is no any E2T which doesn’t respond and need to be killed. examines the Orphan List is not empty and there is a E2T Instance to associate them. If yes, it is updating the Routing Manager and if OK - sends Setup on each RAN

    • Also, on Red Button, we empty the Orphan List.

    RAN State, Association and Orphan

    To summarize, RAN can be in the following:

    Associated with E2 Instance

    RAN can be in the following state:

    • Connected

    • Connected Failure

    • Disconnect, but with Count of Retry less then Max, Usually this is in case RMR Failure

    Associated with the Orphan List

    RAN should be in a Disconnect State, while Count of Retry is 0.

    Disassociated

    RAN can be in the following state:

    • Shut Down

    • Disconnect, but with Count of Retry equal to Max