Interactive Source Blending (ISB)

The purpose of this article is to introduce the Interactive Source Blending feature available in TREP 3.0 or above. It also demonstrates how to setup this feature and test it with EMA C++ examples.

Overview

TREP currently supports static record augmentation and publisher augmentation features which allows the client to augment and contribute pre-configured fields/values to the cache for items that match configurations file rules. Interactive Source Blending extends these features by providing the ability to blend data from multiple interactive providers without pre-configured fields/values. For example, data from Elektron can be augmented with additional content published by a local interactive publisher.

Figure 1: Interactive Source Blending

Figure 1 illustrates augmenting data from EED (Elektron) with content from an EMA IProvider application. From the picture, ADH 3.x has two routes (route1 and route2).

  • The first route (route1) connects to the primary provider which provides ELEKTRON_DD service from the EED server
  • The second route (route2) connects to the secondary provider which is an augmentation service (DIRECT_FEED) providing the augmenting fields (x, y, and z) for the ELEKTRON_DD service configured in ADH

When items are subscribed, ADH will retrieve the data from both providers so the cache in ADH will contain all fields published by both providers. Therefore, when a consumer application subscribes data from this service (ELEKTRON_DD), it will retrieve all fields including augmented fields from ADH’s cache and merge them into the Refresh message. However, the Update messages from the primary provider and secondary provider will be sent separately. The Update messages from the individual providers will not be merged.

Prerequisite

This article will explain how to configure TREP 3.0 to support this feature and how to run the example in EMA package to test it. Therefore, to follow steps in this article, you need to have the followings:

  • Elektron feed. We will configure ADH 3.0 to connect to this feed as a primary provider
  • TREP 3.0 package (ADH and ADS). You can download TREP 3.0 from the Download Center. It is in Category: MDS – Infra and Products: Advanced Data Hub (ADH), and Advanced Distribution Server (ADS). You need to download version 3.0 or higher
  • Elektron Message API (EMA) C++ package. We will modify and run the example (IProvider100) in the EMA package as a secondary provider. Therefore, you need to have this package and compiler to build and run the examples. You can download EMA package from the Developer Community website

Moreover, you must have an understanding of TREP installation and setup utilities.

In the next section, we will show you how to configure ADH 3.0 to create new routes to the primary and secondary providers and enable the Interactive Source Blending feature.

Configuring TREP 3.0

To follow steps in this section, we expect that you are familiar with TREP installation tools and configurations so we will only explain the relevant configuration parameters for the ADH that relate to the Interactive Source Blending feature. If you are not familiar with TREP installation and configuration, please refer to ADH installation guide in ADH package or contact your local TREP support team.

This section demonstrates the steps required to configure TREP 3.0 to support Interactive Source Blending according to the diagram in figure 1. In summary, we will add two routes to ADH. The first route will connect to the primary provider which supports ELEKTRON_DD service and the second route will connect to the secondary provider which supports the DIRECT_FEED service (an arbitrary name). Both routes will provide the data for ELEKTRON_DD service. Then, we will enable Interactive Source Blending on ELEKTRON_DD service.

Configuring ADH

1. Initialize a configuration session for the ADH

tsetup adh

This is the first step to configure the ADH. It defines and initializes the temporary configuration database for the ADH.

2. Configure the transport and the global source variables

tconfig rmds_adh_rrcp

This command is used to configure the backbone transport (RRCP) for TREP source components and global parameters for all the infrastructure components, including a transport type, data dictionary files, license file and broadcast address.

3. Configure ADH

tconfig adh_rrcp –n <hostname>
  • Specify routeList to route1, route2

This command is used to configure the ADH on the node specified by the hostname parameter. In the routeList parameter, please specify two routes (route1 and route2). Please change hostname parameter to the name of server that the ADH will be installed.

4. Configure the service (ELEKTRON_DD)

tconfig service_edge –s ELEKTRON_DD

From the figure 1, the ADH will support the ELEKTRON_DD service from Elektron feed. Therefore, we use this command to configure a service sourced from an EED.

5. Configure the first route (route1)

tconfig rssl_route –n <hostname> –i route1
  • Set serviceList to ELEKTRON_DD

This command is used to configure the first route (route1) on the node specified by the hostname parameter. The first route will connect to the EED server - so you need to know the hostname or IP address of the EED server, RSSL TCP port listening on the server, and a valid username for connecting to the server. For serviceList parameter, please enter the same service configured in the 4th step, such as ELEKTRON_DD. Please change hostname parameter to the name of server that ADH will be installed.

6. Configure the second route (route2)

tconfig rssl_route –n <hostname> –i route2
  • Set serviceList to ELEKTRON_DD

This command is used to configure the second route (route2) on the node specified by the hostname parameter. The second route will connect to the EMA IProvider example so you need to know the hostname or IP address that runs the example. For the RSSL port, you can use the default TCP port which is rmds_rssl_sink (14002). For serviceList parameter, please enter the same service configured in the forth step, such as ELEKTRON_DD. Please also change hostname parameter to the name of server that ADH is installed on.

7. Configure the server for route1

tconfig server_edge -n <hostname> -i route1 -s ELEKTRON_DD

The command is used to configure the source server for ELEKTRON_DD service on the first route. Please change hostname parameter to the name of server that ADH is installed on.

8. Configure the server for route2

tconfig server_edge -n <hostname> -i route2 -s ELEKTRON_DD

The command is used to configure the source server for ELEKTRON_DD service on the second route. Please change hostname parameter to the name of server that ADH is installed on.

9. Configure source blending

tconfig source_blending -s ELEKTRON_DD
  • Set enableSourceBlending to True

The command is used to configure the Interactive Source Blending for ELEKTRON_DD service. Please set enbleSourceBlending to true to enable this feature for ELEKTRON_DD service.

10. Commit the configurations

tcommit

This command replaces the configuration file (rmds.cnf) with the temporary configuration database and creates a new backup file of the previously committed configuration file. The configuration file will be created at $RMDS_SW/config/local/rmds.cnf

11. Edit rmds.cnf

Finally, we need to modify few default values in rmds.cnf created from previous steps.

  • Change the cacheLocation of ELEKTRON_DD service to ssl. cacheLoction for the Interactive Source Blending service must be ssl
  • Set the maxCache of ELEKTON_DD service on both routes to 5000
  • Change the appServiceName of route2 to DIRECT_FEED which is the service name provided by the EMA IProvider example
  • Change primaryProvider setting on the second route to False. The second route is not a primary provider
*ELEKTRON_DD*cacheLocation : ssl
…
host1*adh*route1.route*ELEKTRON_DD*maxCache : 5000
host1*adh*route2.route*ELEKTRON_DD*maxCache : 5000
…
host1*adh*route2.route*ELEKTRON_DD*appServiceName : DIRECT_FEED
…
host1*adh*route2.route*ELEKTRON_DD*primaryProvider : False

The previous steps configure Interactive Source Blending with the default values. However, you can control the behavior of this feature by modifying the following parameters in the rmds.cnf configuration file.

Parameter Name Description Default Value
enableSourceBlending Enable interactive source blending per-service False
primaryProvider Specify on a per-server basis, the primary versus secondary providers when interactive source blending is enabled True
secondaryRetryTime The amount of time in seconds an item is recalled on secondary providers 120
secondaryRetryLimit The number of times an item is recalled on secondary providers 3
waitForSecondaryResponses Whether interactive source blending will wait for responses from all secondary providers or send the response from the primary immediately True
protectPrimaryFields Whether interactive source blending will protect primary fields from being modified by secondary providers False
blankSecondaryFields Whether interactive source blending will blank fields from secondary providers on server and item outages. protectPrimaryFields must be enabled for this feature False
logSecondaryOpenFailures Whether to log open and outage failures for secondary items False
secondaryBlankRate The number of updates which interactive source blending will send per second per thread to blank secondary field values 1000

Sample Configurations

The followings show the sample Interactive Source Blending configuration in rmds.cnf after completing the above steps.

...
*adh*ELEKTRON_DD*blankSecondaryFields : False
*adh*ELEKTRON_DD*enableSourceBlending : True
*adh*ELEKTRON_DD*logSecondaryOpenFailures : False
*adh*ELEKTRON_DD*protectPrimaryFields : False
*adh*ELEKTRON_DD*secondaryBlankRate : 1000
*adh*ELEKTRON_DD*secondaryRetryLimit : 3
*adh*ELEKTRON_DD*secondaryRetryTime : 120
*adh*ELEKTRON_DD*waitForSecondaryResponses : True
...
host1*adh*route1.route*ELEKTRON_DD*primaryProvider : True
host1*adh*route2.route*ELEKTRON_DD*primaryProvider : False
...

After configuring the ADH, the next step is to configure the ADS to distribute ELEKTRON_DD service to the consumer.

Configuring ADS

Interactive Source Blending is a feature of the ADH and therefore the ADS does not require any special configuration for this feature. Please follow steps in the ADS installation guide to install and configure ADS as per normal.

Modifying EMA IProvider to provide augmenting fields

At this point, TREP 3.0 has been installed and configured properly to test Interactive Source Blending feature. Next, we will modify the example in EMA C++ package to test this feature. We use the IProvider100 example in Ema\Examples\Training\IProvider\100_Series_Examples\100MarketPriceStreaming directory as a secondary provider because this example is simple and easy to modify.

For the purpose of this exercise, we will use the RIC THB= and add augment it with a few extra FIDs. We will modify the app to provide data for the following additional fields: 92 (PRV_YRHIGH), 93 (PRV_YRLOW), 94 (LIFE_HIGH), and 95 (LIFE_LOW).

Modifying 100_MarketPrice_Streaming (IProvider100)

To demonstrate Interactive Source Blending feature, IProvider100 example is modified to provide augmenting fields. This example demonstrates the basic usage of the EMA library in providing OMM MarketPrice data to the Thomson Reuters Advanced Distribution Hub. It illustrates how to create and publish a single OMM streaming item. Originally, this example provides data for the following fields: 22 (BID), 25 (ASK), 30 (BIDSIZE), and 31 (ASKSIZE). We will modify it to provide data for the following fields: 92, 93, 94, and 95 which are not available in the template of our example THB= RIC. The definitions for these fields are:

PRV_YRHIGH "PREV YEAR HIGH"        92  NULL        PRICE              17  REAL64           7
!
! The highest value during the previous calendar year.
!
PRV_YRLOW  "PREV YEAR LOW"         93  NULL        PRICE              17  REAL64           7
!
! The lowest value during the previous calendar year.
!
LIFE_HIGH  "LIFETIME HIGH"         94  NULL        PRICE              17  REAL64           7
!
! The highest value ever achieved by this issue.
!
LIFE_LOW   "LIFETIME LOW"          95  NULL        PRICE              17  REAL64           7
!
! The lowest value ever achieved by this issue.
!

First, we will modify AppClient::processMarketPriceReqest method in the example to publish these fields in the refresh message. This method is used to handle the MarketPrice request and create a field list for the request message. We have modified it to create a field list with these fields.

IProvider.cpp

void AppClient::processMarketPriceRequest( const ReqMsg& reqMsg, const OmmProviderEvent& event )
{
    if ( itemHandle != 0 )
    {
        processInvalidItemRequest(reqMsg, event);
        return;
    }

    event.getProvider().submit( RefreshMsg().name( reqMsg.getName() ).serviceName( reqMsg.getServiceName() ).solicited( true ).
        state( OmmState::OpenEnum, OmmState::OkEnum, OmmState::NoneEnum, "Refresh Completed" ).
        payload( FieldList().
            addReal( 92, 3990, OmmReal::ExponentNeg2Enum ).
            addReal( 93, 3994, OmmReal::ExponentNeg2Enum ).
            addReal( 94, 9, OmmReal::Exponent0Enum ).
            addReal( 95, 19, OmmReal::Exponent0Enum ).
            complete() ).
        complete(), event.getHandle() );

    itemHandle = event.getHandle();
}

Next, we will modify the update message to update the values of fields of just two of the augmented fields: 92 and 93. The code that sends the update messages is in the main function.

IProvider.cpp

int main( int argc, char* argv[] )
{
    try
    {
        AppClient appClient;

        OmmProvider provider( OmmIProviderConfig().port( "14002" ), appClient );

        while ( itemHandle == 0 ) sleep(1000);

        for ( Int32 i = 0; i < 60; i++ )
        {
            provider.submit( UpdateMsg().payload( FieldList().
                    addReal( 92, 3391 + i, OmmReal::ExponentNeg2Enum ).
                    addReal( 93, 10 + i, OmmReal::Exponent0Enum ).
                    complete() ), itemHandle );

            sleep( 1000 );
        }
    }
    catch ( const OmmException& excp )
    {
        cout << excp << endl;
    }

    return 0;
}

To run the example, the configuration file (EmaConfig.xml) is required. The configuration file defines the source directory used by the example. The default configuration supports DIRECT_FEED service which corresponds to the appServiceName configuration of the second route (route2) in ADH.

<Directory>
    <Name value="Directory_2"/>
    <Service>
        <Name value="DIRECT_FEED"/>
        <InfoFilter>
            <ServiceId value="1"/>
            <DictionariesProvided>
                <DictionariesProvidedEntry value="Dictionary_3"/>
            </DictionariesProvided>

            <DictionariesUsed>
                <DictionariesUsedEntry value="Dictionary_3"/>
            </DictionariesUsed>
…
…
</Directory>

Test Interactive Source Blending

Now, we are ready to test this feature. To test it, we use the Consumer100 example in Ema\Examples\Training\Consumer\100_Series_Examples\100MarketPriceStreaming directory. We have modified Consumer100 to connect to ADS server and subscribe to THB= from ELEKTRON_DD service.

Consumer.cpp

int main( int argc, char* argv[] )
{ 
    try { 
        AppClient client;
        OmmConsumer consumer( OmmConsumerConfig().host( "hostnameOfADS:14002" ).username( "user" ) );
        consumer.registerClient( ReqMsg().serviceName( "ELEKTRON_DD" ).name( "THB=" ), client );
        sleep( 60000 );                // API calls onRefreshMsg(), onUpdateMsg(), or onStatusMsg()
    } catch ( const OmmException& excp ) {
        cout << excp << endl;
    }
    return 0;
}

 

Please change hostnameOfADS to the IP address or host name of ADS. Then, please follow these steps to test Interactive Source Blending.

  1. Run 100_MarketPrice_Streaming (IProvider100)
  2. Start ADH 3.0 to connect to the EED server and 100_MarketPrice_Streaming (IProvider100) If the settings are correct, you will see that ELEKTRON_DD service is up with two source servers via adhmon.

  3. Start ADS 3.0. If the settings are correct, you will see that the ELEKTRON_DD service is up via adsmon.
  4. Run 100_MarketPrice_Streaming (Consumer100) to connect to ADS and subscribe to THB= from ELEKTRON_DD service

After running 100_MarketPrice_Streaming (Consumer100), the output shows the refresh of THB= contains the augmenting FIDs: 92, 93, 94, and 95. Moreover, the consumer will receive the update of FIDs: 92 and 93 every second.

RefreshMsg
    streamId="5"
    domain="MarketPrice Domain"
    Solicited
    RefreshComplete
    ClearCache
    state="Open / Ok / None / 'All is well'"
    itemGroup="00 01"
    permissionData="03 01 01 69 33 C0"
    qos="RealTime/TickByTick"
    seqNum="24398"
    name="THB="
    nameType="1"
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList FieldListNum="99" DictionaryId="1"
            FieldEntry fid="1" name="PROD_PERM" dataType="UInt" value="6933"
            FieldEntry fid="2" name="RDNDISPLAY" dataType="UInt" value="153"
            FieldEntry fid="3" name="DSPLY_NAME" dataType="Rmtes" value="BROKER      GFX"
…
            FieldEntry fid="79" name="HSTCLSDATE" dataType="Date" value="24 MAY
2017"
            FieldEntry fid="90" name="YRHIGH" dataType="Real" value="36.00"
            FieldEntry fid="91" name="YRLOW" dataType="Real" value="34.18"
            FieldEntry fid="92" name="PRV_YRHIGH" dataType="Real" value="39.90"
            FieldEntry fid="93" name="PRV_YRLOW" dataType="Real" value="39.94"
            FieldEntry fid="94" name="LIFE_HIGH" dataType="Real" value="9"
            FieldEntry fid="95" name="LIFE_LOW" dataType="Real" value="19"
            FieldEntry fid="105" name="BCKGRNDPAG" dataType="Rmtes" value="    "
…
            FieldEntry fid="14208" name="BID_HR_MS" dataType="Time" value="10:00
:00:376:000:000"
        FieldListEnd
    PayloadEnd
RefreshMsgEnd

UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    seqNum="0"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="92" name="PRV_YRHIGH" dataType="Real" value="33.91"
            FieldEntry fid="93" name="PRV_YRLOW" dataType="Real" value="10"
        FieldListEnd
    PayloadEnd
UpdateMsgEnd

UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    seqNum="0"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="92" name="PRV_YRHIGH" dataType="Real" value="33.92"
            FieldEntry fid="93" name="PRV_YRLOW" dataType="Real" value="11"
        FieldListEnd
    PayloadEnd
UpdateMsgEnd

UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    DoNotConflate
    seqNum="24430"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="3284" name="EURO_NETCH" dataType="Real" value="-0.15"
        FieldListEnd

    PayloadEnd
UpdateMsgEnd


UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    DoNotConflate
    seqNum="24462"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="114" name="BID_NET_CH" dataType="Real" value="-0.11"
            FieldEntry fid="372" name="IRGPRC" dataType="Real" value="-0.32"
        FieldListEnd
    PayloadEnd
UpdateMsgEnd

UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    seqNum="0"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="92" name="PRV_YRHIGH" dataType="Real" value="33.93"
            FieldEntry fid="93" name="PRV_YRLOW" dataType="Real" value="12"
        FieldListEnd
    PayloadEnd
UpdateMsgEnd

The output shows that the 100_MarketPrice_Streaming (Consumer100) retrieves the refresh message that contains the augmenting FIDs: 92, 93, 94, and 95. It also retrieves the update of FIDs: 92 and 93 from the secondary provider: 100_MarketPrice_Streaming (IProvider100). Moreover, from the output, the update messages from the primary and secondary providers are retrieved separately.

References

  • Advanced Data Hub 3.0 : Software Installation Manual in ADH 3.0 package. You can download it from the Download Center. It is in Category: MDS – Infra and Products: Advanced Data Hub (ADH)
  • Advanced Distribution Server 3.0 : Software Installation Manual in ADS 3.0 package. You can download it from the Download Center. It is in Category: MDS – Infra and Products: Advanced Distribution Server (ADS)
  • 100_MarketPrice_Streaming (IProvider100) and 100_MarketPrice_Streaming (Consumer100) in the Elektron SDK – C/C++ (EMA) package. The package is available at the developer community.