ARTICLE

Introduction to Interactive Source Blending (ISB) feature in ADH 3.0 or above using EMA C++

July 16,2017

Jirapongse Phuriphanvichai

Interactive Source Blending (ISB)

The purpose of this article is to introduce the Interactive Source Blending feature available in ADH 3.0 or above. It also demonstrates how to setup this feature and test it with EMA C++ examples.

Overview

Refinitiv Real-Time Distribution System currently supports static record augmentation and publisher augmentation features which allows the client to augment and contribute pre-configured fields/values to the cache for items that match configurations file rules. Interactive Source Blending extends these features by providing the ability to blend data from multiple interactive providers without pre-configured fields/values. For example, Real-Time data can be augmented with additional content published by a local interactive publisher.

Figure 1: Interactive Source Blending

Figure 1 illustrates augmenting data from EED (Real-Time Feed) with content from an EMA IProvider application. From the picture, ADH 3.x has two routes (route1 and route2).

The first route (route1) connects to the primary provider which provides ELEKTRON_DD service from the EED server

The second route (route2) connects to the secondary provider which is an augmentation service (DIRECT_FEED) providing the augmenting fields (x, y, and z) for the ELEKTRON_DD service configured in ADH

When items are subscribed, ADH will retrieve the data from both providers so the cache in ADH will contain all fields published by both providers. Therefore, when a consumer application subscribes data from this service (ELEKTRON_DD), it will retrieve all fields including augmented fields from ADH’s cache and merge them into the Refresh message. However, the Update messages from the primary provider and secondary provider will be sent separately. The Update messages from the individual providers will not be merged.

Prerequisite

This article will explain how to configure ADH and ADS 3.0 to support this feature and how to run the example in EMA package to test it. Therefore, to follow steps in this article, you need to have the followings:

Real-Time feed. We will configure ADH 3.0 to connect to this feed as a primary provider

ADH and ADS packages. You can download ADH and ADS from the Download Center. It is in Category: MDS – Infra and Products: Advanced Data Hub (ADH), and Advanced Distribution Server (ADS). You need to download version 3.0 or higher

Enterprise Message API (EMA) C++ package. We will modify and run the example (IProvider100) in the EMA package as a secondary provider. Therefore, you need to have this package and compiler to build and run the examples. You can download EMA package from the Developer Community website

Moreover, you must have an understanding of ADH and ADS installation and setup utilities.

In the next section, we will show you how to configure ADH 3.0 to create new routes to the primary and secondary providers and enable the Interactive Source Blending feature.

Configuring ADH 3.0

To follow steps in this section, we expect that you are familiar with ADH installation tools and configurations so we will only explain the relevant configuration parameters for the ADH that relate to the Interactive Source Blending feature. If you are not familiar with ADH installation and configuration, please refer to ADH installation guide in ADH package or contact your local ADH support team.

This section demonstrates the steps required to configure ADH 3.0 to support Interactive Source Blending according to the diagram in figure 1. In summary, we will add two routes to ADH. The first route will connect to the primary provider which supports ELEKTRON_DD service and the second route will connect to the secondary provider which supports the DIRECT_FEED service (an arbitrary name). Both routes will provide the data for ELEKTRON_DD service. Then, we will enable Interactive Source Blending on ELEKTRON_DD service.

Configuring ADH

1. Initialize a configuration session for the ADH

    	
            tsetup adh

This is the first step to configure the ADH. It defines and initializes the temporary configuration database for the ADH.

2. Configure the transport and the global source variables

    	
            tconfig rmds_adh_rrcp

This command is used to configure the backbone transport (RRCP) for source components and global parameters for all the infrastructure components, including a transport type, data dictionary files, license file and broadcast address.

3. Configure ADH

    	
            tconfig adh_rrcp –n <hostname>

Specify routeList to route1, route2

This command is used to configure the ADH on the node specified by the hostname parameter. In the routeList parameter, please specify two routes (route1 and route2). Please change hostname parameter to the name of server that the ADH will be installed.

4. Configure the service (ELEKTRON_DD)

    	
            tconfig service_edge –s ELEKTRON_DD

From the figure 1, the ADH will support the ELEKTRON_DD service from Real-Time feed. Therefore, we use this command to configure a service sourced from an EED.

5. Configure the first route (route1)

    	
            tconfig rssl_route –n <hostname> –i route1

Set serviceList to ELEKTRON_DD

This command is used to configure the first route (route1) on the node specified by the hostname parameter. The first route will connect to the EED server - so you need to know the hostname or IP address of the EED server, RSSL TCP port listening on the server, and a valid username for connecting to the server. For serviceList parameter, please enter the same service configured in the 4th step, such as ELEKTRON_DD. Please change hostname parameter to the name of server that ADH will be installed.

6. Configure the second route (route2)

    	
            tconfig rssl_route –n <hostname> –i route2

Set serviceList to ELEKTRON_DD

This command is used to configure the second route (route2) on the node specified by the hostname parameter. The second route will connect to the EMA IProvider example so you need to know the hostname or IP address that runs the example. For the RSSL port, you can use the default TCP port which is rmds_rssl_sink (14002). For serviceList parameter, please enter the same service configured in the forth step, such as ELEKTRON_DD. Please also change hostname parameter to the name of server that ADH is installed on.

7. Configure the server for route1

    	
            tconfig server_edge -n <hostname> -i route1 -s ELEKTRON_DD

The command is used to configure the source server for ELEKTRON_DD service on the first route. Please change hostname parameter to the name of server that ADH is installed on.

8. Configure the server for route2

    	
            tconfig server_edge -n <hostname> -i route2 -s ELEKTRON_DD

The command is used to configure the source server for ELEKTRON_DD service on the second route. Please change hostname parameter to the name of server that ADH is installed on.

9. Configure source blending

    	
            tconfig source_blending -s ELEKTRON_DD

Set enableSourceBlending to True

The command is used to configure the Interactive Source Blending for ELEKTRON_DD service. Please set enbleSourceBlending to true to enable this feature for ELEKTRON_DD service.

10. Commit the configurations

    	
            tcommit

This command replaces the configuration file (rmds.cnf) with the temporary configuration database and creates a new backup file of the previously committed configuration file. The configuration file will be created at $RMDS_SW/config/local/rmds.cnf

11. Edit rmds.cnf

Finally, we need to modify few default values in rmds.cnf created from previous steps.

Change the cacheLocation of ELEKTRON_DD service to ssl. cacheLoction for the Interactive Source Blending service must be ssl
Set the maxCache of ELEKTON_DD service on both routes to 5000
Change the appServiceName of route2 to DIRECT_FEED which is the service name provided by the EMA IProvider example
Change primaryProvider setting on the second route to False. The second route is not a primary provider

    	
            *ELEKTRON_DD*cacheLocation : ssl
…
host1*adh*route1.route*ELEKTRON_DD*maxCache : 5000
host1*adh*route2.route*ELEKTRON_DD*maxCache : 5000
…
host1*adh*route2.route*ELEKTRON_DD*appServiceName : DIRECT_FEED
…
host1*adh*route2.route*ELEKTRON_DD*primaryProvider : False

The previous steps configure Interactive Source Blending with the default values. However, you can control the behavior of this feature by modifying the following parameters in the rmds.cnf configuration file.

Parameter Name	Description	Default Value
enableSourceBlending	Enable interactive source blending per-service	FALSE
primaryProvider	Specify on a per-server basis, the primary versus secondary providers when interactive source blending is enabled	TRUE
secondaryRetryTime	The amount of time in seconds an item is recalled on secondary providers	120
secondaryRetryLimit	The number of times an item is recalled on secondary providers	3
waitForSecondaryResponses	Whether interactive source blending will wait for responses from all secondary providers or send the response from the primary immediately	TRUE
protectPrimaryFields	Whether interactive source blending will protect primary fields from being modified by secondary providers	FALSE
blankSecondaryFields	Whether interactive source blending will blank fields from secondary providers on server and item outages. protectPrimaryFields must be enabled for this feature	FALSE
logSecondaryOpenFailures	Whether to log open and outage failures for secondary items	FALSE
secondaryBlankRate	The number of updates which interactive source blending will send per second per thread to blank secondary field values	1000

Sample Configurations

The followings show the sample Interactive Source Blending configuration in rmds.cnf after completing the above steps.

    	
            ...
*adh*ELEKTRON_DD*blankSecondaryFields : False
*adh*ELEKTRON_DD*enableSourceBlending : True
*adh*ELEKTRON_DD*logSecondaryOpenFailures : False
*adh*ELEKTRON_DD*protectPrimaryFields : False
*adh*ELEKTRON_DD*secondaryBlankRate : 1000
*adh*ELEKTRON_DD*secondaryRetryLimit : 3
*adh*ELEKTRON_DD*secondaryRetryTime : 120
*adh*ELEKTRON_DD*waitForSecondaryResponses : True
...
host1*adh*route1.route*ELEKTRON_DD*primaryProvider : True
host1*adh*route2.route*ELEKTRON_DD*primaryProvider : False
...

After configuring the ADH, the next step is to configure the ADS to distribute ELEKTRON_DD service to the consumer.

Configuring ADS

Interactive Source Blending is a feature of the ADH and therefore the ADS does not require any special configuration for this feature. Please follow steps in the ADS installation guide to install and configure ADS as per normal.

Modifying EMA IProvider to provide augmenting fields

At this point, ADH and ADS 3.0 have been installed and configured properly to test Interactive Source Blending feature. Next, we will modify the example in EMA C++ package to test this feature. We use the IProvider100 example in Ema\Examples\Training\IProvider\100_Series_Examples\100MarketPriceStreaming directory as a secondary provider because this example is simple and easy to modify.

For the purpose of this exercise, we will use the RIC THB= and add augment it with a few extra FIDs. We will modify the app to provide data for the following additional fields: 92 (PRV_YRHIGH), 93 (PRV_YRLOW), 94 (LIFE_HIGH), and 95 (LIFE_LOW).

Modifying 100_MarketPrice_Streaming (IProvider100)

To demonstrate Interactive Source Blending feature, IProvider100 example is modified to provide augmenting fields. This example demonstrates the basic usage of the EMA library in providing OMM MarketPrice data to the Refinitiv Advanced Distribution Hub. It illustrates how to create and publish a single OMM streaming item. Originally, this example provides data for the following fields: 22 (BID), 25 (ASK), 30 (BIDSIZE), and 31 (ASKSIZE). We will modify it to provide data for the following fields: 92, 93, 94, and 95 which are not available in the template of our example THB= RIC. The definitions for these fields are:

    	
            PRV_YRHIGH "PREV YEAR HIGH"        92  NULL        PRICE              17  REAL64           7
!
! The highest value during the previous calendar year.
!
PRV_YRLOW  "PREV YEAR LOW"         93  NULL        PRICE              17  REAL64           7
!
! The lowest value during the previous calendar year.
!
LIFE_HIGH  "LIFETIME HIGH"         94  NULL        PRICE              17  REAL64           7
!
! The highest value ever achieved by this issue.
!
LIFE_LOW   "LIFETIME LOW"          95  NULL        PRICE              17  REAL64           7
!
! The lowest value ever achieved by this issue.
!

First, we will modify AppClient::processMarketPriceReqest method in the example to publish these fields in the refresh message. This method is used to handle the MarketPrice request and create a field list for the request message. We have modified it to create a field list with these fields.

IProvider.cpp

    	
            void AppClient::processMarketPriceRequest( const ReqMsg& reqMsg, const OmmProviderEvent& event )
{
    if ( itemHandle != 0 )
    {
        processInvalidItemRequest(reqMsg, event);
        return;
    }
 
    event.getProvider().submit( RefreshMsg().name( reqMsg.getName() ).serviceName( reqMsg.getServiceName() ).solicited( true ).
        state( OmmState::OpenEnum, OmmState::OkEnum, OmmState::NoneEnum, "Refresh Completed" ).
        payload( FieldList().
            addReal( 92, 3990, OmmReal::ExponentNeg2Enum ).
            addReal( 93, 3994, OmmReal::ExponentNeg2Enum ).
            addReal( 94, 9, OmmReal::Exponent0Enum ).
            addReal( 95, 19, OmmReal::Exponent0Enum ).
            complete() ).
        complete(), event.getHandle() );
 
    itemHandle = event.getHandle();
}

Next, we will modify the update message to update the values of fields of just two of the augmented fields: 92 and 93. The code that sends the update messages is in the main function.

IProvider.cpp

    	
            int main( int argc, char* argv[] )
{
    try
    {
        AppClient appClient;
 
        OmmProvider provider( OmmIProviderConfig().port( "14002" ), appClient );
 
        while ( itemHandle == 0 ) sleep(1000);
 
        for ( Int32 i = 0; i < 60; i++ )
        {
            provider.submit( UpdateMsg().payload( FieldList().
                    addReal( 92, 3391 + i, OmmReal::ExponentNeg2Enum ).
                    addReal( 93, 10 + i, OmmReal::Exponent0Enum ).
                    complete() ), itemHandle );
 
            sleep( 1000 );
        }
    }
    catch ( const OmmException& excp )
    {
        cout << excp << endl;
    }
 
    return 0;
}

To run the example, the configuration file (EmaConfig.xml) is required. The configuration file defines the source directory used by the example. The default configuration supports DIRECT_FEED service which corresponds to the appServiceName configuration of the second route (route2) in ADH.

    	
            <Directory>
    <Name value="Directory_2"/>
    <Service>
        <Name value="DIRECT_FEED"/>
        <InfoFilter>
            <ServiceId value="1"/>
            <DictionariesProvided>
                <DictionariesProvidedEntry value="Dictionary_3"/>
            </DictionariesProvided>
 
            <DictionariesUsed>
                <DictionariesUsedEntry value="Dictionary_3"/>
            </DictionariesUsed>
…
…
</Directory>

Test Interactive Source Blending

Now, we are ready to test this feature. To test it, we use the Consumer100 example in Ema\Examples\Training\Consumer\100_Series_Examples\100MarketPriceStreaming directory. We have modified Consumer100 to connect to ADS server and subscribe to THB= from ELEKTRON_DD service.

Consumer.cpp

    	
            int main( int argc, char* argv[] )
{ 
    try { 
        AppClient client;
        OmmConsumer consumer( OmmConsumerConfig().host( "hostnameOfADS:14002" ).username( "user" ) );
        consumer.registerClient( ReqMsg().serviceName( "ELEKTRON_DD" ).name( "THB=" ), client );
        sleep( 60000 );                // API calls onRefreshMsg(), onUpdateMsg(), or onStatusMsg()
    } catch ( const OmmException& excp ) {
        cout << excp << endl;
    }
    return 0;
}

Please change hostnameOfADS to the IP address or host name of ADS. Then, please follow these steps to test Interactive Source Blending.

Run 100_MarketPrice_Streaming (IProvider100)
Start ADH 3.0 to connect to the EED server and 100_MarketPrice_Streaming (IProvider100) If the settings are correct, you will see that ELEKTRON_DD service is up with two source servers via adhmon.

3. Start ADS 3.0. If the settings are correct, you will see that the ELEKTRON_DD service is up via adsmon.

4. Run 100_MarketPrice_Streaming (Consumer100) to connect to ADS and subscribe to THB= from ELEKTRON_DD service

After running 100_MarketPrice_Streaming (Consumer100), the output shows the refresh of THB= contains the augmenting FIDs: 92, 93, 94, and 95. Moreover, the consumer will receive the update of FIDs: 92 and 93 every second.

    	
            RefreshMsg
    streamId="5"
    domain="MarketPrice Domain"
    Solicited
    RefreshComplete
    ClearCache
    state="Open / Ok / None / 'All is well'"
    itemGroup="00 01"
    permissionData="03 01 01 69 33 C0"
    qos="RealTime/TickByTick"
    seqNum="24398"
    name="THB="
    nameType="1"
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList FieldListNum="99" DictionaryId="1"
            FieldEntry fid="1" name="PROD_PERM" dataType="UInt" value="6933"
            FieldEntry fid="2" name="RDNDISPLAY" dataType="UInt" value="153"
            FieldEntry fid="3" name="DSPLY_NAME" dataType="Rmtes" value="BROKER      GFX"
…
            FieldEntry fid="79" name="HSTCLSDATE" dataType="Date" value="24 MAY
2017"
            FieldEntry fid="90" name="YRHIGH" dataType="Real" value="36.00"
            FieldEntry fid="91" name="YRLOW" dataType="Real" value="34.18"
            FieldEntry fid="92" name="PRV_YRHIGH" dataType="Real" value="39.90"
            FieldEntry fid="93" name="PRV_YRLOW" dataType="Real" value="39.94"
            FieldEntry fid="94" name="LIFE_HIGH" dataType="Real" value="9"
            FieldEntry fid="95" name="LIFE_LOW" dataType="Real" value="19"
            FieldEntry fid="105" name="BCKGRNDPAG" dataType="Rmtes" value="    "
…
            FieldEntry fid="14208" name="BID_HR_MS" dataType="Time" value="10:00
:00:376:000:000"
        FieldListEnd
    PayloadEnd
RefreshMsgEnd
 
UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    seqNum="0"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="92" name="PRV_YRHIGH" dataType="Real" value="33.91"
            FieldEntry fid="93" name="PRV_YRLOW" dataType="Real" value="10"
        FieldListEnd
    PayloadEnd
UpdateMsgEnd
 
UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    seqNum="0"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="92" name="PRV_YRHIGH" dataType="Real" value="33.92"
            FieldEntry fid="93" name="PRV_YRLOW" dataType="Real" value="11"
        FieldListEnd
    PayloadEnd
UpdateMsgEnd
 
UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    DoNotConflate
    seqNum="24430"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="3284" name="EURO_NETCH" dataType="Real" value="-0.15"
        FieldListEnd
 
    PayloadEnd
UpdateMsgEnd
 
 
UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    DoNotConflate
    seqNum="24462"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="114" name="BID_NET_CH" dataType="Real" value="-0.11"
            FieldEntry fid="372" name="IRGPRC" dataType="Real" value="-0.32"
        FieldListEnd
    PayloadEnd
UpdateMsgEnd
 
UpdateMsg
    streamId="5"
    domain="MarketPrice Domain"
    updateTypeNum="0"
    seqNum="0"
    name="THB="
    serviceId="257"
    serviceName="ELEKTRON_DD"
    Payload dataType="FieldList"
        FieldList
            FieldEntry fid="92" name="PRV_YRHIGH" dataType="Real" value="33.93"
            FieldEntry fid="93" name="PRV_YRLOW" dataType="Real" value="12"
        FieldListEnd
    PayloadEnd
UpdateMsgEnd

The output shows that the 100_MarketPrice_Streaming (Consumer100) retrieves the refresh message that contains the augmenting FIDs: 92, 93, 94, and 95. It also retrieves the update of FIDs: 92 and 93 from the secondary provider: 100_MarketPrice_Streaming (IProvider100). Moreover, from the output, the update messages from the primary and secondary providers are retrieved separately.

References

Advanced Data Hub 3.0 : Software Installation Manual in ADH 3.0 package. You can download it from the Download Center. It is in Category: MDS – Infra and Products: Advanced Data Hub (ADH)
Advanced Distribution Server 3.0 : Software Installation Manual in ADS 3.0 package. You can download it from the Download Center. It is in Category: MDS – Infra and Products: Advanced Distribution Server (ADS)
100_MarketPrice_Streaming (IProvider100) and 100_MarketPrice_Streaming (Consumer100) in the Refinitiv Real-Time SDK – C/C++ (EMA) package. The package is available at the developer community.