Migrate your Deployed Streaming Application to the Cloud
Umer Nalla
Developer Advocate Developer Advocate

Prerequisite

Familiar with RT-SDK programming and configuration e.g. having already developed and deployed an Enterprise Message API application.

Migrating your Deployed application to the Cloud

Cloud-based platforms have become the standard for Enterprise solutions, offering superior manageability, scale, and service to that offered over a traditional company-internal data centre.

Since the launch of  Refinitiv's Real-Time - Optimized (RTO) cloud-based streaming data service back in 2018 (formerly known as ERT in Cloud), more and more customers have migrated or are considering migrating their deployed real-time streaming applications to utilise our cloud-based offerings.

 The RTO service is fully aligned with the Refinitiv strategy of providing a consistent fit-for-purpose delivery standard, partnering with companies offering public cloud services and delivering an open platform with self-service capabilities accessible through strategic APIs.

As well as reducing duplication of technologies across Real-Time and simplifying the customers' technology requirements, Refinitiv Real-Time - Optimized can deliver multiple benefits:

  • Supporting customer cloud transformations which deliver improved service, better time to market and simplified commercials
  • Building a solution fit for the evolving customer needs
  • Massively reducing internal costs - both operational and capital expenditure

For a smaller customer who does not have a deployed Refintiv Real-Time Distribution System (RTDS) and is using our previous EZD or Elektron Connect service, the migration is relatively straightforward and my colleague Gurpreet Bal has written a comprehensive article which includes such a migration scenario. 

However, for medium to larger customers with existing RTDS deployment, the migration can be done in stages - for example:

Stage 1

Option 1 - Start with migrating a simple use case of a non-Trading application,  connecting to Real-Time – Optimized to explore sourcing data from the cloud. As I go on to explain below, with minimal change to the application's configuration + code, the client can start to consume RTO data in the application. This can help the client's understanding of the endpoint options, network connectivity and authentication and security requirements.
Option 2 - If you want to try out RTO without making code changes at the application level, there is another path. One or more of your RTDS servers could be configured to connect to RTO and make the RTO data available on a new service name. You could then modify just the service name used by the application, so that is sources it's data (indirectly) from RTO rather than from your existing deployed real-time feed (Edge Device). 

Stage 2

As you become more comfortable with RTO, the next step could be to move the above application up to the cloud - rather than being deployed on a local server.
Or you could continue migrating more critical applications to RTO and potentially also deploying those applications to the cloud.

As you can expect, both options could help lower the Total Cost of Ownership by removing server and data centre costs and replacing them with lower cloud costs. Those early-adopter applications to the cloud may be selected for their data, latency or workflow needs.

Stage 3

As confidence in the RTO and cloud-based deployments grow, more and more applications could be migrated to consume RTO data and potentially being deployed in the cloud.

For this article, I will be focusing on using Option 1 with applications developed using the Refinitiv Real-Time SDK - as the older APIs such as RFA and SFC do not directly support RTO and the migration path for such apps is covered by my colleague Gurpreet's article.

Code and Config changes?

One of the key concerns that arise when considering such migration from RTDS to RTO is just how much effort will the developers have to expend making code and config changes at the application level?

Thankfully the only code changes required are at the initialisation stage of the application. The code concerned with requesting the data and receiving + decoding the data should remain unchanged - as the data request mechanism and response handling + payloads remain identical when moving from RTDS to RTO.

The config changes can also be minimal or even simplified - depending on how the application is currently configured.

Configuration changes

Most applications consuming data from RTDS are configured to use a ChannelSet. As described in this article, a ChannelSet consists of a list of two or more Channels where the API will automatically try each Channel in the set until a successful connection is made.

For example, your existing (RTDS) Consumer configuration - named Consumer_2 may be referencing two Channel configs - Channel_1 and Channel_2. These two Channels specify the hostname of your two chosen servers e.g. 'prod_ads1' and 'prod_ads2'

    	
            
<ConsumerGroup>


    <DefaultConsumer value="Consumer_2"/>


    <ConsumerList>


        <Consumer>


            <Name value="Consumer_2"/>


            <!-- ChannelSet specifies an ordered list of Channels to which OmmConsumer will attempt --> 


            <!-- to connect, one at a time, if the previous one fails to connect-->


            <ChannelSet value="Channel_1, Channel_2"/>


            <Dictionary value="Dictionary_2"/>


            <XmlTraceToStdout value="0"/>


            <Logger value="Logger_1"/>


        </Consumer>


    </ConsumerList>


</ConsumerGroup>


...


...


<ChannelGroup>


    <ChannelList>


        <Channel>


            <Name value="Channel_1"/>


            <ChannelType value="ChannelType::RSSL_SOCKET"/>


            <CompressionType value="CompressionType::None"/>




            <GuaranteedOutputBuffers value="5000"/>


            <ConnectionPingTimeout value="30000"/>




            <TcpNodelay value="1"/>




            <Host value="prod_ads1"/>


            <Port value="14002"/>


        </Channel>


        <Channel>


            <Name value="Channel_2"/>


            <ChannelType value="ChannelType::RSSL_SOCKET"/>


            <CompressionType value="CompressionType::None"/>


            <GuaranteedOutputBuffers value="5000"/>


            <Host value="prod_ads2"/>


            <Port value="14002"/>


        </Channel>


    </ChannelList>


</ChannelGroup>

When migrating the above to RTO, whilst it is possible to use specific RTO endpoints in each Channel config - in place of the ADS hostname - this is not the recommended or simplest approach.
It is much simpler to use the Session Management and Service Discovery facility offered by the Message API and let it take care of the details. In summary, Service Discovery allows you to specify the location of the RTO servers you wish to connect to. The API will then Discover the most appropriate RTO endpoints to use - no worrying about endpoint names etc.
The added benefit of this is that if Refinitiv does ever need to change the endpoint details - you don't have to - the Service Discovery will return the latest endpoint details and the API will use them to establish the channel.
You could still use ChannelSets - where each channel specifies a different location - to benefit from cross-regional resiliency - as shown in my example below:

    	
            
<ChannelGroup>


    <ChannelList>


        <Channel>


            <Name value="Channel_1"/>


            <ChannelType value="ChannelType::RSSL_ENCRYPTED"/>


            <CompressionType value="CompressionType::None"/>


            <GuaranteedOutputBuffers value="5000"/>


            <!-- EMA discovers a host and a port from RDP service discovery for the specified location


 when both of them are not set and the session management is enabled. -->


            <Location value="eu-west-1"/>


            <EnableSessionManagement value="1"/>


            <EncryptedProtocolType value="EncryptedProtocolType::RSSL_SOCKET"/>


            <!-- ObjectName is optional: defaulted to ""-->


            <ObjectName value=""/>


        </Channel>


        <Channel>


            <Name value="Channel_2"/>


            <ChannelType value="ChannelType::RSSL_ENCRYPTED"/>


            <CompressionType value="CompressionType::None"/>


            <GuaranteedOutputBuffers value="5000"/>


            <Location value="us-east-1"/>


            <EnableSessionManagement value="1"/>


            <EncryptedProtocolType value="EncryptedProtocolType::RSSL_SOCKET"/>


        </Channel>


    </ChannelList>


</ChannelGroup>

In the above example, I have defined a channel set with my primary location choice of EU-West and the secondary location choice of US-East.

At the time of writing, RTO is deployed in the following geographic locations.

GEOGRAPHIC LOCATION CLOUD PROVIDER REGION CONFIG VALUE
AMERS  AWS N. Virginia us-east-1
AMERS AWS Ohio us-east-2
EMEA AWS Ireland eu-west-1
APAC AWS Singapore ap-southeast-1

You can find the latest list of Locations and RTO Endpoints as well as further details about the RTO service in the Real-Time Optimized Install and Config Guide.

Using Specific Endpoints

If for some reason, you don't wish to use Service Discovery and prefer to use specific RTO Endpoints, then this is also possible - as demonstrated in the configuration snippet below - by simply specfiying the Host and Ports for the chosen Endpoint:

    	
            
<ChannelGroup>


    <ChannelList>


        <Channel>


            <Name value="Channel_4"/>


            <ChannelType value="ChannelType::RSSL_ENCRYPTED"/>


            <CompressionType value="CompressionType::None"/>


            <GuaranteedOutputBuffers value="5000"/>


            <Host value="eu-west-1-aws-3-med.optimized-pricing-api.refinitiv.net"/>


            <Port value="14002"/>


            <EnableSessionManagement value="1"/>


            <EncryptedProtocolType value="EncryptedProtocolType::RSSL_SOCKET"/>


        </Channel>


    </ChannelList>


</ChannelGroup>

NOTE: I am still enabling Session Management - the only difference is that I am NOT specifying a Location and specifying an RTO endpoint and Port instead.
As before, I would recommend you use a ChannelSet and specify 'standard' (Refinitiv Managed) endpoints in different regions for each channel in your ChannelSet (to benefit from cross-regional resiliency) e.g. I could use 3 Channels in my ChannelSet and specify the above eu-west-1-aws-3-med.optimized-pricing-api.refinitiv.net and something like us-east-1-aws-3-med.optimized-pricing-api.refinitiv.net and ap-southeast-1-aws-3-med.optimized-pricing-api.refinitiv.net etc.
The exact endpoints to use will differ depending on your licence/watchlist limits - see the RTO install + config guide for further details.

Code Changes

As well as the config changes described above, you will also need to make some code changes in order for an RTDS connecting application to connect to RTO. As mentioned previously, the changes are mostly to the initialisation/connection-related code of the application. The remainder of the code should remain unchanged.

Assuming you are using the Session Management + Service Discovery functionality of EMA to select the appropriate RTO Endpoint, the key code sections you will need to add will be to specify user credentials (and proxy details if being used in your organisation).

If you compare one of the most basic EMA examples - 100_MP_Streaming (an RTDS example) with the 113_MP_SessionMgmt example - you will note that the bulk of the extra code deals with reading in the command line parameters and applying them to the OmmConsumerConfig instance - which is subsequently passed in when creating the OMMConsumer instance.

Unless you are using a Proxy server the following few lines could be all you need to add/amend to convert your RTDS application to connect to RTO (along with the config changes I detailed ealier):

    	
            
OmmConsumerConfig config;


 


config.username("<Your MachineID>");


config.password("<Your RTO Password>");


config.clientId("<Your Client ID/AppKey>");


 


OmmConsumer consumer( config.consumerName("<Name of Consumer Configuration in EMA Config Schema>");

Refer to the RT-SDK QuickStart for details on where you can find your MachineID, set your password and create an AppKey.

Migrating Enterprise Transport API applications

I have focused on Enterprise Message API applications as the majority of our RT-SDK users develop using EMA. If, however, your application was developed using the Enterprise Transport API, then this can also be migrated - although it will require more effort - just how much effort will depend on whether you are using the ValueAdd layer of ETA.

If you are using, the ValueAdd layer, then the code changes required should be relatively straightforward.
Referring to the source code for the ETA examples VAConsumer and WatchlistConsumer, you will note that these examples can connect to RTDS or RTO by simply providing alternative command line parameters.
If you study the code in a bit more detail, you will note that the key changes for connecting to RTO are related to providing the credentials (as we did with EMA), enabling Session Management and optionally specifying a location for the Service Discovery.

So, for example, the following lines of code would be some of the key changes you would need to make:

    	
            
RsslReactorOAuthCredential oauthCredential;


RsslReactorOMMConsumerRole consumerRole;


RsslReactorConnectInfo reactorConnectInfo;


 


oauthCredential.clientId = clientId;


consumerRole.pOAuthCredential = &oauthCredential;


 


loginRequest.userName = watchlistConsumerConfig.userName;


if(queryEndpoint) 


    serviceDiscoveryOpts.userName = loginRequest.userName;


 


loginRequest.password = password;


if(queryEndpoint) 




    serviceDiscoveryOpts.password = loginRequest.password;


 


reactorConnectInfo.enableSessionManagement = enableSessionMgnt;


reactorConnectInfo.location = location;

NOTE: The above is just a highlight of the code changes you would need to make to an RTDS application to migrate it to use RTO instead. I recommend you study the code for VAConsumer and WatchListConsumer examples- as well as read the Transport API C Value Added Developers Guide (or the Transport API J Value Added Developers Guide) shipped with the RT-SDK packages.

 

Migrating your Contributor application to the Cloud

Before I finish, I should mention that if you have a Contribution (Posting / Inserting) application to distribute data outside of your organisation - currently using a deployed Contribution engine/server - this too can be migrated to use our Cloud-based Refinitiv Contributions Channel system.

Your application can be migrated to connect directly to RCC, or you can configure your existing RTDS to connect to RCC instead. This 2nd option would minimise any code changes for your application - as it would continue to contribute to the RTDS system - perhaps only a change of service name may be required at the application level.

The following articles and tutorials cover the various paths for contributing to RCC.

Contributing your data to Refinitiv with WebSocket API via a Refinitiv Real-Time Distribution System | Refinitiv Developers

Contributing your data directly to Refinitiv Contributions Channel with WebSocket API | Refinitiv Developers

Contributing your data to Refinitiv with the Refinitiv Real-Time SDK via a Real-time Distribution System | Refinitiv Developers

Tutorial on Contributing directly to RCC via RT-SDK Java | Refinitiv Developers

Tutorial on Contributing directly to RCC via RT-SDK C++ | Refinitiv Developers

 

Summary

I hope you find this article useful in helping you consider and plan your migration of deployed applications to the cloud. As I have tried to highlight, there are various options and routes open to you for reducing hardware footprint, cost of ownership etc - with potentially very little effort on your part. 

If you have any further questions, feel free to ask on the Refinitiv Developer Community Forums.