A Quick Start Guide to Playback and Recording Data for the Elektron SDK

Overview

The ability to capture and replay data from a real-time service such as the Thomson Reuters Elektron Real-time data feed is a very useful tool for exploring your understanding of the Elektron APIs and as a means to provide a basic test of an application under development without the need to access an infrastructure.

This article describes the use of rmdstestclient and sink_driven_src which belong to a set of tools known as the infra tools package, rmdstestclient will enable you to record data from a live service and sink_driven_src will use that data to provide a replay service.

The arguments described here are for the purposes of recording and playback, for a comprehensive set of arguments accepted by rmdstestclient and sink_driven_src please refer to the install.pdf document which is contained in the infra tools package.

What You Will Need

You will need the infra tools package, the Load Id of the infra tools package used with this article is infra_tools0.0.6.L1, this can be downloaded from the developer portal here.

Access to a machine on which to install the infra tools that meets the Hardware/OS requirements described by the infra tools README, these are;

  • Intel or AMD
  • Solaris x86 - 64 bits (Intel or AMD)
  • Red Hat Enterprise Linux 5.x - 64 bits
  • Red Hat Enterprise Linux 6.x - 64 bits
  • Oracle Enterprise Linux 6.x
  • SuSE SLES 11 - 64 bits
  • Windows 64-bit - limited support

For recording data you will need:

  1. A list of instruments and the corresponding domain model numbers
  2. Access to an Elektron Data Feed from which to request the instruments OR
  3. Access to a TREP with a service that can provide the required instruments
  4. A user account with appropriate entitlements for a service that can provide the required instruments. User accounts on TREP are issued by the DACS administrator.

To replay data you will need

  • The data captured using rmdstestclient,
  • The data dictionary files that the data was captured with or compatible equivalent,
  • The file containing the list of instruments and domains used during the data capture and
  • The RSSL consumer under test to request the captured data from the replay service.

Installation

Installation of the infra tools is simply a process of unpacking the downloaded zip file into a location of your choosing.

When you unzip the downloaded infra tools package infra_tools0.0.6.L1.zip it will result in a tar file:

Setup/infra_tools0.0.6.L1.tar

Unpack the tar file in the chosen location, e.g.

                tar xvf Setup/infra_tools0.0.6.L1.tar

This will create a directory infra_tools0.0.6.L1 in the current directory containing the infra tools.

The infra tools provides rmdstestclient and sink_driven_src in the following subdirectories:

rhel5_x86_64    win_x86_64      rhel6_x86_64    sol10_x86_64

Based on the directory name, select the most appropriate for your platform and Operating System. On Solaris and Linux ensure that the LD_LIBRARY_PATH environment variable includes this directory so that rmdstestclient and sink_driven_src can load the runtime libraries they require.

The installation process is now complete.

Recording Data Using rmdstestclient

When using rmdstestclient you need to specify the following information;

  • The host to connect.
  • The port on which the server is listening for connection requests.
  • The protocol to use for the connection. We are only considering rssl in this article.
  • The user name, if you are connecting to a TREP environment this will be a DACS login name which is issued by the DACS Administrator.
  • A service name available on the server from which to request the data to be recorded.
  • The path and file name containing the list of instruments to request.
  • The path and file name to store the recorded data.

 

Rmdstestclient command syntax

A description of the parameters we will use to capture data using rmdstestclient is given below;

rmdstestclient -S Service [-ef RicFile] [-h host] [-p port]  [-ct ctype] [-u user] [-rf req_flags] [-obf outputBinaryFile] [-of outputXMLfile.dat]

rmdstestclient options

-S Service Specifies Service as the service name from which the data is requested.
-ef RicFile

Specifies RicFile as the path and name of a file that contains items to be requested belonging to different domains.

See Instrument List File Format below for details.

-h host Specifies host as the hostname on which the server is deployed.
-p port Specify port as the port number or name to connect to the server.
-u user

Specifies user as the user name to login on the server.

-ct ctype Specify ctype as the type of connection used by rmdstestclient to connect to the server. We will only consider connection type “rssl”.
-rf req_flags     Specify additional RSSL request flags. In order to support consumers that rely on the presence of the key in updates (service and item name) we will set req_flags to 8 : Update Key.
-obf outputBinaryFile      Records RWF data in binary format to the specified file “outputBinaryFile”. You can replay this data later using sink_driven_src.
-of outputXMLfile Records RWF data to the specified file “outputXMLfile”. You can replay this data later using sink_driven_src.

 

Instrument List File Format

The file containing the instruments to request consist of line separated entries, each entry is in the following format:

Domain|Instrument

 

Where

                Domain : The domain number of the instrument required.

                Instrument : The instrument name/RIC required.

                The symbol ‘|’ (0x7c) is used as the field separator.

On a UNIX environment please make sure that this file contains newline (\n) for end of line termination and not a CR/LF (\r\n) combination as in the case with DOS text file format.

Domain Numbers

Valid domain numbers for use in the instrument list file are given in the table below. For more detail on these RDMs please refer to the RDM Usage Guide supplied with your API and also available on the developer portal for EMA C++, EMA Java, ETA C and ETA Java

Reuters Domain Model Model Number
MMT_MARKET_PRICE 6
MMT_MARKET_BY_ORDER 7
MMT_MARKET_BY_PRICE 8
MMT_MARKET_MAKER               9
Example List File

6|.OEXA

6|AAPL.OQ

6|ABBV.N

6|ABT.N

6|ACN.N

6|AGN.N

6|AIG.N

6|ALL.N

6|AMGN.OQ

6|AMZN.OQ

6|AXP.N

6|BA.N

6|BAC.N

6|BIIB.OQ

 

The Recorded Data File

The rmdstestclient can record in binary or XML format. The binary format produces a smaller capture file, you should consider using this format particularly when larger quantities of data are captured for the sake of retrieval performance when this data is replayed. To generate a data file in binary format use the “-obf” argument, to generate an XML data file use the “-of” argument instead.

Running rmdstestclient

Example Binary Format Capture Command

infra_tools0.0.6.L1\win_x86_64\rmdsTestClient.exe -S ELEKTRON -ef ./rics1.txt -h ads1 -u rmds -ct rssl -rf 8 -obf recBin.dat -p 14002 

 

In this example rmdstestclient will connect to host ads1 on port 14002 using the RSSL protocol and login with user name “rmds”, it will then request the RICs contained in rics1.txt from the service called ELEKTRON and record the data in the file “recBin.dat” in binary format.

Example Recording Data in XML Format

To record data in XML format we substitute the “-obf” argument with “-of”;

rmdstestclient -S ELEKTRON -ef ./rics1.txt –rf 8 -h ads1 -p 14002 -ct rssl -u rmds -of recXml.dat 

This example will perform as described in the previous example above with the exception that it will record the data in the file “recXml.dat” in XML format.

Playback of Captured Data using sink_driven_src

To playback data captured using rmdstestclient as described above we need to run the sink_driven_src as a server listening for connection requests from consumer clients. To do this requires the following information;

  • The port on which sink_driven_src should listen for connection requests.
  • The protocol to use for the connection. We are only considering rssl in this article.
  • The service name sink_driven_src should advertise for the playback service.
  • The path and file name containing the captured data.

sink_driven_src Command Syntax

A  description of the parameters we will use to replay the captured data using sink_driven_src;

sink_driven_src -S Service [-bdf binaryDataFile] [-Q xmlDataFile ] –U [ rate ] –N [ port ] [-K] [-ik]

Where

-S Service

Specifies the service name advertised and on which the data is published.

-bdf   binaryDataFile

Specifies binaryDataFile as the name of the binary data file that contains previously captured data which sink_driven_src will replay.

-Q xmlDataFile

Specifies xmlDataFile as the name of the XML data file that contains previously captured data which sink_driven_src will replay.

-U rate

Specifies the number of updates sink_driven_src sends per second.

-N port

Listen on Port Number port

-K

Instructs sink_driven_src to listen for and accept client connections. This is required otherwise sink_driven_src will not listen for connection requests.

-ik

Include rssl key in update messages. This is to cater for those consuming applications that require the key data in update messages. Applications should use the stream Id.

-E protocol

Specify the server protocol. Optional because the default is rssl as required.

 

The Recorded Data File

Using binary format results in a smaller capture file compared to XML format for the same amount of data captured.  When larger quantities of data are captured, binary format provides better retrieval performance however the XML format has the advantage that you can edit the content, if you need to introduce particular values during replay.

To use a data file captured in XML format we use the “-Q xmlDataFile” argument, for a capture file in binary format we use “-bdf   binaryDataFile” instead.

When using the XML format the service name, as specified by the –S parameter and the service name used by the consumer requests must match the XML file content, i.e. if the data was captured from a service called ELEKTRON_EDGE then it must be replayed on the sink_driven_src as the same, although it is possible to edit the content in order to change the service name if required; This restriction does not apply to data captured in binary format.

Running sink_driven_src

Example Playback Using Binary Capture File.

sink_driven_src –S ELEKTRON –bdf EMEA.dat –U 1 –N 14002 –K -ik

In this example sink_driven_src will listen for consumer connections on port 14002, provide the service name “ELEKTRON” and replay data at an update rate of one per second from file EMEA.dat, previously recorded in binary format. Updates will contain the key data, Service and RIC name.

Example Playback Using XML Capture File

sink_driven_src –S ELEKTRON –Q EMEA_Xml.dat –U 1 –N 14002 –K -ik

This example will operate as described in the previous example above with the exception that it will read from the file EMEA_Xml.dat previously recorded in XML format using rmdstestclient.

Running a Consumer

When using sink_driven_src as your data source you will need to specify a local data dictionary for the consumer application under test, this should be the same dictionary used to capture the data, or a compatible equivalent. To be compatible, the dictionary must contain all the fields in the capture file with no difference in the data-types or reduction in the size.

EMA C++ Consumer Example Using sink_driven_src

To illustrate the use of sink_driven_src as a playback service we are going to look at the EMA C++ “100__MarketPrice__Streaming” example from the Elektron SDK version 1.1.0.linux. Although we are using Linux in this example there is enough similarity to be of value if you are using Solaris or Windows.

Add the location of the infra tools to the PATH and LD_LIBRARY_PATH environment variables, in this example the required infra tools directory is installed in /opt/infra_tools0.0.6.L1/rhel6_x86_64

e.g.

$ PATH=${PATH}:/opt/infra_tools0.0.6.L1/rhel6_x86_64
$ LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/infra_tools0.0.6.L1/rhel6_x86_64

Start sink_driven_src running on the local machine with the following arguments:

sink_driven_src –S ELEKTRON –bdf EMEA.dat –U 1 –N 14002 –K -ik

The recorded data file EMEA.dat was generated using the following instrument list :

6|BT.L

6|VOD.L

Update Consumer.cpp to reflect the playback service name and a RIC name from the instrument list, the file location relative to the Elektron SDK installation directory is; 

Ema/Examples/Training/Consumer/100_Series_Examples/100__MarketPrice__Streaming /Consumer.cpp

In method main() :

consumer.registerClient( ReqMsg().serviceName( "ELEKTRON" ).name( "VOD.L" ), client );

The host setting “localhost:14002” in Consumer.cpp is correct for this arrangement and user name “user” can remain unchanged since the sink_driven_src will accept any user name.

Once the example is built with the above updates copy Ema/Examples/EmaConfig.xml, Ema/Etc/RDMFieldDictionary and Ema/Etc/enumtype.def into the resulting build directory, containing the Consumer100 binary, e.g.:

Ema/Examples/Training/Consumer/100_Series_Examples/100__MarketPrice__Streaming/CENTOSL7_64_GCC482/Optimized

The content of this directory should look like this:

$ ls -p
Consumer100  EmaConfig.xml  enumtype.def  obj/  RDMFieldDictionary

Edit the EmaConfig.xml to use the local data dictionary files.

This EMA example uses the default consumer name which is set to “Consumer_1” as can be seen in the EmaConfig.xml extract below. The Dictionary entry for Consumer_1 is updated to use “Dictionary_2” from the DictionaryGroup instead of “Dictionary_1”.

<DefaultConsumer value="Consumer_1"/>
	<ConsumerList>
		<Consumer>
			<Name value="Consumer_1"/>
			<Dictionary value="Dictionary_2"/>

The “Dictionary_2” entry as shown in the EmaConfig.xml extract below will now be used by the Consumer example. This entry is configured to use a File based Dictionary instead of downloading from the channel, the filename settings below for the RDMFieldDictionary and enumtype.def dictionary files specify the current directory of the EMA Consumer binary:

<DictionaryGroup>
	<DictionaryList>
		<Dictionary>
			<Name value="Dictionary_2"/>
			<DictionaryType value="DictionaryType::FileDictionary"/>
			<RdmFieldDictionaryFileName value="./RDMFieldDictionary"/>
			<EnumTypeDefFileName value="./enumtype.def"/>
		</Dictionary>

When you run the snk_driven_src you should see an output on the console similar to the following:

$ sink_driven_src -S ELEKTRON -bdf EMEA.dat -U 1 -N 14002 -K -ik
Initialized RSSL library.
begin CPU time: 0.003003 sec user, 0.005005 sec system
Total number of messages processed: 245
  end CPU time: 0.003142 sec user, 0.006284 sec system
.
Loaded  pkts: 0 items: 2 images: 2 image_length: 0 unsol images: 0 updates: 243 update_length: 0 memory (MB): 0.0616217
setup_updates:  cfg_cfg_ticks_per_second=100 cfg_ticks_per_second=100 usec_per_tick=10000 updates_per_tick=0 extra_updates=1 max_updates_to_pack=10

When you run Consumer100 you should see an output on the sink_driven_src console like this:

sessionAct fd=4
RSSL Login Request from User : user
Sending Login Response
Sent Login Response Success
RSSL Source Directory request

SockOptions: TCP_NODELAY=0 SNDBUF=2626560 RCVBUF=1061488 bytes

UPA peer component version: eta3.1.0.L1.rrg|ema3.1.0.L1.linux 64-bit Static(Optimized_Assert)
Sending Source Directory
Sent Source Directory

This indicates that the Consumer100 has connected to sink_driven_src, logged in and has been sent a source directory which will contains the service on which sink_driven_src will publish.

The console output from the Consumer100 will appear similar to the following, showing a refresh response message followed by update messages:

$ ./Consumer100 
loggerMsg
    TimeStamp: 15:36:53.757
    ClientName: ChannelCallbackClient
    Severity: Success
    Text:    Received ChannelUp event on channel Channel_1
        Instance Name Consumer_1_1
        Connected component version: upa7.6.0.L1.linux.rrg 64-bit Static
loggerMsgEnd
RefreshMsg
    streamId="5"
    domain="MarketPrice Domain"
    Solicited
    RefreshComplete
    ClearCache
    state="Open / Ok / None / 'All is well'"
    itemGroup="00 01"
    permissionData="03 01 01 56 25 C0"
    qos="RealTime/TickByTick"
    seqNum="1"
    name="VOD.L"
    nameType="1"
    serviceId="258"
    serviceName="ELEKTRON"
    Payload dataType="FieldList"
        FieldList FieldListNum="194" DictionaryId="1"
            FieldEntry fid="1" name="PROD_PERM" dataType="UInt" value="5625"
            FieldEntry fid="2" name="RDNDISPLAY" dataType="UInt" value="115"
            FieldEntry fid="3" name="DSPLY_NAME" dataType="Rmtes" value="VODAFONE GROUP"
            FieldEntry fid="4" name="RDN_EXCHID" dataType="Enum" value="64"
...

 

Conclusion

The rmdstestclient supplied with the infra tools package provides us with a means to record data of interest and the sink_driven_src allows us to replay this data for problem solving and more general testing of Elektron SDK applications, but it is not a substitute for testing those applications with the Thomson Reuters Enterprise Platform or the Elektron Real-time Datafeed.

For information on other capabilities of the infra test tools please see the install.pdf contained in the infra tools package.

If you have any questions regarding this article or the Elektron SDK please refer to our Q&A Forums on the Developer Portal.