A Quick Start Guide to Playback and Recording Data for the Refinitiv Real-Time SDK
Last Updated: October 2021
Overview
The ability to capture and replay data from a real-time service such as the Refinitiv Real-Time data feed is a very useful tool for exploring your understanding of the Refinitiv Real-Time APIs and as a means to provide a basic test of an application under development without the need to access an infrastructure.
This article describes the use of testclient and testserver which belong to a set of tools known as the infra tools package, testclient will enable you to record data from a live service and testserver will use that data to provide a replay service.
The arguments described here are for the purposes of recording and playback, for a comprehensive set of arguments accepted by testclient and testserver please refer to the Refinitiv_Real_Time_Distribution_System_Demo_Tools_xxx.pdf document which is contained in the infra tools package.
What You Will Need
You will need the infra tools package, the Load Id of the infra tools package used with this article is infra_tools3.5.3.L1, this can be downloaded from the developer portal here. You can also download the AMERS, EMEA, and ASIA canned data from here.
Access to a machine on which to install the infra tools that meet the Hardware/OS requirements described by the infra tools README, these are;
- Intel or AMD
- Red Hat Enterprise Linux 7.x (64-bit) (Qualification)
- Oracle Linux 7.x (Red Hat compatible kernel, 64 bit)
- CentOS Version 7 (Qualification)
- Amazon Linux 2 in AWS (Qualification)
- Red Hat Enterprise Linux 8.x (64-bit)
For recording data you will need:
- A list of instruments and the corresponding domain model numbers
- Access to a Refinitiv Real-Time Data Feed from which to request the instruments OR
- Access to a Refinitiv Real-Time Distribution System (formerly TREP) with a service that can provide the required instruments
- A user account with appropriate entitlements for a service that can provide the required instruments. User accounts on Real-Time Distribution System (formerly TREP) are issued by the DACS administrator.
To replay data you will need
- The data captured using testclient,
- The data dictionary files that the data was captured with or compatible equivalent,
- The file containing the list of instruments and domains used during the data capture and
- The RSSL consumer under test to request the captured data from the replay service.
Installation
Installation of the infra tools is simply a process of unpacking the downloaded zip file into a location of your choosing.
When you unzip the downloaded infra tools package infra_tools3.5.3.L1.linux.rrg.zip it will result in a tar file:
Setup/infra_tools3.5.3.L1.linux.rrg.tar
Unpack the tar file in the chosen location, e.g.
tar xvf Setup/infra_tools3.5.3.L1.linux.rrg.tar
This will create a directory infra_tools3.5.3.L1.linux.rrg in the current directory containing the infra tools.
The infra tools provide testclient and testserver in the following subdirectories:
linux7_x86_64/demo linux8_x86_64/demo
Based on the directory name, select the most appropriate for your platform and Operating System. For Linux we need to ensure that the LD_LIBRARY_PATH environment variable includes this directory so that testclient and testserver can load the runtime libraries they require.
The installation process is now complete.
Recording Data Using testclient
When using testclient you need to specify the following information;
- The host to connect.
- The port on which the server is listening for connection requests.
- The protocol to use for the connection. We are only considering rssl in this article.
- The user name, if you are connecting to a Real-Time Distribution System (formerly TREP) environment this will be a DACS login name which is issued by the DACS Administrator.
- A service name available on the server from which to request the data to be recorded.
- The path and file name containing the list of instruments to request.
- The path and file name to store the recorded data.
testclient command syntax
A description of the parameters we will use to capture data using testclient is given below;
testclient -S Service [-ef RicFile] [-h host] [-p port] [-ct ctype] [-u user] [-rf req_flags] [-obf outputBinaryFile] [-of outputXMLfile.dat]
testclient options
| -S Service | Specifies Service as the service name from which the data is requested. |
| -ef RicFile | Specifies RicFile as the path and name of a file that contains items to be requested belonging to different domains. |
| See Instrument List File Format below for details. | |
| -h host | Specifies host as the hostname on which the server is deployed. |
| -p port | Specify port as the port number or name to connect to the server. |
| -u user | Specifies user as the user name to login on the server. |
| -ct ctype | Specify ctype as the type of connection used by testclient to connect to the server. We will only consider connection type “rssl”. |
| -rf req_flags | Specify additional RSSL request flags. In order to support consumers that rely on the presence of the key in updates (service and item name) we will set req_flags to 8 : Update Key. |
| -obf outputBinaryFile | Records RWF data in binary format to the specified file “outputBinaryFile”. You can replay this data later using testserver. |
| -of outputXMLfile | Records RWF data to the specified file “outputXMLfile”. You can replay this data later using testserver. |
The file containing the instruments to request consist of line separated entries, each entry is in the following format:
Domain|Instrument
Where
Domain : The domain number of the instrument required.
Instrument : The instrument name/RIC required.
The symbol ‘|’ (0x7c) is used as the field separator.
On a UNIX environment please make sure that this file contains newline (\n) for end of line termination and not a CR/LF (\r\n) combination as in the case with DOS text file format.
Domain Numbers
Valid domain numbers for use in the instrument list file are given in the table below. For more detail on these RDMs please refer to the RDM Usage Guide supplied with your API and also available on the developer portal for EMA C++, EMA Java, ETA C and ETA Java
| Reuters Domain Model | Model Number |
| MMT_MARKET_PRICE | 6 |
| MMT_MARKET_BY_ORDER | 7 |
| MMT_MARKET_BY_PRICE | 8 |
| MMT_MARKET_MAKER | 9 |
Example List File
6|.OEXA
6|AAPL.OQ
6|ABBV.N
6|ABT.N
6|ACN.N
6|AGN.N
6|AIG.N
6|ALL.N
6|AMGN.OQ
6|AMZN.OQ
6|AXP.N
6|BA.N
6|BAC.N
6|BIIB.OQ
The Recorded Data File
The testclient can record in binary or XML format. The binary format produces a smaller capture file, you should consider using this format particularly when larger quantities of data are captured for the sake of retrieval performance when this data is replayed. To generate a data file in binary format use the “-obf” argument, to generate an XML data file use the “-of” argument instead.
Running testclient
Example Binary Format Capture Command
./testclient -S ELEKTRON -ef ./rics1.txt -h ads1 -u rtds -ct rssl -rf 8 -obf recBin.dat -p 14002
In this example testclient will connect to host ads1 on port 14002 using the RSSL protocol and login with user name “rtds”, it will then request the RICs contained in rics1.txt from the service called ELEKTRON and record the data in the file “recBin.dat” in binary format.
Example Recording Data in XML Format
To record data in XML format we substitute the “-obf” argument with “-of”;
./testclient -S ELEKTRON -ef ./rics1.txt –rf 8 -h ads1 -p 14002 -ct rssl -u rtds -of recXml.dat
This example will perform as described in the previous example above with the exception that it will record the data in the file “recXml.dat” in XML format.
Playback of Captured Data using testserver
To playback data captured using testclient as described above, we need to run the testserver as a server listening for connection requests from consumer clients. To do this requires the following information;
- The port on which the testserver should listen for connection requests.
- The protocol to use for the connection. We are only considering rssl in this article.
- The service name testserver should advertise for the playback service.
- The path and file name containing the captured data.
testserver Command Syntax
A description of the parameters we will use to replay the captured data using testserver ;
testserver -S Service [-bdf binaryDataFile] [-Q xmlDataFile ] –U [ rate ] –N [ port ] [-K] [-ik]
Where
| -S Service | Specifies the service name advertised and on which the data is published. |
| -bdf binaryDataFile | Specifies binaryDataFile as the name of the binary data file that contains previously captured data that testserver will replay. |
| -Q xmlDataFile | Specifies xmlDataFile as the name of the XML data file that contains previously captured data that testserver will replay. |
| -U rate | Specifies the number of updates testserver sends per second. |
| -N port | Listen on Port Number port |
| -K | Instructs testserver to listen for and accept client connections. This is required otherwise testserver will not listen for connection requests. |
| -ik | Include rssl key in update messages. This is to cater for those consuming applications that require the key data in update messages. Applications should use the stream Id. |
| -E protocol | Specify the server protocol. Optional because the default is rssl as required. |
The Recorded Data File
Using binary format results in a smaller capture file compared to XML format for the same amount of data captured. When larger quantities of data are captured, binary format provides better retrieval performance however the XML format has the advantage that you can edit the content, if you need to introduce particular values during replay.
To use a data file captured in XML format we use the “-Q xmlDataFile” argument, for a capture file in binary format we use “-bdf binaryDataFile” instead.
When using the XML format the service name, as specified by the –S parameter and the service name used by the consumer requests must match the XML file content, i.e. if the data was captured from a service called ELEKTRON_EDGE then it must be replayed on the testserver as the same, although it is possible to edit the content in order to change the service name if required; This restriction does not apply to data captured in binary format.
Running testserver
Example Playback Using Binary Capture File.
testserver –S ELEKTRON –bdf EMEA.dat –U 1 –N 14002 –K -ik
In this example testserver will listen for consumer connections on port 14002, provide the service name “ELEKTRON” and replay data at an update rate of one per second from file EMEA.dat, previously recorded in binary format. Updates will contain the key data, Service and RIC name.
Example Playback Using XML Capture File
testserver –S ELEKTRON –Q EMEA_Xml.dat –U 1 –N 14002 –K -ik
This example will operate as described in the previous example above with the exception that it will read from the file EMEA_Xml.dat previously recorded in XML format using testclient.
Running a Consumer
When using testserver as your data source you will need to specify a local data dictionary for the consumer application under test, this should be the same dictionary used to capture the data, or a compatible equivalent. To be compatible, the dictionary must contain all the fields in the capture file with no difference in the data-types or reduction in the size.
EMA C++ Consumer Example Using testserver
To illustrate the use of testserver as a playback service we are going to look at the EMA C++ “100_MP_Streaming” example from the Refinitiv Real-time SDK.
Add the location of the infra tools to the PATH and LD_LIBRARY_PATH environment variables, in this example the required infra tools directory is installed in /home/radmin/infra_tools3.5.3.L1.linux.rrg/linux8_x86_64/demo
$ PATH=${PATH}:/home/radmin/infra_tools3.5.3.L1.linux.rrg/linux8_x86_64/demo
$ LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/radmin/infra_tools3.5.3.L1.linux.rrg/linux8_x86_64/demo
Start testserver running on the local machine with the following arguments:
testserver –S ELEKTRON –bdf EMEA.dat –U 1 –N 14002 –K -ik
The recorded data file EMEA.dat was generated using the following instrument list :
6|BT.L
6|VOD.L
Update Consumer.cpp to reflect the playback service name and a RIC name from the instrument list, the file location relative to the Refinitiv Real-Time SDK installation directory is;
Ema/Examples/Training/Consumer/100_Series/100_MP_Streaming/Consumer.cpp
In method main() :
consumer.registerClient( ReqMsg().serviceName( "ELEKTRON" ).name( "VOD.L" ), client );
The host setting “localhost:14002” in Consumer.cpp is correct for this arrangement and the user name “user” can remain unchanged since the testserver will accept any user name.
Once the example is built with the above updates copy Ema/EmaConfig.xml, Etc/RDMFieldDictionary and Etc/enumtype.def into the resulting build directory, containing the Consumer100 binary, e.g.:
Ema/Executables/CENTOSL7_64_GCC482/Optimized
The content of this directory should look like this:
$ ls -p
Cons100 EmaConfig.xml enumtype.def RDMFieldDictionary
Edit the EmaConfig.xml to use the local data dictionary files.
This EMA example uses the default consumer name which is set to “Consumer_1” as can be seen in the EmaConfig.xml extract below. The Dictionary entry for Consumer_1 is updated to use “Dictionary_2” from the DictionaryGroup instead of “Dictionary_1”.
<DefaultConsumer value="Consumer_1"/>
<ConsumerList>
<Consumer>
<Name value="Consumer_1"/>
<Dictionary value="Dictionary_2"/>
The “Dictionary_2” entry as shown in the EmaConfig.xml extract below will now be used by the Consumer example. This entry is configured to use a File based Dictionary instead of downloading from the channel, the filename settings below for the RDMFieldDictionary and enumtype.def dictionary files specify the current directory of the EMA Consumer binary:
<DictionaryGroup>
<DictionaryList>
<Dictionary>
<Name value="Dictionary_2"/>
<DictionaryType value="DictionaryType::FileDictionary"/>
<RdmFieldDictionaryFileName value="./RDMFieldDictionary"/>
<EnumTypeDefFileName value="./enumtype.def"/>
</Dictionary>
When you run the testserver you should see an output on the console similar to the following:
$ ./testserver -S ELEKTRON -bdf EMEA.dat -U 1 -N 14002 -K -ik
Initialized RSSL library.
begin CPU time: 0.003003 sec user, 0.005005 sec system
Total number of messages processed: 245
end CPU time: 0.003142 sec user, 0.006284 sec system
.
Loaded pkts: 0 items: 2 images: 2 image_length: 0 unsol images: 0 updates: 243 update_length: 0 memory (MB): 0.0616217
setup_updates: cfg_cfg_ticks_per_second=100 cfg_ticks_per_second=100 usec_per_tick=10000 updates_per_tick=0 extra_updates=1 max_updates_to_pack=10
When you run Consumer100 you should see an output on the testserver console like this:
sessionAct fd=4
RSSL Login Request from User : user
Sending Login Response
Sent Login Response Success
RSSL Source Directory request
SockOptions: TCP_NODELAY=1 SNDBUF=2626560 RCVBUF=131072 bytes
UPA peer component version: rtc3.5.1.L1.linux.rrg 64-bit
Server fd=3: New client fd=5 newSession
Sending Source Directory
Sent Source Directory
This indicates that the Consumer100 has connected to testserver, logged in and has been sent a source directory which will contains the service on which testserver will publish.
The console output from the Consumer100 will appear similar to the following, showing a refresh response message followed by update messages:
$ ./Consumer100
loggerMsg
TimeStamp: 15:36:53.757
ClientName: ChannelCallbackClient
Severity: Success
Text: Received ChannelUp event on channel Channel_1
Instance Name Consumer_1_1
Connected component version: upa7.6.0.L1.linux.rrg 64-bit Static
loggerMsgEnd
RefreshMsg
streamId="5"
domain="MarketPrice Domain"
Solicited
RefreshComplete
ClearCache
state="Open / Ok / None / 'All is well'"
itemGroup="00 01"
permissionData="03 01 01 56 25 C0"
qos="RealTime/TickByTick"
seqNum="1"
name="VOD.L"
nameType="1"
serviceId="258"
serviceName="ELEKTRON"
Payload dataType="FieldList"
FieldList FieldListNum="194" DictionaryId="1"
FieldEntry fid="1" name="PROD_PERM" dataType="UInt" value="5625"
FieldEntry fid="2" name="RDNDISPLAY" dataType="UInt" value="115"
FieldEntry fid="3" name="DSPLY_NAME" dataType="Rmtes" value="VODAFONE GROUP"
FieldEntry fid="4" name="RDN_EXCHID" dataType="Enum" value="64"
...
Conclusion
The testclient supplied with the infra tools package provides us with a means to record data of interest and the testserver allows us to replay this data for problem solving and more general testing of Refinitiv Real-Time SDK applications, but it is not a substitute for testing those applications with the Real-Time Distribution System (formerly TREP) or the Refinitiv Real-Time Datafeed.
For information on other capabilities of the infra test tools please see the install.pdf contained in the infra tools package.
If you have any questions regarding this article or the Refinitiv Real-Time SDK please refer to our Q&A Forums on the Developer Portal.