Understanding Elektron Websocket API - Batch and View feature

Introduction

Refinitiv has introduced websockets as a new mechanism to get streaming/non-streaming market data from Thomson Reuters Enterprise Platform (TREP) infrastructure. By using websockets, a market data application developer is free to choose any language/technology which supports websockets. Almost all major languages have integrated or community supported websockets library. This article will walk through basics of what a websocket is and how the request/responses are structured in a typical session. We will focus on the batch requests and view feature, to request multiple items at a time and to limit the amount of data received which our application has to process.

Dependency on ADS

The websockets support is available in TREP 3.2 and higher only.

The View functionality described in this article is a licensable features and required additional license in ADS.

Websockets protocol in brief

Websocket is TCP connection which is compatible with HTTP proxies and intermediaries, thus making it compatible with the HTTP protocol. To achieve compatibility, the WebSocket handshake uses the HTTP Upgrade header to change from the HTTP protocol to the WebSocket protocol. The protocol specification defines ws (WebSocket) and wss (WebSocket Secure) as two new uniform resource identifier (URI) schemes which can be accessed as ws://hostname:port/uri.

The handshake resembles HTTP in allowing servers to handle HTTP connections as well as WebSocket connections on the same port. Once the connection is established, communication switches to a bidirectional binary protocol which does not conform to the HTTP protocol.

Sample interaction from client to server starts just like a regular HTTP GET request with upgrade flag (non websocket specific headers are omitted here):

GET ws://ADS_1:15000/WebSocket HTTP/1.1
Host: ADS_1:15000
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: Q+02DCm/LuXrcxxQH4r+yA==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Protocol: tr_json2

The ADS which support websockets, responds by accepting the request:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Protocol: tr_json2
Sec-WebSocket-Accept: e3xfcwIg33R5bNO7tthsTM0aWeY=

After this point the binary bi-directional transfer between the client and ADS consists of Websocket frames. The frames contain headers, flags, masks, payload length and payload data. Everything besides the payload data is handled by the underlying client's implementation of the protocol, while the data is passed on to the application layer. The payload data in the frames is structured as JavaScript Object Notation (JSON) object. So from application builder's perspective, a request is sent as JSON object containing fields which comply to tr_json2 protocol and result is delivered in JSON object.

JSON is a simple textual representation of data in String key/value data pairs. Originally developed for JavaScript for use with web services; have now been adopted by major languages and systems like databases. A typical JSON object looks like this (formatted for readability):

{
    "Key1": 1,
    "Key2": "Data in key 2",
    "Key3": {
        "Name": "This is key3",
        "Description": "Contains a nested JSON object"
    }
}

All major browsers support websockets natively and have developer tools to view and troubleshoot requests. Here is a screen shot of Chrome's developer tools (Network tab), showing bi-directional frames between a consumer and ADS:

Developer tools can be accessed using keyboard shortcut:  CRTL-SHIFT-J in Chrome, or CRTL-SHIFT-E in Firefox.

Description of request and response

Once the websocket connection to ADS is established, Login is the very first message which can be sent to the server. The login message consists of Login domain and contains the user's ID and position. This information must match the entitlements defined in the ADS authorization system.

Request:

{
    "ID": 0,
    "Domain": "Login",
    "Key": {
        "Name": "cons1",
        "Elements": {
            "ApplicationId": "256",
            "Position": "10.116.80.111"
        }
    }
}

Here a user with ID cons1 is requesting login to server, using an application #256 from a machine with IP address 10.116.80.111.

ADS accepts the login request and responds with:

[{
        "ID": 0,
        "Type": "Refresh",
        "Domain": "Login",
        "Key": {
            "Name": "cons1",
            "Elements": {
                "Position": "10.116.80.111",
                "SupportOptimizedPauseResume": 1
            }
        },
        "State": {
            "Stream": "Open",
            "Data": "Ok",
            "Text": "Login accepted by host torsrmds11."
        },
        "Elements": {
            "PingTimeout": 30,
            "MaxMsgSize": 61430
        }
    }
]

The state information shows that ADS is ready to provide market data. Additional elements indicate, that user's application is expected to maintain a 30 seconds ping to keep connection open.

Next, the application subscribes to a batch of instruments. Batch allows the convenience of subscribing to multiple instruments with a single handle. The whole stream can be started with a single request. Further, a view feature allows limiting a set of data which can be delivered to the application. This helps simplify the application design, by not having to deal with unwanted data fields, and reducing the data volume. A request like:

{
    "ID": 2,
    "Domain": "MarketPrice",
    "Key": {
        "Name": ["TRI.N", "IBM.N", "T.N"]
    },
    "Streaming": true,
    "View": ["BID", "ASK", "BIDSIZE"]
}

tells the ADS to start a stream with ID# 2 for level 1 quotes for three equities, and only interested in the BID, ASK and BIDSIZE prices.

If the instruments are valid, ADS responds with a REFRESH message (some keys omitted for simplification):

[
  {
    "ID": 2,
    "Type": "Status",
    "State": {
      "Stream": "Closed",
      "Data": "Ok",
      "Text": "Processed 3 total items from Batch Request.  3 Ok."
    }
  },
  {
    "ID": 4,
    "Type": "Refresh",
    "Key": {
      "Service": "IDN_SELECTFEED",
      "Name": "IBM.N"
    },
    "State": {
      "Stream": "Open",
      "Data": "Ok",
      "Text": "All is well"
    },
    "Qos": {
      "Timeliness": "Realtime",
      "Rate": "TickByTick"
    },
    "PermData": "AwAXYsA=",
    "SeqNumber": 9344,
    "Fields": {
      "BID": 156.69,
      "ASK": 156.76,
      "BIDSIZE": 1
    }
  },
  {
    "ID": 5,
    "Type": "Refresh",
    "Key": {
      "Service": "IDN_SELECTFEED",
      "Name": "T.N"
    },
    "SeqNumber": 27792,
    "Fields": {
      "BID": 36.56,
      "ASK": 36.57,
      "BIDSIZE": 7
    }
  },
  {
    "ID": 3,
    "Type": "Refresh",
    "Key": {
      "Service": "IDN_SELECTFEED",
      "Name": "TRI.N"
    },
    "SeqNumber": 56816,
    "Fields": {
      "BID": 39.06,
      "ASK": 39.07,
      "BIDSIZE": 3
    }
  }
]

Here the ADS responds to the subscribe request (ID# 2), indicates that three subscriptions were successful. The image response (Refresh) which contains all the fields for that instrument is also packed in this response. Since this is a view request, the refresh message is a subset of all available fields. All three instruments receive their own ID which can be used to individually control those streams.

A refresh message is followed by update messages. Update frequency is based on how often the underlying instrument changes and also the Timeliness and Rate provided by the ADS service. In this example above, we get Realtime tick-by-tick updates. Often time users may want to slow down the amount of data being received by their application, by requesting a conflated service - which combines multiple updates for an interval and provides reduced data rate.

An update packet looks like:

[{
        "ID": 5,
        "Type": "Update",
        "UpdateType": "Quote",
        "Key": {
            "Service": "IDN_SELECTFEED",
            "Name": "T.N"
        },
        "SeqNumber": 27808,
        "Fields": {
            "BID": 36.56,
            "ASK": 36.57,
            "BIDSIZE": 7
        }
    }
]

This packet contains the update for a single instrument, but updates for multiple instruments can be packed together as well. Important thing to note here is that even though the batch response has and ID#2 (the ID in request message), the streaming instruments have their own ID numbers and can be individually paused or stopped.

Sample consumer using JavaScript/Browser

See the attached source code for a sample web page, which uses JavaScript to login and request and process market data. The minimal code in the sample, which is packaged in the html file itself, exists to demonstrate the core functionality only. The users should use the sample code provided with API as a starting point when developing their own applications. The sample webpage allows a user to connect to ADS and then send any request message which is typed into the text boxes. The ID element from server response is extracted and is displayed in corresponding box. The raw information and request/response messages are displayed in the log panel at the bottom of the page.

To run the sample:

1. Type in the Websocket enabled ADS hostname:port and click connect

2. Next send the login request (ID# 1)

3. Upon getting a successful login response, send the batch subscribe request (ID# 2)

4. Optionally, close all the streams by sending Login close message

Since there is no API associated with websockets, the sample demonstrates how easy it is to form and send request messages and parse the JSON responses from server.

Connect to server:

ws = new WebSocket('ws://ADS:15000/WebSocket', "tr_json2");
ws.onopen = ...
ws.onerror = ...
ws.onclose = ...
ws.onmessage = onMessage;

Once the server is open, the omMessage function peeks into the message and pushes it into appropriate display box.

var parsedMsg = JSON.parse(evt.data.toString());

for (var i = 0; i < parsedMsg.length; ++i)    {
    switch(parsedMsg[i].Type)    {
        case "Ping":
        {
            ws.send('{"Type":"Pong"}');
            log("SENT: Ping reply");
            break;
        }

        default:
        {
            // extract the ID from message and show the message in that text box
            var destID = "RES" + parsedMsg[i].ID;
            // show the result in that textbox
            var tbox = document.getElementById(destID)
            tbox.value = JSON.stringify(parsedMsg[i], null, 2);
        }

Since response message can contain more than one nested message, the onMessage function loops through the response array and checks the message type. The Ping requests from the server are automatically responded with pong reply.

As can be seen in the screenshot above, a batch request with an ID of 2 resulted in a Stream Closed reply. Because the request was accepted by server, this caused opening three streams with ID 3, 4 and 5. The original request also asked for limiting the set of data that our application receives in response. Here we requested for only three Fields: BID, ASK and BIDSIZE. The ADS honored this "VIEW" request, and responded with reduced dataset, which can be seen in the response messages.

Once the streams are open, the individual subscriptions can be unsubscribed by sending a close request like:

{
  "ID": 5,
  "Type": "Close"
}

Sending a close request on login domain will close all the open streams. Note that, the request stream was already closed, so sending a close on ID# 2 will not close any other stream.

Reference

Elektron Websocket API - https://developers.refinitiv.com/elektron/websocket-api
Websockets Interface in browser - https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API
NodeJS Websockets - https://www.npmjs.com/package/nodejs-websocket
Python Websockets - https://pypi.python.org/pypi/websockets