Elektron SDK - Java

API Family: Elektron

EMA Consumer - Request & Decode Machine Readable News

Download tutorial source code

Click here to download

Last updated August 2018
Environment Windows, Linux
Compilers JDK 1.7 or greater
Prerequisite Parsing and Decoding MarketPrice data

Introduction

This tutorial assumes you are familiar with consuming MarketPrice data from TREP using EMA Java and as a minimum, you have worked through the EMA Consumer Tutorials - Parsing and Decoding MarketPrice data. These tutorials are available on the Refinitiv Developer portal under the Elektron Java SDK section.  

If you have not worked through the above or are not familiar with the concepts covered, it is strongly recommended that you complete the tutorial first.

For this MRN tutorial, we will use the source code from the EMA Consumer - Parsing and Decoding MarketPrice data example (Tutorial 3) provided on the Refinitiv Developer portal as the starting point. We will amend/add the additional required code to illustrate how to Request and Decode News Text Analytics (NTA) domain from a Machine Readable News (MRN) service.

The News Text Analytics domain is designed for publishing large complex nested data structures over the ERT network using a FieldList based Envelope. 

Whilst TREP can deliver large payloads, to ensure optimized transport of NTA data items (which can extend to several kilobytes), NTA data items may be fragmented and delivered as multiple envelope messages. Each envelope message has several fields to hold metadata and one field to hold the actual data (fragment) itself. To further optimize the message transportation, the NTA data item is converted to a JSON UTF-8 string, compressed using zlib prior to splitting into fragments (if required) as part of the message encoding process. The consumer needs to re-assemble the fragments and unzip the completed buffer before it can be used. 

In view of the above, processing a response to NTA request is more involved than a typical MarketPrice response.

Description

The News Text Analytics Domain Model describes how the NTA data items are modeled using OMM. It makes use of the FieldList OMM container for the metadata and the Java’s built-in zlib library to decompress the compressed NTA fragments.

Initial Refresh Message

The Initial Refresh response does not contain any NTA data – but the relevant feed related or other static Fields are populated with appropriate values e.g. an MRN_STORY Refresh could contain: 

Item Name: MRN_STORY
Service Name: API_ELEKTRON_EPD_RSSL
Item State: Open / Ok / None / 'All is well'
    Fid: 1 Name = PROD_PERM DataType: UInt Value: 10001
    Fid: 17 Name = ACTIV_DATE DataType: Date Value: 18 / 8 / 2018
    Fid: 259 Name = RECORDTYPE DataType: UInt Value: 30
    Fid: 1709 Name = RDN_EXCHD2 DataType: Enum Value: 1370
    Fid: 4148 Name = TIMACT_MS DataType: UInt Value: 78729054
    Fid: 4271 Name = GUID DataType: Rmtes Value:  blank
    Fid: 5357 Name = CONTEXT_ID DataType: Real Value: 3752.0
    Fid: 6401 Name = DDS_DSO_ID DataType: UInt Value: 8328
    Fid: 6480 Name = SPS_SP_RIC DataType: Ascii Value: .[SPSML0L1
    Fid: 8506 Name = MRN_V_MAJ DataType: Rmtes Value: 2
    Fid: 8593 Name = MRN_TYPE DataType: Rmtes Value: STORY
    Fid: 11787 Name = MRN_V_MIN DataType: Rmtes Value: 10
    Fid: 12215 Name = MRN_SRC DataType: Rmtes Value: HTC_PRD_A
    Fid: 32479 Name = FRAG_NUM DataType: UInt Value: 1
    Fid: 32480 Name = TOT_SIZE DataType: UInt Value: 0
    Fid: 32641 Name = FRAGMENT DataType: Buffer Value:  blank

As you can see, the FRAGMENT field which would contain the NTA data fragments is empty – and the GUID, FRAG_NUM, and TOT_SIZE fields are zero or empty.

Update Messages

However, in the subsequent Update messages, these fields will be populated - but none of the feed related / static fields (contained in the Refresh response) will be, for example: 

Item Name: MRN_STORY
Service Name: ELEKTRON_AD
    Fid: 4148 Name = TIMACT_MS DataType: UInt Value: 33542212
    Fid: 17 Name = ACTIV_DATE DataType: Date Value: 29 / 8 / 2018
    Fid: 8593 Name = MRN_TYPE DataType: Rmtes Value: STORY
    Fid: 8506 Name = MRN_V_MAJ DataType: Rmtes Value: 2
    Fid: 11787 Name = MRN_V_MIN DataType: Rmtes Value: 10
    Fid: 32480 Name = TOT_SIZE DataType: UInt Value: 715
    Fid: 32479 Name = FRAG_NUM DataType: UInt Value: 1
    Fid: 4271 Name = GUID DataType: Rmtes Value: RSDJPZyl__1808292QIj+RWGGWImCljqNqvgZqmPvFLL/Keqw64aqJ
    Fid: 12215 Name = MRN_SRC DataType: Rmtes Value: HTC_PRD_A
    Fid: 32641 Name = FRAGMENT DataType: Buffer Value: <BUFFER>, limit = 715, length = 715

Some of the key Fields to note: 

  • MRN_TYPE : Type of NTA item - Story, News Analytics, News Sentiment Indices
  • TOT_SIZE : Total size in bytes of fragmented data
  • FRAG_NUM : Sequential fragment number
  • GUID : Globally Unique Identifier for the NTA item
  • MRN_SRC : Source component that published this NTA item
  • FRAGMENT : zlib compressed data fragment

Multi Fragment Items

The other point to note is that (for a Multi fragment item), Update messages with FRAG_NUM >1 will have fewer FIDs as the metadata is included in the first Update message (FRAG_NUM=1) for that item as shown below:

Item Name: MRN_STORY
Service Name: ELEKTRON_AD
    Fid: 4148 Name = TIMACT_MS DataType: UInt Value: 33552377
    Fid: 17 Name = ACTIV_DATE DataType: Date Value: 29 / 8 / 2018
    Fid: 8593 Name = MRN_TYPE DataType: Rmtes Value: STORY
    Fid: 8506 Name = MRN_V_MAJ DataType: Rmtes Value: 2
    Fid: 11787 Name = MRN_V_MIN DataType: Rmtes Value: 10
    Fid: 32480 Name = TOT_SIZE DataType: UInt Value: 3196
    Fid: 32479 Name = FRAG_NUM DataType: UInt Value: 1
    Fid: 4271 Name = GUID DataType: Rmtes Value: DJR5PlH9r_18082928oVHG+NheQpFJiajG3vE0EVgHgU1Hzbe8fMDr
    Fid: 12215 Name = MRN_SRC DataType: Rmtes Value: HTC_PRD_A
    Fid: 32641 Name = FRAGMENT DataType: Buffer Value: <BUFFER>, limit = 2600, length = 2600
Init:: Expected total buffer size: 3196 current size: 2600

Item Name: MRN_STORY
Service Name: ELEKTRON_AD
    Fid: 4271 Name = GUID DataType: Rmtes Value: DJR5PlH9r_18082928oVHG+NheQpFJiajG3vE0EVgHgU1Hzbe8fMDr
    Fid: 12215 Name = MRN_SRC DataType: Rmtes Value: HTC_PRD_A
    Fid: 32479 Name = FRAG_NUM DataType: UInt Value: 2
    Fid: 32641 Name = FRAGMENT DataType: Buffer Value: <BUFFER>, limit = 596, length = 2600
Additional fragment size: 596
Add Fragment:: Expected total buffer:3196 current 3196
<<Multi Fragment Completed>>

In the above example, you can see that only the essential fields are repeated i.e. the unique identifier, source service name, fragment number and off course the fragment payload. We will use these fields later when re-assembling multi-fragment NTA item to ensure that we are appending fragments to the correct NTA item.   

Decoding & Re-assembly Overview

As well as using standard TREP Status & State indicators to identify connectivity or data issues, the FRAG_NUM and TOT_SIZE fields should be used to detect missing fragments.

The FRAG_NUM FID is set to 1 for the first Update of each item and is incremented in each subsequent Update for that item. This allows you to you to detect a missing fragment (and ensure correct order of the fragments for re-assembly). 

Additionally, the TOT_SIZE contains the total size of the complete set of fragmented data in bytes. By comparing TOT_SIZE with the sum of the fragment sizes received, it should allow you to confirm when all the fragments for an item have been received and the NTA item is complete.

Using the FRAG_NUM and TOT_SIZE to detect outages requires the consumer to implement timeout functionality because we need to allow time for the missing fragments to reach the consumer before assuming they have been missed. This functionality is outside the scope of this tutorial.

Finally, as the FRAGMENT field contains compressed data, we will need to use the zlib library to decompress the payload to access the true data content. Note that for a multi-fragment NTA item, you will have to ensure you have received all the fragments before unzipping the complete multi-fragment buffer.

To request NTA domain items and decode & re-assemble the NTA item we will modify the com.thomsonreuters.ema.tutorials.consumer.tutorial4.basicConsumer tutorial example application as follows:

  1. Explicitly specify NEWS_TEXT_ANALYTICS as the domain model type when creating our ReqMsg.
  2. Specify MRN_STORY as the RIC Code for the ReqMsg (to request NTA type of  Story)
  3. Create a new class to represent a News Text Analytics item with some helper methods to assemble multi-fragment items.
  4. Process and decode the FieldList payload of NEWS_TEXT_ANALYTICS message. 
  5. Uncompress buffer data to get the content.

NewsTextAnalytics Msg Model Type Request & Response Handlers

To request a News Text Analytics domain model MRN_STORY item we need to change the run() method to explicitly specify the domain and specify the item name when creating our ReqMsg:

private void run()
{		
		...
		// When registering, we also need to specify where we want to capture all the events generated by 
		// this request.  
		OmmConsumerClient eventHandler = this;
		
		consumer.registerClient(
			  EmaFactory.createReqMsg()
							.domainType(EmaRdm.MMT_NEWS_TEXT_ANALYTICS)
							.serviceName("ELEKTRON_AD")
							.name("MRN_STORY"), 
			  eventHandler
		);		
		...		
}

As mentioned earlier, the Refresh message does not contain any NTA data fragments so we can use the existing decodePayload() method to output the meta-data fields to the console.

For the Update message, however, we will need to use a custom decodeNewsTextAnalytics() method which we will define later on: 

// onUpdateMsg
// After receiving the initial image, we capture any updates to the item based on market events.
// This callback parses and decodes the message and displays to the terminal.
public void onUpdateMsg(UpdateMsg updateMsg, OmmConsumerEvent event)
{
	System.out.println("UpdateMsg");
	
	// Decode the Payload
	// The data for an update message is conveyed by the payload. As with the refresh message, payload 
	// will contain a complex data structure that must be decoded and parsed to retrieve the individual 
	// fields of interest. However, the updatehMsg will contain only those fields that have changed 
	// based on the market conditions.
	decodeNewsTextAnalytics( updateMsg.payload() );
}

In its standard form the existing decodeAndPrint() method does not dump any Rmtes and Buffer type fields (e.g. GUID and FRAGMENT field contents) the console, but contents of FRAGMENT are garbage anyway. We can change the decodeAndPrint() method to handle both Rmtes and Buffer type fields:

// decodeAndPrint - FieldList
// The FieldList is a list of entries commonly used in level 1 (MarketPrice) data responses.
private void decodeAndPrint(FieldList fieldList)
{
			...
			case DataTypes.RMTES :
				System.out.println(fieldEntry.rmtes().toString());
				break;
			case DataTypes.BUFFER :
				System.out.println( "<BUFFER>"
				     + ", limit = "
				     + fieldEntry.buffer().buffer().limit()
				     + ", length = "
				     + fieldEntry.buffer().buffer().array().length );
				break;
			...
	}
}

In addition to the FRAGMENT field, a number of other fields are used to deliver the metadata for the NTA item. So, we need methods to extract the metadata and to re-assemble the data fragment(s). 

To ease readability and simplify the basicConsumer methods we can create a new Class to represent the NTA item with some methods to help with the decoding and re-assembly of the NTA data fragments.

The NewsTextAnalyticsItem Class

The key functions of the Class would be to simplify the decoding process and the re-assembly of multi-fragment messages.

class NewsTextAnalyticsItem
{
	public void init(FieldList fieldList) { }
	public void addFragment(NewsTextAnalyticsItem fragment) { }
	public int getTotalSize() { }
	public boolean isComplete() { }
	
	public ByteBuffer getNTABuffer() { return _ntaBuffer; }
	public String getGUID() { return _guid; }
	public String getMRNSource() { return _mrnSource; }
	public String getMRNType() { return _mrnType; }	
	public long getFragNum() { return _fragNum; }
	public long getExpectedSize() { return _expectedSize; }
	public ArrayList<ByteBuffer> getFragmentCollection() { return _fragmentCollection; }
	
	private ByteBuffer _ntaBuffer;
	private String _guid;
	private String _mrnSource;
	private String _mrnType;
	private long _fragNum;
	private long _expectedSize;
	private ArrayList<ByteBuffer> _fragmentCollection;
	
	final int FID_GUID 		= 4271;
	final int FID_MRN_TYPE 	= 8593;
	final int FID_MRN_SRC 	= 12215;
	final int FID_FRAG_NUM 	= 32479;
	final int FID_TOT_SIZE 	= 32480;
	final int FID_FRAGMENT 	= 32641;

}

Firstly we define the method used to initialize the NewsTextAnalyticsItem instance and extract the fields from the envelope

public void init(FieldList fieldList)
{ 
	int fidNum = 0;
	long tmpUint = 0;
	String tmpString;
	
	for (FieldEntry fieldEntry : fieldList)
	{
		// Get the Field ID for this field
		fidNum = fieldEntry.fieldId();
		// System.out.println("fidNum: " + fidNum + " " + fidToString(fidNum));
		// We are only interested in a few key fields 
		// so we only need to handle the corresponding field types
		
		switch (fieldEntry.loadType())
		{
		case DataTypes.UINT:
			tmpUint = fieldEntry.uintValue();
			System.out.println(tmpUint);
			if (fidNum == FID_FRAG_NUM)
				_fragNum = tmpUint;
			else if (fidNum == FID_TOT_SIZE)
				_expectedSize = tmpUint;
			break;
		case DataTypes.RMTES:
			tmpString = fieldEntry.rmtes().toString();
			if (fidNum == FID_GUID)
				_guid = tmpString;
			else if (fidNum == FID_MRN_SRC)
				_mrnSource = tmpString;
			else if (fidNum == FID_MRN_TYPE)
				_mrnType = tmpString;
		break;
		case DataTypes.BUFFER:
			if (fidNum == FID_FRAGMENT)
				_ntaBuffer = clone(fieldEntry.buffer().buffer());
			break;
		default:
			break;			
		}
	}
	
	if (_fragNum == 1)
		System.out.println("Init:: Expected total buffer size: " + _expectedSize + " current size: " + _ntaBuffer.limit());
	else
		System.out.println("Additional fragment size: " + _ntaBuffer.limit());
}

When a data type is Buffer, EMA uses a shared ByteBuffer object from the pool, and it may be reused later. So, the application should deep copy its value instead of pointing a variable to the ByteBuffer object to prevent the race condition from accessing to the same reference. The snippet code below shows how to perform a deep copy from a ByteBuffer instance.

private ByteBuffer clone(ByteBuffer original) {
	ByteBuffer clone = ByteBuffer.allocate(original.capacity());
	original.rewind();		// copy from the beginning
	clone.put(original);
	original.rewind();
	clone.flip();
	return clone;
}

For multi-fragment messages, we need a method to check the validity of the received fragment and to build up the fragment buffer.

public void addFragment(NewsTextAnalyticsItem fragment)
{
	if (!fragment._guid.equals(_guid))
		throw new RuntimeException("Cannot add fragment to item: mismatching GUID - " + fragment._guid + " : " + _guid);
	if (!fragment._mrnSource.equals(_mrnSource))
		throw new RuntimeException("Cannot add fragment to item: mismatching data source - " + fragment._mrnSource + " : " + _mrnSource);
	if (fragment._fragNum != _fragNum + 1)
		throw new RuntimeException("add fragment to item: fragment number is not in sequence");
		
	// Update an _fragNum field.
	_fragNum = fragment._fragNum;
	if (_fragmentCollection == null) {
		_fragmentCollection = new ArrayList<>();
		_fragmentCollection.add(_ntaBuffer);
		_fragmentCollection.add(fragment._ntaBuffer);
	} else {
		_fragmentCollection.add(fragment._ntaBuffer);
	}
	
	System.out.println("Add Fragment:: Expected total buffer:" + _expectedSize + " current " + getTotalSize());
}

In case of multi-fragment news, the application saves fragment items to a collection object for assembly later. Therefore, there are two different logics to determine total fragments' size as shown in the snippet code below:

public int getTotalSize() {
	if (_fragNum == 1) {
		// For a single fragment, return ByteBuffer limit.
		return _ntaBuffer.limit();
	} else {
		// For multi-fragment, return summation of all ByteBuffer in collection.
		int totalSize = 0;
		for (ByteBuffer buffer: _fragmentCollection) {
			totalSize+=buffer.limit();
		}
		return totalSize;
	}
}

Decode Update Message Envelope

With the NewsTextAnalyticsItem class now defined, we can continue with the decoding of the Update message to extract the fields, build up the NTA item and output to console when complete:

private NewsTextAnalyticsItem _multiFragItem;

void decodeNewsTextAnalytics(FieldList envelope)
{
	decode(envelope); // Dump the FieldList first (for informational purposes)
	NewsTextAnalyticsItem item = new NewsTextAnalyticsItem();
	// Now let’s extract the relevant fields into a NTA item - including the Fragment
	item.init(envelope);
	// Is this 1st fragment of a NTA item?
	if (item.getFragNum() == 1)
	{
		// Is it the complete NTA item?
		if (item.isComplete())	// i.e. check length of this single fragment buffer == expected total size of NTA item
		{
			System.out.println("<<Single Fragment>>");
			String unzipped;
			unzipped = uncompressData(item.getBuffer(), unzipped);	// decompress the buffer
			System.out.println("<Content>");
			System.out.print("\t");
			System.out.println(unzipped);					// Display the decompressed NTA item
			System.out.println("</Content>");
			return;
		}
		else
		{							// Start of a multi-fragment item
			_multiFragItem = item;	// so copy this first fragment into our pending item instance
		}
	}
	else
	{
		...
		...
	}
}

As you can see, if the NTA item is completed in a single fragment, we can immediately unzip and output to the screen. 

However, if FRAG_NUM=1 and buffer are incomplete then this is the first of a multi-fragment message. We initialize our multi-fragment storage instance with this initial set of fields and buffer, in expectation of one or more additional Update messages

Build up Multi Fragment until complete

When we receive a subsequent Update message, we extract the fragment and add it to the multi-fragment storage instance (_multiFragItem) until all fragments have been received i.e. the total buffer size equals the expected total size. Once the buffer is complete we can unzip the full buffer and output to the screen.

...
	else
	{
		// FRAG_NUM > 1 so continue to build up NTA item
		try {
			// Add the newly received fragment to the collection of the pending item instance
			_multiFragItem.addFragment(item);
		} catch (Exception e) {
			System.out.println("Multi Fragment Error: " + e.getMessage());
			_multiFragItem = null;
		}
		if (_multiFragItem!= null && _multiFragItem.isComplete())	// Does length of the buffer now == expected total size of NTA item ?
		{
			System.out.println("<<Multi Fragment Completed>>");
			ByteArrayOutputStream baos = new ByteArrayOutputStream();
			for (ByteBuffer buffer: _multiFragItem.getFragmentCollection()) {
				// for debug only
				// System.out.println("limit: " + buffer.limit());
				baos.write(buffer.array(), 0, buffer.limit());
			}
			String unzipped;
			unzipped = unzip(baos.toByteArray());			// Decompress the complete buffer
			System.out.println("<Content>");
			System.out.print("\t");
			System.out.println(unzipped);					// Display the decompressed NTA item
			System.out.println("</Content>");
				
			_multiFragItem = null;
			unzipped = null;
			return;
		}
	}
...

The content of NTA item output is in the form of a JSON UTF-8 string compressed by zlib algorithm.

Uncompress buffer data to get the content

To unzip the content and output the result to the screen, Java Provides classes for reading and writing the standard ZIP and GZIP file formats. You can find them in a java.util.zip package.

public String uncompressData(byte[] bytes)
{
	StringBuffer result = new StringBuffer();
	try {
		GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(bytes));
		BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(gis, "UTF-8"));

		String line = "";
            while ((line = bufferedReader.readLine()) != null) {
            	result.append(line);
            }
	} catch (IOException e) {
		e.printStackTrace();
	}
	return result.toString();
}

Build and Run

Set ELEKTRON_JAVA_HOME system variable Elektron SDK Java software installation path. For example,

On Windows:

set ELEKTRON_JAVA_HOME=D:\Elektron-SDK1.2.1.0.java

On Unix:

export ELEKTRON_JAVA_HOME=/local/Elektron-SDK1.2.1.0.java

Then, run buildConsumer.bat/.ksh to build the tutorial. All classes under an src folder are compiled and created in a bin folder.

As the connectivity parameters, service name etc are hard-coded into the run() method you will need to amend these to reflect your environment before running the application. 

The RIC code has been hardcoded to request MRN_STORY items – this can be changed to request alternative items such as MRN_TRNA, MRN_TRSI.

In the example output shown below, we requested News Text Analytics Story items. 

Example output from an MRN_STORY response

Item Name: MRN_STORY
Service Name: API_ELEKTRON_EPD_RSSL
    Fid: 4148 Name = TIMACT_MS DataType: UInt Value: 33548376
    Fid: 17 Name = ACTIV_DATE DataType: Date Value: 29 / 8 / 2018
    Fid: 8593 Name = MRN_TYPE DataType: Rmtes Value: STORY
    Fid: 8506 Name = MRN_V_MAJ DataType: Rmtes Value: 2
    Fid: 11787 Name = MRN_V_MIN DataType: Rmtes Value: 10
    Fid: 32480 Name = TOT_SIZE DataType: UInt Value: 2459
    Fid: 32479 Name = FRAG_NUM DataType: UInt Value: 1
    Fid: 4271 Name = GUID DataType: Rmtes Value: TOPEQU____17103067zn1WGFz85JDxS3E4DPKpIMX6jFL1ee9iqJvh
    Fid: 12215 Name = MRN_SRC DataType: Rmtes Value: HTC_PRD_A
    Fid: 32641 Name = FRAGMENT DataType: Buffer Value: <BUFFER>, limit = 2459, length = 2459
Init:: Expected total buffer size: 2459 current size: 2459
<<Single Fragment>>
<Content>
    {"altId":"nTOPEQU","audiences":["NP:E","NP:U","NP:NAW","NP:PSC"],"body":"> White House probes Google after Trump accuses it of bias                          [nL2N1VJ0HS]\n> Restaurant operator Yum China rejects $17.6 bln Hillhouse-led offer - sources     [nL3N1VK1DQ]\n> FOCUS-Proxy fight looms as new investors ramp up pressure on Campbell Soup        [nL2N1VI0LD]\n> Canada rejoins NAFTA talks as U.S. autos tariff details emerge                    [nL2N1VJ0VX]\n> Indian panel wants Johnson & Johnson to pay compensation over recalled implants   [nL3N1VK2JP]\n> GLOBAL MARKETS-Stocks hover as investors still on tenterhooks over trade deal     [nL8N1VK1Y5]\n> Altice, Dish urge U.S. to intervene in T-Mobile-Sprint deal                       [nL2N1VJ139]\n> Slowing online sales hurt Best Buy's 2nd qtr, shares drop                         [nL2N1VJ0AD]\n> Thomson Reuters launches $9 billion buyback, sending shares higher                [nL2N1VJ0QU]\n> Amazon plans new video app for Fire TV users - The Information                    [nL3N1VJ5R4]\n> Tesla wins court case against Ontario government over rebate cancellation         [nL2N1VJ18U]\n> EXCLUSIVE-Sterilization firm Sotera Health explores sale for nearly $5 bln        [nL2N1VJ1BV]\n> Ex-CEO sues Barnes & Noble over ouster linked to alleged harassment               [nL2N1VJ141]\n..............................................................................\n For a richer, multimedia version of Top News visit:\n * Eikon: https://amers1.views.cp.thomsonreuters.com/Explorer/NEWSxTOPNEWS.aspx?page=SP_PAGE_002&group=SP_GROUP_002&pageName=U.S.Companies\n        \n * Thomson ONE: Top News tab\n \n ..............................................................................\n For latest top breaking news across all markets                        [NEWS1]\n..............................................................................\nTOP NEWS SUMMARIES ON OTHER SUBJECTS\n\n  Front Page               [TOP/NEWS]   Central Banks & Global Economy [TOP/CEN]\n  World News                  [TOP/G]   Global Markets              [TOP/GLOMKT]\n  Foreign Exchange          [TOP/FRX]   Fixed Income                   [TOP/DBT]   \n  Emerging Markets         [TOP/EMRG]   Financial Services             [TOP/FIN]   \n  Investment Banking      [TOP/DEALS]   Wealth Mgmt-North America   [TOP/WEALTH]\n  IFR Markets              [TOP/NOW2]   U.S. Companies                 [TOP/EQU]\n  European Companies        [TOP/EQE]   Asian Companies                [TOP/EQA]\n  Commodities                [TOP/CE]   Energy                           [TOP/O] \n  Metals                    [TOP/MTL]   Agricultural Commodities       [TOP/GRO] \n  Global Gas Power & Gas    [TOP/ELE]   Industrials                  [TOP/BASIC] \n  Basic Mterials           [TOP/BMAT]   Healthcare                  [TOP/HEALTH]\n  Technology, Media,Telecoms[TOP/TMT]   Consumer Goods              [TOP/RETAIL]   \n  Sport                     [TOP/SPO]   Lifestyle & Entertainment     [TOP/LIFE]   \n  Top News Directory           [TOP/]\n  \n\nREGIONAL TOP NEWS PAGES\n\n  Greater China              [TOP/CN]   Deutschland (in German)         [TOP/DE]\n  Britain & Ireland          [TOP/GB]   France (in French)             [TOP/FRA]\n  Canada                    [TOP/CAN]   Latin America                [TOP/LATAM]\n  Brazil (in Portuguese)     [TOP/BR]   Middle East & North Africa   [TOP/MEAST]\n  Africa                     [TOP/ZA]   Australia & New Zealand        [TOP/ANZ]\n  Reuters BREAKINGVIEWS          [BRV]\n.............................................................................\n Access to some items may depend on subscription level\n ..............................................................................\n\n UP-TO-THE-MINUTE HEADLINES\n Company results                   [RES]  Economic indicators            [ECI]\n Mergers & acquisitions            [MRG]  Interest rates                 [INT]\n \n LIVE PRICES & DATA\n World Stocks                 <0#.INDEX>  Dow Jones industrials         <.DJI>\n Nasdaq                          <.IXIC>  FTSEurofirst 300            <.FTEU3>\n Nikkei                          <.N225>  FTSE 100                     <.FTSE>\n Currency rates            <EFX=> <NFX=>  Debt             <0#USBMK=><EURIBOR>\n U.S. crude oil                   <CLc1>          \n\n HOW TO FIND INFORMATION YOU NEED:\n|<REUTERS> | <NEWS> | <PHONE/HELP> |   <ENERGY>    |\n|<EQUITY>  | <BONDS>|   <MONEY>    |   <COMMODITY> |\n    \n..............................................................................\nContact Page Editor: Bangalore-pages@thomsonreuters.com; phone +1 651 848 5900 or +91 80 6749\n1306\n...............................................................................\n","firstCreated":"2017-10-30T01:50:14.000Z","headline":"*TOP NEWS*-U.S. Companies","id":"TOPEQU____17103067zn1WGFz85JDxS3E4DPKpIMX6jFL1ee9iqJvh","instancesOf":["RR:1305","NI:TOP/EQU"],"language":"en","messageType":6,"mimeType":"text/plain","provider":"NS:RTRS","pubStatus":"stat:usable","subjects":["A:1","A:T","G:6J","G:8W","M:3Z","M:MR","M:NP","M:PG","G:4","G:9","A:2","A:U","E:1","a1312cat:f","M:1QD","R:.FTSE","R:.DJI","R:.IXIC","R:.FTEU3","R:.N225","N2:STX","N2:CORPD","N2:US","N2:CA","N2:TOPNP","N2:TOPCMB","N2:REP","N2:CTXT","N2:AMERS","N2:NAMER","N2:DBT","N2:CDM","N2:BACT","N2:XRNP","N2:XREF","N2:LEN"],"takeSequence":11,"urgency":3,"versionCreated":"2018-08-29T09:19:08.000Z"}
</Content>

Item Name: MRN_STORY
Service Name: API_ELEKTRON_EPD_RSSL
    Fid: 4148 Name = TIMACT_MS DataType: UInt Value: 33549207
    Fid: 17 Name = ACTIV_DATE DataType: Date Value: 29 / 8 / 2018
    Fid: 8593 Name = MRN_TYPE DataType: Rmtes Value: STORY
    Fid: 8506 Name = MRN_V_MAJ DataType: Rmtes Value: 2
    Fid: 11787 Name = MRN_V_MIN DataType: Rmtes Value: 10
    Fid: 32480 Name = TOT_SIZE DataType: UInt Value: 2731
    Fid: 32479 Name = FRAG_NUM DataType: UInt Value: 1
    Fid: 4271 Name = GUID DataType: Rmtes Value: TOPDEALS__17103067ruER4+H+kwVqA2Q/TUXrvySeZkMceAapGtqN
    Fid: 12215 Name = MRN_SRC DataType: Rmtes Value: HTC_PRD_A
    Fid: 32641 Name = FRAGMENT DataType: Buffer Value: <BUFFER>, limit = 2600, length = 2600
Init:: Expected total buffer size: 2731 current size: 2600

Item Name: MRN_STORY
Service Name: API_ELEKTRON_EPD_RSSL
    Fid: 4271 Name = GUID DataType: Rmtes Value: TOPDEALS__17103067ruER4+H+kwVqA2Q/TUXrvySeZkMceAapGtqN
    Fid: 12215 Name = MRN_SRC DataType: Rmtes Value: HTC_PRD_A
    Fid: 32479 Name = FRAG_NUM DataType: UInt Value: 2
    Fid: 32641 Name = FRAGMENT DataType: Buffer Value: <BUFFER>, limit = 131, length = 1267
Additional fragment size: 131
Add Fragment:: Expected total buffer:2731 current 2731
<<Multi Fragment Completed>>
<Content>
    {"altId":"nTOPDEALS","audiences":["NP:C","NP:D","NP:E","NP:M","NP:O","NP:T","NP:U","NP:OIL","NP:NAT","NP:UKI","NP:GRO","NP:MTL","NP:SOF","NP:G","NP:RINVM","NP:PSC"],"body":"> Restaurant operator Yum China rejects $17.6 bln Hillhouse-led offer - sources     [nL3N1VK1DQ]\n> Aston Martin plans to go public as turnaround picks up speed                      [nL8N1VK0ST]\n> FOCUS-Proxy fight looms as new investors ramp up pressure on Campbell Soup        [nL2N1VI0LD]\n> Deutsche Bank CEO stands by global ambitions                                      [nS8N1LG01I]\n> Chinese biotech valuations likely hit as trailblazer's Hong Kong shares sink      [nL8N1VF1X4]\n> Altice, Dish urge U.S. to intervene in T-Mobile-Sprint deal                       [nL2N1VJ139]\n> Steinhoff's board to meet this week over asset sales to pay debt                  [nJ8N1UT02K]\n> Dutch insurer ASR may bid for rival Vivat if Chinese owner sells up               [nL8N1VK1YQ]\n> Japan's Itochu to buy stake in EV startup Singulato -sources                      [nL2N1VJ18V]\n> Chinese EV maker Nio expects to raise $1.32 bln in IPO                            [nL3N1VJ58A]\n> EXCLUSIVE-Sterilization firm Sotera Health explores sale for nearly $5 bln        [nL2N1VJ1BV]\n> Emergent BioSolutions to buy Narcan maker Adapt Pharma                            [nL3N1VJ5RY]\n> PJT Partners to buy CamberView as activism defense heats up                       [nL2N1VJ19C]\n  \n ..............................................................................\n For a richer, multimedia version of Top News visit:\n * Eikon: https://emea1.views.cp.thomsonreuters.com/Explorer/NEWSxTOPNEWS.aspx?page=SP_PAGE_012&group=SP_GROUP_011&pageName=FinancialServices\n    \n * Thomson ONE: Top News tab\n ...............................................................................\n For latest top breaking news across all markets                 [NEWS1]\n ............................................................................... \n TOP NEWS SUMMARIES ON OTHER SUBJECTS\n\n  Front Page               [TOP/NEWS]   Central Banks & Global Economy [TOP/CEN]\n  World News                  [TOP/G]   Global Markets              [TOP/GLOMKT]\n  Foreign Exchange          [TOP/FRX]   Fixed Income                   [TOP/DBT]   \n  Emerging Markets         [TOP/EMRG]   Financial Services             [TOP/FIN]   \n  Investment Banking      [TOP/DEALS]   Wealth Mgmt-North America   [TOP/WEALTH]\n  IFR Markets              [TOP/NOW2]   U.S. Companies                 [TOP/EQU]\n  European Companies        [TOP/EQE]   Asian Companies                [TOP/EQA]\n  Commodities                [TOP/CE]   Energy                           [TOP/O] \n  Metals                    [TOP/MTL]   Agricultural Commodities       [TOP/GRO] \n  Global Gas Power & Gas    [TOP/ELE]   Industrials                  [TOP/BASIC] \n  Basic Mterials           [TOP/BMAT]   Healthcare                  [TOP/HEALTH]\n  Technology, Media,Telecoms[TOP/TMT]   Consumer Goods              [TOP/RETAIL]   \n  Sport                     [TOP/SPO]   Lifestyle & Entertainment     [TOP/LIFE]   \n  Top News Directory           [TOP/]\n  \n\nREGIONAL TOP NEWS PAGES\n\n  Greater China              [TOP/CN]   Deutschland (in German)         [TOP/DE]\n  Britain & Ireland          [TOP/GB]   France (in French)             [TOP/FRA]\n  Canada                    [TOP/CAN]   Latin America                [TOP/LATAM]\n  Brazil (in Portuguese)     [TOP/BR]   Middle East & North Africa   [TOP/MEAST]\n  Africa                     [TOP/ZA]   Australia & New Zealand        [TOP/ANZ]\n  Reuters BREAKINGVIEWS          [BRV]\n    \n ..............................................................................\n Access to some items may depend on subscription level\n ..............................................................................\n UP-TO-THE-MINUTE HEADLINES \n Economic news                  [M]   Economic indicators               [M-ECI]\n G7 live indicators          <ECON>   Central bank news                 [M-CEN]\n U.S. company news              [U]   European companies             [E-EUROPE]\n Asian companies           [E-ASIA]   Mergers & acquisitions            [E-MRG]\n \n LIVE PRICES & DATA\n World Stocks            <0#.INDEX>   S&P 500                            <.SPX>\n Dow Jones Average           <.DJI>   NASDAQ                            <.IXIC>\n FTSEurofirst 300          <.FTEU3>   Nikkei                            <.N225>\n FTSE 100                   <.FTSE>   Currency rates                <EFX=><NFX=>\n Debt            <0#USBMK=><EURIBOR>   U.S. light crude oil              <CLc1>\n Brent crude                <LCOc1>   Gold                               <XAU=>\n                      \n GUIDES ON HOW TO FIND THE DATA AND NEWS INFORMATION YOU NEED\n| <REUTERS>  | <NEWS>   | <PHONE/HELP>   | <USER/HELP>   | <GB/EQUITY>|\n| <EQUITY>   | <BONDS>  | <MONEY>        | <COMMODITY>   | <SHIPPING> |\n| <ENERGY>   | <DERIV>  | <TREASURY>     |\n    \n................................................................................\nContact Page Editor: Bangalore-pages@thomsonreuters.com, +1 651 848\n5900 or +91 80 6749 1306\n................................................................................\n","firstCreated":"2017-10-30T00:55:58.000Z","headline":"*TOP NEWS* Investment Banking","id":"TOPDEALS__17103067ruER4+H+kwVqA2Q/TUXrvySeZkMceAapGtqN","instancesOf":["RR:3181","NI:TOP/DEALS"],"language":"en","messageType":6,"mimeType":"text/plain","provider":"NS:RTRS","pubStatus":"stat:usable","subjects":["A:Q","B:207","E:1","E:J","E:R","G:3","G:6J","G:7J","G:A","M:3Z","M:CV","M:MR","M:NP","M:P9","M:PG","M:Z","A:5","B:125","B:126","B:130","B:1645","B:208","B:261","B:262","B:263","G:4","G:9","B:1643","MCCL:OEC","MCC:OEC","M:1QD","R:.FTSE","R:.SPX","R:.DJI","R:.IXIC","R:.FTEU3","R:.N225","N2:PVE08","N2:INVBR","N2:BACT","N2:MRG","N2:DEAL1","N2:WEU","N2:US","N2:GB","N2:EUROP","N2:TOPNP","N2:CFIN","N2:TOPCMB","N2:REP","N2:WLTH08","N2:CTXT","N2:CMPNY","N2:FUND","N2:FINS08","N2:BISV08","N2:INVS08","N2:PVE","N2:INVM","N2:FINS","N2:BISV","N2:INVBIS","N2:AMERS","N2:NAMER","N2:WLTH","N2:XREF","N2:LEN"],"takeSequence":10,"urgency":3,"versionCreated":"2018-08-29T09:19:09.000Z"}
</Content>

Note: This tutorial will automatically exit after ~60 seconds.  Alternatively, you can (Ctrl+C) to exit at any time. Besides, you may occasionally see what looks like garbage output in the completed NTA console output; this is due to foreign language characters which cannot be represented correctly on the console. Displaying this using the appropriate character set is outside the scope of this tutorial. 

Tutorial Summary

So now that we've come to the end of the tutorial let's summarise what this example has demonstrated and what we've learned. The key goal was to show how to request and parse News Text Analytics data. We learned that the Initial Refresh message contains only metadata. Subsequent Update messages deliver the NTA item in one or more fragments. If delivered as a single fragment we extract the buffer and unzip to obtain the NTA item. If delivered as multiple fragments, they need to be concatenated in sequence to obtain the complete buffer, before unzipping.

Unix Version

A Unix build file, buildConsumer.ksh, has been provided under emaj-consumer folder.

Multiple NTA items

This tutorial deals with requesting a single NTA item which makes the handling of multi-fragment items relatively straightforward. When requesting multiple NTA items you need to allow for the Update messages for the different items arriving interspersed with each other. In view of this, the GUID and Source should be used to confirm a fragment is part of the same NTA item. As Update messages for multi-fragment messages arrive you would need to ensure the additional fragments are added to the correct incomplete item.

Thank you for reading.

References

For more information on the NTA domain and MRN Data model, please take a look at the following notes:

MRN Data Model and Elektron Implementation Guide - Summary of NTA domain, MRN data model, sample output, and implementation commentary.

Tutorial Group: 
EMA Consumer