Focusing on Significant Developments in News - Part 1

September 29,2022

Zoya Farberov

Introduction

Significant Developments provide real-time summaries and categorizations of crucial, market-moving company events – a news analysis and filtering service to help simplify your workflow.

Use Significant Developments (SigDev) information to perform in-depth research on specific topics like mergers, earnings, officer changes, and products.

Our Task "in a Nutshell"

Our primary interest is to absorb programmatically, or to filter, only the news that are marked with NS:SIGDEV code.

Therefore, a query expression in the simplest form could look like this, or this query stanza can also be as part of a more complex query filter:

    	
            GET https://api.refinitiv.com/data/news/{{RDP_VERSTION_NEWS}}/headlines?query=NS:SIGDEV&limit=50&dateFrom=2021-10-12T00:00:00Z&dateTo=2021-10-20T23:59:59Z

We plan to go over some of the most common use cases for obtaining Significant Developments:

From RDP News service
- Python jupyter notebook
- Postman request collection
Using SQS queues in Python script
Using realtime streaming service with Refinitiv-Realtime Optimized in python

Integration with Refinitiv Data Platform News Service

1. Request Significant Development headlines, query reflects the required filtering criteria

2. Requesting stories per selected headlines

Let us first examine an ease of use approach, using Refinitiv Data Library Python. The link to learning more about RD Library python is included in References section at the end of this artcile.

Significant Developments with Refinitiv Data Library Python

We include the required modules, setup the credentials, next, we open an RDP session and request significant developments headlines:

    	
            rd.open_session('platform.rdp')
 
response = news.headlines.Definition(
    query="NS:SIGDEV AND FB.O AND ES:1 BETWEEN 2021-10-21 AND 2022-10-01"
).get_data()
response.data.df

in this example we are looking at

NS:SIGDEV = Significant Developments headlines
FB.O = on Facebook
ES:1 = of High Significance only
Dated between 2021-10-21 and 2022-10-01

and our results in response Pandas dataframe reflect the requested:

next we can request the first story:

    	
            response2 = news.story.Definition(response.data.df['storyId'][0]).get_data() #"urn:newsml:reuters.com:20211003:nNRAgvhyiu:1").get_data()
#print(response2.data.raw)
if 'News Alert' in str(response2.data.raw):
    print('<<<This is an ALERT>>>')
else:
    print('TITLE: ',response2.data.story.title, '\n')
    print(response2.data.story.content)

and the output will reflect the first story of the result:

however, most commonly, we wish to iterate over the headlines and retrieve all the stories that match the criteria:

    	
            count = 0;
for storyIdIter in response.data.df['storyId']:
    print('Story ID #'+str(count)+' is : '+storyIdIter)
    response3 = news.story.Definition(storyIdIter).get_data()
    if 'News Alert' in str(response3.data.raw):
        print('<<<This is an ALERT>>>')
    else:
        print('TITLE: ',response3.data.story.title, '\n')
        print(response3.data.story.content)
    print('---------------------------------------------')
    count = count+1

Significant Developments with Postman

This is also a very easy and quick retrieval per required criteria.

RDP News Significant Development collection (the link to download it from GitHub is included in References section) consists from just 3 requests:

Get RDP Access Token
Get Headlines and Stories For Significant Developments
Get Story Per StoryId

Postman post-request tests and collection runner are used as a glue between the discrete requests, to enact the continuous run and to retrieve all the stories per criteria.

Get RDP Access Token request picks up credentials from Postman environment, requests and obtains an access token and stores it back into Postman environment
Get Headlines and Stories For Significant Developments, as reflected in the name, requests the stories, from RDP news, per required criteria:

    	
            GET https://api.refinitiv.com/data/news/{{RDP_VERSTION_NEWS}}/headlines?query=FB.O AND NS:SIGDEV&imit=50&dateFrom=2021-06-12T00:00:00Z&dateTo=2021-10-20T23:59:59Z

while in it's post-request test, we prepare to retrieve stories based on the storyIds contained in the current request's results that we store into Postman environment:

    	
            var jsonData = pm.response.json();
var arrStoryIds = [];
for(let i = 0; i < jsonData.data.length; i++) {
//    console.log(jsonData.data[i].storyId)
    arrStoryIds[i] = jsonData.data[i].storyId;
}
 
console.log('StoryIDs length='+arrStoryIds.length+',\n storyIDs='+arrStoryIds);
//console.log('StoryIDs 7th-'+arrStoryIds[7])
postman.setEnvironmentVariable("STORY_IDS", JSON.stringify(arrStoryIds));
postman.setEnvironmentVariable("CURRENT_ID", 0);

3. Get Story per StoryId is responsible for retrieving the single story:

    	
            GET https://api.refinitiv.com/data/news/{{RDP_VERSTION_NEWS}}/stories/{{CURRENT_STORY_ID}}

but also in it's post-request test it will help us iterate over to the next StoryId and trigger the same request with it:

    	
            let storyIds = JSON.parse(postman.getEnvironmentVariable("STORY_IDS"));
let currentId = Number(postman.getEnvironmentVariable("CURRENT_ID"));
 
console.log('Current story ID ='+ storyIds[currentId])
var jsonData = pm.response.json();
// pretty print to console is very costly/slow, so unless required, keep it commented out
console.log('Current story ='+ JSON.stringify(jsonData))//,null, 2))  
 
let nextCurrentId = currentId + 1
if (nextCurrentId < storyIds.length) {
    postman.setEnvironmentVariable("CURRENT_ID", nextCurrentId);
    postman.setNextRequest("Get Story per StoryId");
    console.log("nextCurrentId="+nextCurrentId)
} else {
    postman.setNextRequest(null)
    console.log('This was the last story<<<>>>')
}