Use Case Two

Refinitiv Data Platform News - Middle Office Compliance Use Cases

May 15,2022

Zoya Farberov

Introduction to Use Case

How a user can make a RDP API Request/Response call for news:

Published in the last 24 hours, where
The messages are tagged with PermIds (1, 2, 3, 4)
And the messages contain topic codes (x, y or z)

To impelement the use case we are going to explore RDP news filters. RDP news filtering is the primary focus of this use case, while specific filter example is chosen to illustrate it.

Approach

We are going to use RDP API ( HTTP REST) calls directly, as opposite to using Refinitiv Data Library Endpoint representation or Eikon Data Library Python get_news method.

Setup for RDP API Access
Obtain a Valid RDP Authentication Token
Request News Headlines - Define a Function
Request Headlines per Filter Requirements (by PermIds and Topics, for the last 24 hrs

Request News Headlines - Define a Function

    	
            def getNewsHeadlines(query, numLines, return_as_text=False):
    news_category_URL = "/data/news"
    headlines_endpoint_URL = "/headlines?"
 
    REQUEST_URL = base_URL + news_category_URL + RDP_version + headlines_endpoint_URL+ query # +"&limit="+ str(numLines);
    
    accessToken = getToken();
    print("Requesting: ",REQUEST_URL)
    
    acceptValue = "*/*"
    dResp = requests.get(REQUEST_URL, headers = {"Authorization": "Bearer " + accessToken, "Accept": acceptValue});
    
    if dResp.status_code != 200:
        print("Unable to get data. Code %s, Message: %s" % (dResp.status_code, dResp.text));
        if dResp.status_code != 401:   # error other then token expired
            return("Error "+str(dResp.status_code)) 
        accessToken = getToken();     # token refresh on token expired
        dResp = requests.get(REQUEST_URL, headers = {"Authorization": "Bearer " + accessToken, "Accept": acceptValue});
        if dResp.status_code == 200:
            print("Resource access successful")
    else:
        print("Resource access successful")
    if return_as_text:
        return dResp.text
    jResp = json.loads(dResp.text);
    return jResp

next, we are going to use the defined function to

Request Headlines per Filter Requirements (by PermIds and Topics, for the last 24 hrs)

Some of the aspects to note:

Max headlines request size at this time is defined at 100 headlines, that is also what is returned by default
If the expected required result set of headlines is larger then 100, the result is paginated and the received portions of the result are assemebled into a single result dataframe
Each headline contains a storyID which allows to retrieve the corresponding story

    	
            N = 100
query = 'query=(4295905573, 5030853586) and (searchIn:HeadlineOnly) and (last 24 hours)';
#query = 'query=(4295905573, 5030853586) and (Topic:E:F) and (searchIn:HeadlineOnly) and (last 24 hours)';
#query = 'query=(4295905573, 5030853586) and (Topic:SIG OR Topic:WEA) and (searchIn:FullStory) and (last 24 hours)';
#query = 'query=(4295905573, 5030853586) and (Topic:E:F) and (searchIn:HeadlineOnly) and (last 24 hours)';
jRespHeadlines = getNewsHeadlines(query, N)
 
dataAll = []
dfAll = pd.DataFrame()
iterations_count = 0;
while True:
    cursor = ""
    dfH = pd.json_normalize(jRespHeadlines,record_path =['data'])
    if not dfH.empty:
        dataAll.append(dfH)
        if 'meta' in jRespHeadlines:
            if 'next' in jRespHeadlines['meta']:
                # print('Meta.next=', jRespHeadlines['meta']['next'])
                cursor = urllib.parse.quote(jRespHeadlines['meta']['next'])
                print('Iteration= ',iterations_count,'Cursor= ', cursor)   
                query = 'cursor='+cursor;
    else:
        print('No results that qualify filter requirements')
    if not cursor:
        break
    iterations_count += 1
    jRespHeadlines = getNewsHeadlines(query, N)
 
if dataAll:    
    dfAll = pd.concat(dataAll,ignore_index=True)
    print('Displaying full result:')
    display(dfAll)

we may wish to mark the iterations, for a large result set, this allows us to knowhow we progress:

Once the headlines result is received in total, and there is no more results to be received, that qualify our filter requirement, we assess the received result in dataframe:

the columns of the result that are of most interest to us are likley newsItem.contentMeta.subject and storyID. As newsItem.contentMeta.subject is what the story is about should allow us to further zero in on the news stories that we require and storyID is used to request the stories that we select:

next we can just simply request a story or stories per storyID

Request a Story - Define a Function

    	
            def getStory(storyId, jsonOrHtml):
    news_category_URL = "/data/news"
    story_endpoint_URL = "/stories"
 
    REQUEST_URL = base_URL + news_category_URL + RDP_version + story_endpoint_URL+ "/" + storyId 
 
    accessToken = getToken();
    print("Requesting: ",REQUEST_URL)
    
    acceptValue = "application/json"
    if jsonOrHtml != True:
        acceptValue = "text/html"
    dResp = requests.get(REQUEST_URL, headers = {"Authorization": "Bearer " + accessToken, "Accept": acceptValue});
    if dResp.status_code != 200:
        print("Unable to get data. Code %s, Message: %s" % (dResp.status_code, dResp.text));
        if dResp.status_code != 401:   # error other then token expired
            return("") 
        accessToken = getToken();     # token refresh on token expired
    else:
        print("Resource access successful")
        return dResp.text

we can display the story in Json format:

for instance, we can see that the story we have selected, is not part of Top News ( for more on Top News and Top News Images see References). Or display the selected story of interest to us in HTML:

Request Story for Display

    	
            txt = getStory(dfAll.iloc[[0]]['storyId'].item(), False)
from IPython.core.display import display, HTML


display(HTML(txt))

or apply any required formatting on display. To gain more insight into stories and associated rich metadata that is made available via RDP News service, we may wish to use tools such as Json Viewer, see References for more information.

We conclude our present brief discussion of RDP News filtering middle office compliance use case, to discuss more, you can always find us on Refinitiv Developers Forums