Thomson Reuters Tick History (TRTH) - REST API

.Net SDK Tutorial 4: GUI control calls: Validate, extraction

Last update July 2019
Environment Windows
Language C#
Compilers Microsoft Visual Studio 2012/2013
Prerequisites DSS login, internet access, having done the previous tutorial
Source code Download .Net SDK Tutorials Code

Tutorial purpose

This is the fourth tutorial in a series of .Net SDK tutorials. It is assumed that the reader has acquired the knowledge delivered in the previous tutorials before following this one.

This tutorial looks at more programmatic GUI control. Building on the previous tutorial, it covers the following new topics:

  • Checking instrument identifiers validation results when creating an instrument list.
  • Launching a data extraction on the DSS server, waiting for it to complete, and checking its status.
  • Retrieving and displaying the retrieved data and extraction notes.

 

Table of contents

 

Getting ready

Opening the solution

The code installation was done in Tutorial 1.

Opening the solution is similar to what was done in Tutorial 1:

  • Navigate to the \API TRTH REST\Tutorial 4\Learning folder.
  • Double click on the solution file restapi_set_runtime.sln to open it in Microsoft Visual Studio

 

Referencing the DSS SDK

Important: this must be done for every single tutorial, for both the learning and refactored versions.

This was explained in the tutorial 2; please refer to it for instructions.

For this and the following tutorial an additional library must be referenced: SharpZipLib from ICSharpCode. The DLL is delivered with the code samples, but if you prefer you can also use NuGet.

 

Viewing the C# code

In Microsoft Visual Studio, in the Solution Explorer, double click on Program.cs and on DssClient.cs to display both file contents. Each file will be displayed in a separate tab.

 

Setting the user account

Before running the code, you must replace  YourUserId  with your DSS user name, and  YourPassword  with your DSS password, in these 2 lines of Program.cs::

private static string dssUserName = "YourUserId";
private static string dssUserPassword = "YourPassword";

Important reminder: this will have to be done again for every single tutorial, for both the learning and refactored versions.

Failure to do so will result in an error at run time (see Tutorial 1 for more details).

 

Understanding extraction files

Once a schedule has run, and the requested data was extracted from the DSS server database, several resulting extraction files are generated. They are:

  1. The extracted data itself. The file is compressed.
  2. The extraction notes. This file contains details of the extraction.

 

The names of the files (generated by a single extraction) share a common root, with differing extensions added. The root file name can be specified when creating a schedule; as this is optional, if none was specified the DSS server will automatically generate a (very complex) root file name.

Example file names, for a specified root file name myExtraction:

  1. Extracted data:               myExtraction.csv.gz
  2. Extraction notes:           myExtraction.csv.gz.notes.txt

 

The extraction files are stored on the DSS server. They can be retrieved using various mechanisms:

  • Manually, through the web GUI.
  • Programmatically, through the API.

This tutorial and the following demonstrate the 2rd method, using the API. Manual retrieval is out of the scope of these tutorials; for these, more information is available in the help pages of the web GUI.

 

When using the API:

  • The list of extraction files can be retrieved, as a property of the extraction context. This list allows us to see how many files there are, and what the individual file names are.
  • We use API calls to read the file contents on the server, and return these contents to our program, for further treatment. That could be as simple as saving the file to disk, or be more involved and imply on the fly treatment. Note that the extraction note file is a text file, whereas the data file is a gzipped CSV file. This difference must be taken into account when treating the files.

Read on to understand how this is done.

 

Understanding the code

We shall only describe what is new versus the previous tutorial.

 

DssClient.cs

This is nearly the same as the refactored version of Tutorial 3, except for the leading comment, and 2 small changes:

We expose our extractions context to our main program:

public ExtractionsContext extractionsContext;

We add Monday to our recurring schedule creation method (to avoid the demo failing on a Monday).

No additional explanations are required as the rest of the code was described in the previous tutorials.

 

Program.cs

At the top of the code we see the using directives. As we are using a whole set of API calls, several using directives are required to give us access to the types of the DSS API namespace, so that we do not have to qualify the full namespace each time we use them. As in the previous tutorials, these are followed by a using directive referring to the namespace of our code. In addition, we have a directive for the SharpZipLib library:

using ICSharpCode.SharpZipLib.GZip;
using ThomsonReuters.Dss.Api.Extractions;
using ThomsonReuters.Dss.Api.Content;
using ThomsonReuters.Dss.Api.Extractions.SubjectLists;
using ThomsonReuters.Dss.Api.Extractions.ReportTemplates;
using ThomsonReuters.Dss.Api.Extractions.Schedules;
using ThomsonReuters.Dss.Api;
using ThomsonReuters.Dss.Api.Extractions.ReportExtractions;
using ThomsonReuters.Dss.Api.Core;
using DssRestfulApiTutorials;

In the main code block, after connecting to the server, we expose the extractions context from DssClient (to avoid having to use the dssClient. prefix before each use of extractionsContext):

ExtractionsContext extractionsContext = dssClient.extractionsContext;

 

Validating instrument identifiers after creating the instrument list

In the previous tutorial we created an instrument list, but did not worry about validating the instrument identifiers; we just assumed they were successfully appended to the list. In this tutorial we want to check if any issues occurred.

To illustrate this, this time we intentionally include 2 invalid instrument identifiers (in an array of 7):

new InstrumentIdentifier
    { Identifier = "INVALID", IdentifierType = IdentifierType.Cusip,
      UserDefinedIdentifier = "INVALID INSTRUMENT_1" },
new InstrumentIdentifier
    { Identifier = "JUNK.JUNK", IdentifierType = IdentifierType.Ric,
      UserDefinedIdentifier = "INVALID INSTRUMENT_2" }

To validate the instrument identifiers, we capture the result returned by the API call that appends instrument identifiers to an instrument list:

InstrumentsAppendIdentifiersResult appendResult =
    extractionsContext.InstrumentListOperations.AppendIdentifiers(
        instrumentList, instrumentIdentifiers, false);

This result contains useful instrument identifier validation information. For each invalid instrument identifier a message and severity are generated. We can retrieve and display them:

foreach (InstrumentValidationMessage message in appendResult.ValidationResult.Messages)
    Console.WriteLine("- " + message.Message, message.Severity);

We can also get the valid instrument count:

DebugPrintAndWaitForEnter("Valid instrument count: " +
    appendResult.ValidationResult.ValidInstrumentCount + "\nCheck The DSS GUI...");

The output therefore displays 1 line for each invalid instrument, and the valid instrument count:

 

Important note: inactive and historical instruments might be rejected, like a matured bond or an instrument that changed name. This tutorial includes a matured bond (Cusip 438516AC0). You can decide if historical and inactive instruments are rejected by setting user preferences in the web GUI:

This API feedback data can be used to log errors when managing instrument lists using the REST API.

In the web GUI the instrument list contains only the validated instrument identifiers, those not found were automatically rejected:

The results also contain some high level information on the segmentation by asset class of the instrument identifiers. The following code displays this information:

Console.WriteLine("Asset classes count: " + appendResult.ValidationResult.StandardSegments.Count);
Console.WriteLine("Asset classes codes and descriptions: ");
for (int i = 0; i < appendResult.ValidationResult.StandardSegments.Count; i++)
{
    Console.WriteLine("- Code: " + appendResult.ValidationResult.StandardSegments[i].Code +
        " - Description: " + appendResult.ValidationResult.StandardSegments[i].Description);
}

The valid instruments in this tutorial are all equity stocks. The segmentation results will be:

 

Creating a report template

This is exactly the same as what we did in the previous tutorial. In the next step we use the returned report template ID.

 

Creating an extraction schedule

We create a daily recurring schedule. This is a variation of the second schedule we created in the preceding tutorial, the difference being in the definition of the trigger. The preceding one was set to run at 2 am. This one is also for a set time, but for testing purposes it is set to run 2 minutes in the future, that way we quickly get results every time we run the tutorial:

int waitMinutes = 2;
DateTimeOffset dateTimeNow = DateTimeOffset.Now;
DateTimeOffset dateTimeSchedule = dateTimeNow.AddMinutes(waitMinutes);
int scheduleHour = dateTimeSchedule.Hour;
int scheduleMinute = dateTimeSchedule.Minute;

Schedule recurringSchedule = new Schedule
{
    Name = "myRecurringSchedule",
    TimeZone = TimeZone.CurrentTimeZone.StandardName,
    Recurrence = ScheduleRecurrence.CreateWeeklyRecurrence(
        new[] { DayOfWeek.Monday, DayOfWeek.Tuesday, DayOfWeek.Wednesday,
                DayOfWeek.Thursday, DayOfWeek.Friday }),
    // This trigger is on a set time:
    Trigger = ScheduleTrigger. CreateTimeTrigger(
        false,                                   //Limit to today's data: no
        new[] { new HourMinute { Hour = scheduleHour, Minute = scheduleMinute } }),
    ListId = instrumentList.ListId,
    ReportTemplateId = reportTemplateId
};

Note: you can also specify a time zone by name:

TimeZone = "Eastern Standard Time",

As in the previous tutorial, we create the extraction schedule, using the extraction schedule definition, ad retrieve the returned schedule ID:

extractionsContext.ScheduleOperations.Create(recurringSchedule);
string recurringScheduleId = recurringSchedule.ScheduleId;
DebugPrintAndWaitForEnter("Returned recurring schedule ID: " +
    recurringScheduleId + "\nCheck the DSS GUI...");

In the previous tutorial we defined the output file name for the schedule. As we did not do it here, a default name will be generated by the DSS server. This is not important, because we do not intend to retrieve the file manually, we want to get the data programmatically, via the API, as described next.

 

 

Waiting for the extraction to complete

The extraction will be launched by the schedule.

The next step is to wait for the extraction to complete. This part of the code is inside a try / catch block, to trap exceptions.

We use WaitForNextExtraction, a blocking API call which only returns when the extraction is complete (or times out), so there is no need for code to check for extraction completion.

To achieve this, the API call automatically polls the server at regular intervals, to check if the extraction has completed. The poll period is configurable; it should not be too frequent so as not to unnecessarily burden the network and server. The timeout period is also configurable. In the tutorial code we poll once a minute, and set the timeout to 5 minutes:

TimeSpan timeOut = TimeSpan.FromMinutes(5);
int pollPeriod = 60;
ReportExtraction extraction =
    extractionsContext.ScheduleOperations.WaitForNextExtraction(recurringSchedule, pollPeriod, timeOut);

This is one way to do it, but it is also possible to program this differently, using other API calls to check the status of an extraction by Id, or to retrieve completed extractions for a date range, etc. These calls are documented in the REST API Reference tree, and in the Scheduled Extractions examples in the C# example application.

 

Checking the extraction results

Once the extraction has completed, we can display its status and detailed status:

Console.WriteLine("\nStatus: {0} - Detailed status: {1}",
    extraction.Status, extraction.DetailedStatus);

Here is the result:

 

Retrieving the extracted files

We can then retrieve the list of files generated by the extraction. Be aware that the toolkit only populates this list in the extractions context when you explicitly call LoadProperty(), so we must do that systematically before checking the file count value which tells us how many extraction files were generated:

extractionsContext.LoadProperty(extraction, "Files");
Console.WriteLine("File count: " + extraction.Files.Count + "\n");

Here is the result:

We see that 2 files were generated. As explained above in the section on Understanding extraction files, they are:

  1. The extraction notes, containing details of the extraction. The file is of type Note.
  2. The extracted data itself. The file is of type Full, and is in compressed format.

 

If we wanted to retrieve only the data file, and ignore the other file, we could search for the first file where the file type is “Full” (only the data file will be of this type):

ExtractedFile extractedDataFile =
    extraction.Files.FirstOrDefault(f => f.FileType == ExtractedFileType.Full);

 

But in our sample code we decided to retrieve all files instead:

IEnumerable<ExtractedFile> extractedFiles = extraction.Files;

Then we treat each file, using a helper method:

int fileNumber = 0;
foreach (ExtractedFile extractedFile in extractedFiles)
{
    fileNumber++;
    DisplayFileDetailsAndContents(extractionsContext, extractedFile, fileNumber);
}
if (fileNumber == 0) { DebugPrintAndWaitForEnter("ERROR: extracted data file was null"); }

The helper method is defined further in the code, it displays the file name, type, receive time stamp (in UTC) and its size. Furthermore, if the file size is greater than zero, it reads the file contents, and then displays them. Finally, if it is a notes file it checks for successful processing.

:Note that we disable automatic decompression. This is because we want to download the compressed data, and, if required, decompress it using a dedicated package. We used SharpZipLib, other libraries to consider include zlib and DotNetLib. The result is more reliable and robust code that can handle large data sets.

static void DisplayFileDetailsAndContents(
    ExtractionsContext extractionsContext, ExtractedFile file, int fileNumber)
{
    //Do not automatically decompress data file content:
    extractionsContext.Options.AutomaticDecompression = false;

    Console.WriteLine("================================ FILE DETAILS =================================" +
        "\nFile number: " + fileNumber +
        "\nFile name: " + file.ExtractedFileName +
        "\nFile type: " + file.FileType +
        "\nReceived date utc:" + file.ReceivedDateUtc +
        "\nFile size:" + file.Size);
    if (file.FileType.ToString() == "Note")
    {
        Console.WriteLine("=================================== COMMENT ===================================");
        Console.WriteLine("File type is \"Note\": it contains the extraction notes.");
    }
    if (file.FileType.ToString() == "Full")
    {
        Console.WriteLine("=================================== COMMENT ===================================");
        Console.WriteLine("File type is \"Full\": it contains the data.");
    }
    DebugPrintAndWaitForEnter("===============================================================================");

    //Don't fetch the content if the file is empty:
    if (file.Size == 0) return;

    Console.WriteLine("================================ FILE CONTENTS ================================");

    //Direct download from AWS ?
    if (awsDownload && file.FileType.ToString() == "Full")
    {
        extractionsContext.DefaultRequestHeaders.Add("x-direct-download", "true");
    };

    DssStreamResponse streamResponse =
        extractionsContext.ExtractedFileOperations.GetReadStream(file);

    if (file.FileType.ToString() == "Full")
    //The data file is compressed. Here we decompress it in stream.
    //For robust decompression we use SharpZipLib, a 3rd party package.
    {
        int count = 0;
        string line = "";
        using (GZipInputStream gzip = new GZipInputStream(streamResponse.Stream))
        {
            using (StreamReader reader = new StreamReader(gzip, Encoding.UTF8))
            {
                line = reader.ReadLine();
                if (string.IsNullOrEmpty(line))
                {
                    Console.WriteLine("WARNING: no data returned. Check your request dates.");
                }
                else
                {
                    //The first line is the list of field names:
                    Console.WriteLine(line);
                    count++;
                    //The remaining lines are the data:
                    while ((line = reader.ReadLine()) != null)
                    {
                        Console.WriteLine(line);
                        count++;
                    }
                }
            }
        }
        Console.WriteLine("Line count: " + count);
    }

    if (awsDownload && file.FileType.ToString() == "Full")
    {
        extractionsContext.DefaultRequestHeaders.Remove("x-direct-download");
    };

    if (file.FileType.ToString() == "Note")
    // The notes file is small and uncompressed.
    {
        string streamContents;
        using (StreamReader reader = new StreamReader(streamResponse.Stream))
            Console.WriteLine(streamContents = reader.ReadToEnd());

        Console.WriteLine("=================================== COMMENT ===================================");
        if (streamContents.IndexOf("Processing completed successfully") != -1)
        {
            Console.WriteLine("SUCCESS: Processing completed successfully.");
        }
        else
        {
            Console.WriteLine("ERROR: Processing did not complete successfully !");
        }
    }
    DebugPrintAndWaitForEnter("===============================================================================");
}

Note that if we use AWS downloads, we set the additional request header for the data file only, because that is the only one that can be delivered by AWS (AWS will not deliver the Note file).

The order of the returned files might vary.

Here is the result for the note file:

As no data was returned for the bond, it is logged in the extraction note file:

Instrument <CSP,438516AC0 expanded to 0 RICs.

Here is the beginning of the result for the data file. Note that it is a compressed file, which was uncompressed on the fly by our code, to display on screen:

The resulting data might not contain all instruments, even though they were not rejected when we added them to the instrument list. This simply depends on the availability of data for the requested period.

In our specific case here there is no returned data for the bond, because it matured on the 6th January 2016, and we requested data for the 29th September 2016.

 

Cleaning up

Like in the previous tutorial, the instrument list, report template and schedule are deleted.

 

Full code

The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.

 

Summary

List of the main steps in the code:

  1. Authenticate by creating an extraction context.
  2. Create an array of financial instrument identifiers.
  3. Create an instrument list.
  4. Append the array of financial instrument identifiers to the instrument list, to populate it.
  5. Check the instrument identifiers validation results.
  6. Define a record template.
  7. Create the report template, using the defined template.
  8. Create an extraction schedule.
  9. Wait for the extraction to complete.
  10. Check the extraction results.
  11. Retrieve the extracted data.
  12. Cleanup.

 

Code run results

Build and run

Don’t forget to reference the DSS SDK and the SharpZipLib, and to set your user account in Program.cs !

Successful run

After running the program, and pressing the Enter key when prompted, the final result should look like this:









Intermediary results are discussed at length in the code explanations in the previous section of this tutorial.

Press Enter one final time to close the pop-up and end the program.

 

Potential errors and solutions

If the user name and password were not set properly, an error will be returned. See Tutorial 1 for details.

If the program attempts to create an instrument list that has the same name as that of an instrument list that is already stored on the DSS server, an error will be generated. Similar errors will arise for report template or extraction schedule that has the same name as an existing one. See Tutorial 3 for details.

 

Understanding the refactored version

Explanations

DSS client helper class file: DssClient.cs

Obvious comments were removed. We modified the method that appends identifiers, to return validation results:

public InstrumentsAppendIdentifiersResult AppendIdentifiersToInstrumentList(
    IEnumerable<InstrumentIdentifier> instrumentIdentifiers, InstrumentList instrumentList)
{
    InstrumentsAppendIdentifiersResult appendResult =
        extractionsContext.InstrumentListOperations.AppendIdentifiers(
            instrumentList, instrumentIdentifiers, false);
    return appendResult;
}

 

The method to create a recurring schedule was modified to take the schedule hour and minute as parameters:

public string CreateNextBusinessDayRecurringSchedule(
    string scheduleName, string instrumentListId, string reportTemplateId,
    int scheduleHour, int scheduleMinute, string outputFileName)
{
    Schedule schedule = new Schedule
    {
        Name = scheduleName,
        TimeZone = TimeZone.CurrentTimeZone.StandardName,
        Recurrence = ScheduleRecurrence.CreateWeeklyRecurrence(
            new[] { DayOfWeek.Monday, DayOfWeek.Tuesday, DayOfWeek.Wednesday,
                    DayOfWeek.Thursday, DayOfWeek.Friday, DayOfWeek.Saturday }),
        //This trigger is on a set time (regardless of data availability):
        Trigger = ScheduleTrigger.CreateTimeTrigger(
            false,                                   //Limit to today's data: no
            new[] { new HourMinute { Hour = scheduleHour, Minute = scheduleMinute } }),
        ListId = instrumentListId,
        ReportTemplateId = reportTemplateId,
        OutputFileName = outputFileName
    };

    extractionsContext.ScheduleOperations.Create(schedule);
    return schedule.ScheduleId;
}

 

A new cleanup method was added, to delete the instrument list, report template and schedule, with minimal error trapping:

public void Cleanup(
    DssClient dssClient, string instrumentListId, string reportTemplateId, string scheduleId)
{
    try { dssClient.DeleteInstrumentList(instrumentListId); }  catch { };
    try { dssClient.DeleteReportTemplate(reportTemplateId); }  catch { };
    try { dssClient.DeleteSchedule(scheduleId); }              catch { };
}

 

 

Main program file: Program.cs

We removed the on screen comments telling us to check the DSS GUI.

To populate the instrument list and retrieve validation results, we call our new DSS client helper method:

InstrumentsAppendIdentifiersResult appendResult =
    dssClient.AppendIdentifiersToInstrumentList(
        instrumentIdentifiers, instrumentList);

 

To display the validation results, we call a helper method:

DisplayAppendResults(appendResult);

As this helper method is specific to our workflow, we declare it in Program.cs instead of DssClient.cs, after the main code:

static void DisplayAppendResults(InstrumentsAppendIdentifiersResult appendResult)
{
    Console.WriteLine("Instrument identifiers validation results:");
    Console.WriteLine("Validation messages:");
    foreach (InstrumentValidationMessage message in appendResult.ValidationResult.Messages)
        Console.WriteLine("- " + message.Message, message.Severity);

    DebugPrintAndWaitForEnter("Valid instrument count: " +
        appendResult.ValidationResult.ValidInstrumentCount);

    //The standard segments are separations of data by asset class.
    //Display them:
    Console.WriteLine("Asset classes count: " + appendResult.ValidationResult.StandardSegments.Count);
    Console.WriteLine("Asset classes codes and descriptions: ");
    for (int i = 0; i < appendResult.ValidationResult.StandardSegments.Count; i++)
    {
        Console.WriteLine("- Code: " + appendResult.ValidationResult.StandardSegments[i].Code +
            " - Description: " + appendResult.ValidationResult.StandardSegments[i].Description);
    }
    DebugPrintAndWaitForEnter("");
}

 

To create the schedule, we call our new DSS client helper method:

string recurringScheduleId = dssClient.CreateNextBusinessDayRecurringSchedule(
    "myRecurringSchedule", instrumentListId, reportTemplateId,
    scheduleHour, scheduleMinute, "myTimedRecurringExtractionOutput.csv");

As we need the schedule to launch the extraction, we retrieve it by name:

Schedule recurringSchedule =
    extractionsContext.ScheduleOperations.GetByName("myRecurringSchedule");

 

To clean up, we call our new DSS client helper method:

dssClient.Cleanup(dssClient, instrumentListId, reportTemplateId, recurringScheduleId);

 

Full code

The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.

 

Build and run

Don’t forget to reference the DSS SDK, and to set your user account in Program.cs !

 

Conclusions

This tutorial shows how to do a scheduled data extraction, like we could do in the web GUI.

 

Now move on to the next tutorial, which shows how to do an On Demand data extraction.

 

Tutorial Group: 
.Net SDK Tutorials