Thomson Reuters Tick History (TRTH) - REST API

.Net SDK Tutorial 5: On Demand extraction, file IO

Last update July 2019
Environment Windows
Language C#
Compilers Microsoft Visual Studio 2012/2013
Prerequisites DSS login, internet access, having done the previous tutorial
Source code Download .Net SDK Tutorials Code

Tutorial purpose

This is the fifth tutorial in a series of .Net SDK tutorials. It is assumed that the reader has acquired the knowledge delivered in the previous tutorials before following this one.

This tutorial moves away from the scheduled extractions which mimic what can be done using the GUI control, and covers a new topic: On Demand extractions, using as example an intraday summaries (bar) extraction.

In this tutorial we therefore use the simplified high level API calls for On Demand extractions. We also include some file input and output, and some error handling:

  • Populate a large instrument list array read from a CSV file, outside of the DSS server. Errors encountered when parsing the input file are logged to an error file.
  • Create an array of data field names, outside of the DSS server.
  • Optionally set instrument validation parameters, for instance to allow historical instruments.
  • Create and run an on demand data extraction on the DSS server, wait for it to complete.
  • Retrieve and display the data and extraction notes. We also write them to separate files, and analyse the notes file to find issues.

The file input and output do not illustrate additional TRTH REST API functionality, but serve to put it in the context of slightly more productized code.

 

Table of contents

 

Getting ready

Opening the solution

The code installation was done in Tutorial 1.

Opening the solution is similar to what was done in Tutorial 1:

  • Navigate to the \API TRTH REST\Tutorial 5\Learning folder.
  • Double click on the solution file restful_api_more_advanced_topics.sln to open it in Microsoft Visual Studio

 

Referencing the DSS SDK

Important: this must be done for every single tutorial, for both the learning and refactored versions.

This was explained in the tutorial 2; please refer to it for instructions.

For this and the preceding tutorial an additional library must be referenced: SharpZipLib from ICSharpCode. The DLL is delivered with the code samples, but if you prefer you can also use NuGet.

 

Viewing the C# code

In Microsoft Visual Studio, in the Solution Explorer, double click on Program.cs and on DssClient.cs to display both file contents. Each file will be displayed in a separate tab.

 

Setting the user account

Before running the code, you must replace  YourUserId  with your DSS user name, and  YourPassword  with your DSS password, in these 2 lines of Program.cs::

private static string dssUserName = "YourUserId";
private static string dssUserPassword = "YourPassword";

Important reminder: this will have to be done again for every single tutorial, for both the learning and refactored versions.

Failure to do so will result in an error at run time (see Tutorial 1 for more details).

 

Understanding the On Demand calls

The On Demand high level calls we use in this tutorial depart from what was done in the previous tutorials.

As stated in the Tutorial Introduction, in the note on programming approaches, there are 2 approaches:

  • Create (and modify when required) an instrument list, report template and extraction schedule, all stored on the DSS server, and retrieve your data once the extraction has completed. This mimics what you can do with the web GUI. This approach was used in tutorials 3 and 4.
  • Using On Demand advanced API calls, it is also possible to do on the fly data requests without storing an instrument list, report template and extraction schedule on the DSS server. It is a more direct and dynamic way of doing things. In this case the instrument list, report template and extraction schedule are ephemeral, and cannot be seen in the DSS web GUI. This approach is used in this tutorial.

The choice is yours, and will depend on your use case and personal preferences.

 

The second approach we explore now has two consequences:

  1. We cannot see any instrument list, report template, schedule or result file using the web GUI.
  2. Cleanup on the DSS server is not required, as there is nothing to delete.

 

Once an extraction has completed, the results returned by the API contain:

  1. The extracted data itself, in compressed format.
  2. The extraction notes. This contains details of the extraction.

These results are returned by the server to our program, for further treatment. That could be as simple as saving the file to disk, or be more involved and imply on the fly treatment. Note that the extraction notes are pure text, whereas the data is gzipped CSV. This difference must be taken into account when treating the files.

Read on to understand how this is done.

 

Important warning on data usage

Tick history data request quota

DSS accounts have a quota of instruments that can be requested during a defined time period. The subscription fees are proportional to the size of the quota. If the quota is reached, excess fees will be due. That situation must be avoided !

Data usage can be displayed in the web GUI:

Contact your local account manager or sales specialist if you have queries on your account quota or usage.

 

Instrument list files delivered with the tutorials

The code associated with this tutorial can handle large instrument lists, read from a file.

Care must be exercised when running this tutorial, to avoid over consuming data.

For this reason, several files are delivered with the tutorials code:

  • DSS_API_1942_input_file.csv
  • DSS_API_86_input_file.csv
  • DSS_API_10_input_file.csv

The first one contains slightly less than 2000 valid instrument identifiers, the second one 86, the last one only contains 10.

The text and illustrations that follow use the larger file. To avoid consuming too much data, we recommend you use the smaller file for your own tests, which will deliver similar results.

 

Instrument expansion

The medium and large instrument list files do not seem to contain so many instruments, but they contain chains, ISINs and Sedols. When the extraction request is processed by the server, the chains will be expanded to their constituents, and the ISINs will be expanded to the list of all corresponding RICs. The expansion mechanism therefore increases the number of instruments in the request, in some cases quite dramatically.

 

Understanding the code

We shall only describe what is new versus the previous tutorials.

Note: contrary to previous tutorials, this one does not have a refactored version.

DssClient.cs

This is the same as the refactored version of Tutorial 4, except for the leading comment.

 

Program.cs

At the top of the code we see the using directives. As we are using a whole set of API calls, several using directives are required to give us access to the types of the DSS API namespace, so that we do not have to qualify the full namespace each time we use them. As in the previous tutorials, these are followed by a using directive referring to the namespace of our code. In addition, we have a directive for the SharpZipLib library:

using ICSharpCode.SharpZipLib.GZip;
using ThomsonReuters.Dss.Api.Extractions;
using ThomsonReuters.Dss.Api.Content;
using ThomsonReuters.Dss.Api.Extractions.ReportTemplates;
using ThomsonReuters.Dss.Api.Extractions.ExtractionRequests;
using ThomsonReuters.Dss.Api.Core;
using DssRestfulApiTutorials;

 

Member declarations

We add a whole set of declarations. The first is related to where we want to download the data from. It is now possible to choose to download the data file either from the TRTH servers, or directly from the AWS (Amazon Web Services) cloud. It is recommended to use AWS as it delivers faster downloads:

private static bool awsDownload = true;

The following define the input and output files and their locations. As output we will create a data file and a separate error log file. As input we have an instrument identifier list:

private static string outputDirectory = "C:\\API TRTH REST\\";
private static string gzipDataOutputFile = outputDirectory + "TRTH_API_tutorial_5l_data.csv.gz";
private static string csvDataOutputFile = outputDirectory + "TRTH_API_tutorial_5l_data.csv";
private static string csvDataOutputFile2 = outputDirectory + "TRTH_API_tutorial_5l_data2.csv";
private static string noteOutputFile = outputDirectory + "TRTH_API_tutorial_5l_note.txt";
private static string errorOutputFile = outputDirectory + "TRTH_API_tutorial_5l_errors.txt";

private static string inputDirectory = outputDirectory;
private static string instrumentIdentifiersInputFile = inputDirectory + "TRTH_API_10_input_file.csv";

The default directories and file names above can be used as is if you installed the tutorials code as described in Tutorial 1, using the directory structure defined in the Quick Start. If you are using a different folder structure, change the declarations accordingly.

 

The input file is a CSV file. 3 samples are included with the code, but you can also make your own:

  • File format: 1 instrument per line.
  • Line format: <identifier type>, <code>, <optional comment>

Example lines, first two are for a Cusip (CSP) and a Reuters Instrument Code (RIC):

CSP,31352JRV1
RIC,ALVG.DE
CHR,0#.FCHI,CAC40
CIN,G93882192,Vodaphone
COM,001179594,Carrefour
ISN,CH0012138530,CS
SED,B1YW440,3i Group
VAL,24476758,UBS
WPK,A11Q05,Elumeo

The sample input files contains lines with correct line format syntax. They also include lines that are comments, empty or badly structured, to test the error handling included in the code.

To avoid consuming too much data, we recommend you use small files for your own tests. For more information, refer to the previous section: Important warning on data usage.

 

A final parameter allows you to limit the number of lines of data that will be output to screen:

private static int maxDataLines = 1000000;  //To set a limit to the output data size

 

Initial file and directory management

After authenticating, we start with some basic input checks.

First we verify if the input file exists. If it does not, we display an error and exit the program:

if (!File.Exists(instrumentIdentifiersInputFile))
{
    DebugPrintAndWaitForEnter("FATAL: cannot access " + instrumentIdentifiersInputFile +
        "\nCheck if file and directory exist.");
    return;  //Exit main program
}

Then we verify if the output directory exists. If it does not, we display an error and exit the program:

if (!Directory.Exists(outputDirectory))
{
    DebugPrintAndWaitForEnter("FATAL: cannot access directory " + outputDirectory +
         "\nCheck if it exists.");
    return;  //Exit main program
}

Then we clear the output files of any content left over from a previous run, and create them if they do not exist. Here is the code for the first file, the others are done in a similar manner:

StreamWriter sw = new StreamWriter(dataOutputFile, false);
sw.Close();
Console.WriteLine("Cleared data output file " + dataOutputFile + "\n");

The second parameter of StreamWriter defines if content is to be appended to the file. By setting a value of false we clear the contents if the file already exists.

Next, we open the error output file and initialise it with a first message:

//Initialise the error output file:
StreamWriter swErr = new StreamWriter(errorOutputFile, true);
swErr.WriteLine("INFO: list of errors found in input file: " + instrumentIdentifiersInputFile + ":");

We also open the input file:

StreamReader sr = new StreamReader(instrumentIdentifiersInputFile);

 

Creating the instrument list by reading an input file

Instead of creating an array of defined length like in the previous tutorial, we shall create a list for the instrument identifiers.

The reason is that, at this stage, we do not know how many instrument identifiers we will have, because:

  • We don't know how many instruments are in the file.
  • We shall validate the file entries, and only keep the good ones.

It is not possible to define an array of undefined size, and it would be a waste to define a huge one and then resize it.

So we define a list of undefined size, which we shall fill from the file. Once all instrument identifiers have been added to the list, we convert it to an array, as that is what the On Demand extraction API call requires.

The first step is to create the empty instrument identifiers list:

List<InstrumentIdentifier> instrumentIdentifiersList = new List<InstrumentIdentifier> ();

Then we populate the list, by reading one line at a time, from the input file. First we initialize some variables:

int fileLineNumber = 0;
string fileLine = string.Empty;
string identifierTypeString = string.Empty;
string identifierCodeString = string.Empty;
IdentifierType identifierType;
int i = 0;

Then we read all lines until we get to the end of the file, parse the data, and add the validated data to the list:

bool endOfFile = false;
while (!endOfFile)
{
    //Read one line of the file, test if end of file:
    fileLine = sr.ReadLine();
    endOfFile = (fileLine == null);
    if (!endOfFile)
    {
        fileLineNumber++;
        try
        {
            //Parse the file line to extract the comma separated instrument type and code:
            string[] splitLine = fileLine.Split(new char[] { ',' });
            identifierTypeString = splitLine[0];
            identifierCodeString = splitLine[1];
            //Add only validated instrument identifiers into our list.
            if (identifierTypeString != string.Empty)
            {
                if (identifierCodeString != string.Empty)
                {
                    //DSS can handle many types, here we only handle a subset:
                    switch (identifierTypeString)
                    {
                        case "CHR": identifierType = IdentifierType.ChainRIC; break;
                        case "CIN": identifierType = IdentifierType.Cin; break;
                        case "COM": identifierType = IdentifierType.CommonCode; break;
                        case "CSP": identifierType = IdentifierType.Cusip; break;
                        case "ISN": identifierType = IdentifierType.Isin; break;
                        case "RIC": identifierType = IdentifierType.Ric; break;
                        case "SED": identifierType = IdentifierType.Sedol; break;
                        case "VAL": identifierType = IdentifierType.Valoren; break;
                        case "WPK": identifierType = IdentifierType.Wertpapier; break;
                        default:
                            identifierType = IdentifierType.NONE;
                            DebugPrintAndWriteToFile("ERROR: line " + fileLineNumber +
                                ": unknown identifier type: " + identifierTypeString, swErr);
                            break;
                    }
                    if (identifierType != IdentifierType.NONE)
                    {
                        instrumentIdentifiersList.Add(new InstrumentIdentifier
                        {
                            IdentifierType = identifierType,
                            Identifier = identifierCodeString
                        });
                        Console.WriteLine("Line " + fileLineNumber + ": " + identifierTypeString +
                            " " + identifierCodeString + " loaded into array [" + i + "]");
                        i++;
                    }
                }

Note: for more information on identifier types, refer to Tutorial 3: Creating an instrument list.

The code that follows is for error logging. Each error is displayed on screen and written to file, using a helper method:

                //Error handling messages grouped here:
                else
                {
                    DebugPrintAndWriteToFile("ERROR: line " + fileLineNumber +
                        ": missing identifier code in line: " + fileLine, swErr);
                }
            }
            else
            {
                DebugPrintAndWriteToFile("ERROR: line " + fileLineNumber +
                    ": missing identifier type in line: " + fileLine, swErr);
            }
        }
        catch
        {
            DebugPrintAndWriteToFile("ERROR: line " + fileLineNumber +
                ": bad line format: " + fileLine, swErr);
        }
    }
    else
    {
        if (fileLineNumber == 0)
            { DebugPrintAndWriteToFile("ERROR: empty file: " + instrumentIdentifiersInputFile, swErr); }
    }
}  //End of while loop

The helper method which displays errors and writes them to file is defined at the end of the main code:

static void DebugPrintAndWriteToFile(string messageToPrintAndWriteToFile, StreamWriter sw)
{
    Console.WriteLine(messageToPrintAndWriteToFile);
    sw.WriteLine(messageToPrintAndWriteToFile);
}

 

Extract of the output for successfully added instrument identifiers (using the largest sample input file):

Extract of the output for invalid lines in the largest input file:

The error file contains all the ERROR messages that appear on screen, in the same format.

 

Back to the main code, the next step is to close the input file and the error output file, and exit if we do not have at least one valid entry:

sr.Close();
int validIdentifiersCount = i;
if (validIdentifiersCount == 0)
{
    DebugPrintAndWriteToFile("\nFATAL: program exit due to no valid identifiers in the list.", swErr);
    DebugPrintAndWaitForEnter("");
    swErr.Close();
    return;  //Exit main program
}
swErr.Close();

Console.WriteLine("\n" + validIdentifiersCount +
    " valid instruments were loaded into an array, outside of DSS,\n" +
    "for use in the extraction.");

Result:

Now we convert our instrument identifiers list to an array, as that is what the On Demand extraction API call requires:

//Convert the instrument identifiers list to an array:
InstrumentIdentifier[] instrumentIdentifiers = instrumentIdentifiersList.ToArray();

 

Creating the field name array

To create the field name array we proceed just like in the previous tutorials, calling the helper method we created in Tutorial 3:

string[] requestedFieldNames = CreateRequestedFieldNames();

We will use this array when we define the extraction.

 

No report template or schedule creation

We do not create a report template, because the On Demand extraction calls implicitly define it.

We do not create a schedule, because an On Demand extraction is on the fly.

If you require recurring extractions, you can either use the calls we saw in the previous tutorials, or programmatically recurrently run an On Demand extraction.

 

Creating an On Demand extraction

First we define the On Demand request conditions.

In this example we want Intraday Summaries (bar) data, so we declare a new TickHistoryIntradaySummariesCondition. It defines the bar interval (in this case one hour), the historical date and time range for the data we want, and a few other parameters:

TickHistoryIntradaySummariesCondition condition = new TickHistoryIntradaySummariesCondition
{
    SummaryInterval = TickHistorySummaryInterval.OneHour,
    DaysAgo = null,
    MessageTimeStampIn = TickHistoryTimeOptions.GmtUtc,
    QueryStartDate = new DateTimeOffset(2016, 09, 29, 0, 0, 0, TimeSpan.FromHours(0)),
    QueryEndDate = new DateTimeOffset(2016, 09, 30, 0, 0, 0, TimeSpan.FromHours(0)),
    ReportDateRangeType = ReportDateRangeType.Range,
    ExtractBy = TickHistoryExtractByMode.Ric,
    SortBy = TickHistorySort.SingleByRic,
    Preview = PreviewMode.None,
    DisplaySourceRIC = true,
    TimebarPersistence = true
};

If we were requesting a different type of data (like Time and Sales, market depth, or other), a few of the available parameters to define in the condition could be slightly different. Go to the API Reference Tree and select the appropriate extraction request value to see more details.

Important note: inactive and historical instruments might be rejected, like a matured bond or an instrument that changed name. As explained in the Tutorials Introduction under heading Instrument Validation, contrary to scheduled requests, On Demand requests do not take user preferences set in the GUI into account. So, when required, we must specify the instrument validation behavior in the code. There are many available options. In this example we only set 3 of them, and override any other user preferences:

InstrumentValidationOptions validationOptions = new InstrumentValidationOptions
{
    AllowHistoricalInstruments = true,
    AllowInactiveInstruments = true,
    AllowOpenAccessInstruments = true
};
bool useUserPreferencesForValidationOptions = false;

The 2 parameters defined above are optional; if not set, simply remove them from the next call.

Next we define a new TickHistoryIntradaySummariesExtractionRequest, using as parameters the arrays we just defined for instrument identifiers, field names and conditions:

TickHistoryIntradaySummariesExtractionRequest extractionRequest =
    new TickHistoryIntradaySummariesExtractionRequest
{
    IdentifierList = InstrumentIdentifierList.Create(instrumentIdentifiers, validationOptions,
                                                     useUserPreferencesForValidationOptions),
    ContentFieldNames = requestedFieldNames,
    Condition = condition
};

The extraction request is specific to the type of data we want, just like the conditions.

For other types of data, the extraction request type would be different, the conditions might be slightly different, but the overall process is exactly the same. The Example Application (explained in the Quick Start) illustrates many more On Demand extractions.

 

Running an On Demand extraction

We first define an option for the request, to disable automatic decompression. This is because we want to download the compressed data, and, if required, decompress it using a dedicated package. We used SharpZipLib, other libraries to consider include zlib and DotNetLib. The result is more reliable and robust code that can handle large data sets:

//Do not automatically decompress data file content:
extractionsContext.Options.AutomaticDecompression = false;

To actually run the extraction we use the ExtractRaw call. It takes as parameter the extraction request we created. This API will only return when the extraction is complete, so there is no need for code to check for extraction completion.

RawExtractionResult extractionResult = extractionsContext.ExtractRaw(extractionRequest);

 

Retrieving the extracted data, writing it to file

Once the On Demand extraction has completed, the data and notes can be retrieved.

Let us start with the data. As data sets can be very large, the API implements a streaming mechanism to retrieve the data. Using streams means your program will not use huge amounts of memory, and data can be processed inline.

That said, on the fly processing is not recommended for large data sets. Instead, it is recommended to save the compressed data to file, and then to read, decompress and treat it from there.

The code sample illustrates several possibilities:

  • Saving the data directly as a compressed file, on hard disk.
  • Saving the data as a CSV file, for later usage (out of scope here).
  • Reading and decompressing the data from the compressed file that was saved to hard disk.
  • Reading and decompressing the data on the fly, as it is received from the server. This variant also saves the data as a second CSV file (identical to the one created by the other variant above).

It also shows how to download the data from AWS (the Amazon Web Service cloud), which is faster than the standard download.

On the topics of decompression and download, and AWS downloads, see the 2 advisories available in the documentation under the Alerts and Notices section.

 

To save the data in compressed format, the code is very simple; we just copy the result stream into a file.

If we want to download from AWS, we first set an additional request header, which we remove afterwards:

//Direct download from AWS ?
if (awsDownload) { extractionsContext.DefaultRequestHeaders.Add("x-direct-download", "true"); };

DssStreamResponse streamResponse = extractionsContext.GetReadStream(extractionResult);
using (FileStream fileStream = File.Create(dataOutputFile))
    streamResponse.Stream.CopyTo(fileStream);

//Reset header after direct download from AWS ?
if (awsDownload) { extractionsContext.DefaultRequestHeaders.Remove("x-direct-download"); };

Console.WriteLine("Saved the compressed data file to disk:\n" + gzipDataOutputFile);

 

Alternatively we can decompress the data on the fly, and save the result in a CSV file, for later usage:

//Direct download from AWS ?
if (awsDownload) { extractionsContext.DefaultRequestHeaders.Add("x-direct-download", "true"); };

streamResponse = extractionsContext.GetReadStream(extractionResult);
using (GZipInputStream gzip = new GZipInputStream(streamResponse.Stream))
    using (FileStream fileStream = File.Create(csvDataOutputFile))
        gzip.CopyTo(fileStream);

//Reset header after direct download from AWS ?
if (awsDownload) { extractionsContext.DefaultRequestHeaders.Remove("x-direct-download"); };

Console.WriteLine("Saved the uncompressed data file to disk:\n" + csvDataOutputFile);

 

After saving the data to hard disk in compressed format, we can read and decompress it for further treatment. This is the recommended way to proceed, especially for large data sets. Here we decompress the data, and display it line by line on screen. The first line is the header line, which is the list of fields. The remaining lines contain the data.

int lineCount = 0;
string line = "";
Console.WriteLine("Read compressed data file\n" + gzipDataOutputFile + "\nfrom disk, and decompress contents:");
Console.WriteLine("==================================== DATA =====================================");
swErr = new StreamWriter(errorOutputFile, true);
using (FileStream fileStream = File.OpenRead(gzipDataOutputFile))
{
    using (GZipInputStream gzip = new GZipInputStream(fileStream))
    {
        using (StreamReader reader = new StreamReader(gzip, Encoding.UTF8))
        {
            line = reader.ReadLine();
            if (string.IsNullOrEmpty(line))
            {
                Console.WriteLine("WARNING: no data returned. Check your request dates.");
                swErr.WriteLine("WARNING: no data returned. Check your request dates.");
            }
            else
                //The first line is the list of field names:
                Console.WriteLine(line);
            lineCount++;
            //The remaining lines are the data:
            //Variant 1 (no limit on screen output): write all lines to console
            //while ((line = reader.ReadLine()) != null)
            //{
            //    Console.WriteLine(line);
            //    lineCount++;
            //}
            //Variant 2: write lines individually to console (with a limit on number of lines we output)
            for (lineCount = 1; !reader.EndOfStream; lineCount++)
            {
                line = reader.ReadLine();
                if (lineCount < maxDataLines) { Console.WriteLine(line); };
                if (lineCount == maxDataLines)
                {
                    Console.WriteLine("===============================================================================");
                    Console.WriteLine("Stop screen output now, we reached the max number of lines we set for display:\n" + maxDataLines);
                    Console.WriteLine("Counting the remaining lines, please be patient ...");
                }
            }
            Console.WriteLine("Total line count for data read from disk: " + lineCount);
        }
    }
}
swErr.Close();

Inside the loop, the commented code variant 1 is shorter, but the second one is more appropriate for a demo.

 

The last variant is to decompress the data on the fly when receiving it from the server, without the intermediary step of saving it to disk. We immediately treat it, line by line. This method is not recommended for large data sets.

In this code we just display it on screen, and also save it to a CSV file:

lineCount = 0;
Console.WriteLine("Read data from DSS, decompress contents on the fly, save to csv and display:");
Console.WriteLine("==================================== DATA =====================================");
//Direct download from AWS ?
if (awsDownload) { extractionsContext.DefaultRequestHeaders.Add("x-direct-download", "true"); };

swErr = new StreamWriter(errorOutputFile, true);
sw = new StreamWriter(csvDataOutputFile2, false);

streamResponse = extractionsContext.GetReadStream(extractionResult);
{
    using (GZipInputStream gzip = new GZipInputStream(streamResponse.Stream))
    {
        using (StreamReader reader = new StreamReader(gzip, Encoding.UTF8))
        {
            line = reader.ReadLine();
            if (string.IsNullOrEmpty(line))
            {
                Console.WriteLine("WARNING: no data returned. Check your request dates.");
                swErr.WriteLine("WARNING: no data returned. Check your request dates.");
            }
            else
            {
                //The first line is the list of field names:
                sw.WriteLine(line);
                Console.WriteLine(line);
                lineCount++;
                //The remaining lines are the data:
                //Variant 1 (no limit on screen output): write all lines to console
                //while ((line = reader.ReadLine()) != null)
                //{
                //    Console.WriteLine(line);
                //    lineCount++;
                //}
                //Variant 2: write lines individually to console (with a limit on number of lines we output)
                for (lineCount = 1; !reader.EndOfStream; lineCount++)
                {
                    line = reader.ReadLine();
                    sw.WriteLine(line);
                    if (lineCount < maxDataLines) { Console.WriteLine(line); };
                    if (lineCount == maxDataLines)
                    {
                        Console.WriteLine("===============================================================================");
                        Console.WriteLine("Stop screen output now, we reached the max number of lines we set for display:\n" + maxDataLines);
                        Console.WriteLine("Counting the remaining lines, please be patient ...");
                    }
                }
                Console.WriteLine("Total line count for data read on the fly: " + lineCount);
            }
        }
    }
}
//Reset header after direct download from AWS ?
if (awsDownload) { extractionsContext.DefaultRequestHeaders.Remove("x-direct-download"); };
sw.Close();
swErr.Close();
Console.WriteLine("\nSaved the uncompressed data file to disk:\n" + csvDataOutputFile2);

 

Here is a small extract of the first lines of the output, with the list of fields and the beginning of the data:

The resulting data might not contain all instruments, even though they were not rejected when we added them to the instrument list. This simply depends on the availability of data for the requested period.

 

Retrieving and analyzing the extraction notes, writing them to file

Note: AWS download only applies to data files, which explains why the following code makes no provision for it.

It is recommended to examine the extraction notes. They deliver useful information on the extraction, and might contain errors or warnings.

The notes are contained in a collection of strings that can easily be retrieved from the extraction result. The collection usually contains only one element, in other words all the notes are in a single string that contains new line characters. To analyze the notes contents line by line, the string must be split. This is illustrated in the sample code, which parses the notes file for the successful processing message, as well as errors, warnings, and permission issues:

sw = new StreamWriter(noteOutputFile, true);
Console.WriteLine("==================================== NOTES ====================================");
Boolean success = false;
string errorMsgs = "";
string warningMsgs = "";
string permissionMsgs = "";
foreach (String notes in extractionResult.Notes)
{
    sw.WriteLine(notes);
    Console.WriteLine(notes);
    //The returned notes are in a single string. To analyse the contents line by line we split it:
    string[] notesLines =
        notes.Split(new string[] { "\n", "\r\n" }, StringSplitOptions.RemoveEmptyEntries);
    foreach (string notesLine in notesLines)
    {
        success = success || (notesLine.Contains("Processing completed successfully"));
        if (notesLine.Contains("ERROR")) { errorMsgs = errorMsgs + notesLine + "\n"; }
        if (notesLine.Contains("WARNING")) { warningMsgs = warningMsgs + notesLine + "\n"; }
        if (notesLine.Contains("row suppressed for lack of"))
        {
            permissionMsgs = permissionMsgs + notesLine + "\n";
        }
    }
}
sw.Close();

The results of the analysis are also output to file and display:

swErr = new StreamWriter(errorOutputFile, true); 
if (success) { Console.WriteLine("SUCCESS: processing completed successfully.\n"); }
else { errorMsgs = "ERROR: processing did not complete successfully !\n" + errorMsgs; }
sw = new StreamWriter(errorOutputFile, true);
if (errorMsgs != "")
{
    swErr.WriteLine("\nERROR messages in notes file:\n" + errorMsgs);
    Console.WriteLine("ERROR messages:\n" + errorMsgs);
}
if (warningMsgs != "")
{
    swErr.WriteLine("WARNING messages in notes file:\n" + warningMsgs);
    Console.WriteLine("WARNING messages:\n" + warningMsgs);
}
if (permissionMsgs != "")
{
    swErr.WriteLine("PERMISSION ISSUES messages in notes file:\n" + permissionMsgs);
    Console.WriteLine("PERMISSION ISSUES messages:\n" + permissionMsgs);
}
swErr.Close();

This analysis is just an example of what one could do to extract some of the meaningful messages from the notes.

Here is the first part of the output, with generic information about the extraction:

 

Note that in the above, the notes state that the input list has 199 items. That count is before Instrument expansion. The next part of the file delivers details of the expansions. Simple instruments expand to 1 RIC (i.e. they do not expand):

Instruments that are not valid for the requested range of dates will expand to 0 RICs (i.e. they are removed from the list):

Chains will expand to multiple RICs:

Several instrument types, like ISINs, Cusips and Sedols, will also expand to multiple RICs:

 

Data access is subject to permissioning. Depending on the entitlements of the account, some data might not be delivered. Applied permission restrictions are detailed per impacted instrument:

The amount of data that can be retrieved in a month is subject to a quota, which is also part of the account permissioning. Quota related messages are also delivered in the notes:

 

The last section of the notes gives a summary list of all the instruments.

This includes the status of each one, i.e. if it was active or not during the time range of the request.

The last element is the count of delivered data lines per instrument. Obviously, the data count is 0 for inactive instruments:

 

Our sample code analyzes all the notes, and delivers a summary. Here is the first part of that:

 

No cleaning up

As stated previously, cleanup on the DSS server is not required, as there is nothing to delete in this case.

 

Full code

The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.

 

Summary

List of the main steps in the code:

  1. Authenticate by creating an extraction context.
  2. Check input file and output directory existence, clear output files.
  3. Create an array of financial instrument identifiers, populate it from a file. Manage and log errors.
  4. Create an array of field names.
  5. Create an extraction, after setting instrument validation options (if required, for instance for historical instruments).
  6. Run the extraction, wait for it to complete.
  7. Retrieve the extracted data, display it and write it to file.
  8. Retrieve the extraction notes, display them, analyze them and write them to file.

We do not create a report template or schedule, and there is no need to cleanup.

 

Code run results

Build and run

Don’t forget to reference the DSS SDK and the SharpZipLib, and to set your user account in Program.cs !

Successful run

After running the program with the smallest instrument list, and pressing the Enter key when prompted, here is the result:











Intermediary results are discussed at length in the code explanations in the previous section of this tutorial.

Press Enter one final time to close the pop-up and end the program.

 

We find the output files here; their contents reflect what was displayed on screen:

 

Potential errors and solutions

If the user name and password were not set properly, an error will be returned. See Tutorial 1 for details.

 

Conclusions

This tutorial introduced the simplified high level calls for On Demand extraction requests.

It showed how instrument identifiers can be read from a CSV file, and how to log errors in the process.

It demonstrated how to set instrument validation options.

It also showed how extracted data could be written to file.

 

Tutorial Group: 
.Net SDK Tutorials