DataScope Select - REST API

Close Menu
Expand All Collapse All
Introductory Tutorials Tutorials Introduction Programming without SDK Tutorial REST API Tutorials REST API Tutorials Introduction REST API Tutorial 1: Connecting to the DSS server REST API Tutorial 2: On Demand End of Day Extraction REST API Tutorial 3: On Demand intraday extraction, embargo REST API Tutorial 4: On Demand price history extraction REST API Tutorial 5: On Demand corporate actions extraction REST API Tutorial 6: On Demand ownership data extraction REST API Tutorial 7: On Demand T&C extraction REST API Tutorial 8: On Demand composite extraction REST API Tutorial 9: On Demand extraction: instrument list REST API Tutorial 10: GUI control calls: immediate extract REST API Tutorial 11: Search by Instrument REST API Tutorial 12: Search for an Equity REST API Tutorial 13: Search for a Future or Option REST API Tutorial 14: On Demand price history extraction raw .Net SDK Tutorials .Net SDK Tutorial 1: Connecting to the DSS server .Net SDK Tutorial 2: GUI control calls: List, report, sched .Net SDK Tutorial 3: GUI control calls: Validate, extraction .Net SDK Tutorial 4: GUI control calls: Embargo, note files .Net SDK Tutorial 5: On Demand: EoD extraction .Net SDK Tutorial 6: On Demand: EoD extraction, file I/O .Net SDK Tutorial 7: On Demand: large instrument lists .Net SDK Tutorial 8: On Demand: Terms & Conditions .Net SDK Tutorial 9: On Demand: Composite extraction .Net SDK Tutorial 10: Search by Instrument .Net SDK Tutorial 11: Search for Equity .Net SDK Tutorial 12: Search for Future or Option

.Net SDK Tutorial 7: On Demand: large instrument lists

Last update November 2019
Environment Windows
Language C#
Compilers Microsoft Visual Studio 2012/2013
Prerequisites DSS login, internet access, having done the previous tutorials
Source code Download .Net SDK Tutorials Code

Tutorial purpose

This is the seventh tutorial in a series of .Net SDK tutorials. It is assumed that the reader has acquired the knowledge delivered in the previous tutorials before following this one.

In this tutorial we build on the previous two by overcoming the limit on the number of instrument identifiers:

 

  • Manage an instrument list larger than the extraction limit, using loops.
  • Like the previous one, this sample also contains 2 extraction requests, one with the ExtractWithNotes endpoint, the other with the ExtractRaw endpoint, to illustrate the differences between the two.

Extraction limits vary depending on the type of requested data, from 500 to 150000 instruments. This code example is for an End of Day request where the limit is 75000 instruments (150000 after chain expansion). Refer to the DataScope User Guide for all the details on the limits that apply by extraction type, or see them online.

In the tutorial we do not actually use such a large list of instruments; instead we simulate a lower limit, to avoid consuming too much (billable) data. For more information, refer to section: Important warning on data usage.

 

Table of contents

 

Getting ready

Opening the solution

The code installation was done in Tutorial 1.

Opening the solution is similar to what was done in the previous tutorials:

  • Navigate to the \DSS REST API\Tutorial 7\Learning folder.
  • Double click on the solution file rest_api_bulk_looping.sln to open it in Microsoft Visual Studio.

 

Referencing the DSS SDK

Before anything else, you must reference the DSS REST API .Net SDK in the Microsoft Visual Studio project.

Important: this must be done for every single tutorial, for both the learning and refactored versions.

This was explained in the tutorial 2; please refer to it for instructions.

 

Viewing the C# code

In Microsoft Visual Studio, in the Solution Explorer, double click on Program.cs and on DssClient.cs to display both file contents. Each file will be displayed in a separate tab.

 

Setting the user account

Before running the code, you must replace  YourUserId  with your DSS user name, and  YourPassword  with your DSS password, in these 2 lines of Program.cs:

        private static string dssUserName = "YourUserId";
        private static string dssUserPassword = "YourPassword";

Important reminder: this must be done for every single tutorial, for both the learning and refactored versions.

Failure to do so will result in an error at run time (see Tutorial 1 for more details).

 

Important warning on data usage

DSS data request quota

DSS accounts have a quota of instruments that can be requested during a defined time period. The DSS subscription fees are proportional to the size of the quota. If the quota is reached, excess fees will be due. That situation must be avoided !

Data usage can be displayed in the DSS web GUI:

Contact your local account manager or sales specialist if you have queries on your DSS account quota or usage.

 

Instrument list files delivered with the tutorials

This tutorial (and the previous one) handle large instrument lists, read from a file.

Care must be exercised when running these tutorials, to avoid over consuming data.

For this reason, several files are delivered with the tutorials code:

  • DSS_API_1500_input_file.csv
  • DSS_API_60_input_file.csv
  • DSS_API_10_input_file.csv

The first one contains 1500 valid instrument identifiers, the second one 60, the last one only contains 10.

The text and illustrations that follow use the larger file. To avoid consuming too much data, we recommend you use the smaller file for your own tests, which will deliver similar results.

 

Understanding the code

We shall only describe what is new versus the previous tutorials.

DssClient.cs

This is the same as the refactored version of Tutorial 6, except for the leading comment.

No additional explanations are required as the rest of the code was described in the previous tutorial.

 

Program.cs

Member declarations

We add one declaration, the maximum size of an instrument identifier list DSS can handle in one request:

private static int maxDssIdentifiersListSize = 75000;

Extraction limits vary depending on the type of requested data, from 500 to 150000 instruments. This code example is for an End of Day request where the limit is 75000 instruments (150000 after chain expansion). Refer to the DataScope User Guide for all the details on the limits that apply by extraction type.

 

As stated in the previous tutorial, to avoid consuming too much data for tests, we recommend you use a small input file:

private static string instrumentIdentifiersInputFile = inputDirectory + "DSS_API_10_input_file.csv";

For an explanation why, refer to the previous section: Important warning on data usage.

The purpose of this tutorial is to manage very large instrument lists. To test the algorithm without consuming huge amounts of data, simulate a lower limit by setting a lower value for maxDssIdentifiersListSize, smaller than the number of instruments in the input file, like:

private static int maxDssIdentifiersListSize = 7;

For production code set the value back to 75000 !

 

Creating the instrument list by reading an input file

Like in the previous tutorial’s learning version we first create a list for the instrument identifiers, and populate it from the input file. This list will contain all the instrument identifiers, even if there are more than the 75000 DSS can handle in one request:

List<InstrumentIdentifier> instrumentIdentifiersList =
    PopulateInstrumentIdentifiersListFromFile(instrumentIdentifiersInputFile, errorOutputFile);

Later in the code we will extract slices of 75000 instrument identifiers from this full list. As in C# it is easier to create sub-lists from a large list than sub-arrays from a large array, we don’t want to convert the list to an array at this stage. That is why our helper method returns a list, contrary to the helper method in the previous tutorial’s refactored version, which returned an array. Apart from this small difference, the methods are the same:

static List<InstrumentIdentifier> PopulateInstrumentIdentifiersListFromFile(
    string instrumentIdentifiersInputFile, string errorOutputFile)
{
    //Open the input file:
    StreamReader sr = new StreamReader(instrumentIdentifiersInputFile);

    //Initialise the error output file:
    StreamWriter sw = new StreamWriter(errorOutputFile, true);
    sw.WriteLine("List of errors found in input file: " + instrumentIdentifiersInputFile + "\n");

    //Instead of an array (of defined length), create a list for the instrument identifiers.
    //We do this because:
    //  we don't know how many instruments are in the file,
    //  we filter the file entries, to only keep the validated ones.
    //Create the empty instrument identifiers list:
    List<InstrumentIdentifier> instrumentIdentifiersList = new List<InstrumentIdentifier> ();

    //Populate the list, reading one line at a time from the file:
    int fileLineNumber = 0;
    string fileLine = string.Empty;
    bool commaExistsInFileLine = false;
    string identifierTypeString = string.Empty;
    string identifierCodeString = string.Empty;
    IdentifierType identifierType;
    int i = 0;

    //Loop through all lines until we get to the end of the file:
    bool endOfFile = false;
    while (!endOfFile)
    {
        Errors errorCode = Errors.NoError;

        //Read one line of the file, test if end of file:
        fileLine = sr.ReadLine();
        endOfFile = (fileLine == null);
        if (endOfFile && fileLineNumber == 0) { errorCode = Errors.EmptyFile; };
        fileLineNumber++;

        //Parse the file line to extract the comma separated instrument type and code:
        if (errorCode == Errors.NoError && !endOfFile)
        {
            commaExistsInFileLine = (fileLine.IndexOf(",") >= 0);
            if (commaExistsInFileLine)
            {
                string[] splitLine = fileLine.Split(new char[] { ',' });
                identifierTypeString = splitLine[0];
                identifierCodeString = splitLine[1];
            }
            else
            {
                errorCode = Errors.BadLineFormat;  //Missing comma
                identifierTypeString = string.Empty;
                identifierCodeString = string.Empty;
            }
        }
        if (identifierTypeString == string.Empty && errorCode == Errors.NoError)
            { errorCode = Errors.EmptyType; }
        if (identifierCodeString == string.Empty && errorCode == Errors.NoError)
            { errorCode = Errors.EmptyCode; }

        identifierType = IdentifierType.NONE;
        if (errorCode == Errors.NoError && !endOfFile)
        {
            //DSS can handle many types, here we only handle a subset:
            switch (identifierTypeString)
            {
                case "CHR": identifierType = IdentifierType.ChainRIC; break;
                case "CIN": identifierType = IdentifierType.Cin; break;
                case "COM": identifierType = IdentifierType.CommonCode; break;
                case "CSP": identifierType = IdentifierType.Cusip; break;
                case "ISN": identifierType = IdentifierType.Isin; break;
                case "RIC": identifierType = IdentifierType.Ric; break;
                case "SED": identifierType = IdentifierType.Sedol; break;
                case "VAL": identifierType = IdentifierType.Valoren; break;
                case "WPK": identifierType = IdentifierType.Wertpapier; break;
                default: errorCode = Errors.UnknownIdentifier; break;
            }
        }

        if (errorCode == Errors.NoError && !endOfFile)
        {
            //Add validated instrument identifier into our list:
            instrumentIdentifiersList.Add(new InstrumentIdentifier
            {
                IdentifierType = identifierType,
                Identifier = identifierCodeString
            });
            Console.WriteLine("Line " + fileLineNumber + ": " +
                identifierTypeString + " " + identifierCodeString + " loaded into array [" + i + "]");
            i++;
        }

        if (errorCode != Errors.NoError)
        {
            DebugPrintAndWriteToFileErrorMessage(
                errorCode, fileLineNumber, fileLine, identifierTypeString, sw);
        }
    }  //End of while loop

    sr.Close();
    sw.Close();

    return instrumentIdentifiersList;
}

The outputs of this method, for successfully added instrument identifiers, are exactly the same as for the previous tutorial:

The outputs of this method, for invalid lines in the input file, are exactly the same as for the previous tutorial:

Same goes for the error file which contains all the ERROR messages.

 

Similarly to the previous tutorial, this is followed with a check on the number of valid instrument identifiers:

int validIdentifiersCount = instrumentIdentifiersList.Count();
if (validIdentifiersCount == 0)
{
    DebugPrintAndWaitForEnter("Exit program due to no identifiers in the list.");
    return;  //Exit main program
}
Console.WriteLine("\n" + validIdentifiersCount +
    " valid instruments were loaded into an array, outside of DSS,\n" +
    "for use in the extraction.");

Result:

 

Creating the field name array

To create the field name array we proceed just like in the previous tutorials, calling the helper method we created in Tutorial 2:

string[] requestedFieldNames = CreateRequestedFieldNames();

We will use this array when we define the extraction.

 

No report template or schedule creation

Like in the previous tutorial, we do not create a:

  • Report template, because the On Demand extraction calls implicitly define it.
  • Schedule, because an On Demand extraction is on the fly.

 

Creating and running the On Demand extractions, retrieving the data

This section explains how we code a loop to process all the instruments in several chunks.

We create a big loop, inside which we run one or more On Demand extractions, each immediately followed by whatever data processing or treatment the use case requires. DSS has a limit on the number of instruments in a list. That limit is 75000, we set it in variable maxDssIdentifiersListSize. If the file contains more, we split the list into:

  • 1 or more chunks of 75000.
  • An eventual remainder.

We run an extraction and process the data for each sub-list, inside a big loop. At each iteration of the loop we:

  • Create an instrument identifier sub-list (a subset of the main one).
  • Create an on demand EOD extraction for that instrument identifier sub-list.
  • Retrieve the data and extraction notes, treat them and save them to file. This is done using 2 code variants, one for JSON formatted data, the other for compressed CSV formatted data.

 

Let us look at the code ...

Determine the number of loops of 75000 instruments:

int loopsToRun = validIdentifiersCount / maxDssIdentifiersListSize;

Determine the number of instruments in the remainder loop:

int sizeOfInstrumentIdentifiersRemainder = validIdentifiersCount % maxDssIdentifiersListSize;

Add a loop for the eventual remainder:

if (sizeOfInstrumentIdentifiersRemainder > 0) { loopsToRun++; }

For testing purposes, display the calculated numbers:

Console.WriteLine("Next we will launch " + loopsToRun +
    " on demand EOD pricing extraction(s),\n" +
    "each followed by data treatment.\n" +
    "The " + loopsToRun + " iteration(s) take into account the " +
    maxDssIdentifiersListSize + " instruments limit.");
if (sizeOfInstrumentIdentifiersRemainder > 0)
{
    DebugPrintAndWaitForEnter("The last iteration has " + sizeOfInstrumentIdentifiersRemainder +
        " instruments.\n");
}
else
{
    DebugPrintAndWaitForEnter("The last iteration has " + maxDssIdentifiersListSize + " instruments.\n");
}

As our sample instrument lists have less than 75000 instruments, there will only be 1 iteration of the loop.

To test this part of the code without using an instrument identifier list of more than 75000 instruments, we can change the value of maxDssIdentifiersListSize (do not do this in production).

Setting it to 1000, with an input file of 1500 instruments, we get:

Setting it to 7, with an input file of 10 instruments, we get:

 

Before the loop, declare some helper variables:

IDssEnumerable<ExtractionRow> extractionDataRows;
int bigLoopIteration = 1;
int firstListElementIndex = 0;
int numberOfListElements = 0;

To extract slices of 75000 instrument identifiers from our full list we use 2 variables:

  • firstListElementIndex:         index (in the full list) of the first instrument identifier of the current slice (starts at 0).
  • numberOfListElements:       number of instrument identifiers in the current slice (75000 or the remainder).

 

At the start of the loop, determine the values of these 2 variables for the current iteration:

while (bigLoopIteration <= loopsToRun)
{
    //Fill the instrument identifiers subset array with a set of instruments,
    //either 75000, or less if it is the remainder:
    firstListElementIndex = (bigLoopIteration - 1) * maxDssIdentifiersListSize;
                
    if (bigLoopIteration < loopsToRun || sizeOfInstrumentIdentifiersRemainder == 0)
        { numberOfListElements = maxDssIdentifiersListSize; }
    else
        { numberOfListElements = sizeOfInstrumentIdentifiersRemainder; }

Extract the sub-list of instrument identifiers from the full list (this is much easier using lists than arrays):

    List<InstrumentIdentifier> instrumentIdentifiersSubList =
        instrumentIdentifiersList.GetRange(firstListElementIndex, numberOfListElements);

Convert this list to an array, as that is what the On Demand extraction API call requires:

    InstrumentIdentifier[] instrumentIdentifiersSubArray = instrumentIdentifiersSubList.ToArray();

 

For testing purposes, display the relevant numbers:

    Console.WriteLine("Loaded identifiers into a sub array, iteration number: " + bigLoopIteration +
        "\nFirst element index: " + firstListElementIndex + " - Number of elements: " +
        numberOfListElements);

Going back to the test with a value of 1000 for maxDssIdentifiersListSize, we get at first iteration:

And at second iteration:

 

Extract and treat JSON data

Like in the previous tutorial, we shall run 2 extraction variants:

  • The first one uses the ExtractWithNotes endpoint, which delivers the data in JSON format.
  • The second one uses the ExtractRaw endpoint, which delivers compressed CSV formatted data.

In real life you would only run one of these, chosen depending on your use case.

 

To run the extraction we call our helper method, like in the previous tutorial:

    ExtractionResult extractionResult =
        dssClient.CreateAndRunEodPricingExtraction(instrumentIdentifiersSubArray, requestedFieldNames);
    extractionDataRows = extractionResult.Contents;

Finally we process the data from this extraction, using the helper method from the previous tutorial:

    if (bigLoopIteration == 1)
        { DisplayAndLogExtractedDataFieldNames(dataOutputFile, extractionDataRows); }
    DisplayAndLogExtractedDataFieldValues(dataOutputFile, extractionDataRows);

Note we only display the field names at the first iteration, as they will always be the same.

The results on screen are the same as in the previous tutorial (except that they may come in one or several iterations, depending on the instrument identifier list size, and the value of maxDssIdentifiersListSize).

 

Extract and treat compressed data

To run the extraction for compressed CSV data, we call our helper method, like in the previous tutorial:

    RawExtractionResult rawExtractionResult =
        dssClient.CreateAndRunEodPricingRawExtraction(instrumentIdentifiersSubArray, requestedFieldNames);

Then we process the data from this extraction, using the helper methods from the previous tutorial:

    dssClient.SaveCompressedData(rawExtractionResult, gzipDataOutputFile);
    DisplayAndLogAndAnalyzeRawExtractionNotes(gzipNotesOutputFile, errorOutputFile, rawExtractionResult);

 

Post extraction code

After the extractions, the loop ends:

  //Increment the big while loop to process the next set of identifiers:
  bigLoopIteration++;
}

A final message signals that all extractions were completed:

DebugPrintAndWaitForEnter("All extractions completed.");

 

No cleaning up

As stated in the 2 previous tutorials, cleanup on the DSS server is not required, as there is nothing to delete in this case.

 

Full code

The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.

 

Summary

List of the main steps in the code:

  1. Authenticate by creating an extraction context.
  2. Check input file and output directory existence, clear output files.
  3. Create an array of financial instrument identifiers, populate it from a file. Manage and log errors.
  4. Create an array of field names.
  5. Inside a big loop:
    1. Create a sub-list of financial instrument identifiers, convert it to an array.
    2. Create an extraction.
    3. Run the extraction, wait for it to complete. Retrieve the extracted data and extraction notes, display them and write them to files.
    4. Variant of step 5.3, for compressed data.

We do not create a report template or schedule, and there is no need to cleanup.

 

Code run results

Build and run

Don’t forget to reference the DSS SDK, and to set your user account in Program.cs !

Successful run

The following is a run using the largest input file, with maxDssIdentifiersListSize set to 1000. Remember this is only for testing, in production maxDssIdentifiersListSize would be set to its real value of 75000. Remember: 75000 is the limit for End of Day requests. Refer to the DataScope User Guide for all the details on the limits that apply by extraction type.

After running the program, and pressing the Enter key when prompted, the final result should look like this:

This goes on with the next instruments of the file ...

The instrument identifiers import messages end like this, with quite a few errors generated by the wrongly formatted lines we included in the input file to test the error handling:

This is followed by the first iteration of the big loop, and the display of the returned field names:

Next come the data values for this first set of 1000 instruments:

This goes on with all the returned data for the first iteration, followed by the statistics and messages for this first slice:

After that, the extraction notes for this extraction are displayed, followed by their analysis results.

The second extraction is then performed, and the compressed data saved to file. After that, the extraction notes for the second extraction are displayed, followed by their analysis results.

This is followed by the next iteration of the big loop, and the display of the data values (field names are not displayed):

This goes on with all the returned data for that iteration, followed by the statistics, messages and notes for that slice.

The program ends after the last slice.

We can then check out the output files; their contents reflect what was displayed on screen, and is the same as what was generated by the previous tutorial (except that data values will have changed if you ran it on a different day).

 

Potential errors and solutions

If the user name and password were not set properly, an error will be returned. See Tutorial 1 for details.

 

Understanding the refactored version

Explanations

DSS client helper class file: DssClient.cs

The maxDssIdentifiersListSize declaration has been moved here (from Program.cs), as it is specific to DSS, and not likely to change:

public int maxDssIdentifiersListSize = 75000;

There are no other changes.

 

Main program file: Program.cs

As the maxDssIdentifiersListSize declaration has been moved to the DssClient.cs class, we expose it here:

int maxDssIdentifiersListSize = dssClient.maxDssIdentifiersListSize;

Comments that explicitly mentioned the number 75000 (for readability) were modified to mention the variable name maxDssIdentifiersListSize.

 

Inside the big loop, we call a helper method to populate the instrument identifiers sub-array:

InstrumentIdentifier[] instrumentIdentifiersSubArray =
    PopulateInstrumentIdentifiersSubArray(
        bigLoopIteration, loopsToRun,
        maxDssIdentifiersListSize, sizeOfInstrumentIdentifiersRemainder,
        instrumentIdentifiersList);

This helper method is declared in Program.cs, after the main code:

static  InstrumentIdentifier[] PopulateInstrumentIdentifiersSubArray(
    int bigLoopIteration, int loopsToRun,
    int maxDssIdentifiersListSize, int sizeOfInstrumentIdentifiersRemainder,
    List<InstrumentIdentifier> instrumentIdentifiersList)
{
    //Fill the instrument identifiers subset array with a set of instruments,
    //either maxDssIdentifiersListSize, or less if it is the remainder:
    int firstListElementIndex = (bigLoopIteration - 1) * maxDssIdentifiersListSize;
    int numberOfListElements = 0;

    if (bigLoopIteration < loopsToRun || sizeOfInstrumentIdentifiersRemainder == 0)
        { numberOfListElements = maxDssIdentifiersListSize; }
    else
        { numberOfListElements = sizeOfInstrumentIdentifiersRemainder; }

    List<InstrumentIdentifier> instrumentIdentifiersSubList =
        instrumentIdentifiersList.GetRange(firstListElementIndex, numberOfListElements);

    Console.WriteLine("Loaded identifiers into a sub array, iteration number: " + bigLoopIteration +
        "\nFirst element index: " + firstListElementIndex +
        " - Number of elements: " + numberOfListElements);

    //Convert the sublist to an array, as that is what the API needs.
    InstrumentIdentifier[] instrumentIdentifiersSubArray = instrumentIdentifiersSubList.ToArray();
    return instrumentIdentifiersSubArray;
}

Note also that this helper method includes the list conversion to an array.

 

Full code

The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.

 

Build and run

Don’t forget to reference the DSS SDK, and to set your user account in Program.cs !

 

Conclusions

This tutorial showed how very large instrument identifiers lists can be easily handled inside a loop, to manage the maximum extraction size supported by the DSS servers.

Extraction limits vary depending on the type of requested data, from 500 to 150000 instruments. This code example is for an End of Day request where the limit is 75000 instruments (150000 after chain expansion). Refer to the DataScope User Guide for all the details on the limits that apply by extraction type.

For large instrument lists, using an endpoint that delivers compressed data, like was illustrated in the second extraction in this sample, can be an advantage.

Tutorial Group: 
.Net SDK Tutorials