DataScope Select - REST API

Close Menu
Expand All Collapse All
Introductory Tutorials Tutorials Introduction Programming without SDK Tutorial REST API Tutorials REST API Tutorials Introduction REST API Tutorial 1: Connecting to the DSS server REST API Tutorial 2: On Demand End of Day Extraction REST API Tutorial 3: On Demand intraday extraction, embargo REST API Tutorial 4: On Demand price history extraction REST API Tutorial 5: On Demand corporate actions extraction REST API Tutorial 6: On Demand ownership data extraction REST API Tutorial 7: On Demand T&C extraction REST API Tutorial 8: On Demand composite extraction REST API Tutorial 9: On Demand extraction: instrument list REST API Tutorial 10: GUI control calls: immediate extract REST API Tutorial 11: Search by Instrument REST API Tutorial 12: Search for an Equity REST API Tutorial 13: Search for a Future or Option REST API Tutorial 14: On Demand price history extraction raw .Net SDK Tutorials .Net SDK Tutorial 1: Connecting to the DSS server .Net SDK Tutorial 2: GUI control calls: List, report, sched .Net SDK Tutorial 3: GUI control calls: Validate, extraction .Net SDK Tutorial 4: GUI control calls: Embargo, note files .Net SDK Tutorial 5: On Demand: EoD extraction .Net SDK Tutorial 6: On Demand: EoD extraction, file I/O .Net SDK Tutorial 7: On Demand: large instrument lists .Net SDK Tutorial 8: On Demand: Terms & Conditions .Net SDK Tutorial 9: On Demand: Composite extraction .Net SDK Tutorial 10: Search by Instrument .Net SDK Tutorial 11: Search for Equity .Net SDK Tutorial 12: Search for Future or Option

.Net SDK Tutorial 5: On Demand: EoD extraction

Last update July 2019
Environment Windows
Language C#
Compilers Microsoft Visual Studio 2012/2013
Prerequisites DSS login, internet access, having done the previous tutorials
Source code Download .Net SDK Tutorials Code

Tutorial purpose

This is the fifth tutorial in a series of .Net SDK tutorials. It is assumed that the reader has acquired the knowledge delivered in the previous tutorials before following this one.

In this tutorial we start using the simplified high level API calls for On Demand extractions. We:

  • Create an instrument list array, outside of the DSS server.
  • Create an array of data field names, outside of the DSS server.
  • Create and run an on demand EoD data extraction on the DSS server, wait for it to complete.
  • Retrieve and display the data in JSON format, and the extraction notes (including maintenance notes).

 

Table of contents

 

Getting ready

Opening the solution

The code installation was done in Tutorial 1.

Opening the solution is similar to what was done in the previous tutorials:

  • Navigate to the \DSS REST API\Tutorial 5\Learning folder.
  • Double click on the solution file rest_api_advanced_topics.sln to open it in Microsoft Visual Studio.

 

Referencing the DSS SDK

Before anything else, you must reference the DSS REST API .Net SDK in the Microsoft Visual Studio project.

Important: this must be done for every single tutorial, for both the learning and refactored versions.

This was explained in the tutorial 2; please refer to it for instructions.

 

Viewing the C# code

In Microsoft Visual Studio, in the Solution Explorer, double click on Program.cs and on DssClient.cs to display both file contents. Each file will be displayed in a separate tab.

 

Setting the user account

Before running the code, you must replace  YourUserId  with your DSS user name, and  YourPassword  with your DSS password, in these 2 lines of Program.cs:

        private static string dssUserName = "YourUserId";
        private static string dssUserPassword = "YourPassword";

Important reminder: this must be done for every single tutorial, for both the learning and refactored versions.

Failure to do so will result in an error at run time (see Tutorial 1 for more details).

 

Understanding the On Demand calls

The On Demand high level calls we use in this tutorial (and the following) depart from what was done in the previous tutorials.

As stated in Tutorial 2, in the note on DSS programming approaches, there are 2 approaches:

  • Create (and modify when required) an instrument list, report template and extraction schedule, all stored on the DSS server, and retrieve your data once the extraction has completed. This mimics what you can do with the DSS web GUI. This approach was used in tutorials 2 to 4.
  • Using On Demand advanced API calls, it is also possible to do on the fly data requests without storing an instrument list, report template and extraction schedule on the DSS server. It is a more direct and dynamic way of doing things. In this case the instrument list, report template and extraction schedule are ephemeral, and cannot be seen in the DSS web GUI. This approach is used in this tutorial, and the following ones.

The choice is yours, and will depend on your use case and personal preferences. See the tutorials introduction section on the .Net SDK tutorials content for some examples.

The second approach we explore now has two consequences:

  1. We cannot see any instrument list, report template or schedule using the DSS GUI.
  2. Cleanup on the DSS server is not required, as there is nothing to delete.

 

Understanding the code

We shall only describe what is new versus the previous tutorials.

DssClient.cs

This is the same as the refactored version of Tutorial 4, except for the leading comment.

As from here on we are changing the programming approach, most of these methods will not be used. Nevertheless we will not remove them, as the idea is to build a library of useful methods.

No additional explanations are required as the rest of the code was described in the previous tutorial.

 

Program.cs

At the top of the code we see the DSS using directives list is slightly different and shorter than for the previous tutorial, because we are now using different API calls.

Creating the instrument array

Instead of creating an instrument list in DSS, we create a local array of instrument identifiers, in memory. This array is not stored on the DSS server.

In the next tutorial we will use such an array to read a large instrument list from file. But here we start by simply creating an array of size 5, and populating it manually:

//Create an array for the instrument identifiers, of size 5:
InstrumentIdentifier[] instrumentIdentifiers = new InstrumentIdentifier[5];

//In this example we populate it manually:
instrumentIdentifiers[0] = new InstrumentIdentifier
{
    IdentifierType = IdentifierType.Ric,
    Identifier = "IBM.N",
    UserDefinedIdentifier = "EQUITYTEST"
};
instrumentIdentifiers[1] = new InstrumentIdentifier
{
    IdentifierType = IdentifierType.Cusip,
    Identifier = "438516AC0",
    UserDefinedIdentifier = "BONDTEST"
};
//The next 2 do not add the optional field, user defined identifier:
instrumentIdentifiers[2] = new InstrumentIdentifier
{
    IdentifierType = IdentifierType.Ric,
    Identifier = "0001.HK"                
};
instrumentIdentifiers[3] = new InstrumentIdentifier
{
    IdentifierType = IdentifierType.Ric,
    Identifier = "VOD.L"
};
//The next one is a non existing identifier code, it will not return any data:
instrumentIdentifiers[4] = new InstrumentIdentifier
{
    IdentifierType = IdentifierType.Ric,
    Identifier = "DUMMY.DUMMY"
};

We will use this array when we define the extraction.

 

Creating the field name array

To create the field name array we proceed just like in the previous tutorials, calling the helper method we created in Tutorial 2:

string[] requestedFieldNames = CreateRequestedFieldNames();

For this demo the helper method includes 4 fields:

static string[] CreateRequestedFieldNames()
{
    string[] requestedFieldNames = { "Instrument ID", "Security Description",
                                     "Universal Close Price Date", "Universal Close Price" };
    return requestedFieldNames;
}

As a reminder, an explanation on how to choose field names is available in Tutorial 2, at the end of the section on report templates, under the heading: How to choose a report template format, and find out what field names are available ?

We will use this array when we define the extraction.

 

No report template or schedule creation

We do not create a report template, because the On Demand extraction calls implicitly define it.

We do not create a schedule, because an On Demand extraction is on the fly.

If you require:

  • Recurring extractions: either use the calls we saw in the previous tutorials, or run your program recurrently.
  • Extractions triggered by data availability: use the calls we saw in Tutorial 2.

 

Creating an On Demand extraction

In this example we want EoD (End of Day) pricing, so we declare a new EndOfDayPricingExtractionRequest, using as parameters the instrument identifiers and requested field names arrays we just defined:

EndOfDayPricingExtractionRequest extractionEod = new EndOfDayPricingExtractionRequest
{
    IdentifierList = InstrumentIdentifierList.Create(instrumentIdentifiers),
    ContentFieldNames = requestedFieldNames
};

This is just one of many possibilities. The Example Application explained in the Quick Start illustrates many more On Demand extractions. The list at time of writing is:

  • IntradayPricingExtractionRequest
  • FixedIncomeAnalyticsExtractionRequest
  • TechnicalIndicatorsExtractionRequest
  • LegalEntityAuditExtractionRequest
  • CompositeExtractionRequest
  • MBSFactorHistoryExtractionRequest
  • TrancheFactorHistoryExtractionRequest
  • OwnershipExtractionRequest
  • TermsAndConditionsExtractionRequest
  • TimeSeriesExtractionRequest
  • BondScheduleExtractionRequest
  • RatingsExtractionRequest
  • CorporateActionsStandardExtractionRequest
  • CorporateActionsIpoExtractionRequest
  • CorporateActionsIsoExtractionRequest
  • LegalEntityDetailExtractionRequest
  • LegalEntityHierarchyExtractionRequest
  • EstimatesSummaryExtractionRequest
  • EstimatesActualExtractionRequest
  • EstimatesDetailExtractionRequest
  • EstimatesFootnotesCompanyExtractionRequest
  • EstimatesFootnotesDetailExtractionRequest
  • NewsAnalyticsExtractionRequest
  • NewsAnalyticsCommoditiesExtractionRequest
  • NewsItemsExtractionRequest
  • PremiumEndOfDayPricingExtractionRequest
  • PremiumPricingExtractionRequest
  • SingleHistoricalPriceExtractionRequest
  • FundAllocationExtractionRequest

Thanks to this large panel of declarations the report template becomes implicit, we do not have to create it manually.

 

Running an On Demand extraction

The Extract call takes as parameter the extraction we created. It is a blocking API call which only returns when the extraction is complete, so there is no need for code to check for extraction completion.

ExtractionResult extractionResult = extractionsContext.ExtractWithNotes(extractionEod);
DssCollection<ExtractionRow> extractionDataRows = extractionResult.Contents;

 

Retrieving the extracted data

The On Demand extraction returns extraction data rows, which simplifies the code, as we do not have to deal with file names and types like we did in the previous tutorials.

We can extract the field names from the first row of the returned data. This is not really required, because we defined them in the field name array, but for training purposes we include code to show how this is done. Here we simply store the field names in a string, separated with commas, and display the result, which reflects what we put in the field name array:

if (!extractionDataRows.Any())
    { DebugPrintAndWaitForEnter("No data returned ! Is it a banking holiday ?"); }
else
{
    //Extract the names of the data fields, from the first row of returned data:
    StringBuilder returnedFieldNames = new StringBuilder();
    ExtractionRow firstRow = extractionDataRows.First();
    foreach (string fieldName in firstRow.DynamicProperties.Keys)
    {
        returnedFieldNames.Append(fieldName);
        returnedFieldNames.Append(",");
    }
    DebugPrintAndWaitForEnter("Returned list of field names:\n" + returnedFieldNames.ToString());

We retrieve the data inside a foreach loop, by selecting the values from the key/value pairs contained in each extraction data row, one row at a time. Here we simply put the values in a string, separated with commas, and display the result:

    //Extract the actual data values, from all rows of returned data:
    Console.WriteLine("Returned field values:");
    Console.WriteLine("======================");
    string rowValuesInString = string.Empty;
    int validDataRows = 0;
    foreach (ExtractionRow row in extractionDataRows)
    {
        IEnumerable<object> rowValues = row.DynamicProperties.Select(dp => dp.Value);
        rowValuesInString = String.Join(", ", rowValues);
        if (rowValuesInString != string.Empty)
        {
            Console.WriteLine(rowValuesInString);
            validDataRows++;
        }
    }

We have 5 instrument identifiers, of which 1 is intentionally invalid; this results in 1 empty row that we do not display.

Returned data rows might be empty for other reasons, like in case of a banking holiday, or non volatile instrument that are not quoted on the day of the request.

We therefore include code to display the total number of data rows, and to count and display the valid data rows, i.e. those that contain data:

    //The number of data rows could be less than the number of instruments,
    //if data is not available for some instruments:
    DebugPrintAndWaitForEnter("\nExtraction completed." +
        "\nNumber of data rows: " + extractionDataRows.Count() +
        "\nNumber of valid (non empty) data rows: " + validDataRows);
}

Next we retrieve the extraction notes, which deliver useful information, warnings and errors:

//Display the extraction notes:
if (extractionResult.Notes.Any())
{
    Console.WriteLine("Extraction Notes:");
    Console.WriteLine("=================");
    foreach (String note in extractionResult.Notes)
        Console.WriteLine(note);
}
else Console.WriteLine("Error: no extraction notes returned");

 

Side note on embargoes

What would happen if an On Demand request was subject to an embargo ?

In this tutorial we make an End of Day request, so there is no embargo !

But it is possible to make an On Demand intraday request, using:

IntradayPricingExtractionRequest extractionEod = new IntradayPricingExtractionRequest
{
    IdentifierList = InstrumentIdentifierList.Create(instrumentIdentifiers),
    ContentFieldNames = requestedFieldNames
};

Requests for Intraday data can be embargoed, as we saw in Tutorial 4.

An On Demand extraction call is blocking. As a consequence, if the instrument list contains embargoed instruments, then the call will only return after all embargoes are lifted. If we take the same example as in Tutorial 4, i.e. an embargo on the German exchange Xetra (instrument identifier ALVG.DE), the call will only return after 15 minutes.

If you want to experiment with this by modifying the code of this tutorial, you must:

  • Change the declaration of the extraction, as illustrated in the code snippet above.
  • Adapt the instrument list, to use at least one that is subject to embargo, with your DSS account permissions.
  • Adapt the field list, because the Universal Close Price fields are not valid for an intraday request. You can use fields like Ask Price, Ask Size, etc. like we did in tutorial 4.

Concerning the DSS General preferences for this use case, it is recommended to disable partial files, or alternatively to enable partial files and delta deliveries:

 or 

Either one of these settings will deliver this result:

If your settings enable partial files but disable deltas capabilities, then the result will contain useless duplicate data lines, which will require additional handling:

 

No cleaning up

As stated previously, cleanup on the DSS server is not required, as there is nothing to delete in this case.

 

Full code

The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.

 

Summary

List of the main steps in the code:

  1. Authenticate by creating an extraction context.
  2. Create an array of financial instrument identifiers.
  3. Create an array of field names.
  4. Create an extraction.
  5. Run the extraction, wait for it to complete.
  6. Retrieve the extracted data and extraction notes.

We do not create a report template or schedule, and there is no need to cleanup.

As you can see, if your use case only requires On Demand extractions, your code will be shorter and easier to write.

 

Code run results

Build and run

Don’t forget to reference the DSS SDK, and to set your user account in Program.cs !

 

Successful run

After running the program, and pressing the Enter key when prompted, the final result should look like this:



 

Potential errors and solutions

If the user name and password were not set properly, an error will be returned. See Tutorial 1 for details.

 

Understanding the refactored version

Explanations

DSS client helper class file: DssClient.cs

Using statements were added, for the new API calls in the new method.

The extraction context is set to private, as it is not required to expose it to the main program:

private ExtractionsContext extractionsContext;

 

One new method was added, to create and run an On Demand EoD extraction:

public ExtractionResult CreateAndRunEodPricingExtraction(
    InstrumentIdentifier[] instrumentIdentifiers, string[] contentFieldNames)
{
    //Create an on demand extraction, of type EndOfDayPricingExtractionRequest:
    EndOfDayPricingExtractionRequest extractionEod = new EndOfDayPricingExtractionRequest
    {
        IdentifierList = InstrumentIdentifierList.Create(instrumentIdentifiers),
        ContentFieldNames = contentFieldNames
    };
    //Run the extraction.
    //This call is blocking, it returns when the extraction is completed:
    return extractionsContext.ExtractWithNotes(extractionEod);
}

It would be very easy to create additional methods for different extraction types, as variations of this one.

 

Main program file: Program.cs

One using statement was removed, as API calls were moved to DssClient.cs.

The statement that exposed the extraction context was removed.

 

To populate the array of instrument identifiers, we call a helper method:

InstrumentIdentifier[] instrumentIdentifiers = PopulateInstrumentIdentifiers();

As this helper method is specific to our workflow, we declare it in Program.cs instead of DssClient.cs, after the main code:

static InstrumentIdentifier[] PopulateInstrumentIdentifiers()
{
    //Create an array for the instrument identifiers, of size 5:
    InstrumentIdentifier[] instrumentIdentifiers = new InstrumentIdentifier[5];

    //In this example we populate it manually:
    instrumentIdentifiers[0] = new InstrumentIdentifier
    {
        IdentifierType = IdentifierType.Ric,
        Identifier = "IBM.N",
        UserDefinedIdentifier = "EQUITYTEST"
    };
    instrumentIdentifiers[1] = new InstrumentIdentifier
    {
        IdentifierType = IdentifierType.Cusip,
        Identifier = "438516AC0",
        UserDefinedIdentifier = "BONDTEST"
    };
    //The next 2 do not add the optional field, user defined identifier:
    instrumentIdentifiers[2] = new InstrumentIdentifier
    {
        IdentifierType = IdentifierType.Ric,
        Identifier = "0001.HK"
    };
    instrumentIdentifiers[3] = new InstrumentIdentifier
    {
        IdentifierType = IdentifierType.Ric,
        Identifier = "VOD.L"
    };
    //The next one is a non existing identifier code, it will not return any data:
    instrumentIdentifiers[4] = new InstrumentIdentifier
    {
        IdentifierType = IdentifierType.Ric,
        Identifier = "DUMMY.DUMMY"
    };
    return instrumentIdentifiers;
}

To create and run the On Demand EoD extraction, we call our new DSS client helper method:

ExtractionResult extractionResult =
    dssClient.CreateAndRunEodPricingExtraction(instrumentIdentifiers, requestedFieldNames);
DssCollection<ExtractionRow> extractionDataRows = extractionResult.Contents;

 

To display the extracted data field names, we call a helper method:

DisplayExtractedDataFieldNames(extractionDataRows);

We decided to create 2 separate methods, one for the field names, the other for the field values, because most use cases will not require the field names.

As this helper method is specific to our workflow, we declare it in Program.cs instead of DssClient.cs, after the main code:

static void DisplayExtractedDataFieldNames(IDssEnumerable<ExtractionRow> extractionDataRows)
{
    //Error handling:
    if (!extractionDataRows.Any())
    {
        DebugPrintAndWaitForEnter("No data returned ! Is it a banking holiday ?");
        return;
    }

    //Extract the names of the data fields, from the first row of returned data:
    StringBuilder returnedFieldNames = new StringBuilder();
    ExtractionRow firstRow = extractionDataRows.First();
    foreach (string fieldName in firstRow.DynamicProperties.Keys)
    {
        returnedFieldNames.Append(fieldName);
        returnedFieldNames.Append(",");
    }
    DebugPrintAndWaitForEnter("Returned list of field names:\n" + returnedFieldNames.ToString());
}

 

To display the extracted data values, we call another helper method:

DisplayExtractedDataFieldValues(extractionDataRows);

As this helper method is specific to our workflow, we declare it in Program.cs instead of DssClient.cs, after the main code:

static void DisplayExtractedDataFieldValues(DssCollection<ExtractionRow> extractionDataRows)
{
    //Error handling:
    if (!extractionDataRows.Any())
    {
        DebugPrintAndWaitForEnter("No data returned ! Is it a banking holiday ?");
        return;
    }

    //Extract the actual data values, from all rows of returned data:
    Console.WriteLine("Returned field values:");
    Console.WriteLine("======================");
    string rowValuesInString = string.Empty;
    int validDataRows = 0;
    foreach (ExtractionRow row in extractionDataRows)
    {
        IEnumerable<object> rowValues = row.DynamicProperties.Select(dp => dp.Value);
        rowValuesInString = String.Join(", ", rowValues);
        if (rowValuesInString != string.Empty)
        {
            Console.WriteLine(rowValuesInString);
            validDataRows++;
        }
    }

    //The number of data rows could be less than the number of instruments,
    //if data is not available for some instruments:
    DebugPrintAndWaitForEnter("\nExtraction completed." +
        "\nNumber of data rows: " + extractionDataRows.Count() +
        "\nNumber of valid (non empty) data rows: " + validDataRows);
}

 

To display the extraction notes, we call another helper method:

DisplayExtractionNotes(extractionResult);

As this helper method is specific to our workflow, we declare it in Program.cs instead of DssClient.cs, after the main code:

static void DisplayExtractionNotes(ExtractionResult extractionResult)
{
    //Error handling:
    if (!extractionResult.Notes.Any())
    {
        DebugPrintAndWaitForEnter("Error: no extraction notes returned");
        return;
    }

    Console.WriteLine("Extraction Notes:");
    Console.WriteLine("=================");
    foreach (String note in extractionResult.Notes)
        Console.WriteLine(note);
    DebugPrintAndWaitForEnter("");
}

 

Full code

The full code can be displayed by opening the appropriate solution file in Microsoft Visual Studio.

 

Build and run

Don’t forget to reference the DSS SDK, and to set your user account in Program.cs !

 

Conclusions

This tutorial introduced the simplified high level calls for On Demand extraction requests.

 

Now move on to the next tutorial, which builds on the present tutorial by adding file input and output:

  • Populate a large instrument list array from a file.
  • Write the received data to a file.

These added capabilities do not illustrate additional DSS REST API functionality, but serve to put it in the context of productized code.

As of tutorial 8 we will return to illustrating more API functionality and use cases.

Tutorial Group: 
.Net SDK Tutorials