question

Upvotes
Accepted
GoGoGroundhog avatar image
23 ●3 ●3 ●7

Inconsistent number of data points returned for historical data request

I would like to raise your attention on an unexpected behaviour regarding the API's time series data request.

I can reproduce the problem with the DataApiUsageExampleTimeseriesData that can be downloaded here with slight code changes.

In TimeSeriesRequestExample.Launch() I set up a request like this:

request = timeSeries.SetupDataRequest("CFI2Z7") 
.WithView("TRDPRC_1")
.WithAllFields()
.WithInterval(CommonInterval.Intraday1Minute)
.WithNumberOfPoints(20000)
.OnDataReceived(chunk => { Console.WriteLine("Count={0}",chunk.Records.Count()); })
.CreateAndSend();

When requesting 20000 data points now (27th Oct 2017, 14:30 CEST), the number of records provided will be 14945.
Note that the number returned is less then the number requested. However, when requesting 30000 data points, the result set is much larger, 18586 data points.
Also note that when requesting 40000 data points you still get only 18586 records delivered, so I assume this is all data available for that symbol.
Why don't we get back the full amount of data back when requesting 20000 records even though it fully fits the requested number?(Also note that in all case I will get two chunks sent, with the second having 0 records and IsLast == true).
eikoneikon-com-apirequest
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

4 Answers

Upvotes
Accepted
Alex Putkov.1 avatar image
39.2k ●75 ●11 ●27

Well, DTEGn.DE is not as liquid as you think. If you look close you'll see gaps in the minute bars returned. E.g. today Nov 7 there were no trades on this stock between 16:31:47 and 16:33:11 exchange time, which results in the 16:33 bar missing. But you're right that what I mentioned is not the full story. There are other factors affecting how many rows are returned. Those factors include predictable things such as trading sessions. E.g. right now if I request 1,000 rows, which is less than the number of rows I would get for two full trading sessions if this stock traded every minute, I get back 994 rows going to the start of trading session yesterday. But if I request 1,100 rows, which is more than the number of rows I would get for two full trading sessions if the stock traded every minute, I get back full 1,100, of which 106 rows belong to the session 3 business days ago. There are also factors that cannot be predicted such as what's ready available in the cache in the Timeseries Proxy, which is an element of distributed delivery infrastructure. The exact same request submitted multiple times may return different number of rows. The bottom line though is that you can rely on this parameter to limit the number of rows returned to below the value you specify, but you cannot rely on it to return the exact number of rows. Unless the instrument is so liquid that it's guaranteed to have a trade in every 1 minute interval. Try your experiment with IBM.N.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
Alex Putkov.1 avatar image
39.2k ●75 ●11 ●27

WithNumberOfPoints parameter can be used to limit the number of rows in the response to below the specified threshold, but you cannot rely on this parameter for the request to return exactly the number of rows you specify unless the instrument is very liquid and is guaranteed to have at least one trade within every minute of every trading session. For less liquid instruments such as this future there are minutes when no trades occur, which you can think of as "empty rows" that still count towards the threshold specified in WithNumberOfPoints parameter. The other limit you need to be aware of is the depth of price history stored on the backend, which is provided here. As you can see for 1 minute interval the depth is 90 days, and this is the limit you hit when you increased the value of WithNumberOfPoints parameter to 30K. As of 8pm New York time today Oct 31st 2017 price history in 1 minute aggregation goes back to Aug 1st 2017 and this entire timeseries is 18,652 rows.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
GoGoGroundhog avatar image
23 ●3 ●3 ●7

Hi Alex,

thanks a lot for your response.

In my opinion though, this cannot explain how the API behaves in this regard.
For example, requesting a very liquid equity, DTEGn.DE 1 Minute data successively increasing WithNumberOfPoints by 1000 for each request yields the following:

DTEGn.DE Intraday1Minute 1Min Requested: 1000 Received: 1000
DTEGn.DE Intraday1Minute 1Min Requested: 2000 Received: 1986
DTEGn.DE Intraday1Minute 1Min Requested: 3000 Received: 3000
DTEGn.DE Intraday1Minute 1Min Requested: 4000 Received: 4000
DTEGn.DE Intraday1Minute 1Min Requested: 5000 Received: 5000
DTEGn.DE Intraday1Minute 1Min Requested: 6000 Received: 6000
DTEGn.DE Intraday1Minute 1Min Requested: 7000 Received: 6665
DTEGn.DE Intraday1Minute 1Min Requested: 8000 Received: 8000
DTEGn.DE Intraday1Minute 1Min Requested: 9000 Received: 8929
DTEGn.DE Intraday1Minute 1Min Requested: 10000 Received: 10000

Note that you do only get 8929 data points back when requesting 9000 points, but you do get the full 10000 when requesting 10000 points.
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
GoGoGroundhog avatar image
23 ●3 ●3 ●7

Thanks for the additional explanation. I guess if one's app needs to provide a certain amount of data points, it boils down to making multiple requests, stitching results together until you reach your required number of data points (or no additional data is delivered by the API).

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Click below to post an Idea Post Idea