I'm using the DDS REST API to perform a search for all available equity options. I'm expecting close too 200k rows in the response. The result is paginated and the issue I'm seeing is that the further I get along with fetching pages, the slower the response gets. e.g. fetching page 1 takes around 16 seconds, wheras fetching page 320 takes 53 seconds. The attached graph shows the slow down compared to ideal linear scaling.
At this rate it looks like I won't be able to fetch the result in a reasonable amount of time.
is there a better way of approaching a queary like this one?
