Let's say I want to get around 500 fields for around 4000 instruments (stocks) for about 20 year (SDate = 2020-01-01). Given the multiple limits I cannot just use

get_date(instrument, fields, {'SDate': '2000-01-01', 'EDate': today_as_string}) 

as this will give all kinds of errors / limits breaching.

Currently I will chunk the instruments 50 at a time, select 1 field and query for the full date range.

dfs = []
for field in fields:
    for instruments_chunk in split_list(instruments, max_items_in_list=50): 
        df = ek.get_data(instruments_chunk, [field + 'date', field], {'SDate': '2000-01-01', 'EDate': today_as_string})
        df.rename(columns={df.columns[1]: 'date'}, inplace=True)
        df['date'] = pd.to_datetime(df['date']).dt.date.astype('datetime64')
        dfs.append(df)
final_df = pd.concat(dfs)

Although this works it is very slow and will take several days to complete.

Is there a better solution to this kind of problem?