GATEWAY TIMEDOUT ERROR WHILE COLLECTING HISTORICAL DATA

sahilsingh

from datetime import timedelta,date
import time
a = date(2015,1,1)
s1 = []
for i in range(len(w)):
 
 if(i==1):
        break
 while(a<=date(2017,8,31)):
     s = pd.DataFrame(kite.historical(w.instrument_token[i],str(a),str(a+timedelta(29)),'5minute'))
     a = a+timedelta(30)
     s1.append(s)
     time.sleep(7) 
 temp = pd.concat(s1)
 j = str(w.tradingsymbol[i])+'test1.csv'
 temp.to_csv(j)

This was the following test code i wrote to snoop data of the first instrument in the instrument_token file.I also added time.sleep(7) to give an interval of 7 second between each iteration or each call(if many requests are sent within a short period of time then also the error would come) thinking i would get rid of the gateway timedout error,but no even then it is showing the error after sometime.However it works fine when i do it for a period of just 1 year.So please tell me what rules do you have regarding how many queries can be sent in a given amount of time so i can accordingly design my code.

sujith

Hi @sahilsingh,
I think the request is timing out before data is being fetched from the database. Can you try with less time interval?

sahilsingh

I tried with 2 seconds and 3.Still the same result.

sahilsingh

The funny thing is that in the window of 2015 to 2016 end its working fine.But with 2017 I get the error after 10-12 seconds that's just after extracting 5 months of data.Its like it 'Knows' I wanna extract 2.5 years data.
UPDATE:
Seems to be working with an interval of 4 seconds atleast for now!!!

sahilsingh

Now i am getting this new error even though previously it was working
InputException: No candles found based on token and time and candleType.

sujith

Hi @sahilsingh,
Can you give us params to reproduce this? We will check and let you know.

sahilsingh

I just simply executed the code which i had already posted.I took the token names from downloading the instrument list by zerodha.
w = pd.DataFrame(kite.instruments())
token name was produced by w.instrument_token[i]//iterate through the first 50 instruments in the list

sujith

Hi @sahilsingh,
It is not an error. It means candles doesn't exist for the instrument user has requested for.
Maybe the instrument was not traded on the day you had requested for.

sahilsingh

Not possible.A few day ago the same code had no problem in retrieving data.So did the data suddenly disappear?.Also btw this is happening for many other instrument lists too.

sujith

Hi @sahilsingh,
There many instruments which are of T2T segments in that dump. Normally instruments are traded on EQ segment. So, if you are requesting data for T2T segments then it will not have data. There can be F&O instruments also if you are requesting data for NIFTY17SEPFUT with date 2015 then it will give you the same error.

sahilsingh

Yes i know that and thats why i changed the file to contain only equities.The following problem is being experienced with equities.

sujith

Hi,
T2T segment instruments also belong to Equity segment.
We use the same APIs on all our platforms, it is working fine.

sujith

If the error message says no candles then it means it was not traded on the day you are requesting for.

sahilsingh

Then tell me how to eliminate the T2T instruments from the file or tell me how do you identify such instruments

sujith

Hi @sahilsingh,
Check out this thread for more information about T2T segments. Usually, tradingsymbol will have the series name like BE, BL appended to it.
For EQ segment instruments, series is not appended.