Historical data - too many iterations to get the data

ramatius
Hi,
For many scripts, the historical data API is returning data only after several tries. For example, when I try to get the data for HINDZINC between 2017-03-01 and 2017-03-31, the call returned error in first 12 calls (same calls), but returned data on 13th attempt.

Some scripts take many more attempts, sometimes 20+.

While this trial-and-error process has been automated, it takes nearly one day to get the data (to a reasonable extent) for the scripts of interest & time. The real issue is that if the call fails, it is impossible to tell whether data is available or not.

Please review this issue & fix it. If the data is present, return it in the first call itself.
  • mohit77bansal
    I am too facing the similar issue. Also, i tried to extract data using multithreading, it always show Too many request error. Please resolve these issue and suggest some alternate solutions.
  • sujith
    sujith edited April 2017
    Hi @ramatius,
    I would suggest trying with from date 2017-03-01 and to date 2017-03-20.

    @mohit77bansal,
    All Kite connect http requests are limited to 3 requests per second.
  • ankur0101
    ankur0101 edited April 2017
    @sujith , but why not 2017-03-31?
  • sujith
    Hi @ramatius,
    I think in your scenario what happens is you are placing a http request to fetch data and before database call completes fetching the row, your request is failing with the timeout. Hence I would suggest you try with less time interval.
  • ramatius
    Hi @sujith The timeout is 3 minutes in my script. The server does return error long before that. Sometimes I do get timeout (i.e. server doesn't respond for 3 mins), but this is quite rare, may be once in 1000+ calls.

    Please check.
  • Vivek
    @ramatius Can you post the error response you are getting? You need to wrap your historical call with try catch block and print the error response.
  • ramatius
    The error says "Error GET ing http:..... : Bad request
  • sujith
    @ramatius,
    Bad request means the request sent from client to server is malformed. Maybe you should check the params you are sending.
  • ramatius
    The same call succeeds after a few tries. There is nothing to get malformed - the URL call is very straightforward.

    My guess is that you have some internal timeout mechanism and it returns an error if the data is not there in the server cache. When the data gets into the cache, then the call succeeds.

    I request you to try this yourself and see. It will save lot of time & resources on both sides to get the data in the first call itself, if it is present.
Sign In or Register to comment.