Historical API gives frequent errors

Encrypted
Encrypted edited September 2017 in API clients
Hi, I was doing testing of my app off market hours. I found historical data is giving frequent errors.
The errors are "The remote server returned an error: (429)."
"The operation timed out".
I am using queue to hit server.(using dotnetAPI)
From forum I found that many of users are facing same issue.
Is this historical API still under development phase?
Please do needful.
Thanks for understanding our pain point.
Please let us know if you need more info to replicate errors.
(if you try to get 25 stock's 1 min interval data I hope you can easily replicate)
  • sujith
    Hi @Encrypted,
    HTTP error code 429 means you are getting rate limited.
    All Kite Connect HTTP calls are limited to 3 requests per second.
  • Encrypted
    Encrypted edited September 2017
    Hi@sujith but if I implement queue that should not happen.if you need I can share my code.please suggest if you we need to implement something different.i want to avoid threading.
  • sujith
    Hi,
    Can you paste your code?
  • Encrypted
    Encrypted edited October 2017
    created one queue class and inherited kite class and trying to invoke it singleton instance
    public class RequestQueue : KiteConnect.Kite
    {
    private readonly Queue<Historical> _requestHistory;
    private RequestQueue() : base(APIkey)
    {
    _requestHistory = new Queue<Historical>();
    }
    private static RequestQueue _singleton;
    private static readonly string APIkey;

    public static RequestQueue Instance()
    {
    if (_singleton == null)
    _singleton = new RequestQueue();
    return _singleton;
    }

    public void Enqueue(Historical request)
    {
    _requestHistory.Enqueue(request);
    }
    public Historical Dequeue()
    {
    return _requestHistory.Dequeue();
    }
    public List<Historical> Flush()
    {
    List<Historical> listHist = new List<Historical>();
    Historical hist = new Historical();
    while (_requestHistory.Count > 0)
    {
    //hist = _requestHistory.AsParallel().WithExecutionMode(ParallelExecutionMode.ForceParallelism).d

    hist = _requestHistory.Dequeue();
    listHist.Add(hist);

    }
    return listHist;
    }
    }

    //from the main class taking stocks name from xml file as below and calling enque deque function
    RequestQueue req = RequestQueue.Instance();
    //req.
    List<Historical> historical = new List<Historical>();
    string date = DateTime.Now.ToString("yyyy-MM-dd", System.Globalization.CultureInfo.GetCultureInfo("en-US"));
    foreach (var node in nodeList)//all my stocks from xml stored nodelist
    {
    stockName = node.SelectSingleNode("STOCKNAME").InnerText;
    string strStock = "select instrumenttoken,convert(date,getdate(),112) from tblInstrument where tradingsymbol= '" + stockName + "'";
    dsStock = new DataSet();

    SqlDataAdapter objStock = new SqlDataAdapter(strStock, conn);


    //conn.Open();
    objStock.Fill(dsStock);


    try
    {//NEED TO CHECK CAREFULLY BELOW CODE
    req.Enqueue(kite.GetHistorical(dsStock.Tables[0].Rows[0][0].ToString(), date, date, "minute", stockName));

    ......

    historical = req.Flush();
    for (int i = 0; i < historical.Count; i++)
    {

    storing in database log

    Extract(strTableName, historical);
    }
    }//for each loop close
  • Encrypted
    Encrypted edited October 2017
    public Historical GetHistorical(
    string InstrumentToken,
    string FromDate,
    string ToDate,
    string Interval,string stockname)
    {
    var param = new Dictionary<string, string>();

    param.Add("instrument_token", InstrumentToken);
    param.Add("from", FromDate);
    param.Add("to", ToDate);
    param.Add("interval", Interval);

    var historicaldata = Get("market.historical", param);

    List<Historical> historicals = new List<Historical>();

    foreach (ArrayList item in historicaldata["data"]["candles"])
    historicals.Add(new Historical(item));
    int cnt = historicals.Count;
    clsDBCreation.clsDBCreate objDBCreate = new clsDBCreation.clsDBCreate();
    SqlConnection conn = new SqlConnection();
    conn.ConnectionString = "....";
    string strSqlCreate = objDBCreate.createMainTable("tbl" + stockname);
    ...
    objcmdCreate.ExecuteNonQuery();
    string strSqlCOUNT = objDBCreate.checkMainTable("tbl"+ stockname);
    SqlDataAdapter objDS = new SqlDataAdapter(strSqlCOUNT, conn);
    DataSet dscount = new DataSet();
    ..
    objDS.Fill(dscount);
    ...
    int finalcountcheck = 0;
    if (dscount.Tables[0].Rows.Count > 0)
    {
    finalcountcheck = Convert.ToInt16(dscount.Tables[0].Rows[0][0].ToString());
    string currTime = dscount.Tables[0].Rows[0][1].ToString();
    }
    return historicals[finalcountcheck];
    }
  • Encrypted
    tested yesterday and today with 60 stocks 1 min data.from 188th candle started giving error.upto 188 th worked fine.
  • tonystark
    Hi @Encrypted,

    Even if you are using queue this may happen. Because sometimes requests might complete in short amount of time. And next request will be fired immediately. If more than 3 requests are called in a second like this you will get 429.

    You can either add ~300ms delay in between each request or add a retry mechanism if you are getting 429 to solve this issue.
  • Encrypted
    Hi,getting invalid api credentials.I am now testing my application.Suddently it started since last 5 mins.I regenerated tokens.But it is giving forbidden error.
  • Encrypted
    from yesterday night till now I am getting same error.not able to anything.
  • darkknight
    same error for me as well, last weekend as well as this weekend. hardly works 10% of the time.
  • Encrypted
    Encrypted edited October 2017
    Are we doing something wrong.i can see 4-5 people reporting same error.is it working fine for others?or we are only using it..
  • Encrypted
    Encrypted edited October 2017
    further debugging I found there is code issue from my app end.but they have also appended below line in Kite.cs
    if (((HttpWebResponse)webResponse).StatusCode == HttpStatusCode.Forbidden)
    _sessionHook?.Invoke();
    if invalid api error comes that is taken care.
    but is it good practice ?
  • sujith
    @Encrypted,
    In .NET client session hook is invoked for every 403 errors which can be any other issue like invalid api_key(ForbiddenError in KiteConnect's context). We will change it to only invoke session hook when there is TokenException.
This discussion has been closed.