Historical API gives frequent errors

Encrypted

Hi, I was doing testing of my app off market hours. I found historical data is giving frequent errors.
The errors are "The remote server returned an error: (429)."
"The operation timed out".
I am using queue to hit server.(using dotnetAPI)
From forum I found that many of users are facing same issue.
Is this historical API still under development phase?
Please do needful.
Thanks for understanding our pain point.
Please let us know if you need more info to replicate errors.
(if you try to get 25 stock's 1 min interval data I hope you can easily replicate)

sujith

Hi @Encrypted,
HTTP error code 429 means you are getting rate limited.
All Kite Connect HTTP calls are limited to 3 requests per second.

Encrypted

Hi@sujith but if I implement queue that should not happen.if you need I can share my code.please suggest if you we need to implement something different.i want to avoid threading.

sujith

Hi,
Can you paste your code?

Encrypted

created one queue class and inherited kite class and trying to invoke it singleton instance

public class RequestQueue : KiteConnect.Kite
    {
        private readonly Queue<Historical> _requestHistory;
        private RequestQueue() : base(APIkey)
        {
            _requestHistory = new Queue<Historical>();
        }
        private static RequestQueue _singleton;
        private static readonly string APIkey;

        public static RequestQueue Instance()
        {
            if (_singleton == null)
                _singleton = new RequestQueue();
            return _singleton;
        }

        public void Enqueue(Historical request)
        {
            _requestHistory.Enqueue(request);
        }
        public Historical Dequeue()
        {
            return _requestHistory.Dequeue();
        }
        public List<Historical>  Flush()
        {
            List<Historical> listHist = new List<Historical>();
            Historical hist = new Historical();
            while (_requestHistory.Count > 0)
            {
                //hist = _requestHistory.AsParallel().WithExecutionMode(ParallelExecutionMode.ForceParallelism).d

                 hist = _requestHistory.Dequeue();
                listHist.Add(hist);
                
            }
            return listHist;
        }       
    }

//from the main class taking stocks name from xml file as below and calling enque deque function
 RequestQueue req =  RequestQueue.Instance();
                //req.
               List<Historical> historical = new List<Historical>();
               string date = DateTime.Now.ToString("yyyy-MM-dd", System.Globalization.CultureInfo.GetCultureInfo("en-US"));  
                foreach (var node in nodeList)//all my stocks from xml stored nodelist
                {
                    stockName = node.SelectSingleNode("STOCKNAME").InnerText;
string strStock = "select instrumenttoken,convert(date,getdate(),112) from tblInstrument where tradingsymbol= '" + stockName + "'";
                    dsStock = new DataSet();

                    SqlDataAdapter objStock = new SqlDataAdapter(strStock, conn);
                 

                    //conn.Open();
                    objStock.Fill(dsStock);

                   
                try
                    {//NEED TO CHECK CAREFULLY BELOW CODE
                        req.Enqueue(kite.GetHistorical(dsStock.Tables[0].Rows[0][0].ToString(), date, date, "minute", stockName));

......

historical = req.Flush();
                    for (int i = 0; i < historical.Count; i++)
                    {

                        storing in database log
                      
                        Extract(strTableName, historical);
                    }
}//for each loop close

Encrypted

public Historical GetHistorical(
            string InstrumentToken,
            string FromDate,
            string ToDate,
            string Interval,string stockname)
        {
            var param = new Dictionary<string, string>();

            param.Add("instrument_token", InstrumentToken);
            param.Add("from", FromDate);
            param.Add("to", ToDate);
            param.Add("interval", Interval);

            var historicaldata = Get("market.historical", param);

            List<Historical> historicals = new List<Historical>();

            foreach (ArrayList item in historicaldata["data"]["candles"])
                historicals.Add(new Historical(item));
            int cnt = historicals.Count;
            clsDBCreation.clsDBCreate objDBCreate = new clsDBCreation.clsDBCreate();
            SqlConnection conn = new SqlConnection();
            conn.ConnectionString = "....";
            string strSqlCreate = objDBCreate.createMainTable("tbl" + stockname);
          ...
objcmdCreate.ExecuteNonQuery();
            string strSqlCOUNT = objDBCreate.checkMainTable("tbl"+ stockname);
            SqlDataAdapter objDS = new SqlDataAdapter(strSqlCOUNT, conn);
            DataSet dscount = new DataSet();
        ..
objDS.Fill(dscount);
         ...
int finalcountcheck = 0;
            if (dscount.Tables[0].Rows.Count > 0)
            {
                finalcountcheck = Convert.ToInt16(dscount.Tables[0].Rows[0][0].ToString());
                string currTime = dscount.Tables[0].Rows[0][1].ToString();
            }
            return historicals[finalcountcheck];
        }

Encrypted

tested yesterday and today with 60 stocks 1 min data.from 188th candle started giving error.upto 188 th worked fine.

tonystark

Hi @Encrypted,

Even if you are using queue this may happen. Because sometimes requests might complete in short amount of time. And next request will be fired immediately. If more than 3 requests are called in a second like this you will get 429.

You can either add ~300ms delay in between each request or add a retry mechanism if you are getting 429 to solve this issue.

Encrypted

Hi,getting invalid api credentials.I am now testing my application.Suddently it started since last 5 mins.I regenerated tokens.But it is giving forbidden error.

Encrypted

from yesterday night till now I am getting same error.not able to anything.

darkknight

same error for me as well, last weekend as well as this weekend. hardly works 10% of the time.

Encrypted

Are we doing something wrong.i can see 4-5 people reporting same error.is it working fine for others?or we are only using it..

Encrypted

further debugging I found there is code issue from my app end.but they have also appended below line in Kite.cs

if (((HttpWebResponse)webResponse).StatusCode == HttpStatusCode.Forbidden)
                        _sessionHook?.Invoke();

if invalid api error comes that is taken care.
but is it good practice ?

sujith

@Encrypted,
In .NET client session hook is invoked for every 403 errors which can be any other issue like invalid api_key(ForbiddenError in KiteConnect's context). We will change it to only invoke session hook when there is TokenException.

Encrypted

Thanks.