API rate limits

rakeshr
rakeshr edited July 2023 in General
We always had API rate limits in order to prevent abuse and bad actors from accessing our API servers. This helps us to control and allow legitimate traffic only.

In order to further improve our security, we've tightened the existing limits only for bad requests i.e HTTP 400 Bad Request and HTTP 403 Forbidden status codes. We've observed some instances where there are API calls being made without a proper authentication token or with bad input fields. These are the only two new cases where we have started throttling these requests. In all other cases, the previous rate limits continue to work as applicable before.

Edit:

Throttling Period

We have introduced a small throttling period to reduce the bad requests load effect. There's a cooldown period of 10s that must be respected whenever you receive a 429 Status Code. It's a sliding window of 10s, so if the user sends more requests before the cooldown period is over, the throttling window of 10s is further extended. This only affects cases where people are firing API requests in "infinite loop" scenarios and not properly handing response codes.

If you are continuously getting 429 because of the above sliding throttling period, you can follow the steps below:
Debug your codebase for the request point continuously sending 40X(invalid input) exceeding the rate limit defined here. Add a proper sleep period in the loop to accommodate the rate limits. Then wait for 10 seconds and restart the program.

PS: This won't affect users within the previous rate limits anyhow. If you are facing 429, you need to check your existing scripts/codebases and validate/sanitize the request fields in order to avoid sending 40X requests.
  • AbhyudayaCodes25
    Ok. Thanks for the clarification. Yes, our system did place lots of 403 requests to the API yesterday. But, due to that, we are unable to use the API till now. What to do to resolve this. Kindly guide us.
  • rakeshr
    What to do to resolve this. Kindly guide us.
    As said above you need to find and kill those requests.
    You need to check your existing scripts/codebases and validate/sanitize the request fields in order to avoid sending 40X requests.
  • AbhyudayaCodes25
    AbhyudayaCodes25 edited October 2022
    No, the program is not even starting now. Every request is responding 429 Too many requests since yesterday.
  • Vaga
    @rakeshr There is no change in the way we have been requesting the data, yet we are facing too many request error since yesterday. Is there something else that has been changed in rate limit or api requests?
  • AbhyudayaCodes25
    @rakeshr Can you elaborate what is the limit that is applied for 400 and 403 requests ?
  • AbhyudayaCodes25
    AbhyudayaCodes25 edited October 2022
    As said above you need to find and kill those requests.

    @rakeshr We already removed those extra requests, but still too many requests is coming.
    And once too many requests error starts coming in, we are not able to do anything for the rest of the day. Every request gets same error continuously for the whole day. If we once trigger the new rate limits, then after how much time can we request again ? We would like you to clarify and kindly solve this asap.
  • 4v5
    @rakeshr
    Can we have a zoom call to take this forward?
    I pay 2000 every month to use Zerodha
    Your own support team is denying to take up service request
    Its been 3 days since i have been paying with no service
    If you are unable to handle this please engage someone else, and have this taken forward

    If not, let me know the escalation path
    I would like to have this resolved and have zerodha pay for non working for 3 days now
  • sujith
    @4v5,
    We don't provide one to one support for Kite Connect. You can write technical queries here and we will help you out.
  • AbhyudayaCodes25
    Can you check if this is the case at your end?

    Yes @sujith. This is exactly what we were doing, but before yesterday. Yesterday, according to @rakeshr 's suggestion, we changed it so that now it only starts requesting after user logs in to zerodha, by adding a time flag. Therefore, it makes no 403 requests now before the user is logged in.
    And today that function worked fine, and the system did recognize that user has logged in to zerodha without sending any 403 requests. But after login, as soon as it started fetching quotes, too many requests error started popping up again. Therefore, my query is
    1) Can you elaborate the new rate-limit that you have added from your end for 400 and 403 requests ?
    2) Also, what is the timespan after which we can request again if the new rate-limit is hit ? Like in your previous rate limit of 10 req/sec. If we hit it, we can retry again from the very next second.
  • sujith
    Please private message your IP, we will check stats and get back to you.
  • sujith
    @AbhyudayaCodes25, @sandesh, @YashV
    We had escalated this with Cloudflare team. This should be fine now. Can you check the let us know?
  • AbhyudayaCodes25
    AbhyudayaCodes25 edited October 2022
    No @sujith. It is still not fine. Its still not working. Do we get banned for the whole day if we trigger any 40X error ? It worked for sometime today morning, but after that, one too many requests error came, and then too many requests error is coming till now.
    We would request you to handle and solve this asap.
  • abhishek26
    Still facing 429 issues!
    Please look into it urgently.
  • sujith
    sujith edited October 2022
    Please re-read the original post on this thread, we have updated the initial write up with more details.
  • AbhyudayaCodes25
    AbhyudayaCodes25 edited October 2022
    We have added 20 secs sleep, still too many requests error is coming. What to do now ?
    Kindly guide.
  • sujith
    You need to enable debug logs of pykiteconnect and see if you are actually making requests after 20s or not.
  • AbhyudayaCodes25
    Ok let us check
  • namratasonawane
    namratasonawane edited October 2022
    @sujith

    I understand the measures taken to reduce load caused by bad requests. I have a a few questions:

    1. Is the IP limit still active? (250 requests in 10seconds from single IP). I am assuming this limit is for valid requests.
    2. How many bad requests in how many seconds before the throttling starts? What is the exact rule? Asking this here, because it is not documented in api rate limits.
    3. Could you please elaborate what are the causes for 400 bad requests. I understand it could be invalid orders, but is there any set of checks that you can provide. For example, I noticed that when a limit price is outside the circuit limits OR when a non-amo is sent outside market hours, even that results in a bad request error.
  • AbhyudayaCodes25
    AbhyudayaCodes25 edited October 2022
    @sujith @rakeshr
    Now, we seem to get response from Kite API, but sometimes as soon as we start the program, too many requests start to come. Even if no requests were made before from that IP. And we have to wait for multiple 10s, like 100 seconds, after that it will respond. Can you check over it once ?
    It seems like too many requests is coming without even making any requests initially.
  • namratasonawane
    namratasonawane edited October 2022
    @sujith

    "Some" of the causes of bad requests:

    1. 400-Buy orders are blocked for this strike price due to [open interest (OI)]
    2. Order price outside circuit limits
    3. Non-amo order placed outside market hours

    For reasons like point 2 & 3, obviously we can do checks on our side (provided you had warned us in advance).

    But if you look at the cause in point 1, there is no way for us to find this cause or do checks before placing a request. These are just samples, there could be many reasons like these. And this is the primary reason why many of your genuine customers are complaining.


    The change you made had a valid intension but it was not implemented correctly. It may have reduced load, but it is affecting genuine customers.

    On top of that you have not even shared complete details of how the check is done.

    We know this would not cause a big hole in your brokerage revenues, but it is affecting your reputation.

    So unless you come up with an intelligent solution, you should rollback this change.


  • AbhyudayaCodes25
    AbhyudayaCodes25 edited October 2022
    Yes, I completely agree with @namratasonawane . You should roll back the change for now if you don't have some concrete solution to our problems.
  • namratasonawane
    namratasonawane edited October 2022
    @sujith @AbhyudayaCodes25

    1. Order cannot be modified as it is being processed. Try later.
    2. MARKET orders are blocked for stock options due to illiquidity

    Two more examples of Bad Request for which you cannot have validation upfront.
    Error 1 - nothing we can do
    Error 2 - Market orders blocked is a RMS policy which can change from time to time

    How do you expect us to limit these?

    I believe we have given you enough proof to deeply consider a rollback or a more intelligent solution.
  • jeevan_bhatt
    jeevan_bhatt edited October 2022

    hey @sujith @rakeshr I am also facing this issue because of this
    MARKET orders are blocked for stock options due to illiquidity

    I have stopped the program for today because i was not able to place orders, do i need to raise ticket or i will be able to hit your api successfully from tomorrow?
    please help its Urgent, I have some positions opened.
  • rakeshr
    @namratasonawane @AbhyudayaCodes25
    If you follow the rate limits for order placement, in no way you will be affected, even for any 40X status orders as you won't encounter with throttling slide period, as these get triggered only if you breach the above rate limits.
    If in any case, you are handling multiple users from the same server/IP, you can DM me the app detail and server IP.
  • rakeshr
    @jeevan_bhatt
    You are exceeding the order limits of the day as described here. Debug your program and make sure to not exceed the rate limits. You can follow the steps mentioned above in the top thread with header If you are continuously getting 429 because of the above sliding throttling period, you can follow the steps ...
    I have some positions opened.
    For the manual close of position, you can reach support here.
  • jeevan_bhatt
    @rakeshr ,I have solved the issue on my end , will I be able to place orders tomorrow?
  • jeevan_bhatt
    @sujith @rakeshr I am still getting the same error, though i have fixed the issue, I am able to get the quote and prices but whenever I am trying to place order still getting :

    kiteconnect.exceptions.NetworkException: Maximum allowed order requests exceeded.

    please help it is costing me money
  • namratasonawane
    @rakeshr

    I have given you list of IP addresses.

    I still wonder what changes were made recently. Could you please shed some light on that?

    Because the API limits were always present, we never got "Too many requests" error. And these days even after taking many precautions, the errors are still coming.
  • rakeshr
    @namratasonawane
    As said above in the announcement thread, we have started throttling 40X(invalid requests) per IP based. So, in your case(of handling multiple users from the same IP), even if you don't breach previous rate limits per user, but there can be a chance of triggering a limit of 40X, if all API users 40X requests are combined at any instance. We are checking on your sent IP detail if this is a scenario.
  • AbhyudayaCodes25
    @rakeshr We have sent only one 40X request since morning, but too many requests is coming. We are messaging you the IP.
  • rakeshr
    We have sent only one 40X request since morning, but too many requests is coming.
    This is not possible at all. You need to re-check all your programs spun up.
  • AbhyudayaCodes25
    @rakeshr I have messaged the IP. Firstly, kindly check all the requests made by this IP.
  • namratasonawane
    @rakeshr

    As per the throttling notes, I see following line.

    We have introduced a small throttling period to reduce the bad requests load effect.

    What it tells us is that when there are bad requests, then it will cause throttling. But we need more clarity on the following.
    1. When does throttling starts? On how many bad requests in how much time duration?
    2. Does your bad request check exclude some valid causes like the ones mentioned below. Ideally, your bad request check should exclude these causes.

    Causes
    - price outside circuit limits
    - market orders not allowed in certain options
    - buying is blocked in certain options (only exiting existing positions is allowed)
  • vatsarj
    Hi API Support Team
    I have multiple family accounts and I have a program that sends orders / trades to all these accounts using your API, but recently I have started getting Login Error, so wanted to check do you have some rate limit on logging into multiple accounts one after the other from a particular IP address ?
  • sujith
    @vatsarj,
    You can private message the IP and we will check and get back to you.
  • namratasonawane
    @sujith @rakeshr

    https://kite.trade/docs/connect/v3/exceptions/#api-rate-limit

    Are these limits given on above link per user or per IP?

    Could you please shed some light on what are limitations per IP?
  • vatsarj
    @sujith How do I send a private message ? Please guide. Thnx
  • namratasonawane
    @sujith @rakeshr

    Patience are running out now... Even after multiple questions...still no concrete answer on what are the limits applicable per IP.

    Just like other vendors, we also have no choice now to ask our clients to start shifting to other brokers.

    Anyways, thanks for all these years. Zerodha was the best in terms of speed and reliability.
  • naveen278
    my patience is also running out zerodha is not letting me post.
    getQuote is not working for I tried with NFO:NIFTY03NOV2217800CE

    Can some one help me?

  • sujith
    @naveen278,
    Please don't hijack the thread. If you are unable to post then you can search for the similar queries on the forum and comment there. You need to give request and response logs with query if you want someone to help.
  • AbhyudayaCodes25
    @naveen278 Your input is wrong. The input is different for monthly expiry options and weekly expiry options. If your expiry is the last expiry of the month, then you need this scrip: NFO:NIFTY{YY}{MMM}17800CE. For ex: NFO:NIFTY22NOV17800CE. For any other expiry of the month, you need this scrip: NFO:NIFTY{YY}{M}{DD}17800CE. For ex, NFO:NIFTY22N0317800CE. Also, in the weekly expiries, the M is more complex. It needs to be 1 - 9 for Jan to Sep. But, for October, Nov, Dec, it needs to be O, N, D. Hope it clarifies.
  • naveen278
    Thank you I was looking for this format NFO:NIFTY{YY}{M}{DD}17800CE..
    I see that in the earlier query I had, the folks did not give the right information and just closed and blocked from posting. Thank You Abhuudaya
  • vatsarj
    @sujith could you please let me know how do I DM you my IP address, please refer to my previous post for my issue. Thnx
  • AbhyudayaCodes25
    @vatsarj Go to @sujith's profile and click on the small message button on the upper right side. Then, in recipients, it should show sujith and in the message box, type your message, then click on send. It will send a private message.
  • vatsarj
    thnx a lot @AbhyudayaCodes25

    @sujith I have sent you a private message with my IP details, please check and let me know.
  • AbhyudayaCodes25
    @namratasonawane
    https://kite.trade/docs/connect/v3/exceptions/#api-rate-limit
    I think the limits given on this link are per user based and not IP based.

    I reduced the number of requests the algo sends and put some handlers on unnecessary bad requests and changed to websocket for ltp fetching instead of quote function. And it fixed my issues mostly. I also added a 12 second sleep if 429 error comes as extra safety. You can also try these changes and see if these work.
  • amit_aggrawal321
    @sujith @AbhyudayaCodes25 In our app, we read orderbook every 2 seconds for multiple users.. When we read orders for 2-3 accounts it does not throw error, but if we try to fetch for more accounts (Simultaneously using threading from same IP address), it throws error.. So can you please confirm, whether the rate limit is account based or IP based ? Moreover suggest if this approach is fine and if it not then please suggest how shall we go about it?
  • sujith
    @amit_aggrawal321,
    Can you private message the api_key and IP? We will check and get back to you.
  • AbhyudayaCodes25
    @sujith and @rakeshr Although mostly the programs are running now, but still sometimes too many requests error is coming randomly. Can you verify from your end that everything is fine or not ?
  • amit_aggrawal321
    @sujith Sent your several DMs but did not get reply from you.. Please assist to solve the issue or please connect me with technical person from your team.
  • Harry
    Is api rate limit for a user or an IP. Because I am planning to place order for 8 users from ome IP?
  • rakeshr
    rakeshr edited November 2022
    Because I am planning to place order for 8 users from ome IP?
    No, should not be an issue, for each user having an individual API key.
  • explorer
    https://kite.trade/docs/connect/v3/exceptions/#api-rate-limit is showing nginx 404 not found page. Entire API docs are not accessible. Can somebody please look into it?

    I need to understand more about how rate limiter is implemented i.e. when exactly the counting window is reset. Is it reset at start of every second (i.e. 1, 2, 3 second etc.) or is it sliding log where the timestamps are recorded for each request so that rate limiter can detect if there were permissible requests made in 1 second sliding window?

    I see there are multiple ways to implement rate limiters so I need to know exactly how Kite implemented so that I can implement throttling on my side to honour the rate limitations.

    For order limitation of 10 orders/second
    Fixed window - Reset requests counter at the start of every second (e.g. 1, 2, 3 second etc.)
    Sliding log - Orders accepted if < 10 orders in last 1 second (1 second counted from current timestamp i.e. if now is 50s10ms, 1 second window started 1 second in past i.e. from 49s10ms, not from 50th second)
    Some other algorithm?

    Can somebody from Zerodha confirm and explain with small example. It would be extremely beneficial for others as well. Thanks in advance and also for rock solid APIs!
  • rakeshr
    showing nginx 404 not found page
    Should be fine now. Apologies for this.
    I need to know exactly how Kite implemented
    It's sliding throttling period. Go through Throttling Period section above, it's explained in detail.
  • explorer
    Thanks for quick action.
    It's sliding throttling period. Go through Throttling Period section above, it's explained in detail.
    This explains the scenario when the API rate limit is breached. I need to understand how do I avoid running into that situation. If I know how exactly 1 second period/window is calculated on kite backend then I can batch my API requests accordingly and never run into rate limited errors.
  • Swarn
    I am having the below error very frequently . Please help to resolve.
    {"exc_type": "ReadTimeout", "exc_message": ["ReadTimeoutError(\"HTTPSConnectionPool(host='api.kite.trade', port=443): Read timed out. (read timeout=7)\",)"], "exc_module": "requests.exceptions"}
  • rakeshr
    HTTPSConnectionPool(host='api.kite.trade', port=443): Read timed out. (read timeout=7)
    Can you let us know, which specific API request is throwing this error? Is it happening for all the requests?
Sign In or Register to comment.