Order Placement Errors 9:00 am to 9:05 am

I see weird behavior on order placements on start of currency market.
Orders placed using kite.order_place at 9 am or slightly after don't return an order_id. I get an exception stating "Gateway timeout"
However I see that the very same order shows up in order book in maybe 1-2 minutes and gets executed if the price is reached. This is a big problem for me.
So is this a known bug? Or is this a system limitation?
In either case do you have any workarounds?

  • razcads
    Any updates on this issue? Continues to happen every day.
  • sujith
    Our OMS processes tens of thousands of pre-market orders at this time. We are in talks with OMS vendors to come up with a feasible solution.
  • Shaha
    try placing AMO if that helps.
  • razcads
    AMO order doesn't do as I take make my strategies based on current market rates. Most of currency action gets done in the first 5 mins and not having ability to track orders at this stage put paid to any strategy used. Hope to get a speedy solution to this. Please keep this ticket open till that happens. Thanks .
  • sujith
    We tried placing orders from 9:00:00 AM to 9:05:00 AM for three to four days. We observed this during the first 30 seconds of 9:00:00 AM.
    We place tens of thousands of AMOs at 9:00:00 AM and RMS will be busy in handling those ordes which will result in this. Now, OMS team is working on prevalidating orders at 8:48:00 AM which will resolve the issue.
  • razcads
    @Sujith Thanks for simulating this at your end. Yes this happens during the first 30 seconds on low volume days and extends beyond 9:01 on high volume days. Look forward to your team pre-validating the orders earlier. This will solve a major (for me) glitch.
  • razcads
    Adding on..
    Sometimes I have also seen this error around same time. This happens after the order has been placed and order id retrieved and post that I query using kite.orders for that order_id.
    Couldnt find that `order_id`
  • razcads
    @sujith This is still not resolved. I continue to get these errors. What is the update here?
  • sujith
    How often does it happen? Can you give me order id?
  • razcads
    Happens for orders placed at 09:00 am. Happened today as well. 180216000015312 .

    Situation is like this - I place order using order_place, get order_id and then when I try to retrieve the order to see if it has been executed I get this error - it cant find the order id!
  • sujith
    The OMS team is working on it. They are testing it out. Maybe in a couple of weeks, we will be prevalidating AMOs at 8:48 AM and that should resolve this issue.
  • razcads
    Hi @sujith
    Any update on this?
  • sujith
    There were some issues and OMS team had to re-do things.
    Hopefully, if everything goes well we will take it live in a couple of weeks.
  • razcads
    @sujith any further progress with this?
  • sujith
    We are working on this.
  • sudhirshettyk
    @sujith , if you don't mind can you please add more details of root cause of why a order which was responded with a order id , later gets "can't find the order id" error .
  • sudhirshettyk
    @sujith , any comments ? It will help us to try and catch or mitigate failure of our product
  • sudhirshettyk
    @sujith, please comment
  • sujith
    It seems like a bottleneck at core OMS. We have tens of thousands of AMOs being sent at the market opening. But the exchange line has some limit on the number of messages that can be sent wherein each order is a message.

    We are thinking of speaking to the exchange about the architecture and infrastructure stuff.
  • sudhirshettyk
    Hi Sujith,

    Sorry , that doesn't explain the issue right?
    The issue - Your system issues a order-id and subsequently when queried says order-id not found. How any external dependency come in to play unless you are doing something systemically wrong.

    My understanding of general simplified sequence as i would see will be :
    1) Your system generates a order-id
    2) Your system records the order with its order-id in your backend database.
    From this point , even if your system crashes, exchanges has issues , the order-id should not be lost (database is commited)
    3) you respond to user with order-id (this should be the guaranteed indication to user that my order request got recorded in your system )
    4) you place order to exchange

    Now , how is exchange bottleneck stops you from retreiving the record corresponding to the order-id from your own database ? Let me know if i am getting it wrong.

    But , in case you are responding with a order-id before commiting it to your database , OR you respond with order id , place order to exchange and then commit , you are doing something systemically wrong and it is a very dangerous situation w.r.t the integrity of the data . user gets the order-id and assumes order is placed , what if in between of you commiting the record to your database , exchange line crashes or your system crashes ....

    Please help address my concerns.

    Sudhir Shetty

  • sujith
    We don't create order_id. It happens at the OMS.
    We send a place order request on behalf of the client at OMS and we get order_id from the OMS.

    Even though the order is created, OMS is not returning any data for the first 30 seconds of the market opening.
  • sudhirshettyk
    Hi Sujith,

    Thanks , So receiving a order_id at our end doesn't gurantee that the order got recorded at your system ?
    BTW, OMS is not currently your own implementation ?
  • sujith
    Ideally getting order_id means there is an entry in the orderbook. But at market opening, it is taking time to update orderbook. This is an exception which is being looked into.
    We use OMS provided by the Thomson Reuters.
  • sudhirshettyk
    Hi Sujith,
    Thanks , this unfolds a lot more scenerio that i need to ring fence in my system -
    first of all , though you are seeing it at market opening , this can essentially happen whenever OMS is over-loaded . So from my system design i should expect that it can happen anytime .
    As you say , the order-id is generated by OMS and not at your end , then one scenerio that is the most important for me to know is following -
    say ,
    1) i place a order
    2) you request the OMS .
    3) OMS places a order and generates a order-id
    4) But you dont receive this order-id due to a blip in the network (blip i mean short duration issue like a packet drop)

    In this case what would be your response ? How will i eventually get to know this order indeed got placed at the OMS ? Will this order-id be orphaned in the OMS ? How do you suggest should i implement by system to ring fence this scenerio ?

    Thanks for your time and patience with my long questions.

  • sujith
    One of the scenario wherein order is placed and Kite Trade doesn't get the response from OMS then it will result in request timeout.
    In another scenario wherein you got order_id but the order doesn't exist in orderbook then you will see the following error
    status: "error",
    message: "Couldn't find that `order_id`.",
    data: null,
    error_type: "GeneralException"
  • sudhirshettyk
    Thanks. Then later when we request orderbook , the orderbook is fetched directly from OMS or your local recorded copy ? If former is the case then i must be assured that the order will show up eventually in the orderbook . Can you please confirm.
  • sujith
    It is a push message from the OMS. We just show whatever we get from the OMS.
  • sudhirshettyk
    okey , so do you guarantee that in the former case that you specified above (order is placed and Kite Trade doesn't get the response from OMS then it will result in request timeout) ,
    the order would "eventually" show up in the orderbook (please do consider the situation where the Kite could miss a PUSH message because of its internal issue) .
  • sujith
    Yes, even when there is a request timeout while placing an order. We would recommend double-checking orderbook. The order will show up there. This is the behavior I have seen so far.
  • razcads
    @sudhirshettyk I can confirm the behavior that @sujith alludes to. As I see it this error wont go away anytime soon as its to do with a different vendor who most likely wont change his OMS behavior for a border case considering there are only a minor fraction (a small subset of retail algo traders on Zerodha) who complain about it... so not much return on investment for them.
  • sudhirshettyk
    Thanks @sujith @razcads .. i am fine with errors (any async system would) as long as we have a clear understanding of it , i can try and ring-fence it. So in this case as long as it is guranteed that a order doesn't get orphaned in OMS and eventually shows in orderbook i am fine .
    Coming back to this case , we have a issue if Kite recieves the PUSH notification and uses this to build a local orderbook which is what is queried when we request the orderbook . @Sujith , can you please confirm that this is not the case and you always fetch the orderbook from OMS and dont build your local orderbook based on PUSH notification you recieve .
  • sudhirshettyk
    Hi @Sujith , Can you please confirm the above question when you get some time.
    Best Regards,
    Sudhir Shetty
  • sujith
    Yes, it is a PUSH from OMS to Kite Trade but these are interprocess communications that happen internally. It shouldn't matter to end user. The whole flow is very complex to put it in one or two paragraph. Obviously, we won't take one year to do Kite Connect 3 if it is so simple right :)

    To answer your query whether there will be an orphan order in the OMS after you get order id. This is not possible.
    We are talking to OMS and the exchange to come up with a feasible solution to overcome the line capacity issue. In fact there some major backend things to be done which might require us to get the exchange approvals as well.

    We will update this thread once a fix is deployed.
  • sudhirshettyk
    Thanks @Sujith . BTW, I do appreciate the exponential level of complexity you would be facing and have been solving :-)
  • SBK
    SBK edited November 2018
    Hi @sujith ,

    I am facing similar Problem in cash market. I placed a cover order which is giving me an order id.
    Now, to find out the child order id, I have below python code:
    def get_cover_order_id( parent_order_id):
    all_orders = self.kite.orders()
    df = pd.DataFrame(all_orders)
    df = df[['order_id','parent_order_id']]
    orders_dict = df.set_index('parent_order_id')['order_id'].to_dict()
    o_id = orders_dict[parent_order_id]
    return o_id

    This code is not working for my early few trades (usually in the very first minute 9.15), but it works fine later on throughout the day. It returns "Couldn't find that `order_id`"

    Is it similar problem which has been discussed in this thread?
    If so can you suggest an alternative way of doing this.

  • sujith
    Can you share more details? Like for which instrument this is happening and at what time exactly you place the order?
    We had done some changes to avoid this but it seems like this is not resolved still. I will forward details to the concerned team and get this checked.
  • SBK
    Hi @sujith,

    Sure, PFA the screenshot of the error from last week.

    I placed the order successfully (cover order). Got order id. You can check the timings as well.
    Passed the order id to the code (pasted above already) to retrieve child order id. It could not provide that.
    Then I tried to modify that child order then it says "Couldn't find that order id".

    Let me know your comments.

  • jkoracle23
    @sujith - I am also facing two exceptions (mentioned below) time to time at random time in the MIDDLE of log due to kiteConnect.getOrderHistory(order.orderId)
    My code handles these exception and continue trading. But I would like to know why these come occasionally at random time ?
    (1)Couldn't find that `order_id` (2)Invalid `api_key` or `access_token`
    Details are:
    Example: Time is mentioned in the below log:
    2019-05-22 10:57:09 ERROR BuyStocks:1357 - kite exception in placeSellOrder in Buystocks class -->Couldn't find that `order_id`.
    2019-05-22 10:57:10 ERROR BuyStocks:1357 - kite exception in placeSellOrder in Buystocks class -->Couldn't find that `order_id`.

    2019-05-22 10:57:07 ERROR BuyStocks:1357 - kite exception in placeSellOrder in Buystocks class -->Invalid `api_key` or `access_token`.
  • sujith
    It could be because the order record is not propagated to all the systems yet. You can add some milliseconds delay and check.
    Invalid `api_key` or `access_token`
    This could be because your access token or api_key might be empty. Can you send the status code when this happens?
Sign In or Register to comment.