Hi I run my scripts on jupyter notebook. After log in flow , we get request token using which we create kite instance and generate access as well as public tokens. However when I test my code and I make any changes in the main script and /or in modules , I am required to restart the kernel. In the process all variables in the session are lost. I enabled auto reload also but of no help. Though access token persists but kite instance needs to be recreated every time kernel is started by generating request token. This is very inefficient. I am not sure if I am doing the things correctly. Any guidance in this regard shall be of great appreciation and help. Also which IDE is recommended for such type of tasks involving live data and analysis thereof? regards.
I am utterly unaware of Python internals. But from the perspective of commonality amongst languages. How is instance creation expensive?
Senarios
1. Instance is of a class which on creation will instantiate abnormally large number of identifiers inside, which might consume time.
on modern processor this can only happen if you are creating such instances in a loop over and over again. even then you would come out winner because thousands of instances would be created that is if a class is abnormally large. otherwise you wont notice the difference. ( This does not happen in kite variable )
2. Instance of a class does some deserialization from disk which consumes time. ( not with kite ).
3. Instance of a class is trying to connect over network at the time of instantiation and thereby consuming time. ( no with kite ).
Kite instance variable only connects to Kite server when requested. So every instance is just an address in memory of your computer occupying some space according to the size of variables which is insignificant.
Once you have stored Session information you can use it freely while doing anything else. Or if it suites your style make a Global Kite variable. until you figure out some other way.
Creating of Kite instance to throw a request at Kite server is not a expensive operation unless it has something to do with Python being a dynamic language.because instance is being created directly in your computers memory and not by first seeking permission from Kite server.
Why not make a class which instantiates kite variable inside it and then use that class to put requests to Kite server. through that variable.
@tonystark : Thanks for your reply . Perhaps my question was not clear as else your reply would not have been so simplistic in nature. Please explore if you could have a better one in view of the restatement of the issue I am facing. I have not dealt with oop in whatever little experience I have while working with C. Now I strictly followed the examples provided in the docs. For example (I am reproducing sequentially): 1) kite=KiteConnect(api_key,api_secret) 2)After creating this instance I execute - url = kite.login_url() 3) With this url I request the request token 4) With the request token so obtained , I get access data : kitedata = kite.request_access_token(request token,api_key) 5) Now I have a dictionary " kitedata"from where I pull access token and public token for setting connection to web socket and access to oms etc - kite.set_access_token(--------------) and kws = WebSocket(......,........,USER_ID) . 6) I run all the above steps in separate cells in jupyter notebook.
So far so good, I get all the results as expected. But when I am testing some changes in the task (being handled by celery) - for any modification in the task file, I have to restart the kernel and in the process the instance of KiteConnect class created at step no. 1) is lost and I get the error message "kite is not defined". Then I follow all the steps again , finally getting the same access token and public token and then execute the code. This is where I feel I am not following the correct way by simply following the example. I have the tasks (functions ) written in one file which I import in the jupyter cell. Regards
@trade_then , thanks a lot. As explained to tonystark , my problem is to preserve the instance variable while changing and executing the code. I think I have to find a solution based on your suggestion of making a class. But again, all variables are lost when I restart the kernel. So even this class instance would also be lost. The kind of changes I make in my code are like changing the resampling frequency or Moving average periods or adding / dropping an indicator. For doing all these I go back to my task file and follows all those steps which I feel should be avoidable.
@RP3436 Preserving the instance object is a *bad* idea and you shouldn't be storing the object anywhere. When you refresh your ipython kernel (jupyter notebook is a frontend for ipy kernel) releases the old objects and basically starts a fresh session. It's your responsibility to store the access_token somewhere, where on a refresh of the session, this data isn't flushed. (eg store in redis, database, env variable, file etc.)
Now when you set the Kiteconnect object, check for access token first, wherever you have stored it. If access_token is None, only then you generate a new access token by doing the whole login-flow. Else you simply set the access token and continue using KiteConnect object.
@mrkaran thanks a lot for the reply. I have no confusion about what you said. I need not check for access token , as it is for sure , is flushed when the kernel is restarted. Let us assume , it is lost , then one has to go through all the steps - login url , request token , access token ...up to kite.set_access_token. Right? If noting down and storing the access token somewhere imply the same thing , then yes, I store it in a text file. But the kite object itself is lost . So is this usual while working on jupyter? Is there a way I can re run my script on jupyter cell after modifying the script or the module , without restarting the kernel ?
Apart from the python stuff ; You are missing only on one point. you are not storing the userinfo aquired from GenerateSession external to your scripting enviornment. so that it can be retrieved when you start again. you don't have to go through all the stuff again starting from login -> request token -> access token. you only have to start from access token and instantiate the Kite. That is once you have obtained the access token and the relevent userinfo . You should store it outside of your scripting environment. and retrieve it whenever required for the rest of the day. Indeed if your Kite variable is lost ( for whatever reason ) you would have to create it again. but this time only supplying it with already retrieved information from previous generate session.
Thanks Regards
just on the lighter side of it: How does one start with C and end up with Python and not with C#
1. Python -> six lettered word 2. C# -> One more word
Is there a way I can re run my script on jupyter cell after modifying the script or the module , without restarting the kernel ?
A very hacky (and insecure) way is to use pickle to serialize kiteconnect object. But you will have to store this serialized object somewhere externally, load it when you restart the kernel, deserialize it and then only you can use.
I don't really see any point or rather benefit with this approach. Why don't you just re-use the access token?
Thanks both of you @trade_then and @mrkaran for the insights. Now I will do some research to understand the approach. From the replies - it appears that restarting the kernel may be a required step but then why do I get messages like "invalid session" or "name kite is not defined". And when I repeat all the steps from beginning - it works fine. I will be rather happy to re use the access token , but honestly I do not know how to do that.
@trade_then : C was a no halt station (in 1990) in my long journey from mech engg - Electrical engg - Project & Structured Finance - Business Education - Stock Trading. C# a probable next halt.
And when I repeat all the steps from beginning - it works fine
That's how a Jupyter notebook is supposed to run It lets you be flexible with the code and allow you to execute any line from any order. But if _your_ program expects a variable to be in the memory space for the line to execute, jupyter magically can't load that up. You need to execute all the required commands in the order your program expects for that to happen.
Senarios
1. Instance is of a class which on creation will instantiate abnormally large number of identifiers inside, which might consume time.
on modern processor this can only happen if you are creating such instances in a loop over and
over again. even then you would come out winner because thousands of instances would be created
that is if a class is abnormally large. otherwise you wont notice the difference.
( This does not happen in kite variable )
2. Instance of a class does some deserialization from disk which consumes time.
( not with kite ).
3. Instance of a class is trying to connect over network at the time of instantiation and thereby consuming time.
( no with kite ).
Kite instance variable only connects to Kite server when requested. So every instance is just an address in memory of your computer occupying some space according to the size of variables which is insignificant.
Once you have stored Session information you can use it freely while doing anything else. Or if it suites your style make a Global Kite variable. until you figure out some other way.
Creating of Kite instance to throw a request at Kite server is not a expensive operation unless it has something to do with Python being a dynamic language. because instance is being created directly in your computers memory and not by first seeking permission from Kite server.
Why not make a class which instantiates kite variable inside it and then use that class to put requests to Kite server. through that variable.
Thanks
Regards
I have not dealt with oop in whatever little experience I have while working with C. Now I strictly followed the examples provided in the docs. For example (I am reproducing sequentially):
1) kite=KiteConnect(api_key,api_secret)
2)After creating this instance I execute - url = kite.login_url()
3) With this url I request the request token
4) With the request token so obtained , I get access data : kitedata = kite.request_access_token(request token,api_key)
5) Now I have a dictionary " kitedata"from where I pull access token and public token for setting connection to web socket and access to oms etc - kite.set_access_token(--------------) and kws = WebSocket(......,........,USER_ID) .
6) I run all the above steps in separate cells in jupyter notebook.
So far so good, I get all the results as expected. But when I am testing some changes in the task (being handled by celery) - for any modification in the task file, I have to restart the kernel and in the process the instance of KiteConnect class created at step no. 1) is lost and I get the error message "kite is not defined". Then I follow all the steps again , finally getting the same access token and public token and then execute the code. This is where I feel I am not following the correct way by simply following the example. I have the tasks (functions ) written in one file which I import in the jupyter cell.
Regards
access_tokensomewhere, where on a refresh of the session, this data isn't flushed. (eg store in redis, database, env variable, file etc.)Now when you set the Kiteconnect object, check for access token first, wherever you have stored it. If
access_tokenisNone, only then you generate a new access token by doing the whole login-flow. Else you simply set the access token and continue using KiteConnect object.Here's a very simple visual illustration:
Thanks
Regards
just on the lighter side of it: How does one start with C and end up with Python and not with C#
1. Python -> six lettered word
2. C# -> One more word
kiteconnectobject. But you will have to store this serialized object somewhere externally, load it when you restart the kernel, deserialize it and then only you can use.I don't really see any point or rather benefit with this approach. Why don't you just re-use the access token?
@trade_then : C was a no halt station (in 1990) in my long journey from mech engg - Electrical engg - Project & Structured Finance - Business Education - Stock Trading. C# a probable next halt.