Help with Data Handling in Python

heyitsabi

Hey guys,
1. I am very new to python and algo trading. I have recently subscribed to the API and I am having issue with handling the data.

Example:
When I use KiteConnect - kite.quote to fetch data - I am getting a set of dictionaries and I have used a loop to bifurcate the data, which is working fine but is inefficient because of how dictionaries work in python, sometimes the data received has different location for different keys.

t = kite.quote("NIFTY50","12111106","NSE:HDIL","NSE:HINDALCO","NSE:ITI","NSE:SETCO","SDBL","NSE:MTNL","NSE:DAAWAT","NSE:KOHINOOR")
columns = ['symbol']
temp = list()

for i in t.keys():
    values = []
    counter = 1
    if i == "12111106":
        values.append("NIFTY18AUGFUT")
    else:
        values.append(i)
    for j in t[i].keys():
        #print(j,t[i][j])
        if j == 'ohlc':
            for k in t[i][j].keys():
                if k not in columns:
                    columns.append(k)
                values.append(t[i][j][k])
        elif j == 'depth':
            for k in t[i][j].keys():
                #print(k)
                if counter >= 5:
                    counter = 1
                for items in t[i][j][k]:  #this portion handles the buy/sell depth 
                                                      #for some reason it is a dictionary inside a list
                    for l in items.keys():
                        if (k+l+str(counter)) not in columns:
                            columns.append(k+l+str(counter))
                        values.append(items[l])

                    counter += 1
        else:
            if j not in columns:
                columns.append(j)
            values.append(t[i][j])
    temp.append(values)

tdf = pd.DataFrame(temp,columns=columns)

The above code does the job but is suffering with the fundamental flaw that I am assuming the data received will always have the same location within the dictionary. Also while using the instrument_token of an instrument the quote doesn't return the name of the instrument, I had to manually do that. And when I stream data the format changes and I had to write another set of loops to handle that, which failed badly because of how dictionary works.

Please help me with this - I want to format the data into pandas for easier computation. If there is some other modules that help with the same please let me know about that, I will try my best to learn that module before asking here again.

2. I want to subscribe for the historical API as well, does that api provide data in nested list/dictionary format? I am having trouble bifurcating these format.