Hi, For currency options symbol USDINR20FEB71.5PE, there was a spurious wrong tick data at 13:28:46, showing a 300% jump from previous tick price, from 0.2075 to 0.7925. The next tick was again at 0.2075.
This wrong tick data caused the algo to exit positions incorrectly.
No, there is nothing wrong on our side; the streaming system is well tested and has been working fine for years. We've never seen such an error till now.
Pls look into this and let me know if there are any insights.
Hi, Did a deep-dive into this. There were 206 data tick errors in currency data for Feb20 (from 01Feb to 20Feb). On further analysis, looks like the data gets shifted by one or more places in the data array, thereby messing up the values. Issue is seen across quote and full mode data.
While the error rate is insignificant (like 206 errors in 8090993 records), it is still non-zero.
Note that this error can't happen at our end - the JSON comes with instrument token as well as the value and is simply unpacked and stored. If there is any issue in this simple process, it will show up in every record, not just in 0.006% of the records.
Such data errors are seen in 2019 data as well, but error rates were less than that of 2020.
Appreciate if you can look at this. It is critical that the data has zero errors.
@ramatius We stream ticks, as we receive it from the exchange. We haven't received the above-misaligned tick at our charting system. Anyhow,we are logging all ticks for above contract USDINR20FEB71.5PE for today, to see if any discrepancy. Do let us know, when you encounter above discrepancy again with DateTime and instrument_name.
Thanks @rakeshr The error rate is like 206 errors in 8090993 records, captured over 20 days across 146 currency symbols. So you need to check across all currency option symbols over a few days.
Since we haven't observed this scenario, we have started storing ticks dump for yesterday and today for USDINR20FEB71.5PE. Can you run the same and let us know the data points where you see the irregularities?
Did a deep-dive into this. There were 206 data tick errors in currency data for Feb20 (from 01Feb to 20Feb). On further analysis, looks like the data gets shifted by one or more places in the data array, thereby messing up the values. Issue is seen across quote and full mode data.
While the error rate is insignificant (like 206 errors in 8090993 records), it is still non-zero.
Note that this error can't happen at our end - the JSON comes with instrument token as well as the value and is simply unpacked and stored. If there is any issue in this simple process, it will show up in every record, not just in 0.006% of the records.
Such data errors are seen in 2019 data as well, but error rates were less than that of 2020.
Appreciate if you can look at this. It is critical that the data has zero errors.
Regards,
Ramakrishnan S
quantindia.in
We stream ticks, as we receive it from the exchange. We haven't received the above-misaligned tick at our charting system.
Anyhow,we are logging all ticks for above contract
USDINR20FEB71.5PEfor today, to see if any discrepancy.Do let us know, when you encounter above discrepancy again with DateTime and instrument_name.
Here is one log file that shows the full details of the errors seen - quantindia.in/misc_afjhauweroiuafdlkjqew/check_data_for_errors_2020_02_22_21_51_15.log
The errors were corrected iteratively. Manual corrections were made when auto-correction was not possible.
Hope this helps in zero-ing in on the problem.
Regards,
Ramakrishnan S
quantindia.in