I am trying to fetch historical continuous data of multiple instruments over a period. The initial fetch went through perfect. After a while, when I increased the number of workers, it started giving out 504 errors. Timeout on my HTTP client is 10 seconds.
Rate limit is fine. Give a proper http error (429) and please mention the api rate limits in the docs so that devs can plan accordingly
If you are pulling data continuously and getting 504 then you can add some delay and fetch it again. It is possible that a request would have failed by the time DB fetch is completed.
All the rate limits are based on the api_key. I am afraid we can't provide data in the file dump. You will have to fetch it as JSON and convert it at your end.