FDSNException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
I have modified my codes to automatically increase the wait time between requests, but it seems that the wait time needs to be many tens of seconds to succeed in fulfilling requests. As I am trying use RS data for research, and may request 100’s or 1000’s of data windows, such rate limiting makes large requests very slow to impractical.
Can you confirm the minimum wait time set on the server?
If this wait time cannot be reduced (e.g. to 1 sec or less), can you suggest other request mechanisms for large data sets?
Thank you for taking the time to write to us regarding this element of our ecosystem. I have passed all the information you’ve provided to our management team, who are looking into it as I write.
Support of scientific research is one of our main pillars, so I’m sure that a solution will be found. I’ll update you as soon as possible.
Thanks for your rapid reply.
I have a bit more information. I think that many of my requests were for data windows that were not available:
WARNING: Skipping channel: AM_R8EFE_00_EHZ : FDSNException: No data available for request.
Including these warnings, the needed request wait time may be much less than 10’s of seconds.
I make many requests for data that ends up being unavailable, despite checking first start and end times in the ObsPy inventory. I suppose that these start and end times do not reflect the true availability, perhaps when the station/channel data was intermittent. Maybe the fundamental problem is reasonable limitations of the FDSN protocol or ObsPy with respect to completely listing all granular start and end times. But this means one can make many requests for unavailable data, which collides with the rate limiting feature.