FDSN dataselect server 'rate limiting' too strict?

Developers
1

It seems that since about a year or so, there is now server rate limiting for FDSN dataselect requests, e.g.

(https://fdsnws.raspberryshakedata.com/fdsnws/dataselect/1/query?network=AM&station=R1719&location=00&channel=EHZ&starttime=2022-01-25T20%3A32%3A08.000000Z&endtime=2022-01-25T20%3A34%3A48.000000Z)

can result in error like:

FDSNException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.

I have modified my codes to automatically increase the wait time between requests, but it seems that the wait time needs to be many tens of seconds to succeed in fulfilling requests. As I am trying use RS data for research, and may request 100’s or 1000’s of data windows, such rate limiting makes large requests very slow to impractical.

Can you confirm the minimum wait time set on the server?
If this wait time cannot be reduced (e.g. to 1 sec or less), can you suggest other request mechanisms for large data sets?

Thanks,
Anthony Lomax

2 Likes
2

Hello Anthony, and welcome to our community!

Thank you for taking the time to write to us regarding this element of our ecosystem. I have passed all the information you’ve provided to our management team, who are looking into it as I write.

Support of scientific research is one of our main pillars, so I’m sure that a solution will be found. I’ll update you as soon as possible.

Thank you for your patience and understanding.

3

Thanks for your rapid reply.
I have a bit more information. I think that many of my requests were for data windows that were not available:

WARNING: Skipping channel: AM_R8EFE_00_EHZ : FDSNException: No data available for request.

Including these warnings, the needed request wait time may be much less than 10’s of seconds.

I make many requests for data that ends up being unavailable, despite checking first start and end times in the ObsPy inventory. I suppose that these start and end times do not reflect the true availability, perhaps when the station/channel data was intermittent. Maybe the fundamental problem is reasonable limitations of the FDSN protocol or ObsPy with respect to completely listing all granular start and end times. But this means one can make many requests for unavailable data, which collides with the rate limiting feature.

Thanks,
Anthony

2 Likes
4

Great to see you on the forum @alomaxnet

We are working on a solution for this. Stay tuned.

branden

1 Like
5

Is there any news on this?

I haven’t been hit with this before, but got it first thing today. I don’t think I made any requests yesterday and got these messages straight up today…

Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:28:38) [MSC v.1929 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 8.13.2 -- An enhanced Interactive Python.

runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:191
    trace0 = rs.get_waveforms('AM', stn, '00', ch, start, end)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:




runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:130
    inv = rs.get_stations(network=nw, station=stn, level='RESP')  # get the instrument response

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:748 in get_stations
    data_stream = self._download(url)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1846 in raise_on_error
    raise FDSNBadGatewayException("Service responds: Bad gateway ",

FDSNBadGatewayException: Service responds: Bad gateway 
HTTP Status code: 502
Detailed response of server:




runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:191
    trace0 = rs.get_waveforms('AM', stn, '00', ch, start, end)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:




runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:191
    trace0 = rs.get_waveforms('AM', stn, '00', ch, start, end)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:

Thanks,

Al.

1 Like
6

Hello Al,

I think this is related to the server issues we had during the (UTC) night, which we are solving as I write. It’s possible that it’s a false flag because of that, as I see error 502 mixed with error 429.

However, if it happens again, please do let us know. Thank you as usual!

1 Like