FDSN dataselect server 'rate limiting' too strict?

alomaxnet · September 18, 2023, 2:43pm

It seems that since about a year or so, there is now server rate limiting for FDSN dataselect requests, e.g.

(https://fdsnws.raspberryshakedata.com/fdsnws/dataselect/1/query?network=AM&station=R1719&location=00&channel=EHZ&starttime=2022-01-25T20%3A32%3A08.000000Z&endtime=2022-01-25T20%3A34%3A48.000000Z)

can result in error like:

FDSNException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.

I have modified my codes to automatically increase the wait time between requests, but it seems that the wait time needs to be many tens of seconds to succeed in fulfilling requests. As I am trying use RS data for research, and may request 100’s or 1000’s of data windows, such rate limiting makes large requests very slow to impractical.

Can you confirm the minimum wait time set on the server?
If this wait time cannot be reduced (e.g. to 1 sec or less), can you suggest other request mechanisms for large data sets?

Thanks,
Anthony Lomax

Stormchaser · September 19, 2023, 9:11am

Hello Anthony, and welcome to our community!

Thank you for taking the time to write to us regarding this element of our ecosystem. I have passed all the information you’ve provided to our management team, who are looking into it as I write.

Support of scientific research is one of our main pillars, so I’m sure that a solution will be found. I’ll update you as soon as possible.

Thank you for your patience and understanding.

alomaxnet · September 19, 2023, 9:26am

Thanks for your rapid reply.
I have a bit more information. I think that many of my requests were for data windows that were not available:

WARNING: Skipping channel: AM_R8EFE_00_EHZ : FDSNException: No data available for request.

Including these warnings, the needed request wait time may be much less than 10’s of seconds.

I make many requests for data that ends up being unavailable, despite checking first start and end times in the ObsPy inventory. I suppose that these start and end times do not reflect the true availability, perhaps when the station/channel data was intermittent. Maybe the fundamental problem is reasonable limitations of the FDSN protocol or ObsPy with respect to completely listing all granular start and end times. But this means one can make many requests for unavailable data, which collides with the rate limiting feature.

Thanks,
Anthony

branden · September 21, 2023, 3:46pm

Great to see you on the forum @alomaxnet

We are working on a solution for this. Stay tuned.

branden

sheeny · July 26, 2024, 1:49am

Is there any news on this?

I haven’t been hit with this before, but got it first thing today. I don’t think I made any requests yesterday and got these messages straight up today…

Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:28:38) [MSC v.1929 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 8.13.2 -- An enhanced Interactive Python.

runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:191
    trace0 = rs.get_waveforms('AM', stn, '00', ch, start, end)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:




runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:130
    inv = rs.get_stations(network=nw, station=stn, level='RESP')  # get the instrument response

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:748 in get_stations
    data_stream = self._download(url)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1846 in raise_on_error
    raise FDSNBadGatewayException("Service responds: Bad gateway ",

FDSNBadGatewayException: Service responds: Bad gateway 
HTTP Status code: 502
Detailed response of server:




runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:191
    trace0 = rs.get_waveforms('AM', stn, '00', ch, start, end)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:




runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:191
    trace0 = rs.get_waveforms('AM', stn, '00', ch, start, end)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:

Thanks,

Al.

Stormchaser · July 26, 2024, 9:35am

Hello Al,

I think this is related to the server issues we had during the (UTC) night, which we are solving as I write. It’s possible that it’s a false flag because of that, as I see error 502 mixed with error 429.

However, if it happens again, please do let us know. Thank you as usual!

sheeny · August 2, 2024, 11:26pm

It looks like this is back again:

runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:191
    trace0 = rs.get_waveforms('AM', stn, '00', ch, start, end)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:




runfile('D:/Documents/Python/QR13any.py', wdir='D:/Documents/Python')
Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\qr13any.py:191
    trace0 = rs.get_waveforms('AM', stn, '00', ch, start, end)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:

Al.

sheeny · August 3, 2024, 5:01am

Seems OK again by 05:00 UTC.

Al.

Stormchaser · August 5, 2024, 7:59pm

Thank you for the continued reporting as usual sheeny!

I don’t think I have seen a server issue by that time, but I may be wrong, so the server team is checking if that was the source of the message you were seeing or not.

sheeny · August 7, 2024, 11:44pm

I am getting that message again today,

Al.

Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:28:38) [MSC v.1929 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 8.13.2 -- An enhanced Interactive Python.

runfile('D:/Documents/Python/LocalStns9.py', wdir='D:/Documents/Python')
d:\documents\python\localstns9.py:250: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed two minor releases later. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap(obj)`` instead.
  cmap = get_cmap('Paired', lut=12)
['#1f78b4', '#33a02c', '#e31a1c', '#ff7f00', '#6a3d9a', '#a6cee3', '#b2df8a', '#fb9a99', '#fdbf6f', '#cab2d6']
Station R21C0 (Raspberry Shake Citizen Science Station)
	Station Code: R21C0
	Channel Count: None/None (Selected/Total)
	2022-02-12T09:46:05.109000Z - 
	Access: open 
	Latitude: -33.7658, Longitude: 149.8820, Elevation: 1114.0 m
	Available Channels:
	  .00.HDF       100.0 Hz  2022-02-12 to None
	  .00.EHZ       100.0 Hz  2022-02-12 to None

Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\localstns9.py:277
    buildStream(st, output, rplots)

  File d:\documents\python\localstns9.py:131 in buildStream
    trace = rs.get_waveforms('AM', station=slist[i], location = "00", channel = '*HZ', starttime = start, endtime = end) #vertical geophone could be EHZ or SHZ

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:




runfile('D:/Documents/Python/LocalStns9.py', wdir='D:/Documents/Python')
d:\documents\python\localstns9.py:250: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed two minor releases later. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap(obj)`` instead.
  cmap = get_cmap('Paired', lut=12)
['#1f78b4', '#33a02c', '#e31a1c', '#ff7f00', '#6a3d9a', '#a6cee3', '#b2df8a', '#fb9a99', '#fdbf6f', '#cab2d6']
Station R21C0 (Raspberry Shake Citizen Science Station)
	Station Code: R21C0
	Channel Count: None/None (Selected/Total)
	2022-02-12T09:46:05.109000Z - 
	Access: open 
	Latitude: -33.7658, Longitude: 149.8820, Elevation: 1114.0 m
	Available Channels:
	  .00.HDF       100.0 Hz  2022-02-12 to None
	  .00.EHZ       100.0 Hz  2022-02-12 to None

Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\localstns9.py:277
    buildStream(st, output, rplots)

  File d:\documents\python\localstns9.py:131 in buildStream
    trace = rs.get_waveforms('AM', station=slist[i], location = "00", channel = '*HZ', starttime = start, endtime = end) #vertical geophone could be EHZ or SHZ

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:




runfile('D:/Documents/Python/LocalStns9.py', wdir='D:/Documents/Python')
d:\documents\python\localstns9.py:250: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed two minor releases later. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap(obj)`` instead.
  cmap = get_cmap('Paired', lut=12)
['#1f78b4', '#33a02c', '#e31a1c', '#ff7f00', '#6a3d9a', '#a6cee3', '#b2df8a', '#fb9a99', '#fdbf6f', '#cab2d6']
Station R21C0 (Raspberry Shake Citizen Science Station)
	Station Code: R21C0
	Channel Count: None/None (Selected/Total)
	2022-02-12T09:46:05.109000Z - 
	Access: open 
	Latitude: -33.7658, Longitude: 149.8820, Elevation: 1114.0 m
	Available Channels:
	  .00.HDF       100.0 Hz  2022-02-12 to None
	  .00.EHZ       100.0 Hz  2022-02-12 to None

Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\localstns9.py:277
    buildStream(st, output, rplots)

  File d:\documents\python\localstns9.py:131 in buildStream
    trace = rs.get_waveforms('AM', station=slist[i], location = "00", channel = '*HZ', starttime = start, endtime = end) #vertical geophone could be EHZ or SHZ

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:




runfile('D:/Documents/Python/LocalStns9.py', wdir='D:/Documents/Python')
d:\documents\python\localstns9.py:250: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed two minor releases later. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap(obj)`` instead.
  cmap = get_cmap('Paired', lut=12)
['#1f78b4', '#33a02c', '#e31a1c', '#ff7f00', '#6a3d9a', '#a6cee3', '#b2df8a', '#fb9a99', '#fdbf6f', '#cab2d6']
Station R21C0 (Raspberry Shake Citizen Science Station)
	Station Code: R21C0
	Channel Count: None/None (Selected/Total)
	2022-02-12T09:46:05.109000Z - 
	Access: open 
	Latitude: -33.7658, Longitude: 149.8820, Elevation: 1114.0 m
	Available Channels:
	  .00.HDF       100.0 Hz  2022-02-12 to None
	  .00.EHZ       100.0 Hz  2022-02-12 to None

Traceback (most recent call last):

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File d:\documents\python\localstns9.py:277
    buildStream(st, output, rplots)

  File d:\documents\python\localstns9.py:131 in buildStream
    trace = rs.get_waveforms('AM', station=slist[i], location = "00", channel = '*HZ', starttime = start, endtime = end) #vertical geophone could be EHZ or SHZ

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:872 in get_waveforms
    data_stream = self._download(url, use_gzip=False)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1486 in _download
    raise_on_error(code, data)

  File ~\anaconda3\envs\ObspyCartopy\lib\site-packages\obspy\clients\fdsn\client.py:1838 in raise_on_error
    raise FDSNTooManyRequestsException(msg, server_info)

FDSNTooManyRequestsException: Sent too many requests in a given amount of time ('rate limiting'). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:

Stormchaser · August 9, 2024, 7:06pm

Hello sheeny,

I have received an update from our team, and Errors 429 are related to the fact that, in all likelihood, you made a lot of data queries in a short amount of time, triggering them.

As it is written in the error code: Sent too many requests in a given amount of time. For more information about our FDSNWS traffic optimization, you can check this page in the manual: FDSN Web Services

You can avoid this Error 429 by adding a slight delay between your queries (I use 5 seconds, for example), and you should be good to go again.

sheeny · August 10, 2024, 12:58am

G’Day Stormchaser,

These errors occurred from my first attempt of the day. I hadn’t tried to access the data for at least 8-10 hours.

I haven’t had this error appear when I’ve been processing data i.e. I don’t think I’ve ever actually exceeded the trigger frequency for the error.

It has only occurred first thing in the morning or maybe I’ve made one successful data call.

Unfortunately, when I copy the text from the console it doesn’t copy the numbered console prompt. In future I’ll make sure I add a screenshot so the numbered console prompt at least is visible.

Al.

Stormchaser · August 11, 2024, 11:35am

Yes, that would be helpful, thank you!

Also, when it appears again, could you also detail the data request you are making, so that I can use the same code as you are, and try to replicate the event on my PC?

Thanks again.