Requesting more than 12h of data from the fdsnws

Hello,

I have noted the usage rules on the FDSN dedicated page: Raspberry Shake FDSNWS - DataSelect limit downloads to 12h per request per 100hz channel.

Is this new? Until last sunday 9th of June I was being able to download a full day of data for several stations on a daily basis.

Will it go back to a more relaxed limits at some point once the server issues are resolved ? or is this permanent ?

It can be a bit unpractical for users wanting to download several days of data at a time for extensive data analysis.

2 Likes

I have modified my code to download by half days. My code would loop for each station and channel and day and download a days worth. So now I just put two consecutive requests for each half day, one after the other within the loop. Instead of one.

I still get an error for some stations relating to ‘rate limiting’

An exception occurred: Sent too many requests in a given amount of time (‘rate limiting’). Wait before making a new request.
HTTP Status code: 429
Detailed response of server:

How do I avoid this ? Is it because of the consecutive requests?

my code:

for day in pbar:
        for count, station in enumerate(STATIONS):
                for channel in CHANNELS:

                          # first half of day
                          st  = c.get_waveforms(network, station, location, channel, starttime = 
UTCDateTime(day), endtime = UTCDateTime(day)+43200)
                          
                          #second half of the day
                          st += c.get_waveforms(network, station, location, channel, starttime = UTCDateTime(day)+43200, endtime = UTCDateTime(day)+86400)
                          
                          st.write(fn,format="MSEED")
2 Likes

Hello mdpw,

As far as I know, the team is still working on the new server infrastructure optimization, so it is possible that the specific limit (12h per request per 100hz channel) will change in the future. Nonetheless, I have passed your feedback on to them so that they can take it into account.

Regarding your second message, yes, I think that is because there were too many consecutive requests in a short amount of time, and the server limited you as a consequence. I think putting a small (5-10 seconds should probably be enough) delay between data downloads will allow you to complete your bulk download without any more issues.

Again, this is all precious feedback, so rest assured that I’ll pass everything on to the team. If we change anything, we will communicate it promptly, as usual.

Thank you again!

1 Like

thanks! I’ll try that for the time being.

2 Likes

Hello mdpw,

To update you as promised, we are now upgrading our query capabilities to be able to accommodate entire 24h data requests from FDNSWS, without the need to split it into two 12h tranches.

When you have the chance, could you try to see if your original download program works as intended now?

Thank you for your patience.

I can download 24h worth of data at a time now , thanks!

2 Likes

No problem at all mdpw!