Hello, I’m not sure if this question belongs in this category.
I have been retrieving archived data from the fdsnws server (using obspy) from RS stations that show up on FDSN station meta data queries) for a phd project. I have successfully managed to download data for some stations going back to 2017. But for others I’m struggling to download the data even though they appear as available in the station metadata queries.
Also some stations that don’t show an end date, which would suggest they are still operating(?), in fact don’t even show up online (in Data view or Station view web apps) nor can I download the data for any period since their start date from the archive.
It seems to be happening with quite a few stations (i’ve listed them below) , while not for others, so I was wondering if some station data is removed from the RS servers or whether its a question of station name changes, as I read in the blogs, happened with R63B4. Is there a way to know what should be retrievable and what not?
Station |type| Start date | End date |
|R10BA|4D|2021-04-12|
|R44BB|4D|2021-04-12|
|R4640|1D|2021-10-28|
|R63B4|1D|2019-02-08|
|R6D71|4D|2021-04-12|
|R800E|4D|2021-04-12|
|R87DF|4D|2021-03-28|
|RA680|4D|2021-03-28|
|RA8B1|4D|2021-04-12|
|RAB15|4D|2021-04-12|
|RC14F|1D|2021-02-14|
|RC8CD|4D|2021-03-28|
|RCA2F|4D|2021-03-28|
|S2798|3D|2023-06-29|
That is correct; if the “End Date” parameter is missing, that means that stations should still transmit data to our network. I will ask our server team to investigate what is happening for the stations you have listed (thank you for compiling that!)
However, to quickly check what data is available or not, you can use DataView (https://dataview.raspberryshake.org/) to browse through our station list, go directly to the timeframe you are interested in, and check if data is present.
If a station doesn’t appear in the list when you search for it, you may need to click on the “Filters” button and enable the “Show Offline” slider.
Thanks for looking into it.
Just to add I did the check you suggested in dataview.
So all those stations listed with no end dates appear as offline and additionally no data shows up when navigating to dates where supposedly they ought to have.
just wondering if the server team got back regarding the missing data for the stations I mentioned?, I realise there is a still ongoing live data issue with some services (StationView, DataView, FDSNWS) . So i guess that won’t help.
fyi I know that my supervisor managed to download data from the following x3 stations (no longer streaming) back in 2020. So at some point the data for these was available.
Station |type| Start date | End date |
|R0887|4D|2017-11-26|2019-09-22|
|R9621|1D|2017-02-22|2019-06-11|
|R5982|1D|2017-04-14|2019-07-10|
As for the following 10x 4D stations (supposedly streaming) , I pinned them all down to the same location on station view in Greenwich in London, (whilst on the phone app they all appear spread out on a north south alignment across the River Thames, which suggests they are all linked to the same “user”?)
So just wondered if there was a simple answer like the data being removed on purpose? in which case I can put the issue to rest. But if there is a chance to download the data it would be worth getting to the bottom of it. The more data the better!
as of now, I have not received feedback on this particular issue, as, as you imagined, it is strictly correlated to the server-side outage we are still working on.
As for the following 10x 4D stations (supposedly streaming) , I pinned them all down to the same location on station view in Greenwich in London, (whilst on the phone app they all appear spread out on a north south alignment across the River Thames, which suggests they are all linked to the same “user”?)
If they are very close it is possible that they are arranged in some array for specific projects, as other Shakers have done that in the past.
So just wondered if there was a simple answer like the data being removed on purpose? in which case I can put the issue to rest. But if there is a chance to download the data it would be worth getting to the bottom of it.
You can rest assured that no data is deleted from our servers, so if it has been uploaded, it is there. Naturally, the discrepancy that you highlighted about the missing “End Date” parameter still needs to be checked out.
Thank you for providing further details about the stations you are interested in. I think our server team will take care of this as soon as our other priorities (re-establish full data access functionalities for everyone) have been completed. We apologize for the disruption and any inconvenience this outage may have caused.
Thanks for keeping us updated about the progress of the server transfer. It looks like it turned out to be a major overhaul of the system.
I noticed there was data now available from Jan 2022. So I have gone and tested if I can download data for the stations I had mentioned previously were unavailable.
Yes, the overhaul of our server infrastructure has been (and still is) a major one, but we’re surely glad to have reached the current milestone. And we are already working on the next one.
Regarding the list you shared in your post (thank you for it!), I am not sure why those stations are still displaying an open epoch while most of them are clearly offline. After checking, I can confirm that all stations in the list are currently offline, bar for:
Shake R800E, which is online,
Shake RAB15, which is online too,
Shake S2798, which is not online, but has data for 2023.
If you provide the date/times you were interested in, I can try to search a bit more. But if the FDSN service is answering with that message, it means that those Shakes have not sent data to our servers for the period of time you have entered.
I want to thank you for your notification. I have passed it on to our team so they can look at what’s needed to fix this apparent bug in our overall station metadata.
Checking in more detail, as far as I can see, and starting from today (all days are in Julian format):
Shake R800E has data for the last four days;
Shake RAB15 has data for the last 29 days;
Shake S2798 has data for 2 days in 2023 (180,203).
R10BA - no data between 2022 and today
R44BB - no data between 2022 and today
R4640 - no data between 2022 and today
R63B4 - no data between 2022 and today
R6D71 - has data for 2 days in 2024 (84,85)
R87DF - has data for 2 days in 2024 (84,85)
RA680 - no data between 2022 and today
RA8B1 - no data between 2022 and today
RC14F - no data between 2022 and today
RC8CD - no data between 2022 and today
RCA2F - no data between 2022 and today
So, most stations above did not transmit data to our servers in the available timeframe at the time of this message (January 1st, 2002, to June 14th, 2024).