Gaps in data from server

Technical Support
1

I was pleased to learn about how to obtain and view data in Python using ObsPy:

from obspy.clients.fdsn import Client
from obspy import UTCDateTime

client = Client(base_url='https://fdsnws.raspberryshakedata.com/')
starttime = UTCDateTime("2019-05-06 21:00:01")
endtime = starttime + 60*60*3
wave1= client.get_waveforms('AM', 'RF7DC', '00', 'SHZ', starttime, endtime)
wave1.plot(type='dayplot',outfile='190506_21H_RF7DC.png')

190506_21H_RF7DC
190506_21H_RF7DC800Ă—600 67.2 KB

however I notice the output shows a number of gaps in the data. These gaps are not present in the helicorder GIF stored locally on the device, so I assume my internet link must have been dropping out at those times. I have the log files but trying to upload the .zip I get the message “Sorry, new users cannot upload attachments” :frowning:

Since some history is stored locally on the device, would it make sense for the R-Shake to play “catch-up” once the network connection comes back online (at least for some limited time range), so the data is not gone forever?

2

@jbeale—sorry about that. You should be able to attach the logs now I think.

@ivor will field this one when he’s free since he makes our data policy decisions.

3

Thanks- log files attached here. Occasional network outages are a fact of life on most residential connections, so if the network storage is to be useful for any analysis, it seems worthwhile to fill in the gaps when possible and not unduly burdensome.

log.tar.zip (277.3 KB)

4

hello,

there are likely two things going on here: 1) local network outages, and 2) server-side issues.

for the first problem, the system has been designed to queue messages locally when a connection to the server is lost (but not so much that the Pi will run out of memory), and when connection to the server is re-established, all queued messages are sent to the server. depending on the length of the outage, gaps may be non-existent (short outage), to small (medium), to extensive (long). in this case, memory is the defining resource.

for the second problem, there is a known issue that is being investigated and will be addressed, intent to resolve is sooner than later. it turns out one learns a lot about computing when creating a global network of instruments delivering real-time data 24 hours / day, where the number of units is ever-increasing.

apologies for the inconvenience. once a remedy has been applied server-side, and its efficacy confirmed, i will report back to this post.

cheers,

richard

1 Like