Hi guys, my station AM.R271A is misbehaving since this morning. It’s in an inaccessible location and only accessible by reverse SSH’ing into the pi. When I first connected to it the pi was sitting completely idle, didn’t seem to have any docker activity going at all - so something apparently crashed.
I opted for a reboot and it came back for about an hour and a half, but since then is again been missing data. I managed to get the logs out of /opt/log but don’t really know what I’m looking for in there - here’s a link to the logs if anyone can help decipher what’s happening?
Thank you for posting the logs from your Shake. From what I can see from our data, there seem to have been issues between Feb. 12th and 13th, but now everything appears back to normal, with constant live transmission: RS StationView
The only issue listed in the logs was a loss of contact between the Shake and the time synchronization NTP server, which has likely caused the data gaps you have noticed. The reboot seems to have put everything back normal.
If this problem appears again, feel free to write here on the forum!
Ok thanks - the odd thing is I’m uploading the data to an external server every 10 minutes and it looks like that data was corrupt - all I got was flatline data. See https://nciassets.inside.net/shakeandsway/R271A/2022/02/14/2022-02-14_R271A_EHZ_VEL_dayplot.png for Feb 14. Oddly, even though the data had come back for the live feed (the link of which you posted) the data remained corrupt for the remainder of Feb 14.
It is indeed curious since I can see normal data (and not flat) for Feb. 14th on our DataView service: RS DataView BETA
Is it possible that the data you uploaded to your external server has been corrupted somehow, thus creating that representation? In any case, except for the interruptions that I saw between the 13th and the start of the 14th, everything looks good on our servers, so if you need, you can re-download the data from these days from there.
Thanks - I’m not overly concerned with data continuity from this station. I was mostly wondering whether there’s a resilience problem to file-write errors with the miniseed files. If you want I can make the data file available so you can inspect it? I’m using just the standard obspy framework to read the files, and don’t really have time at the moment to dig into this. But: Happy to share the data file if you want to take a look.
I’m sorry to hear that the problem with flat data appears to be back. Could you please send a screenshot of what you can see together with the logs from the Shake?
Because, as it had happened before, on DataView I am able to see normal (not flat) data being stored on our servers (image attached).
I saw that the data seems fine in the online data viewer, but the miniseed file being interpreted serverside just flatlines. At midnight UTC (once a new miniseed file is being written) this recovers. Looks like this is a miniseed file corruption issue. Here’s the miniseed file if you want to have a look:
and the associated plot:
On the plot you can see there’s missing data around 22:30, that’s when I hard rebooted the shake. I’d expect data to be OK from after that time.
While creating the plot, obspy complains:
/home/ubuntu/miniconda3/envs/obspy/lib/python3.7/site-packages/obspy/io/mseed/headers.py:828: InternalMSEEDWarning: readMSEEDBuffer(): Not a SEED record. Will skip bytes 19750016 to 19750143.
Which repeats for a half a dozen records or so. The code being run where this happens:
st = obspy.read(tracefile)
Doesn’t look like there’s much that can be passed to the read function in terms of fault tolerance options.
Thanks Branden, that is interesting to see. Could this be due to write errors on the SD card? I’ve been pondering replacing the samsung SD card with a swissbit industrial grade one, I’ve had great success doing that with other devices.
However, recovery wise when that happens again: Is anyone else here using the obspy framework and has a suggestion on how to make obspy.core.read more resilient to data flubs? There do not appear to be any options on what to do when read errors are encountered.
“this” refers to the miniseed file corruption. The question I guess is: the data you see that’s online, did that save to disk first before being uploaded to your servers or does that come straight from the sensors? If the latter, it’s not file corruption but sensor data corruption. Not sure what would cause that.
Ah ok - I thought my shake was also uploading to your network and that’s where you got the data from. So it could still be a corrupt sd card. thanks for looking at it with a different tool. It’s doing it again today. I really need to address this… need to go find an obspy support forum.
