Station flatlining

Hi guys, my station AM.R271A is misbehaving since this morning. It’s in an inaccessible location and only accessible by reverse SSH’ing into the pi. When I first connected to it the pi was sitting completely idle, didn’t seem to have any docker activity going at all - so something apparently crashed.

I opted for a reboot and it came back for about an hour and a half, but since then is again been missing data. I managed to get the logs out of /opt/log but don’t really know what I’m looking for in there - here’s a link to the logs if anyone can help decipher what’s happening?

Hello zeedoktor,

Thank you for posting the logs from your Shake. From what I can see from our data, there seem to have been issues between Feb. 12th and 13th, but now everything appears back to normal, with constant live transmission: RS StationView

The only issue listed in the logs was a loss of contact between the Shake and the time synchronization NTP server, which has likely caused the data gaps you have noticed. The reboot seems to have put everything back normal.

If this problem appears again, feel free to write here on the forum!

Ok thanks - the odd thing is I’m uploading the data to an external server every 10 minutes and it looks like that data was corrupt - all I got was flatline data. See https://nciassets.inside.net/shakeandsway/R271A/2022/02/14/2022-02-14_R271A_EHZ_VEL_dayplot.png for Feb 14. Oddly, even though the data had come back for the live feed (the link of which you posted) the data remained corrupt for the remainder of Feb 14.

Guess my shake’s not one for love. :slight_smile:

It is indeed curious since I can see normal data (and not flat) for Feb. 14th on our DataView service: RS DataView BETA

Is it possible that the data you uploaded to your external server has been corrupted somehow, thus creating that representation? In any case, except for the interruptions that I saw between the 13th and the start of the 14th, everything looks good on our servers, so if you need, you can re-download the data from these days from there.

Thanks - I’m not overly concerned with data continuity from this station. I was mostly wondering whether there’s a resilience problem to file-write errors with the miniseed files. If you want I can make the data file available so you can inspect it? I’m using just the standard obspy framework to read the files, and don’t really have time at the moment to dig into this. But: Happy to share the data file if you want to take a look.

This problem is back - I hard rebooted the shake an hour ago, still only uploads flatline miniseed data.

Hello zeedoktor,

I’m sorry to hear that the problem with flat data appears to be back. Could you please send a screenshot of what you can see together with the logs from the Shake?

Because, as it had happened before, on DataView I am able to see normal (not flat) data being stored on our servers (image attached).

I saw that the data seems fine in the online data viewer, but the miniseed file being interpreted serverside just flatlines. At midnight UTC (once a new miniseed file is being written) this recovers. Looks like this is a miniseed file corruption issue. Here’s the miniseed file if you want to have a look:

and the associated plot:

On the plot you can see there’s missing data around 22:30, that’s when I hard rebooted the shake. I’d expect data to be OK from after that time.

While creating the plot, obspy complains:
/home/ubuntu/miniconda3/envs/obspy/lib/python3.7/site-packages/obspy/io/mseed/headers.py:828: InternalMSEEDWarning: readMSEEDBuffer(): Not a SEED record. Will skip bytes 19750016 to 19750143.

Which repeats for a half a dozen records or so. The code being run where this happens:

st = obspy.read(tracefile)

Doesn’t look like there’s much that can be passed to the read function in terms of fault tolerance options.

Hello @zeedoktor

Just opened that same file in PQL. Works fine. So this is definitely a software issue, not a file issue.

Notice the gaps (yellow flags) and overlap (red flag). Your script is not recovering from there being a gap in the data …

branden

Thanks Branden, that is interesting to see. Could this be due to write errors on the SD card? I’ve been pondering replacing the samsung SD card with a swissbit industrial grade one, I’ve had great success doing that with other devices.

However, recovery wise when that happens again: Is anyone else here using the obspy framework and has a suggestion on how to make obspy.core.read more resilient to data flubs? There do not appear to be any options on what to do when read errors are encountered.

Good evening @zeedoktor

You ask, “Could this be due to write errors on the SD card?” In your question, what is “this” referring to? Thank you!

branden

“this” refers to the miniseed file corruption. The question I guess is: the data you see that’s online, did that save to disk first before being uploaded to your servers or does that come straight from the sensors? If the latter, it’s not file corruption but sensor data corruption. Not sure what would cause that.

Good morning @zeedoktor

You uploaded a file to this issue- the same file that when you read it in with your Python script shows up flat-lined.

I downloaded the file.

I viewed it with PQL. All works- no flat-lined data.

The problem, therefore, is your script, not the file.

branden

Ah ok - I thought my shake was also uploading to your network and that’s where you got the data from. So it could still be a corrupt sd card. thanks for looking at it with a different tool. It’s doing it again today. I really need to address this… need to go find an obspy support forum.

@zeedoktor

maybe upload the code here and ask the community for an audit. There are a lot of Shakers on this forum who navigate Python and ObsPy well.

branden