Two shakes, a RSD3 running .20, and a RS4D running .21, randomly stop reporting data on StationView. They do, however, continue to produce data locally, and both show active server connections. The co-located Boom running .21 does not exhibit this behavior.
All three machines have active NTP sync, albeit to differing pools of servers.
I note the machines are uploading via a Starlink connection that is subject to random interruptions, but I’d expect that the links would recover, particularly since the internet and connection statuses are showing as good.
Any hints on where to start attacking this problem would be appreciated. Nothing glaringly obvious jumps out of the logs.
Could you post the logs from all three instruments when you can? Having one that doesn’t show these transmission issues could give us a clue for the other two.
Sure thing! Should be attached, upload gods willing.
A few things:
The 3D that fell off StationView magically showed up again. This is the same 3D that’s steadfastly refusing to update to .21 that’s reported in another ticket, so I won’t upload the same logs over on that ticket.
The 4D .21 splash page indicated that services were down. I’ve attached logs taken at that time, and then again after reboot.
The Boom log is for the instrument that is always happy with life
Thank you for all the logs wiley42, they were very informative!
Indeed, the RBOOM is just coasting through its great life; the others should take an example!
Going into what I’ve found:
all Shakes display (every now and then) a disconnection from the NTP time synchronization server. This doesn’t usually last long, and the units always reconnect to the server, so it could be related to the Starlink connection (with all three Shakes together)
other than this, the RBOOM is fine
however, both the RS3D and the RS4D show some file corruption from the logs. And for the 3D, this is likely the reason why the Shake doesn’t manage to update on its own
Thus, for both the 3D and the 4D, I recommend re-burning the microSD card. You can find instructions on how to do so here.
If the microSD cards continue to show inconsistent behavior, then it may be worth using different ones.
Once the reburning is done, simply reinsert the cards into the Shakes, turn them on, and then everything should be fine now.
I have local stratum-one NTP servers, so I’ll just whack all three of them to point to one of those rather than whatever pool they’re pointing to by default. I’ll need to pull the 3D and 4D out of their vault and I’m traveling most of next week, so I’l update my observations sometime the week of the 8th. Hopefully a couple of new mlc cards and refreshed images will make them as happy as the RBOOM
Replying to my own message is bad form but, well, here I go.
Turns out the 4D eventually decided to report that local services were down. Logs are attached (hopefully). RSH.R4EFC.2026-02-15T18_56_21.logs.tar (2.9 MB)
This (Monday) UTC morning I’m seeing all three units not transmitting live data, so I don’t know if you have temporarily pulled them or if you’re having an outage.
However, while I can confirm (as you have seen) that the 4D has been correctly updated to v0.21.1, the logs still show continued NTP errors such as the one below:
2026 046 10:38:34>> Time adjustment M0: HARD RESET. This will result in a one-time time-tear.
The average is approximately once every 1-2 minutes, so the NTP connection (or the overall internet connection) behaves quite erratically, and the result is stations dropping out of StationView.
If possible, it could be interesting to see what happens using a standard internet connection (non-Starlink one) to see if we can isolate the possible cause.
Also, if you can attach the logs from the other two instruments you have, I can confirm if this is the reason why all three are not reporting, or if there are other factors at play.
I’m increasingly skeptical that loss of NTP sync explains the issue, given that all three units are on the same switch, with the same backhaul, to the same switch, to the same Starlink terminal. It also doesn’t explain why the 4D’s local status page occasionally reports that services have died.
If I had an alternative, I wouldn’t be using Starlink
Now, on the NTP front, I did observe that the Rboom was configured to use a different pool than the 3D and 4D. However, to factor this mess out entirely, all three are now talking to a pair of stratum one servers installed on the same switch as the Shakes; all three have sync with the local stratum-one servers as well as various pool servers. Currently the 3D and 4D are reporting data locally and claim connections to the server, but are not reporting on Station View; the Boom is showing a server connection and is reporting on Station View, but is showing nothing in the local activity preview…
I have been monitoring your Shakes for the past 24 hours or so, trying to gather more information about your situation:
The Boom continued without issues, as far as I could see
The 3D appears to be behind real-time, but is slowly catching up
The 4D, instead, has transmitted for a bit, and then seemingly stopped
So, overall, at the logs snapshot, the Shakes were connected to the server and could communicate with it. I have asked our server team to check things on our side, and I’ll let you know if they find anything.
In the meantime, could I ask you to SSH into the Shake and execute these commands without rebooting the instruments or your Starlink router?
SHM(0) .GPS. 0 l 10 16 377 0.000 408.520 1.569
SHM(1) .PPS. 0 l - 16 0 0.000 0.000 0.000
*tick.mainecoon. .PPS. 1 u 170 512 377 0.292 0.992 2.624
+tock.mainecoon. .PPS. 1 u 541 512 377 0.274 -0.822 1.742
+usscz2-ntp-002. .GPSs. 1 u 60 64 377 22.750 2.338 8.135
At the moment, all three are up, all three are finding each other on the splash screen, but only the Boom’s data is being reported by Station View. Rebooting the 3D/4D will result in their data being reported (despite an NTP solution not yet being computed), but eventually they fall off. It’s a bit like the wire protocol doesn’t recover nicely from network interruptions.
Thank you for all the command-line reports, wiley42.
Our server team has checked things on our side, and I can report that there are no server-side issues at the moment. So, we need to focus on why the connection is behaving in this way.
As reflected client-side (your logs), our servers get many data transmission timeouts that can also be seen from all the broken pipeline lines in the logs. This means that the Shake is actively trying to transmit, but then the interruptions you’ve seen happen.
Let’s try this test by checking what happens when the Shakes are connectet singularly:
Disconnect the 3D and 4D, and leave the Boom on for 24h
Then disconnect it, and reconnect only the 3D for 24h
And, last, disconnect that and connect only the 4D for 24h
This is to check if the data volume that the three instruments are trying to update is behind what we are seeing. We already know that the Boom works fine, but it will be interesting to see the result of the experiment above.
Additionally, another element you can check (if you haven’t already) is this.
Yep; interruptions are expected; the vault is located in mountainous terrain with visibility to the north partially obscured by trees and rising terrain, which means transient drop-outs, up to several minutes in duration, are normal. There’s a 40 meter tower waiting to be erected that should get the terminal above the trees (if only just), but there’s not much we can do about terrain but hope that build-out of the constellation eventually fills the gaps.
Currently, everything in the vault is down thanks to storms and the UPS eventually running dry. Everything is under two meters of snow, but clearing is expected soon, at which point we’ll excavate things and get back on the air – at which point we’ll run the proposed experiment.
While the Shakes are backhauling via Starlink, it’s not “naked” Starlink; they live on a RFC1918 network with a firewall at the edge that builds a tunnel via Starlink to a data center in a more -er- civilized part of the world. This gives us a way around the sometimes unexpected behavior of the Starlink terminal, and allows us to have a fixed external IP address.
I’ll be back once I have more data to share. As ever, thank you for the support!
Quite understandable, thank you for all the additional details wiley42!
Probably, the additional height provided by the tower will address most of the issues, as the terminal should also be able to connect to more satellites (while waiting for the network up there to densify).
It’s no trouble at all. If you need further assistance when the snow melts, just reach out to us!
Okay, after an extended set of distractions we’re back to looking at this problem. The status quo is definitely being maintained, which is to say the Boom just sort of works away quietly while the two shakes locally report that they’re happy with life, but no data contributions show up on Station View.
While this has historically been mildly annoying, it wasn’t much of an issue because we could pull the local helicorder logs. However, it appears that functionality has been lost with the last major upgrade, which means these things are currently effectively useless to us.
Is it possible for us to backrev these things? While we’ll perform the experiement suggested in the message from 19 Feb, we’re skeptical. Granted, we still see the occasional drop-out on Starlink, but your wire protocol should expect network interruptions, so the expectation is that’s not a contributing factor.
So, while the shakes are currently up and happy locally, we’ll go take them offline and start the rotating isolation process. See you in ~72 hours
The local helicorder is still being produced and is available. You can reach it via either of these two addresses:
rs.local/heli
ShakeIPAddress/heli
You’ll reach the same panel you had access to before the latest update. If you want to access the actual image files in Shake, they are stored at /opt/data/gifs.
Regarding your second query, no, it’s not possible to revert back to a previous Shake OS version.
Boot 4D with 3D already running, after 15 minutes StationView is still claiming “No data” from the 4D, although the map icon is occasionally changing colors and new data is no longer being reported for the 3D while old data scrolls off the display. At 17 minutes the icon for the 4D is white. At 25 minutes the 3D is now reporting no data, although for the moment the icon is still changing color. By 30 minutes (possibly earlier, I was distracted), the map icons for both the 3D and 4D are white.
Shut down the 4D. After about ten minutes the 3D started reporting again, before subsequently falling off the air again. EDIT: It fell off the air, then about ten minutes later it started reporting again, and then fell off the air again – although the ground motion values appear to still be updating and the map icon is changing colors. EDIT2: Around 40 minutes after downing the 4D, the 3D appears to have fully recovered with respect to data contribution to StationView.
Helicorder data (thanks for the hint on where to find it!) is still being generated and persisted.
This kinda feels like the shakes are stepping on each other. All three shakes have static IP assignments and assignments in internal DNS. The connection to the outside world is NAT’ed, so all three appear to be reporting from the same routable IP address, and all three have the same default hostname of “raspberryshake”.
Having looked at this more closely, it appears that while the waveform windows display “no data” and there is no dataview data being presented, there looks to be valid ground motion data. We also spent a silly amount of time investigating the tunnel and can confirm that the two shakes in question have distinct outbound port numbers, while the shakes themselves seem to be presenting close, but not identical, source port numbers.
The fact that every Shake works well on its own is already good news! We can remove that “player” from the “board”.
What you report after this new extensive testing phase (thank you for it) is more or less what my colleague and I were expecting to see, with the instruments working fine on their own, but then starting to exhibit unusual behavior when the others are added to the connection.
Do you have the most recent logs from all the instruments? If so, could you attach them again?
Some more info that could help us determine why this continues to happen with multiple Shakes:
Is the tunnel multiplexing all outbound traffic (basically all two/three instruments’ TCP streams riding inside one outer connection), or does each device effectively get its own independent outer session/socket?
Is the tunnel running over TCP or UDP? And, if you know, does the firewall/tunnel software apply any bandwidth restrictions/shaping or per-flow buffering/queuing limits? Or idle-timeout/keepalive settings?
Could you try to run this sudo tcpdump -i <interface> -w shake_capture.pcap -s 96
from the firewall/tunnel machine, but from the side that sees the streams of the two Shakes coming in (not the Starlink one) and then send me the .pcap file?
Some notes:
replace the <interface> with your LAN-facing network interface name, commonly eth0, but it could be different, so better check first with your network manager using ip a
start recording with the command before you turn on/connect both Shakes, let it run throughout the entire test (a similar 30-40 minutes one), then stop it with a simple Ctrl+C
if the file is too large, you can use gzip shake_capture.pcap to shrink it before sending it
and, if you could, the Starlink operational logs from the same time of the testing would be good to have too, but only if they are retrievable
So we did the cost-benefit analysis of continuing to chase this problem and decided that it was more economical to just source route all of the Shakes around the tunnel. That fixed the immediate problem, but we’ve been hanging fire on declaring victory, which maybe turns out to be a good thing, as today there was a notification from your side that one of the Shakes had fallen off the planet. Dashboard showed the shake running and connected to the server, but services had dropped offline; a reboot and everything is back to normal.
Logs from the failed Shake are attached; this is likely an unrelated issue, so let us know if we need to start a new thread.
