RSUDP crashes on alert

I’ve been displaying the RSUDP live data on a monitor in my classroom for a while now. I have it set up to give alerts for events. Whenever there is an alert that’s somewhat significant the data output freezes and won’t ever catch up.

I suspect that this is a mere data processing issue and that the Mac mini I’m running it on can’t handle the calculations for the alert thresholds.

Have any of you been successful with running RSUDP on a newer raspberry pi (like a 4 or 5) and been able to handle alerts?

Is there an alert methodology I should follow to get better processing efficiency for alerts?

Thanks

1 Like

Hello there,

Do you have the technical specifics of the Mac mini that you are using? Alarm calculations should work unless the system is really loaded.

I have RSUDP on a 3B+ with a 64-bit OS, and there are no issues, albeit I only display a single channel trace/spectrogram. Also, I only show the last 120 seconds, which may be a factor too. I think a Pi 4 should be more than enough for your needs.

What I would recommend with the current system:

  • shorten the amount of time that is being displayed; maybe show only the last 5 minutes or so so that the system will have to do less work;
  • test if the alarm feature works in this case, or test it again with a single displayed channel.

Is this setup now working, or does the alarm feature still delay the real-time data after it is triggered?

Yeah, I’m showing 10 minutes on three channels.
I’ll reduce it to 5 and give it a shot.

It’s a 1.4 ghz 4gb ddr3 ram (1600MHz).
It’s quite on the low end.

I like the longer traces for class viewing.

I’ve got a pi 5 in the works for next school year.

Do you have a suggestion for low frequency alert thresholds that minimize sensitivity to noise?

1 Like

Yes, let’s make this test and see how it goes.

The hardware is a bit on the low specs, but memory seems to be enough (even if it’s a DDR3). When you upgrade to a Pi5 you should definitely see some improvement, but traces this long can always introduce some kind of strain on the processing systems, so I’m curious about the test results with the new machine.

The noise exclusion threshold is always something hard to define without having personal experience with the Shake location. For example, many RSUDP users are fine with the default value in the settings .json file, but others have had to do some back-and-forth experimenting with it.

As another test, you can try with an STA parameter of 4 (instead of the default 6), and also set "deconvolve": false, to save some processing power. In a classroom, you could try to set the "threshold" parameter at 1.5; it should make it more sensitive (even for some jumping/stomping classroom experiments), while raising it closer to 2 should only show larger events activations.

However, as said, you’ll have to play with these settings to perfectly capture how your location “works” in terms of local noise.

I have my threshold at 2
high pass 0.2
low pass 3
STA/LTA were default.

I’ll drop the STA a bit. It’s pretty obvious that the STA/LTA calculation is what’s breaking this.

2 Likes

Let us know how it goes!

I have a similar problem on two separate instances of RSUDP running on two separate Raspberry Pi 4s. Both rPis are receiving data from the same Raspberry Shake 3D. One instance on one rPi is set to display 60 seconds with the spectrogram showing 0-50 Hz. The second instance on the second rPi is set to display 10 minutes with the spectrogram showing 0-10 Hz. (I do this in order for the first rPi to display near-by earthquakes - it is near a seismically active hydrothermal area - and the second rPi to display more distant, larger earthquakes). Both are set to display all three RS3D channels.

Both instances crash on a regular basis (-both- the 60-second and 10 minute traces). Actually, the 60-second trace seems to crash more frequently, but only slightly. Most of the time, they just randomly crash and quit (not necessarily on an alert), which is not too bad since I have them set to simply restart. Sometimes, however, they will hang, usually as SouthLewisHighSchool describes, on an alert. Then I must manually close the terminal window in which they are running, after which they will then automatically restart.

All the values in the rsudp_settings.json file for both instances are identical except the duration (the first is set to 60 and the second is set to 600). Both instances are monitoring the EHE channels and use identical values for:
“sta”: 6,
“lta”: 30,
“threshold”: 2.9,
“reset”: 1.5,
“highpass”: 0.4,
“lowpass”: 6,
“deconvolve”: false,
“units”: “VEL”

1 Like

So far, my reduction of the trace time has not improved the crashes. What setting did you change to narrow your displayed frequencies? I want to omit gym class from my display.

Unfortunately, there is no setting in the rsudp_settings.json file that controls the frequency range of the spectrogram. In order to change the frequency range, two lines in the source code must be changed. They are both in the “ac_plot.py” file.

line 450: change “self.sps/2” to “self.sps/(100/range)
line 562: change “self.sps/2” to “self.sps/(100/range)

In other words, for the default frequency range of 0-50 Hz, it is “self.sps/2”
For the frequency range to be 0-25 Hz, it should be “self.sps/4”
I wanted a frequency range of 0-10 Hz, so I use “self.sps/10”

2 Likes

Hello dlfowler,

As you described the crashes happening “on a regular basis”, have you noticed a particular pattern regarding them, or are they completely randomized? Do they happen in both standard window and kiosk mode, or only on one of these possible setup displays?

Do you have any other observations that could help us pinpoint the cause of the crashes? I ask because I have an RSUDP installation on a Pi3B+ in a 60 (but also 120 at times) second settings, in kiosk mode, and it never crashed in years of usage. So, the more details you can provide, the more I can dive into possible causes. Thank you!

The additional frequency range settings of the spectrogram sound very interesting. I’ll make some tests and then submit them for enhancement to our developer so they can be added in future updates.

@ SouthLewisHighSchool, could you try for some time (just as a further test) to see if the crashes still happen with a standard 60/90/120 seconds display?

Hello Stormchaser,

The full crashes seem to be random. Since I run rsudp as a systemctl daemon, it automatically restarts after full crashes. Often, the only way I know it has crashed is that the “detected events” count has reset and the terminal and “live output” windows have reopened in random locations.

Less frequently, RSUDP will occasionally freeze and hang. Since the program has not quit, it will not automatically restart. It seems to hang on an alert, just before saves an image file (but the image file is not saved). The stdout message in the terminal window indicates that it is “saving png XXX seconds after alarm,” but no png is saved.

I have not checked the log files, so I have not determined the cause of the either the full crashes or hangs.

I have not run rsudp in kiosk mode, so I don’t know if it would make any difference.

I spoke with Ian back in 2020 about the issue. At that time, he said it was a known issue on rPis, but he didn’t know the cause. I always assumed it probably had been resolved since then, but since I’ve been running RSUDP continuously (except for the crashes), I’ve never upgraded either the OS or RSUDP. My two RSUDP rPi 4s are still running 32-bit Buster and RSUDP 1.01. I’ve thought about upgrading to rPi 5s running the latest 64-bit OS and RSUDP 2.0, which may correct the issues. (over the holidays may be a good time to do so, and then report back on how it works)

On the frequency edits, the first edit is in the “_format_axes,” the routine to plot the axes scales, the second edit is in “_update_specgram,” the routine to actually draw the sprectrogram,

1 Like

Yes, I think that the next step should be doing more tests, maybe with more recent Pi OS/RSUDP versions, to see if this is something that updated systems have solved.

On my side, I’ll continue to “play” with my RSUDP installation a bit to see if I can manage to “trigger” these crashes, and see what the cause could be.

As you don’t use the kiosk mode, I’ll try the same and see if the normal window mode helps in finding out what’s happening.

EDIT: A suggestion that the software team put forward would be to monitor the memory usage on your Pi/miniPC during the time leading up to the crash. It could give us more information on its causes.