RSudp errors

hcintron · April 15, 2020, 2:53pm

Hi Everyone,

I have 2 errors, Last night I was playing with some of the settings on the alert module to avoid less alerts due to noise using the docs as a reference.

"alert": {
    "enabled": true,
    "channel": "HZ",
    "sta": 6,
    "lta": 30,
    "threshold": 4.5,
    "reset": 0.5,
    "highpass": 0.8,
    "lowpass": 9,
    "deconvolve": false,
    "units": "VEL"},

when I try these settings RSudp will eventually error out with

2020-04-15 04:02:54 [Init] Waiting for UDP data on port 8888...
2020-04-15 04:02:56 [Init] Available channels: ['EHZ', 'ENZ', 'ENN', 'ENE']
2020-04-15 04:02:56 [get_inventory] Fetching inventory for station AM.R7078 from Raspberry Shake FDSN.
2020-04-15 04:02:57 [get_inventory] Inventory fetch successful. Station region is VIRGIN ISLANDS
2020-04-15 04:02:57 [Plot] WARNING: Running on armv7l machine, using Tk instead of Qt
2020-04-15 04:02:57 [Plot] Starting.
2020-04-15 04:02:57 [Plot] Plotting channels: ['EHZ']
2020-04-15 04:02:57 [Plot] Signal deconvolution set to CHAN
2020-04-15 04:02:57 [Plot] Seismogram units are channel-specific
2020-04-15 04:02:57 [Alert] Alert stream units are Voltage counts
2020-04-15 04:02:57 [Alert] Starting Alert trigger with sta=6s, lta=30s, and threshold=4.5 on channel=?HZ
2020-04-15 04:02:57 [Alert] Alert stream will be bandpass filtered from 0.8 to 9 Hz
2020-04-15 04:02:57 [Telegram] Starting.
2020-04-15 04:02:57 [Consumer] Starting.
2020-04-15 04:02:57 [Producer] Starting.
2020-04-15 04:02:57 [Producer] Receiving UDP data from 192.168.40.71
2020-04-15 04:02:57 [Alert] Listening to channel EHZ
2020-04-15 04:02:57 [Alert] Earthquake trigger warmup time of 30 seconds...
2020-04-15 04:02:58 [Plot] WARNING: Failed to set PNG icon image, trying .ico instead
2020-04-15 04:03:28 [Alert] Earthquake trigger up and running normally.
**2020-04-15 04:07:12 [Alert] Threshold: 4.5; Current max STA/LTA: 0.0972malloc_consolidate(): invalid chunk size**
**unix-start-rsudp.sh: line 28: 25539 Aborted                 rs-client -s $settings 2> >(tee -a /tmp/rsudp/rsudp.log >&2 2>&1)**

now when I restore back to the original config basically changing the high/low filters I get what looks like error from the server side

"alert": {
    "enabled": true,
    "highpass": 0,
    "lowpass": 50,
    "deconvolve": false,
    "units": "VEL",
    "sta": 6,
    "lta": 30,
    "threshold": 1.576,
    "reset": 1.570,
    "exec": "eqAlert",
    "channel": "HZ",
    "win_override": false},

getting a ssl cert error

2020-04-15 14:44:53 [Init] Available channels: ['EHZ', 'ENZ', 'ENN', 'ENE']
2020-04-15 14:44:53 [get_inventory] Fetching inventory for station AM.R7078 from Raspberry Shake FDSN.
**2020-04-15 14:44:54 Certificate did not match expected hostname: %s. Certificate: %s**
**2020-04-15 14:44:54 [get_inventory] ERROR: Inventory fetch failed!**
**2020-04-15 14:44:54 [get_inventory]        Error detail: HTTPSConnectionPool(host='fdsnws.raspberryshakedata.com', port=443): Max retries exceeded with url: /fdsnws/station/1/query?network=AM&station=R7078&starttime=2020-04-15T10:44:53.530746Z&level=resp&nodata=404&format=xml (Caused by SSLError(CertificateError("hostname 'fdsnws.raspberryshakedata.com' doesn't match either of '*.raspberryshake.org', 'raspberryshake.org'",),))**
2020-04-15 14:44:54 [Plot] WARNING: Running on armv7l machine, using Tk instead of Qt
2020-04-15 14:44:54 [Plot] Starting.
2020-04-15 14:44:54 [Plot] Plotting channels: ['EHZ']
2020-04-15 14:44:54 [Plot] Seismogram units are Voltage counts
2020-04-15 14:44:54 [Alert] Alert stream units are Voltage counts
2020-04-15 14:44:54 [Alert] Starting Alert trigger with sta=6s, lta=30s, and threshold=1.576 on channel=?HZ
2020-04-15 14:44:54 [Telegram] Starting.
2020-04-15 14:44:54 [Consumer] Starting.
2020-04-15 14:44:54 [Producer] Starting.
2020-04-15 14:44:54 [Producer] Receiving UDP data from 192.168.40.71
2020-04-15 14:44:54 [Alert] Listening to channel EHZ
2020-04-15 14:44:54 [Alert] Earthquake trigger warmup time of 30 seconds...
2020-04-15 14:44:55 [Plot] WARNING: Failed to set PNG icon image, trying .ico instead
2020-04-15 14:45:24 [Alert] Earthquake trigger up and running normally.
2020-04-15 14:45:47 [Alert] Threshold: 1.576; Current max STA/LTA: 1.5717

Thanks

iannesbitt · April 15, 2020, 6:22pm

Hi Hector, first off, the SSL error should be fixed now. That was on our end and is unrelated to your other problem.

Second, I have seen that abort error before (and others like it). Are you using Raspberry Pi 3? For some reason Pi 3B/3B+ are prone to memory errors in rare cases when using matplotlib (and since the plot redraws so much, the longer you run it the more likely it is to error).

So far I have not found a solution for this, I think the bug that causes it is buried in the C code of matplotlib. My hack solution is to run rsudp as a daemon so that it restarts after it crashes.

hcintron · April 16, 2020, 12:31am

Hi Ian,

Cool on the SSL I just restarted the software and I did not see the SSL error.

Yes I’m running it on a Pi3B+ I’ll try the hack solution and also I might play with a Pi2B i have laying around. Once I have it sort it out I’ll turn on twitter feed. the last few days I was getting too many false alarms. Thanks

hcintron · April 16, 2020, 6:19am

Hi Ian,

I got an error while trying to start the daemon. what I find curios on a new install on a different Pi trying to start manually I receive the same traceback error.

This was the original Pi with the daemon

pi@earthquakepi:~ $ systemctl --user daemon-reload
pi@earthquakepi:~ $ sudo loginctl enable-linger "$USER"
pi@earthquakepi:~ $ systemctl --user start rsudp.service
pi@earthquakepi:~ $ tail -n 30 -f /tmp/rsudp/rsudp.log
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing ./bin/rsudp
Requirement already satisfied: obspy in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from rsudp==1.0.0) (1.1.1)
Requirement already satisfied: numpy in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from rsudp==1.0.0) (1.15.1)
Requirement already satisfied: matplotlib==3.1.1 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from rsudp==1.0.0) (3.1.1)
Requirement already satisfied: pydub in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from rsudp==1.0.0) (0.23.1)
Requirement already satisfied: twython in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from rsudp==1.0.0) (3.7.0)
Requirement already satisfied: python-telegram-bot in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from rsudp==1.0.0) (12.4.2)
Requirement already satisfied: future>=0.12.4 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from obspy->rsudp==1.0.0) (0.16.0)
Requirement already satisfied: setuptools in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from obspy->rsudp==1.0.0) (40.2.0)
Requirement already satisfied: lxml in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from obspy->rsudp==1.0.0) (4.2.4)
Requirement already satisfied: scipy>=0.9.0 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from obspy->rsudp==1.0.0) (1.0.0)
Requirement already satisfied: sqlalchemy in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from obspy->rsudp==1.0.0) (1.2.11)
Requirement already satisfied: requests in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from obspy->rsudp==1.0.0) (2.23.0)
Requirement already satisfied: decorator in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from obspy->rsudp==1.0.0) (4.4.1)
Requirement already satisfied: python-dateutil>=2.1 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from matplotlib==3.1.1->rsudp==1.0.0) (2.6.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from matplotlib==3.1.1->rsudp==1.0.0) (2.2.0)
Requirement already satisfied: kiwisolver>=1.0.1 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from matplotlib==3.1.1->rsudp==1.0.0) (1.1.0)
Requirement already satisfied: cycler>=0.10 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from matplotlib==3.1.1->rsudp==1.0.0) (0.10.0)
Requirement already satisfied: requests-oauthlib>=0.4.0 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from twython->rsudp==1.0.0) (1.3.0)
Requirement already satisfied: cryptography in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from python-telegram-bot->rsudp==1.0.0) (2.8)
Requirement already satisfied: certifi in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from python-telegram-bot->rsudp==1.0.0) (2018.8.24)
Requirement already satisfied: tornado>=5.1 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from python-telegram-bot->rsudp==1.0.0) (6.0.3)
Requirement already satisfied: chardet<4,>=3.0.2 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from requests->obspy->rsudp==1.0.0) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from requests->obspy->rsudp==1.0.0) (1.24.2)
Requirement already satisfied: idna<3,>=2.5 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from requests->obspy->rsudp==1.0.0) (2.9)
Requirement already satisfied: six>=1.5 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from python-dateutil>=2.1->matplotlib==3.1.1->rsudp==1.0.0) (1.11.0)
Requirement already satisfied: oauthlib>=3.0.0 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from requests-oauthlib>=0.4.0->twython->rsudp==1.0.0) (3.1.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from cryptography->python-telegram-bot->rsudp==1.0.0) (1.11.5)
Requirement already satisfied: pycparser in ./berryconda3/envs/rsudp/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography->python-telegram-bot->rsudp==1.0.0) (2.18)
Building wheels for collected packages: rsudp
  Running setup.py bdist_wheel for rsudp: started
  Running setup.py bdist_wheel for rsudp: finished with status 'done'
  Stored in directory: /tmp/pip-ephem-wheel-cache-mgxrpsm3/wheels/96/51/7f/31c1c2db9498304e7bdaf78a5cbec199ccb81599ee71b037d2
Successfully built rsudp
Installing collected packages: rsudp
  Found existing installation: rsudp 0.4.3
    Uninstalling rsudp-0.4.3:
      Successfully uninstalled rsudp-0.4.3
Successfully installed rsudp-1.0.0
Qt import failed. Trying Tk...
detail: No module named 'PyQt5'
2020-04-16 01:12:40 [Init] Logging initialized successfully.
2020-04-16 01:12:40 [Main] Logging initialized successfully.
2020-04-16 01:12:40 Using settings file: /home/pi/.config/rsudp/rsudp_settings.json
2020-04-16 01:12:40 Output directory is: /home/pi/rsudp
2020-04-16 01:12:40 [Init] Initializing.
2020-04-16 01:12:40 [openSOCK] Opening socket on localhost:8888 (HOST:PORT)
2020-04-16 01:12:40 [Init] Waiting for UDP data on port 8888...
2020-04-16 01:12:42 [Init] Available channels: ['EHZ', 'ENZ', 'ENN', 'ENE']
2020-04-16 01:12:42 [get_inventory] Fetching inventory for station AM.R7078 from Raspberry Shake FDSN.
2020-04-16 01:12:43 [get_inventory] Inventory fetch successful. Station region is VIRGIN ISLANDS
2020-04-16 01:12:43 [conn_stats] Initialization stats:
2020-04-16 01:12:43 [conn_stats]                 Port: 8888
2020-04-16 01:12:43 [conn_stats]   Sending IP address: 192.168.40.71
2020-04-16 01:12:43 [conn_stats]     Set station name: R7078
2020-04-16 01:12:43 [conn_stats]   Number of channels: 4
2020-04-16 01:12:43 [conn_stats]   Transmission freq.: 250 ms/packet
2020-04-16 01:12:43 [conn_stats]    Transmission rate: 4 packets/sec
2020-04-16 01:12:43 [conn_stats]   Samples per second: 100 sps
2020-04-16 01:12:43 [conn_stats]            Inventory: AM.R7078 (Raspberry Shake Citizen Science Station)
2020-04-16 01:12:43 [Plot] WARNING: Running on armv7l machine, using Tk instead of Qt
2020-04-16 01:12:43 [Plot] Plotting 1 channels: ['EHZ']
2020-04-16 01:12:43 [Plot] Signal deconvolution set to CHAN
2020-04-16 01:12:43 [Plot] Seismogram units are channel-specific
2020-04-16 01:12:43 [Plot] Starting.
2020-04-16 01:12:43 [Alert] Alert stream units are counts
Traceback (most recent call last):
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 281, in run
    if settings['custom']['enabled']:
KeyError: 'custom'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/berryconda3/envs/rsudp/bin/rs-client", line 11, in <module>
    sys.exit(main())
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 451, in main
    run(settings, debug=debug)
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 295, in run
    raise KeyError(e)
KeyError: KeyError('custom',)
tail: /tmp/rsudp/rsudp.log: file truncated
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple

this was the new Pi install on a diferent unit running it manually

pi@raspberrypi:~ $ cd ~/bin/rsudp
pi@raspberrypi:~/bin/rsudp $ 
pi@raspberrypi:~/bin/rsudp $ bash unix-start-rsudp.sh
Installing from the git directory...
Done.
Qt import failed. Trying Tk...
detail: No module named 'PyQt5'
2020-04-16 05:39:24 [Main] Logging initialized successfully.
2020-04-16 05:39:24 Using settings file: /home/pi/.config/rsudp/rsudp_settings.json
2020-04-16 05:39:24 Output directory is: /home/pi/rsudp
2020-04-16 05:39:24 [Init] Initializing.
2020-04-16 05:39:24 [openSOCK] Opening socket on localhost:8888 (HOST:PORT)
2020-04-16 05:39:24 [Init] Waiting for UDP data on port 8888...
2020-04-16 05:39:26 [Init] Available channels: ['EHZ', 'ENZ', 'ENN', 'ENE']
2020-04-16 05:39:26 [get_inventory] Fetching inventory for station AM.R7078 from Raspberry Shake FDSN.
2020-04-16 05:39:28 [get_inventory] Inventory fetch successful. Station region is VIRGIN ISLANDS
2020-04-16 05:39:28 [conn_stats] Initialization stats:
2020-04-16 05:39:28 [conn_stats]                 Port: 8888
2020-04-16 05:39:28 [conn_stats]   Sending IP address: 192.168.40.71
2020-04-16 05:39:28 [conn_stats]     Set station name: R7078
2020-04-16 05:39:28 [conn_stats]   Number of channels: 4
2020-04-16 05:39:28 [conn_stats]   Transmission freq.: 250 ms/packet
2020-04-16 05:39:28 [conn_stats]    Transmission rate: 4 packets/sec
2020-04-16 05:39:28 [conn_stats]   Samples per second: 100 sps
2020-04-16 05:39:28 [conn_stats]            Inventory: AM.R7078 (Raspberry Shake Citizen Science Station)
2020-04-16 05:39:28 [Plot] WARNING: Running on armv7l machine, using Tk instead of Qt
2020-04-16 05:39:28 [Plot] Plotting 1 channels: ['EHZ']
2020-04-16 05:39:28 [Plot] Signal deconvolution set to CHAN
2020-04-16 05:39:28 [Plot] Seismogram units are channel-specific
2020-04-16 05:39:28 [Plot] Starting.
2020-04-16 05:39:28 [Alert] Alert stream units are counts
Traceback (most recent call last):
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 281, in run
    if settings['custom']['enabled']:
KeyError: 'custom'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/berryconda3/envs/rsudp/bin/rs-client", line 11, in <module>
    sys.exit(main())
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 451, in main
    run(settings, debug=debug)
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 295, in run
    raise KeyError(e)
KeyError: KeyError('custom',)
pi@raspberrypi:~/bin/rsudp $ conda activate rsudp
(rsudp) pi@raspberrypi:~/bin/rsudp $ rs-client
Qt import failed. Trying Tk...
detail: No module named 'PyQt5'
2020-04-16 05:42:57 [Main] Logging initialized successfully.
2020-04-16 05:42:57 Using settings file: /home/pi/.config/rsudp/rsudp_settings.json
2020-04-16 05:42:57 Output directory is: /home/pi/rsudp
2020-04-16 05:42:57 [Init] Initializing.
2020-04-16 05:42:57 [openSOCK] Opening socket on localhost:8888 (HOST:PORT)
2020-04-16 05:42:57 [Init] Waiting for UDP data on port 8888...
2020-04-16 05:42:59 [Init] Available channels: ['EHZ', 'ENZ', 'ENN', 'ENE']
2020-04-16 05:42:59 [get_inventory] Fetching inventory for station AM.R7078 from Raspberry Shake FDSN.
2020-04-16 05:43:00 [get_inventory] Inventory fetch successful. Station region is VIRGIN ISLANDS
2020-04-16 05:43:00 [conn_stats] Initialization stats:
2020-04-16 05:43:00 [conn_stats]                 Port: 8888
2020-04-16 05:43:00 [conn_stats]   Sending IP address: 192.168.40.71
2020-04-16 05:43:00 [conn_stats]     Set station name: R7078
2020-04-16 05:43:00 [conn_stats]   Number of channels: 4
2020-04-16 05:43:00 [conn_stats]   Transmission freq.: 250 ms/packet
2020-04-16 05:43:00 [conn_stats]    Transmission rate: 4 packets/sec
2020-04-16 05:43:00 [conn_stats]   Samples per second: 100 sps
2020-04-16 05:43:00 [conn_stats]            Inventory: AM.R7078 (Raspberry Shake Citizen Science Station)
2020-04-16 05:43:00 [Plot] WARNING: Running on armv7l machine, using Tk instead of Qt
2020-04-16 05:43:00 [Plot] Plotting 1 channels: ['EHZ']
2020-04-16 05:43:00 [Plot] Signal deconvolution set to CHAN
2020-04-16 05:43:00 [Plot] Seismogram units are channel-specific
2020-04-16 05:43:00 [Plot] Starting.
2020-04-16 05:43:00 [Alert] Alert stream units are counts
Traceback (most recent call last):
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 281, in run
    if settings['custom']['enabled']:
KeyError: 'custom'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/berryconda3/envs/rsudp/bin/rs-client", line 11, in <module>
    sys.exit(main())
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 451, in main
    run(settings, debug=debug)
  File "/home/pi/berryconda3/envs/rsudp/lib/python3.6/site-packages/rsudp/client.py", line 295, in run
    raise KeyError(e)
KeyError: KeyError('custom',)
(rsudp) pi@raspberrypi:~/bin/rsudp $

Thanks

iannesbitt · April 16, 2020, 2:02pm

This means your settings file is outdated on both machines (this is my fault, I added a section to the settings file in the newest update). Please back up your existing settings files and re-export the rsudp settings like so:

conda activate rsudp
cp ~/.config/rsudp/rsudp_settings.conf ~/.config/rsudp/rsudp_settings.conf.bak
rs-client -d default

Then you will need to copy the old settings over to the new file.
I think there may be a reliable way to automate this (I’ll look into it) but for now you will have to do it by hand.

Alternatively you can copy these lines to ~/.config/rsudp/rsudp_settings.conf just below the “alert” block:

"custom": {
    "enabled": false,
    "codefile": "n/a",
    "win_override": false},

Technically order doesn’t matter, but this block is ended by a comma so it cannot go at the end of the file, which is why I suggest putting it after the “alert” block.