Filesystem Full, Station Not Reporting Data

Hello All, I hope you can shed some light on this problem. Here are the details…

Station: R907E
Hardware: Raspberry Shake RS4D
Storage: 8 GB microSD card
Operating system: “Raspbian GNU/Linux 8 (jessie)”

Background

My Raspberry Shake (R907E) stopped reporting seismic activity about two years ago. I finally got the unit back inside, connected a monitor and keyboard, and was able to boot the Raspberry Pi successfully. The Linux operating system appears to be functioning, but the station is not reporting data as expected.

During troubleshooting, I discovered that the root filesystem is completely full.

1. Filesystem Status

Running df -h reports:

Filesystem   Size   Used   Avail   Use%
/dev/root    7.4G   7.2G      0   100%

Other mounted filesystems show little or no usage.

2. Directory Usage

Running sudo du -xhd1 / identified the following major directories:

/opt          4.0G
/var          2.0G
/usr          831M
/lib          430M

Further investigation of /opt showed:

/opt/data     3.9G
/opt/log      1.9M
/opt/DL        47M

And /opt/data contains:

/opt/data/gifs       3.8G
/opt/data/archive    144M

The GIF directory contains approximately 10,997 GIF files .

The oldest files are dated September 20, 2021, and the newest files are dated October 30, 2026.

Interestingly, some helicorder GIFs are still being generated, suggesting that at least portions of the Raspberry Shake software remain operational.

3. System Log Usage

The /var/log directory occupies approximately 1 GB.

The largest files are:

daemon.log.1     468M
daemon.log       229M
syslog           229M
syslog.1          89M

Both daemon.log and syslog are actively being updated.

I have not yet examined their contents to determine whether recurring errors account for their unusually large sizes.

4. Kernel Messages

One of the last kernel messages I found was:

ttyS ttyS0: 1 input overrun(s)

I understand this indicates a serial-port receive overrun, although I don’t know whether it is related to the reporting failure.

I also checked for undervoltage or throttling messages using:

dmesg | grep -i -E 'voltage|under-voltage|throttl'

No messages were returned.

5. Questions

at this point, I cannot even create a temp file: “No space left on device”
Before deleting files or rebuilding the microSD card, I would appreciate some guidance.

  1. GIF retention: Is it normal for /opt/data/gifs to accumulate several years of helicorder images? Should older GIF files be automatically deleted?
  2. Disk recovery: Is there a recommended procedure for safely removing older GIF files and recovering disk space without affecting the Raspberry Shake installation?
  3. System logs: Are the large daemon.log and syslog files indicative of a known Raspberry Shake problem?
  4. Data acquisition: Since helicorder GIFs are still being generated, is there a recommended method for determining whether the seismic acquisition hardware is operating correctly?
  5. Network reporting: After recovering disk space, what is the recommended procedure for verifying or restoring data transmission to the Raspberry Shake network?
  6. Software recovery: Given the age of this installation, would you recommend repairing the existing system or installing a current Raspberry Shake image on a larger microSD card?

My preference is to recover the existing installation if possible, particularly to preserve its configuration and any diagnostic information that might explain the original failure.

Any recommendations would be appreciated. Thanks!

Jim
AM.R907E

6. More information:

While there are A LOT of .gif files, I found that the log file was being flooded with avahi-daemon errors. This is what I am seeing, and the .103 device is my home PC Windows 11 PC.

Nov 1 09:06:29 raspberryshake avahi-daemon[419]: Invalid response packet from host 192.168.1.103.

And there are lot of error records all occurring multiple times a second…

sudo grep -c 'Invalid response packet' /var/log/daemon.log.1
4948823

For the moment, I stopped the avahi-daemon.service and avahi-daemon.socket processes. Is it possible that Raspbian GNU/Linux 8 (jessie) is running an older and incompatible version of Avahi that is causing the error message flood?

Hello, welcome back to the community, and thank you for such a detailed report!

Your diagnosis is correct: a 100% full system will stop the Shake from working properly. In a recent, similar case, the Shake service failed to start on a full disk. The station came back after the data was saved, the old archive directories were removed, and the unit was rebooted.

Before doing anything, if you want to preserve the current state of the microSD card, you can turn off the Shake, extract the card, and make a 1:1 copy using one of the many SD card imagers that are available online (Win32 Disk Imager for Windows or DD for both Linux and Windows are good options).

Regarding your queries:

  1. The number of GIFs in /opt/data/gifs is correlated to (double) the number of saved days you have input in Settings - Waveform Data Saving. The default is 7, which leaves plenty of space for the system to keep running without space issues. Do you remember/know how many save days were last set in your Shake?

  2. Yes, you can do it in two steps. First, run this command after you SSH into the Shake:

    sudo find /opt/data/gifs -name '*.gif' -mtime +30 | head

    This will list all files that are older than 30 days (you can check their datetime stamp in the filename. There will be two gifs per day, ending in 00 and 12).

    Once you are sure those are all old images, use this other command:

    sudo find /opt/data/gifs -name '*.gif' -mtime +30 -delete

    That will delete all GIFs older than 30 days. If you want to change the number, feel free to do so, just remember that it has to match between the two commands.

    Check how much space has been freed, and then reboot the unit.

  3. Having those two logs with that size could be a symptom of some underlying issue, but I would need to check what they say (a snippet or a full upload to a remote hosting service, like Dropbox or Mediafire) to be 100% sure. A full disk could be causing general issues that, as you correctly surmised, keep the OS from performing optimally across the board.

  4. As far as I could see from the last data uploaded to our servers (on October 31th), the data produced by the Shake was in perfect condition, with no particular issues that I could find.

    A quick way to compare data is to use DataView across different days. Another option is to compare the size of the daily data files (found in /opt/data/archive) and, in both methods, look for discrepancies (data appearance in the waveform/spectrogram, or file sizes well below or above the general average).

  5. After you delete the old GIF images and reboot, the Shake should start operating nominally again (since we have probably deleted about 3.0 to 3.5 GB). Thus, it should be able to reconnect to the server on its own within 3-5 minutes of the reboot.

  6. MicroSD cards can be quite resilient, so in the end it is your choice. If you want to save all the produced helicorder images, then you can use the 1:1 copy you made before starting these steps (or copy them directly from the microSD or via scp before reburning.

    Otherwise, if the images are not essential, you could either: a) re-burn the current microSD card with a fresh Shake OS copy, here, or b) burn a new microSD card in the same way, keeping the current one.

Regarding the avahi-daemon, this could also be a symptom of the full storage. So, after you clean up, if logs continue to be flooded with messages, you can send them to us (with the full zipped log files you can download from rs.local/), and we can take a look.

For now, let’s focus on freeing up some space, and we’ll see what comes after.

Let me know!

Hi SC,
Thanks for the reply and answers. I did delete the old .gif’s per your instructions and went from 10,997 to 229 gifs; this definitely freed up space. The ‘df /’ command now shows:

df /
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        7706968 4077692   3297324  56% /

While the trace is not very interesting (sitting on a towel on a bench in the garage), I do see traces on the Data View and Station View. The avahi-daemon did restart with the reboot and the avahi errors have disappeared.

BTW… the new dashboard looks great! As for the Settings - Waveform Data Saving… how can I get to that?

Thanks again for the assist!

regards,
jim

Great news Jim!

And thank you for the feedback on the dashboard, I’ll pass it along to our team!

You can find the Waveform Data Saving option by clicking the gear icon in the top right, and then scrolling to the bottom of the General tab.

It was a pleasure! Enjoy your Shake!