Shake reboots, server starts, but data no longer recorded

After a few recent power failures, my shake seems stuck. It will reboot and the webserver shows up, but after 7 hours it still reports
System Status: BOOTING
Data Producer: OFF
Data Consumer: OFF
Server Connection: Not Connected
and I notice that no new helicorder plots have been saved since the first power outage three days ago (Jan 13 2024). I have tried power-cycling, and rebooting through the web page, without solving the problem. Any tips? My unit is AM.RC93C.00.HDF and it is actually a “R-Boom” (infrasound detector) running on a R-Pi 2 model B. Prior to this it had been running OK for a long time, possibly over a year.

Hello jbeale, and welcome back to the community.

I’m sorry to hear about this issue, so let’s see what we can do.

I think the recent power failures you described may have affected some files on the microSD card, causing the problem you are seeing. Could I please ask you, as you can access rs.local/, to download the Shake logs and post them here?

They should provide more insight into what is happening so I can advise you better. If needed, instructions on how to get them are here: Please read before posting!

Thank you!

log fileset is attached.
RSH.RC93C.2024-01-17T16_16_32.logs.tar.zip (210.1 KB)

Thank you for the logs jbeale, they have confirmed my suspicion about some corruption going on in the microSD card files.

I would then recommend re-burning the microSD card after formatting and erasing all its data/partitions first (you can use DISKPART for this as it is very efficient), and see how the Shake behaves with the newly installed system. This should solve the data issues you are seeing.

I will leave the burning instructions link here for your convenience: microSD card topics

If it doesn’t, wait for around 30 minutes after the last post-reburn Shake reboot, download the new logs and then send them to me.

Thank you for your collaboration.

Thank you for your review of the situation. I am also curious if there is any possibility of in-place remote-update, given that the computer itself seems to boot up and communicate. Physically accessing that system is non-trivial, and given the current weather conditions locally, it may be some time before it can be done.

1 Like

You’re more than welcome jbeale.

A full OS reset is currently impossible without physically reaching the Shake and extracting the microSD card, as it needs to be fully formatted before reinstalling the OS itself.

What is the installation of this Shake? Is it out in the field (I see a GPS connection?), and if so, is it connected via GPRS/3G to the network?

Can you try to reboot after accessing the Shake via SSH (thus not through the web page)? You can find instructions on how to do so here: How to access your Raspberry Shake’s computer via ssh and, after this, reboot with the following command?

sudo shutdown -r 2

The Shake should reboot after two minutes, and then, can you please wait for around 30 min, download the new logs and send them here?

I wish to see if there are any notable differences between the two log sets and then see if we can try anything from there.

Thank you.

In case it is of any use, I did the shutdown -r 2 command via ssh as requested and later I downloaded the logs again.
RSH.RC93C.2024-01-21T03_38_54.logs.tar.zip (211.2 KB)

1 Like

hi john,

sorry to hear your box has gone wonky.

i had a look at the log files and can say pretty confidently that this is not due to SD card corruption. rather, what’s happening is that at bootup, the startup procedure makes an attempt to download the support files for the GPS module that is detected and attached to the USB port.

is there, in fact, a GPS module attached to the Pi? if so, this problem can be dealt with by hand, assuming that logging into the box is possible.

i will be thinking what the most straightforward way might be to get the box back up and running and get back to you. in the meantime, can you confirm the following?:

  • that there is indeed a GPS module attached, and
  • if so, also provide the details why? (i ask because when the unit is connected to the internet, i.e., is able to use an NTP server instead, a GPS module is not necessary.)
  • logging into the unit via ssh is possible

thanks in advance, cheers,
richard

Aha! Very interesting. There is no GPS device connected to my outdoors R-Boom, which is connected to my home network by buried ethernet cable using PoE power. It does have a serial device on the USB port: an Arduino that periodically reports temperature and humidity readings inside the outdoor enclosure, being read by a separate Python program I installed. I usually do this with any Pi-based projects I have installed outdoors, to check on possible case integrity issues. I thought it would run entirely separately from the R-Shake software- in fact it did for at least a year- but I gather for some reason, the system now interprets my particular serial device as a GPS unit.
Is there a way to turn off detection of attached serial devices as GPS units?
Also, yes I can log in remotely via ssh.

Linux raspberryshake 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l

WELCOME TO RASPBERRY SHAKE!

Developed by:
 Raspberry Shake: https://raspberryshake.org
Boaz Consultancy: https://sqlx.science

STATION:	AM.RC93C.00
IP-ADDR:	192.168.1.110

Last login: Sat Jan 20 01:57:10 2024 from 192.168.1.141
1 Like

hi john,

sorry for the delay, lots of “stuff” going on…

i think there is a fairly straightforward way to get the boot-up script to “think” there’s no GPS module attached.

in the directory /usr/local/bin you will find a program named gpsd-mgr. the boot-up program runs this program to determine if a GPS module is attached. if it returns a value of 1, i.e., false in bash script world, it will proceed assuming that no GPS module is attached.

therefore, you can try the following (apologies for not being able to check this myself beforehand):

  • move the program /usr/local/bin/gpsd-mgr to the side, i.e., copy to a different directory or rename it
  • in its place, make a simple bash script that will look something like:
#!/bin/bash
exit 1
  • make sure it is marked as executable:
    cl-prompt > chmod a+x /usr/local/bin/gpsd-mgr
  • reboot

the boot-up program should then assume that there is no GPS module attached to the USB port, thereby getting past the problem you are experiencing.

please let me know if this is resolves the issue or not. if not, the log files are always appreciated.

cheers,
richard

1 Like

Ok, having replaced /usr/local/bin/gpsd-mgr as described, it does now actually reach the Data Producer and Data Consumer ON state, and even Server Connection: Connected. And at the moment, in fact I see my station is actually online:

However it looks like I do have SD card problems, so I presume I will have to uncover my housing, open up the system and replace the SD card after all.

[  212.950971] mmc0: cmd op 17 arg 0xc6a61d flags 0xb5 - resp 00000900 00000000 00000000 00000000, err 0
[  212.950980] mmc0: data blocks 1 blksz 200 - err 0
[  212.950987] mmc0: =========== REGISTER DUMP ===========
[  212.950993] mmc0: SDCMD  0x00004051
[  212.950999] mmc0: SDARG  0x00c6a61d
[  212.951006] mmc0: SDTOUT 0x017d7840
[  212.951012] mmc0: SDCDIV 0x00000003
[  212.951019] mmc0: SDRSP0 0x00000900
[  212.951025] mmc0: SDRSP1 0x00001133
[  212.951031] mmc0: SDRSP2 0x7fffffff
[  212.951037] mmc0: SDRSP3 0x0002482e
[  212.951044] mmc0: SDHSTS 0x00000080
[  212.951050] mmc0: SDVDD  0x00000001
[  212.951056] mmc0: SDEDM  0x00010801
[  212.951063] mmc0: SDHCFG 0x0000041e
[  212.951069] mmc0: SDHBCT 0x00000200
[  212.951075] mmc0: SDHBLC 0x00000001
[  212.951081] mmc0: ===========================================
[  212.951285] mmcblk0: error -110 transferring data, sector 13018653, nr 27, cmd response 0x900, card status 0x0
[  212.951318] print_req_error: I/O error, dev mmcblk0, sector 13018653

1 Like

ugh, the dreaded register dump :face_with_diagonal_mouth:

sorry about that…

let us know if you need anything else from us.,
richard