Standalone RS, GPS, Consumer/Producer not starting

Technical Support
1

Hello,

As the title suggests, I have a Raspberry shake that is designed to operate in a remote mountain-top location, without internet, and operating on solar setup. The shake is configured to run in standalone mode, and it has a GPS antenna attached (USB purchased from Raspberry shake).

It seems that when booting up, the Data Consumer and Data Producer do not start. It seems they stay off indefinitely. However, if I issue the rsh-restart command, the producer and consumer start up and the data starts being written to the archive.

As this is a remote instrument and may shut down over the winter periodically if the battery power drops, I am trying to sort out why the producer/consumer are not starting up.

The simple, ugly fix is simply to put an rsh-restart command in the crontab, and run that some number of times per day, just to make sure that they start up if not running. However, I assume that restart will result in a few seconds of data loss during the restart cycle. Not a deal breaker by any means, but if it can be avoided, that would be great.

I will attach some log files to this thread tomorrow, but I wanted to post in case there was some obvious answer as to what may be stopping these from starting up. Ie, if there is no NTP time enabled, and the GPS does not have a timing lock when the device boots, will the producer and consumer wait until the GPS has a lock to start? In the brief testing I did, even if the GPS was not yet locked, issuing the rsh-restart command resulted them both starting up, even though there was no GPS PPS signal.

Also, for full disclosure, this RS is in a custom enclosure and has two other boards in the GPIO stack, with the shake board still being on the top. We are using a Power Hat connected to the 12v solar system to power the pi via the GPIO pins. And also in the stack is a custom PCB also connected through the GPIO that monitors the battery voltage of the 12 volt power system, acting as a smart LVD/LVR. Essentially, it is an LVD/LVR that gives the PI time to shutdown gracefully. Then, when the LVR condition is reached, it issues a reboot to the PI board to start it up again. As best we can tell, none of the GPIO pins are interfering with each other. But if we cannot find any other reason why this is happening, then we will start diving further into any possible issues with the sleep/wake controller.

Thanks,
Andrew

2

RSH.R7C23.2025-05-14T23_56_00.logs.tar (3.0 MB)

Here are the log files. The point where I tried to boot it “yesterday” without any GPS connection, and had to force the consumer/producer to start would have been noted as May 4th 2025. Then it will have suddenly gotten a GPS lock and jumped to the correct UTC Date/Time.

3

Hello Andrew,

Thank you for the extensive explanation and the log files you sent. They were surely enlightening.

The main issue you’re facing is a power supply one. The logs are full of these messages:

2025 124 19:40:55>>	No Data has been received from the MCU in 12 read attempts.It appears the MCU is not transmitting data.  This is a fatal condition and should be investigated if this condition persists!
2025 124 19:40:55>>	Data has been successfully received, fatal condition resolved.
2025 124 19:40:55>>	internal error: buffer overflow!  cannot process read data...
2025 124 19:40:55>>	buf: {'MA': 'RS3D-7-4.11','DF': '1.0','CN': 'EH1','TS': 0,'TSM': 0,'TQ': 45,'SI':  5000,'DS': ['5A91B','5879A','41E09','2A112','193F8','FFFF7EFE','FFFC3362','FFF9A621','FFFA32DC','FFFD71C2','E608','24ED8','2636F','1A5C0','3113','FFFFD4F0','2353A','46ADC','506CD','5D736','75DB3','84263','72735','417C6','FFFD47B2','FFF4CD8E','FFF0D418','FFF3F7E8','FFFB6001','FFFFD283','FFFE9943?E???ÂȘc??W:

As you can see, errors are mixed with some ‘gibberish’ that cannot be successfully interpreted. My recommendation would be to check and see if the current power supply you are using is continuing to deliver a stable voltage between 5.0 and 5.2V and a current of at least 2.5A at all times, as a decrease in power could lead to data services interruption. If similar errors continue to happen to happen, and if you have another Pi power supply that you know is in working condition, please try to exchange the current one with that, and see if the Shake now properly works for longer periods.

Due to lack of power supply, you’ll then have (in a chain) GPS timing and Consumer/Producer issues, where the GPS will not be constantly powered up (or receive less power than what’s needed to operate nominally), and cause what you’re seeing:

2025 124 19:46:26: GPS clock not yet fixed...
2025 124 19:47:24: GPS clock not yet fixed...
2025 124 19:48:25: GPS clock not yet fixed..

To reply to this:

Ie, if there is no NTP time enabled, and the GPS does not have a timing lock when the device boots, will the producer and consumer wait until the GPS has a lock to start?

If an NTP server is not available, the timing will start incorrect and will remain incorrect until an internet connection is found (or GPS lock is acquired) and the Raspberry Shake rebooted. When the computer time is very much different from real time, the computer must be rebooted in order for NTP to reset the clock successfully, i.e., resetting the clock can only happen during the computer’s boot-up sequence. Please note: this is a Raspberry Pi feature, not a Shake feature.

Data will still flow locally and use the system time for time stamping (It always uses the computer clock, whether NTP is running or not. The difference is if the clock disciplined by NTP). If there is no NTP, however, Raspberry Shake will recognize that the timing quality has been compromised and will not stream the data to the community server.

If nothing changes after examining/changing the power supply and other custom equipment, then we’ll have to explore other avenues.

Let us know what you find!

4

Thank you very much @Stormchaser. That’s interesting, I’m running it off a benchtop power supply right now, at 13v or so, connected to a pi hat that should be creating a stable 5v 3a. I’ll look into that.

Is there an easy way to monitor if my system time matches the time from GPS PPS. Ie, does the GPS PPS time get noted anywhere, if there is a lock, but ntp hasn’t set the clock. I’m wondering about creating my own daemon I can run that monitors the GPS clock and system clock, and when there is a mismatch when the GPS PPS is locked, initiates a PI reboot in order to force the clock to sync properly.

1 Like
5

No trouble at all Andrew!

To check for any undervoltage events, you can exec this from the command line:

sudo zgrep -a -i voltage /var/log/syslog*

As it will provide all the lines in the log files where voltage issues have occurred. The power requirements (also for other readers) are 5.1V @ 2.5A (for Pi 3B) and 3A (for Pi 4B), and if you’re using a benchtop power supply, it should be easy to check things out. Also, try to replace the power cable(s) if the supply is providing what’s needed, but undervoltage events still happen.

Always from the command line, if you execute

ntpq -p

you’ll have a list of the NTP servers the daemon knows about (both local GPS and internet-based NTP servers), also indicating which server it is using. In particular, the line with * is the server that the NTP daemon is currently using while the line with + is a server that is a valid candidate if the NTP daemon needs to change to another.

You can find more details about NTP and GPS timing here: NTP and GPS timing details.

If you have any other questions, always feel free to ask.

6

This is great, thanks Stormchaser. I think that was the issue. :slight_smile:

2 Likes