I shut down my shake and connected a Ublox 7-based GPS receiver to the USB port. After restarting, I quickly accumulated 1.3 GB of error messages in odf_SL_plugin.err. Most lines read like the following" :2020 320 23:54:48>> No Data has been received from the MCU in 2191 read attempts.It appears the MCU is not transmitting data. This is a fatal condition and should be investigated if this condition persists!"
This is not an NTP issue–myshake.out shows 3 network NTP servers, plus the gps and pps devices. Will try to upload the logs tarball, but it is so large the server may not allow it.
Without logs, it’s a bit difficult to discern the exact cause of this. Can you try to edit the ‘odf_SL_plugin.err’ file, leaving maybe only the lines of the last day and then try to upload the zipped package again?
You’re right about the fact that a MCU issue is not related to NTP servers connection, as we explored in our last discussions in the other topic.
Here is a zip of the logs, with the biggest one edited down. Some files seem to be missing from the original tarball. Something is seriously wrong here…
Thank you for the logs, and yes, I am seeing that many files are definitely missing.
Are these logs fresh from a restart or are they taken while the Shake was operating? If the second is trye, can you also please try to do this: shut down your Shake from the rs.local interface, wait for 5 minutes, then turn it on and after a couple of minutes download the logs again.
If the odf_SL_plugin.err file is still big, cut it down in the same way you did for this one.
I want to see if there are discrepancies after a restart compared to what I am seeing now.
Thank you for the new logs James, they confirmed what I was suspecting from the partial ones that you sent yesterday.
It seems that there is some kind of corruption in the Shake OS in the SD card. These strange lines in `odf_SL-plugin.err’ file demonstrate this:
2020 322 19:23:58>> QAºEBaƒÐ‰ÃÔE! E 2020 322 19:23:58>> !˜ªQš 2020 322 19:23:58>> 16168 269 2020 322 19:23:58>> No Data has been received from the MCU in 12 read attempts.It appears the MCU is not transmitting data. This is a fatal condition and should be investigated if this condition persists!
Usually, the causes of such a report are two:
unstable connections between the Shake board, the Raspberry Pi and the sensors
microSD card corruption
Since we already checked the first in our last topic, what you can try to do is to format the microSD card and reinstall the Shake OS from scratch. Here’s the procedure:
Please take the microSD card you have and format it before burning the new Shake OS image
Make sure, when you format it, that the shown capacity is the maximum (i.e. if the SD card is 32GB like yours, then the capacity should be near or equal to that value). This is to check that no partitions have been involuntarily and erroneously created. They could be the cause of the error we see now
Take care to format the SD card in a FAT32 File System (or exFAT if the SD card is 64GB or larger -not your case)
I suspected I would have to rebuild the system image. OK, that has been done. The shake is not trying to rapidly fill up the sd card any more, and it looks like it is using the gps as a source of time data.
But the original problem of (large) gaps in the local data, data overruns and intermittent mcu not transmitting has returned. It is not a temperature issue this time (current outside temp is 76 F/ 24 C), I think we have thoroughly eliminated NTP as an possibility, and I am not able to locate any bad connections between the geophone, shake board, and the Raspberry Pi.
Attached logs are after about 1 hour of operation. What do we try next?
Thank you for the new logs, James, now they indeed work and display the files as they should. I can confirm what you said about the operations of the Shake.
Data overruns are there, but to my memory, they seem much less in number compared to before, which can be classified as a good thing.
The strange characters, however, are still there in the odf_SL-plugin.err file, as I am sure that you have noticed. We have re-burned the card, we have checked the connections, it is not a temperature issue, so now the only thing remaining to check is to see if the fault is in the Raspberry Pi board.
If you have another one available around, please swap them (possibly, you will have to re-burn the microSD card again) and see if the issue continues to be present.
I ordered a bare shake board from the online store today. It will probably be a week or two before it arrives but when it does, I will swap it in and see if there is any improvement. I have long suspected a bad solder joint on the shake board as the cause of the problem because of the temperature sensitivity and because the Pi appears to otherwise be working properly. If there is no improvement, then I will try another Pi, since they are easily obtained. Will post an update when I have new info.