We have remote systems collecting in India and Bhutan, In India there is an always on internet connection used to sanity check in ZeroTier things are live still, Bhutan is more infrequent Starlink when enough Power. They do not stream to the Shake server as issues around data share in these regions.
We logged in this week to the India station and confirmed the save data setting was 360 days, but, it was purging on the 7 day loop and has been since install (March). Looking at the log files it looks like 4/03 the OS went to 21.1 with the bug around save data days not sticking, so the 360 setting was doing nothing. 29/03 the OS went to 21.2 with the bug fix (we did not log in, but assume internet connection even if not sending to Shake Servers allowed update). BUT, we of course did not reboot as no reason to we thought and did not know 21.2 installed. It looks like despite 21.2 auto-install the previous bug of the 360 days data retention not sticking persisted as the patch did not apply the data save policy? I rebooted this week with 30 days and that has stuck and working as expected
Here is what my ChatGPT analyses of logs gets: the most likely story is that the 360-day value did not stick under 21.1, the 21.2 update did not automatically re-apply it, and the unit carried on purging on the old 7-day setting until the setting was later changed/applied as 300 days around 17 June with a reboot.
This is the only way I can explain what is happening accross units. This is pretty catastrophic for the projects given the crtical data lost due to be recorded at the same time as other sensors, and, the cost of doing Indian/Bhutan installs. What we have is a folder full of .gifs (that kept recording despite any setting changes - normal??) but only 7 days of miniseed logged.
So,
i) can we confirm that updates are autoserved on any internet connection, even if not sending to Shake servers
ii) from the log files what I think happened is what happened or something else
iii) when 21.2 fixed that data save bug moving forward, it did not force apply the setting that showed on the rs.local and SSH but was not applied. ie it needed a manual reboot or it did reboot but deafaulted to 7 days again as the 360 was not saved.
iv) if the above is true, what is the way to stop bugs causing critical issues like this for users in future?
i’ve had a look at your log files and can report that the unit did reboot after the upgrade to V21.2, containing the fix for the save data days issue in V21.1. for all system updates, the update sequence always ends with a system reboot. thus, that the setting is now working correctly for you is not a result of the manual reboot. more likely, it is due to the reset of the value itself. fyi: the myshake.out file indicates that the setting is now 300 days, not 30 days as you say in your note.
to answer your questions:
i) yes, updates to all shakes will take place if the unit is connected to the internet, whether the unit is sending data to the data servers or not
ii) as stated above, since the unit was rebooted, it is unclear why the setting was not restored to 360 after the update to V21.2. again, a second reboot, manual or otherwise, would not cause a change in how the fix would be applied or not
iii) the fix in V21.2 was thoroughly tested before being released. that is, for those units which were referencing the wrong value in V21.1, after the update V21.2 was installed and the unit rebooted, the new value was correctly put into effect
iv) for this particular issue, the bug fix has been indeed implemented such that this will not be possible to be reintroduced by any new update in the future. for the totality of all functionality provided by the Shake-OS, all updates are put through a series of regression tests in order to avoid situations such as these. however, from time to time, an unforeseen consequence of a change can make it into the release. when we become aware of this, it is always our intent to issue an update to address the issue as soon as we can.
i hope this helps to answer your questions about how the Shake-OS updates are tested and released.
Thanks for the response. Sorry for the typo, it was 300 I changed to recently, not 30.
Ok, so it did reboot when the upgrade happened - that is good.
So, not sure I followed though - if the setting we put in did not ‘stick’, when it rebooted what value would it use? Would it be the 360 value we had entered when we had the 21.1 that did not stick/apply the 360 despite showing it, or, would just apply the 7 days that was ‘stuck’ in the system incorrectly as the setting, as 21.1 stopped the 360 being applied? If the latter, it explains our data loss.
yes, the bug in V21.1 was that the value set in the front-end configuration app would not end up being applied, using the default value of 7 instead. V21.2 fixed this where the config app’s value, previously set, would be used correctly. that is, no further user intervention was necessary for the fix to take effect.