Dear everybody,
not even a week ago I received my new raspberry shake 3D V5 (R21E0). And I have to say I am extremely pleased with the build quality and the ease of use!
Most issues occuring are related to the swarm software whose developers I will contact directly. However, there is an extremely weird signal occuring in the low frequency range, which I am particularly interested in.
With the correct, quite specific settings (spectrogram, 50s window length, 0-0.5Hz) one sees bursts every 124.2-124.5s in the 10-150mHz range (up to 17th harmonic!).
This effect does not change no matter where I locate the device. Actually I was able to see it in many RS3D devices around the world, e.g. R77C3, R0BBA, R84CF. Yet I couldn’t find this signal in the probably more professional devices of the Akutan network which leads me to the assumtion that it is specific to the RS3D or its couplings (magnetic field?).
The bursts increase in strength until 00:00 UTC and will happen a very last time after that before almost completely dying down and only slowly coming up over the course of the next ‘day’. As the bursts are mostly in synch around the world, it seems one can exclude large scale (global) magnetic and electric fields, so it must happen from the assembly itself.
The bursts will mostly show up in EHE, less in EHN and not at all in EHZ.
One guess we came up with is it’s a full 1-day data chunk being sent over the header pins over and over again. After 00:00 the day would be ended and less data would be sent…
Since I know that professional broad band seismometers can severly suffer from magnetic fields I assumed this issue might be related to magnetic fields caused by the rpi/shield. Therefore, I shoved a sheet of 150um mu metal foil into the case in between rpi and sensors which didn’t help too much.
Do you have any insights for us what this signal could be, where (on the board) it originates from and how I could get rid of it?
Cheers,
Tobias
Some graphs:
R21E0 EHE 10-100mHz 2nd order zero lag bandpassed, always noise->negative->positive->noise
R21E0 EHE around 2020/01/05 00:00UTC - ‘signal’ dying down shortly after midnight
R21E0 EHE before mu-metal installation started 2020/01/07 13:26UTC
R21E0 EHE after mu-metal installation finished 2020/01/07 13:41UTC
hi tobias,
i notice that you have the WiFi module turned ON. is this what you want? we have seen that for some Pi models the built-in WiFi module can have an adverse effect, producing a spike approximately every two minutes when it attempts to scan for new networks.
To disable the WiFi from being ON when using ethernet, see the file /opt/settings/user/enable-wifi.conf and adjust accordingly.
If you do want to use WiFi, we recommend attaching an external dongle; in this case, the module is far enough away from the hardware that there is no spike.
If you still see this after WiFi is disabled, please post your log files again, along with any other information you may discover.
warm regards,
richard
Hi Richard,
thanks a lot for your reply. In fact I read the advice not to switch wifi on as it may cause noise, but I expected it was off by default. Wouldn’t it make sense to at least add a checkbox to the webserver?
Anyway, wifi (and bluetooth) is switched off in our device now and I was waiting for late hours - as I said the height of the bursts increases over the course of the day.
20 min around 01/08 20:00 UTC - wifi off
It appears to be a bit more windy out there which is why I decided to add spectra for comparison
To me it does not appear that the ‘signal’ (8mHz comb) changed significantly.
90min around 01/07 20:00 UTC - wifi on
90min around 01/08 20:00 UTC - wifi off
Maybe one piece of additional information: at this moment we have to use a USB network dongle. However, as the issue occurs at many (all RS3D?) stations I guess this is not the origin and one should concentrate on what these stations have in common?
It seems the shield layout changed over time which might change electromagnetic couplings. Is there a way to filter for RS3D stations earlier than V5?
Cheers,
Tobias
The latest log file:
RSH.R21E0.2020-01-08T22_33_47.logs.tar (1.5 MB)
Tobias:
Interesting find! Excellent observations. I took a look at the test data for your unit and I also see this phenomena during the testing period here in the lab. I confirm your observations that:
-
The bursts will mostly show up in EHE, less in EHN and not at all in EHZ.
-
There are multiple harmonics. I count ~20. Their amplitude rises from 0.01 Hz and then falls as you approach 0.2 Hz:
-
This is frequency band limited to to the 0.01 to ~0.2 frequency band, or below the corner frequency of the RS3D, which is ~0.7 Hz.
-
This effect does not change no matter where the device is located, as evidenced by the presence of the signal here in the test lab and at your home.
-
This is not unique to your RS3D but characteristic of them.
-
The bursts increase in strength until 00:00 UTC and will happen a very last time after that before almost completely dying down and only slowly coming up over the course of the next ‘day’. This suggests to me that the problem is firmware/ software related (and not wifi-related).
-
The bursts are mostly in sync around the world - they are not coincident in absolute time, but the spacing is consistently ~125 ms between peaks.
-
Signal pattern: always noise->negative->positive->noise
I will add:
- Same spectral spike frequencies are seen across all units in a test bed
We will look into it and see what we can determine. Any additional observations are welcome.
branden
Yesterday I decided to investigate this issue once more.
Diving deeper into Brandon(1) I had a look at the ADC-shield. The coupling strength decreases with distance from RPi USB/Lan ports. But then there is also the SMSC LAN9514 chip right underneath the east channel electronics. The Geophone cables come out of the shield’s bottom and run right across the Lan/Usb chip. Hence I decided to re-solder the Geophones from the top and even added some copper shield over the bottom of the solder pads to close the broken ground plane.
(the bottom got a second kapton layer where only the hole-surrounding gnd pads were exposed)
90min around 01/10 19:45 UTC - cables from top
This didn’t improve the situation
In my desperation I added a (not quite closed) mumetal shield around the ADC-shield for magnetic and electric shielding with a layer of isolation to the LAN port. The hope was that at low frequencies the holes don’t reduce effectiveness.
90min around 01/10 22:50 UTC - mu-metal shield
This time it didn’t just not help, we got additional signals which also seem to die down at midnight UTC. So I’ll revert this second change next week, but the cables will remain at the top of the shield…
At this point I feel that the problem is less likely to be hardware coupling to geophones related.
hello tobias,
first off, thank you very much for the effort you have put into tracking down this problem yourself; we are a small outfit here in panama and are very appreciative when users help us out with their talents and expertise in tracking down these sorts of hard-to-find issues.
after running some process-of-elimination testing this weekend, we are fairly confident that we have found the source of the spikes you identified. and the good news is that this is software-related. good news, obviously, since SW is much easier to tweak to solve the issue than HW.
to get an independent confirmation, can you run one more test and let us know what your spectra looks like after running for a while? please issue the following command on the command line of the Shake Pi:
> sudo systemctl stop rsh-data-consumer
this will stop the OWS process which provides data to the helicorder program and a local Swarm connection (if Swarm is running). since these data-producing services will not be active, there will be no helicorder data nor will a local Swarm connection work.
however, data will still be written to disk using seedlink, so the trace files will be accessible either from the data server (when that is turned on), or by copying the trace files using scp directly off the Pi itself.
cheers, please let us know your findings,
richard
Hi Richard,
thank you so much! We apprechiate the efforts you put in and the short response times very much.
We applied your fix at 20:55UTC
so I would judge this a full success.
When looking at the spectra before
and after
it’s not only the 8mHz comb that is disappearing, but also the 34mHz comb which we started seeing recently. The 2Hz line seems to be an unrelated mess, occasionally originating from our lab…
It’s a pitty one cannot have the near realtime local swarm connection AND good low frequency performance in the stored data at the same time.
But yay it’s finally time to have a look whether there are systematics (seismic activity) below 100mHz, or whether its pure readout noise 
…and the mu-metal pimp will stay for now until we see actual drawbacks apart from looking less clean than the virgin.shake 
Cheers,
Tobias
hi tobias,
thanks for the confirmation from your side.
we will be actively looking to find a SW-based solution to this problem, where access to the data will still be possible (helicorder and local swarm), though not causing these undesirable spikes.
cheers, we will keep you updated,
richard
Hello,
I was wondering if this issue was corrected in an update, or if I would still need to stop the OWS process to avoid this noise source?
Stefano
hi,
an update on this: we have not yet been able to identify a solution where both OWS is running and it doesn’t cause the spikes in question. until we are able to solve this, it is recommended to stop the OWS process and retrieve the data directly from the server using swarm of FDSNWS data-fetch services.
cheers,
richard