Local data retrieval - C bindings or SDK?

Technical Support
1

Proud owner of a new 4D. Unless I’ve misread the documentation, it looks like the Shake requires an internet connection to ShakeNet in order to record and retrieve data? Unfortunately that won’t be possible in my application. Instead, do you have a local API or a C program I can bind to retrieve data on the pi itself? Streaming is a preferred option since we want to send it over LoRa in real-time.

My problem is that the station will be remotely installed and our communication is going to use a low power protocol known as LoRa and we’ll be writing our own networking layer on top of it.

We’re writing in Rust so bindings to C/C++ are totally doable. Alternatively, if you could point me to some docs that may cover this already that’s helpful, too.

Thanks for your help.

Edit: it looks like there may be a way to tap into raw UDP packets, perhaps locally? https://github.com/raspishake/rsudp/tree/master/rsudp

2

Hi @crockpotveggies—all data is recorded to the SD card, and saved for 7 days by default, under /opt/data/archive. The easiest and most reliable thing to do may be to pull that data over SSH.

You can increase the days saved by default in the web front end settings. For a 4D with an 8GB card I wouldn’t recommend much over 60 days as the card will fill up.

Yes, rsudp will also work locally, however because it relies on UDP it is not as reliable as grabbing the data directly from disk. If you do want to try using rsudp to retrieve data and save to disk, the following settings configuration will work for you (although I recommend changing "station": "Z0000", to your own station callsign, Rxxxx). Data will be recorded to daily miniseed files (same format as on disk) in the ~/rsudp directory on your local machine:

{
"settings": {
    "port": 8888,
    "station": "Z0000",
    "output_dir": "~/rsudp",
    "debug": true},
"printdata": {
    "enabled": false},
"write": {
    "enabled": true,
    "channels": "all"},
"plot": {
    "enabled": false,
    "duration": 300,
    "spectrogram": true,
    "fullscreen": true,
    "kiosk": false,
    "eq_screenshots": true,
    "channels": ["HZ", "HDF"],
    "deconvolve": true,
    "units": "ACC"},
"forward": {
    "enabled": false,
    "address": "127.0.0.1",
    "port": 13000,
    "channels": ["EHZ"]},
"alert": {
    "enabled": false,
    "highpass": 0,
    "lowpass": 8,
    "deconvolve": false,
    "units": "ACC",
    "sta": 6,
    "lta": 30,
    "threshold": 1.581,
    "reset": 1.574,
    "exec": "eqAlert",
    "channel": "HZ",
    "win_override": false},
"alertsound": {
    "enabled": false,
    "mp3file": "doorbell"},
"tweets": {
    "enabled": false,
    "tweet_images": true,
    "api_key": "n/a",
    "api_secret": "n/a",
    "access_token": "n/a",
    "access_secret": "n/a"}
}

And of course you will need to forward data to port 8888 on your local machine by changing the UDP datacast settings in the Shake’s web front end.

3

Out of curiosity why would the UDP data be unreliable on a local machine? Is this Shake-specific or are you referring to the nature of the UDP protocol itself?

4

It has weaknesses as an archival tool due to the UDP protocol itself, the fact that rsudp doesn’t have any data backfilling algorithm, and the fact that the process relies on constant network availability in order to maintain gap-free data archival on the remote machine. UDP is fantastic for sub-second event detection. For complete data archival however, it’s much more reliable to grab the records from the source. (if I’ve misunderstood your use case please correct me)

edit 2020-01-30: corrected the phrase “gap-free data archival on the local machine” to “gap-free data archival on the remote machine”

5

I think we’re almost there, but if I understand you correctly are you saying the Shake does network requests to backfill gaps in local data? Otherwise let’s assume I do the following:

  1. Open rs.local and create Datacast called 127.0.0.1:8080
  2. Run a simple, local application named UDPtoLoRa on the same Shake bound to port 8080
  3. UDPtoLoRa then has a lossless way of transmitting the data to an external server

Because I’m not relying on external UDP connections I can take full responsibility for data loss. The goal is to detect events in real-time and only transmit waveform when there’s enough activity (helping to save battery power). I hope that makes sense?

So unless there’s an external network doing local backfilling then I think data loss would ultimately become my responsibility?

6

No, sorry for the miscommunication, I mean network availability is required to maintain gap-free data in the remote, non-Shake box, when using rsudp. Data is automatically recorded locally, i.e. on the Shake’s SD card.

Yes, this would be the way I would do it. rsudp has a lot of code that is along these lines and may be helpful (albeit written in Python), so feel free to take a look at that codebase for inspiration.

7

Sounds great thanks for the back-and-forth. This will probably be open-source so I hope to share on the forums when it’s ready.

2 Likes
8

Fantastic! We can’t wait for you to share it.

9

hi,

i’m arriving a bit late for the show, but one thing i would like to add:

  • the data-producer program automatically sends data to the local machine over port 8888; there is no need to explicitly specify ‘localhost’ to have the data case locally using UDP protocol.

  • so, just configure rsudp to read from port 8888 on localhost and data will be ingested as expected.

cheers, and thanks for being willing to contribute your solution to the community!

richard