Using Filespooler Without Queues to Pass More Metadata

One frustration people sometimes have with ssh or NNCP[1] is that they'd like to pass along a lot of metadata to the receiving end. Both ssh and nncp-exec allow you to pass along command-line parameters, but neither of them permit passing along more than that. What if you have a whole host of data to pass? Maybe a dozen things, some of them optional? It would be very nice if you could pass along the environment.

Filespooler[2] lets you do just that. As mentioned in the Filespooler Reference[3], any environment variable beginning with `FSPL_SET_` that's set when `fspl prepare` is called is written into the header and will also be set for the command run by `fspl queue-process` or `fspl stdin-process`.

The support added in Filespooler 1.0.8 for not using a sequence file on the sender makes this easier.

Consider this example:

FSPL_SET_FOO=123 fspl prepare -s - | \
  fspl stdin-process bash -- -c set | grep FSPL_SET_FOO
FSPL_SET_FOO=123

Of course, this can be combined with ssh:

FSPL_SET_FOO=123 fspl prepare -s - | \
  ssh otherhost fspl stdin-process bash -- -c 'set' | \
  grep FSPL_SET_FOO
FSPL_SET_FOO=123

Of course, you can both set the environment *and* use stdin. Let's say you wrote this script and saved it in ~/test.sh:

#!/usr/bin/env bash

set -euo pipefail

echo "Hex dump of $FSPL_SET_FILE:"
hd

Now you can run this service remotely:

echo hi | \
  FSPL_SET_FILE=hi.txt fspl prepare -i - -s - | \
  ssh otherhost fspl stdin-process ~/test.sh
Hex dump of hi.txt:
00000000  68 69 0a                                          |hi.|
00000003

This technique also works with nncp-exec; see also Using Filespooler over NNCP[4].

--------------------------------------------------------------------------------

Links to this note

• Using Filespooler over NNCP[5]

NNCP[6] is a powerful tool for building Asynchronous Communication[7] networks. It features end-to-end Encryption[8] as well as all sorts of other features; see my NNCP Concepts[9] page for some more ideas.

• Filespooler[10]

Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing[11], NNCP[12], ssh, UUCP[13], USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler.

Filespooler is particularly suited to distributed and Asynchronous Communication[14].

More on www.complete.org

(c) 2022-2024 John Goerzen