Using Filespooler over Syncthing
Filespooler[1] is a way to execute commands in strict order on a remote machine, and its communication method is by files. This is a perfect mix for Syncthing[2] (and others, but this page is about Filespooler and Syncthing).
This page also functions as a tutorial for Filespooler.
While I talk about Syncthing in particular here, the instructions here apply to pretty much any directory-synchronization tool, such as Dropbox, Box, etc. All that's required is that it meet the requirements laid out in Guidelines for Writing To Filespooler Queues Without Using Filespooler[3]. The most important one is that when a file appears in the `jobs` directory with a name matching `fspl-*.fspl`, it must be ready to process. Most sync tools will save files to a temporary filename while syncing them, then rename them when done, which meets these requirements.
Preparation
Before you can use Filespooler, of course you will need to install it. This is quick and easy; see the Installation section on the Filespooler homepage[4] for details.
We are going to assume we have two machines: one that is the "sender" and one that is the "receiver". We're going to implement a service that can encode or decode a payload using base64(1). The base64 command is part of most Unix/Linux distributions, and is included in GNU coreutils. We will be using Syncthing as the transport for this service!
Creating the queue
For the sake of this demonstration, we will create the queue in an already-existing directory. Let's say you have a directory already being synced between the two computers called `~/sync`. Let's make a queue in there:
sender$ fspl queue-init -q ~/sync/b64queue
As with most Unix commands, if it has no output, it worked properly.
All `fspl` commands that work with a queue take a `-q queuedir` parameter (or `--queuedir` if you prefer). In this case, it gives the path to the queue we will be creating.
Let's see what's in that directory:
sender$ ls -lR ~/sync/b64queue /home/jgoerzen/sync/b64queue: total 3 drwxr-xr-x 2 jgoerzen jgoerzen 2 May 16 20:17 jobs -rw-r--r-- 1 jgoerzen jgoerzen 2 May 16 20:17 nextseq -rw-r--r-- 1 jgoerzen jgoerzen 0 May 16 20:17 nextseq.lock /home/jgoerzen/sync/b64queue/jobs: total 0
The `~/sync/b64queue/jobs` directory will hold the data about the actual jobs we want to process. It is written to by the sender and read from by the receiver. `nextseq` and `nextseq.lock` are files that help the queue processor do its job. They should only be touched by the receiver, never by the sender.
Sending a first request
Now, we're going to use `fspl prepare` to prepare a request. This command will:
- Write a `fspl` job file (or "packet") to stdout
- Optionally read input from stdin and send it to stdout as a "payload". This payload will later be piped to the command we run at queue processing time.
- Optionally supply environment variables or command-line parameters to be used at queue processing time.
Now since this command writes its output to stdout, we need a way to save it to the queue. You can, of course, Write To Filespooler Queues Without Using Filespooler[5]. But it's easier to just use a Filespooler command: `fspl queue-write`. This is separate from `fspl prepare` because you might want to encode the job file in your pipeline (see, for instance, Compressing Filespooler Jobs[6] and Encrypting Filespooler Jobs with GPG[7]).
That may have sounded complicated, but let's put the pieces together and it will be easy:
sender$ echo Hi | fspl prepare -s ~/b64seq -i - | fspl queue-write -q ~/sync/b64queue
Let's break that down:
- First, we pipe "Hi" to `fspl prepare`.
- `fspl prepare` takes two parameters:
* `-s seqfile` gives the path to a *sequence file* used on the sender side. This file has a simple number in it that increments a unique counter for every generated job file. It is matched with the `nextseq` file within the queue to make sure that the receiver processes jobs in the correct order. It MUST be separate from the file that is in the queue and should NOT be placed within the queue. There is no need to sync this file, and it would be ideal to not sync it.
* The `-i` option tells `fspl prepare` to read a file for the packet payload. `-i -` tells it to read stdin for this purpose. So, the payload will consist of three bytes: "Hi\n" (that is, including the terminating newline that `echo` wrote)
- Now, `fspl prepare` writes the packet to its stdout. We pipe that into `fspl queue-write`:
* `fspl queue-write` reads stdin and writes it to a file in the queue directory in a safe manner. The file will ultimately match the `fspl-*.fspl` pattern and have a random string in the middle.
At this point, wait a few seconds (or however long it takes) for the queue files to be synced over to the recipient.
Inspecting the queue
On the receiver, we can see if any jobs have arrived yet:
receiver$ fspl queue-ls -q ~/sync/b64queue ID creation timestamp filename 1 2022-05-16T20:29:32-05:00 fspl-7b85df4e-4df9-448d-9437-5a24b92904a4.fspl
If you have an empty output, you could examine the `~/sync/b64queue/jobs` directory to see if it actually does contain files synced over from the sender.
Let's say we'd like some information about the job. Try this:
receiver$ $ fspl queue-info -q ~/sync/b64queue -j 1 FSPL_SEQ=1 FSPL_CTIME_SECS=1652940172 FSPL_CTIME_NANOS=94106744 FSPL_CTIME_RFC3339_UTC=2022-05-17T01:29:32Z FSPL_CTIME_RFC3339_LOCAL=2022-05-16T20:29:32-05:00 FSPL_JOB_FILENAME=fspl-7b85df4e-4df9-448d-9437-5a24b92904a4.fspl FSPL_JOB_QUEUEDIR=/home/jgoerzen/sync/b64queue FSPL_JOB_FULLPATH=/home/jgoerzen/sync/b64queue/jobs/fspl-7b85df4e-4df9-448d-9437-5a24b92904a4.fspl
This information is intentionally emitted in a format convenient for parsing.
Running the queued jobs
Well, now that we've done all this, let's get to business and run the jobs!
receiver$ fspl queue-process -q ~/sync/b64queue --allow-job-params base64 SGkK
There are two new parameters here:
- `--allow-job-params` says that the sender is trusted to supply additional parameters for the command we will be running.
- `base64` is the name of the command that we will run for every job. It will:
* Have environment variables set as we just saw in `queue-info`
* Have the text we previously prepared - "Hi\n" - piped to it
By default, `fspl queue-process` doesn't do anything special with the output; see Handling Filespooler Command Output[8] for details on other options. So, the base64-encoded version of our string is "SGkK". We successfully sent a packet using Syncthing as a transport mechanism!
At this point, if you do a `fspl queue-ls` again, you'll see the queue is empty. By default, `fspl queue-process` deletes jobs that have been successfully processed.
Demonstrating Ordering
Let's make a demo to see if ordering is always preserved. We'll intentionally inject jobs into the queue in the wrong order:
sender$ echo first | fspl prepare -s ~/b64seq -i - > /tmp/first sender$ echo second | fspl prepare -s ~/b64seq -i - | fspl queue-write -q ~/sync/b64queue
So we only injected the second packet into the queue. What happens on the receiver?
receiver$ fspl queue-ls -q ~/sync/b64queue ID creation timestamp filename 3 2022-05-16T20:22:19-05:00 fspl-7671ed24-8cb7-46fc-baff-1c99ceb6cfc9.fspl
The packet we saved off as "first" is ID 2; we can verify it this way:
sender$ fspl stdin-info < /tmp/first FSPL_SEQ=2 FSPL_CTIME_SECS=1652750533 FSPL_CTIME_NANOS=412072059 FSPL_CTIME_RFC3339_UTC=2022-05-17T01:22:13Z FSPL_CTIME_RFC3339_LOCAL=2022-05-16T20:22:13-05:00
`fspl stdin-info` does the same thing as `fspl queue-info`, just for things on stdin. Most `fspl queue` commands have a `fspl stdin` version also.
OK, so we have set up an out-of-order situation in the queue. What happens when we try to process the queue:
receiver$ fspl queue-process -q ~/sync/b64queue --allow-job-params base64
Nothing! Because we are missing packet 2 in the queue. If we increase the log level, we can see why:
receiver$ $ fspl --log-level debug queue-process -q ~/sync/b64queue --allow-job-params base64
DEBUG prepare_seqfile_lock{path="/home/jgoerzen/sync/b64queue/nextseq"}: filespooler::seqfile: Attempting to prepare lock at "/home/jgoerzen/sync/b64queue/nextseq.lock"
DEBUG open{path="/home/jgoerzen/sync/b64queue/nextseq"}: filespooler::seqfile: Attempting to acquire write lock
DEBUG open{path="/home/jgoerzen/sync/b64queue/nextseq"}: filespooler::seqfile: Attempting to open file "/home/jgoerzen/sync/b64queue/nextseq"
DEBUG scanqueue_map{queuedir="/home/jgoerzen/sync/b64queue" decoder=None}: filespooler::jobqueue: Reading header from "/home/jgoerzen/sync/b64queue/jobs/fspl-7671ed24-8cb7-46fc-baff-1c99ceb6cfc9.fspl"
DEBUG filespooler::cmd::cmd_exec: Sequence processing: Stopping processing as there is no job with the next sequence ID to process
OK, let's move our saved packet into the queue:
sender$ fspl queue-write -q ~/sync/b64queue < /tmp/first sender$ rm /tmp/first
And now we should see it process two packets:
receiver$ fspl queue-process -q ~/sync/b64queue --allow-job-params base64 Zmlyc3QK c2Vjb25kCg==
And indeed, it processed two packets -- and in the correct order! (You can tell because "first" is shorter than "second").
Passing Parameters
I mentioned before that you can pass parameters. I also indicated that we would build an encoder *and decoder*. The base64 command accepts a `-d` to decode. So, how about we try it? Let's use the encoded strings we've been playing with and see if we can get the original back out:
sender$ echo SGkK | fspl prepare -s ~/b64seq -i - -- -d | fspl queue-write -q ~/sync/b64queue sender$ echo Zmlyc3QK | fspl prepare -s ~/b64seq -i - -- -d | fspl queue-write -q ~/sync/b64queue sender$ echo c2Vjb25kCg== | fspl prepare -s ~/b64seq -i - -- -d | fspl queue-write -q ~/sync/b64queue
OK, now we should have 3 jobs in the queue:
receiver$ fspl queue-ls -q ~/sync/b64queue ID creation timestamp filename 4 2022-05-16T20:30:40-05:00 fspl-ef38c718-59b1-4302-ae80-df3ec8e9ee46.fspl 5 2022-05-16T20:31:10-05:00 fspl-34063b9b-9393-4f60-9a2e-d7ff1b8456bf.fspl 6 2022-05-16T20:31:27-05:00 fspl-f330e9ac-16ba-4517-b6bb-523c81ef6fa6.fspl
Good. Let's inspect one:
receiver$ fspl queue-info -q ~/sync/b64queue -j 4 FSPL_SEQ=4 ... FSPL_PARAM_1=-d
I omitted most of the output, but notice that `FSPL_PARAM_1`. Since we have been processing the queue with `--allow-job-params`, anything after the `--` on the `fspl prepare` command line will be passed on to the command we execute: in this case, `base64`. So we can mix encode and decode jobs in a single queue. Let's add a fourth job, an encode one, just to be sure:
sender$ echo foo | fspl prepare -s ~/b64seq -i - | fspl queue-write -q ~/sync/b64queue
All right! Let's process the queue:
receiver$ fspl queue-process -q ~/sync/b64queue --allow-job-params base64 Hi first second Zm9vCg==
Yes! That is absolutely correct! The three decode jobs did indeed decode, and the encode job did indeed encode, and they were all processed in the correct order.
Dealing with errors
Thus far, everything we have done has been with commands that succeed. What happens when a command fails? Let's try it.
`base64` can encode anything, but it can only decode things that are valid base64 strings. What if we accidentally send plain text to the decoder? `base64 -d` will exit with an error. Let's see how that works:
sender$ echo Hi | fspl prepare -s ~/b64seq -i - -- -d | fspl queue-write -q ~/sync/b64queue
And on the recipient, we've got one item in the queue:
receiver$ fspl queue-ls -q ~/sync/b64queue ID creation timestamp filename 8 2022-05-16T20:36:49-05:00 fspl-5e8fe338-3c9c-4c77-b207-e7c296fb9ca3.fspl
Let's see what happens when we process this:
fspl queue-process -q ~/sync/b64queue --allow-job-params base64 base64: invalid input ERROR filespooler::exec: Command exited abnormally with status ExitStatus(ExitStatus(256)) Error: Aborting processing due to exit status ExitResult(ExitStatus(ExitStatus(256))) from command
Well we certainly detected the error! Let's decode what is happening:
- The line that says `base64: invalid input` was sent to stderr from `base64` itself. `base64` then exited with an error.
- Filespooler detected that the command exited with an error and told you about it (`Command exited abnormally`)
- Additionally, because of the default behavior for errors, `fspl queue-process` aborted queue processing entirely, and exited with an error.
The packet is still in the queue (run `fspl queue-ls`) to confirm. You can keep running `fspl queue-process` as much as you like, but it will never proceed because this command will always exit with an error. Even if you add 100 more packets after it, because the *first* one has an error, none of the others will be processed because of the default of strict ordering.
This is perfect for many use cases. For instance, if you are Using Filespooler for Backups[9], you probably *want* an incremental that fails because of a full disk to keep failing until you've fixed the disk problem.
What can you do now? There are quite a few choices:
- You can use the `--on-error` parameter to `fspl queue-process` to tell it what to do when there is an error. The Filespooler Reference[10] discusses these options in more detail. One option, `Delete`, will cause the error to be ignored, the file deleted, and the queue processing to continue. However, this may not be what you want, because it will handle *all* errors like that. You might want to combine it with `--maxjobs 1`.
- You can tell Filespooler to skip the file.
Let's look at the second option. In this example, the offending job is ID 8. Recall that Filespooler has a sequence file that tells it what to process next. We can inspect that file:
receiver$ fspl queue-get-next -q ~/sync/b64queue 8
That's as it should be. We can tell it to just skip job 8:
receiver$ fspl queue-set-next -q ~/sync/b64queue 9
That's fine. But job 8 is still sitting in the queue. There is (intentionally, for now) no `fspl queue-rm` command, but it is easy enough to do get rid of job 8 manually:
receiver$ fspl queue-info -q ~/sync/b64queue -j 8 | grep ^FSPL_JOB_FULLPATH= FSPL_JOB_FULLPATH=/home/jgoerzen/sync/b64queue/jobs/fspl-5e8fe338-3c9c-4c77-b207-e7c296fb9ca3.fspl receiver$ rm /home/jgoerzen/sync/b64queue/jobs/fspl-5e8fe338-3c9c-4c77-b207-e7c296fb9ca3.fspl
And done!
Advanced topics and observations
Only syncing the jobs/ subdirectory with an append-only queue
In this example, we synced the entire `b64queue` directory. This is unnecessary. It would be ideal to sync only the `jobs` subdirectory of it, to prevent it looking like a valid queue for processing on the sender. To do this, see Filespooler Append-Only Queues[11].
Getting help
Besides the Filespooler Reference[12], you can get a list of all `fspl` subcommands with `fspl help`. Summaries of options valid for each subcommand are available with `fspl SUBCOMMAND --help`.
Additional topics
- Although the default is strict ordering, this is not required. The Filespooler Reference[13] discusses the `--order-by Timestamp` option to `fspl queue-process` which is a significant change from what's described here.
- Filespooler easily supports a One-To-Many[14] configuration using the concepts laid out above. You can also achieve Many-To-One with Filespooler[15] (or even many-to-many) by using that timestamp ordering.
- You may want to handle output differently; see Handling Filespooler Command Output[16].
- See the other subpages at Filespooler[17] and the Introduction to Filespooler[18].
--------------------------------------------------------------------------------
Links to this note
- Building an Asynchronous, Internet-Optional Instant Messaging System[19]
I loaded up this title with buzzwords. The basic idea is that IM systems shouldn't have to only use the Internet. Why not let them be carried across LoRa radios, USB sticks, local Wifi networks, and yes, the Internet? I'll first discuss how, and then why.
- Filespooler Append-Only Queues[20]
In the Using Filespooler over Syncthing[21] example, we synced the entire `b64queue` directory. This is unnecessary.
- Guidelines for Writing to Filespooler Queues Without Using Filespooler[22]
Filespooler[23] provides the `fspl queue-write` command to easily add files to a queue. However, the design of Filespooler intentionally makes it easy to add files to the queue by some other command. For instance, Using Filespooler over Syncthing[24] has Syncthing do the final write, the nncp-file (but not the nncp-exec) method in Using Filespooler over NNCP[25] had NNCP do it, and so forth.
- Using Filespooler over Rclone and S3, Rsync.Net, Etc.[26]
You can use Filespooler with a number of other filesystems and storage options. s3fs, for instance, lets you mount S3 filesystems locally. I can't possibly write about every such option, so I'll write about one: rclone.
- Gitsync-Nncp over Filespooler[27]
You can use gitsync-nncp[28] (a tool for Asynchronous[29] syncing of git[30] repositories) atop Filespooler[31]. This page shows how. Please consult the links in this paragraph for background on gitsync-nncp and Filespooler.
- Compressing Filespooler Jobs[32]
Filespooler[33] has a powerful concept called a *decoder*. A decoder is a special command that any Filespooler command that reads a queue needs to use to decode the files within the queue. This concept is a generic one that can support compression, encryption, cryptographic authentication, and so forth.
- Introduction to Filespooler[34]
It seems that lately I've written several shell implementations of a simple queue that enforces ordered execution of jobs that may arrive out of order. After writing this for the nth time in bash, I decided it was time to do it properly. But first, a word on the *why* of it all.
- Creating the Signed and Encrypted Packets[35]
Sometimes, one wants to verify the integrity and authenticity of a Filespooler[36] job file before processing it.
- Filespooler[37]
Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing[38], NNCP[39], ssh, UUCP[40], USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler.
Filespooler is particularly suited to distributed and Asynchronous Communication[41].
More on www.complete.org
(c) 2022-2024 John Goerzen