Filespooler

What is Filespooler?

1.

Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing[1], NNCP[2], ssh, UUCP[3], USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler. Filespooler is particularly suited to distributed and Asynchronous Communication[4].

2.

Filespooler is a tool in the Unix tradition of "do one thing and do it well." It is designed to integrate nicely with decoders (to handle compressed or Encrypted[5] packets, for instance). It can send and receive packets by pipes. Its on-disk format is simple and is designed to interface well with other tools.

3.

Filespooler is *strictly ordered* by default; that is, it executes jobs in the order they were created, even if they arrive out of order. However, it also supports looser operation modes for scenarios such as certain Many-To-One[6] setups.

4.

Filespooler is an example of scalable Small Technology[7]:

* The file format is lightweight; less than 100 bytes overhead in most cases.

* The queue format is lightweight; even a Raspberry Pi[8] could easily process thousands of different queues if needed.

* The main CLI tool, fspl, uses less than 10MB RAM on x86_64

5.

Filespooler processes packets as streams, and can easily accommodate multi-terabyte payloads.

6.

Filespooler is extremely versatile. In addition to the various transports it can easily work with, it can also easily work with encoders/decoders such as compressors and encryption tools. Basically, if you can pipe stuff to or from it, Filespooler can integrate with it. Thanks to its flexible design (it's the "find of command execution"), Filespooler also supports advanced queue topologies supporting One-To-Many[9], Many-To-One[10], Feeding Queues from Other Queues[11], Parallel Processing[12], etc. with its simple design -- all with very little effort.

Main Links

• Filespooler homepage - this is it!
• Filespooler Git Repo[13] - includes pre-built Linux binaries for x86_64, armhf (Raspberry Pi), and aarch64 (64-bit ARM)
• Filespooler Reference[14]
• crates.io page[15]

Learning about Filespooler

• Start with Introduction to Filespooler[16] - my blog post about it
• The Filespooler Git repo[17] contains a detailed reference[18], which is also renderable as a manpage. It includes installation instructions.

Once installed, learn about using it in different situations.

• The Using Filespooler over Syncthing[19] is one of the more comprehensive tutorials for Filespooler, and is a great place to start (even if you don't use Syncthing.)

Installation

The Filespooler Reference[20] discusses this. You can install with Rust with a one-line command, but binaries are also available from the releases[21] page. They are built for these platforms:

• Linux x86_64 (common for PCs and laptops)
• Linux aarch64/arm64 (64-bit ARM)
• Linux armhf (for 32-bit ARM and Raspberry Pi[22]). Even though this isn't technically a backport, the compatibility notes from Installing Debian Backports on Raspberry Pi[23] apply here as well.

These binaries are built using the trusted infrastructure maintained by Debian, using the official Rust docker images, and the build logic contained within the Filespooler repo.

Integrations

Transports

These pages are introductions that explain how to use Filespooler with different transports:

• Using Filespooler over Syncthing[24] - one of the tutorials for Filespooler. Also applicable to other directory syncing tools such as Dropbox, box.com, and unison.
• Using Filespooler over NNCP[25] - one of the design goals of Filespooler. This is a quite powerful combination!
• Using Filespooler over rclone and S3, rsync.net, etc.[26]
• Guidelines for Writing To Filespooler Queues Without Using Filespooler[27]

Encoders and Decoders

• Compressing Filespooler Jobs[28]
• Encryption and authentication can also be expressed as decoders; see more below under Security.

Programs to Execute

• Using Filespooler for Backups[29]
• gitsync-nncp over Filespooler[30]

Security

By default, Filespooler packets are un-encrypted and unsigned. But Filespooler is designed to integrate nicely with encryption tools, thanks to the `--decoder` option. Here are some examples.

• Encrypting Filespooler Jobs with GPG[31]
• Encrypting Filespooler Jobs with Age[32]
• Verifying Filespooler Job Integrity[33]

Management

• Filespooler in cron and systemd[34]
• Handling Filespooler Command Output[35]

Tips and Tricks

• Parallel Processing of Filespooler Queues[36]
• Feeding Filespooler Queues from Other Queues[37]
• One-To-Many with Filespooler[38] - sending jobs from one system to many systems
• Many-To-One with Filespooler[39] (and many-to-many) - sending jobs from many systems to one queue

* Includes a conversation about non-sequence-based queue processing and accepting duplicate sequence numbers

• Processing Multiple Commands in a Single Filespooler Queue[40]
• Processing Filespooler Queues Without Filespooler[41]
• Filespooler Append-Only Queues[42] and their use on sending systems
• Using Filespooler Without Queues to Pass More Metadata[43] for things like ssh and NNCP

--------------------------------------------------------------------------------

Links to this note

• How Gapped Is Your Air?[44]

Sometimes we want better-than-firewall security for things. For instance:

• An Asynchronous Rsync With Dar[45]

In my writing about dar[46], I recently made that point that dar is a filesystem differ and patcher.

• Roundup of Data Backup and Archiving Tools[47]

Here is a comparison of various data backup and archiving tools. For background, see my blog post[48] in which I discuss the difference between backup and archiving. In a nutshell, backups are designed to recover from a disaster that you can fairly rapidly detect. Archives are designed to survive for many years, protecting against disaster not only impacting the original equipment but also the original person that created them. That blog post goes into a lot of detail on what makes a good backup or archiving tool.

• Building an Asynchronous, Internet-Optional Instant Messaging System[49]

I loaded up this title with buzzwords. The basic idea is that IM systems shouldn't have to only use the Internet. Why not let them be carried across LoRa radios, USB sticks, local Wifi networks, and yes, the Internet? I'll first discuss how, and then why.

• Dead USB Drives Are Fine: Building a Reliable Sneakernet[50]

"OK," you're probably thinking. "John, you talk a lot[51] about things like Gopher[52] and personal radios[53], and now you want to talk about building a reliable network out of... *USB drives*?"

• The PC & Internet Revolution in Rural America[54]

Inspired by several others (such as Alex Schroeder's post[55] and Szczeżuja's prompt[56]), as well as a desire to get this down for my kids, I figure it's time to write a bit about living through the PC and Internet revolution where I did: outside a tiny town in rural Kansas. And, as I've been back in that same area for the past 15 years, I reflect some on the challenges that continue to play out.

• Using Filespooler Without Queues to Pass More Metadata[57]

One frustration people sometimes have with ssh or NNCP[58] is that they'd like to pass along a lot of metadata to the receiving end. Both ssh and nncp-exec allow you to pass along command-line parameters, but neither of them permit passing along more than that. What if you have a whole host of data to pass? Maybe a dozen things, some of them optional? It would be very nice if you could pass along the environment.

• Dar[59]

dar is a Backup[60] and archiving tool. You can think of it as as more modern tar. It supports both streaming and random-access modes, supports correct incrementals (unlike GNU tar's incremental mode), Encryption[61], various forms of compression, even integrated rdiff deltas.

• Gnupg (GPG)[62]

GnuPG (also known by its command name, gpg) is a tool primarily for public key Encryption[63] and cryptographic authentication.

• Processing Filespooler Queues Without Filespooler[64]

All of the Filespooler[65] examples so far have focused on using `fspl queue-process` to process queue items.

• Guidelines for Writing to Filespooler Queues Without Using Filespooler[66]

Filespooler[67] provides the `fspl queue-write` command to easily add files to a queue. However, the design of Filespooler intentionally makes it easy to add files to the queue by some other command. For instance, Using Filespooler over Syncthing[68] has Syncthing do the final write, the nncp-file (but not the nncp-exec) method in Using Filespooler over NNCP[69] had NNCP do it, and so forth.

• Gitsync-Nncp[70]

gitsync-nncp is a tool for using Asynchronous Communication[71] tools such as NNCP[72] or Filespooler[73], or even (with some more work) Syncthing[74] to synchronize git[75] repositories.

• Many-to-One With Filespooler[76]

Since Filespooler[77] is an ordered queue processor by default, it normally insists on a tight mapping between the sequence numbers in job files and execution order in a queue.

• Handling Filespooler Command Output[78]

By default, Filespooler[79] doesn't do anything special with the output from the commands that `fspl queue-process` executes. If they write to stdout or stderr, you'll see this on the controlling terminal or wherever you have piped or redirected it.

• Filespooler in Cron and Systemd[80]

Filespooler[81] is designed to work well in automated situations, including when started from cron or systemd. It is a fairly standard program in that way. I'll discuss a few thoughts here that may help you architect your system.

• Gitsync-Nncp over Filespooler[82]

You can use gitsync-nncp[83] (a tool for Asynchronous[84] syncing of git[85] repositories) atop Filespooler[86]. This page shows how. Please consult the links in this paragraph for background on gitsync-nncp and Filespooler.

• Using Filespooler for Backups[87]

Filespooler[88] makes an *excellent* tool for handling Backups[89]. In fact, this was the case the prompted me to write it in the first place.

• Processing Multiple Commands in a Single Filespooler Queue[90]

You'll notice that Filespooler[91]'s `fspl queue-process` command takes a single command. What if you want to permit the sender to select any of several commands to run?

• Compressing Filespooler Jobs[92]

Filespooler[93] has a powerful concept called a *decoder*. A decoder is a special command that any Filespooler command that reads a queue needs to use to decode the files within the queue. This concept is a generic one that can support compression, encryption, cryptographic authentication, and so forth.

• Filespooler Reference[94]

The reference documentation for Filespooler[95] is here:

• Introduction to Filespooler[96]

It seems that lately I've written several shell implementations of a simple queue that enforces ordered execution of jobs that may arrive out of order. After writing this for the nth time in bash, I decided it was time to do it properly. But first, a word on the *why* of it all.

• One-to-Many With Filespooler[97]

In some cases, you may want to use Filespooler[98] to send the data from one machine to many others. An example of this could be using gitsync-nncp over Filespooler[99] where you would like to propagate the changes to many computers.

• Feeding Filespooler Queues from Other Queues[100]

Sometimes with Filespooler[101], you may wish for your queue processing to effectively re-queue your jobs into other queues. Examples may be:

• Parallel Processing of Filespooler Queues[102]

Filespooler[103] is designed around careful sequential processing of jobs. It doesn't have native support for parallel processing; those tasks may be best left to the queue managers that specialize in them. However, there are some strategies you can consider to achieve something of this effect even in Filespooler.

• Verifying Filespooler Job Integrity[104]

Sometimes, one wants to verify the integrity and authenticity of a Filespooler[105] job file before processing it.

• Encrypting Filespooler Jobs With Age[106]

Like the process described in Encrypting Filespooler Jobs with GPG[107], Filespooler[108] can handle packets Encrypted[109] with Age (Encryption)[110]. Age may be easier than GnuPG in a number of cases, particularly because it can use a person's existing SSH keypairs for encryption.

• Encrypting Filespooler Jobs With GPG[111]

Thanks to Filespooler[112]'s support for decoders, data for filespooler can be Encrypted[113] at rest and only decrypted when Filespooler needs to scan or process a queue.

• Using Filespooler over NNCP[114]

NNCP[115] is a powerful tool for building Asynchronous Communication[116] networks. It features end-to-end Encryption[117] as well as all sorts of other features; see my NNCP Concepts[118] page for some more ideas.

• Using Filespooler over Syncthing[119]

Filespooler[120] is a way to execute commands in strict order on a remote machine, and its communication method is by files. This is a perfect mix for Syncthing[121] (and others, but this page is about Filespooler and Syncthing).

• Syncthing[122]

Syncthing is a serverless, peer-to-peer file synchronization tool. It is often compared to Dropbox. However, unlike Dropbox, there is no central server with Syncthing; your devices talk directly to each other to sync data. Syncthing has various effective methods for firewall traversal, including public relays for the worst case. All Syncthing traffic is fully encrypted and authenticated.

• Old and Small Technology[123]

Old technology is any tech that's, well... old.

• John Goerzen's Software[124]

This page gives you references to software by John Goerzen[125].

More on www.complete.org

(c) 2022-2024 John Goerzen