Linewise Scripts

On my laptop, it's Windows, and while I could have run the same shell script in Cygwin, instead I tried writing it as a batch file. It seemed do-able because it's just calling some external commands and checking if they worked. Easy. But when I run it and it checks out the pages branch, that's when things get tricky. The batch file no longer exists. Windows batch files are NOT read at startup. When calling an external command, the command processor just remembers an offset into the file. When it's time to continue the script it can read the next line from there. Or not, in my case.

Unix shell scripts do the exact same thing. As a consequence the shell script can be changed on the fly. The reasoning I heard for this was that shell scripts are supposed to emulate commands typed in at a terminal, hence the care taken (if you process trace a shell and watch how it does file I/O) to line up the offset at the beginning of the next line.

These days many programs slurp entire file contents into memory, or you can find programs that pass a reference to an array somewhere, then create a new array from the reference… maybe the programmer did not know how to iterate over the reference, or otherwise did not know what they were doing? If modern systems had less memory such practices would be much less viable. There were some human genome perl scripts that ran out of memory by copying and copying and copying so biologists trying to program were doing this back in the 1990s. Well, the script worked, provided the machine had enough memory and the genome was not too large. Another problem they ran into was putting too many files into a single directory which made the filesystem or NFS hilariously slow. Modern filesystems better cope with this with directory hashing, but you still may want to make a T/TR/TREE of files to split the files into subdirectories. A .git/objects directory should show this pattern.

Downsides with making a copy of the file include the usual /tmp attacks, the copy failing (oops, the disk was full), permissions problems, etc.