Local IRC Services

These are not difficult to setup. One use for such would be local chat in a small company; another would be for monitoring of services where a bot or bots can report on the status of things (new help requests, email incoming, high CPU issues, the weather, etc). Notifications, but without the hassle and distraction of OS-level notifications.

Of course this all works better if you're already running IRC for chat, so adding a local server and some more channels to be in is a solved problem. IRC is also light on resources, and unlike the Apple notification spam you can easily turn it off or tune it in various ways. That macOS still spams you with notifications even with "do not disturb" set and all the notification knobs I could find turned off, yeah, I don't know. Buggy code gonna buggy.

Two Martians came forward and took greeting stances ten yards away—which meant, between those of roughly equal status, something that looked like shaking hands with yourself. Martians certainly didn't shake hands with each other; this culture put a premium on social distance.
And the gesture was the local equivalent of waving, whistling between your teeth, and shouting, Hey, Mac!
— "In The Courts of the Crimson Kings". S. M. Sterling. 2008.

A Server

There are a bunch of IRC servers, I forget why I picked ngircd. Maybe it was in the ports system and easy to install? You may want SSL and authentication for users, but that's more work. The following server listens on localhost and let's pretend a wireguard interface so that other systems can easily connect over that. Notably the server is not exposed to the internet, nor even the local wifi or whatever; you need to be on the server or on the VPN to get at it, or to tunnel to it over SSH. Security through obscurity can help, as long as it's not the only measure taken.

    [Global]
            Name = example.org
            Info = blah blah blah
            AdminInfo1 = fixme
            AdminInfo2 = fixme
            AdminEMail = fixme@example.org
            Listen = 127.0.0.1,192.0.2.42
            MotdPhrase = "blah de blah blah"
    [Limits]
            MaxNickLength = 9
            MaxListSize = 9
    [Options]
            AllowRemoteOper = no
            DNS = no
            Ident = no
            OperCanUseMode = yes
            PAM = no
    [Operator]
            Name = someone
            Password = Hunter2
    [Channel]
            Name = #swamp

A Bot

There are a lot of IRC client libraries and bot implementations. eggdrop is traditional and appears to have all sorts of bells and whistles. The Bot::IRC Perl module is what I picked; it has a lot of dependencies but allowed for a quick prototype monitoring bot to be written without too much hassle.

    #!/usr/bin/perl
    # monbot.pl - example IRC monitoring bot
    use 5.40.0;
    use Bot::IRC;

    my $ch = '#swamp';    # the swamp was important in MUD1

    # How long to disable alerts once an alert has gone out.
    my $shadow_secs = 1789;

    # Toggle for when the alerts are annoying, or mainly to show how to
    # setup an interaction with the bot.
    my $do_checks = 1;
    # ... but have the checks not happened in a while?
    my $last_check = time();

    # Track when various files have been modified. This could also be done
    # with an inode change something that gets the news to this bot.
    my %mtime = (
        '/var/mail/someone' => 0,
        '/etc/passwd'       => 0,
    );
    my %shadow;
    # NOTE the files may not exist, but usually should; in theory you have
    # configuration management to ensure that the file exist, have the right
    # permissions, etc.
    for my $path ( keys %mtime ) {
        $mtime{$path}  = ( stat $path )[9] // 0;
        $shadow{$path} = 0;
    }

    sub checks {
        my ($bot) = @_;
        my $now = time;
        if ( $now > $last_check + $shadow_secs ) {
            $bot->msg( $ch, "checks $do_checks and no checks in a while..." );
            $last_check += $shadow_secs;
        }
        return unless $do_checks;
        my @updates;
        for my $path ( keys %mtime ) {
            my $mtime = ( stat $path )[9];
            if ( defined $mtime and $mtime != $mtime{$path} ) {
                if ( $now > $shadow{$path} ) {
                    push @updates, $path;
                    $shadow{$path} = $now + $shadow_secs;
                }
                # NOTE no new notifications whilst shadowing is on. Others
                # may want to be notified about a change that happened during
                # the shadow period when the window drops.
                $mtime{$path} = $mtime;
            }
        }
        $bot->msg( $ch, "updated: @updates" ) if @updates;
        $last_check = $now;
    }

    sub toggle {
        my ( $bot, $in, $m ) = @_;
        my $old = $do_checks;
        if ( $m->{word} eq 'on' ) {
            $do_checks = 1;
        } elsif ( $m->{word} eq 'off' ) {
            $do_checks = 0;
        }
        if ( $do_checks != $old ) {
            $bot->reply("checks now $m->{word}");
        } else {
            $bot->reply("checks $do_checks");
        }
    }

    Bot::IRC->new(
        connect => {
            nick   => 'monbot',
            name   => 'status monitoring bot',
            server => '192.0.2.42',
            join   => $ch,
        },
        daemon => {
            stderr_file => 'monbot.out',
            stdout_file => 'monbot.err',
        },
        plugins => [ {
            hooks => [ [ { to_me => 1, text => qr{\b(?on|off)\b} },
                         \&toggle ] ],
            ticks => [ [ 59, \&checks ] ]
        } ],
    )->run;

Probably a database should be used to store events so a summary can be obtained via other methods (email, for example). Otherwise more generic "show alerts, but not too many" library caching code could be used, plus means to add more things besides file modification times. Other scripts could perhaps write status files for the bot to read and report on, among other such ideas.

As the infrastructure becomes more critical (more people, more services, more Mammons on the line) more care should be put into standing up such monitoring services, like would you notice if the monitoring bot hadn't been present on the channel for a while? Some amount of duplication of monitoring is okay, or at least to have a small local monitoring system that checks whether the main monitoring system and other critical systems are actually up and doing things, while at the same time remembering to avoid the "boy who cried wolf" problem that many monitoring (or notification) systems suffer from: too many alerts may result in the thing being ignored or turned off. (And where "too many" may be somewhere below one for certain notifications, like, I don't know, "game mode" being turned on on macOS.)

Notification

What does a notification look like for me in irssi?

     [thrig(+Ziw)] [1:libera (change with ^X)] [8]
And the gesture was the local equivalent of waving, whistling between your teeth, and shouting, Hey, Mac!
C bot instead of Perl