Blacklistd not recognizing probing attemtps

[Conrad Meyer]

Hi László,

On Tue, Feb 26, 2019 at 5:53 PM László Károlyi <laszlo at karolyi.hu> wrote:
> I'm on 12.0-RELEASE-p3 and I have configured blacklistd with sshd to
> lock out those random IPs that are probing my server. The problem is, I
> noticed that in many cases, blacklistd does not put the offending IP on
> its list.
>
> I've contacted Christos Zoulas in email to see if he has anything to
> tell about it, and after putting blacklistd in debug mode and
> reproducing the issue, he suggested to contact you with the it. So here
> it is. I'll paste a couple lines from the sshd log, I get these, which
> aren't registered for some reason:

I think there are a couple suspicious issues in FreeBSD blacklist(d)
and sshd's blacklist support, some of which explains what you see.

First, FreeBSD blacklistd ignores BAD_USER failures entirely.[1.1]
This is not ideal.  That includes things like "Invalid user FOO from
XXX port YYY" in your logs.  Those attempts will never result in IPs
being filtered.  2 of 5 blacklist failure notifications in ssh are
BAD_USER types (i.e., nops).  The rest are ordinary AUTH_FAIL.

Second, libblacklist opens reporting connections SOCK_CLOEXEC (or
FD_CLOEXEC if the sock macro is unavailable), and the FreeBSD sshd
libblacklist patches only invoke BLACKLIST_INIT() in one location.
The problem is that sshd loves to fork.  The patch happens to
initialize the BL reporting socket in the primary process associated
with any given connection (post-accept fork), but sshd forks multiple
times *after* this in privsep mode for preauth (i.e., the
authentication process), and even something postauth.  When it forks,
the logging socket is closed by the operating system.  *Privsep is
enabled by default in sshd.*  BLACKLIST_NOTIFY -> blacklist_r ->
bl_send has a mechanism to detect failures, reset the connection, and
retry.  However, it will (a) spuriously close an invalid fd that no
longer represents the cloexec socket, and (b), execute in a context
where the _PATH_BLSOCK (/var/run/blacklistd.sock) is no longer
available due to chroot, capsicum, or other sandboxing.

Third, sshd creates the login grace timeout handler, then begins
initial packet version exchange, then actually initializes
libblacklist.  There is a BLACKLIST_NOTIFY in the login grace timeout
handler.  This means an abusive client that stalls initial version
exchange can't be reported, because the timeout handler runs before
libblacklist is initialized.  (It also means we can't notify
libblacklist if initial version exchange detects something bad — but
we don't try to at this time, AFAICT.)

The 1st seems like an issue local to FreeBSD — upstream blacklistd
does not appear to have a BAD_USER concept at all[1.2].  Importantly,
upstream treats all of these as AUTH_FAIL.

The 2nd issue is shared with NetBSD, as far as I can tell.  The
details are a little different, but both close arbitrary invalid fds,
and both may be chrooted when attempting to reinitialize a forked sshd
process's blacklist reporting socket.

NetBSD has the same cloexec behavior[2.1] and also defaults to using
privsep[2.2][2.3].  NetBSD's pfilter.c[2.4] looks slightly different
from our blacklist.c[2.5].  However, they are not significantly
different.

The 3rd issue is sort of unique to FreeBSD: NetBSD initializes
libblacklist much earlier in sshd:main(), via server_accept_loop().
In fact, NetBSD initializes blacklist (pfilter) prior to the fork()
for the client connection.  This means that in NetBSD, the initial
blacklist socket is closed before any sshd authentication begins at
all!  bl_send() -> bl_reset / bl_init may recover in some contexts,
but maybe not others.

Another important distinction is that NetBSD has a catch-all
pfilter_notify() in cleanup_exit() when the exitcode matches 255;
FreeBSD's sshd is missing that one.

Best regards,
Conrad

[1.1]: https://github.com/freebsd/freebsd/blob/master/contrib/blacklist/bin/blacklistd.c#L263
[1.2]: https://github.com/NetBSD/src/blob/trunk/external/bsd/blacklist/include/bl.h#L40

[2.1]: https://github.com/NetBSD/src/blob/trunk/external/bsd/blacklist/lib/bl.c#L130-L161
[2.2]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh/dist/sshd.c#L232
[2.3]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh/dist/servconf.c#L526
[2.4]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh/dist/pfilter.c#L32-L38
[2.5]: https://github.com/freebsd/freebsd/blob/master/crypto/openssh/blacklist.c#L83-L96