Security Advisory - Recent compromise of freefall.freebsd.org

[Jordan K. Hubbard]

Overview:

The following advisory documents a recent security compromise on
freefall.freebsd.org, the FreeBSD Project's master source repository
machine, discussing some of the potential ramifications of the event
and the recovery measures which are being carried out in its
aftermath.

Since investigation is still ongoing and at least one law enforcement
agency is currently involved, some details will, of necessity, need to
be deliberately vague or even omitted entirely for now.  We apologize
for this and promise to keep everyone as up-to-date as possible on
events as the situation progresses, releasing information as we're
allowed and deem it prudent.

Anyone with an account on freefall.freebsd.org is strongly advised to
*CHANGE THEIR PASSWORD*, both on freefall and on any other machines
where the same password is used.  Based on the Trojan horses we found,
you should assume that your password was grabbed and transmitted to a
hostile 3rd party if you logged in at any time on or after January
18th, 1997.  It does not matter if you logged in with ssh or with
telnet, you should assume that your password has been collected.
Furthermore, if you used ssh, rlogin or telnet on freefall to go *out*
to other machines then you should assume that password information
given to these programs was also compromised.


Details:

The break-ins occurred on at least 2 cdrom.com machines, root being
compromised in both instances, and numerous system binaries had Trojan
horses inserted for the purpose of gathering and sending back password
information.  The method of entry used by the attacker(s) is not so
important given that both systems were vulnerable to several
significant, now known, security exploits at the time and any one of
them could have been used to gain entry & root privilege.  What is
more interesting about this attack is the sophistication of the Trojan
horses left behind, assembled as they were from a rather sophisticated
"kit" put together by someone who clearly knew their way around a BSD
system.  This told us that we should not take this attack as just
another incident of juvenille pranksterism but as something rather
more serious.

Since the CVS master repository machine was attacked, it would also be
an immediate and obvious concern that the intruder may have taken
advantage of their temporary root privileges to make modifications to
the FreeBSD master source repository, possibly to introduce back-doors
for later use or cause deliberate embarrassment by introducing
catastrophic failure modes.

Fortunately, neither scenario is as fearsome as it might seem.  For
one thing, the CVS repository is replicated on hundreds of machines
now, all syncing up with varying degrees of (deliberate) latency, and
"CTM deltas" are also made continuously from this repository.  These
streams of CTM information can show exactly what changed from moment
to moment in the source tree, entirely independently of the CVS
mechanisms (which might be compromised) for doing so.

There is also the fact that there are many, many eyes on the FreeBSD
source tree right now, more than most of us probably ever thought
possible in the beginning, and it's hard to believe that someone would
be able to slip a significant attack past the eyes of that many
people, watching their daily CTM deltas come by and reviewing, as they
do, each change with heavy skepticism before bringing it into their
own source trees.  To date, no reports of anything suspicious have
been received.


In summary:

We will continue to review our CTM deltas and we will look for signs
of skullduggery, but we frankly feel that the real dangers here lie
not so much in recently introduced changes, which are easily reviewed
for and caught, but in those accidental security holes which have been
buried in the BSD code for months or possibly years.  Since security
seems to have become the theme of the month, and many people have
volunteered (in light of our recent 2.1.6 security fracas) to begin a
much more serious and comprehensive security audit, we will take
advantage of this opportunity to see that all code in the FreeBSD
source tree, old and new alike, is reviewed line by line for buffer
overflows, unguarded copies, back doors, whatever.  We may not make it
through every last byte, but we can certainly focus on the "hot spots"
(suid programs and system utilities) and do our best to prevent
problems like those which caused our recent headaches from reoccuring.

This advisory is simply to inform those people who have used freefall
in the last 40 days or so that they should change their passwords and
to explain to people that yes, there was a break-in to
freefall.freebsd.org and yes, we're aware of the issues this raises,
both now and in the immediate future, and that we will be exerting
significant effort over the next few weeks in dealing aggressively
with security issues, both in FreeBSD and on the FreeBSD project
machines.


FreeBSD Auditing Project:

Those interested in participating in "The Great Code Sweep", more
officially known as the FreeBSD Auditing Project, should also send
mail to me <jkh at freebsd.org>.  I'll be working over the next 2 days on
dividing /usr/src into reasonable, prioritized, chunks (there, I used
"prioritized" in a sentence - I've always wanted to do that) and
talking with the volunteer auditors about how to split the work up
amongst everyone.  Then we'll dive in and go to work!

I'll be posting more details on just what it is we're looking for, and
how to communicate changes back if you don't have commit access, in
the coming days on the current at freebsd.org mailing list.  Highlights
will also be sent to announce at freebsd.org, including a second call for
auditors and full instructions on how to participate, so hopefully no
one should miss it.

Thanks.

					Jordan