Sudden mbuf demand increase and shortage under the load
l adrielt
ladrielt at gmail.com
Tue Jul 13 08:18:38 UTC 2010
Hi, I'm not sure but reading the advisory that just came out today
sounds like it could have something to do with your mbuf issues.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
=============================================================================
FreeBSD-SA-10:07.mbuf Security Advisory
The FreeBSD Project
Topic: Lost mbuf flag resulting in data corruption
Category: core
Module: kern
Announced: 2010-07-13
Credits: Ming Fu
Affects: FreeBSD 7.x and later.
Corrected: 2010-07-13 02:45:17 UTC (RELENG_8, 8.1-PRERELEASE)
2010-07-13 02:45:17 UTC (RELENG_8_1, 8.1-RELEASE)
2010-07-13 02:45:17 UTC (RELENG_8_0, 8.0-RELEASE-p4)
2010-07-13 02:45:17 UTC (RELENG_7, 7.3-STABLE)
2010-07-13 02:45:17 UTC (RELENG_7_3, 7.3-RELEASE-p2)
2010-07-13 02:45:17 UTC (RELENG_7_1, 7.1-RELEASE-p13)
CVE Name: CVE-2010-2693
For general information regarding FreeBSD Security Advisories,
including descriptions of the fields above, security branches, and the
following sections, please visit <URL:http://security.FreeBSD.org/>.
I. Background
An mbuf is a basic unit of memory management in the FreeBSD kernel
inter-process communication and networking subsystem. Network packets
and socket buffers are dependent on mbufs for their storage.
Data can be embedded directly in mbufs, or mbufs can instead reference
external buffers. The sendfile(2) system call uses external mbuf storage
to directly map the contents of a file into a chain of mbufs for
transmission purposes. The mbuf object supports a read-only flag that
must be honored to prevent modification or writes to buffer data in
cases like these.
II. Problem Description
The read-only flag is not correctly copied when a mbuf buffer reference
is duplicated. When the sendfile(2) system call is used to transmit
data over the loopback interface, this can result in the backing pages
for the transmitted file being modified, causing data corruption.
III. Impact
This data corruption can be exploited by an local attacker to escalate
their privilege by carefully controlling the corruption of system files.
It should be noted that the attacker can corrupt any file they have read
access to.
NOTE: While systems without untrusted local users are not affected by
the security aspects of this issue, the potential for data corruption
implies that this should still be treated as a critical erratum.
IV. Workaround
No workaround is available.
V. Solution
Perform one of the following:
1) Upgrade your vulnerable system to 7-STABLE or 8-STABLE, or to the
RELENG_8_1, RELENG_8_0, RELENG_7_3, or RELENG_7_1 security branch dated
after the correction date.
2) To update your vulnerable system via a source code patch:
The following patches have been verified to apply to FreeBSD 7.1, 7.3,
8.0 and 8.1 systems.
a) Download the relevant patch from the location below, and verify the
detached PGP signature using your PGP utility.
# fetch http://security.FreeBSD.org/patches/SA-10:07/mbuf.patch
# fetch http://security.FreeBSD.org/patches/SA-10:07/mbuf.patch.asc
b) Apply the patch.
On 7/11/10, Ali Mashtizadeh <mashtizadeh at gmail.com> wrote:
> Hi Maxim,
>
> I experienced the same issue recently on 8-STABLE branch and it seems
> it has been fixed since 8.1-RC2 and above. I couldn't track down the
> root cause in the code nor could I find a commit that seems to be the
> obvious fix.
>
> Thanks,
> ~ Ali
>
> 2010/2/15 Maxim Sobolev <sobomax at freebsd.org>:
>> Hi,
>>
>> Our company have a FreeBSD based product that consists of the numerous
>> interconnected processes and it does some high-PPS UDP processing (30-50K
>> PPS is not uncommon). We are seeing some strange periodic failures under
>> the
>> load in several such systems, which usually evidences itself in IPC (even
>> through unix domain sockets) suddenly either breaking down or pausing and
>> restoring only some time later (like 5-10 minutes). The only sign of
>> failure
>> I managed to find was the increase of the "requests for mbufs denied" in
>> the
>> netstat -m and number of total mbuf clusters (nmbclusters) raising up to
>> the
>> limit.
>>
>> I have tried to raise some network-related limits (most notably maxusers
>> and
>> nmbclusters), but it has not helped with the issue - it's still happening
>> from time to time to us. Below you can find output from the netstat -m few
>> minutes right after that shortage period - you see that somehow the system
>> has allocated huge amount of memory for the network (700MB), with only
>> tiny
>> amount of that being actually in use. This is for the
>> kern.ipc.nmbclusters:
>> 302400. Eventually the system reclaims all that memory and goes back to
>> its
>> normal use of 30-70MB.
>>
>> This problem is killing us, so any suggestions are greatly appreciated. My
>> current hypothesis is that due to some issues either with the network
>> driver
>> or network subsystem itself, the system goes insane and "eats" up all
>> mbufs
>> up to nmbclusters limit. But since mbufs are shared between network and
>> local IPC, IPC goes down as well.
>>
>> We observe this issue with systems using both em(4) driver and igb(4)
>> driver. I believe both drivers share the same design, however I am not
>> sure
>> if this is some kind of design flaw in the driver or part of a larger
>> problem with the network subsystem.
>>
>> This happens on amd64 7.2-RELEASE and 7.3-PRERELEASE alike, with 8GB of
>> memory. I have not tried upgrading to 8.0, this is production system so
>> upgrading will not be easy. I don't believe there are some differences
>> that
>> let us hope that this problem will go away after upgrade, but I can try it
>> as the last resort.
>>
>> As I said, this is very critical issue, so I can provide any additional
>> debug information upon request. We are ready to go as far as paying
>> somebody
>> reasonable amount of money for tracking down and resolving the issue.
>>
>> Regards,
>> --
>> Maksym Sobolyev
>> Sippy Software, Inc.
>> Internet Telephony (VoIP) Experts
>> T/F: +1-646-651-1110
>> Web: http://www.sippysoft.com
>> MSN: sales at sippysoft.com
>> Skype: SippySoft
>>
>>
>> [ssp-root at ds-467 /usr/src]$ netstat -m
>> 17061/417669/434730 mbufs in use (current/cache/total)
>> 10420/291980/302400/302400 mbuf clusters in use (current/cache/total/max)
>> 10420/0 mbuf+clusters out of packet secondary zone in use (current/cache)
>> 19/1262/1281/51200 4k (page size) jumbo clusters in use
>> (current/cache/total/max)
>> 0/0/0/25600 9k jumbo clusters in use (current/cache/total/max)
>> 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
>> 25181K/693425K/718606K bytes allocated to network (current/cache/total)
>> 1246681/129567494/67681640 requests for mbufs denied
>> (mbufs/clusters/mbuf+clusters)
>> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
>> 0/0/0 sfbufs in use (current/peak/max)
>> 0 requests for sfbufs denied
>> 0 requests for sfbufs delayed
>> 0 requests for I/O initiated by sendfile
>> 0 calls to protocol drain routines
>>
>> [FEW MINUTES LATER]
>>
>> [ssp-root at ds-467 /usr/src]$ netstat -m
>> 10001/84574/94575 mbufs in use (current/cache/total)
>> 6899/6931/13830/302400 mbuf clusters in use (current/cache/total/max)
>> 6899/6267 mbuf+clusters out of packet secondary zone in use
>> (current/cache)
>> 2/1151/1153/51200 4k (page size) jumbo clusters in use
>> (current/cache/total/max)
>> 0/0/0/25600 9k jumbo clusters in use (current/cache/total/max)
>> 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
>> 16306K/39609K/55915K bytes allocated to network (current/cache/total)
>> 1246681/129567494/67681640 requests for mbufs denied
>> (mbufs/clusters/mbuf+clusters)
>> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
>> 0/0/0 sfbufs in use (current/peak/max)
>> 0 requests for sfbufs denied
>> 0 requests for sfbufs delayed
>> 0 requests for I/O initiated by sendfile
>> 0 calls to protocol drain routines
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>
>
>
> --
> Ali Mashtizadeh
> علی مشتی زاده
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"
>
More information about the freebsd-hackers
mailing list