Unexplainable behaviour after upgrading from 9.1 to 10.0

[Tommy Scheunemann]

Hello @list,

back in February this year I did an upgrade from 9.1-RELEASE to 
10.0-RELEASE.
Recompiled all ports (of course), using the same options as before. Also 
the configuration is exactly the same - both of base and ports.
Still since then, and following all updates up to 10.0-RELEASE-p7, the 
system has increasingly way more load.
Example top output:

last pid: 44452;  load averages:  0.57,  0.46,  0.42
up 0+02:07:00 14:02:29
75 processes: 1 running, 74 sleeping
CPU:  0.4% user,  0.0% nice,  0.6% system,  0.2% interrupt, 98.9% idle
Mem: 691M Active, 1803M Inact, 119M Wired, 8988K Cache, 94M Buf, 138M Free
Swap: 2048M Total, 15M Used, 2033M Free

The OS starts swapping out processes while there's still memory available, 
as example Nginx (which acts as reverse proxy), php-fpm processes and a 
couple of other instead of just keeping them in memory.
This behaviour is the same using the GENERIC kernel as well as a 
self-compiled one (Xen drivers removed, unused NIC drivers, VirtIO and USB 
support).
The hardware is of course the same as before so physically nothing 
changed.
Short after rebooting swap isn't used, though the longer the system runs 
the more swap is used, up to around 900 MB e.g. by swapping MySQL.
The system itself is an i386 FreeBSD 10.0-RELEASE, 2 GB RAM, 2 SATA 
drivers using a Gmirror.
The attached images show the difference in the entire system behaviour, 
while the traffic the webserver gets, things MySQL should do, mails 
Postfix sends and recieves didn't changed.
Just let the webserver deliver something takes of course more time, since 
the process seems to run from swap.
Maybe interesting the sysctl.conf.local:

kern.coredump=0
kern.randompid=348

security.jail.chflags_allowed=0
security.jail.allow_raw_sockets=0
security.jail.enforce_statfs=2
security.jail.set_hostname_allowed=0
security.jail.param.persist=1
security.jail.param.securelevel=3

security.bsd.see_other_uids=0
security.bsd.see_other_gids=0

net.inet.icmp.drop_redirect=1
net.inet.ip.portrange.randomized=1
net.inet.ip.random_id=1

Which is the same as under 9.1

Anything else I can do to let the kernel stop swapping out processes 
without much need ?
And of course if one needs more information, really no problem.
Would just like to get this issue solved before moving along with 10.0 on 
the way more critical machines.

Thanks in advance
-------------- next part --------------
A non-text attachment was scrubbed...
Name: memory.png
Type: application/octet-stream
Size: 31596 bytes
Desc: 
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20140718/b20eee1c/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: load.png
Type: application/octet-stream
Size: 38050 bytes
Desc: 
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20140718/b20eee1c/attachment-0001.obj>