Where has my gbde write performance gone?

[Joseph Gleason]

Sometime between FreeBSD 7.2-RELEASE-p4 and 8.0-RELEASE write
performance of gbde encrypted devices seems to have dropped
significantly.  A system I have running 7.2 seems to run gbde drives
at or near the drive max rate (30-40MB/s) while I am seeing less than
10% of that on 8.0 systems.

I get the same slow writes on 8.0-RELEASE-p2 as well as 8.0-RELEASE.

Here is an example on a fresh 8.0 install which shows gbde taking the
drive write performance of 40 MB/s down to 2.6 MB/s:

lab# uname -a
FreeBSD lab.int.fireduck.com 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat
Nov 21 15:02:08 UTC 2009
root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64

lab# dd if=/dev/urandom of=/dev/ad4s1d bs=32k count=32k
32768+0 records in
32768+0 records out
1073741824 bytes transferred in 25.130537 secs (42726577 bytes/sec)

lab# gbde init /dev/ad4s1d
Enter new passphrase:
Reenter new passphrase:

lab# gbde attach /dev/ad4s1d
Enter passphrase:

lab# dd if=/dev/urandom of=/dev/ad4s1d.bde bs=32k count=32k
32768+0 records in
32768+0 records out
1073741824 bytes transferred in 401.097004 secs (2677013 bytes/sec)

iostat from while that last 'dd' was running:

       tty             ad4             cpu
 tin  tout  KB/t tps  MB/s  us ni sy in id
   0    22  5.67 483  2.67   0  0  4  1 96
   0    66  5.67 509  2.82   0  0  4  1 95
   0    22  5.69 514  2.86   0  0  6  1 94
   0    22  5.67 506  2.80   0  0  6  1 93
   0    22  5.67 472  2.61   0  0  4  1 95


iostat on a FreeBSD 7.2-RELEASE-p4 box doing a similar operation:

 tin  tout  KB/t tps  MB/s  us ni sy in id
   0   22 29.54 1208 34.86   3  0 56  2 39
   0   22 29.56 1177 33.97   3  0 57  1 39
   0   22 29.54 1201 34.64   3  0 58  2 37
   0   22 29.57 1144 33.04   2  0 51  3 44
   0   22 29.56 1126 32.52   3  0 54  2 42
   0   22 29.53 1179 34.01   3  0 53  2 42
   0   22 29.57 1165 33.65   2  0 58  2 38

One thing I notice is the larger block size the 7.2 writes but I don't
imagine that would be that significant.

I've been using FreeBSD in various amateurish and wrong ways since
2.2, so I wouldn't rule out me doing something stupid.  If so, I'd
love to know what.