Performance issues with 8.0 ZFS and sendfile/lighttpd

[Ivan Voras]

Miroslav Lachman wrote:
> Ivan Voras wrote:
>> gnukix at alltel.blackberry.com wrote:
>>> I can send in more documentation later but I am seeing severe zfs
>>> performance issues with lighttpd. Same machine using UFS will push
>>> 1gbit or more but same content and traffic load can not hit 200mbit.
>>> Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes
>>> the disks into the ground with 50+ megabytes/sec dusk i/o. No
>>> compression no atime no checksums on zfs and still same IO levels. Ufs
>>> with soft updates and atime on. Orders of magnitude more disk IO...
>>> Like zfs isn't using cache or isn't coalescing disk reads or both.
>>> Has anyone else seen this or have any recommendations? Lighttpd config
>>> remains exactly the same as well FYI. Only difference is ufs vs zfs.
>>
>> AFAIK, ZFS is incompatible (currently) with some advanced VM operations
>> (like mmap, and I think sendfile relies on the same mechanism as mmap),
>> so that could be a cause of the slowdown. Though I'm surprised you can
>> only get 200 MBit/s - that's 25 MB/s and I think that even with multiple
>> memcpy-ing data around the kernel you should be able to get hundreds of
>> MB/s on newer hardware (which normally really can achieve tens of
>> gigabytes/s of sustained memory access).
> 
> I have more strange issue with Lighttpd in jail on top of ZFS. Lighttpd 
> is serving static content (mp3 downloads thru flash player). Is runs 
> fine for relatively small number of parallel clients with bandwidth 
> about 30 Mbps, but after some number of clients is reached (about 50-60 
> parallel clients) the throughput drops down to 6 Mbps.
> 
> I can server hundereds of clients on same HW using Lighttpd not in jail 
> and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe more).
> 
> I don't know if it is ZFS or Jail issue.

Do you have actual disk IO or is the vast majority of your data served 
from the caches? (actually - the same question to the OP)