No physical znode address

[Patrick Proniewski]

Hi all,

I'm running a FreeBSD 8.2 server, with Apache 2.2 hosting around 260 web sites. It's a virtual machine, running on top of ESXi and a SAN storage.
The OS is installed on UFS, and a dedicated ZFS disk holds every web sites. Each web site is a ZFS volume created from the zpool "tank".

# zpool list  
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank   149G  53.6G  95.4G    35%  ONLINE  -

# zpool status          
  pool: tank
 state: ONLINE
 scrub: scrub completed after 0h19m with 0 errors on Fri May 13 22:57:10 2011
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  da1       ONLINE       0     0     0

errors: No known data errors

Today, I've noticed an httpd process, stuck, using 100% CPU for hours. It looks like the process has opened non-existing files. Here is a part of the output of lsof:

# lsof -p 10453
COMMAND   PID USER   FD     TYPE             DEVICE SIZE/OFF   NODE NAME
httpd   10453  www  cwd                                             No physical znode address: 0xffffff0013019c60
httpd   10453  www  rtd     VDIR               0,87      512      2 /
httpd   10453  www  txt     VREG               0,87  1321703 406618 /usr/local/sbin/httpd
httpd   10453  www  txt     VREG               0,87   246776 235521 /libexec/ld-elf.so.1
httpd   10453  www  txt     VREG               0,87   154320 659461 /lib/libm.so.5
../..
httpd   10453  www  120r                                            No physical znode address: 0xffffff00132e2840
httpd   10453  www  121r                                            No physical znode address: 0xffffff0013019c60
httpd   10453  www  122r                                            No physical znode address: 0xffffff00132e2840
httpd   10453  www  123r                                            No physical znode address: 0xffffff0013019c60
httpd   10453  www  124r                                            No physical znode address: 0xffffff00132e2840
httpd   10453  www  125r                                            No physical znode address: 0xffffff0013019c60
httpd   10453  www  126r                                            No physical znode address: 0xffffff00132e2840
httpd   10453  www  127r                                            No physical znode address: 0xffffff0013019c60
httpd   10453  www  128r                                            No physical znode address: 0xffffff00132e2840
httpd   10453  www  129r                                            No physical znode address: 0xffffff0013019c60
httpd   10453  www  130r                                            No physical znode address: 0xffffff00132e2840
httpd   10453  www  131r                                            No physical znode address: 0xffffff0013019c60
../..

Reading a part of lsof's source code, it seems to relate to ZFS (dnode2.c - FreeBSD ZFS node functions for lsof).
Using truss, I've discovered that the process is trying to stat a non-existing file, with a way too long path. truss output is a infinite repetition of:

stat("/Sites/sites//spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core-vh/sites/edhum/bd/.Trashes//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////",0x7fffffffcc90) ERR#63 'File name too long'

(I had to kill -9 truss process...)

Obviously, there is something wrong with this particular web site. But I'm afraid it could come from the file system, or impact the FS. 
Any idea is welcome.

patpro