tmpfs out of space (ZFS related?)

[Ivan Voras]

I got a curious error today while starting PostgreSQL, complaining about 
"out of space" errno while creating lock file on /tmp.

/tmp on this machine is mounted as tmpfs and indeed, here is the statistic:

biggie:/# df -i
Filesystem    1M-blocks  Used  Avail Capacity iused     ifree %iused 
Mounted on
/dev/mfid0s1a      9912  5193   3926    57%  306079   1012831   23%   /
devfs                 0     0      0   100%       0         0  100%   /dev
fdescfs               0     0      0   100%       4     11092    0% 
/dev/fd
tmpfs                 0     0      0   100%       9         0  100%   /tmp
tank             376044     0 376044     0%       4 770138347    0%   /tank
tank/ports       376658   614 376044     0%  145919 770138347    0% 
/usr/ports
tank/mysql       376073    29 376044     0%     102 770138347    0% 
/var/db/mysql
tank/pgdata90    400469 24425 376044     6%    1047 770138347    0% 
/tank/pgdata90

On the other hand, "top" reports this:

last pid: 79667;  load averages:  0.08,  0.68,  0.77    up 1+09:12:13 
00:11:33
44 processes:  1 running, 43 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 13M Active, 46M Inact, 15G Wired, 232K Cache, 1458M Buf, 8358M Free
Swap: 1024M Total, 1024M Free

Note the "8358M free" report.

The server has ZFS and was doing IO intensive database work on it; the 8 
GB free memory comes from PostgreSQL being restarted and freeing the 
memory (but failing to start again...).

Starting PostgreSQL gets me this message:
Nov 22 00:18:24 biggie postgres[79696]: [1-1] FATAL:  could not write 
lock file "/tmp/.s.PGSQL.5432.lock": No space left on device

This is 8-STABLE amd64.

Running "touch /tmp/abc" works, and creates a file. Running "echo abc > 
/tmp/abc" doesn't return an error but *doesn't write anything to the 
file*, just creates a directory entry.

The status doesn't change over time, i.e. "df" on tmpfs always shows "0 
free".