mfi0 timeout error zfs boot mount problem
Jürgen Weber
jurgen.weber at theiconic.com.au
Tue Oct 23 04:06:59 UTC 2012
Hi
Well, while this thread has been very quiet I have resolved my issues.
With patience changing:
kern.maxfiles=5000000
kern.maxvnodes=5000000
vfs.zfs.zil_disable="1"
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
The above solves the system unable to import or mount the pool.
I have also gone into the Card settings BIOS and changed under advanced
settings "Forward Read" to "none". This solves the mfi0 timeout.
Once I had the system up, I then added a l2arc cache via a usb2 SSD HDD.
I then shut hte system down and it took 3 hours to shut down.. but it
eventually did.
When I turned the system back on again, it booted as normal.
The lesson learnt?! Do not turn on deduping on a large file system
unless you have a lot of RAM or L2ARC! I would say 32GB of RAM/L2ARC for
every 10TB as a good rule of thumb, if not... double.
Thanks
Jurgen
On 22/10/12 12:30, Jürgen Weber wrote:
> Some more updates!
>
> on the bootloader I have also tried:
> kern.maxfiles=5000000
> kern.maxvnodes=5000000
>
> I have also gone into the Card settings BIOS and changed under
> advanced settings "Forward Read" to "none".
>
> Now the systems gets to
>
> "Trying to mount root from zfs:tank/root []..... " and then after
> maybe 1 to 5 minutes the next couple of lines load like its working!
>
> eg:
> "Setting hostuuid: xxxxx"
> "Setting hostid: xxxxxx"
> "Entropy harvesting:interrupts ethernet point_to_point kickstart"
> "Starting file system checks:"
> "Mounting local file systems:."
>
> and stops. I have had the machine on my desk all morning observing it
> and I can see the disk access is going crazy,, it is doing something.
>
> I have found this article:
>
> http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe
>
> I have a 15TB file system which has dedup on from the start (10TB. I
> feel its trying to load the DDT and its going to swap/there is not
> enough RAM (only have 16GB's). Hopefully my 64GB RAM upgrade is enough.
>
> Thanks
>
> Jurgen
>
>
>
> On 22/10/12 09:06, Jürgen Weber wrote:
>> This is still a problem for me, is anyone there? :)
>>
>> I have tried the following at the bootime loader.
>>
>> vfs.zfs.zil_disable="1"
>> vfs.zfs.prefetch_disable="1"
>> vfs.zfs.txg.timeout="5"
>>
>> Any other suggestions on how to get this zpool to import and mount
>> again?
>>
>> Thanks
>>
>> On 21/10/12 07:50, Jurgen Weber wrote:
>>> Hi
>>>
>>> Lastly, is there a way at boot time, some sysctl's or something I
>>> can set to bring zfs to a minimalistic state? Turn off features, etc
>>> to get this to mount?
>>>
>>> Any ideas appreciated.
>>>
>>> Thanks
>>>
>>> Jurgen
>>> On 20/10/2012 9:02 AM, Jurgen Weber wrote:
>>>> Guys
>>>>
>>>> Some more details on this, some insight would be greatly appreciated.
>>>>
>>>> As my day wore on trying to get this zpool to import or mount I
>>>> have learnt a few things. I think over time this issue has came
>>>> about as more and more data was added to the file systems.
>>>>
>>>> Some further details:
>>>>
>>>> Its a 8 disk raidz pool that the system boots from as well. The
>>>> disk are all 2TB.
>>>> The server has 16GB Of RAM, I notcied the day before this happen
>>>> the server was struggling with its RAM griding to a halt and
>>>> dumping its RAM.
>>>> The issue is not hardware because I found another server (same one)
>>>> swapped the harddrives out took another 8GB of RAM and I have the
>>>> same problem.
>>>> The main data file systems have dedup and gzip compression on.
>>>>
>>>> I have booted from open/Oracle Solars 11 adn attempted to import
>>>> and the Solaris live CD will not import either. In the Solaris
>>>> system the disk detach from the system.
>>>>
>>>> I get the feeling that ZFS is hitting some root limit when
>>>> attempting to mount and its not finishing the job.
>>>>
>>>> Thanks
>>>>
>>>> Jurgen
>>>>
>>>> On 19/10/2012 10:29 AM, Jürgen Weber wrote:
>>>>> Team
>>>>>
>>>>> I have googled around for a solution and I see a lot of posts
>>>>> about firmware versions and patches for FreeBSD 8.*.
>>>>>
>>>>> I have a FreeBSD 9.1rc1 system, which was beta1 orginally and has
>>>>> been running for months.
>>>>>
>>>>> Now it will not boot, I get the following:
>>>>>
>>>>> "Trying to mount root from zfs:tank/root [].....
>>>>> mfi0: COMMAND 0Xffffff8000cb83530 TIMEOUT AFTER xxx SECONDS
>>>>> (this just repeats).
>>>>>
>>>>> I have not seen this error before during normal runtime, _only_
>>>>> during boot.
>>>>>
>>>>> Originally when I had the problem I could boot off a USB stick
>>>>> (9.1beta1 or rc1), run a 'zpool import -f tank' and it would work
>>>>> on the livecd. Rebooting and the main system would work.
>>>>>
>>>>> This time this work around does not work for me. When I am on the
>>>>> USB stick I can run a 'zpool import' and all of the disk are
>>>>> recognised, the pool is recognised and the file system is healthy.
>>>>>
>>>>> The Card is a H700 PERC, with 12.10.3 firmware in a Dell R515.
>>>>> Running FreeBSD 9.1-RC1, latest zfs and zpool versions.
>>>>>
>>>>> I have tried disabling the cache (mfiutil cache xxx disable). I
>>>>> have also gone into the Card settings and changed under advanced
>>>>> settings "adaptive forward read" to "read only".
>>>>>
>>>>> Any help, appreciated.
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>
>>
>
> --
> Jürgen Weber
>
> Systems Engineer
> IT Infrastructure Team Leader
>
> THE ICONIC | Ejurgen.weber at theiconic.com.au |www.theiconic.com.au
--
Jürgen Weber
Systems Engineer
IT Infrastructure Team Leader
THE ICONIC | E jurgen.weber at theiconic.com.au | www.theiconic.com.au
More information about the freebsd-fs
mailing list