stuck /etc/rc autoboot processes

[Damien Fleuriot]

On 12/28/11 11:50 AM, Sergey Kandaurov wrote:
> On 27 December 2011 22:46, Damien Fleuriot <ml at my.gd> wrote:
>> Hello list,
>>
>>
>>
>> Yesterday and today, I've been busy either patching boxes for the BIND
>> advisory that we received on the 23rd (when they were running 8.1 or
>> 8.2-RELEASE), or upgrading them (when running 8.0-RELEASE).
>>
>>
>> Today I've come across 2 boxes running 8.2-STABLE and of course, the
>> BIND patch wouldn't apply correctly.
>>
>> I've decided to cvsup them to 8.2-RELEASE and "upgrade" them to it.
>>
>>
>> I've gone through the following steps:
>> - make buildworld
>> - make buildkernel
>> - make installkernel
>> - nextboot -k my new kernel, to ensure it worked fine
>> - rebooted again with the new kernel, this time correctly installed as
>> /boot/kernel
>> - installed the world
>> - run mergemaster -FiPU
>> - rebuild ports
>>
>>
>> Now, I'm facing this odd situation where, just after booting, I get this
>> on the 2 boxes:
>>
>>
>> root         22  0.0  0.0  8256  1876  v0  Is+   7:32PM   0:00.03 sh
>> /etc/rc autoboot
>> root       1250  0.0  0.0 18000  2576  v0  I+    7:32PM   0:00.04
>> /usr/local/sbin/rsyslogd -a /var/run/log -a /var/named/var/run/log -i
>> /var/run/syslog.pid -f /usr/local/etc/rsyslog.conf
>> root       1790  0.0  0.0  8256  1952  v0  I+    7:32PM   0:00.00 sh
>> /etc/rc autoboot
>> root       1793  0.0  0.0  8256  1952  v0  I+    7:32PM   0:00.00 sh
>> /etc/rc autoboot
>>
>>
>> Does anybody have an idea why I get these stuck "sh /etc/rc autoboot"
>> processes ?
>>
>> Any pointers as to where I should look ?
> 
> Check if the box has a working resolving during boot.
> This is a main reason why it may stuck in /etc/rc phase.
> When on physical console, type ^T. Usually it will get you
> the name of offending process.
> 
> You posted output from ps aux. It would be nice if you post
> ps auxl, so values of MWCHAN ps keyword will be also seen,
> which can add an additional debugging info.
> 


Find below the info:


# ps aufx
http://pastebin.com/iLy0Hs8s

# ps aufxl
http://pastebin.com/3meFWvRH

# dmesg.boot
http://pastebin.com/rFEsPfD5

Again, the box gets stuck at "Local package initialization:" from
/etc/rc.d/localpkg


I then run the following:
# sh -x /etc/rc.d/localpkg


A snip from the end of the script's output (stuck) yields:
+ logger 'localpkg: DEBUG: run_rc_command: doit: pkg_start '
+ echo 'localpkg: DEBUG: run_rc_command: doit: pkg_start '
localpkg: DEBUG: run_rc_command: doit: pkg_start
+ eval 'pkg_start '
+ pkg_start
+ local initdone
+ initdone=''
+ find_local_scripts_old
+ zlist=''
+ slist=''
+ [ -d /usr/local/etc/rc.d ]
+ grep '^# PROVIDE:' '/usr/local/etc/rc.d/[0-9]*.sh'
+ zlist=' /usr/local/etc/rc.d/[0-9]*.sh'
+ grep '^# PROVIDE:' /usr/local/etc/rc.d/relayd_check.sh
+ slist=' /usr/local/etc/rc.d/relayd_check.sh'
+ [ -z '' -a -f '/usr/local/etc/rc.d/[0-9]*.sh' ]
+ [ -x '/usr/local/etc/rc.d/[0-9]*.sh' ]
+ [ -f '/usr/local/etc/rc.d/[0-9]*.sh' -o -L
'/usr/local/etc/rc.d/[0-9]*.sh' ]
+ [ -z '' -a -f /usr/local/etc/rc.d/relayd_check.sh ]
+ echo -n 'Local package initialization:'
Local package initialization:+ initdone=yes
+ [ -x /usr/local/etc/rc.d/relayd_check.sh ]
+ set -T
+ trap 'exit 1' 2
+ /usr/local/etc/rc.d/relayd_check.sh start


relayd_check.sh is a custom script that I wrote to monitor relayd for
crashes, log them to /var/log/ and restart the process.

This script does *not* contain any "PROVIDE / REQUIRE / KEYWORD"
initialization info and I'm beginning to think this may be the problem.

I shall try further, thanks for all the pointers so far :)