application hangs in STABLE from time to time

[Ganbold]

Kris,

Kris Kennaway wrote:
> On Fri, Dec 01, 2006 at 01:02:33PM +0800, Ganbold wrote:
>   
>> Kris Kennaway wrote:
>>     
>>> On Fri, Nov 24, 2006 at 10:02:23AM +0800, Ganbold wrote:
>>>
>>>  
>>>       
>>>> So do I have interrupt storms here and it is something related to bge?
>>>>    
>>>>         
>>> No, your interrupts look fine.
>>>
>>>  
>>>       
>>>> What else should I check when application hangs again?
>>>>    
>>>>         
>>> The most important thing to know is what is the application doing when
>>> it hangs.  Unfortunately none of the information you provided shows
>>> this.  Next time use the -o wchan argument to ps to find out what
>>> state the process is blocked in.  
>>>       
>> Ok, Here it is:
>>
>>  573  ??  Is     0:00.02 /usr/sbin/inetd -wW -C 60
>> 78721  ??  I      0:00.01 /usr/local/Radiator-3.15/hooks/PSA
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> 78744  ??  Is     0:00.05 sshd: tsgan [priv] (sshd)
>> 78747  ??  S      0:00.02 sshd: tsgan at ttyp0 (sshd)
>>  591  v0  Is+    0:00.00 /usr/libexec/getty Pc ttyv0
>>  592  v1  Is+    0:00.00 /usr/libexec/getty Pc ttyv1
>>  593  v2  Is+    0:00.00 /usr/libexec/getty Pc ttyv2
>>  594  v3  Is+    0:00.00 /usr/libexec/getty Pc ttyv3
>>  595  v4  Is+    0:00.00 /usr/libexec/getty Pc ttyv4
>>  596  v5  Is+    0:00.00 /usr/libexec/getty Pc ttyv5
>>  597  v6  Is+    0:00.00 /usr/libexec/getty Pc ttyv6
>>  598  v7  Is+    0:00.00 /usr/libexec/getty Pc ttyv7
>> 16099  p0- I     20:29.05 perl /usr/local/Radiator-3.15/radiusd 
>> -log_file /var/log/radius/logfile -config_file 
>> /usr/local/Radiator-3.15/voip.cfg -pid_file /
>> 78748  p0  Is     0:00.01 -sh (sh)
>> 78750  p0  I      0:00.01 su
>> 78751  p0  S      0:00.04 _su (csh)
>> 78761  p0  R+     0:00.00 ps ax
>>
>> voiprad#ps axHlwww|grep PSA
>>
>>    0 78721 16099   0   4  0  1696  1184 sbwait I     ??    0:00.01 
>> /usr/local/Radiator-3.15/hooks/PSA
>>
>> voiprad#
>> voiprad#
>> voiprad# ps -o wchan
>> WCHAN
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> piperd
>> wait
>> pause
>> -
>>     
>
> Well, I meant a more complete command than that one ;-) Fortunately
> it's also included in your previous output above ("sbwait").  This
> means that the process is waiting for network traffic (usually waiting
> for another local or remote process to send it data).  So it's not
> obviously pointing to a problem.
>
> Remind me again how you know this isn't an application bug (sorry,
> I've forgotten context)?
>   

This application connects to remote mysql-4.0.x server
and sends some queries and does some calculations and returns.
I tried to run this application from console, and it works
fine. It runs from radius server and it works fine serving
user access requests except sometimes it hangs. It used to
work fine on FreeBSD 5.2-STABLE before upgrading to RELENG-6.
So I guess there is something else.

>   
>>> You can also use kgdb to find out
>>> where it is waiting in the kernel:
>>>
>>> kgdb /dev/mem /boot/kernel/kernel.symbols
>>> info threads
>>> <find the thread corresponding to the process that is blocked>
>>> thread <tid>
>>> bt
>>>  
>>>       
>> Oh, I don't have kernel.symbols file, how to enable it?
>>     
>
> It might be in your kernel compilation directory (possibly called
> kernel.debug).  Otherwise, you'll have to build a new kernel and
> trigger the problem again.
>   
I tried with with kernel.debug without success:

voiprad# kgdb /dev/mem /usr/obj/usr/src/sys/VOIPRAD/kernel.debug
kgdb: bad namelist

thanks,

Ganbold

> kris
>