cron pile up! Lot's of "cron: running job (cron)"

[Kevin Kinsey]

Rudy wrote:
> 
> cron jobs seem to get stuck.  Not always, but within a day, there are at
> least 20 stuck.  It is not always the same cronjob that does the
> sticking.  :)   When this occurs, I can run ps ax| grep cron and get a
> bunch of lines like this:
> 
> 51921  ??  D      0:00.00 cron: running job (cron)
> 51922  ??  IVs    0:00.00 cron: running job (cron)
> 52544  ??  D      0:00.00 cron: running job (cron)
> 52545  ??  IVs    0:00.00 cron: running job (cron)
> 54418  ??  D      0:00.00 cron: running job (cron)
> 54419  ??  IVs    0:00.00 cron: running job (cron)
> 54667  ??  D      0:00.00 cron: running job (cron)
> 54668  ??  IVs    0:00.00 cron: running job (cron)
> 55835  ??  D      0:00.00 cron: running job (cron)
> 55836  ??  IVs    0:00.00 cron: running job (cron)
> 
> What is going on?  Please help me remedy this situation.
> 
> The PID numbers next to cron's with a STATE of "IVs" show up in
> /var/log/cron, for example:
> 
> # grep 54668 /var/log/cron
> Dec  2 22:32:00 pita /usr/sbin/cron[54668]: (root) CMD
> (/root/bin/raid-status.sh CRON)
> # grep 55836 /var/log/cron
> Dec  2 22:40:00 pita /usr/sbin/cron[55836]: (root) CMD
> (/root/bin/10minutes.mail.sh | mail -E -s "[ERROR]
> mail.monkeybrains.net"  example at example.com)
> 
> 
> If I run 'lsof' I can find these open handles:
> 
> cron      54668             root  cwd   VDIR       0,80        512   
> 471040 /var/cron
> cron      54668             root  rtd   VDIR       0,77        
> 512        2 /
> cron      54668             root  txt   VREG       0,82      32496   
> 122864 /usr/sbin/cron
> cron      54668             root  txt   VREG       0,77     162712    
> 49929 /libexec/ld-elf.so.1
> cron      54668             root  txt   VREG       0,77      44788    
> 49922 /lib/libutil.so.5
> cron      54668             root  txt   VREG       0,77     941952    
> 49923 /lib/libc.so.6
> cron      54668             root  txt   VREG       0,82      19277   
> 826439 /usr/local/lib/nss_mysql.so.1
> cron      54668             root  txt   VREG       0,82     413626   
> 826986 /usr/local/lib/mysql/libmysqlclient.so.15
> cron      54668             root  txt   VREG       0,77      64604    
> 49928 /lib/libz.so.3
> cron      54668             root  txt   VREG       0,77     107432    
> 49918 /lib/libm.so.4
> cron      54668             root  txt   VREG       0,77      28648    
> 49916 /lib/libcrypt.so.3
> cron      54668             root    0u  PIPE 0xca02c660      
> 16384          ->0xca02c718
> cron      54668             root    1u  PIPE 0xcc473250          
> 0          ->0xcc473198
> cron      54668             root    2u  PIPE 0xcc473250          
> 0          ->0xcc473198
> cron      54668             root    5u  unix 0xc6665858        
> 0t0          ->0xc67e89bc
> cron      54667             root  cwd   VDIR       0,80        512   
> 471040 /var/cron
> cron      54667             root  rtd   VDIR       0,77        
> 512        2 /
> cron      54667             root  txt   VREG       0,82      32496   
> 122864 /usr/sbin/cron
> cron      54667             root  txt   VREG       0,77     162712    
> 49929 /libexec/ld-elf.so.1
> cron      54667             root  txt   VREG       0,77      44788    
> 49922 /lib/libutil.so.5
> cron      54667             root  txt   VREG       0,77     941952    
> 49923 /lib/libc.so.6
> cron      54667             root  txt   VREG       0,82      19277   
> 826439 /usr/local/lib/nss_mysql.so.1
> cron      54667             root  txt   VREG       0,82     413626   
> 826986 /usr/local/lib/mysql/libmysqlclient.so.15
> cron      54667             root  txt   VREG       0,77      64604    
> 49928 /lib/libz.so.3
> cron      54667             root  txt   VREG       0,77     107432    
> 49918 /lib/libm.so.4
> cron      54667             root  txt   VREG       0,77      28648    
> 49916 /lib/libcrypt.so.3
> cron      54667             root    0u  VCHR       0,26        0t0       
> 26 /dev/null
> cron      54667             root    1u  VCHR       0,26        0t0       
> 26 /dev/null
> cron      54667             root    2u  VCHR       0,26        0t0       
> 26 /dev/null
> cron      54667             root    3u  PIPE 0xca02c660      
> 16384          ->0xca02c718
> cron      54667             root    4u  PIPE 0xca02c718          
> 0          ->0xca02c660
> cron      54667             root    5u  unix 0xc6665858        
> 0t0          ->0xc67e89bc
> cron      54667             root    6u  PIPE 0xcc473198      
> 16384          ->0xcc473250
> cron      54667             root    7u  unix 0xc67e86f4        
> 0t0          ->(none)
> cron      54667             root    8u  PIPE 0xcc473250          
> 0          ->0xcc473198
> 
> What is going on?  Is my libnss_mysql acting up?

What scripts are running?  Care to sanitize the crontab
file and show it as well?  Barring hardware issues (disk
errors, etc.), I'd suspect the scripts.

What about server load averages?

KDK
-- 
Law of Continuity:
	Experiments should be reproducible.
	They should all fail the same way.