cron pile up! Lot's of "cron: running job (cron)"

Dan Nelson dnelson at allantgroup.com
Mon Dec 3 23:44:48 PST 2007


In the last episode (Dec 03), Support (Rudy) said:
> Below is part of the cron...  Seems like any random cronjob can get
> clogged up... load varies from 0.2 to 1.0 on this dual-core box.  I
> rebooted the box -- cron's continue to slowly pile up.
> 
> One of the cronjobs that is 'stuck' is this one: /root/bin/raid-status.sh
> which can be found here:
>  http://www.monkeybrains.net/~rudy/example/raid_status.html
> 
> Forgot to mention, I am running:
>   6.2-STABLE FreeBSD 6.2-STABLE #3: Thu May 31 01:18:15 PDT 2007
> 
> OH, ps shows this:
> 58383  ??  D      0:00.00 cron: running job (cron)
> 58384  ??  IVs    0:00.00 cron: running job (cron)

In general, when troubleshhoting, "ps axlw" is a more useful command.
It adds among other columns, the MWCHAN one, which details exactly why
a process is stuck in the D state.  

Anyway, cron does a fork and then a vfork creating a child and a
grandchild process.  I'm sort of surprised at the amount of code
between vfork and exec in the grandchild in
/src/usr.sbin/cron/cron/do_command.c .  Since process 3 is actually
using process 2's address space one must be extremely careful not to
modify static variables or change other global state that would affect
the parent once it resumes execution, and all the logging,
environment-setting, and user-context calls are certain to mess with
the parent's state, especially with nss modules in the mix.  I'd
personally recompile cron with all vforks replaced with fork and see
what happens.

It couldn't hurt to update to a newer kernel version along the RELENG_6
branch as a test, I guess.  Note that your uname will change to
6.3-PRERELEASE, but apart from causing lsof to complain, you should be
okay.

> /var/log/cron has this entry:
> Dec  3 20:16:00 pita /usr/sbin/cron[58384]: (root) CMD  (/root/bin/raid-status.sh CRON)
> 
> BUT there is no 'raid-status.sh' stuck in the "ps axw".  Seems like the 
> vfork set off the cronjob, it ran, but then cron didn't 'stop' executing.  
> Any debuggin tips?

Can you tell if raid-status.sh ever ran?  i.e. is process 2
stuck at the start of vfork or at the end.

BTW, here's a minimal example of the danger of putting code between
vfork and exec:

#include <err.h>
#include <stdio.h>
#include <unistd.h>

int main(void)
{
	int i = 1;
	switch (vfork())
	{
	case -1:
		err(1, "vfork failed");
		break;
	case 0:
		/* child */
		i = 2;
		execl("/usr/bin/true", "true", NULL);
		_exit(0);
		break;
	default:
		break;
	}
	printf("in parent, i is %d\n", i);
	return 0;
}


-- 
	Dan Nelson
	dnelson at allantgroup.com


More information about the freebsd-questions mailing list