Debugging periodic scripts

[Jeremy Chadwick]

(Please keep me CC'd as I'm not subscribed)

Hi folks,

Does anyone know how to go about debugging periodic scripts, such as
getting useful debug output from start to finish?

Basically the situation is this:

- We have 5 systems which are RELENG_8 (some 8.2-STABLE, and a couple
  8.3-PRERELEASE).  These are all bare metal boxes, not VMs.
- All the machines have WITHOUT_IPFILTER=true defined in src.conf.
- All the machines also have this in their /etc/periodic.conf :
  daily_status_security_ipfdenied_enable="no"
- All the machines run ntpd and do not have clock skew problems or other
  odd anomalies (they all work great hardware-wise) or filesystem issues
  (all are using UFS2).

On 2 of the systems, /etc/periodic/security/510.ipfdenied gets run
during "periodic security" even though it's explicitly shut off in
periodic.conf.  Thus on these 2 systems, our security mails contain this
line: ipfstat: not found

I've checked permissions on everything I can think of (from / all the
way down) but it all looks fine.  I even wrote a small forloop to check
all the systems' periodic.conf files and ensure the ipfdenied_enable
line is proper (no weird trailing or preceding spaces, high-bit
characters, DOS CRs, etc.) and they all check out (1 line, 44 characters
long).

One of the boxes was even recently rebuilt from scratch (full format +
OS reinstall); it exhibited this problem prior to the rebuild, as well
as after the rebuild.

None of the systems have any unique changes to /root dotfiles nor the
shell adjustments in things like /etc/profile, /etc/csh*, etc..

I've tried doing this:

(sh -x /etc/periodic/security/510.ipfdenied >& /dev/stdout) | grep ipfdenied

Which returns exactly what I would expect:

+ daily_status_security_ipfwdenied_enable=YES
+ daily_status_security_ipfdenied_enable=YES
+ daily_status_security_ipfwlimit_enable=YES
+ daily_status_security_ipf6denied_enable=YES
+ daily_status_security_ipfdenied_enable=no

The first 4 come from /etc/defaults/periodic.conf, the last comes from
/etc/periodic.conf.

Running /etc/periodic/security/510.ipfdenied from a root shell results in
no output.

Editing /etc/periodic/security/510.ipfdenied's hashbang line to use -x
doesn't change the behaviour either (maybe stderr gets sent to
/dev/null?), whether I run it by hand as a script or via "periodic
security".

Other settings in periodic.conf are in fact honoured, such as
daily_status_smart_enable and some others, so I'm inclined to believe
periodic.conf is indeed being read.  I don't know what's making this
situation.

I haven't resorted to using ktrace yet but will down the road assuming
nobody has any other ideas.  Otherwise something tells me I'm going to
have to go look at the periodic source code to figure out what's going
on under the hood.

Thoughts/ideas?

-- 
| Jeremy Chadwick                              jdc at parodius.com |
| Parodius Networking                     http://www.parodius.com/ |
| UNIX Systems Administrator                 Mountain View, CA, US |
| Making life hard for others since 1977.             PGP 4BD6C0CB |