ktrace and KTR_DROP
peadar.edwards at gmail.com
Sun Jul 3 19:54:08 GMT 2005
On 7/1/05, Robert Watson <rwatson at freebsd.org> wrote:
> There are two benefits to the current ktrace dispatch model:
> (1) Avoiding untimely sleeping in the execution paths of threads that are
> being traced.
> (2) Allowing the traced thread to run ahead asynchronously, hopefully
> impacting performance less.
Agreed, as long as the rate it needs to trace at doesn't exceed the rate the
trace can be recorded at: otherwise, it needs to be flow controlled, either by
calling vn_write, or by waiting for another thread to do so. Just to re-iterate,
I understand that not slowing down the target process is probably beneficial in
some circumstances, but you can only do that by loosing trace information, and
in the vast majority of cases that's exactly what you don't want.
> One of the things I've been thinking for a few years is that I think I
> actually preferred the old model better, there processes (now threads)
> would hang a "current record" off of their process (now thread) structure,
> and fill it in as they went along. The upsides of this have to do with
> the downsides of the current model: that you don't allow fully
> asynchronous execition of the threads with respect to queueing the records
> to disk, so you don't run into "drop" scenarios, instead slowing down the
> process. Likewise, the downsides.
> In the audit code, we pull from a common record queue, but we allocate the
> record when the system call starts for each process -- if there aren't
> records available (or various other reliability-related conditions fail,
> such as adequate disk space), we stall the thread entering the kernel
> until we can satisfy its record allocation requirements.
I don't follow: looking at the code from RELENG_4, it looks pretty similar to
what's there already: create a requst, call ktrwrite, free request. By adding
the flow control with the patch, you just end up waiting for the free queue to
be non-empty, rather than waiting to aquire the vnode lock and do the write. Am
I missing something? Do I need to go back further in time?
> There are two cases where I really run into problems with the current
> (1) When I'm interacting with a slow file system, such as NFS over
> 100mbps, I will always lose records, because it doesn't take long for
> the process to get quite ahead of the write-behind.
It doesn't even need NFS: syscall throughput is much better than I/O
> (2) When I trace more than one process at a time, the volume of records
> overwhelms the write-behind.
As long as you need to do physical writes for the syscalls, I don't
know how you'l ever avoid that...
> Write coalescing/etc is already provided "for free" by pushing the writes
> down into the file system, so other than slowing down the traced process a
> little, I think we don't lose much by moving back to this model. And if
> we pre-commit the record storage on system call entry (with the exception
> of paths, which generally require potential sleeps anyway), we probably
> won't hurt performance all that much, and avoid sleeping in bad places.
I'm not sure I follow: are you saying you'd rather move the ktrace
resource aquisition out of where the sleep will be more critical? I
don't see how you can avoid having the ktrace'd thread wait for as
long as it takes for the writer thread to catch up with the system to
More information about the freebsd-arch