Is a successful call to write(2) atomic?
Ronald F. Guilmette
rfg at tristatelogic.com
Thu Jun 17 19:12:05 UTC 2021
I need to say "Thank you all!" to all of the folks who chimed in and who
have graciously tried to help me to get this problem sorted out.
Although I didn't try the idea of passing buffers/lines back up from
the "worker bee" child processes to the parent (which would then write
those lines to FD 1 in a nice sequentialized fashion) I did try all
of the other suggesions that have been made here, specifically (a) setting
O_APPEND on the FD, and also (b) wrapping my calls to write() in calls
to flock() -and- also (c) embedding in that flock section an additional
call to lseek() to position next following write, reliably, at the
current actual EOF of the output file.
Every step along the way, the "garbled output lines" problem seemed to
get smaller, but at the end, it was still present. So I added some
more code to do more & better checking of the integrity of the data
lines before I even tried to write them to FD #1.
And sure enough, once I had that additional "input sanitizing" code in
place, the problem went away entirely.
Now I'm re-running the whole process again, but with all of the extra
stuff that people suggested to me, i.e. the code that sets O_APPEND
on the FD, the calls to flock() and the call to lseek() all #ifdef'd
out. The results aren't final yet, but for the first 18,000+ lines of
output, the "garbled output lines" problem does indeed seem to be gone
So, bottom line, it now appears that I owe everyone an apology for taking
up your time. My bad! It now appears that the -actual- problem was
most probably related to "logic errors" in my own code, and more
specifically my failure to adequately sanitize input data (which is
being acquired from remote network hosts that I do not control) before
even calling write().
I'm suitably embarassed to admit this, but I feel morally compelled to
confess my sins and to take responsibility for my own screw ups.
In any case, this has all been quite educational, and for that I thank
More information about the freebsd-questions