sed - remove nul lines from file

James B. Byrne byrnejb at
Tue Nov 7 17:13:07 UTC 2017

I have a data file created by an ancient proprietary scripting
language called QTP.  There is a bug in this program which, on
occasion, manifests itself by inserting output records consisting
entirely of nul (^@) (\x00) bytes at regular intervals.  In the
present case every 47th. record consists entirely of nuls.

The purpose of this data file is to feed a psql COPY statement for
loading into a PostgreSQL database.  The presence of the NUL
characters prevents this.  I have previously used the tr utility to
remove the NUL characters but this requires me to manually remove the
residual empty lines.

I have tried various permutations of the sed invocation reproduced
below to remove these lines directly but without success.  The
examples that I have found on StackExchange and various other
self-help sites do not give the results claimed, at least not for me
on FreeBSD. So, I would appreciate if anyone here can point out what I
am doing wrong or how the sed on FreeBSD differs in behaviour for that
used in the examples I have found.

Given a file INFILE with records containing the following:

. . .
*93566000008166*,*CCTL*,*3072 49534494                 *
*93566000008166*,*CCTL*,*3072 49534493                 *
*93566000008166*,*CCTL*,*3072 49534497                 *
*93566000015962*,*CCTL*,*8156 4171000541               *
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ . . .
*93566000198850*,*CCTL*,*417 1003874                   *
*93566000010320*,*CCTL*,*8084 2601553853102            *
. . .

I wish to remove (all) the line(s) with the nul (^@) characters.  I
have tried this:

sed '/^\x00*$/d' INFILE > INFILE.sed

and this:

sed _E '/^\x00*$/d' INFILE > INFILE.sed

but neither these nor the many other combinations that I have tried
remove the lines.  What is the method of accomplishing this in sed or
is it not possible?

***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:ByrneJB at
Harte & Lyne Limited
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3

More information about the freebsd-questions mailing list