dd is returning different content using same inputs

[Arrigo Marchiori]

Hello,

On Thu, Jun 08, 2017 at 05:15:15PM +0800, Christopher Hall wrote:

> Hello Dewayne,
> 
> On Thu, 8 Jun 2017 18:10:11 +1000, Dewayne Geraghty
> <dewaynegeraghty at gmail.com> wrote:
> 
> > I'm attempting to skip the first three bytes of a file and then pull
> > the header out of message. Please note the file size varies between
> > 105 and 54 bytes, even though the input file and everything else
> > remains constant, on
> > 
> > FreeBSD hathor 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #0 r319562M:
> > Mon Jun  5 03:13:08 AEST 2017
> > root at hathor:/110007/D/K8/hqdev-amd64-smp-vga amd64 1100514 1100514
> > 
> > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384
> > count=1 of=encheader ; ls -lrth  encheader
> > 0+1 records in
> > 0+1 records out
> > 105 bytes transferred in 0.000026 secs (4071661 bytes/sec)
> > -rw-r--r--  1 root  wheel   105B Jun  8 16:35 encheader
> > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384
> > count=1 of=encheader ; ls -lrth  encheader
> > 0+1 records in
> > 0+1 records out
> > 67 bytes transferred in 0.000026 secs (2554912 bytes/sec)
> > -rw-r--r--  1 root  wheel    67B Jun  8 16:35 encheader
> > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384
> > count=1 of=encheader ; ls -lrth  encheader
> > 0+1 records in
> > 0+1 records out
> > 68 bytes transferred in 0.000024 secs (2867263 bytes/sec)
> > -rw-r--r--  1 root  wheel    68B Jun  8 16:35 encheader
> > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384
> > count=1 of=encheader ; ls -lrth  encheader
> > 0+1 records in
> > 0+1 records out
> > 54 bytes transferred in 0.000025 secs (2167717 bytes/sec)
> > -rw-r--r--  1 root  wheel    54B Jun  8 16:35 encheader
> > # rm encheader;dd if=/tmp/tLbdoJ/encmessage bs=1 skip=3 | dd bs=384
> > count=1 of=encheader ; ls -lrth  encheader
> > 0+1 records in
> > 0+1 records out
> > 91 bytes transferred in 0.000024 secs (3751494 bytes/sec)
> > -rw-r--r--  1 root  wheel    91B Jun  8 16:35 encheader
> > 
> > I used this around May 8th (from a OS and ports rebuild on 4th May)
> > and was performing some final testing on an 11.1 Prerelease built on
> > Jun 5.
> > 
> It works similarly on TrueOS (FreeBSD 12) I suspect it is because you
> block sizes are different on the input and output dd commands.
> I would think that the output block size (obs) on 1st dd needs to be
> specified the same input block size on receiving dd to make it work
> correctly.

I also agree; the ``funny'' behavior is probably given by the pipe and
the different block sizes.

The manual page of dd(1) says about the bs parameter:

[...]
>     bs=n     Set both input and output block size to n bytes, superseding the
>              ibs and obs operands.  If no conversion values other than
>              noerror, notrunc or sync are specified, then each input block is
>              copied to the output as a single block without any aggregation
>              of short blocks.

The second dd seems to receive non-deterministic amounts of data from
the first, and due to the bs parameter, it does not wait until 384
bytes are received, but it outputs its partial block immediately and
then quits.

> > This works correctly so perhaps I'm not using dd correctly?
> > dd if=/tmp/tLbdoJ/encmessage skip=1 bs=3  | dd bs=1 count=384
> > of=encheader ; hd encheader|head -n1; ls -l encheader

If I understood correctly, you could issue a single command:

$ dd if=inputfile bs=1 skip=3 count=384 of=outputfile

I hope this helps.
-- 
rigo

http://rigo.altervista.org