[Bug 288755] sysutils/debootstrap: missing packages due to pcregrep buffer size

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288755

            Bug ID: 288755
           Summary: sysutils/debootstrap: missing packages due to pcregrep
                    buffer size
           Product: Ports & Packages
           Version: Latest
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: Individual Port(s)
          Assignee: nc@FreeBSD.org
          Reporter: bugzilla@tunedal.net
          Assignee: nc@FreeBSD.org
             Flags: maintainer-feedback?(nc@FreeBSD.org)

Created attachment 262833
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=262833&action=edit
Patch for debootstrap

Installing Debian Bullseye using debootstrap version 1.0.128+nmu2 (via bastille
in my case but presumably also without bastille) produces a broken environment
with several packages missing, e.g. zlib, tar, login and passwd.

The resulting log file contains lots of messages about prcegrep exceeding its
buffer size:

  root@host:~ # cat
/usr/local/bastille/releases/Debian11/debootstrap/debootstrap.log|grep "too
long"|uniq -c
    28 pcregrep: line 933034 of file
/usr/local/bastille/releases/Debian11/var/lib/apt/lists/http:__deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages
is too long for the internal buffer

The default buffer size in pcregrep is 20 kB, according to the docs. The line
in question is 65653 characters long:

  root@host:~ # head -933034
/usr/local/bastille/releases/Debian11/var/lib/apt/lists/http:__deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages|tail
-1|wc -c
     65653

Running apt in the created environment fails because zlib is missing:

  root@host:~ # bastille cmd debianjail apt update
  [debianjail]:
  apt: error while loading shared libraries: libz.so.1: cannot open shared
object file: No such file or directory
  [debianjail]: 127

Changing the buffer size to 1 MB (or some other large value) by replacing all
three occurrences of "pcregrep" to "pcregrep --buffer-size=1M" in
/usr/local/share/debootstrap/functions fixes the problem. I'm attaching a patch
to do this.

Diffing the debootstrap logs before and after the fix shows that more URLs are
processed and the pcregrep messages are gone:

  root@host:~ # grep -c ' URL:' debootstrap-orig.log
  126
  root@host:~ # grep -c ' too long ' debootstrap-orig.log
  28

  root@host:~ # grep -c ' URL:' debootstrap-fixed.log
  166
  root@host:~ # grep -c ' too long ' debootstrap-fixed.log
  0

Diffing the bastille logs shows these additional packages installed after the
fix (i.e. missing without the fix):

  root@host:~ # diff -u bastille-bootstrap-orig.log
bastille-bootstrap-fixed.log|grep '^[+-].*Extracting'
  +I: Extracting bsdutils...
  +I: Extracting libblkid1...
  +I: Extracting liblzma5...
  +I: Extracting libmount1...
  +I: Extracting libsemanage-common...
  +I: Extracting libsemanage1...
  +I: Extracting libsepol1...
  +I: Extracting libsmartcols1...
  +I: Extracting libsystemd0...
  +I: Extracting libudev1...
  +I: Extracting libuuid1...
  +I: Extracting libxxhash0...
  +I: Extracting login...
  +I: Extracting lsb-base...
  +I: Extracting mount...
  +I: Extracting passwd...
  +I: Extracting sed...
  +I: Extracting sysvinit-utils...
  +I: Extracting tar...
  +I: Extracting tzdata...
  +I: Extracting util-linux...
  +I: Extracting zlib1g...

Steps to reproduce:

  root@host:~ # bastille bootstrap debian-bullseye
  ... say "y" to everything ...

  root@host:~ # bastille create -L my-debian-jail debian-bullseye
192.168.123.123
  ... lots of error messages ...

  root@host:~ # bastille cmd my-debian-jail apt update

While the attached patch fixes the problem (assuming lines are not longer than
1 MB), ideally, to make troubleshooting easier, this sort of error would bubble
up and cause the command to fail rather than seem to succeed and produce a
broken environment. Maybe the error handling in the script can be reworked
somehow to improve this?

-- 
You are receiving this mail because:
You are the assignee for the bug.