Re: Possible regression in main causing poor performance

On Fri, Aug 18, 2023 at 07:09:10PM -0700, Mark Millard wrote:
> Glen Barber <gjb_at_freebsd.org> wrote on
> Date: Sat, 19 Aug 2023 00:10:59 UTC :
> 
> > I am somewhat inclined to look in the direction of ZFS here, as two
> > things changed:
> > 
> > 1) the build machine in question was recently (as in a week and a half
> > ago) upgraded to the tip of main in order to ease the transition from
> > this machine from building 14.x to building 15.x;
> > 2) there is the recent addition of building ZFS-backed virtual machine
> > and cloud images.
> > 
> > . . .
> > The first machine runs:
> > # uname -a
> > FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \
> > amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023
> 
> I'm confused:
> 
> "the build machine in question was recently (as in a week and a half
> ago) upgraded to the tip of main in order to ease the transition from
> this machine from building 14.x to building 15.x"? But the above
> kernel is from mid July? (-aKU was not used to also get some clue
> about world from the pair of 140009? that would show.)
> 

Err, right.  I'm confused too, and apparently looked at a different
machine but copied 'uname -a' from the correct machine.  The kernel and
userland are always in sync on these machines.

  # uname -UK
  1400093 1400093

> > Last week's snapshot builds were completed in a reasonable amount of
> > time:
> > 
> > root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./builds-14.conf ; echo ^G
> > 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs
> > 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots
> > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release
> > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports
> > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc
> > 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release
> > [...]
> > 20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap
> > 20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap
> > 20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap
> > 
> > Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE.
> > 
> > With no changes to the system, no stale ZFS datasets laying around from
> > last week (everything is a pristine environment, etc.), this week's
> > builds are taking forever:
> 
> My confusion may extend to this "no changes" status vs. the uname
> output identifying the kernel is from mid July.
> 

It is admittedly a month versus two weeks difference here.  My fault.

> > root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./builds-14.conf ; echo ^G
> > 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs
> > 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots
> > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release
> > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports
> > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc
> > 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release
> > [...]
> > 20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap
> > 20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap
> > 20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap
> > 
> > Note, it is just about 4 minutes past 00:00 UTC as of this writing, so
> > we are about to cross well over the 24-hour mark, and cloud provider
> > images have not yet even started.
> > 
> > . . .
> 
> In:
> 
> https://lists.freebsd.org/archives/freebsd-current/2023-August/004314.html
> ("HEADS UP: $FreeBSD$ Removed from main", Wed, 16 Aug 2023)
> 
> Warner wrote:
> 
> QUOTE
> . . . , but there's no incremental building
> with this change, . . . Also: expect long build times, git fetch times, etc
> after this.
> END QUOTE
> 
> Might this be contributing? How long did those two
> "Checking out . . ." take? Similar time frames?
> 

Checkout times are negligible.  They are pristine environments, so there
are no delta resolutions, etc.  Everything is clean from the start.

Last week:
20230811-00:03:13       INFO:   Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release
20230811-00:05:27       INFO:   Creating ZFS snapshot zroot/releng/14-src-snap@clone
20230811-00:05:33       INFO:   Checking out https://git.FreeBSD.org//ports.git (main) to /releng/scripts-snapshot/ports
20230811-00:09:44       INFO:   Creating ZFS snapshot zroot/releng/14-ports-snap@clone

This week:
20230818-00:15:46       INFO:   Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release
20230818-00:17:57       INFO:   Creating ZFS snapshot zroot/releng/14-src-snap@clone
20230818-00:18:01       INFO:   Checking out https://git.FreeBSD.org//ports.git (main) to /releng/scripts-snapshot/ports
20230818-00:22:09       INFO:   Creating ZFS snapshot zroot/releng/14-ports-snap@clone

Glen