Re: 504 gateway time-outs
- In reply to: Mark Millard : "Re: 504 gateway time-outs"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 03 Mar 2026 07:03:59 UTC
On Sat, Feb 28, 2026 at 8:55 PM Mark Millard <marklmi@yahoo.com> wrote: > On 2/28/26 17:57, Kevin Oberman wrote: > > On Sat, Feb 28, 2026 at 5:48 PM Kevin Oberman <rkoberman@gmail.com > > <mailto:rkoberman@gmail.com>> wrote: > > > > On Sat, Feb 28, 2026 at 3:33 AM Graham Perrin > > <grahamperrin@gmail.com <mailto:grahamperrin@gmail.com>> wrote: > > > > On 28/02/2026 10:32, Mark Millard wrote: > > > … > > > > > > The following got "504 Gateway Time-out" when I tried them: > > > > > > <https://pkg-status.freebsd.org/beefy24/build.html? > > mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d > > <https://pkg-status.freebsd.org/beefy24/build.html? > > mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d>> > > > > > > <https://pkg-status.freebsd.org/beefy23/build.html? > > mastername=150amd64-default&build=df4f957ea181 <https://pkg- > > status.freebsd.org/beefy23/build.html?mastername=150amd64- > > default&build=df4f957ea181>> > > > > > > Both somewhat slow to load, however they do load for me. > > > > Thanks > > > > > > beefy22 is showing hte same issues. Something is tying up the > > builders with builds hanging in "build-depends" and "lib-depends" > > for very long periods of time, as long as over 4 hours) with high > > load averages. This has been going on for months and seems to be > > getting worse. Chromium builds which used to take around 20 hours > > are now running around 40 and I really don't think that this is just > > code bloat. At times, about half of the active builds are in some > > state other than "build"; mostly one of the depends states which > > should never take long as all dependencies are built before a build > > is started. > > > > After some time passes (often hours) things clear up. Dependencies > > are suddenly loaded in a few minutes or less. Note that things do > > continue to complete, but the 10 minute averages are in single > > digits and frequently go to 0. During this time, my connection to > > beefy22 will timeout and sometimes I can't establish a connection. > > > > I don't have any special access to the machines... just via pkg- > > status, so I'm fairly limited in any analysis. > > > > > > Just before I started my last message, all was well, but I last saw > > were a dozen builds in one of the "depends" states and "impulse" at 74 > > when I lost my connection. I can reconnect to the beefy22 status page, > > but it has not updated for several minutes. > > [Do not expect that I edited my all notes in presentation-text-order.] > > Via <https://pkg-status.freebsd.org/builds?type=package&all=1> : > > beefy22: 143amd64-default 17:21:38 > > Accurate would be over 27 hrs (see below). > > Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space. > > < > https://pkg-status.freebsd.org/beefy22/build.html?mastername=143amd64-default&build=df4f957ea181 > > > : > > Load Averages Swapinfo Elapsed > (100%) 63.94 69.19 70.56 0.24% 27:37:33 > > > -- > > Kevin Oberman, Part time kid herder and retired Network Engineer > > E-mail: rkoberman@gmail.com <mailto:rkoberman@gmail.com> > > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 > > > I see such on beefy24 (main-amd64-default) currently but also got lucky > for an detailed page display. Notably: > > < > https://pkg-status.freebsd.org/beefy24/build.html?mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d > > > > shows: > > Load Averages Swapinfo Elapsed > (107%) 102.71 106.93 107.91 55.15% 26:35:05 > > but <https://pkg-status.freebsd.org/builds?type=package&all=1> shows > 23:10:02 for Elapsed. > > (Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The > poudriere configuration is allowing 45 builders at once. Each builder is > likely to be configured to allow, say, up to 3 make processes, > MAKE_JOBS_NUMBER=3 . It is difficult to look at logs now to check on the > figure. I've no information about the amount of RAM or the size of the > SWAP space for beefy24 or beefy23.) > > The longest-still-active port-package builders elapsed-time-so-far > figures shown were: > > qt6-webengine-6.10.2 23:33:08 > chromium-145.0.7632.109 20:43:03 > electron39-39.7.0 20:41:59 > apache-openoffice-devel-4.2.1768900765_1,4 11:20:05 > > Those are likely the major users of tmpfs space that competes for > RAM+SWAP for USE_TMPFS=all (or other large USE_TMPFS settings). > > I'd guess that the paging is thrashing for the above but have no way of > checking. It could be that some use of TMPFS_BLACKLIST to avoid the use > of TMPFS for the port-packages that have huge TMPFS involved could help > (without otherwise disabling USE_TMPFS=all to still speed up most > builders). > > (I will note that TMPFS_BLACKLIST entries do not necessarily end up with > no tmpfs use, just less. But that less can help avoid paging that ends > up thrashing.) > > Note during my editing/research and other activities, the web page had > not updated at all after its initial display. It is now significantly > later than when I started editing this note. > > Later: Hmm. > > https://pkg-status.freebsd.org/builds?type=package&all=1 > > 05:48:27 beefy23 (150amd64-default) > > but . . . > > < > https://pkg-status.freebsd.org/beefy23/build.html?mastername=150amd64-default&build=df4f957ea181 > > > > Load Averages Swapinfo Elapsed > (112%) 107.81 100.28 97.16 0.04% 27:42:2 > > > -- > === > Mark Millard > marklmi at yahoo.com > beefy22 started a full ports build Saturday at 00:10 UTC. Most of last Saturday and early Sunday had 'impulse' of no more than double digits and over a 20 hour period the number of completed builds was just over 1000. Chromium and electron were killed when they reached 48 hours. The build now exceeds 72 hours. Last month's fill build was about 64.5 hours. It really looks like things are getting worse. I suspect some resource is reaching exhaustion. One suggestion I saw that looks possible is tmp which I believe is a tempfs, but I really can't tell too much with what is visible on pkg-status. On to wild speculation. Maybe reduce the number of concurrent builds. If a small amount of DRAM could help, that would be great, but DRAM is getting very expensive and reducing the number of builds might speed things up at no cost. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683