(Retitled!) /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p (example) USE_TMPFS="data" tmpfs usage for poudriere

In reply to: Mark Millard : "Re: Any clue why "df -m" vs. "du -xsAm" get such different results for the tmpfs in question (403 MiBytes vs. 101 MiBytes)?"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Mark Millard <marklmi_at_yahoo.com>
Date: Sat, 11 Jun 2022 22:03:26 UTC
On 2022-Jun-11, at 13:50, Mark Millard <marklmi@yahoo.com> wrote:

> [I've deleted prior parts of the exchange.]
> 
> I finally have a hypothesis with some evidence about what
> primarily contributes to the 400 or so "1M-blocks Used" in:
> 
> # df -mi /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/
> Filesystem 1M-blocks Used Avail Capacity iused   ifree %iused  Mounted on
> tmpfs           1024  400   623    39%  103659 3828501    3%   /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p
> 
> --where the poudriere-devel is using USE_TMPFS="data" for a
> "bulk -a -c" (that has been running for a little over 2 weeks
> now and is past 1/3 done).
> 
> Taking a quick estimate of the file count in . . ./ref/.p/var/cache/
> I get:
> 
> # ls -Tla /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/ | wc
>   82146  821452 7942485
> 
> Taking 82146*4096 I get: 336,470,016
> Taking 336,470,016 / 1024 / 1024 I get around: 320 Mi
> 
> By contrast:
> 
> # du -xsAm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/
> 46	/usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/
> 
> But:
> 
> # du -xsm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/
> 295	/usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/
> 
> and:
> 
> # du -xsm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/
> 319	/usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/

I see that a little relevant material you have
not been able to see because of what I omitted
from earlier messages --messages not sent to you
at the time.

In my earlier messages I'd shown:

# du -xsAm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p
101	/usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p

# du -xsm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p
68	/usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p

so -xsm resulsts were smaller than -xsAm results and way smaller than
the "df -m" results. The 101-ish -xsAm result for .p/ has stayed fairly
consistent.

It appears that the swapoff activity mentioned below changed
the -xsm results by forcing tmpfs material back into RAM. This
ends up explaining the original 403 MiBytes vs. 101 MiBytes
question that was mentioned in the original message subject.

I mention this as it would appear to be important to trying
to replicate or otherwise test the issue: Context matters
for interpreting "du -xsm" results for tmpfs from what I
can tell.

> This is all after having done a swapoff for the 2 swap partitions
> and a swap on for both and then waiting a while with the "bulk -a -c"
> still running throughout and after. (So under 82 MiBytes is in
> use in swap space at this point.)

The above seems to be what lead to my "du -xsm" result being
fairly close to the "df -m" result.

> poudriere's tmpfs usage reporting does not seem to cover this
> area's tmpfs usage. But, at least for "bulk -a -c" kinds of
> activity, it ends up being the majority of the tmpfs usage for
> USE_TMPFS="data" types of configuration.
> 
> For reference:
> 
> # ls -Tla /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/ | more
> total 301234
> drwxr-xr-x  2 root  wheel  5257152 May 28 02:31:26 2022 .
> drwxr-xr-x  4 root  wheel      128 May 28 01:56:42 2022 ..
> -rw-r--r--  1 root  wheel        2 May 28 02:26:40 2022 builder_active%01
> -rw-r--r--  1 root  wheel        2 May 28 02:31:24 2022 builder_active%02
> -rw-r--r--  1 root  wheel        2 May 28 02:31:26 2022 builder_active%03
> -rw-r--r--  1 root  wheel        2 May 28 02:31:26 2022 builder_active%04
> -rw-r--r--  1 root  wheel       18 May 28 01:56:51 2022 originspec-pkgname%accessibility_accerciser
> -rw-r--r--  1 root  wheel       19 May 28 01:56:51 2022 originspec-pkgname%accessibility_at-spi2-atk
> -rw-r--r--  1 root  wheel       20 May 28 01:56:51 2022 originspec-pkgname%accessibility_at-spi2-core
> -rw-r--r--  1 root  wheel       11 May 28 01:56:51 2022 originspec-pkgname%accessibility_atk
> -rw-r--r--  1 root  wheel       13 May 28 01:56:51 2022 originspec-pkgname%accessibility_atkmm
> -rw-r--r--  1 root  wheel       17 May 28 01:56:51 2022 originspec-pkgname%accessibility_caribou
> . . .
> -rw-r--r--  1 root  wheel       17 May 28 01:56:55 2022 pkgname-originspec%zutils-1.11
> -rw-r--r--  1 root  wheel       15 May 28 02:03:32 2022 pkgname-originspec%zxfer-1.1.7
> -rw-r--r--  1 root  wheel       14 May 28 02:03:00 2022 pkgname-originspec%zxid-1.42_1
> -rw-r--r--  1 root  wheel       19 May 28 02:04:03 2022 pkgname-originspec%zxing-cpp-1.3.0
> -rw-r--r--  1 root  wheel       12 May 28 01:59:41 2022 pkgname-originspec%zydis-3.1.0
> -rw-r--r--  1 root  wheel       18 May 28 01:57:13 2022 pkgname-originspec%zynaddsubfx-3.0.6,2
> -rw-r--r--  1 root  wheel        9 May 28 02:02:07 2022 pkgname-originspec%zyre-2.0.1
> -rw-r--r--  1 root  wheel       14 May 28 01:59:41 2022 pkgname-originspec%zziplib-0.13.72_1
> -rw-r--r--  1 root  wheel       14 May 28 02:03:00 2022 pkgname-originspec%zzuf-0.13_1
> -rw-r--r--  1 root  wheel       13 May 28 01:56:45 2022 ports_metadata%top_git_hash
> -rw-r--r--  1 root  wheel        4 May 28 01:56:49 2022 ports_metadata%top_unclean



===
Mark Millard
marklmi at yahoo.com