32GB limit per swap device?

[Matthew Dillon]

    The limitation was ONLY due to a *minor* 32-bit integer overflow in one
    or two *intermediate* calculations in the radix tree code, which I
    long ago fixed in DragonFly.

    Just find the changes in the DFly codebase and determine if they need
    to be applied.

    The swap space radix code (which I wrote long ago) is in page-sized
    blocks, so you actually probably want to keep using a 32-bit integer for
    the block number there to keep the physical memory reservation required
    for the radix tree low.  If you just pop the base block id up to 64 bits
    without adjusting the radix code to overlay a 64 bit bitmap on it you
    waste a lot of physical memory for the same amount of swap reservation.
    This is NOT where the limitation lies.  It was strictly an intermediate
    calculation that caused the original limitation.

    With 32 bit block numbers stored in the radix tree nodes in the swap
    code the physical limitation is something like 1 to 4 TB of total swap.
    I forget exactly but it is at least 1TB.  I've tested 1TB swap partitions
    on DragonFly with just the minor fixes to the original radix tree code.

    --

    Also note that I believe FreeBSD has done away with the interleaved swap.
    I'm not sure why, I guess geom can interleave the swap for you but I've
    always thought that it would be easier to just specify and add the
    partitions separately so one has the flexibility to swapon and swapoff
    the individual partitions on a live system.  Interleaving is important
    because you get an almost perfect performance multiplier.  You don't
    want to just append the swap partitions after each other.

    --

    One last thing:  The amount of wired physical memory required is still
    on the order of ~1MB per ~1GB of swap.  A 32-bit kernel is thus still
    limited by available KVM, effectively limiting you to around ~32G of
    swap depending on various factors if you do not want to run the system
    out of KVM.  I've run upwards of 128G of swap on 32-bit systems but it
    really pushed the KVM use and I would not recommend it.

    A 64-bit kernel is *NOT* limited by KVM.  Swap is effectively limited to
    ~1TB or ~2TB using the original radix code with the one or two intermediate
    overflow fixes applied.  The daddr_t in the original radix code can remain
    32-bits (in DragonFly I typedef'd another name so I could explicitly make
    it 32-bits regardless of daddr_t).

    Large amounts of swap space are becoming important as things like tmpfs
    (and swapcache in DragonFly as well) can really make use of it.  Swap
    performance (the ability to interleave the swap space) is also important
    for the same reason.  Interleaved swap on two SATA-III SSDs is just
    insane... gives you something like 800MB/sec of aggregate read bandwidth.

						-Matt