Closet Server Consolidation

You asked, so here it is… this is a work in progress, function over form for now…

Fitting old things/life into new location - so making adjustments incrementally - had 2 very loud rack-mount servers move from a nice cool/isolated basement to a hot and the middle of living space closet. So, now I have some serious heat/noise issues.

Step 1: “the build”

  • swap out 2x2690v4 MB from disk array case
  • swap in asrock 10G epyc board EPYCD8-2T
  • 7351p (16core 2.5-2.9GHz) + 64G of ECC 2966 ram
  • 10x8T disks
  • optane 280G cache drive
  • some more nvme later…
  • 10G and 40G networking (intel/mellanox)
    This is where we start - the existing server in this case
    Yank out the guts that will move over
    This board/setup will move to a compute-only node and no longer handle the disk array
    The pile of new goodies…
    This board is very thick and stiff as a board - significantly thicker than prior AsRock rack boards I’ve had.
    CPU in - oops - tray CPU doesn’t come with torque wrench… approximate torque engaged…
    Fan orientation less than ideal - we’ll address that in the future… works fine for now.
    Case cleaned out - posts to be moved around from SSI-EEB
    All stuffed in and ready to boot… going to set it up entirely from IPMI as I don’t have a VGA or second monitor (or cable of any form that works) handy to connect it up. So, over the network it is! Ghetto fan for peripherals - I don’t trust there is enough flow in this relatively quiet cheap case to support 40G, 10G, optane and 16 port SAS/SATA HBA/Raid controller heat removal. So, I’ve always just put a 120mm fan in there to keep air moving.

Step 2: The Setup
Nice Asrock KVM/IPMI - web UI - no Java required for KVM console!

KVM Console optionally runs in browser window without java (java is still an option, but didn’t need it or want it). Fresh install of CentOS7.6 from USB thumb-drive using IPMI without a hitch…


Step 3: The Performance?
Copying over ~25T of files, so it will be a bit before I can run any proper perf tests…

I can share some sanity checks I did for now using dd - very crude TFIW - zfs is VERY hard to benchmark reliably:

Write (per block size):
4K ~500MB/s sustained over a 16G file
8K ~650MB/s sustained over a 16G file
64K ~1.1GB/s sustained over a 32G file

64K ~2.5GB/s

So, this showed me the L2ARC on a 280G optane drive doing its thing - those are roughly what I’d expect to see reading/writing to the optane drive directly.

I could/may add a second to improve that to match 40G network capabilities in the future, but being able to read/write to a compressed, raid6 (raidz2) filesystem of 50T in useable uncompressed space at those speeds is justfinethanks.

1 Like

That’s absolutely beautiful. I know beauty isn’t everything, but man am I a sucker for a sexy UI.

1 Like

Is Rome supposed to be a drop in replacement?

1 Like

Not sure… but for this particular server, it wouldn’t matter much to me. I’d use rome to get cheaper naples.

1 Like

isnt milian supposed to be here before 2020?


1 Like

Well, the last I heard was “late 2020 or early 2021”… but YMMV

My goal with this server is as many cores at as high a frequency as I can get without:
a. overheating my closet
b. spending more than ~$800 for a CPU at any given time

More cores is nice as I can run more VMs… but with 16 cores now, its not struggling… So, neither rome nor milan are on my horizon at this point for this node.

Well, except that a large copy + VMs and my not having reserved ram for VM is showing a flaw with ZFS… dammit… swapped all the hell despite plenty of memory. hmmm…


Do you not have the arc Max set to a hard limit?

1 Like

I didn’t - goofed… it is now… had to reboot and kill my rsync’s. Limited it to ~50G in arc to match the 50T in free, uncompressed space (80T of spinning disk after radiz2 and “what’s a gigabyte” computation method tax).

That leaves 14G for VMs/OS right now… Will adjust more later, but that should be fine for now…

1 Like

How are you calculating disk space --> ram usage?

1 Like

Not sure what you mean?

How am I computing how much ram I need? just targeting ~1000:1 (TB:GB), though - its under that:
du -h

volume1                    56T  3.5T   53T   7%

Right now I’ve set max arc to 50G…

Thinking I might try lowering that drastically given the optane L2arc and see what happens… but after I’ve finished rsyncing from the other array.


You said you have 64GB of ram installed?
last I knew… setting the arc max was set in Bytes… and that’s measured in used memory.
Correlating arc/mem usage to actual disk space… is my question.

1 Like

more /etc/modprobe.d/zfs.conf
options zfs zfs_arc_min=4294967296
options zfs zfs_arc_max=53687091200

cat /proc/spl/kstat/zfs/arcstats
c_min 4 4294967296
c_max 4 53687091200

I believe you are correct as well, that there are ratios used that can be tuned… I just used the hard-cap for now.


ah ok… 50GB

1 Like

Yep - 50*(1024^3)

1 Like

Did the swapping stop?

also… are you using the optane as cache for ZFS or as swap space?

1 Like

Not entirely… definitely slower, but…

KiB Mem : 65855732 total, 16681460 free,  6296164 used, 42878108 buff/cache
KiB Swap:  3002364 total,  2971892 free,    30472 used. 58259572 avail Mem 

2 partitions on optane - one is ZIL (relatively small) the others is L2ARC (200G+).

zpool status
  pool: volume1
 state: ONLINE
  scan: none requested

        NAME         STATE     READ WRITE CKSUM
        volume1      ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            sdk      ONLINE       0     0     0
            sdl      ONLINE       0     0     0
            sdm      ONLINE       0     0     0
            sdn      ONLINE       0     0     0
            sdo      ONLINE       0     0     0
            sdp      ONLINE       0     0     0
            sdq      ONLINE       0     0     0
            sdr      ONLINE       0     0     0
            sds      ONLINE       0     0     0
            sdt      ONLINE       0     0     0
          nvme0n1p1  ONLINE       0     0     0
          nvme0n1p2  ONLINE       0     0     0

with all that free space in RAM now… you can do a

swapoff -a && swapon -a

to get the rest cleaned up.
FYI… if you have a third partition on the optane… (can be few gigs ) using optane for your swap space… (in promox for me) is a HUGE improvement when something actually ends up using swap.