Closet Server Consolidation

You asked, so here it is… this is a work in progress, function over form for now…

Fitting old things/life into new location - so making adjustments incrementally - had 2 very loud rack-mount servers move from a nice cool/isolated basement to a hot and the middle of living space closet. So, now I have some serious heat/noise issues.

Step 1: “the build”

  • swap out 2x2690v4 MB from disk array case
  • swap in asrock 10G epyc board EPYCD8-2T
  • 7351p (16core 2.5-2.9GHz) + 64G of ECC 2966 ram
  • 10x8T disks
  • optane 280G cache drive
  • some more nvme later…
  • 10G and 40G networking (intel/mellanox)
    This is where we start - the existing server in this case
    image
    Yank out the guts that will move over
    image
    This board/setup will move to a compute-only node and no longer handle the disk array
    image
    The pile of new goodies…
    image
    This board is very thick and stiff as a board - significantly thicker than prior AsRock rack boards I’ve had.
    image
    CPU in - oops - tray CPU doesn’t come with torque wrench… approximate torque engaged…
    image
    Fan orientation less than ideal - we’ll address that in the future… works fine for now.
    image
    Case cleaned out - posts to be moved around from SSI-EEB
    image
    All stuffed in and ready to boot… going to set it up entirely from IPMI as I don’t have a VGA or second monitor (or cable of any form that works) handy to connect it up. So, over the network it is! Ghetto fan for peripherals - I don’t trust there is enough flow in this relatively quiet cheap case to support 40G, 10G, optane and 16 port SAS/SATA HBA/Raid controller heat removal. So, I’ve always just put a 120mm fan in there to keep air moving.
    image
7 Likes

Step 2: The Setup
[placeholder]
Nice Asrock KVM/IPMI - web UI - no Java required for KVM console!
asrock_kvm_login
asrock_kvm_sensors

KVM Console optionally runs in browser window without java (java is still an option, but didn’t need it or want it). Fresh install of CentOS7.6 from USB thumb-drive using IPMI without a hitch…
asrock_kvm_conole

3 Likes

Step 3: The Performance?
[placeholder]
Copying over ~25T of files, so it will be a bit before I can run any proper perf tests…

I can share some sanity checks I did for now using dd - very crude TFIW - zfs is VERY hard to benchmark reliably:

Write (per block size):
4K ~500MB/s sustained over a 16G file
8K ~650MB/s sustained over a 16G file
64K ~1.1GB/s sustained over a 32G file

Read:
64K ~2.5GB/s

So, this showed me the L2ARC on a 280G optane drive doing its thing - those are roughly what I’d expect to see reading/writing to the optane drive directly.

I could/may add a second to improve that to match 40G network capabilities in the future, but being able to read/write to a compressed, raid6 (raidz2) filesystem of 50T in useable uncompressed space at those speeds is justfinethanks.

1 Like

That’s absolutely beautiful. I know beauty isn’t everything, but man am I a sucker for a sexy UI.

1 Like

Is Rome supposed to be a drop in replacement?

1 Like

Not sure… but for this particular server, it wouldn’t matter much to me. I’d use rome to get cheaper naples.

1 Like

isnt milian supposed to be here before 2020?

image

1 Like

Well, the last I heard was “late 2020 or early 2021”… but YMMV

My goal with this server is as many cores at as high a frequency as I can get without:
a. overheating my closet
b. spending more than ~$800 for a CPU at any given time

More cores is nice as I can run more VMs… but with 16 cores now, its not struggling… So, neither rome nor milan are on my horizon at this point for this node.

Well, except that a large copy + VMs and my not having reserved ram for VM is showing a flaw with ZFS… dammit… swapped all the hell despite plenty of memory. hmmm…

2 Likes

Do you not have the arc Max set to a hard limit?

1 Like

I didn’t - goofed… it is now… had to reboot and kill my rsync’s. Limited it to ~50G in arc to match the 50T in free, uncompressed space (80T of spinning disk after radiz2 and “what’s a gigabyte” computation method tax).

That leaves 14G for VMs/OS right now… Will adjust more later, but that should be fine for now…

1 Like

How are you calculating disk space --> ram usage?

1 Like

Not sure what you mean?

How am I computing how much ram I need? just targeting ~1000:1 (TB:GB), though - its under that:
du -h

volume1                    56T  3.5T   53T   7%

Right now I’ve set max arc to 50G…

Thinking I might try lowering that drastically given the optane L2arc and see what happens… but after I’ve finished rsyncing from the other array.

2 Likes

You said you have 64GB of ram installed?
last I knew… setting the arc max was set in Bytes… and that’s measured in used memory.
Correlating arc/mem usage to actual disk space… is my question.

1 Like

Setting:
more /etc/modprobe.d/zfs.conf
options zfs zfs_arc_min=4294967296
options zfs zfs_arc_max=53687091200

Result:
cat /proc/spl/kstat/zfs/arcstats
c_min 4 4294967296
c_max 4 53687091200

I believe you are correct as well, that there are ratios used that can be tuned… I just used the hard-cap for now.

2 Likes

ah ok… 50GB

1 Like

Yep - 50*(1024^3)

1 Like

Did the swapping stop?


also… are you using the optane as cache for ZFS or as swap space?

1 Like

Not entirely… definitely slower, but…

KiB Mem : 65855732 total, 16681460 free,  6296164 used, 42878108 buff/cache
KiB Swap:  3002364 total,  2971892 free,    30472 used. 58259572 avail Mem 
2 Likes

2 partitions on optane - one is ZIL (relatively small) the others is L2ARC (200G+).

zpool status
  pool: volume1
 state: ONLINE
  scan: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        volume1      ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            sdk      ONLINE       0     0     0
            sdl      ONLINE       0     0     0
            sdm      ONLINE       0     0     0
            sdn      ONLINE       0     0     0
            sdo      ONLINE       0     0     0
            sdp      ONLINE       0     0     0
            sdq      ONLINE       0     0     0
            sdr      ONLINE       0     0     0
            sds      ONLINE       0     0     0
            sdt      ONLINE       0     0     0
        logs
          nvme0n1p1  ONLINE       0     0     0
        cache
          nvme0n1p2  ONLINE       0     0     0
2 Likes

with all that free space in RAM now… you can do a

swapoff -a && swapon -a

to get the rest cleaned up.
FYI… if you have a third partition on the optane… (can be few gigs ) using optane for your swap space… (in promox for me) is a HUGE improvement when something actually ends up using swap.

2 Likes