IT Security is hard. You have to keep hacking away.

World’s Smallest VMware ESXi Server

PICT1246

World’s Smallest VMware ESXi Server

I am not the first person to build a super-small ESXi server, and I won’t be the last.  But for a few moments in history, this might be as small as you can get for an ESXi server with latest Xeon processor, 16GB of ECC memory, 100% SSD, and 2 server-class Ethernet controllers.  Easily able to run 16 active VM’s.

So far (8 weeks now) it works perfectly reliably.

Two servers in my carry-on…

I started this project because I organize cyber competitions and may need to travel by plane.  Pre-shipping the servers takes time, money, and foresight… and I’m too lazy and broke to do all that.  Plus, baggage-handlers make me nervous, so checking my servers in my regular luggage seems like a bad idea.  So the solution for me is to fit two servers in my carry-on bag.  Must have room for protective padding as well.  Plus the smallest toothbrush I can find.

My national airline’s limit for a carry-on is:  23 cm x 40 cm x 55 cm (9 in x 15.5 in x 21.5 in) and 10 kg (22 lb).  (You also get a “personal item”, which is my laptop bag, but could fit a 3rd ESXi server in there also, if I had to.  This one fits in my laptop bag’s front pocket.)

The Specs…

After way, way too many hours research, here’s the exact config I chose for this home build.  Works for me, running VMware ESXi 5.1 free edition.  (Shout-outs below to the many sites that helped make this possible.)

MotherBoard Intel S1200KPR $155 Mini-ITX Board, for Xeon-v2, ECC RAM, 2xGbE
CPU Xeon E3-1245v2 $280 Intel Xeon 4-core/8-threads plus graphics
Memory 16GB DDR3 1600 ECC $155  2 x 8GB, Unbuffered Kingston KVR16E11/8
SSD 256GB OCZ Vertex 4 $135 256GB Solid State Drive
CPU Cooling Dynatron K199 $32 Dynatron K199 80mm 2 Ball CPU Cooler (Blower)
Case w/power Mini-Box M350 $102 M350 Case + Pico PSU 150W + power adapter
40mm Fans x 2 $8 Case Fans 40mm (1 is a spare)
$2 Mounting bracket for fan.
TOTAL: $869 Plus Tax and (sometimes) Shipping
Everything arrives. Look at all that  packaging!

Everything arrives. Look at all that packaging!

Why this motherboard?

S1200KPR is a true Intel server-class mother board.  I wanted a true server for the ECC RAM (stable), Xeon processor (excellent multitasking) and, especially, server-class Ethernet controllers.

It’s a mini-ITX, so it’s smaller than the whole ATX family – even the Micro-ATX.  The only drawback I hit with this form-factor is 16GB max for the RAM.  I wish it allowed more, but *small* was my goal.

The KPR version of this board supports the Ivy Bridge technology and v2 Xeon processors.  (Do not settle for the “S1200KP”.)  It has plenty of SATA and USB ports: SATA up to 6Gbps, though USB only up to 2.0.  (Currently, I don’t believe ESXi supports USB 3.0 anyway, but it would have been nice for future-proofing.)

The GbE controllers were crucial.  Last time I tried to add a NIC to my i7 box, the computer store dude said “This one is popular.  It costs $15”.  All I could respond was “But I wanted to pay a lot more than that.”  We agreed on $95 for an IntelPRO/1000GT with the 82572 chip to add to my i7 desktop.PICT1231

 

So for this server I wanted the Ethernet controllers to be at least that powerful.  Glad to see the S1200KPR comes with both an 82574L and an 82579LM controller already on-board.  (There is a PCIe slot on this motherboard, but no room for add-in cards in this case.  Remember… *small*.)

 

Who cares about server-class Ethernet?

I do.  Server-class controllers offload all sorts of work from the CPU – even TCP/UDP checksums.  And they support VLAN tags.  I know desktop NIC’s don’t bother with VLAN tags because tagging stops at the switch.  But in my VMware setup, I need to pass the tags to the vSwitches inside the server, so I really did need server-class Ethernet.

VMware 5.1 does not have a driver for the 82579.  To add the driver to your ESXi installation package you’ll find great instructions here. Yes, you will get the error message.  Yes, I ignored it (as recommended).  I’ve had no problems.  And no speed degredation on either Ethernet port (though I haven’t got a rigorous test for that.)

Just the parts.  Beer can is only to add perspective to the photo.  Honest.

Just the parts. Beer can is only to add perspective to the photo. Honest.

 Why those other components?

This Xeon E3-1245-v2 is the latest generation Xeon.  3.4GHz, 4 core, 8 threads.  LGA 1155 socket.  Ivy Bridge.  This CPU (and motherboard) support the fastest DDR3 1333/1600 ECC memory (unbuffered).  They’re not as fast as the E5 Xeon’s, but much less expensive, and draws less power.  Power generates heat, and heat is one of the problems you need to deal with in a *small* case.

They also have VT-x and Hyperthreading, of course.  [Edit:  Although this Xeon and the mobo chipset both support vt-d, the mobo BIOS does not.  I had this confirmed by Intel, unfortunately.  They said “…for the S1200KP and KPR. No luck for VT-d on these boards.”]

The 1225/45/75 Xeon’s have on-board graphics, which we absolutely need in this build, since we have no room for an add-in graphics card.  I chose th1245v2 chip as a trade-off between price of lower models and performance of the higher ones.  Also considered the 1265L-v2 because the “L” means low power.  Could have dropped from 77W down to 45W.  Might mean a smaller PSU – which would be nice.  But the 1265L uses less power because it runs at only 2.5GHz.  (I love saying that “only 2.5Ghz…”)

Snowy day outside.

Kids building a snow fort across the street today. Must prepare my battle plan for later. But right now, back to work….

I researched pretty hard to find the fastest ECC memory possible, to get the most out of this server.  Glad to get some good advice (CMTL Labs) and a good price (DirectCanada) on some 1600MHz DDR3 which was sure to work with this mobo.  I haven’t done any performance tests on it though, but I’d love to hear from anyone who’s tried.

The storage had to be SSD, in my mind.  Cost of SSD is way down.  And when I watch for performance bottlenecks in my apps, the disk always tops-out long before the CPU.  In my case 256GB is enough so that’s where I settled for the price/size tradeoff.

messy-build

In the middle of any build, things can get a little ugly.

The CPU cooler was crucial.  Have I mentioned “small”?  The case I chose has severe height restrictions, so the standard CPU coolers just wouldn’t fit.  I saw some low-rise coolers that might have worked but. to be sure, I chose a blower.  (Blows hot air out the side of the case.)  This is noisier than a regular cooler, so all you Home Theater buffs should stay away.  In my home office, I moved the server away from my desk.  In a classroom, you can’t even hear it.

Only complication during assembly was the pressure-bars that came with this CPU cooler.  They can not be used with this mobo – because they would crush some IC’s.  So I trimmed down the bars (with a jigsaw) so I could use only the nut at each end.  Then it was easy to screw the CPU cooler in place, through the motherboard, as usual.

 

That case is really small.

So small the entire server fits into the motherboard's anti-static bag.

This is the entire server. So small that it fits into the motherboard’s anti-static bag.

 

 

The guys at mini-box.com.au have focused on doing one thing really well, and it was exactly what I needed.  (I have absolutely no ties to them.)  I especially liked that they can bundle the power supply because I’m not an expert on that component.  In addition to getting the size right, I also got help from Zap the Super Moderator at AnandTech forum.  Apt moniker.

 

 

For assembly, I have to give a huge shout-out to Mini-Box guys for their video on how to assemble a server using the M-350 case.  The video features a different mobo, but my assembly went perfectly well following the same steps.  You can start their most-excellent video here.

Ready to be flung into recycling.

Ready to be flung into recycling bin.

 

 

As an homage to their great work, I present photographic evidence of all my nasty boxes gagged and bound – and slightly abused – before being flung dramatically into the recycling bin.

 

Other small cases…

I was inspired to try for an ultra-small build by these photos from Lime Technologies.  Their case is only 3.8L, and looked great.  But for my build, I went for the M-350, which is only 2.5L.

Note that for an insanely small ESXi box, though not a true server, the guys at Virtually Ghetto are trying to get ESXi 5.1 running on a Mac Mini.  Not clear to me if it’s working smoothly yet, but for a desktop box, the Mac Mini has some awesome features!

I ruled it out as a server though, since it doesn’t have the dual server-class Ethernet ports I need, nor the ECC RAM and Xeon processors I prefer.  Also, they chose a 2.6GHz instead of 3.4GHz CPU.  And you pay for the name:  currently $1,600 for a Mac-Mini config that approaches this home build (16GB memory and 256SSD) but ultimately lacks the features mentioned.

[Edit:  Excellent blog series describing VMware on an Intel NUC is here.  If you don’t mind using a single NIC and a few speed limitations, those things are really, really small!]

PICT1228

This server is so small, I could eat it for a snack:

Is it powerful enough?

So far, I’ve got 30 VM’s copied onto the disk.  16 of them are running during my events.  (70% Linux, 30% Windows).  During my events, as many as 6 or 8 are being exercised really hard at any given time.  I watch the performance graphs in VMware, and I’m nowhere near any ceilings yet.  Am thinking of shooting for 24 active VM’s, and 12 participants blasting simultaneous Nmap scans at them.  If anyone tries this, I’d be interested to hear your feedback.  [Edit:  Ran 24 VM’s at last event.  No problems.  Again though, only 6 to 8 really active at any one time.]

Caveats…

Not everyone needs an ultra-small  ESXi server.  I made some trade-offs while building this bundle.  The 16GB memory is a severe limitation if you want a lot of VM’s.  And there is no way (with the M-350 case) to add a card to the PCIe slot on the motherboard.  Plus, the blower is much louder than a normal CPU fan (though much quieter than any full-size server).  And there is no place for an optical drive.

Oh yes, just one glitch:  the BIOS won’t find the hard drive during boot-up search, so I have to hit [F10] to have it display the “Boot Device Menu”.  Only the SSD shows up on the list, so I don’t know why it doesn’t get selected automatically.  (Still.  I can boot the server – even with no monitor – by hitting [F10] then [Enter] within 15 seconds of boot-up.  Never misses.)

But I really needed an ultra-small ESXi server, so I am satisfied with these trade-offs, and very pleased with the result.  Now I just need to find a really small toothbrush.

42 Comments
  1. Anyone else have experience with an ultra-small ESXi server?

  2. You could just buy a Mac Mini. You can get 10GBe Thunderbolt NICs, not sure if they’d be compatible with ESXi though.

    • Hi Sam,
      I considered the Mac Mini for a while. Apple gear is excellent quality, and their device is super-small. Plus, as I mentioned in the “Other Cases” section above, a bunch of people are running VMware on the Minis for sure.
      But the drawbacks started with the price – almost double, when I bought mine.
      And then their were the questions I couldn’t answer: would it support VLAN’s? And your own question about Thunderbolt. Even if that i/f was were supported by VMware, I couldn’t afford to swap-out all my switches and routers, so I’d probably get an adapter and drop the Thunderbolt down to GigE anyway.
      So I figured that Mac Mini, unless you have spare cash and no way to build your own alternative, just isn’t as good a solution.

      • For future reference, yes the minis support VLANs, tagged VLAN trunks, etc. The 1Gb thunderbolt NIC also works swell with a simple config file tweak as well.

  3. Hi,
    Nice solution for a “small” ESXi.

    Are you absolutely sure that S1200KPR ( not S1200KP ) supports VT-d ?
    I have found a lot of messages in Intel forums suggesting that it doesn’t.

    I noticed that there is a new BIOS from june 2013.
    Maybe it has something to do with that?

    Best regards
    Niels

    smed60 -at- hot mail -dot- com

    • @Niels. Thanks for pointing this important mistake out. I made a bad leap in logic, and I was wrong about the vt-d. Updated article above.

  4. hi,
    anothet option can be Intel NUC which can be installed with so-dimm memory. esxi can run from usb stick. 4″ x 4″ is the total foot print.

  5. Lovely work. Really like the server. Great job! A few of these would stack nicely. I think 16GB of RAM is plenty when you only have the 256GB of storage.

  6. I really want to get to 32Gb of rams… damn.

  7. How noisy is this sucker?

    • The sound ratings for the K199 “blower” are:
      At Duty Cycle 0-20%: 20.71 dBA
      At Duty Cycle 50%: 37.5 dBA
      At Duty Cycle 100%: 51.7 dBA
      and I run it at 70%, so it’s over 40dBA. That’s not quiet.

      By contrast, Dynatron’s entry-level, standard cpu cooling fan (K5) goes from 15 up to 30 dBA. (All mfg ratings.)

      Reasons to get it: very low profile (15mm vs. 25mm for the K5… not including the heat sink), and it blows the heat out the *side* of the case, which is very useful in this particular form factor.

      I put it at farthest edge of my desk, and I stopped noticing the noise after the first day. It’s loud for a desktop, but *nothing* like a rack-mount server. I knew if it bugged me I could move it to the floor under the desk, where it’d be about as noticeable as an old-style PC on your desktop. But I didn’t need to do that.

  8. Would there be any benefit into going with Xeon E3-1245-v3 that is based on Haswell arch?

  9. I threw these 2 CPU’s into the comparison tool @ cpu-world.com. For the new chip they say…
    Pros: Supports AVX2 instructions
    Drawbacks: Insignificantly higher power consumption, Slightly more expensive. (Note that is *in*significant.)
    Apparently AVX2 is not supported yet in many apps, but may be in the future. I can not find any declaration by VMware regarding AVX2. My quick, amateur review of the new AVX2 instructions sees some interesting possibilities, but nothing “wow”. This is definitely not my specialty, however.

  10. Also, are missing the lack of VT-D support?

    • Arr, spelling mistake, my bad. I meant whether you miss somehow the VT-D support? Did you stumble upon a use case where you’d really need it in your ESXi setup?
      Thanks a lot for answering my previous questions!
      Cheers!

      • From what I understand of VT-D, it can improve performance by connecting VM’s directly (pass-thru mode) to NICs or storage devices. So I guess it would make my rig run faster, in some cases. But is there a use case where I would “really need it”? I guess I can say no.

    • I know this is quite some time after you posted, so you may well have already made your purchase. If not, consider the ASRock E3C226D2I I bought. It supports VT-D even with a Core i5 (Haswell). The picoPSU 150W works fine with this mobo and an SSD for me as well in this setup. Another nice thing is it supports ECC (although probably not with an i5, anyone care to confirm?) and non-ECC RAM.

  11. One more Q. Could you share with us what power brick did you get for your build? I presume some that can deliver 150W on 12V.

    Also, is 150W enough? I did some quick calculations and it appears that:
    – Intel S1200KPR: 95W
    – Xeon E3-1245v2: 77W
    – RAM KVR16E11/8: 3W*2 = 6W
    – Samsung SSD = 0,069 W

    That amounts to 178,069 Watts + the blower. On the other hand that’s ~28W on top of the available 150W. I guess you’d draw that 180W on 100% utilization, right?

  12. Your 95W for the motherboard is way too high. A good thread is here:
    http://semiaccurate.com/forums/showthread.php?p=172146
    They estimate 10-15W for the MoBo, which I believe is much more accurate. I have no measurements of my own system though.

    I ordered my power brick bundled together with my M350 enclosure. See: http://www.mini-box.com/M350-Enclosure-WITH-PICOPSU-150-XT-and-150W-Adapter-KIT
    Cost: $102.00
    Part No: ENC-M350D-PWR

  13. Oh, another one! Those Linux / Windows VMs are 32bit or 64bit?

  14. Do you think this setup would work with either booting ESXi off of flash mem or USB stick? I will be using NFS datastore so trying to keep the cost down and no drives would keep the size and weight down as well.

    • I have not tried it, but I do like your thinking. I have read many articles by people who boot ESXi off USB sticks, so it really should be no problem. I can say there is indeed an internal USB connector, so you wouldn’t have the USB “sticking” out and getting knocked-off.

      • Thanks for the reply. That MB looks to be discontinued and a little hard to find. Any other mini-ITX boards you recommend?

        • Sorry Caveman. I am only familiar with this one model of mini-ITX.

        • I was wondering the same thing and came across the ASRock E3C226D2I. I have one on order and I’ll try and remember to report back after I set the system up. Only unfortunate things I anticipate are that it has a full PCIe slot instead of a mini-PCIe, and the ethernet cards require a (VMWare supported) different driver to be downloaded from their website for ESXi 5.5.

          • I hope you can let us know. I may need to build another server in next few months.

          • So far it’s working very well. The VIB install for the Intel i210 ethernet chipset worked exactly as it should have. The box is still sitting open because I have to decide on a few things like the heatsink, SSD placement, and some wire length issues, so I won’t list details here yet. Just a few quick things: ESXi 5.5 works well, hardware passthrough is supported, and everything works as it should as far as I can tell with just a few hours of testing.

          • Sorry, a few things I left out. I’m using a Core i5-4570S, and yes hardware passthrough works. Also, since this uses the server-line Intel C226 chipset PCH, things like SATA work as expected out of the box with ESXi 5.5.

  15. Hi,

    Im considering copying your design. Can you tell me if theres room to upgrade: Dynatron K199 > Noctua Nh-L9I?

    • Lomik, I am not familiar with your alternative, so all I could do was open my case and measure what I have.
      I slid a stack of sticky-notes in between top of blower and bottom of SSD, and this rough method seems to show just over 3mm clearance in there. Hopefully, you can find the published specs for the two coolers, and do the math. Be careful though: some coolers have wires coming out of the top which would use-up about 1mm of that gap (if they protrude directly under the SSD).

  16. is this compatible with esxi 5.5?

  17. Sorry. I have not tried 5.5 on this device yet.

    I would say “should be compatible”, based on previous experience of backward compatibility for ESXi releases. But I know if you’re building a new box, it’s nice to hear someone say “I tried it, and it works”.

    Can anyone else answer this for zooo?

  18. HAPiRat,
    were you able to run stressed test?

    the PicoPSU and little air flow makes me a bit nervous.

  19. I have used the server without problem for testing in my lab, and stress-testing during several computer training sessions. No problems.
    In the lab, I monitor CPU and disk activity while doing a lot of VM moving and changing. The CPU has never maxed-out, even during heavy disk operations.
    During the training, there would be more CPU-intensive work, and the memory is definitely over-subscribed by my 20+ VM’s. However, I am too busy then to do performance monitoring. Your mileage would be different than mine anyway.
    For the PSU, I believe 150W is actually more than I need. For my next server, I want to try 120W. I am pushing the lower limits because the 150W brick is the largest I have ever seen. It’s heavy.
    For the blower, I adjusted the fan speed up to a constant 70%, and it keeps the temperature of the CPU acceptable (<70deg). So there is still some headroom if you are nervous and want to adjust the speed even higher. Your ceiling will be 100% fan speed, or personal tolerance for a noisy box.

  20. interesting, I speced out this same board and cpu awhile ago for similar reasons. Curious tho, is there an equivalent that would allow a max of 32GB RAM?

  21. @MO
    Sorry, no. From what I understand, no mini-ITX board can go beyond 16GB of memory. I don’t think the standard says “no” but rather it defines other circuitry that precludes the possibility.

    To get more than 16GB you’ll have to go to a larger micro-ATX board. For now.

    No doubt we’ll see more RAM on smaller boards at some time, but I have no idea if it’s weeks, months, or years away.

  22. FYI: VMware ESXi 5.5.0 build-1331820 works just fine.

  23. great article!

    i would like to build a server similar to yours.

    only problem. seems that the Intel S1200KPR is no longer availeble :/

    Any advice on an alternative motherboard?

    • My little S1200 is humming along perfectly so I haven’t researched the latest boards since then. I’m definitely way out of date. If I get a new one, I’ll post here. And if you get a new one, please feel free. This post is still getting a fair number of hits, so people might be interested to know what you find.

      • Good to hear.

        I have almost decided to make a purchase. all from newegg.com
        Any thoughts on why I should not go for this setup?

        iStarUSA S21-20F2 Black Aluminum / Steel Tower Case Compact Stylish Mini-ITX Enclosure

        ASRock E3C226D2I Mini ITX Server Motherboard LGA 1150 Intel C226 DDR3 1600/1333 $60

        Intel Intel Xeon E3-1245V3 Haswell 3.4GHz LGA 1150 84W Server Processor BX80646E31245V3 $485

        Crucial 16GB (2 x 8GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) ECC Unbuffered Server Memory Model CT2KIT102472BD160B $170

        2 x WD Red WD10JFCX 1TB IntelliPower SATA 6.0Gb/s 2.5″ NAS Internal Hard Drive -Bulk $160 total

        SanDisk 16GB 16G Cruzer Ultra Fit SDCZ43-016G USB 3.0 130MB/s Nano Flash Pen Thumb Drive CZ43 $15

        total $885

        Items should be compatible, i have few concerns though..

        #1 will the Sandisk Cruzer USB3.0 connect to the Onboard Type A USB 2.0 interface?

        #2 are there any better/whiser picks for Motherboard, Processor and Memory? performance/price

        #3 how much noise would the standard CPU cooler generate during work.

        Have a nice weekend.

        Kind regards
        Jesper

Open Events

Open Events are open to anyone who wants to improve their IT Security skills, meet others, and have fun!  We get great feedback from sys admins, network admins, web programmers, and IT-Sec consultants and practitioners - of all skill levels.

Corporate/Departmental

Corporate/Department Events are for groups who want to have a fun and challenging team-building event.  There's no better, faster, or more effective way to build IT Security skills, than to put the team into a competitive environment.  We can do this on- or off-site.  Great pricing if we combine your team event with others.

Student Events

Student Events started at Algonquin College in June 2012.  Since then, we've done similar events at McGill University for Electrical, Computer and Software engineering students, Carleton U "Learn to Win" program, and La Cité Collégiale.  If you are interested in having an event at your school then let us know.

Groups & Associations

Group and Association Events can be an exciting and engaging addition to your calendar.  A hands-on, fun-filled and challenging activity that your members will really look forward to.  They'll improve their IT Security skills, and earn CPE and other professional credits.