Slashdot Mirror
Reasonable Hardware For Home VM Experimentation?
cayenne8 writes "I want to experiment at home with setting up multiple VMs and installing sofware such as Oracle's RAC. While I'm most interested at this time with trying things with Linux and Xen, I'd also like to experiment with things such as VMWare and other applications (Yes, even maybe a windows 'box' in a VM). My main question is, what to try to get for hardware? While I have some money to spend, I don't want to, or need to, be laying out serious bread on server room class hardware. Are there some used boxes, say on eBay to look for? Are there any good solutions for new consumer level hardware that would be strong enough from someone like Dell? I'd be interested in maybe getting some bare bones boxes from NewEgg or TigerDirect even. What kind of box(es) would I need? Would a quad core type processor in one box be enough? Are there cheap blade servers out there I could get and wire up? Is there a relatively cheap shared disk setup I could buy or put together? I'd like to have something big and strong enough to do at least a 3 node Oracle RAC for an example, running ASM, and OCFS."
Xeon based, easy access to multiple drive bays, dual gigabit ethernet, etc. Runs linux, Windows, Mac OS X
...something like this?
I ran into this same situation and found the best cost/performance setup was a Beowulf cluster of netbooks.
You get the cumulative power of those Atom processors and have a huge memory pool to run the VMs within.
Just check the BIOS to make sure that you can set the MB for virtualization.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
Is this a joke? Like seriously is it? No there isn't. Move along. This is the exact opposite of cobble old hardware into a pile.
I ran my first virtual machine on an athlon 2200+ with 768 megs of ram. If it can run windows 7, you can run a VM or 3 (Depending on how heavy you want to get). Essentially take your computer, subtract the cycles and ram required to run the OS and background programs, that's the hardware you have left over to run the os. If the guest OS was compatible with your original hardware, chances are it'll work just fine in the OS.
Is it sad that I am more likely to recognize you and your posts by your sig than your name or UID?
Dell currently have the Studio XPS (2.66Ghz Core i7, 3G RAM, 500G HDD) going for US$800 - for a basic home virtualisation server, it's hard to go past, especially if you spend another US$80 or so to bump the RAM up to 9GB. I can't imagine you could build it yourself for a whole lot less (depending on how you value your time, of course).
(Damn, sometimes I wish I lived in the US. Stuff is just so bloody cheap there.)
64-bit Linux host and as absolutely much memory as you can possibly install.
http://rocknerd.co.uk
I personally use qemu-kvm and im quite happy with it. Thats running on a dual core machine with 2G of ram (probably not enough ram though!).
For the KVM stuff you need have chips which support Intels VT or AMDS AMD-V so your processor is the most important aspect. A quad core would probably be suitable too if you can buy that.
For just experimentation usage its a fantastic alternative to VMWare (I personally got sick of having to recompile the module every time my Kernel got updated).
On my box myself i've had about 6 CentOS VMs running at once but frankly there were not doing much most of the time. Ultimately its going to boil down to how much load you inflict on VMS underneath, my experience with it has not been very load heavy so I could probably stretch to 9vms on my hardware which is probably on the lower end of the consumer range these days.
The most important bits are your CPU and RAM. If your after something low spec you can do dual core 2g ram but you could easily beef that up to quad core 8G RAM to give you something you can throw more at.
Oh and Qemu without KVM is painstakingly slow - I wouldn't suggest it at all.
I currently run VMware Workstation with an Intel Q6600. VMware has a setting to choose to use one or 2 of the cores. Generally, for Linux VMs, one core is enough (unless you decide on GUI). If one goes for Windows Vista/7, 2 is better for performance, but one works okay for XP.
Ram is dirt cheap right now on Newegg as well. I have 8gb of Corsair ddr2 ram I got for 50 dollars after rebates. Non GUI, you can get by with 384-512mb of ram, but otherwise, id go with at least 1024 or more.
The nicer part of VMware Workstation is it now supports Directx 9.0c (but with only shader level 2, still working on 3). Expect a 10 or so perecent in performance droppage though for gaming depending on how many resources you allocate.
Your needs look a bit bigger than mine (mostly trashing VMs and running test software before doing something crazy to the actual box). A bigger CPU such as a Xenon might be more to your liking, since you can have 2 of them for a total of 8 cores (leading to lots of VMs).
My VM server rig is decidedly low-end compared to many I've seen, but it certainly gets the job done. I custom built the box, mostly from components bought on NewEgg; it has a dual-core AMD64 chip (soon to be upgraded to a quad-core), 3 GB RAM, and about 500 GB total drive space between two IDE (yeah, I know, will upgrade to SATA at some point) drives.
The machine runs Ubuntu Server with VMWare Server 2. I can easily run several Debian and Ubuntu VPS nodes on it under light load, and I use it for experimentation with virutal LANs and dedicated purpose VMs. I periodically power up a Windows Server 2003 VM, which uses a lot more resources, but it's still fine for testing purposes.
512 MB RAM, 20 GB disk, 200 GB transfer, five datacenters. $19.95/month.
Is there a relatively cheap shared disk setup I could buy or put together? I'd like to have something big and strong enough to do at least a 3 node Oracle RAC for an example, running ASM, and OCFS."
Er, if you're running VMs, you inherently have "cheap shared disk" - the disk in the host that any of the VMs can access. :)
You can find lots of used servers on eBay that you can mess around with. Sun's v20z servers are pretty cheap and have a decent amount of power.
A lot of the stuff I've run across is rack mounted and keep in mind that rack mounted servers are loud in most cases. So it may not be the best thing to play around with in your home or office.
You don't really need any special CPU to mess around with virtualization, you won't get "full" virtualization but I don't think that will stop you. For more info check out, this page.
I'm currently running a number vm's in my desktop using Sun'x VirtualBox (xVM) whatever they're calling it now. Even within some of the solaris VM's I'm running solaris containers so I'm doing virtualization upon virtualization and my processor doesn't have Virtualization technology support.
If you want to do full virtualization look for server class CPUs. Xeons and Opterons. Using Newegg's power search there is an option to filter by CPU's that support virtualization technology.
If you're primary focus is Oracle RAC, you may want to look at Oracle VM which is Xen based.
Dual Opteron < $600
You can run virtual instances on practically anything. I use VMWare Workstation on an older AMD Athlon 3200+ (the machine on which I'm typing this) and get acceptable performance if I only have one instance booted at a time. You're not going to be playing video-intensive games on the instance, but it'll work find
I maintain a few websites (my blog, a gallery, couple other things) on an old server class machine in the garage. Companies often scrap servers after the 3 year warranty expires, or they've finished depreciating (depending on individual business rules) and they're often fast enough to make reasonable virtual servers. Often you can pick them up at a scrap sale or surplus store, or, if your company has an IT department, get permission to snag a machine that's about to be scrapped.
I recently brought up VMWare's free bare-metal hypervisor ESXi and was surprised at how easy it was to set up and create instances. VMWare has a free Physical-to-Virtual converter you might want to experiment with. It works great with Windows, but is kinda hit-and-miss with Linux.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
Any of the chips with x86 hardware virtualisation should do. >=2Gb ram would also be good
Of course faster CPU and as much RAM as you can get are good things but you don't need high-end stuff for VMware. I was running VMware on a 450 Mhz Celeron with 512 MB RAM back 10 years ago and it worked fine (Linux host and Windows 2000 guest).
ESX whiteboxing information can be found in a number of places but you might want to start here: http://www.ntpro.nl/blog/archives/338-The-complete-white-box-list.html http://ultimatewhitebox.com/
http://virtualize.wordpress.com/
You can do it "well" on a dual core with 4GB of ram. Even less, but with todays prices you can get a system for a couple hundred if you watch for sales. RAM is you biggest killer that you will notice. Then again, with quad cores with VM assistance going for under $200CDN, thats relatively cheap. If you're worried about HD performance a couple 500GB drives striped will give you over 100MB/s of read speed a relatively small investment.
---- aut viam inveniam aut faciam
I get along quite happily running 5 or 6 VMs on a Dell Vostro slimline desktop, Core2Quad, 8GB ram and a 10k RPM disk that cost me no more than £400 6 months ago, and thats using Microsofts HyperV server (free download, runs as the hypervisor itself so theres no Windows Server instance underneath it - don't mistake 'HyperV Server' for 'HyperV for Windows Server 2008', they are not the same :) ).
I got a dual-quad-opteron for my messing about, making sure to get one cpus new enough to have an IOMMU for faster I/O virt.
kvm/qemu is the sweetest virt solution nowadays imo - vms scheduled by linux itself, rather than some other wonky kernel.
we have about 4 machines with 2 quadcore running ESX and about 100 machines (many linux and windows) and 64GB of ram in each esx node... and we have still about 50% of resources free
so grab one quad core machine, with lots of ram (for oracle RAC+ASM+DB you will need at least about 4GB for the 3 RAC nodes, the more the better)
as this is for testing, i would but a plain quadcore PC, with 6 to 8 Gb of ram, install a linux 64bit with xen or vmware esxi
if you have more money, you can buy more ram or even cpu, but you dont really need a blade nor a server, a plain PC will do
ohh i forgot... HD, buy at least 2 HD, to spread the IO load, if you want raid, then you need 4 HD for a raid10... you can also try iscsi with a openfiler based nas/san (another PC, with lot of HDs and several gigabit network card)... of course, the server also need several gigabit network card to increase the IO bandwidth of iscsi
have fun
Higuita
I strongly advise you to do your homework before spending money on non-server class hardware (or before selecting a server for that matter). VMWare runs on a lot of hardware, but it also fails badly on a lot of consumer grade motherboards. There are some list (White Box Hardware Lists such as http://ultimatewhitebox.com/) you can check. After spending some time on name server HW and on White Box gear, I can tell you that the name server gear is a lot more compatible, easier to work with, and worth the money (if you have it). If you are doing casual stuff and don't mind the considerable pain you will have to go through to get patches and select disks systems and other components, consumer gear will let you play a bit. As for doing anything serious with more than one VM on a box - not likely. Xen is a commitment, as is VMWare or any other VM system. It is going to eat the box if you do anything other than dabble in it, and you are going to spend some real money if you intend to do much with VMWare (think $3K - $5K to get very serious). Running a VM is easy. Running multiple servers, backups, external disk systems, etc. is real work and costs real money for all the extra stuff you will need. If you stick to Linux you can save a bunch, but if you intend to do any real work with MS Servers, you are going to need several licenses, and iSCSI targets, back up tools, etc...You won't actually learn too much before you go to that level that you can't learn with VMWare Workstation (a great product but not anything like a production server environment). You can get you feet wet for nothing but time with most of these tools, but you can't get real, in depth experience with what it takes to run a production cluster, replication, remote storage, live replication, and all the rest of the things you need for real production unless you actually set up a production like system - that means real servers (White Box or name brand) and lots of hardware. You won't be able to see much with less than 8 cores, and 16GB plus some local RAID and iSCSI network targets. You can get started, but if you are going to spend money, I really think you should start to buy gear that is going to build towards a real server environment or you should stick to home systems and maybe run VMWare Workstation or some other stand alone VM just to play with it. VM Mode Linux (not very popular today) or some Xen sets for personal use would give you some understanding of VM concepts, but not a lot of basis for real production issues (at least they did not for me and I was a pretty heavy development user). Production VM deployments have a lot of issues that all take real in depth study, and lots of resources (iron) to get right. On the other hand, you can get a Supermicro, a Dell, or HP server with dual Xenon quad cores for less than $4000 with some nice disk. Get 4 or 5 containers under a VM and set up a replication to another server and a remote iSCSI disk and then you have enough to start to actually do real learning on. Of course the license fees will be way more than the hardware costs, if you are using MS tools and VMWare. ESXi is OK but unless you are going to go deep and do it all the hard way (hack the OS) you can't do a lot with the free version. With Xen, if all you want is to run a couple versions of Linux, just get a quad core box and have some fun...it doesn't really give you much production knowledge, but you will have some interesting test you can try. What I am really saying is - with only 4 cores you can do some useful things to support development,and you might make a nice personal server for you private web sites, but you don't have enough iron to experience the real issues of production VM management. If you are going past what a developer does (or a tester) and looking at a operations type environment you will need 8 to 16 cores on multiple boxes. This is a lot more than a home user typically wants to spend. IMO you also can't really expect to be really good on more than one system unless you do it day in a
Amazon EC2 is what I use for stuff like this, both windows and linux boxes everything available at a single push of a button. I
also use it alot for development, fire up a machine load and go.
Got Code?
Reading 'cayenne8', I can't help but imagine a V8 Porsche, and because I'm a car guy, for good or bad, this shifts the focus of my comment toward resources, specifically what is available, versus what is acceptable or tolerable.
Let's say you're a one-man Lab, incorporating all the SA, Developer, and Midware functions into your 'play' with this environment. How much time will each environment spend heavily plowing into loads?
If your intent is to deploy RAC in a multitude of scenarios, in short order, with a minimum of intervention, you may be able to get away with $1500 to $2500 worth of NewEgg parts (think high throughput - RAID, Max. RAM, Short access times, etc.) and the virtualization technology of your choice. Personally, I find VirtualBox capable of everything I need as far as virtualization and deployment goes, however, you need to be able to leverage 'fencing', with likely puts you into VMWare territory.
Fortunately, VMWare Server is 'free', and CentOS and OpenSuSE support some of the more advanced features of HA on Linux. Then again, if we're looking at resources as a major factor, then Redhat and Novell might be worth looking at, as they both offer 60 to 90-day evaluation licenses for their Enterprise Linux products, which may offer a prettier and more 'honest' (per the documentation and common expectations) implementation of their respective HA features than the freely-available, and in some cases, in-flux versions of the same software.
As far as RAC goes, take a look at the requirements for RAC, per Oracle's installation guidelines,, and size/spec from there. I believe you can get away with 16GB - total, if you have the capability to size the VM's memory access, or otherwise configure the amount of addressable memory, or put uo with or hack Oracle's RAC installation pre-flight. There is also valuable documentation available on your chosen OS vendor's sit, which may even be Oracle, who knows..
You may be hell-bent on performance, however, and you may be looking for the ultimate grasp of technological perfection, as it exists at Sun Mar 22 17:29:59 EDT 2009. In this case, you may want to look at Xen, which is available on Solaris as their 'xVM' technology, as well as on various Linuxes and BSDs.
On the other hand, you may be a Mac guy, with a decked-out Octo-core Xeon Mac Pro, where you have the option of Parallels and Virtual PC and something else, in addition to Sun's VirtualBox mentioned above.
Ultimately, things to keep in mind may be shared disk requirements, fencing options, and VM disk and memory access.
YMMV
We're running Xen heavily at my work, using mostly commodity type hardware. One point I will make off the bat, is that Xen can't run Windows on all types of Intel processors. There is a crippled version of their Celeron dual core processor that does not handle multiprocessing well enough to allow you to 'appear' native to the application. All Opteron processors from AMD have this bit available, so the Dell Poweredge T105 will work well for that, or any Opteron-based system. We bought a T105 so that we could have Windows on a virtual Xen instance, and so far it's been working well.
I've been noticing that you can get older P4 based, rack-mount systems pretty cheap on eBay, and the supporting parts to set up your 'lab' environment would not be too bad either.
You can do Oracle with just a single machine running multiple VMs; however, if you really want to get serious, you should consider building two physical machines. One each machine, create a virtual or two with 1-2G of RAM. for the shared disk, use DRBD volumes between the two.
My test RAC cluster has two AMD X2 64-bit systems with two gigabit NICs each. CompUSA has a similar machine for about $212 on sale this week. Newegg prices are similar. You'll need to add a couple extra Gig NIC and some more storage. Still should cost under $400 each.
On each physical system I used CentOS 5.2 with Xen. I created LVMs on the physical machines as the root volumes. Also carved out a separate volume to back the shared volume. Then I carved out a xen virtual machine on each with 1.5G each. I put the DRBD network on one pair of NICs. The other pair was used for the network and heartbeat (virtual ethernet devices).
Make sure your are buying a processor (and motherboard) that supports VT otherwise you may not be able to host other 64 bit systems.
VirtualBox is fairly good even on mediocre hardware. The more RAM and CPU the better, but you don't need a quad-core with 8 gigs of RAM just to run a virtualizer. Heck, you don't even need a dual core for that. Do make sure you have lots of RAM though (I have ~2 gigs, and ~2 gigs swap as well, though Linux never uses it anyway). YMMV, so don't use this info for anything mission-critical.
$ make available
The difference between 'server class' hardware and you beige box PC is that the more expensive 'server' is a lot more reliable and has extra remote access and hardware monitoring features. That's about it. If all you want is to run virtual machines in a test environment, just get a desktop with a hefty CPU and a whole whack of RAM and you're set. A good 'gaming' machine without the video card would be fine. You don't need to spend extra for a 'server'.
Since it doesn't sound like you're planning to actually run any production software on this machine, just about anything will do. Memory will probably be your biggest need, so at least two or three gigs might be in order. Disk space is cheap, and processor power probably won't matter too much for experimenting. As far as shared disk, try iscsi target mode, it's supported on most Linux distributions, and it works with most cluster software.
Look for the sale.
I got mine shipped to the door for $530 USD end of 2008.
I threw in 8 gigs ($100) and an ATI4850 ($150) and the damn thing is still cheap. Only the q6660 models will have the 2-rail power supply with the pci-e power connector. But you may not care for your needs.
Q6600 will VM 64-bit OS on a 32-bit host.
Really for getting started you just need memory. Everything else is just a convenience in terms of performance and won't really buy you more functionality.
I run XP as my host OS with just 2GB of physical RAM, and then do development in a 768MB Linux partition under that using VMWare Workstation. You can do the same thing for free with Xen or VMWare Player or Server.
When 2-4GB is not enough, then either upgrade your workstation to a 64-bit OS and throw in as much memory as you can fit/afford, or bring up another box.
You can get VMWare ESXi for free and do bare metal virtualization if you want to, or run the 64-bit Linux distro of your choice and instances of VMWare Server on top of that, again all for free.
Memory is the most important resource, then having more than one CPU core is nice, but for "home experimenting" you don't really need anything fancy.
If you have money, wait a couple weeks until the Nehalem server chips come out and get a dual-CPU server with 16+GB or ram. That will give you 8 screaming fast cores, ECC memory, and the ability to run as many virtual machines as you could possibly want.
G.
You say you want to go "cheap", that you don't want to spent too much money, yadda yadda... and then you go on to mention things like "cheap" shared disk and "cheap" blade servers?
What you realistically need and want are two different things.
I'd suggest a cheap quad-core AMD Phenom II system with 8G or so of RAM. Nothing too fancy. that I assume you're going to be running a Windows host OS, or VMWare ESX. More RAM will be needed for the Windows host OS, obviously.
Absolute lowest-end hardware you'd want to look at getting is an AMD Athlon 64 x2 or Intel Core (IIRC) based system. In other words, you want/need the VT support, or it'll be purely an emulated environment, and substantially slower than native (30%?), not just marginally (10%?).
I recommend AMD hardware because it's got a better price/performance point, and because unlike the other stuff in the "reasonable midprice" range for Intel, it's got the memory controller/north bridge integrated into the CPU (for newer gen stuff). I'd say go Phenom or Phenom II without any hesitation.
With a CPU like this, there's no reason you couldn't build a full system for around $450-500, sans storage. You could probably find a suitable "starter"/deal system for $300 from TigerDirect that'll do the job just fine with a little more RAM and another drive.
For disk, just go with an SATA RAID card (LSI are good) and 3 1Tb disks. That's about as cheap as you'll get and still have room to work.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
I won't get into the debate on "name brand" verses "whitebox" or do it yourself.
I am a teacher so money is an issue for me.
Things I have found that work for me in a lab (rather than production environment):
dual or quad processor (AMD seems to work and lower $)
processor / mother board support for hardware virtualization
Lots of ram ( I use x64 OS with 8-16G)
Choice of vm technology - Virtual PC/Server is find for MS based guest but
VMWare Server all the way for anything else
I don't get too wrapped up in RAID solutions, HD are cheap and DVD backup is even cheaper
If you need to backup/change out vm machines consider building an Openfiler NAS. It even
supports iSCSI if you have the funds.
First and foremost, vm's can be a fun learning tool.
GP
What I would do is look at what you need in hardware terms to run something like VMware ESXi. Firstly ESXi is at the best price point (it's free) whilst giving you most of the VI3 capabilities. The hardware that I run it on is pretty much whitebox hardware - Intel CPU (Q6600, 8GB RAM), Intel or Supermicro motherboard. The main requirement is that in order to install ESXi there needs to be a hardware controller that it can find and install to. Have a look at the VI3 HCL (http://www.vmware.com/resources/compatibility/search.php?action=base&deviceCategory=server) to find your hardware to ensure things will run. On one system I use an Adaptec 19160 to boot ESXi from, then use iSCSI to an OpenFiler machine where all the VM's are stored. The other machine is a Dell T3400 where the onboard SATA when configured as non raid / AHCI, ESXi can install directly to any of the attached SATA drives. Another solution is to use an Adaptec SATA RAID card for local storage.
Really.. it all boils down to your monthly utility fees and what you are willing to pay...
You can pick up 1-off servers being ditched by corporations (if you degauss the drives and certify that you will destroy them if you ever stop using them, you may get the drives as well) - otherwise, it will probably be sans hard drives, for next to nothing....
I picked up a test platform, Two Dual Core 2.66Ghz 64 Bit Xeons, 16GB RAM, 8 hot swap U320 72GB Drives, battery backed raid caching controller, dvd, floppy, 2 x 720 watt hot swap power supplies, in a nice deskside case, for zilch, nada, zip.. Just haul it away...
Currently running VMWare ESXi, with 16 VMs installed (I normally don't run more than 6 at a time, but it will run 12 without too much latency)...
Solaris x86 (64 bit and 32bit installs), Ubuntu, Nexenta (the v2 beta), OpenSolaris, Windows 7 Beta (32 and 64 bit) - blech, CentOS 4 & 5 (32 and 64 bit), PCLinuxOS 2007 and 2009, ReactOS, Windows XP (32 and 64) - configured as static - ie - no changes ever saved...
It's lots of fun to work with, and a great learning platform.
Who is general failure, and why is he reading my hard drive?
What is VMware? There is no such thing as "VMware".
There's VMware Workstation, Fusion, and Server. All of which are hosted applications (Server/Workstation on Win/Lin, Fusion on OS X).
Then there's ESX 3.5 (or ESXi 3.5), which virtualizes a completely different set of hardware for the VM then any of the hosted solutions do. ESX/ESXi allows for v4 virtual hardware ONLY (based on a 440BX Intel Reference platform), the hosted solutions allow for v4 or v7 virtual hardware (which is similar to v4, but supports 3D acceleration, passthrough PCI devices, etc).
Hosted solutions will run on anything that can boot the above operating systems. ESX/ESXi **REQUIRES** a specific set of hardware, and you'd be best to find something on the VMware HCL as such (a Dell T300, or an HP ML350).
So, again, what is VMware? Hosted or bare metal hypervisor? Two completely different series of products, and despite "generally" virtualizing the same hardware (from the VM's point of view), there are still heavy, radical differences between the two.
-AC
But the question is, how many VM's do you plan on running at once?
I installed a 2 Node RAC environment on Vmware using my laptop which was a 2Gz Intel Core 2 Duo with 2GB of ram. (Instructions here )
So you don't need something super powerful if you don't plan on leaving them all running 24x7 and just startup the ones you are playing with at the time. A Quad core system with at least 4GB of ram should and lots of disk should be plenty.
I would stay away from running any of your environments on external USB drives. I have a 1 TB USB 2.0 drive and its too slow for anything heavy such as installing Oracle E-Business Suite.. However databases and RAC worked fine.
I've played around a fair bit with Oracle and VMware so if you have any questions feel free to ask.
"Thanks to the remote control I have the attention span of a gerbil."
I've been thinking of maxing out my 8 Core Mac Pro with 32GB ram, gobs of disk space and installing XenServer or Vmware ESX server and boot via rEFit. Another option I'm considering is picking up a new box altogether. I have my eye on a Dell server with 2 4x core AMD CPUs. With a 300GB disk and 32GB ram, it goes for just over $3k. Add in a few SAS drives and you're around $4k, but you have a highly capable system capable of running more VMs than you probably need.
A while back I ran across these little boxes. They were being phased out, and were on sale. I bought one, and found that VMware ESXi works great on the... so I got 5 more ;)
I set them up with ESXi, and put 5 1TB drives in a midtower case running Linux with 3 GigE NICs, and setup NFS shares and iSCSI targets (just to play around). Bond the NICs and have ESX use it for datastores... all for $3,600.
Tada! Instant "blade" environment w/SAN! Sure, the performance isn't quite the same, but for proving out concepts and experimenting, it's awesome. And ESX is fun to play with compared to plain old Server (1 or 2). Not to be biased, but VMware is by far the most well stocked, feature wise, virtualization solution out there. I've personally used it since pre-1.0 back in 1999-2000.
I'm mentioning this since you mentioned VMware, and I thing someone above me mentioned it as well, but it's a important point; VMware ESXi is by far more picky about hardware than Linux. If you want to play with it at some point, make sure whatever you buy will work with it. Check out vm-help.com, which gives you more hardware compatibility insight than VMware's documentation.
Have fun!
For modeling something like RAC, a dual-core anything with tons of RAM would be necessary.
However, the devil's advocate in me is saying to not go virtual with this project unless you have some speedy-fast fiber channel SAN at your disposal. Reason being: you aren't going to see the same performance in the VMs as you would with physical hardware. Especially with the database backend that is constantly thrashing your drives depending on load.
The game.
The main requirement is enough RAM for two operating systems plus some extra for the virtualization system. The CPU is less important. I run Windows XP Pro as a virtual system on a Linux host with VMware Workstation 6. It is a 5-year-old Athlon 3000+ box with 1 GB of RAM. I allocate 512 MB to Windows, which is about the minimum for XP. Current Linux distributions need at least 256 MB and VMware is something of a memory hog itself so 1 GB is about the minimum RAM for this setup. Windows is perhaps just a smidgen slower than it would be if running natively on the same hardware but the difference is minimal. It does not have much effect on the speed of Linux apps running simultaneously. Things bog down fast if you try to run more than one virtual system simultaneously but VMware is good at using multiple processors for this. I did some work which involved running up to 6 instances of FreeBSD simultaneously on an 8-core Xeon system with 4 GB RAM. Up to 6 it did not slow down much. Over 6 it got sludgy. Have fun! Mike
Buy all the memory you can afford. Then buy some more.
Virtualization is a memory pig. Cool, fun to play with, but still a memory pig.
...laura
The biggest thing that you have to watch out for with VMWare ESXi is the hardware compatibility list. You will run into trouble with two major components: RAID controllers and network adapters.
The network adapter solution is simple: buy the most plain-jane Intel PCI or PCIe adapter that you can find. Examples of ones that are known to work right out of the box are the Intel PWLA8391 GT (single-port PCI) and the Intel EXPI9402PT (dual-port PCIe). I own both of these and can personally confirm operation with the latest version of VMWare ESXi.
The drive controller situation is both complicated and -- more importantly -- expensive. Overall, Adaptec seems to be the most well-supported controller hardware out there. I have tried LSI controllers, but they often don't play well with desktop boards. Unfortunately for experimenters, the built-in RAID on practically every Intel motherboard is completely unsupported in RAID mode. Obviously no enterprise environment would be using on-board RAID like that, but it would be nice to have for experimentation.
Which brings me to my favorite storage solution for ESXi: Openfiler. Openfiler is an open-source NAS/SAN solution based on rPath Linux. It turns any supported PC into a storage applicance, and can share its storage in a plethora of ways. In the case of a virtualization effort, it has two major things going for it: it supports any storage controller that Linux supports, and it supports iSCSI and NFS.
If, say, you do have a machine sitting there with Intel on-board RAID, you can install Openfiler there. While the hardware might not work under ESXi, it'll work great for Openfiler. Even better, Openfiler also supports Linux software RAID which can be superior when it comes to disaster recovery (no need to have a specific controller card to see your data). With this in mind, you'll be able to get Openfiler running on just about any hunk of shit box you have sitting around.
Once you have Openfiler set up, you can take the next step in virtualization-on-the-cheap: installing ESXi on a USB flash drive. There are a number of tutorials on the web for this (just google 'ESXi USB flash install'), but the basic process amounts to extracting the drive image from the ESXi installation archive and simply writing it to flash with dd (on Linux) or physdiskwrite (on Windows). Once this is done, you can plug the flash drive into nearly *any* recent x86 hardware and it will boot ESXi. A really neat feature that you get along with this is the ability to substitute hardware with ease, and upgrade to later versions of ESXi simply by swapping the flash drive.
Once you have ESXi installed, create an iSCSI volume on your Openfiler box. Then, use the VMWare management software to connect the ESXi box to your Openfiler iSCSI volume. You can then create virtual disks and machines from the actual USB-flash-booted VMWare host, all of which will be stored on your Openfiler machine. You may also want to try experimenting with NFS instead of iSCSI. There are a couple proponents of this out there that say under certain circumstances it's even faster than iSCSI. It also makes backing up your virtual machines a little simpler since an NFS share is generally easier to get to than iSCSI from most machines. Another cool aspect of the Openfiler-based configuration is that you will get access to another whiz-bang feature of VMWare called vMotion. Since the VMs and their disks are stored centrally, you can actually move the VM execution from one ESXi box to another - on the fly.
In all, this is a great way to get your feet wet in virtualization because you can have a pretty sophisticated setup with very basic commodity hardware. If you want to go the extra mile and get really fancy, put a dedicated gigabit NIC (or two, bonded) in each box and enable jumbo frames; the SAN will be more than fast enough most anything you'd like to do.
Good luck!
I've bought a small Shuttle K45 system, adding my own Intel chip and extras in there. Cost about $450 for my setup. About to put VMWare Server on it. I'll let you know how it works out.
--
# Canmephians for a better Linux Kernel
$Stalag99{"URL"}="http://stalag99.net";
Well you don't clearly state what you wish to accomplish nor how much money you have so it is hard to answer. But maybe such setup will be OK.
Build yourself custom PCs.
Storage server:
- good and big enclosure which can fit large ammount of drives
- moderate 64bit AMD processor (really any - you will not be doing any serious processing on storage server)
- any ammount of RAM (really 1 or 2 gigs will be enough)
- mobo with good SATA AHCI support (for RAID) and NIC (any - for management) onboard
- one 1Gb PCI-* NIC with two ports
- 6x SATA2 NCQ HDD (any size you need) dedicated for working in RAID - software based (dmraid) RAID1+0 array configuration
Virtualization servers (2 or more):
- you need the virtualization servers to have the same config
- any decent enclosure you can get
- the fastest 64bit AMD processor you can get preferably tri or quad core (it will do the processing for guests) with VT extensions
- as much RAM as you can get/fit into the machine
- mobo with VT support, one (any - for management) NIC onboard
- one 1Gb PCI-* NIC with two ports
- one moderate SATA disk for local storage (you will be using it just to boot the hypervisor) or disk-on-chip module
Network switch and cables:
- any managed 1Gb switch with VLAN and EtherChannel support, HP are quite good and not as expensive as Cisco
- good CAT6 FTP patchcords
General notes for hardware: :)
- make sure all of the PC hardware is *well* supported by Linux since you will be using Linux
- if you can get better (quality wise) components, good enclosures, power supplies, drives etc. - since it is a semi server setup you don't like it to fail for some stupid reason
Network setup:
- make two VLANS - one for storage, other for management
- plug onboard NICs into management VLAN
- plug HBA NICs into storage VLAN
- configure ports for EtherChannel and use bonding on your machines for greater throughput
Software used:
- for storage server just use Linux
- for virtualization servers use Citrix XenServer5 (it is free, has nice management options, supports shared storage and live motion) or vanilla Xen on Linux, don't bother with VMWare Server, VMware ESX and Microsoft solutions are expensive
Storage server setup:
- install any Linux distro you like (CentOS would not be a bad choice)
- use 64bit version
- use dmraid for RAID and LVM for volume management
- share your storage via iSCSI (iSCSI Enterprise Target is in my opinion best choice)
Virtualization servers setup:
- install XenServer5 (or any distro with Xen - CentOS won't be bad)
- use interface bonding
- dont use local storage for VMs - use storage network instead
Well here it is. Quite powerfull and cheap virtualization solution for you.
I not only run this at home, but at lots of small business customers. Has 3Ghz Pentium D (dual core, 64-bit). Get 2 large SATA drives (500G or more) and 2G or more ECC memory. Starting price is $400, but by the time you get the memory and disk upgraded, it is about $600, $800 with onsite maintenance. A big benefit for me for home use was it is *quiet*. It has a single large (and therefore quiet) fan with ducting to draw air over the CPU heatsink. Look for it in the "small business section" of Dell.
Drawbacks: only 2 drive bays (upgrade to 840 for 4 bays - not as quiet). No sensors - at least that lm_sensors knows about. I just monitor the disk temperature.
Configuration: run the 2 drives with software RAID1, and LVM on top of that. Create a small (100M) RAID1 boot partition at the beginning of the disk. The RedHat/Fedora installer can create this configuration. (I also save and mirror the Dell diagnostics partition, and add it to the grub boot menu.)
Here is a compatibility list for ESXi. If you pull it off then add a second machine and build yourself an ESXi cluster ;). You need to pay attention to the SATA controller.
:).
http://www.vm-help.com/esx/esx3.5/Whiteboxes_SATA_Controllers_for_ESX_3.5_3i.htm This way you are not only getting experience with your line of work applications but also with the VM software most datacenters use. VMWare Server/MS Virtual Server are nice alternatives when you don't have ESX/ESXi compatible hardware, but eat much more HW resources and ESXi takes just 3 minutes to install and you don't have to patch the operating system every other week
You could also checkout FreeNAS as an alternative to Openfiler, just depends on if you want to run Linux or FreeBSD on your NAS.
Second, the cheap crap they put on motherboards and call 'RAID' is generally nothing of the sort. Its almost always handled by the CPU itself, either via the driver or System Management mode of the CPU and as such is no better than using the software RAID provided by your OS. In most cases its better to use the software RAID as its made to work with your OS in a the most efficient manner, rather than however the MB designers wanted to do it. And since the MB designers were just looking for some cheapass way to add the word RAID to their marketing material, its probably not exactly the best way to go about it.
VMotion does not work with ESXi. You must have ESX and Virtual Center to gain access to all the neat stuff involving more than one vmware server.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
You should have at least 2 of these: AMD-V or Intel-VT Capable Motherboard and Processor Combo (for those interested in running Hyper-V or other enhanced VM setups) I prefer the Intel branded boards for my setups, never let me down... At least 4GB of RAM per box Cheap SATA drive 100GB maybe? 2 or more Intel Pro 1000 NICs (can get them for about 35 bucks on newegg) You should also get: Any box with a P4 or similar should work for this. Setup Openfiler or FreeNAS. If you are playing with VMs, shared storage is a must. You will preferably setup RAID0 or something equally as fast (assuming this is purely a test environment and you dont care about redundancy.) It will be hit or miss with the network adapter support for integrated NICs so you may want to pick up an intel pro 1000 for one of these as well. You will also need supporting hardware 2 gigabit switches - 3Com unmanaged 5 or 8 port ones are excellent and support jumbo frames very nicely A cheap 4 port KVM, there is a trendnet 4 port USB KVM for under 75 bucks available at most shops 3 desktop UPSes or maybe 1 good enterprise UPS. Always a smart move to protect your investment, and power failures can easily screw up a RAID0 solution. Don't forget to check out the online HCLs for the VM solutions you want to run. The above configuration should work for VMware ESX or HyperV, ESXi has tighter requirements so it would be worth checking out. Happy hunting!
Nuclear war would really set back cable. - Ted Turner
Simply put, two new cheap servers identically configured, with at least 4-8GB RAM each (really this depends upon how much you want to use--the sky is the limit), and one system to run iSCSI / OpenFiler (can be different / older / repurposed hardware). Make sure to have enough NICS to isolate your iSCSI traffic to its own interface. TCO will probably be around ~$2000, if you have some old hardware laying around to fill in the gaps.
I use a Core 2 Duo laptop. I bought the laptop off of Ebay with a busted screen for $150. I bought 4G of RAM for it, which cost me another $35. Once I got CentOs installed on it with an old monitor plugged into it, I closed the lid and put it on the shelf tied into my network. I have 27 VM images on it that I use for various programming test environments. Granted I never run more than 5 or 6 at the same time, with the Core 2 Duo I can run x86-64 images as well to let me test on as many different types of systems as I can get my hands on.
My 2-cents.
I run a VPS hosting company, my job is to research, setup, and maintain a cluster/grid of servers running Xen with hundreds of guests (virtual machines). For testing and even for deployment, we've used machines as simple as a single-core AMD 3800 with 80GB disks in RAID-1, and 1GB of RAM. These aren't the most profitable machines, as they can only support as many virtual machines as can pay for the electricity and square footage, but they work perfectly fine for up to approximately 12 guests. I do highly recommend a dual-processor or dual-core system, though.
If you want to know how much you can stress a system, for highly-dense numbers of guests, I try not to load more than 15 guests and 2GB of RAM per CPU core. Of course, if you plan to have a low-density of guests (say one guest per core), you'll need to adjust accordingly.
I found that for my home office, where I often have pretty excessive needs such as installing multiple operating systems and performing multiple large compiles at the same time, a dual quad-core system with 16GB of RAM is overkill. Right now, I'm using a single quad-core workstation with 8GB of RAM and it works pretty well for me, and is probably still a bit more than I need.
I find that enabling hardware virtualization in the BIOS and installing enough RAM for my VM needs is enough for low budget. Storing the virtual disks on a hard drive physically separate from my host operating system allows things to run well without much impact to my other processes. You can offload your virtualization to a different computer and run the management client on your primary computer if you require more processing power. I'm sure having a 64bit quad core CPU will help out too, but it isn't necessary for most things. Use whatever VM software you like the most.
I have a dual dual-core Xeon system built on a Tyan Tempest (i5000VF) motherboard with 8GB of RAM that runs XenServer 5. Right now I have it running 2 Windows 2008 domain controllers, an XP instance, an OpenSolaris instance, and several Linux VMs.
From NewEgg, that RAM cost me about $160 total. The 5 500GB drives (at the time I bought them) were $150 a piece. The processors were $150/each and the motherboard was $340. I picked up a 3Ware 9550SX PCI-Express RAID controller from E-bay for about $200.
It is server-class hardware, but can be built in stages (e.g., start with one processor, 4GB of RAM and 1 drive). I'd recommend not skimping; you'll appreciate the stability in the long term. I've been using this setup for about 2 years and am just now looking at starting again with new hardware (I'd like to build a shared-storage setup with OpenSolaris and ZFS).
Amazon Web Services (AWS)
Amazon Web Services (AWS) is a great way to do something like this w/out laying out any money.
As low as 15 cents per hour and ONLY when you are actually using a virtual resource.
Get an AWS account - its free.
Learn how to use their Amazon Machine Instance (AMI) management tool to launch any of the hundreds of publicly available operating system images out there (Windows, Solaris or Linux), 32 bit or 64 bit.
Clone an AMI that you like as a base to make it "yours". Customize it however you like.
Put VMware Workstation on it and you can experiment building Virtual machines all you like.
I've done it with an Ubuntu 64 Bit AMI then after installing VMware on the Ubuntu 64 I have played around with creation of virtual machines of other operating systems and applications such as Ubuntu Linux, Mac OSX, and Windows XP & Vista.
Its basically using the virtual cloud environment that AWS offers to work on developing your own virtual appliances or machines.
Standard AWS Instances
Instances of this family are well suited for most applications.
* Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform
* Large Instance 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform
* Extra Large Instance 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform
High-CPU Instances
Instances of this family have proportionally more CPU resources than memory (RAM) and are well suited for compute-intensive applications.
* High-CPU Medium Instance 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of instance storage, 32-bit platform
* High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform
EC2 Compute Unit (ECU) â" One EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.
Oracle already has a number of guides to building a cheap Oracle RAC setup. One of the more interesting ones used a firewire device that could support multiple logins. Thus creating a cheap and fast shared storage device to use for ASM and OCFS. The article is here: http://www.oracle.com/technology/pub/articles/hunter_rac10gr2.html. The setup here was only a 2 node system. I'm not sure if these cheap firewire drives can handle 3 logins. There is another guide for doing iSCSI, although I would think the firewire setup would be cheaper and faster.
Good comments, but vMotion most certainly does work with ESXi. Yes you need Virtual Center, but ESX is not a prerequisite.
In the long term, I believe that VMWare sees greater uptake of ESXi vs. ESX since it is a lot thinner and plays better in a dense environment.
The biggest problem I see with those getting into virtualization is that they think that virtualizing things makes them magically need fewer resources.
You can share CPU time as most apps will not drive the CPU 100%, having said that it is often best to have as many cores as you can afford.
Do not over allocate your RAM, if you can have as much ram as needed for how much you allocate to the VMs, if you over lap you will get a huge performance hit.
Sparse disk is a fairly new feature only in some VM systems, you will need lots of disk for all of the VMs, also you will probably want to run them on different LUNs or disk groups so you don't get lots of thrashing on the drives.
If you are only running 1 or 2 VMs as a test then really all you need is to up the ram a little and make sure the host meets the minimum specs of the VM applications.
EA David Gardner -"... but the consumers have proven that actually what they want is fun."
installing stuff to run it as a hobby and not push any major data sets through the system you really only need to worry about RAM and disk capacity (for storing the VM files which will house the OS and programs). Just get as much RAM as you can so you can give each VM its own normal amount of RAM (500MB-4GB) depending on which applications are in the VMs. You probably want each VM to have at least 10GB of disk space so calculate that in to your overall disk capacity requirements. Your hardware in the end will be based on how many VMs you will end up running. Each one will have to have an OS installed of course so you are going to have some wasted disk space. Worst thing you can do is skimp on RAM....you don't want your host OS to be swapping because you gave the VMs too much memory and didn't have enough left for the host.
this nation, under God, shall have a new birth of freedom. -- Lincoln, Gettysburg Address
Speaking as an Oracle DBA who has done a little of this, I can tell you to get a lot of RAM. I would say that an MB that can be expanded to at least 8 GB is the way to go. You might get by with only 4GB for a while, but you will eventually want more, give the relative costs of RAM.
Oracle is always RAM hungry, and VM's multiply that.
I just did this myself. I ended up just shooting for cheap hardware on the theory that if it breaks in 2 years I can just replace it. I have a Quad Core Phernom with 8 gig of RAM and two 750gig drives. Chucked VMWare on it and havent had any issues running about 8 or so VM's on it. It also serves up media using TVersity and is a network share dump as well.
The biggest issue I have had so far, is Disk Driver perfomance. If you are planning on running multiple concurrent VM's then go for as many HDD's as you can. Stick the most load intensive ones on seperate drives and you will really see the benefits.
I just built one yesterday that runs vmware quite nicely.
Gigabyte GA-G31M-ES2L Motherboard w/3-D graphics $60
Intel E7400 2.8 GHz dual-core Processor $100
Corsair TWIN2X4096-6400 4 GB DDR2-800 Memory $55 (not counting $30 mail-in rebate)
XION XON-101 Case w/450-W Power Supply & 3 quiet fans $60
OEM DVD +/-RW Drive advertised sale $25
I already had a hard drive.
Came in right at $300 at my local MicroCenter. You should find comparable prices elsewhere if you shop around a bit. TigerDirect has a couple of very similar bare-bones kits at very similar prices.
Great performance for the money!
See the following resources:
http://www.oracle.com/technology/pub/articles/tech_dba.html#linux
-Mark
There are 10 types of people in the world; those who understand binary and those who don't.
The only crucial prerequisite for VMs is having enough extra RAM for the overhead of the host OS to run nicely. The host OS should use spare ram over and above that to cache disk access, which boosts VM performance.
The second rule of thumb is don't blow money on top spec hardware.
DDR2 RAM is cheap, load it up. This is the only real fun killer if you don't have enough, all other advice here is non-essential, any non-dinosaur box is fine for fiddling with VMs.
An interesting note a discrete graphics card (any cheap or old with 128mb buffer) will lift overall system performance. Otherwise some bus bandwidth is sacrificed for onboard graphics. 5-10% overall performance maybe.
As a tip, I find VMs love CPU clock speed, more so than extra cores, so a dual-core is good VM experimentation stuff. It depends on what your doing of course, quad cores are a benefit if you want to run mutliple VMs or some heavyweight multi-threaded tasks in your guest OS.
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.
I was running VM's back in the days of DOS. First with Taskview and later with Deskview running 4 concurrent DOS v5 sessions on a single-core 8088! And if they slowed down I'd just push the turbo button and go from 4.77Mhz to 8Mhz! oooWEEEE! That's right! And I'd tote that 45lbs IBM-XT all the way to school...in the snow! And I LIKED IT!
The main thing you need for VMs is memory. There isn't really any good way for VMs to share memory, they each need their own. So decide what you want to give each system, and make sure you've got that much on the host plus like 1GB for the host OS and VM software. Good news is RAM is cheap. You should be able to pick up plenty for not much money. If you get a system based on a 975X or P35 chipset, you should be able to drop 8GB of RAM in it. Ought to be more than plenty. Those are cheap and plentiful these days too. Plus, they use DDR2 RAM, which is currently the cheapest. An Intel DP35DP motherboard might be a good choice.
As for a processor, kinda depends on how hard the VMs will be working. That they can share. So if they are mostly sitting idle, like say a web server serving up static pages, you can get away with not a whole lot of CPU power. If you want them all to be working all the time, you need more. A Core 2 Duo will probably do just fine if you that's what you've got or you need to keep the cost as low as possible. However, this is a case where a quad core would make more sense so that's a good way to go if you can. Goes double if it's the same price. Like say you can get a 2.4GHz Core 2 Quad for the same price as a 3.0GHz Core 2 Duo. While for a desktop you'd probalby want the duo, get the quad in this case. Might look at the Q6600 or Q8300. Both are under $200 and would do a real nice job. Note that the Q8300 is going to need a P35 board, teh 6600 will work on a 975 board.
Disks are a real big "it depends." VMs can be set to grow as they need more space, and so you can in theory have a bunch of VMs sharing one small disk, along with the OS. However, that can lead to performance problems. Harddrives suck at random access, and if a bunch of VMs get going on it at the same time, that's what you'll get. So ideally you'd have one VM per harddisk. In reality, that's probably overkill unless you've got lots of disks laying around. However if your VMs will be heavy disk access, you might want to consider getting 2 drives for them since drives are cheap. Either way, the best idea is to have the preallocate all the space they need for their virtual drives. You get better performance that way, even though it wastes drive space, but again, drives are cheap. Maybe start off with one drive for the VMs and if you find they are getting bogged down, buy another and move them over. They are just files on the drive so easy to move.
Those are the biggest factors to think about. You get a quad core, good amount of RAM, and enough disk space, you should get great performance. If you need to save money, don't feel like a dual core won't work fine. Really the only thing not to cheap out on is RAM. You need to have enough, virtual memory is WAY too slow. So if you want 4 VMs with 1GB each, have not less than 5GB in the system.
Supposing you do have plenty of cash and want to further increase performance one other thing you can look at is NICs. VMs don't do a great job of sharing NICs presently. VMWare is actually working on that, but right now you get ideal performance with one NIC per VM. Not normally a big deal but if your VMs do lots of traffic it can matter. So if you want, get more NICs. One of those multi-port NIC cards works just as well. This really isn't all that necessary, but you can do it if you are after the best performance.
I'm doing this now, running a company infrastructure on Xen3.2 and 2 non-server class machines. These are Gigabyte and MSI Core 2 Duo motherboards running at 2.6 and 3Ghz. Each with 4GB of RAM, dual GE NICs, RAID1 drives. Nothing special.
Application Systems are:
- enterprise email/calendaring/IM
- CRM
- document management, file/print
- project management
- VPN
- internal website / wiki
- VoIP/PBX
- Monitoring, PKI to manage VPN credentials
- LDAP for authentication across all these systems.
Each VM can be migrated to the alternate host-server with minimal downtime (sub 1-second).
Backups are rdiff-backup based - complete VM backups take less than 2 minutes. Most of the machines are only 10GB disk images. DMS is 20GB since we're a document heavy enterprise.
Total CPU is hardly ever over 20% utilized, basically, only during backups. Because Linux grabs available RAM for disk buffers, it is all used, but everything easily fits on a single 4GB RAM box with excellent performance. This is nice so system upgrades don't impact running systems, but most of the work can still be performed during work hours. Having 2 boxes lets me perform system upgrades without any risk to the "production" system.
I'm running 64-bit Ubuntu for all.
I tried VMware - it wouldn't load ESXi on my hardware and VMware server is "too heavy." For some of out customers, VMware is the best solution, but they are Windows shops.
I use VirtualBox on a laptop, but it isn't ready for enterprise use. Another year or so and it will be stable enough. 3 of my partners also use VBox on their laptops. It's easier to setup a VM with Linux than fighting with cygwin.
Xen and Linux go together nicely. I plan to bring up a few Windows VMs - I've read they work fine under Xen3.2, but haven't had time to try them yet.
I've blogged about most of what I've learned along the way. Learning about systems, applications, Xen, and other virtualization issues.
VMWare can give 1 or 2 cores / guest.
VirtualBox can't ( only 1 core / guest ).
Parallels is crippled to giving only 1.5GB RAM / guest.
Xen I don't know, but since it's a *server* virtualization system, I don't think it's possible to arrange a desktop guest through it, without trickery ( rdesktop?, or giving 1 video card to the guest, via PCI passthrough, and having dedicated displays for it? )
Simply, work out how much CPU each guest needs, .. $$$ )
decide which virtualization SW you need use
( it's gonna be VMWare for me, unfortunately
Decide how much RAM you need for each,
Add in 1 or 2 cores for your host, and enough RAM for it, too, and you then have an *estimate*, rather than a mere guess.
Asus has some AM2+ and AM3 mobos that can do quad-core, 16GB, *and ECC*.
Cheapest deal, if you don't need more than quad-core.
If you *do* need more than quad-core, then you're looking at a 2-socket board, and that costs lots more.
( Supermicro has some nice ones with 5 pcie slots, & lots of DIMMs )
From all I've read on virtualization servers, though, it's RAM people usually underestimate, and CPU people usually overestimate.
Part of the reason for that is
a) if you're stressing 1 server, you are unlikely to simultaneously stressing some *other* server
( that *doesn't* apply to web server + database backend: both get stressed simultaneously, then )
b) most apps are rigged to be single-core, due to coding/language, lack of parallelization, whatever..., and
c) EVERYthing that is running, including the VMs, including the extra overhead for that, and the disk-caching for EVERY OS, oughta be in RAM.
( look into the Linux Terminal Server comments strewn 'round the 'net, for some other's experience with that )
I plan to do a similar experimentation on RAC (and also Data Guard)... waiting for my current computer goes out of order, so that I can get a new one.
I think any commodity CPU with several core can do the job well. You need much RAM, as each RAC instance requires around 400MB, and you need to have memory for each physical / virtual machine. For storage, I plan to share from the host (nfs or iscsi). As a result, when I want to get the best performance, I can run Oracle directly on the host. (You can use 1 host + 2 guest for your 3 node RAC)
One advantage of running RAC experimentation in VM is that you don't need to invest on the internal gigabit network.
quad core, 16G ram, extra cooling, and a good power supply
running Gentoo as KVM host, 2 postfix mail servers, 2 appache web servers.
box2 same hardware gentoo host , 1 xp guest with vsnet, and MX, 1 gentoo with zend studio and test apache server, mythtv backend, and torrent box
275 watts, uptime box 1 359 days, box2 not that much
Proudly Butchering code for 20 years
Also, I should mention that the reason I drug the whole Intel RAID into the mix is that ESX/ESXi does not support software RAID, so if you want RAID you *have* to have some sort of hardware solution (even if the processing is done on the host CPU). So, for those that are experimenting it would be nice to support the Intel RAID since it's free for the having on most recent boards.
For the reasons detailed above in both our posts, it would be even nicer if ESX/ESXi supported software RAID. However, given its enterprise purpose I highly doubt we'll see this in the near future.
For the best performance in Virtualization, buy a Nehalem CPU, a core i7, 4 cores. We are using those at work and the benchmarks are amazing.
If you don't have much money, buy the low end quad core, core i7 920, throw in 8GB of memory and you will have plenty of power and memory to throw in a couple of 1vCPU VMs, the performance is pretty good. If you have money, buy a dual core i7, 8 or 16GB, then you have something that smoke. VMWare esxi is pretty cool. I believe they have in the work a version optimized for Nehalem.
I'm experimenting with XenServer on some enterprise software. Turns out that Oracle will never certify (I'm guessing) anything but their own VM software, which IS Xen based. Think hard about that, I know I am.
Well anyways, I don't remember if it's VMware or XenServer, but all processors in the Resource Pool must have the same processors. All Intel or all AMD in order to take advantage of High Availability. Oh yea, and I think XenServer only supports 64bit systems. Hum, what else.
Oh yes, more processors are better, don't assign more virtual processors than you have physical or strange things start happening. Not sure if you're looking at RAID for shared or local storage, but NAS storage would be handy on a quick network. If you're looking to serve your VMs from shared storage, well unless you're going fiber channel or something else really expensive, then commercial RAID is you're best bet.
If this really is just home experimentation, then RAM won't be a concern, but if you're looking to do some serious work, you're looking at 1GB of RAM per virtual machine if you ask me. Of course, we do some crazy number crunching. Anyways, food for thought.
I have a 1,83 core2duo running 4 vms. A webserver, email server, sql server and a backup system.
Works perfectly.
You don't need a lot of cores for VM hosts. But you do need lots of RAM, since each VM can take a huge chunk.
So, essentially, you don't need anything "special" hardware wise to use VM's. And I recommend using Linux + VirtualBox. http://www.virtualbox.org/
If you want to run linux processes with isolation from your physical machine, install an OpenVZ enabled kernel plus the openvz packages. It nicely isolates processes running inside each container; there is minimal virtualisation overhead (so you don't need a bigger machine).
Also the container root filesystem is an ordinary directory on your host. This means you can put multiple containers into a large filesystem and they share the available space, you can backup or copy containers trivially, and you can extend or reduce the amount of space available in the root filesystem while the container is running.
The amount of storage used per instance depends on your distro size; I start each instance with a minimal set of debian packages from a template occupying about 200 megs, and install more from there.
www.openvz.org.
My current box hosting VMs is the following:
Core2 Duo 6600
4GB Ram
2 100+GB HDs (might be 250s, don't recall off hand)
MB is a Supermicro X7SBA (probably the most expensive part)
I'm using MS Hyper-V server (I'm not a Linux user), and it works quite well. Yeah, I can't run 10 different VMs simultaneously, but I can run what I need for learning / experimentation, etc.
Do you see the FNORDS? I refuse to post anonymously, as I am fireproof!
We have an Xserve where I work. It's a terrible piece of hardware and software... It wasn't doing anything we needed it to do reasonably (uninstalling the default Apache and LDAP installs just to recompile them to play nice was ridiculous), so we bought VMWare Fusion and ran some servers on that (it was easier to authenticate Apache on a Linux VM to the Xserve's open-directory than it was to do it all native on the Xserve). We ran into more snags (not being able to bridge specific VM's to specific NIC's, etc), so we decided it was time for ESXi.
Now, rEFIt, boot camp, etc, all play relatively nicely on the Mac Pro, the MacBook Pro's, and others (from what I hear). Not the Xserve. Apple managed to lock the Xserve down to the degree where anything that isn't OSX server is going to be a BITCH to install. Period. Save yourself a couple grand, a few hours (if not days), and your sanity; don't buy an Apple product for virtualization. We're moving from the Xserve to some more standard hardware.
Both Nehalem and Barcelona (Phenom) are quad-core and most importantly, support EPT and NPT respectively. This feature has significant impact on virtualization performance.
If you want to run 4 VMs, you'll probably want to have a fair bit of memory. 4GB would be good, 8GB would be better.
Memory is a bit more key than processor speed IMO. Any recent Core 2 Duo should be more than adequate to run a VM. You'll find you can run Vista, and XP in 1 GB and 512 MB very comfortably. If you plan on running multiple VM's at the same time, you will definitely need 3-4 GB of RAM.
;)
If you have need for each VM to have access to specific hardware like a DVDRW or whatnot, you can either connect and disconnect it as needed to each VM, or if you need it on both at the same time, you'll want a box that you can physically mount both at once. I go the cheap route and just connect them to the VM that needs them
I run Windows XP, and Ubuntu 8.10 on my iMac (dual core 3.06 Ghz and 4 GB of memory) via VMWare fusion. My XP machine is actually a VM image of my work box. The XP VM uses 1 GB and the Ubuntu 512 MB. XP runs very well including IP phone, even with 512 MB though.
I guess I'm saying you shouldn't go crazy on hardware, but do spend a bit extra for more ram. you'll need it if you plan to run multiple VM's at once.
I bought a Rackable Systems dual proc, each with dual cores with 16 gig of ram and a 500gib hard drive (internal) off of ebay for less than $400 US. It is currently running 4 Checkpoint R65 firewalls, a windows 2003 server, a windows XP workstation and a Provider-1 management server. All that eats about 6-7 gigs of ram and 3 processors when the traffic gets pretty high. When I use the Provider-1 system to manage the firewalls, it boosts up to 4 cores and 8-10 gigs of ram. The system is quite noisy, but I stick it in another room and it helps. I have 2 terabytes of NAS that I run most of the VMs off of with a 24 port gigabit switch.
"My immediate reaction is "WTF? What kind of moron doesn't make things 64-bit safe to begin with?" Linus
Lots of RAM:
and plenty of disk space (I'm at the better part of 1TB on the host for a variety of reasons).
So far I've successfully created VMs for Ubuntu 8.10, Fedora Core 10, Windows 2000 Professional, Windows XP Home, Windows Vista Home Premium, Windows Server 2003 and 2008, CentOS 4.7 and CentOS 5.2. I'm running CentOS 5.2 (x86_64) as the host OS and I had to rebuild the kernel, qemu, KVM and SDL to get everything running correctly. All of the OSes except FC10, Ubuntu 8.10, Vista and Server 2008 run fine on the base system. Ubuntu and FC10 needed the newer kernel. Vista and Server 2008 needed the other software updates. The upgrade to the 2.6.28.7 kernel from kernel.org is usually stable but I still run on the latest CentOS kernel unless I need to play with Ubuntu or FC10 just to make sure things are stable.
More details on my blog. Vista and Server 2008 are both dogs but everything else has acceptable performance in a VM.
Cheers,
Dave
They that can give up essential liberty to obtain a little temporary safety deserve neither safety nor liberty.
Ben
Memory, and lots of it. Nothing else will help as much for running multiple VMs.
Memory is dirt cheap, I recently bought 8 gigs of ECC ram for ~100 USD. Of course, over 3-4 gigs, and you need a 64 bit OS, I use Ubuntu 64, but I know others who use Vista 64 to good effect.
At least 2 cores, 3 or for doesn't hurt either. There's great value in both AMD and Intel at the moment, Intel owns the top end, but at the low end or midrange AMD tends to have the better value.
If possible, get a separate drive for at least your main OS, and run the VMs off their own drive. More spindles == more IO, I run 6 drives in my box, one for the OS, and 4 raid 5 for my homedir for speed, capacity, and safety, and one drive bay I swap out for a spare I keep offsite that holds my backups. Linux software raid is great for this use, and with modern multi-core processors you won't notice the overhead.
If you can only afford maxing out one thing though, make it the memory.
Blessed are the pessimists, for they have made backups.
I litterally just puchased an AMD Phenon II quad core, 8GB ram, etc, etc.. for the exact same purpose. I just installad the db software on the first node. and running gentoo. my entire box is brand new and cost under 500$. I also had a cheap dual proc pentium 3 with xen and had an entire RAC system running without issue. (iscsi for shared media over openfiler) It doesn't take much.. not exactly the fastest cluster, but it worked for a test.
Take your minimum disk and RAM requirements for a single server, multiply by how many VMs you want, these are your minimum disk and RAM requirements for the host. There is no minimum CPU speed, slower will be slower and faster will be faster, but a test rig won't fail to work just because you have a single core.
I'm running a small cluster of load-balanced LAMP VMs on my laptop; 128MB / 3GB each, sharing a single 1GHz CPU :P
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
Just about any machine you can buy these days can do full virtual. If not, get your money back!
Ruby Neural Evolution of Augmenting Topologies
In terms of VMWare, depends on whether you are going to try to use VMWare Server or ESXi.
ESXi is very picky about hard drive controllers, and somewhat picky about NICs, but well give you better performance than VMWare Server.
ESXi claims to support only 2 SATA controllers - Intel ICH7 and Broadcom HT1000. From experience I have also used it with a Promise SATA300 TX4 controller. I *think* it may have recognized the nVidia controller in one of my systems, but I am not 100% certain.
The biggest issues to consider beyond the hard drive and NIC controllers is how many VMs do you plan on running at the same time?
The more you run at the same time, the more RAM you need. In addition, you will find that as one has more simultaneous VMs, access to local disk becomes a huge performance issue (more so than simple bandwidth - the virtualization layer kills I/O performance to local disk). If you want to run several simultaneous VMs and keep your performance up, you will either want SCSI with multiple LUNs or multiple SATA drives with each VM stored to a different drive.
Hardware virtualization for the CPU is required by some hypervisors (Xen, for example requires hardware virtualization if you are not using para-virtualization). VMWare ESXi does not require hardware virtualization - though I suspect that it performs better with the hardware virtualization.
The basics to virtualization comes down to the # of cores, the amount of memory and the # of spindles, though if you've read through the latest reviews of SSD's on Anandtech you can replace spindles with Intel X-25m or X-25e drives. In a virtual environment random reads/writes are FAR more common than any sequential read/write access, therefore you either want a high spindle count or fast SSD drives, depending on your budget.
A quad-core system with 4-8 gb or more (depends all on how much memory you want to allocate per VM) and either alot of SATA drives or one or more Intel SSD's will be very fast.
Most VM's that I've seen miss the boat on the disk throughput but deliver on the amount of memory and RAM. Given those requirements you can find a quad-core processor + mb for ~$200 (I bought mine for 180), 8 GB RAM (PC2-6400) ~$60-80, and one intel ssd for $325. You should be extremely happy with the performance. If you need storage, I'd suggest buying some sata drives in addition.
you said you're not in the market for server hardware, but have you considered 1 or 2 generation older stuff?
you can pick up single and dual socket motherboards for well under 200$. 2ghz chips for 50-100$ each(that's a pure guess, they may be even cheaper), 1gb ram sticks for about 10-15$ each.
I've got tyan s2895 and s2892 boards, both work very well with windows or linux, vmware esxi will install without any fuss if you use a sata drive, and, if you so desire, I hear they make a mean hackintosh. my s2895 has 2 2.6ghz chips and 4gb of ram, running about 6 (mostly idle) linux vm's that all feel as responsive as my desktop machine.
you can get opteron 240-246's or higher for pretty decent prices, possibly even dual cores.
the only downside is they run a bit warm.
that, or I'd suggest a core 2 quad. those can run a little warm too, in my experience. your ram and motherboard will be cheaper than the opterons, but I have no idea how they play with osx or vmware.
http://www.stikc.com/ Pick up a used server. great people. (I don't work for them)
I'd recommend a multiprocessor machine with extra slots to squeeze in more NICs. I got a reasonably priced Dell 6650 and bought 2 2-port gigabit processors so I can map hardware ethernet devices to separate VMs. That'll allow me to span VMs over different subnets.
But, honestly if you're just doing this for fun, you can run the VM server on any old PC. You'll be surprised at the small footprint of your VMs, which validates the whole concept.
I swear to God...I swear to God! That is NOT how you treat your human!
- Get a motherboard with support for loads of RAM and 2x quadcore CPU's.
- Get CPU's that pack some punch but doesn't shred your wallet to pieces.
- Load up on as much RAM as you can lay your hands on. It's dead cheap these days anyways so..
- Install Linux, set up ramdrives, create script that copies VM disk images from disk to RAM and launches VM's on boot.
- ???
- Profit!
No seriously, I've always wanted to try VMWare ESX on a nice computer with the VM's running in RAM. With the tiny overhead from ESX, the ability to assign VM's to CPU cores and the insane speed of RAM, you'd have some ungodly fast virtual machines there!
PlastBox, to lazy to sign up
Forget screamingly fast video - that's not what virtualisation is about (and probably won't be supported as a virtual device, anyway). Likewise you won't need anything more than a basic audio implementation.
Given that you won't be playing games on any of your VMs, processing power will be less of an issue - just make sure you have enough grunt to get Vista loaded.
I run VirtualBox on a 2GHz/3GB Athlon and that's fine for a couple of VMs
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
I built a system rather cheaply that works great with virtualization. Go for a decent quad core (Q9450/Q9550 for example) and a MB that supports DDR2 instead of DDR3 (or both, like this motherboard)
- The Q9450/Q9550 have large caches and overclock (on air) easily and stably to 3.5GHz+ without any hassle.
- Get reasonably fast DDR2 ram (esp. if you are overclocking). You won't reap much/any benefits from going to DDR3 with this setup, so go the cheaper DDR2 route. Get 4GB (or more).
The rest of the parts will be up to you, but this is what I'd recommend at the moment as the core of your system. Make sure that you get a MB with enough internal SATA ports (or eSATA) - the MB I linked to has only 4xinternal SATA and 2xeSATA ports; if you need more, you'll have to look elsewhere.
Cuban Music MP3's - cuband.com
You could go out and by a multiprocessor box with loads of RAM, but you can get by with less. Over the last year I've been doing training for an MCSA and all you get to play with there is a 2.8GHz P4 with 1Gb of RAM which turns out to be enough to run a whole active directory environment (2-3 servers and a client). If you just want to play with a few OS's then pretty much any box you buy today will have loads of RAM and at least a dual core CPU, more than enough imo.
I've got an HP ML110 G5 (dual core) with 1GB RAM running OpenSolaris and has 4 disks in. This is my "storage" server and runs ZFS, making it capable of serving CIFS, NFS and iSCSI. I've also got an HP ML115 G5 (quad core) with 5GB RAM running ESXi off a USB memory stick (internal USB socket). No failover in this configuration, but I'm able to run a complete test Windows 2008 domain (2 x DCs, 1 x Terminal Server, 1 x WSUS server, 2 x Vista clients and 2 x XP clients - all thanks to Technet).
The HP servers are cheap and relatively powerful. See my blog if you want more detail.
I don't know about VMWare, but in VirtualBox you need allocate a certain amount of RAM to each and very virtual machine. So if you have Windows Vista you run out of RAM very quickly if you want to run several at once.
Most of the posters here also seem to forget that Windows likes to access the hard drive quite frequently. Hard drive bandwidth will be a huge bottleneck if you have several virtual machines with Windows running at the same time, because they will all want to access the hard drive to index files, defrag a little bit and so on. You should really get a processor with VT (all new Athlons and most new Intel processors have support for virtual machines, you should still check). I don't think speed will matter too much, because current processors, especially those with VT are fast as heck.
Then you would get lots of RAM. Maybe go with a DDR2 setup. You can get 2 GB of RAM for under 20 bucks. So if you put 4 of them on your board it comes down to 8 Gigs for 80 bucks.
To ease the hard drive bottleneck I would simply spend my money on several hds. The cheap (small as in 160GB) start at 60 or 70 bucks. Get four of them and spread you virtual machines across those.
Also keep in mind that you will need a 64bit operating system to support 8 Gigs of RAM. And a processor that supports a 64bit os. Again, all Athlons support 64bit, not all Intel ones do.
We had a lot of fun making a perspex case for two DG945FC boards with E6300 processors, with a third board to act as the controller. See http://campbell-lange.net/company/forums/february2009/ We put the machines together in a case in order to demonstrate both services and operating systems migrating between machines. Windows 7 moved across well. Xen works well on these. VMWare Infrastructure however doesn't support the boards. As the previous poster mentioned, memory is all important. We've found this little test rig is hugely useful for testing, particularly when one has to throw up different Windows instances to check Linux interoperability.
The only reason you want a oracle RAC at home is for experimentation and getting expierence with it. In that case you either run in on bare hardware and buy 3 cheap machines with lots of memory and dedicated gigabit (or higher) networking.
Or you run in one machine where you run 3 virutal machines .Performance will be lower, but that is not the point. you want to learn oracle RAC.
Note that this is only usefule for non commeercial LEARNING oracle RAC where you get some kind of free (OTN?) license.
If you want to run oracle RAC in a business case take a good look at the oracle license in combination with virtualisation, because it might cost you (A LOT!!) more than you want. If you run RAC for the performance you might want to cut out virtualisation completely, because bare hardware is faster that virtulised hardware (!)
Also take a good look at the cost of a oracle RAC license, because you might want to consider other options to get high availability.
I am also considering to build a virtualization environment at home using commodity hardware and free (as in beer) software. I am considering 2 (or more) virtualization machines (1 x quadcore, 16 GB ram, gigabit lan) and 2 mirrored iscsi hosts (1 x quadcore, 8 GB ram, 5 x 1,5 TB SATA, Solaris+ZFS, gigabit lan). However, I cannot find any way to get the High Availability and Live Migration features without paying horrible license costs.
Is there any way to get these features for free?
Go visit sites like UltimateWhiteBox.com
My personal setup is two systems with Q6600 CPUs, 8 GB memory, intel 1000GT nic. I use a NAS box with an NFS share for shared storage in between them so I can do VMotion.
The big caveat I've found is that most desktop boards embedded eth won't work.
Use a lower end or cheap video card. I picked up some 128meg ATIs for $16 on ebay.
The CPU should support VT extensions and have plenty of RAM. If you figure 1-2 GB of RAM for each VM running concurrently, you should have a good estimate of what you need. A good OS platform to use is OpenSolaris because it's open source. In addition, it supports Xen, Zones, and other virtualization approaches. With OpenSolaris's Crossbow (network virtualization) you can not only have a testbed for VMs, but you can stress-test your VMs under different network characteristics (long latencies, etc.). Throw in Dtrace to track down issues and you have a real winner.
Running 4+ vm on a single machine, doesn't that limit your bandwidth? I always wondered why this is overlooked. Maybe its enough, well i'm not sure, so i'm asking you guys.
I recently bought a Dell 2950 III w/32 GB memory, dual quad-core processors, and (2) 1 GB SAS drives for about $4k brand new from Dell. I am running the Datacenter edition of Windows Server 2008 x64, however, you could also run the free edition of Hyper-V. The box supports x64 clients, and is very fast and stable.
I would suggest the tower model as the fans in the rack model are very loud. I would also suggest the Dell remote management card for an additional $250. It allows you to do just about anything remotely, even loading a new host OS w/virtual ISOs.
-G
Here's some research I did quite recently (less than a month ago). Prices should still be about the same.
Bare minimum:
Dell PowerEdge T100 â" GBP 279
Quad Core Intel Xeon X3220 2.40Ghz
http://configure.euro.dell.com/dellstore/config.aspx?c=uk&cs=ukbsdt1&kc=305&l=en&oc=PE1T1001&s=bsd&sbc=%20server-poweredge-t100
4x 2GB DDR2 RAM â" GBP 82.76
http://www.crucial.com/uk/store/partspecs.aspx?IMODULE=CT25664AA667
2x 1TB 32MB Cache 7200RPM HD â" GBP 163.30
http://www.novatech.co.uk/novatech/specpage.html?WD-10EADS
Total: GBP 525.06
Medium setup:
Dell PowerEdge T300 - GBP 569
Quad Core Intel® Xeon® X3363, 2.83GHz
http://configure.euro.dell.com/dellstore/config.aspx?b=&c=uk&cs=ukbsdt1&kc=N4XT3001&l=en&oc=SV1T300&rbc=SV1T300&s=bsd
6x 2GB DDR2 RAM - GBP 124.14
http://www.crucial.com/uk/store/partspecs.aspx?IMODULE=CT25664AA667
4x 1TB 32MB Cache 7200RPM HD â" GBP 326.6
http://www.novatech.co.uk/novatech/specpage.html?WD-10EADS
Total: GBP 1019.74
Over the top:
Dell PowerEdge 1900 â" GBP 1359
2x Quad Core Intel® Xeon® E5345, 2x4MB Cache, 2.33GHz
http://configure.euro.dell.com/dellstore/config.aspx?b=&c=uk&cs=ukbsdt1&kc=N4XM2301&l=en&oc=SV11901&rbc=SV11901&s=bsd
8x 2GB DDR2 RAM â" GBP 165.52
http://www.crucial.com/uk/store/partspecs.aspx?IMODULE=CT25664AA667
6x 1TB 32MB Cache 7200RPM HD â" GBP 589.9
http://www.novatech.co.uk/novatech/specpage.html?WD-10EADS
Total: GBP 2114.42
I have a Pogo Linux workstation. (Quiet!) with a dual core chip and 8 GB ram.
I run VirtualBox with a copy of WinXP for those applications that there isn't a good linux implementation
for. (Sorry -- Gimp != Photoshop Glom != access) It's allocated 1.5 GB ram, and mounts my home directory from the real machine via Samba. I get acceptable performance on everything I do on it.
For test bed type applications, I think that RAM is your biggest consideration. You don't want the VM to swap to
virtual memory. (VMVM? VM2?)
Third Career: Tree Farmer Second Career: Computer Geek First Career: Teacher, Outdoor Instructor, Photographer.
If you want High Availability for free, you will have to look into shared nothing architectures. You have open source solutions such as Sequoia (http://sequoia.continuent.org) that are database neutral that can do the job. Live migration is doable with shared nothing but it will take longer (basically depends on the size of your VM size).
Basically, as long as each virtual node isn't doing any WORK, you don't need any special hardware. And even if they are doing some work, but just not a lot. We have 5 Linux Xen VMs in production on a 1600Mhz Celeron with 768MB of RAM, works fine, no problems.
The CPU is almost irrelevant - you'll need whatever CPU you'd need to do all the things you're doing, plus some overhead, but it's not like it falls apart.
RAM is the only critical thing. You need at least 96 MB for the host and 24MB for each additional live Xen VM, as I recall (That's probably not precisely right) But you'll naturally be swapping a ton if you do that. A more reasonable VM has 128M - 256MB of RAM itself, so you need that for each active VM. But again, that's only for each one running at a time.
Or if you are going to swap a bunch, get better disks :)
In any case, I definitely wouldn't climb the price curve of equipment to do this; don't buy anything on the bleeding edge - look at arstechnica and just max the RAM on a value box - or maybe upgrade the MB to something that takes more RAM.
Used, commodity computer equipment is usually not price effective compared to the cheap end of what's still available new. But pay attention to the price point where it's cheaper to get (and power, while they're on) TWO value boxes than to pump up the one box you've been thinking of higher.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
Why even buy a computer? Upgrade your machine to 4GB or so and download VirtualBox. It runs on pretty much every OS out there, and it'll run almost anything as a virtual machine too.
If you find it's a bit slow, buy yourself a faster machine, or add more memory. That's pretty much all you need.
I use a Gigabyte EP45 with 8GB of RAM and 4 500 GB disk drives in a RAID10 configuration with a 3.2 GHz Intel Duo2.
I use Sun's Openbox (less issues then VMWare atm) and run 8 VMS at a time without much in the way of performance impact.
Ensure hardware virtualization is on leave IO ACPI virtualzation off. I have nested paging on also.
Each VM is allocated 256 MB of RAM for linux systems and the 2 M$ systems (XP and Vista) are allocated 1GB each.
PARENT OS (XP)
VM1 - Linux MYSQL Host
VM2 - Linux LAMP Host
VM3 - Linux REPORTS Server (Takes data and generates email from all systems)
VM4 - Linux Firewall\IDS (The main system uses wireless with the VMS allocated the two physical NIC ports on the Mobo. This is the default gateway for all systems.)
VM5 - Gentoo SSH DISTCC CCACHE Buddy (My friends use this via SSH to help compile Gentoo Systems)
VM6 - Windows XP VPN Client Container (Strictly Business Locked Down VPN host for remoting into work)
VM7 - Vista Media Host (Uses a mapped drive to the Parent system's MEDIA folder)
VM8 - Linux Game Hoster (Hosts versious game servers via private VPN for easy netplay)
The cool thing is EVER WITH ALL THAT RUNNING I can still play warhammer online with 0 lag or play Fallout with no lag.
There will be an occasional 2 second lag\pause on the system for some mysterious reason but only if the Vista VM is running.
This is not acceptable for production purposes but for home use it works great. For production I would simply change the parent to a lighter weight parent.
-=[ Who Is John Galt? ]=-
I would go with the Intel 920 four core+HT CPU, fully populated RAM (12GB), and at least four TB drives using software RAID (my WD run 10 deg cooler than my Seagates). Run Fedora F10 as the host OS, and knock youself out with VM. Linux handles both multiple cores and HT well, seems to make good allocation choices based on understanding cache.
I write this on a VM desktop machine which moves between several hosts similar to described, or a notebook running an AMD dual-core. All run FC9 or FC10 as host, and I have two desktops for various use, XP, Win7, and my HTTP, DNS, DHCP, and mail servers (using CentOS) available as backups if the dedicated hosts should fail.
Cost of a box like that should be
You can use heartbeat + Xen over 2 boxes to get HA, so long as you have shared storage. There are some notes about how we did Xen live migration on our "perspex box" in my comment here: http://ask.slashdot.org/comments.pl?sid=1171301&cid=27295879 Citrix Xenserver now offers "Live Motion" for free: http://www.citrix.com/English/ps2/products/feature.asp?contentID=1686939
VMs aren't good for much. I found my computer usage is more intense than I realized, and that's where VMs fall down. They're good for test environments, throwaway software installations, and light duty servers. I'd rather have real hardware.
I/O is poor. It takes about 3x as long to do a compile in VMware than directly on that same machine. Downloading an ISO kills performance on pretty much everything as whichever VM is doing the download thrashes the hard drive mightily to save all that data. Thinking of using VMs to learn about clustered file systems or database replication or anything else meant for multiple real servers? Painful.
VMs aren't great with hardware. Can't have both Xen and the Nvidia proprietary drivers. Graphics intensive software is often unusable in a VM. Playing a video in a browser in a VM is a performance killer.
Intellectual Property is a monopolistic, selfish, and defective concept. It is "tyranny over the mind of man"
Any computer with at least two cores or two CPU chips. I have two systems one is a dual xeon powoered HP running Linux with VMware's "server" and the other a dual core Apple iMac running Mac OS X with VMware's "fusion. They both work well. But in any case max out the RAM.
In fact if you are buying a computer just for this purpose, Buy yhr one that has the largest maximum RAM. You want at least 4GB. Also get the best disk system you can afford.
Processor speed is not really what you need but you do want a modern Intel Core2 and loads of RAM
The CPU is almost irrelevant - you'll need whatever CPU you'd need to do all the things you're doing, plus some overhead, but it's not like it falls apart.
I disagree. The CPU is the most relevant element of the whole system for someone who wants to play around with actualization.
If you are buying a system to play with virtualization you really want Intel VT or AMD-V support. Along with execute disable (Intel XD/AMD NX) and Hardware Data Execution Protection (DEP), and other features.
VMWare server will run on almost anything, but if you want to play with Citrix Xenserver, Solaris Virtual Box or Microsoft Hyper-V server too, you really benefit from the extra features, and in some cases you require them.
I'd look at a Xeon 3200 (~$250) for the CPU.
You could also get away with the right Core 2 Quad. A Q9100 or better supports everything... Intel VT, XD, and TXT (virtualization, execute disable, and trusted execution). The old Q6600 supports everything but TXT. (But afaik, you don't need TXT for anything.)
That said, avoid the Q8200, it doesn't support VT.
The current i7's should all be good.
I really don't know my AMD's well enough to advise on them for virtual server roles.
In any case, research the cpu carefully, to give yourself as many options as possible, and then match it to a motherboard that will support the features as well. Some don't.
Any reason you want to go down the (multi)path of OCFS for a 3-node Oracle RAC? Why not just use RAW shared disks? You aren't going to see the difference in a virualization environment anyhow - your disk bottleneck will be the shared access in the Virtual disk controller. OCFS is really only useful when using real HBA's for FibreChannel or iSCSI.
And don't forget to select "Enterprise Edition" when doing your Oracle install. Standard has a 2-node limit.
This isn't an a shameless plug or anything, I have no affiliation with these guys other than yes, I did buy a cheapie server from them and it works great! I use it for game servers like L4D & 2142 & all the old classics (red faction, quake series, etc.) The main thing to watch for in my opinion is the Hard Drive you plan on using with it. These are IDE only & the $99 one only accepts 1/4" single-platter drives or maybe a laptop drive. but it's perfect for me :) I popped a 500GB IDE drive in the dual 3.06 Xeon / 2GB Ram one.
dubba dubba dubba dot weirdstuff.com/cgi-bin/category/10174
More is Better.
In production I have found that RAM is more important then CPU, if you have to make choices.
The *last* thing you want is swapping.
---- Booth was a patriot ----
If your intent is to educate yourself on the various configuration options, then the workload of the "applications" (Oracle DBMS, etc) is modest if it's idle. I have configured a Oracle RAC two node cluster using a pair of basic notebooks and a WD MyBook World Ed NAS, which is pretty weak hardware and low cost. An upgrade would be a couple of PC's with CPU's that support virtualization, such as KVM, like Intels VT or AMDS AMD-V class and a NAS. You could even use FreeNAS, which I recommend regardless, for shared storage. VM's are generally constrained by memory. There is also the option of renting cloud resources, ala Amazon EC2 or Elastra et al, which is probably very cheap considering the light workload and data required. As with most things, it depends on what you want to accomplish. Consider the minimum requirements of the packages or applications and factor for virtualization overhead. Obviously, performance tests require much more resources.
In my testing, I found my cheapo Dell had two bottle-necks: Ram and disk. 1GB/VM, while not required, made playtime considerably more pleasant. I only have 3GB. Also, one VM doing heavy disk I/O pauses the machine. I am thinking about adding a software-striped pair of disks inside for the VMs and backing the VM images up regularly.
-Adam
Computer Science is all about trying to find the right wrench to bang in the right screw. -T.Cumbo?
I had the exact same requirement as you i.e to learn Oracle RAC with cheap commodity hardware in a home environment. I went with an AMD 780G based mobo with a 2.6Ghz AMD Dual Core. I packed with 8GB DDR2-800 which costs less than $100 nowadays. 1TB and 1.5TB disks are also reasonably cheap from newegg. Use Linux software RAID and LVM to speed up the disks..
I chose the recently free'd Citrix XenServer and I am mightily impressed. It just wipes VMware off the floor when it comes to I/O performance due to para-virtualized drivers. The NIC drivers are also fast and efficient. I solved the shared disk problem by making one VM an iSCSI target. I can present as many LUN's as I want to other VM's now. since all traffic goes over the internal NIC, performance is excellent.
The XenServer product also consumes far fewer system resources than VMWare and the quality and polish is really good. not to mention the excellent documentation that is available from Citrix
You can save a lot of money by choosing AMD based chipsets and a cheap Phenom or Athlon X2. CPU clock speed may not buy you as much as fast disks and RAM.
I think you define "play" differently than I do.
Any modern PC will allow you to "play" with virtualization. What you're describing (in my opinion, of course) are things you want if you're going to do serious work with virtualization.
At work we're deploying Windows Server 2003 + 2 VMs on VMware Server to Pentium 4 and older Core 2 systems with 1G of memory. In significant volume. I think we have around 2,000 out so far and we're still rolling.
The preferred solution is to not have a problem.
OK, here I go. I work for VMware so I need to experiment with all kind of VMs all the time. I just put together a system *just* for what you are describing here. I assembled an X58 Biostar motherboard with an Intel i7 920 processor + 6 GB or RAM and a nice 1 TB SATA drive. I spent about 900K with Tigerdirect. The new vSphere leverages every bit of hardware acceleration from the Nehalem cores.
I installed the Release Candidate code of VMware vSphere 4.0 (the new name for ESX) and I am litteraly blown away by the performance I'm getting from this box. I have [no kidding] 16 VMs running and I'm not even hitting 80% CPU utilization. VMware's transparent memory sharing plus VMI paravirtualization for Ubuntu has my system running like a rocket with a minimal use of resources. I'm even running Asterisk for first time on a VM with no degradation whatsoever. I have (among many others) an Opensolaris fileserver with NFS and ZFS connected to one of the native NIC ports via the new Pass-through capabilities of vSphere and it just rocks.
I'll install RAC next week to see how it runs. It works well on regular ESXi 3.5 so I'm sure this will be even better.
ESXi is free. The new version will also be free. It's a dedicated, single purpose hypervisor. I subscribe to the philosophy of "use the best tool for the job", so give it a try if you have access to the beta. Otherwise wait a little bit and you'll get the GA product...
start with 3Ghz dual core (like e8400/e8500),
8GB ddr2-800 of ram,
and raid-0 from any drives you need (320gb already have enough speed)
if you still want to spend more:
add memory;
then add cores;
and only then add more drives.
config decribed above (3ghz, 8gb, 2*500Gb in raid-0) succesfully runs 5 winservers in virtual (wsus, sql, sharepoint, wds, dc) and also run enterprise app and serves as fileserver, and do it all pretty fine.
Any modern PC will allow you to "play" with virtualization. What you're describing (in my opinion, of course) are things you want if you're going to do serious work with virtualization.
If your stated goal is to experiment with virtualization, and you are going to buy hardware for the purpose, it would be absurd to select hardware that doesn't support most of the available solutions.
If I wanted to buy some hardware to play with "Linux", would you recommend a PS3?
I mean sure, you can run linux on it, and you could certainly play with Linux on a PS3.
But surely you agree it would make a LOT more sense to buy an Intel/AMD based PC that will run all the major and minor distros.
Similiarly, if you want to play with virtualization, you should probably choose a cpu with hardware vm support, to give you the option to "play" with Xen, VMware, VirtualBox, etc... all with unmodified guests... if you pick a celeron... you get VMware. The others either don't work at all, or have all kinds of restrictions.
So off the top of my head:
- For HyperV & Xen you need VT-enabled hardware, latest Intel & AMD only
- Go for more than 4gb of RAM, part of this is about pushing a few limits, and you'll want to run a few VM's at the same time
- Get a hardware RAID card - I think the LSI MegaRAID SATA-150 is about as standardized and supported as it gets - ESX/ESXi work fine, and Xen and HyperV should all work
- Use laptop drives for your RAID set - they fit much nicer into cases (4x320's gives you 900gb)
- Go for 2-3 network adapters - Intel or Broadcom only (10/100 is fine)
Pick a base OS and run VMWare Server - trust me on this. Instead of reinstalling the OS off of cd, you're mounting the cd in the VM and doing your installs without the legwork. You can also download pre-built demo appliances so you spend your time dealing with the product (Oracle, IBM, etc), rather than tweaking out your CentOS config.
If you're going to stick with VMWare ESX/ESXi you can get any server hardware from the last 3 years. Sun x4100's, Dell 1750/2650's, HP DL380/385's all work fine, though RAM is mostly still expensive :-p.
Xeon or AMD equivalent processors are best because they have a lot of second and third level cache. Two gigabyte network card are required, but 4 are suggested. One network interface for traffic and the other one for migration or administation. Hard discs faster as you can permit (money question): SCSI, RAID configuration. Memory depends on how many virtual machine you are going to handle (operating system, jobs to carry out, ...). You have to remember you have to attribute some memory to host operating system.
If you plan to use VMware ESX (or free version ESXi), you have to chose a compatible server, because drivers cannot be available for a generic own build server.