Ask Slashdot: Little Boxes Around the Edge of the Data Center?

First time accepted submitter spaceyhackerlady writes "We're looking at some new development, and a big question mark is the little boxes around the edge of the data center — the NTP servers, the monitoring boxes, the stuff that supports and interfaces with the Big Iron that does the real work. The last time I visited a hosting farm I saw shelves of Mac Minis, but that was five years ago. What do people like now for their little support boxes?"

Comments

Little boxes

[Hatta]

I make them with ticky tack.

Re:Little boxes

[don.g]

You do realise no one outside of New Zealand will get that joke...

Re:Little boxes

[SDrag0n]

You do realize that everyone who watched weeds will be humming along right?

Re:Little boxes

[msauve]

Are you sure about that?

Re:Little boxes

[Anonymous Coward]

or watchers of Weeds on Showtime

Re:Little boxes

[TheGratefulNet]

network boxes,
made in china,
network boxes that go sparky-spark
network boxes
exploding boxes
dangerous boxes, all the same.

Re:Little boxes

[JazzHarper]

or old farts who remember Pete Seeger.

Re:Little boxes

[don.g]

Looks like I'm wrong -- see what happens when you trust your own memory over Google/Wikipedia? Someone clearly lied to me in my youth when they told me it was referring to our town :-(

Re:Little boxes

[JoeCommodore]

Iv'e seen Windows 8, I know what ticky tack little boxes look like.

Re:Little boxes

[Anonymous Coward]

Only 32 and got the reference!

Re:Little boxes

[connor4312]

As an American school student, I can say that it has often been shown as part of middle-school-level history class. Maybe there's some note about it in the Teacher's Edition of the textbook or something.

Re:Little boxes

[Em Adespoton]

There are white ones
And more white ones
And they all have those blinky lights
and they're all made out of ticky tacky
and they all fail just the same.

Re:Little boxes

[ThunderBird89]

Little boxes?
"The little boxes will make you angry!"

Re:Little boxes

[Anonymous Coward]

Or old farts who remember Malvina Reynolds.

Re:Little boxes

[renfrow]

Hehehe, why is I don't have mod points today?

Re:Little boxes

[Anonymous Coward]

or old farts who remember Pete Seeger.

Really old farts remember this song from the LP "Another Country Heard From", singer and songwriter Malvina Reynolds.

Re:Little boxes

[Hatta]

Do you have to be an old fart to remember Pete Seeger? That was my reference. I had forgotten that Weeds used it as its theme song, I'm not a fan.

Re:Little boxes

[WillKemp]

I'm in Australia and i got it!

Re:Little boxes

[Jeremiah Cornelius]

The song was written 'bout Daly City - a Philippine colony which forms the buffer-zone between San Francisco and the United States of America.

Re:Little boxes

[Darinbob]

Used to sing it at camp.

Re:Little boxes

[Anonymous Coward]

Little boxes?
"The little boxes will make you angry!"

The little boxes wouldn't like me when I'm angry.

Re:Little boxes

[Hes Nikke]

You do realize that song is significantly older than weeds right?

Re:Little boxes

[Hes Nikke]

Is that why my family always sang that song when we drove past?

Re:Little boxes

[mikael_j]

But for what one might call the current generation it was popularized by the show.

Re:Little boxes

[WGFCrafty]

Uh, and a bit more substantially.... everyone who listened to music in the 1950s.

Re:Little boxes

[sa1lnr]

I'm Scottish and I get it. :)

Re:Little boxes

[adolf]

The version played on Weeds is significantly better* than the original recording of the same, which suffers from a whole lot of scratchy and not-so-good:

On Weeds, it sounds like something that was carefully recorded quite recently. The original...not so much, but it's a lovely song just the same.

I genuinely thought that the opening from Weeds was a modern recording until I went looking and found that it was relatively ancient, but just recently-polished. I imagine that lots of folks might be able to be similarly-confused just as easily as I was.

*: I say this with great reservation because I find that nearly all attempts at remastering old music are ripe with failure on many levels. But, IMHO, whoever did the work for this track knew what they were doing, and was allowed to spend the time to get it done well. I wish I knew who was responsible.

Re:Little boxes

[ThunderBird89]

Wrong reference there. Eh, does nobody watch Doctor Who?

Re:Little boxes

[kcitren]

Every episode had a different rendition of the song.

Re:Little boxes

[Anonymous Coward]

Wrong reference there.

Deliberate cross-pollination of references. Eh, I thought it was funny.

Eh, does nobody watch Doctor Who?

Who'd wanna watch a show about a boy and his box, off to see the universe?

Re:Little boxes

[hackula]

Why, are you my mummy?

Re:Little boxes

[Hes Nikke]

Not in season 1.

Re:Little boxes

[Em Adespoton]

For those still missing the reference:

http://ingeb.org/songs/littlebo.html

A classic from the 60's.

Re:Little boxes

[wwphx]

or old farts who remember Pete Seeger.

Who was on The Colbert Report a couple of months ago, not that this statement doesn't exclude me from the realm of old farts.

VMs

[Anonymous Coward]

put them in VMs!

Re:VMs

[Art Challenor]

put them in VMs!

Great Plan! If all your servers are virtual then you don't have to worry about diesel fuel when there's a hurricane!

Re:VMs

[Nutria]

Call me old school, but Unix/Linux are multi-tasking. Why not just run multiple services on one OS directly on the metal?

Re:VMs

[TwineLogic]

Indeed. For example, does it not seem dumb to virtualize an NTP server?

Re:VMs

[Anonymous Coward]

There are some tasks that you want to run on separate servers. If for no other reason than that you want the services to be up even while the main server is rebooting.

Re:VMs

[nurb432]

Then you dont fully understand how a vmware farm works.

Re:VMs

[nurb432]

Not in the least is it dumb. If you manage your systems properly and their boot order, its a non issue.

Re:VMs

[msauve]

Good luck putting ntp on a VM (a host, maybe, but interrupt latency will kill you).

Re:VMs

[LordLimecat]

VMWare (and, I understand, all of their competitors) have this notion of clustering where one "main server" can be rebooting without causing any of their guests to suffer interruption.

You can stuff those services onto a separate guest, but as long as things are laid out properly and you dont have some dependency for your virtual infrastructure on that guest, you can virtualize it just fine. You can even virtualize the vCenter server, though it makes bringing the virtual infrastructure back up from scratch a little bit more painful (you have to manage the servers individually until vCenter is back up).

Re:VMs

[Anonymous Coward]

put them in VMs!

Great Plan! If all your servers are virtual then you don't have to worry about diesel fuel when there's a hurricane!

Unless you run out of virtual Diesel.

Re:VMs

[Anonymous Coward]

Yes. It is a stupid idea to virtualize an ntp server or any host that needs to keep accurate time. Once the vm server host gets bogged down, you will see time go way off, very quickly.

You can use containers for such things, and gain much of the isolation without the drawbacks of full virtualization.

At work we use containers for this sort of thing, running on a two node cluster for redundancy.

At home, I'm using a single very low power arm board to run everything that is always-on (ntpd (board has an rtc), dns, webserver, nfs, kdc, AP (usb radio), iptables for fw, etc.). My work would never go for it, but a few arm boards could run all the small stuff and consume just a few watts of power.

Re:VMs

[Anonymous Coward]

There are issues with running NTP inside of a VM, but nothing that's impossible to overcome. The NTP page lists gotchas and other info about each virtualization option. The flexibility you get from a VM strategy could very well be worth the minimal amount of adapting to running the services as a guest.

OTOH, it would really suck to have an NTP vulnerability combined with a privilege escalation compromise some other part of your infrastructure. Running it in a VM makes that a lot less likely and allows you to run, say, an SSH gateway on the same physical hardware.

Also, for most applications, having an extremely accurate time isn't that important. It's much more important to have a extremely consistent time so that all the servers are synchronized.

Re:VMs

[mlts]

There are good reasons to separate functions. Mainly security. That way, if someone hacks the NTP server, they don't get control of DNS, nor do they get control of the corporate NNTP server, or other functions.

The ideal would be to run those functions as VMs on a host filesystem that uses deduplication. That way, the overhead of multiple operating systems is minimized.

What would be nice would be an ARM server platform, combined with ZFS for storing the VM disk images, and a well thought out (and hardened) hypervisor. The result would be a server that can take one rack unit, but can handle all the small stuff (DNS caching, NTP, etc.)

Re:VMs

[jrmiller]

Not really. NTP's such a lightweight service that it runs fine on a vm. As other posters have mentioned, you certainly don't want to use the system clock as your time source, but you shouldn't do that anyway. Hopefully you're syncing with an upstream provider that syncs from a non-computer-based source. See http://tycho.usno.navy.mil/ntp.html for a good sync source (among many others). We've successfully virtualized NTP servers serving a 6000-person university.

Re:VMs

[Anonymous Coward]

Uhhh. because the "little boxes" and individual servers run on unicorn farts and angel tears?

Re:VMs

[jellomizer]

It comes down to an issue of scalability.
With the multiple services, on one OS. Means if some of the services gets popular, and needs more power then the server can handle. You will need to decommission and reinstall and configure the service onto an other server... And in the mean time your other services are often getting performance hindered. Virtualizing means if you need to move it from one box to an other it is a file copy away. vs. reconfiguring and testing.

Re:VMs

[dbIII]

IMHO, after coming into a workplace that initially had a very strict boot order, if you manage your systems properly you should be able to boot them in any order you want without having stuff hanging on dependencies. That's not tricky anymore with various ways to do automounting when the volume is needed instead it having to be available at boot.

Re:VMs

[Ost99]

Why dedup? Those VMs should not require more than 500MB-2GB each.
Deduplication (inline) only adds complexity and sources of latency you don't need or want.
Any small pizza box with 2x146GB drives (or 2x256GB ssd) in RAID1 should be able to handle any number of virtualized small utility guests without any deduplication.

Re:VMs

[deniable]

Virtual isn't a problem but Vmotion between hosts that don't have accurate hardware clocks sucks big time.

Re:VMs

[marcosdumay]

Well, one of the reasos is that some services get hold of port 80 (or, a few times other ports), and don't want to share it. With virtualization you can share resources with those too... But yes, those services are a minority, and probably won't need a lot of resources...

Another reason is that you may want to give different people permission to administrate different machines... But again, except for companies that sell hosting, that's an exception.

A third reason is that you may want to replicate your environment for backups and testing... Except that you don't need a VM to do that on Linux. You just copy the files, add two devices to /dev and run the bootloader again. It's easier than backing-up a VM in Windows.

And I've never heard about any other reason for virtualization. I can't also think about any other. I'm lost about why sudenly so much people wants it so badly... Ok, all datacenters added specialized machines for decades because of those first two reasons I gave you above, and get some benefit virtualizing them... But the core of a datacenter (the main databases, web servers - the machies that actualy spend the day working) should run on the metal, and altought I've met several people that arguee otherwise, I've never heard any argument for virtualizing them that holds any water.

But now, I think, maybe the HA people should try to virtualize their clusters. They have a huge amount of redundancy, and consolidating several virtual machines in a single real one can help them reduce their costs. (Ok, if you are in doubt, no, I'm not THAT stupid, it's a joke.)

Re:VMs

[marcosdumay]

That's why you use clusters.

Now, why are you talking about that in a thread about virtualization?

Re:VMs

[Nutria]

I'm lost about why sudenly so much people wants it so badly... Ok, all datacenters added specialized machines for decades because of those first two reasons I gave you above,

I thought it was because young geeks and proto-managers grew up with the Curse Of Windows, where you had to run one service per machine, and then brought that flawed mindset into the Linux world.

Re:VMs

[Anonymous Coward]

The problem is not the vmotion, it is that your hardware clocks are off. Oddly, we have no problems with ESX servers with hardware clocks that are off. The hardware clock on the physical ESX server only come into play if you are using vmtools and have the option to set the system clock from the hardware clock on a repeated basis. I'm not sure why you would do that. Every system can and does keep its own time independent of the hardware clock except at bootup. If your system time is constantly being set set from the hardware clock for some reason, then synchronize your hardware clocks with NTP? That's not hard at all.

I suggest reading this
http://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf

Re:VMs

[Anonymous Coward]

After reading through this VMs sub thread, I've come to the conclusion that /. has many programmers and coders that really do not have a clue what Vmware can and can not do or have really old information or used ESX back in the 2.X days or they only messed with it in their spare time and saw a small subset of the benefits or it was grossly misconfigured. Stick to programming and let the System Administrator/VM guy/ or whoever runs the virtualization run it.

The comments like "I'd never virtualize system critical stuff" or one failure brings everything down etc are comical. Those are comments from people that have never used it in an enterprise environment. There is almost no enterprise software made that is not certified and fully supported to run in a VM environment. If it is not VM ready, it is the exception or really old. Many large companies are 100% virtual.

Re:VMs

[ls671]

Accurate ntp installs indeed use the system clock and rely on it. They just change the frequency on how often OS time is ticked based on information received through the network from other ntp servers.

Ntp in a VM could work under 2 conditions:
1) The VM has raw access to the system clock or a pretty good abstraction of it. VMware has had problems with that.
2) The ntp guest VM process runs at nice -10 to nice -15, and ionice RT while at it ;-)

Nevertheless, no ntp will run accurately with a poor physical system clock, more commonly called an oscillator.

http://img.tfd.com/cde/CLOCK.GIF

http://www.thefreedictionary.com/system+clock

Re:VMs

[ls671]

I forgot: play with adjtimex to see how ntpd plays with the frequency.

man adjtimex

Re:VMs

[Anonymous Coward]

dedupe does NOT scale when performance (even low end) is a concern.

Re:VMs

[ls671]

Jails work fine too, better IMHO.

http://slashdot.org/comments.pl?sid=3224997&cid=41850103

http://slashdot.org/comments.pl?sid=3224997&cid=41850239

Re:VMs

[Anonymous Coward]

Well modern hypervisors like VMWare allow you to prioritize virtual machines so that they get a higher share of scheduling time in an overcomittment scenario. Assign your ntpd server a high priority so that it doesn't have to wait in a long queue to get run time.

Yes running time-sensitive stuff on a hypervisor is tricky but not at all impossible. It's not stupid unless you don't know what you're doing.

Re:VMs

[Z00L00K]

Or use some old servers - many times an old PIII with Linux is good enough for some low demanding work like being an NTP server.

Re:VMs

[profplump]

Or, as I suggested before, you could simply not use the CPU time source as your local NTP clock. There are a whole slew of other oscillators available that work just fine in a VM, for both NTP and kernel timekeeping. Not to mention VMs that will pass through access to a high-precision timing source even if the virtualized tick clock is unreliable. If you need high accuracy (and not just good consistency) you need to know what you're doing, but that's the case with or without a VM -- there really is no good reason an NTP server can't run in a VM even given an unpredictable CPU tick counter.

Re:VMs

[Anonymous Coward]

you seem to assume, the seperation by VMs on the same host is stronger than the seperation by unix users. But i would suggest, unix users is more secure, because there are less points of failure involved.

Re:VMs

[Anonymous Coward]

I've never heard about any other reason for virtualization. I can't also think about any other.

Really? Then you lack serious knowledge of virtualization and possibly also imagination.

1. Load balancing. You can manage your physical resources better. In a cluster of virtualization hosts, the virtualization platform can re-distribute virtual machines to keep the workload balanced.
2. High availability. If your single All Singing, All Dancing Uberserver breaks, all the services running it break. If a VM host breaks, the VMs can be migrated or restarted on a different host automatically. Some platforms such as VMWare even offer Fault Tolerance, where it keeps two copies of the virtual machine running at the same time on different hosts, and keeps them in lock step.
3. Utilisation. With things like page de-duplication I can over-subscribe the virtualization hosts and use more memory than is physically available. With VM scheduling I can squeeze more CPU time out the virtualization hosts, too.
4. Partitioning. You're quick to dismiss it with a wave of a hand, but if someone compromises a virtual machine running Apache, they don't automatically gain access to all the other services running in the other virtual machines. The same applies in situations where you need to keep sets of data separate: customer A is trusting you with their data and doesn't want it stored alongside data from customer B: there are lots of situations where that applies.
5. Point-in-time scaling. It's much quicker to spin up a couple of new virtual machines if I need to add capacity to a certain system to handle an unexpected peak workload (say, my inbound email server comes under attack) than it is to provision new physical servers. Who has physical hardware just lying around waiting to be used anyway? Once the peak has passed I can turn the virtual machines off to save resources.

There's other stuff like Storage DRS, simplified network multipathing (your VMs just see a single NIC!), simplified storage multipathing (your VMs just see a SCSI controller!), roll based access controls, the ability to mix multiple operating systems without needing dedicated hardware for each one etc. etc.

Stop dismissing useful technology just because you haven't found the need for it.

Re:VMs

[Anonymous Coward]

Well... Hardware independence, high availability, easier upgrades and refresh cycles, load balancing, OS isolation, fault tolerance, easier migration and disaster recovery, snapshots... There's more to virtualization than consolidation. Running directly on the metal is increasingly less of an option.

Re:VMs

[pixr99]

But now, I think, maybe the HA people should try to virtualize their clusters. They have a huge amount of redundancy, and consolidating several virtual machines in a single real one can help them reduce their costs. (Ok, if you are in doubt, no, I'm not THAT stupid, it's a joke.)

There's nothing wrong with that at all provided you tell your environment that cluster members cannot be moved onto the same hardware. Try it. You'll like it!

Re:VMs

[Anonymous Coward]

I work in local plant-level IT support. My company is world-wide and made up of bits of other companies my company has smashed and devoured.

One of the four plants I support was owned by a company that employed plant-level full IT support. Everything was done at the plant. Although no one will talk about it, it seems that everytime these guys went to the john they came out with a server. Consequently there are twenty two servers at a plant that supports few than two hundred people and makes less than five million per year in profits. At one point there was a server to test if a different test server could be used as a fail-over for another test server, but in a different environment.

My company, when they bought this outhouse of a plant, decided the best way to handle it was VMware. Except they didn't do a full inventory and process check first, they just counted servers and bought licenses. Now, three years later, I am fighting the plant management (who want to keep everything from 'the old days') corporate server support (who can't be bothered to reduce the number of servers) my manager (who is from one of the crushed companies and too shit-scared to say anything) and my fellow IT support staff (who are also from the same crushed company and love their boss) over reducing support time and costs for all of these stupid little VMware servers.

DO NOT USE VMware. You will be sorry later. VMware is a virus and, once you contract it, you can't get rid of it and you will be tuck with the servers forever.

Re:VMs

[Anonymous Coward]

People still use nntp???

Re:VMs

[Anonymous Coward]

Wait, you're telling us that your management is inept and couldn't deploy VMWare correctly, then you tell us it's VMWare's fault? How?

Re:VMs

[JMandingo]

Because VM images are easy to duplicate and move. Easy to scale up, easy to back up, easy to swap failing hardware. You use multiple VMs and not just one box running everything so that you can easily duplicate or move one service (or group of related services) without touching others.

Re:VMs

[Skweetis]

A friend works for a large server vendor. He told me a while ago that they essentially only provide and support the management tools for their servers as VCenter plugins anymore; because, almost without exception, their customers only use them as ESXi hosts.

Re:VMs

[fifedrum]

there's a lot of "dumb" in this group of answers all throughout. people advocating all sorts of rediculous things.

Re:VMs

[Anonymous Coward]

I'd be leery of that:

To get from one VM to another requires full root/ring 0 access in the client VM, then it takes knowing some weakness in the hypervisor, and that being a weakness that is actually exploitable as opposed to just a crash. After that, it is trying to find a way to get into a VM. This is very difficult, and there have been no recorded incidents of say, a VMWare Workstation VM getting compromised causing the host to get hacked.

To get from one UNIX user to another requires a single hole in a SUID binary.

A hypervisor with a hardened attack surface has far fewer points of attack than a full-featured OS. This is mitigated somewhat by AppArmor or SELinux, but is still present.

Re:VMs

[Nutria]

But you also have to patch multiple (sometimes many) OS instances. My thinking is, "What happens more often, Patch Tuesday or moving a service from one box to another?"

Re:VMs

[cornjones]

who hasn't automated their patching yet? being able to move OS's to other hosts (or datacenters) plus the drastic decrease in time to provision and general hardware abstraction is invaluable.

Re:VMs

[marcosdumay]

Load balancing. ... In a cluster of virtualization hosts, the virtualization platform can re-distribute virtual machines to keep the workload balanced.

Yeah, you can do that in any kind of cluster. You don't need to add virtualization for that.

Utilisation. With things like page de-duplication I can over-subscribe the virtualization hosts and use more memory than is physically available.

It can improve your memory usage up to the point (for a complete and perfect optimization) you'll be if you just consolidated your real machines...

Partitioning. You're quick to dismiss it with a wave of a hand, but if someone compromises a virtual machine running Apache, they don't automatically gain access to all the other services running in the other virtual machines.

Yes, there is some gain in security. In practice, however, your datacenter has a completely open set of doors (it always has), and you are focusing in closing a window.

Point-in-time scaling. ... Who has physical hardware just lying around waiting to be used anyway? Once the peak has passed I can turn the virtual machines off to save resources.

You mean, you don't have extra resources available, but you can spin up a new virtual machine anyway, and it will be able to handle the extra demand?

Re:VMs

[Nutria]

general hardware abstraction is invaluable.

That would be useful...

Re:VMs

[afidel]

Yes, because there's never been an instance of two services on Linux/Unix needing mutually exclusive versions of a shared library....

Re:VMs

[Guspaz]

Dedupe also doesn't work well on VM disk images, because dedupe tends to be block-level, and sparse VM disk images just don't like that. Dedupe also significantly increases RAM requirements (deduplicating a single multi-terabyte hard disk can consume gigabytes of RAM). Ultimately, disk space is cheap, and it's not worth the performance and memory overhead just to save a few gigs of disk space.

Re:VMs

[Anonymous Coward]

You're missing most of his points. Virtualization makes a lot of stuff that you dismiss, much easier.

Re:VMs

[Anonymous Coward]

This isn't a problem in an enterprise. They use configuration management software that handles this for you.

Re:VMs

[ls671]

Not to mention VMs that will pass through access to a high-precision timing source even if the virtualized tick clock is unreliable.

That was my point 1, although I should instead have written:

"The VM has access to an hardware oscillator".

1) The VM has raw access to the system clock or a pretty good abstraction of it. VMware has had problems with that.

I do not care how it is abstracted or if it is called the system clock. I assume that it might be desirable in VM environments to keep both functionalities separated and that is the point you are making.

An average board with an slightly above average above oscillator work fine for me at +/- 5 ms because I am cheap. Search ntp forums for motherboards with reliable oscillators. I would like VMs to use my oscillator, I do not care if the VM views it as the system clock and or CPU time source or whatever one may call it ;-)

http://slashdot.org/comments.pl?sid=3224997&cid=41850103

Re:VMs

[ls671]

"The VM has access to an hardware oscillator".

And it can't miss a tick. By tick, I mean a relaxed version of it. Think about dialups where the tick occurs every second.

Simple on paper, harder to implement.

Re:VMs

[Anonymous Coward]

Yeah, you can do that in any kind of cluster.

A cluster of what, precisely? Bananas? I've never come across a non-VM "cluster" where I can say "Hey DNS server, stop running on machine A and start running on machine B!".

You mean, you don't have extra resources available, but you can spin up a new virtual machine anyway, and it will be able to handle the extra demand?

I refer you back to his point about over committing resources.

Re:VMs

[bingoUV]

Yeah, you can do that in any kind of cluster. You don't need to add virtualization for that.

How do you move an NTP server from one physical machine to another? Without stopping it in between.

It can improve your memory usage up to the point (for a complete and perfect optimization) you'll be if you just consolidated your real machines...

Agreed here. Most of the times, virtualization will only add to the memory requirements, especially in such small services as this ask slashdot is enquiring about. The guest OS will end up requiring the vast majority of memory. De-duplication merely helps in lowering that overhead.

Yes, there is some gain in security. In practice, however, your datacenter has a completely open set of doors (it always has), and you are focusing in closing a window.

I am curious about this. Does "your datacenter" mean any datacenter, or specifically the AC's datacenter? If any, please "walk into" the open door of slashdot datacenter and change the title to dotslash (long needed anyway) to demonstrate.

You mean, you don't have extra resources available, but you can spin up a new virtual machine anyway, and it will be able to handle the extra demand?

True, one needs to have hardware ready in the worst case. But in the event of a particular service starting requiring more resources, it is trivial with zero downtime to
1. Add a powerful server to the pool of virtualization servers
2. Live-Migrate the guest running that particular service to this powerful server.

You can even set this live migration to happen automatically. But even better, if the pool is hosting 20 services and one is requiring more resources, typically you can subtract resources from other services, at least temporarily until you go find the powerful server to replace it. With zero downtime.

Re:VMs

[wwphx]

Mine run on unicorn tears and angel farts: I virtualize on my i7 iMac.

Re:VMs

[Anonymous Coward]

You can do all of that with a couple of blades - Sparc or x86/x64 - Running Solaris 11 - with nonglobal zones - all isolated from each other, one os instance, hack one, don't even see the rest.
Can build entire virtual Datacenters in a box with Solaris 11, Crossbow, Comstar and Containers.

Re:VMs

[Anonymous Coward]

One can do similar with POWER7 machines using LPARs. Hack an IBM LPAR, you know it is a VM, you know the stats of the VM, so if it is uncapped, you can probably slurp up all avalable CPU swishing around that isn't assigned, but it would be extremely difficult to jump to another LPAR unless one hacks the target like a standalone machine, or if the HMC/SDMC is accessible/configured in some brain-dead manner.

POWER and SPARC are expensive though. If I were needing the CPU and hack resistance (both of those are far more secure than x86 at the CPU level), I'd go with them. Otherwise, vSphere is good enough for almost anything.

Re:VMs

[marcosdumay]

How do you move an NTP server from one physical machine to another? Without stopping it in between.

Well, NTP's cluster is called "pool". (Also, you can hack something with a shared IP...)

I am curious about this. Does "your datacenter" mean any datacenter, or specifically the AC's datacenter?

It means it is very probable that the GP's datacenter in particluar has it, because nearly all the datacenters have, and there is no reason to belive the one he administrate doesn't. I'm aware that is a low, but nonzero chance of he being an exception, and thus I'd be wrong.

Also, I'm not wiling to even look at the /. site for vunerabilities. I'll pass the dare.

But even better, if the pool is hosting 20 services and one is requiring more resources, typically you can subtract resources from other services, at least temporarily until you go find the powerful server to replace it. With zero downtime.

Thanks, that's another real reason for using VMs. They give you some flexibility here.

Notice that you should have zero down time anyway, that's not a reason for VMs.

bunch of VMs on a box or two

[Anonymous Coward]

Not using a huge collection of physical boxes any more. Just set up a bunch of VM's and leave them to it.

VMs?

[Anonymous Coward]

Why not make one box a VM host and have your various support boxes VMs (except for the ones that NEED to be physical).

Re:VMs?

[Skapare]

Great. Then we can knock out the data center by unplugging just one box. Brilliant plan.

Re:VMs?

[green1]

Redundancy doesn't mean having different services on separate boxes, it means having the same services in multiple places. In fact it's easier with one VM box hosting everything, because it's easier to keep it backed up and sync'd to a spare then it is to do a whole bunch of individual ones.

Re:VMs?

[rnswebx]

I can only hope you're joking.

You're saying to have one box with all of your services on multiple VMs, right? If you have any sort of service interrupting event on that one machine, all of your services go down. That sounds awesome.

Re:VMs?

[green1]

The alternative is a whole bunch of individual points of failure of individual services. No better really as any one of those boxes going offline is likely critical.

Either way you need a backup of your services. In your scenario you need a dozen primary boxes, and a dozen backups. In mine you need one primary machine and one backup, much simpler to administer, and at least as reliable if not more so. You could even have a tertiary backup and still have fewer physical machines to worry about than just your primaries.

Re:VMs?

[Anonymous Coward]

You're not the first to consider downtime and redundancy in VM scenarios.

There are ways of mitigating that.

Re:VMs?

[rnswebx]

Why can't you run multiple services on one machine and have a secondary? The examples given in the topic were NTP and monitoring. I don't see a need virtualizing those services, or many of the others that have been discussed in this thread. It's standard procedure in any reasonable infrastructure where I've worked to run these sorts of services on a single, bare metal machine. (I currently part of a team managing ~40k linux servers, for example)

Help me understand why would you go through the hassle of virtualizing all of these services into separate VMs and creating extra layers of administration in this particular case.

(The wording in the post I originally responded to, in my opinion, did not indicate that you had any secondary service active, but rather just backed up to some other server.)

Re:VMs?

[berashith]

It really gets good if you decide to feed IPs to all hosts (even your ESX boxes) through DHCP, and then virtualize your DNS/DHCP services. works great until you turn it off!

Re:VMs?

[Anonymous Coward]

Don't worry old man, eventually you will be replaced by someone who does understand it. BARE METAL IS DYING and doing it pretty fast. Netcraft can confirm it. Your point about service consolidation onto fewer machines is valid. Run more things on less hardware. The same reason that point is valid is why it also makes sense to consolidate those entire machines over to a VM farm.

One example. You have an HP blade ceter or a bunch of Proliant G4 servers that are at end of life or getting too expensive to maintain, you are running 20 bare metal machines on them. You want to replace them all with a Cisco UCS system or even a newer HP systems. Have fun.

If that HP blade center or Proliants were running as an ESX cluster and your "servers" were running as VMs, you could fire up that UCS system with ESX and add it to your ESX cluster. Then migrate all of the VMs over to the UCS blades running ESX with nothing more than a few clicks and they would have ZERO downtime. Lets say that new UCS system is in a different building but has decent connectivity, same thing. (gets a little tricky if that other building is not in the same subnet because the VM won't work right away but there is a way around that too) Really, it only takes a few clicks. Eject the HP blade ESX servers from the cluster, unplug it and throw it away.

Re:VMs?

[afidel]

Set your critical VM's to power on with host, highest priority. Now your problem is solved, it's not like the problem hasn't been run into many years ago and solved.

As far as NTP is concerned, I run that on core switches, it's a nice stable location (how often do you change them out versus other boxes) and if they aren't available then the fact that your clock is drifting is the least of your problems.

Re:VMs?

[berashith]

The situation I saw, and laughed at for far too long, was that the ESX host was set to get IP from DHCP ( forced by MAC ). This worked fine until the servers needed to server this were P2Ved, and then a SAN maintenance brought everything down. People were lucky that the Physical box still existed, but was powered down.

Yes to NTP on switches. They are always one hop away

Re:VMs?

[green1]

My point is not about the VMs, I personally agree that there is no need to have many of those services on their own VM. However the original argument was multiple physical machines vs multiple virtual ones, My point was in getting them all on to one box more than it was about using VMs vs multiple services on one box. With them all on one physical box, whether virtualized or not, it makes it easier to have a second physical machine which is a backup of the first, rather than multiple individual primaries with multiple individual backups. more reliable too.

I didn't get in to the VM vs multiple services argument for the simple reason that without further information about the particular setup and exactly which auxiliary services they run, along with who is responsible for what, and the bureaucracy behind it all it's impossible to know what can be consolidated and what can not for this particular example.

Virtual machines.

[Anonymous Coward]

'nuff said.

Re:Virtual machines.

[ls671]

Jails, jails, jails !

cheapest atom board

[Anonymous Coward]

take the cheapest intel atom board, add ram and the cheapest mass storage you can get (stick), done.
yeah yeah, your supplier might not have a 24con-atx 12v-adapter, but that's your problem.

VIrtual little boxes

[kurthill4]

I use virtual machines; very easy to schlep around if needed, very easy to launch/create a new one, etc. Linux vm's for anything needing scripting.

little "virtual" boxes?

[Anonymous Coward]

I guess it depends on the specifics, but sounds like jobs for VMs to me.

Re:little "virtual" boxes?

[Skapare]

Why not just run all the programs in the same OS. I put the DNS server and the NTP server and the Mail server and the Web server in one box without VMs. You can still take down the whole place by unplugging one box so don't worry about lacking a kill switch.

Re:little "virtual" boxes?

[ColdWetDog]

Why is it important to have a kill switch? You working on the Skynet beta or something?

Re:little "virtual" boxes?

[swalve]

Because it makes it WAY easier to manage. You don't have to worry about varying OS requirements, you don't have to worry about devilish interactions between services, you don't have to worry about things like rebooting a mail server taking down DNS and your website. When the hardware shits itself, you don't have as much worry with reinstalling everything when the replacement inevitably has different drivers or HALs or what have you.

Also, if you CAN run all that stuff on one box, you probably aren't the target audience for this information. But even then, with VMs, you can expand more easily. When it comes time that your one box gets too overloaded, you can just split the machines to two boxes and off you go.

Re:little "virtual" boxes?

[Anonymous Coward]

No reasonable BOFH can get along without a kill switch

Virtual Machines for Small Workloads

[ndrw]

For those "little boxes" that you know won't be fully utilized or need extreme resources, I suggest getting a couple of decently sized servers, running some virtualization platform (vmware, xen, windows (lol), and using virtual machines.

Re:Virtual Machines for Small Workloads

[Anonymous Coward]

I'm curious; how much does that really cost, to have a VM farm with redundant hardware & complex management software, compared to the cost of a raspberry pi NTP server at each site?

I can build an NTP server out of discarded trash that will run for ten years with remarkably little maintenance overhead and zero licensing costs.

Virtualization fanatics need not assume I am assaulting their religion - I use and like VMs, they have many wonderful advantages and very few disadvantages.

VM

[Anonymous Coward]

The trend today seems to be a couple of fuck-you-powerful machines running a lot of virtual machines. (Kind of part of the reason Microsoft has been shitting their pants over VM ware, their licensing forces you to constrain your network design by artificial licensing)

Anything, really

[Anonymous Coward]

Any low power box will do, really, if all you plan to do is run NTP or other minor services. Or, as others have pointed out, get one mid-range server and load it up with VMs for the various minor tasks you need to perform.

HP Proliant MicroServer N40L

[steveha]

I don't work in a data center. But I think you might want to look at an HP Proliant MicroServer.

Basically it is an AMD laptop chipset on a tiny motherboard in a cunningly designed compact enclosure. The SATA drives go into carriers that are easily swapped (but not hot-swappable). It's quiet and power-efficient. It supports ECC memory (max 8GB) and supports virtualization.

http://h10010.www1.hp.com/wwpc/us/en/sm/WF06b/15351-15351-4237916-4237918-4237917-4248009-5153252-5153253.html?dnr=1

Silent PC Review did a complete review of an older model (with a 1.3 GHz Turion instead of 1.5 GHz).

http://www.silentpcreview.com/HP_Proliant_MicroServer

SRP is $350, but Newegg has it for $320 (limit 5 per customer).

http://www.newegg.com/Product/Product.aspx?Item=N82E16859107052

Newegg also has 8GB of ECC RAM for about $55, so you can get one of these and max its RAM for under $400.

I just got one and haven't had time to really wring it out, but I did do the RAM upgrade. Despite the tiny enclosure, it wasn't too painful to work on it, and I was impressed by the design. The Turion dual-core processor has a passive heat sink on it, and the single large fan on the back pulls air through to cool everything. (There is also a tiny high-speed fan on the power supply.)

I'm going to use this as my personal mail server. It's cheap enough and small enough that I plan to have at least one put away as a hot spare; if the server dies, I'll power it down, move the hard drives to the spare, and I'll have the mail server back up within 5 minutes. Not bad for a cheap little box.

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

HUGE!!!

why don't you look at something like this?

http://www.e-itx.com/m30-d525ae-6c.html

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

work with 4TB drives?

Re:HP Proliant MicroServer N40L

[ALeader71]

I tried esxi on my NL40, but it doesn't see the mothebaord RAID and I didn't want to shell out $$$ for an add-in RAID card that costs nearly as much as the server. So I'm running Ubuntu 12.04 LTS and I'm running KVM to my virtualized boxes.

1 - Windows XP instance for iTunes 2. ??? Not sure yet.

The rest of my services: samba, ssh, DNS, DHCP, Plex, are running on the host OS and I'm not seeing a need to run them within a VM. I'm using "fake" RAID for my four 1TB drives backed up to a USB 3.0 (via add-in card) external HD and I couldn't be happier.

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

answered my own question :

http://n40l.wikia.com/wiki/Hard_drives

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

About that 8GB memory limit, the community has already validated that the published limit is not the actual limit. There are a few of us around running 16GB RAM in these boxes. It works, no problem.

As it turns out, illumos runs really well on this hardware. Pick your distribution. Mine is running SmartOS from a thumb drive, so all four of my 2TB hard disks are dedicated to virtual machines. I've got several virtual machines (zones) running on mine, and planning to pick up a couple more to expand my little "dollar store cloud".

Even though these HP Microservers are inexpensive, don't write them off as cheap.

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

I tried esxi on my NL40, but it doesn't see the mothebaord RAID and I didn't want to shell out $$$ for an add-in RAID card that costs nearly as much as the server. So I'm running Ubuntu 12.04 LTS and I'm running KVM to my virtualized boxes.

1 - Windows XP instance for iTunes
2. ??? Not sure yet.

The rest of my services: samba, ssh, DNS, DHCP, Plex, are running on the host OS and I'm not seeing a need to run them within a VM. I'm using "fake" RAID for my four 1TB drives backed up to a USB 3.0 (via add-in card) external HD and I couldn't be happier.

look into a perc 5 or perc 6 on ebay?

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

HUGE!!!

why don't you look at something like this?

Not sure if you are joking.

The Turion 1.5 GHz is more powerful than an Atom D525 1.8 GHz.

http://www.cpubenchmark.net/cpu.php?cpu=AMD+Turion+II+Neo+N40L+Dual-Core
http://www.cpubenchmark.net/cpu.php?cpu=Intel+Atom+D525+%40+1.80GHz

The Atom also won't support ECC or virtualization instructions. That tiny box only allows a single drive, so forget about RAID. If the tiny box dies, you will need a screwdriver to move the hard drive to a spare tiny box and it will be a slow process; if the hard drive dies, the server is down (no RAID).

The HP MicroServer has a card slot that can take a system management card. Also, if a data center is already buying HP kit, maybe they will want to buy an HP MicroServer instead of an "E-Box" from a Yahoo store.

But other than that, yeah I guess one is as good as the other.

Re:HP Proliant MicroServer N40L

[Mark of the North]

It's not rack-mountable. No IPMI either. That should be a deal-breaker for anyplace serious enough to have a rack.

We try to virtualize anything that can be virtualized. But for those few tasks that really need to run on bare metal, we've had good luck with little Atom D525 Supermicro rackmountable boxes. We bought a few complete boxes (minus ram and storage) that Newegg billed as fanless (which was a lie). Those ran hot enough to develope problems after a few months. Ever since we've built ours up from parts (SUPERMICRO CSE-510-200B 1U rackmount server case, SUPERMICRO MBD-X7SPE-HF-D525-O server motherboard, SUPERMICRO MCP-220-00051-0N single 2.5" fixed HDD mounting bracket, GELID Solutions Model CA-PWM 350 mm PWM Y Cable, RAM and storage). About $400 and have been really reliable. Only thing I don't like is that they don't have IPMI on a dedicated port.

But honestly, if there is any virtualization going on, there shouldn't be much need for these.

Re:HP Proliant MicroServer N40L

[Mark of the North]

I forgot the PC Engines boxes that we have in a few spots. We use them as the box that monitors the UPS and controlls what gets shut-down when the power goes out and looks like it will be down for a while. Since the box draws about 10 watts, the UPS can run it for days before running out of juice. When the power comes back, the PC Engines box coordinates bringing everything else back up. We haven't found anything else that compares for a low-power box that doesn't have to do much of anything other than run reliably.

Re:HP Proliant MicroServer N40L

[PrimaryConsult]

And in some places that get a little *too* serious, you end up with some stupid proprietary appliance that can't be rack mounted but the PHB swore was needed. And for that, you will have one of these. And in the extra space next to said proprietary POS, you can put something like the abovementioned HP server.

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

FYI despite what it says on the tin it will take 16GB of ram just fine. 4TB hard drives also work just fine (but it needs a =2TB drive to boot from).

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

It has IPMI as an option - see item "8. Remote Access Card slot" in http://h18004.www1.hp.com/products/quickspecs/13716_div/13716_div.html and we use this combo all the time. Have never had one failing (Disks excluded), as all the HW is run so much below its specs.

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

Actually there is an IPMI module (iLO) available for it. Newegg even carries it.

Re:HP Proliant MicroServer N40L

[xt]

It does have a remote access card you can put in, take a look at its manual or a review. I have one N36L installed with this card in a closet back home and it makes one hell of a Proxmox VE machine.

Re:HP Proliant MicroServer N40L

[Andor666]

The HP Microserver has the "HP MicroServer Remote Access Card" option, an IPMI compatible card you can plug in a PCI Express x1 port. It has a virtual KVM, also. Its price may be around 60-70$.

See http://www.livingonthecloud.net/2011/08/hp-microserver-remote-access-card.html

I uso two of this on several locations as a NAS with ZFS, using last FreeNAS releases. There used to be a problem using ZFS with FreeNAS on this machines, that weren't powerful enough for running it and transfers would stall each several seconds, but, once I upgraded to FreeNAS 8.3 and upgraded to 4GB of RAM or more, the problem stopped.

Re:HP Proliant MicroServer N40L

[drinkypoo]

Cunningly designed compact enclosure? It's almost literally an order of magnitude larger than it needs to be.

Re:HP Proliant MicroServer N40L

[fuzzywig]

Only in the very expensive, top-tier data centers have I ever not seen non rackmount kit shoved into racks.

I was in a datacenter recently where we have three racks, and whilst wandering round the room waiting for an install, I don't think I saw a single rack that didn't have stuff piled on top of racked stuff. In our racks it's just external HDDs and spare screws/tools/parts, but I did see a small desktop case with a fedex box on top acting as a shelf for some networking gear.

If it works, it'll do.

Re:HP Proliant MicroServer N40L

[AmiMoJo]

I have one of these too. It seems that HP put much better power supplies in the early ones because the newer ones are quite loud. You are incorrect about the single large fan, they actually have a small one in the PSU as well which is quite noisy.

Generally speaking they are not bad machines. Could have done with a serial port and more USB ports.

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

FWIW, a colleague has a microserver with 16GB RAM in it, and it works. Wish I'd had the guts to splash out and try it first :-)

Re:HP Proliant MicroServer N40L

[steveha]

It's almost literally an order of magnitude larger than it needs to be.

Does it "need" to have room for four standard 3.5" hard drives? Personally, I like using RAID, and I don't think one hard drive is enough. For my purposes, yeah, it "needs" those hard drive bays. That's why I bought the thing.

This thing is designed with a standard 5.25" drive bay instead of a laptop optical drive; you can argue that this is just wasted space, if you like. I might put something other than an optical drive in there, though... I'd love some sort of slot for hot-swapping hard drives, to be used for data backup.

This thing also can take PCI express cards, but I'm going to claim that the space for that is exactly as large as it needs to be and not one cubic millimeter larger.

How can this thing could be made one-tenth the size, without giving up any functionality? It can't, because it isn't literally an order of magnitude bigger than it needs to be.

Now, if you are going to claim that hard drive bays, a 5.25" drive bay, and PCI express card slots are all useless, and that you could replace this whole thing with a single PandaBoard, then sure you could make something a tenth the size of this thing. So if all you "need" is an SBC with a NIC and some ROM, then you don't need something this big. However, that also doesn't make the enclosure less cunningly designed.

So, if you still want to sneer at the enclosure, now it's your turn. Provide specifics on how you would design the thing differently, and what tradeoffs will result from your design changes.

steveha

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

You are incorrect about the single large fan, they actually have a small one in the PSU as well which is quite noisy.

From GP:

the single large fan on the back pulls air through to cool everything. (There is also a tiny high-speed fan on the power supply.)

Re:HP Proliant MicroServer N40L

[drinkypoo]

There's been enclosures that clever or cleverer for a long time. The question was about eensy weensy boxes. Even the Digital Tech Dt168 had a niftier case. If you're not using blades which rack mount which is probably the best thing to do, you're going to want something smaller than that for purposes like time server.

Re:HP Proliant MicroServer N40L

[steveha]

If you're not using blades which rack mount which is probably the best thing to do,

Sure, makes sense. The MicroServer might actually cost a bit less, but if every other server is racked, why make an exception?

you're going to want something smaller than that for purposes like time server.

How many time servers do you need? This thing is 8" by 10" by 10". It's not that large, really.

But okay, your objection to its size is duly noted.

Re:HP Proliant MicroServer N40L

[Anonymous Coward]

Why so much hate?

Re:virtualization is the game now

[Zaelath]

Virtualized NTP is about the dumbest thing I've read on /.

Yes, worse than various conspiracy theories and fanboi wars.

ESXi

[nurb432]

No little unsupportable boxes here.

Re:ESXi

[Capt.DrumkenBum]

ESXi rocks.
It has made it easy to spin up a test server or six as needed. Makes my work life just a little bit easier.

Re:ESXi

[Anonymous Coward]

Yeah, the free version of esx.
Spend a little more on decent hardware, load it with lots of cheap ram. Four core cpus are dirt cheap today too.

Have all the project/test/whatever servers you ever wanted, whenever you want.

Re:ESXi

[Anonymous Coward]

How well does that work for you with respect to NTP?

Re:ESXi

[lewiscr]

I wouldn't run my ntp masters on ESXi. I've always let my firewalls and/or routers handle that task, so I haven't needed to virtualize. On the VMs themselves, monitor to make sure ntpd is running (or set your config manager to start it if it's not running). I've had some problems where one VM causes slowness for the others, and ntpd would loose sync and exit.

Re:ESXi

[Anonymous Coward]

"unsupportable" come on. You use config mgmt software like everyone.

Re:ESXi

[ls671]

ESXi is a very nice tool, it isn't a panacea. Look at jails and other alternatives and always use the best toll for the job.

Re:ESXi

[drunkahol]

2 points. Not all firewalls/routers do NTP. Seems like they're phasing it out. None of our current network kit provides NTP - much to my frustration.
Second, ntpd can be configured to not care about how big a time difference there is and sync no matter what - "tinker panic 0".

Cheers

Re:ESXi

[Anonymous Coward]

The problem with this is, if you use too few sources (1), and it has problems, using tinker panic 0 can cause your entire environment to suddenly adjust to the wrong time.

If you need perfect timing, spend some money on a couple of appliances (they're expensive), otherwise just some white boxes running nothing but ntpd. Put them on different switches, on different subnets, and sync them to different sets of external ntp servers.

Re:ESXi

[Anonymous Coward]

Yes, but instead I swap it for a big unsupportable box called ESXi which isn't "supported" on anything but a handful of old hardware devices. If you aren't careful when you spec your hardware you may find it won't work. (Thanks VMware for being so slow to support modern LSI controllers! Won't even recognize as a bunch of disks yet alone a RAID and your "patches" are even a generation of hardware behind.) No thanks. I switched to KVM and an won't look back.

Re:ESXi

[Skweetis]

If you don't need perfect timing, just consistent timing for kerberos and log file sanity, then running ntpd on ESXi with tinker panic 0 and about four time sources will work well, with no more than a few seconds drift at any time.

Previous gen hardware

[trandles]

Last generation's compute nodes. We keep some around for utility functions after decommissioning a large cluster.

Re:Previous gen hardware

[Anonymous Coward]

Exactly. Most of the people commenting here obviously have never worked in a data center. You use the cheapest thing possible for non-critical servers like this - not overpriced macs or HP-whatever that some guy mentioned.

Re:Previous gen hardware

[pboyd2004]

We tend to keep all of this type of stuff on a couple of smaller/older servers running as VMs. This way it's simple to move them to newer hardware when we retire them and they can be moved so that we don't suffer downtime if we need to service the physical machine.

Get a real time server.

[attemptedgoalie]

Go get a GPS satellite receiver/time server. Actually, get two. Don't screw with time.

THEN, virtualize the rest of the stuff. Monitoring, syslogging, management, patchers, etc.

We've virtualized everything except for
- a Windows DC so that it stays up if the vmware datastores or SAN eats itself in a horrible way.
- The NIS server we have to use on our UX environment due to an ancient regulation. I'm not willing to put up HP-UX VMs for this right now, otherwise it'd be safe in a VM as well.
- Anything we can't virtualize due to licensing/contract/support issues. So our VOIP environments, phone call recording, access control systems for the doors,

My datacenter is getting a lot nicer to look at, and a lot easier to upgrade. I can shift servers or volumes all over the room so I can do live maintenance during the day.

Re:Get a real time server.

[Anonymous Coward]

Note: GPS timeservers can vary widely in quality. Don't assume that the most elegant package, slickest website or cheapest price equates to a solid box (remember, realtime OS's can crash too ;).

Some of the most reliable and precise timeservers I've seen have been home-built PC based boxes.. YMMV.

Re:Get a real time server.

[aaarrrgggh]

What's the purpose of the GPS time server? I understand what it does, but why do you need microsecond level accuracy to a server that distributes time via NTP with an accuracy of about 2-3 milliseconds? The only purpose I understand is when you need to synchronize time with greater accuracy across distances, or with the same accuracy without a network.

Re:Get a real time server.

[AmiMoJo]

GPS alone is not enough for a data centre, or any large operation. If you have two GPS receivers at the same location both could easily lose signal for an extended period of time. Back them up with a cheap atomic clock.

"Obsolete" hardware

[beegle]

Those support tasks don't exactly push hardware to its limit, and most of those tasks are the kind of thing that demands a bunch of redundant servers anyway.

Throw a bunch of "last generation" hardware at the task -- stuff from the "asset reclamation" pile. Leave a few more around as spares. Less disposal paperwork. Works just fine. By the time your last spare fails, you'll have a new generation of obsolete hardware.

Re:"Obsolete" hardware

[sidetrack]

"last generation" hardware usually uses huge amounts of power compared to the current generation. e.g. 150 watts a piece plus. That could quite easily cost US$200 per year to run in a datacentre. You're usually better off scrapping them.

How about:

Raspberry pi - 5 watts
Dell Optiplex 160 - 15 watts
Dell R210-II, with the hard disk removed, and a single Intel SSD instead - ~35 watts

All of those have no spinning disks, and the first two have no moving parts at all.

mini clusters

[Anonymous Coward]

I use several groups of paired 1 U pizza boxes, or as small a server class machine as I can get and still be neat and tidy in the rack. I give each host a primary IP address, then an IP address used for the services that box delivers. I feel free to stick a few services on each one if they can handle it, then stick those cluster IPs in a round robin DNS entry. So each host has a cluster IP that's active all the time, and in the event of an outage on one, the partner takes the cluster IP. In the cluster config, I give each IP a preference, so in normal conditions the cluster is active/active with both hosts working, and in a failure situation, one host takes both sets of IPs and chuggs on like nothing happened.

Sometimes this requires a little IP tables ruleset if your don't want to restart your services to reconnect to every listening IP if that service doesn't support adding listeners on the fly. Sometimes this requires two sets of configuration files for the service.

Most of the time, it just works, you set it up, and forget it. Write a short script that helps you quickly configure the cluster IPs and preferences, so you don't have to go back to offsite memory to build the next set of hosts. It works really well with services that are agnostic and just provide a stateless service. If the service requires some sort of state memory, well, just work it out.

Re:virtualization is the game now

[0123456]

To be fair, if someone cares enough about time accuracy to understand why that's a dumb idea, they should probably be using a GPS receiver instead of a PC.

amazon

[mveloso]

For little boxes that deal with DNS, time, etc - put them in amazon. They're critical servers, but don't really need to be at your site. Put the primaries outside, and slaves on the inside. That way if you have an outage you can always repoint DNS to somewhere else...something you can't do if your primary DNS is on a dead network.

Re:amazon

[sdguero]

Virtualize NTP?

Good luck with that...

Re:amazon

[marcosdumay]

For little boxes that deal with DNS, time, etc - put them in amazon.

besides the NTP problems, also make sure to write on a piece of paper the IP of every computer on IT, then put it on a wall.

When you have internet problems and nobody is able to get any work done anymore because all of the light services don't need to be at your site, you'll need those addresses for the LAN party.

Re:amazon

[Anonymous Coward]

Virtualize NTP?

Good luck with that...

This. If you want to move your local ntp to amazon you could as well drop it and go for a public ntp source.

Re:amazon

[Anonymous Coward]

Yeah, that sounds like a great idea!

Crash Cart

[Jonah Hex]

You have a crash cart with a KVM (for the rare occasions you need to locally access two or more machines simultaneously) and that attaches to all the specialized cables for interfacing with your blades or full size servers, make sure it has a shelf for holding drives/ram/batteries and a bin for more specialized PS2/USB to Server convertors. Otherwise you sit at your desk and remote into EVERYTHING: VMs, Linux, Windows, iLO/etc. - HEX

Re:Crash Cart

[Skapare]

Use OpenStack and you won't even need this.

ARM'd n Dangerous

[Anonymous Coward]

ARM boxes, running many chips with a fraction of the power consumption

taking over a data center near you

Re:ARM'd n Dangerous

[mabhatter654]

That's true. My company uses IBM BladeCenter servers bundled into a VM cluster. The bang-for-buck at the Time were the 4-core Opterons... That easily scaled to 4-cPUS for 16-cores.. (That could probably be higher now). The beauty of AMD. Moving into this space is that the blades could be swappable with the current hardware.

But rather than rowed of boxes, VM is the better way to go.

performance?

[Chirs]

NTP server is all about consistency. If it's running in a VM and can be delayed at the whim of the host, do you think it's going to be a very good source of time?

Re:performance?

[profplump]

I think it will be fine, so long as it's not using the CPU for a timing source.

Re:performance?

[TwineLogic]

Exactly. The latency of response in an NTP server must be consistent in order for the algorithm to converge. It doesn't matter what timing source is used for a reference, if the network communication has variable latency, the NTP precision must degrade. It's revealing that VM proponents don't seem to understand this.

Re:performance?

[I_Wrote_This]

A few years back I had some VM systems that would totally freeze for n*60s (highest n I saw was 15). When it unfroze the system clock was n*60s slow - so it had been a total freeze.
I never did get any explanation.
Those VM systems never did any useful work.
VM systems have their place, but running such services is probably not it, for me.

Re:performance?

[tkrotchko]

Depends on the requirements (the needs).

You don't need cadillac solutions if the requirement is to have logs that are easy to correlate.

Virtual Machines will work fine for most applications.

Re:performance?

[Anonymous Coward]

We use two of our Windows domain controllers for our time source. Those 2008 R2 machines are running on a 10 node ESX farm with about 450 other virtual machines. Those two domain controllers provide time services for about 2000 devices in our worldwide network (not just windows machine either, our switches, routers, SAN, etc). We have NEVER had a problem with NTP and synchronization.

NTP is network time protocol. It is designed with random latency in mind. If you are going over a network, there is random latency. That latency inherent to any network is many orders of magnitude higher than any latency a virtual machine sees running on a hypervisor.

Re:performance?

[Anonymous Coward]

That sounds like a problem with the deployment of the VM hypervisor. That is not at all normal and your experience should not be the basis of judgement of all VMs.

Re:performance?

[Anonymous Coward]

You do understand that kernels and hypervisors are similar in that they get to determine when and which processes get CPU time? A hypervisor determines when your host gets execution time is and a kernel determines when your process gets execution time.

Re:performance?

[ls671]

I have had best results on bare metal indeed.

I run ntpd on bare metal along with other apps but I run ntpd in a jail (chroot like), just in case. I do reply to public requests but I do not allow queries, ntpdate and other stratum servers requests work fine but you can't ntpq -pn me for example.
From ntp.conf:

restrict default noquery

By the way, I am a maniac but I am still satisfied at +/-5 ms. Please do not close my door to hard so it generates a gust of wind towards my ntp server and make it go above +/- 5ms error margin. Not maniac enough to buy a GPS although...

Re:performance?

[Anonymous Coward]

Just run it on your switch. ...if latency really matters to you.

Re:performance?

[Anonymous Coward]

Exactly. The latency of response in an NTP server must be consistent in order for the algorithm to converge. It doesn't matter what timing source is used for a reference, if the network communication has variable latency, the NTP precision must degrade. It's revealing that VM proponents don't seem to understand this.

Uh, it doesn't take a "VM proponent" to host NTP on your network switches. The issue is a VM doesn't have a stable clock, and it's hardly a secret... how would anyone not know that?

Sniping at "VM proponents"? Talk about revealing. That's like saying "sex proponent".

Re:performance?

[Anonymous Coward]

Exactly. The latency of response in an NTP server must be consistent in order for the algorithm to converge. It doesn't matter what timing source is used for a reference, if the network communication has variable latency, the NTP precision must degrade. It's revealing that VM proponents don't seem to understand this.

What do you mean IF? What kind of magical Ethernet are YOU using, because the ones I've seen will drop packets under load just as well as a VM host drops timer interrupts.

Re:performance?

[guruevi]

I know this much from working in science: none of the hypervisors I know off (VMWare, VirtualBox, Parallels, KVM, ...) have a very good track record when it comes to keeping stable time sources (whether ticked or tickless). All of them miss events or delay them which for keeping track of time is a bad thing.

I tend to keep most of my basic necessity networking off VM's. That includes NTP, LDAP, Kerberos and DNS. On my systems they are combined on a single hardware host with old-school security measures to keep the services contained.

Re:performance?

[omnichad]

Well - with VT-d, you could have a PCI add-in card with a stable clock for each VM. Of course that's more like having half VM and half physical.

Re:performance?

[Anonymous Coward]

Well first of all you seem to be comparing desktop virtualization products with real commercial-grade bare-metal hypervisors.

Second of all, Microsoft and VMWare fully supports running a domain controller on a virtual machine. A DC includes timekeeping services for the domain.

Re:performance?

[Guspaz]

Most applications don't need hyper-accurate precision. There are some use-cases out there where it might matter, but for the vast majority of workloads, a few milliseconds of time offset isn't a problem, even in enterprise.

Re:performance?

[Guspaz]

Practical example: the latency to the NTP server I sync to randomly varies between 30ms and 70ms. The range of my offset varies from roughly -5ms to +5ms. Pretty accurate, all things considered.

Re:performance?

[highphilosopher]

By single hardware host are you referring to the 2.4 Ghz pentium beside your desk? If so, you missed this conversation. Grab the next one.

Re:performance?

[Guspaz]

Running ntpd, you don't need your hypervisor to keep good time. ntpd will use adjfreq (speed up or slow down time to counter drift) and adjtime (make small adjustments of a few milliseconds at a time) to keep the VM's clock pretty darned accurate, even running on virtual machines with massive drift. I've seen machines that drift minutes per day be kept to under 1ms off actual time just by throwing ntpd with the default settings on them. It really is trivially easy, unless you need microsecond-level precision.

Re:performance?

[badkarmadayaccount]

I heard about a DTV USB tuner easily reconfigurable as a SDR, ergo GPS reciver for $50 to $80, there was a story here about it.

VMs

[wiedzmin]

VMs

Virtual Machines I suppose

[MichaelSmith]

I think its apalling that we do that. Its a horribly expensive way to work in hardware but we do it because we can't be stuffed to deal with operating systems. Most likely a single box and OS instance could do it for you if it was set up correctly.

Re:Virtual Machines I suppose

[Anonymous Coward]

How do you figure that LESS physical hardware running at or over 50% capacity is MORE expensive than racks full of 1U servers all sitting mostly idle?

virtualizing NTP is dumb

[Chirs]

You want consistently fast behaviour from your time servers. Don't mess with virtualizing them.

Re:virtualizing NTP is dumb

[AK Marc]

If an NTP request is served a little slow, what's the problem?

Re:virtualizing NTP is dumb

[Anonymous Coward]

I think the important part of the GP is 'consistent'. Having your NTP coming over the internet will introduce known unknowns and unknown unknowns.. Also, if your onsite slave/backup unit gets its feed from a GPS, but your primary NTP comes from a lab somewhere, you might have a nasty surprise wrt. leap seconds or something. I don't know and I don't care to think too much at this time of night. But I know that moving what is essentially an off-board RTC replacement offsite is a recipe for new and interesting failure modes :)

Re:virtualizing NTP is dumb

[silas_moeckel]

VM's do bad things to keeping accurate time they do a lot of funny business. There solution so far is it poke a hole though to the main OS to get time.

Re:virtualizing NTP is dumb

[Anonymous Coward]

Chirs is specifically talking about NTP hosts that get their time source over the internet. Which is why this whole debate is stupid but he wants to ride this horse for some reason.

It's common for there to be deltas of several milliseconds on the internet and sometimes random spikes that are much higher. Yet he thinks that the hypervisor's scheduler is going to have greater deltas than that when scheduling execution of the NTP VM.

Now I can understand it if we're talking about a local time source and you need sub-microsecond precision. But few people have a use-case for that. A server farm in a data center or a bunch of desktops in an office do not require it.

If you

[JustOK]

If you can't run it on your iPad, it's probably not worth running.

--Management.

Re:If you

[TubeSteak]

I'm picturing racks of overclocked iPads with a wall of box fans pointed at them.

And then I'm imaginging the conversations that would inevitably ensue:
"I know I fat fingered the fucking IPV6 address. YOU try typing on this goddamn touch screen"

Personally at work for small things...

[pjr.cc]

I personally hate and despise people who put non-rackmount kit in racks...

We use various devices.. mostly all 1ru servers of various configs... for eg there are a couple of mini-itx 1ru servers we have that have e350 based mini-itx boards (i really love the e350/e450 boards)... not quite as cheap as the hp n40 microserver, but at least its a rack format.

Then we have a few that run virtualisation here and there for some tasks using kvm (some of those too have e350's in them as the e350's do have the virt'n extensions unlike the intel atoms)... we also have a few that run intel based i3/i5/i7 mini-itx boards... they're quite nice when you need some extra grunt...

some others are based on super micro boards as well though (which are quite cheap and run core i3/i5/i7 cpus rather the xeons). Then some others are old 1ru xeons we no longer need for server tasks...

Re:Personally at work for small things...

[jrmiller]

I personally hate and despise people who put non-rackmount kit in racks...

Hear, hear! We have a rack that's a big rat's nest of cable boxes and IPTV gear. Most of the devices are single-powered, which means we can't take down a UPS for maintenance without taking them down (not a huge deal), plus they're just a rat's nest of cables. We also have a Mac mini in one of our racks that's attached into some sort of purpose-built chassis. Still's single-psu, though--not ideal for a datacenter unless it's part of a larger cluster.

Re:Personally at work for small things...

[neonmonk]

Sounds like you need an ATS.

Re:Personally at work for small things...

[Skapare]

I totally agree. Just populate your racks and pick some for "special duty" (and put your DNS, NTP, and monitoring daemons on there).

Re:Personally at work for small things...

[green1]

While I agree that the proper solution for a rack is rack mount equipment, the fact that something is not rack-mount is not an excuse for it to be a rat's nest of cables. I have installed non-rack mount equipment, there's no reason the cords can't be just as neat and tidy as the rack-mount stuff if you do it right. That said, the better answer is to smack whoever decided to go with non-rack mounted equipment in the first place...

Rat bait stations

[stevegee58]

These little boxes are very common around data centers.

Re:Rat bait stations

[Hadlock]

What is it rats eat at a data center? Outside of the fishbowl there shouldn't be any water or food (except maybe for the cardboard boxes and packing peanuts servers come in)

Re:Rat bait stations

[Anonymous Coward]

What is it rats eat at a data center?

Dropped packets.

What scale data center?

[sxltrex]

I can't imagine trying to perform network management with a few mac minis so I'm assuming you're referring to a very small facility? Our new data center was built on 10-gig infrastructure and our NM is appropriately scaled--NetScout Infinistreams connected to Gigamon matrix switches. While the Gigamons were quite expensive they allowed us to utilize fewer Infinistreams while also providing some very cool functionality.

It look a long time for our upper management (those with the dollars) to come around to the notion that, in order to realize the full investment made in the data center, true network management needed to be baked in from the start.

Some things must be physical

[ALeader71]

Mac Minis are a good option. For me, it would depend on my environment. If it's Windows, then a few business-class workstations for administrative access and monitoring tools. If it's Unix or Linux, use the same class hardware (or even less for display-only devices) running whatever enterprise OS we're using. For OSX (are there any?) I'd go with Mac Minis or iMacs, but realize I could go the *nix route of my tools aren't OSX specific. I've read a few postings saying "just toss 'em all into a few VMs" and I agree, but for the administrator level access. When the proverbial stuff hits the fan, you need a few good standalone devices to remote or console in to these virtualized towers and figure out what the heck is going on.

Re:Some things must be physical

[Anonymous Coward]

Who the hell would purchase a mac mini for a server room? You want to make a use case for a consumer fine, but as servers? Bollocks.

Fucking shills/fanbois

old and arm

[dtdmrr]

At work (actual machine room), I've just been using old machines (obviously not running critical infrastructure there). At home, I got a raspberry pi to run bind, dhcpd, login, home automation and to wake up my home machines. But I haven't actually moved my dns services to it just yet.

Re:old and arm

[dgatwood]

I just moved my backup DNS server to a Raspberry Pi. I shed 38 Watts of continuous power consumption. If my math is correct, this change will save me about $116 per year in electricity.

Re:old and arm

[tsalmark]

I wont dare tell you how much you pay per KWh but 38 watts at 9 cents per KWh would cost me just shy of $30 per year: http://www.citytrf.net/costs_calculator.htm

Re:old and arm

[Anonymous Coward]

In some parts of the world (e.g. Australia), residential electricity can cost more than US$0.30 per kilowatt hour.

Re:old and arm

[dgatwood]

I'm in Northern California. My top tier of power costs either 35 or 38 cents, I forget which. So a small savings in continuous consumption adds up very quickly. That's also why an electric vehicle would not be cost effective for me. A Nissan leaf, cost-wise, is only equivalent to a car getting 35 MPG, which is significantly worse fuel economy than the similarly sized Prius, has a fraction of the range, and costs half again more.

Soekris

[Xipher]

We are using a couple Soekris boxes for some basic monitoring. They are lightweight atom processors with no active cooling and it's designed with networking in min. 4 Gig-E ports on the 6501, and you can get up to 8 more thanks to 2 PCI-E slots available in the rackmount version. Since we are using an mSATA SSD on the board we have no moving parts, so nothing mechanic to fail.

And I shall call him, mini-server.

[Anonymous Coward]

Depending on the number of systems being monitored, or what the task was, we would use one of three methods.

1. If the device required it's own hardware, a Supermicro 1U Atom system.
2. If the device could be a VM (such as a DNS or DHCP server in this case) it would be.
3. If the device could be a VM, but monitored other VM's, we would use a clustered install of the netmon software.

I think Supermicro makes a 1U twin system, and a 2U quad system. I believe these are available single socket, and should serve well. Given power budgets, usually re-using old hardware, depending on the age, can be a bad idea.

~Another anonymous coward

Re:And I shall call him, mini-server.

[swalve]

I love me some virtual machines, but I would want core services like DNS, DHCP and NAT on a separate machine. Perhaps in VMs if you like, but it seems like a lot of confusion to not have them in a separate "turn this one on first" machine. It seems too recursive to have all your stuff on one machine.

Re:And I shall call him, mini-server.

[Anonymous Coward]

because, you can't just set them higher in the cluster's boot order?

I'm sorry, my virtual clusters don't depend on ANYTHING being up. once the first one comes up after the backup-backup power went out, the CORE services come up first. As more resources come online, the back end stuff fires up, once it's all stable, customer facing stuff comes up, etc.

Forget the "Little Boxes"

[FuegoFuerte]

There aren't many good uses for "little boxes" in a datacenter. For the things you mention, there are dedicated devices, there are big boxes, and there are VMs hosted on big boxes.

1) Time - If you care enough to have your own time server, you don't want this on a generic "little box." If you actually care about accurate time, you'll want a CDMA/GPS/WWVB time device (assuming US, if outside the US use whatever is available for your locale). The easiest setup will be CDMA or WWVB, as long as you get a decent signal where the device sits. I've had good luck with End-Run Technology's gear. GPS works great for time, but won't typically get signal inside a datacenter so you have to run a lot of coax and mount and antenna on the roof, which your datacenter may or may not be OK with. CDMA relies on accurate time, and is usually synchronized directly to GPS, so you can consider it nearly as accurate while being a whole lot easier to set up.

If you don't care enough to buy a dedicated NTP device and are just looking for something to keep all your local gear in sync, a VM will work OK, though it's better to put it on a physical box with a real RTC. If you do host it virtual, make sure you disable any virtual time sync providers your virtualization platform may normally use, or else bad things happen (your NTP server syncs with the RTC on the host, the host syncs with the NTP server, they both drift).

For monitoring, if you have any real number of servers, you'll want it on its own beefy box. Decent monitoring is surprisingly resource-intensive. Any other "little boxes" (dhcp, administrative, etc) are a perfect use for VMs.

What do I like...

[Shoten]

I like the same big boxes as are used for everything else. NTP server, running on a Mac Mini...really? Get a GPS-driven device that serves the purpose. They run an embedded OS, so they're very low-maintenance and straightforward, and they perform extremely well. As far as uptime/network/performance monitoring functions, these need to be at least as reliable as everything else. And the mainframe interfaces are awfully important...imagine how much good you'd be if you maintained you intellect but became paralyzed, deaf, mute, and blind all at the same time? If those fail, your big iron is a big anchor.

Don't skimp on the support infrastructure of a data center. Those systems impact everything.

Re:What do I like...

[tkrotchko]

If you're running mainframes, run a z/VM LPAR, and then run a RedHat guest that is the timeserver for your mainframe.

Of course if you use a sysplex, you'll use STP anyway, probably synced to an atomic clock someplace.

I wouldn't offboard NTP for the mainframe.

Synertron micro boxes

[mike.rimov]

http://www.synertrontech.com/

Some are fanless that I use for linux boxes, some are rackmount with multiple motherboards per 1U case, and their prices are add-ons are cheaper than newegg.

Nope, don't work for 'em, just used their products for about 8 years now.

Those? We virtualized them.

[slacklinejoe]

No idea about others here, but nearly all of our peripheral boxes got pulled into virtualization projects and the "private cloud" thing management bought into. Seems to increase reliability in our systems since they get "free" piggy back rides on high availability systems so one DC failure doesn't take down our even our semi-important systems. To access them we just fire up the VPN with two factor auth over wireless at Starbucks from our laptop/tablet/phone if we need to log in. Beats the hell out of sitting in the chill, roar or heat of the DC.

little support boxes (aka infrastructure)

[Anonymous Coward]

Not being smarmy, mind you (especially not to a first time submitter), but you are talking about infrastructure (DNS, NTP, kickstart and jumpstart servers, internal web servers, etc.). Except for the SPARC jumpstart server, all infrastructure is Linux based and runs on old Sun (Intel) boxes in sufficient quantities for redundancy. This stuff predates the wide acceptance of VMs; if we were to redo it, most of it would move to VMs. There have been NTP issues on VMs, so NTP would likely survive as two boxes also running something else to consume the spare cycles.

Re:virtualization is the game now

[AK Marc]

NTP is not real-time, so a few ms here or there of delay is not a problem. A stand-alone server is overkill. Why wouldn't you set a low-priority service on a VM? Though it is stupid to dedicate a whole VM for it, as it can run as a service on just about anything (routers, and such, though having your authentication server as NTP helps keep down time mismatch errors that can cause authentication issues.

We run them in-switch

[Tugrik]

My datacenter uses Arista gear for top-of-rack and core switching. It's a large cloud-style environment with each rack acting as its own "pod" with self-contained services, so any one pod can be moved to any zone of any of our datacenters with minimal fuss.

Small services like NTP, in-pod DNS, sFlow relay, monitoring, puppet (some of it anyways) and small unixy management tools we just run in the Aristas themselves. They're Fedora-core linux based switches that will run those things happily and do a great job feeding those services to their pods.

As far as NTP, the core pair on the main backbone gets their own GPS inputs, then all the top-of-racks sync to the core pair. Works out quite nicely.

We don't have anything.

[pointyhat]

We don't have any management or service boxes. Everything is appliances (cisco/HP) or off site (exchange, CRM). Our AD servers act as the time servers for the hosting environment. We don't want to manage anything else as it all takes away from the bottom line and eats fairly expensive rack space.

If you care about time...

[DragonWriter]

To be fair, if someone cares enough about time accuracy to understand why that's a dumb idea, they should probably be using a GPS receiver instead of a PC.

Or using both GPS and atomic clocks.

Re:If you care about time...

[Anonymous Coward]

That's what my university does

Re:virtualization is the game now

[Anrego]

It can be done.

Ultimately if you need time more accurate than within a few seconds, you should be using a GPS fed stand alone time server anyway. If you are just running NTP so everyones desktop clock is the same and the log files match up.. VM will work fine.

ZOTAC ZBOXEN Are Nice

[kjhambrick]

This Atom D525 Box:

http://www.zotacusa.com/zboxsd-id13.html

( about $200 ) works well once provisioned with RAM, HD and CentOS 6.

For more throughput( about 4x ), this I3-based box runs very well for about $400:

http://www.zotacusa.com/zbox-id82.html

Tiny, well made and reliable.

-- kjh

NTP servers are NOT about consistency

[tlambert]

NTP servers are NOT about consistency, they are about making badly designed protocols, such as NFS, capable of limping, instead of just falling on their face.

If the requests on these protocols used a client timestamp for the client's idea of the current time, then the server on receiving the request could look at its idea of the current time, and arrive at a delta before it actually did anything other than enqueue the request locally.

Then when the server responded with a non-"now" timestamp in any client response, it could apply this delta to the response value, and as far as the client was concerned, it and the server would have synchronized ideas of "now", without resorting to all of this NTP BS or worrying about clock drift, or anything.

I lobbied very strongly to try to get this fixed in NFSv4; maybe we will get our collective heads out of our butts by NFSv5.

Re:NTP servers are NOT about consistency

[Anonymous Coward]

NTP servers are NOT about consistency, they are about making badly designed protocols, such as NFS, capable of limping, instead of just falling on their face.

If the requests on these protocols used a client timestamp for the client's idea of the current time, then the server on receiving the request could look at its idea of the current time, and arrive at a delta before it actually did anything other than enqueue the request locally.

Then when the server responded with a non-"now" timestamp in any client response, it could apply this delta to the response value, and as far as the client was concerned, it and the server would have synchronized ideas of "now", without resorting to all of this NTP BS or worrying about clock drift, or anything.

I lobbied very strongly to try to get this fixed in NFSv4; maybe we will get our collective heads out of our butts by NFSv5.

Are you all mad? What does improving NFS have to do with intentionally letting PC clocks drift?

Could I go out on a limb and suggest there are reasons besides NFS to keep clocks in sync? Wow.

Re:NTP servers are NOT about consistency

[GeniusDex]

So you say that it is best to solve this problem in each application inidividually instead of, say, running one process on each system which makes sure that the clocks stay in sync for all applications?

Re:NTP servers are NOT about consistency

[tlambert]

So you say that it is best to solve this problem in each application inidividually instead of, say, running one process on each system which makes sure that the clocks stay in sync for all applications?

No more than you appear to be saying that protocols should be implemented in each user space process instance, rather than once in the kernel and shared by all processes, I think. This is a protocol level problem, not an application level problem.

Re:NTP servers are NOT about consistency

[Anonymous Coward]

It's not that simple.

For a start packets don't always take the same time to arrive (thanks to buffers, and differing routes over a number of hops). The delta for one packet would be different to the delta for another.

NTP has to do a lot of work to continue sampling the differences to converge on the true delta.

Simply comparing timestamps in a single request is not sufficient.

Re:NTP servers are NOT about consistency

[adolf]

But if the protocol's time-dependency issues are fixed by an application, along with every other application/protocol's time-dependency issues, then fixing the protocol is superfluous because a functional system will already have a stable sense of what time it currently is courtesy of NTP. One cure for a thousand ailments.

Would you feel better about it if NTP were wholly integrated into the kernel? Why, or why not?

Re:NTP servers are NOT about consistency

[Anonymous Coward]

if nfs misuses ntp, that's on nfs. if you're using time in
your protocol as cheater shared state, you're doing it wrong.
so far we agree.

however, ntp does a pretty good job of keeping a consistent clock.
a consistent clock is good for preventing little irritations, and important
for logging which can be important for determining what went wrong.

Network team must haves.

[nevermindme]

Here are the things network teams needed to much for Virtualization scheduleds and are the first to come back up when the switch/router power is restored.
1 x GPS TIME SERVER MASTERING FOR ALL SWITCH ROUTERS to Provide NTP to the CLIENTS.
2 x TACACS SERVERS FOR SWITCHS , ADMIN VPN, RSA 2 FACTOR... BLAH.
2 x DNS SERVERS BEHIND THAT SERVER AS MASTERS ( to SLAVES BEHIND THE F5s)
2 x ADMIN VPNs (JUNO, CISCO ASA)
2 x CONSOLE SERVERS to everything.
2 x CONFIG SERVERS, IOS, DOCUMENTATION STORES TFTP FTP SFTP
2 x SECURITY SERVERS THAT LET YOU IN THE DOOR
n x HVAC, POWER MONITORING, GENSET TOOLS.
2 x SYSLOG SERVER with local HHD.
2 x JUMPBOX with REAL OUT OF BAND BANDWIDTH (CABLE MODEM, CELL MODEM, VT100 in office space....sucks to work in a dark HOT DC)
1 x TIMECLOCK with cardstock because this is going to take a while.


All should have the least number of transfer switches(evil beasts) and should be at the base of the A and B sides of the Power Plan.

Penny wise and pound foolish

[tlambert]

If by "big iron", you mean "IBM Mainframes or similar kit", then your question has meaning.

If by "big iron", you mean "lots of irritating PCs that I think I can add up into a supercomputer because all problems are amenable to parallel solutions", then your question is meaningless.

Assuming the second, you are much better off just using identical hardware for everything, since it will mean you have the components on hand should anything go wrong, and it will mean that you have a single maintenance SKU. In the long run, that's going to save you a hell of a lot more money than having one or two specialty boxes per rack of set of 8 or 16 racks, since it means you don't lose 8 or 16 racks worth of your "big iron" everytime one of your cheap little specialty boxes fails.

Why seperate boxes for tiny resource requirements?

[dbIII]

Why even have a dedicated server for NTP? It's not as if it's the bad old days of Win NT and one service per box due to memory leaks. If you've got special hardware for an external time source that can be hooked up to an existing server, because the actual software to hand out time consumes buggerall resources. It consumes so little that redundancy is a matter of just configuring whatever machines you've already got to be as many NTP servers as you want just at a lower stratum than whatever you really trust. They'll keep time reasonably well for a fairly long time while the custom time source is off.

If you can't rack it...

[funkboy]

...I don't want it in my datacenter. If you have no budget for non-revenue-generating boxes for services like DNS, NTP, etc. then upgrade the server hardware you tore out of production after the last upgrade cycle with SSDs and low-wattage processors & put it back into service for your internal needs.

Otherwise get a few Dell R210s or some other small cheap rack server with an IPMI 2.0 BMC and get on with your business. Any money saved by buying "mini-PCs" (or whatever you want to call them) for any datacenter computing hardware you plan to rely upon at all will be burned the first time you have to drive to the datacenter and physically babysit some cheap machine because it didn't have IPMI.

Re:If you can't rack it...

[swalve]

Just buy some rack shelves.

Re:If you can't rack it...

[Anonymous Coward]

The only thing that should EVER go on a shelf in a data center is a book.

And even that, SHOULD be in your office.

When we say "data center", we mean squeezing EVERY cubic inch of space into being useful. not leaving a "2U shelf" to hold a poorly cabled device or two that can't even exhaust it's heat properly.

Re:If you can't rack it...

[Bing Tsher E]

I have a rack-mount Tektronix Oscilloscope. It's all vacuum tube.

Don't wee all over yourself thinking about that.

"And they're all made out of ticky-tacky"

[Chris Mattern]

"And they all look just the same"

un-cloud

[drwho]

The use of discreet machines allows for a machine to be specialized for a task. Sometimes you just need fast number-crunchers for special types of numerical problems, and GPUs work well. Other times, tasks can be parallelized so a distributed computing model works well. For the accessory infrastructure of NTP, DNS, and forth, reliability is more important than CPU mips or memory bandwidth. Take the high-end servers of yesteryear, one that would have been put out to pasture, and use that for such things. Debian on an old HP LP1000r makes a nice DNS server. BUT make sure that things which actually wear out, such as hard drives, batteries and fans, have some redundancy. If a battery is used, replace it with a fresh one. Unless it has been in a well-filtered environment for its pervious lifem vaccum the insides and then apply some grease for the fan motor bearings (in that order, not the reverse) to keep the beast going for another ten years or longer.

Install an OS which is well-proven and a version that will receive updates for a long time. For instance, if you are running Ubuntu used the LTS releases. Install only the software needed for the task, and simple remote administration and updating - you probably don't need a compiler, web server, etc. Keep up with critical security patches, automating the task if you can. rsyslog to a remote machine, just the messages that are important. Make sure that logs are rotated on the machine, so that the drive won't fill up.

Create a maintenance calendar. login and check up on the system to be proactive about degradation issues - at least once a year, I'd say. Your server room should be filtered, but that doesn't mean it's not worth checking to see what the dust level is every few years. Make sure the machine is properly labeled and documented - so that no one comes along later and says "What's this old thing still doing here?", and unplugs it.

Write 'Kilroy was here' and the date on the inside of the case, so that future generations will be amused when they come to clean out the dust in fifty years.

SOLVED: Little Boxes

[Jeremiah Cornelius]

Answer: VMware VMs.
 

Re:SOLVED: Little Boxes

[cerberusss]

Answer: VMware VMs.

Yeah, but when virtualizing, I'd put the virtualized DNS servers on their own exclusive hardware.

Re:SOLVED: Little Boxes

[Anonymous Coward]

As a VMW support engineer, this was my first thought.

They have been virtualized.

Re:SOLVED: Little Boxes

[philip.paradis]

I use KVM on Debian hosts for all my production stuff, but yeah, my first thought was "those servers are all virtualized now."

Re:SOLVED: Little Boxes

[philip.paradis]

That doesn't seem to make any sense. The whole point of multiple DNS servers is to have them distributed across separate hosts and preferably separate networks, but putting them on exclusive hardware doesn't provide any gain unless you're handling a ridiculous number of queries per second (in which case there's no point to virtualizing the servers in first place). That is, unless you actually meant to say you make sure the second sentence applies, instead of potentially saying no other VMs run on hosts that happen to host your DNS servers.

Re:SOLVED: Little Boxes

[Anonymous Coward]

For an NTP server? I dunno know why, but alarm bells are going off inside my head....

Re:SOLVED: Little Boxes

[Anonymous Coward]

Two things:

1. You can't virtualize an NTP server
2. I always keep my DNS resolvers on physical hardware, because otherwise you have a bootstrap problem. Starting VMWare without DNS in order to boot your DNS resolvers really sucks. If they're physical, I just have to push the power button.

Re:SOLVED: Little Boxes

[A bsd fool]

Right on the NTP virtualization (which is irrelevant), but wrong on the "bootstrap problem". I run a two private mini-DCs, one fully virtualized, the other almost. In the "almost" DC, only the pfSense box is not virtualized. It handles DNS caching, firewall duties, VPN access, and DHCP. In the second DC, even pfSense runs in a VM. The "trick" is to use the tools you have -- set the VM startup order so the VMs responsible for DNS are started first, or at least soon enough to be up before the VMs that rely on them. The ESX servers themselves do not need DNS for anything. NTP on the VMs is irrelevant. The hypervisors will do NTP to keep themselves synced, and the VMs sync through the (always installed, right?) VMWare tools (or open-vm-tools) since even running an NTP *client* in a VM is problematic and ultimately pointless.

Re:SOLVED: Little Boxes

[Anonymous Coward]

You can do that (bootup ordering), and you're right that ESXi doesn't need DNS to operate: but vSphere does, and managing a cluster of ESXi hosts without vSphere is a pain in the ass, especially if, say, the cluster was powered off ungracefully and vSphere is now refusing to boot.

When you have a cluster of ESXi servers, an MS-SQL server, a vSphere server and a cluster of iSCSI SAN appliances that all need to be booted before your VM cluster is fully operational, I prefer the vast simplicity of having two physical servers that I power on first.

NTP on the VMs is irrelevant. The hypervisors will do NTP to keep themselves synced

I'm talking about NTP servers for the other hosts to sync to. You'd be crazy to run those on virtual machines and expect them not to drift like crazy.

Re:SOLVED: Little Boxes

[A bsd fool]

I assume when you say vSphere you actually mean vCenter? vSphere does not need DNS to work, it runs just fine with IPs. If you've added your ESX hosts to vCenter via hostnames, then those hostnames are going to be needed for you to manage them through vSphere (which manages them in turn through vCenter).

I do have a cluster (3x) of FT/HA ESX servers in one of the DCs, all booting from SD cards with long term storage on an FC SAN, and I have had no problems virtualizing the entire infrastructure. iSCSI may make things a bit tricky if you're using names rather than IPs, which is something most of us prefer to do -- in fact I use DHCP for just about everything, with very few static MAC IP mappings, and names for everything. I don't see the "vast simplicity" of having additional hardware outside the purview of my main management system (vCenter) though.

I also never "power on" the servers. They were powered on the first day, and except for memory upgrades, have been powered on ever since. Updates are accomplished quickly and easily via the vCenter Update Manager, which aside from being slightly annoying unless you have the expensive license, works great. With a normal standard/enterprise (or whatever they are calling it now) license, you must manually vMotion the powered-on VMs to other hosts in the cluster, which is just a minor issue.

I suppose if you're powering your entire infrastructure off and on all the time you may benefit from having a machine or two outside of the VMs, and of course some machines like heavy-load databases will always do better on bare metal, but for most (the OP included I think), the answer is simple: Virtualize everything. That said, I do virtualize everything, even the DBs. What little overhead this costs is more than made up for by the advantages it brings.

Regarding NTP, I still "don't get" what you mean I guess. My ESX hosts sync to the normal NTP pool, and they are the only machines that need to use NTP. All the others are virtual and so sync via the vmware tools and not NTP.

I will say there is something inherently "wrong" about the ESX hosts themselves being firewalled by a VM running on them -- but once you embrace the madness, it's beautiful in a rube goldberg sort of way, and the benefits far outweigh the potential drawbacks.

Re:SOLVED: Little Boxes

[Anonymous Coward]

vSphere needs DNS if you install it with an external database server (Which I have). Yes you can get away with never requiring DNS to start your VMWare cluster, and I've done it, which is why I've decided it's just less effort and pain to have two physical DNS servers instead, which makes it a non-issue entirely.

I also never "power on" the servers. They were powered on the first day, and except for memory upgrades, have been powered on ever since.

I tend to plan for the worst case scenario, which is a restart from a dark data center. Given that a hurricane just passed awfully close by one of them, that seems like a valid assumption for me to make.

Regarding NTP, I still "don't get" what you mean I guess. My ESX hosts sync to the normal NTP pool, and they are the only machines that need to use NTP. All the others are virtual and so sync via the vmware tools and not NTP.

I have a couple of thousand physical servers. They very much need to sync their hardware clocks via. NTP. I need reliable NTP servers. NTP running on a virtual host is not reliable (the clock drifts horribly, although ESX5i is better in this regard).

Re:SOLVED: Little Boxes

[Guspaz]

I have a couple of thousand physical servers. They very much need to sync their hardware clocks via. NTP. I need reliable NTP servers. NTP running on a virtual host is not reliable (the clock drifts horribly, although ESX5i is better in this regard).

A host running ntpd against an external source won't drive at all, virtual or otherwise. Presumably you have to run your own NTP server because you want to save the transit costs of all your servers making requests to external NTP servers. In that case, one of your servers would be synchronized to the root pool, for example, and the rest would synchronize to that one.

If you're running an NTP server that is itself not synchronized to a higher level external NTP server or some physical external clock, that NTP server is going to drift even if it's running on real hardware.

Re:SOLVED: Little Boxes

[Anonymous Coward]

We do this all the time where I work for planned or unplanned power outages. EVERYTHING is virtualized. Including virtual center, our domain controllers etc..

When power is brought back on, someone plugs in or turns on the routers and core switches. 99% of the time, they come up and I connect in through the WAN. If not, we can get in through a modem on an alalog line our have someone patch in the DSL and troublshoot. The person on site then powers up the SAN. I log into the SAN and make sure it is up. I then use ilo into ESX servers and power them up. I connect directly to the ESX server that had the domain controller and the DHCP server running on it with the vSphere client. I verify it has storage. I power on a domain controller and the DHCP server. Then other random servers or ones that may require power before other ones. Eventually the vCenter server comes up and I disconnect from the ESX server and connect to vSphere, reenble DRS and HA and my job is done. Our phone guy connects to the call center server and verifies the phones came up and registered correctly with the offices server. The person on site fires up desktops, copiers, and printers as needed. Piece of cake.

Re:SOLVED: Little Boxes

[Jeremiah Cornelius]

VCenter should be isolated in its own "management cluster" if you virtualise it.

You could do this with vShield - but there are really different operational characteristics for a NOC from the actual Data Center itself...

Re:SOLVED: Little Boxes

[Anonymous Coward]

Er, yes, but in between my stratum 2 servers synchronising with the stratum 1's I'd really rather prefer that they didn't drift +/-2 seconds in the hour. That's the kind of thing that really upsets the other servers.

Re:SOLVED: Little Boxes

[Guspaz]

It won't drift that much with ntpd, since it will try to correct for drift with adjfreq. My VM drifts by minutes per day without ntpd, but with it, I've not seen it get more than 5ms off between ntpd updates.

Re:SOLVED: Little Boxes

[A bsd fool]

vSphere needs DNS if you install it with an external database server (Which I have). Yes you can get away with never requiring DNS to start your VMWare cluster, and I've done it, which is why I've decided it's just less effort and pain to have two physical DNS servers instead, which makes it a non-issue entirely.

Sure, but now the DNS servers are not virtualized, which is counter-productive. There is no downside to having DNS (and, indeed, nearly everything) virtualized. So long as you get the boot order and other cluster settings right, all will be fine. Additionally, since vCenter is a "windows land" thing, there are alternative serverless options for name resolution such as NetBIOS. I'm not saying any of these are the preferred method, but pointing out that DNS is not "required" no matter how you slice up your VI. Personally I use static IPs for the most important infrastructure stuff, which absolutely "makes it a non-issue entirely" in a way that external or virtualized DNS hosts cannot.

I also never "power on" the servers. They were powered on the first day, and except for memory upgrades, have been powered on ever since.

I tend to plan for the worst case scenario, which is a restart from a dark data center. Given that a hurricane just passed awfully close by one of them, that seems like a valid assumption for me to make.

Even restarting from lights out, there is no issue virtualizing DNS if done right. I plan for worst case (who doesn't) but I also recognize that the worst case scenario is rare, so the everyday case is the one where ease-of-management enters the decision making process. Load balanced lightweight virtualized DNS is the better option for day to day operations, and does not introduce any real headaches. It sounds to me like you just didn't do it quite right last time you tried.

Regarding NTP, I still "don't get" what you mean I guess. My ESX hosts sync to the normal NTP pool, and they are the only machines that need to use NTP. All the others are virtual and so sync via the vmware tools and not NTP.

I have a couple of thousand physical servers. They very much need to sync their hardware clocks via. NTP. I need reliable NTP servers. NTP running on a virtual host is not reliable (the clock drifts horribly, although ESX5i is better in this regard).

The discussion was about virtualized servers needing NTP, not physical ones. Of course physical hardware needs NTP. Virtualized hardware does not. VMWare has better ways to keep guest time in sync.

Re:virtualization is the game now

[volxdragon]

That is assuming you can get a GPS signal to where your computer is located - I've seen absolute CRAP reception in many buildings (even putting the GPS receiver ON the inside of the windows, if there is film on them, you commonly will see zero or only 1-2 birds in the constellation, not enough to get a consistent lock for most receivers) and running a pulse trigger or USB down from the roof usually isn't an option (although sometimes it is).

Re:virtualization is the game now

[Anonymous Coward]

Have you noticed how slashdot is getting stupider?
Why does it suck so much nowadays?

Maybe it's because moderation declines fast when there are less eyeballs per story, or something...

2 - 3 redundant "big iron" with VM's

[Fallen Kell]

Use 2 or 3 redundant low power enterprise class servers. Setup vmware or similar with automatic failover, and make all those "little boxes" into virtual machines. The benefit is that you can easily rehost the services even to your production vm solutions in an real emergency. Having them as separate VM's gives the same benefit of having them as separate little boxes (i.e. restarting the ntp server only affects ntp services, not you email as well). You have the added benefit of being able to easily deploy upgrades by simply cloning the existing VM, patching/updating the service in the clone, shutdown the original VM, and if you have a problem, simply turn the original one back on.

We use a lot of VMs for this kind of thing now

[enjar]

One-off boxes become a huge time sink, usually at the absolute worst possible time to do so. With two very viable options with Xen and ESX, put the time and care into setting up a stack with the nifty features you want -- redundancy options, ability to move VMs from one server to another, monitoring, out of band management, RAID, etc.

Then you can set up the little management hosts, set up a VM for each one of those "little things", and also come up with a single way of deploying your operating systems so you can punch VMs out on demand. Both Windows and Linux have ways of doing this, and you can even script VM creation, too.

VMs also let you pretty much side-step the drive issues that will plague old hardware ... that will show up at the wrong time.

Anything Mac is a joke for data center operations since they considered the XServe. You shouldn't need to go to the colo just to power-cycle something, ever. Hardware failure will get anyone (a dead drive or a bad DIMM can ruin anyone's day), but if it doesn't have IPMI/iLO/whatever you call your out-of-band management tool, it shouldn't be used for any infrastructure service at all. You want to have a mac mini set up as a user terminal to check email or whatnot, fine. But don't stick NTP or DNS on hardware that you can't full control elsewhere.

Re:virtualization is the game now

[sfprairie]

We have our core routers sync with public NTP servers (NIST and Naval Observatory). Everything else (servers, phones, ect) goes to two sets of Core routers for time. Dont' see the need for a separate box for NTP.

Re:virtualization is the game now

[green1]

If you care enough to use a GPS receiver instead of a network time source, you should also care enough to get the antenna on to the roof... We have many such time sources controlling timing in the basements of buildings, but the antenna always ends up on the mast.

Re:virtualization is the game now

[marcosdumay]

NTP is not real-time, so a few ms here or there of delay is not a problem.

A few ms on incomming and outcomming lags won't hurt it at all. A few ms on incomming lag without a few ms on the outcomming lag (like what happens when your VM is sent to swap) will completely destroy it's accuracy.

Don't forget to monitor from offsite too

[DrHappyAngry]

NTP is probably a bad idea for virtualization, but most other services it's fine. You can have redundant VMs, or at least snapshot them to another host, to bring them up easily in the event of a failure. And as has been said, you can stack some services into the same instance, For example you could stick munin, mrtg and nagios on one VM. If you're serious about your monitoring though, you've got a monitoring system in there, plus one offsite. This way if the internet connection to the rack goes down, or the monitoring system in the rack goes down, you'll get notified. A distributed nagios system would work well for that. You could even get a cheap micro instance at AWS for the offsite monitoring box, or just run it from the office.

Why not hypervisors?

[SignOfZeta]

I don't operate a datacenter, but for virtualized servers in an office, I always enable the NTP server functionality in the hypervisor, have it sync to a stratum-1 time source, then advertise that address via DHCP and DHCPv6 for my guests and workstations (and visiting cell phones) to use. Being the definitive time source, I also tell the hypervisor to automatically set the clock on the guests, then give a virtualized AD domain controller (if any) the PDC FSMO role to set the Windows domain time. I have sites with two or three hypervisors running NTP, and it seems to work well. Not sure if it will scale to your environment, OP, but it may be worth mulling over.

Re:Why not hypervisors?

[Anonymous Coward]

have it sync to a stratum-1 time source.

Why?

Isn't this like setting up your name servers to query the root servers, but worse?

Re:Why not hypervisors?

[Anonymous Coward]

So, you're saying that you allow visiting devices to talk directly to your hypervisor? Unfortunately, no, that would not scale.

Re:Why not hypervisors?

[Anonymous Coward]

Though with several hypervisors, if you can juggle the order of NTP servers in DHCP responses, this could scale quite well. Of course, you'd have to firewall them to make sure that only NTP is usable by all but the management stations.

time won't be accurate

[Chirs]

The NTP algorithm tries to characterize each time source and use the predicted network latency to arrive at the "true" time. If the NTP server response latency is delayed randomly due to running in a VM then it gives results that are not as accurate.

If you're trying to correlate logs down to the microsecond on multiple machines this can be a problem.

Re:time won't be accurate

[AK Marc]

I guess my problem is that I'd never have 5 servers talk to an NTP server (unless that separate NTP server was needed for GPS level accuracy), I'd just put NTP on one of the 5, and have the other 4 sync to that. Simpler and more reliable than having them all talk to some external source.

Re:time won't be accurate

[Reschekle]

I hope you realize that the internet is going to have much higher latency spikes than your hypervisor will (unless its badly configured or extremely overcommitted).

If you're trying to "correlate logs down to the microsecond" then you should either be using a local time source or should be getting your time from a nearby source on the network.

I'm not sure you understand how hypervisors or NTP really work.

Soekris as serial console

[Anonymous Coward]

Same here, they do great serial console. The machine has no moving part, so it should be quite reliable over time. The weak point is the power supply, though: we replaced a lot of them

We use them with a tweaked NetBSD that boot from flash and uses a RAM disk as root

That's BS

[Anonymous Coward]

If your hypervisor is "randomly delaying" your hosts then you have a deployment problem that you need to fix. That's not how it should work unless you've given your NTP VM the lowest possible priority.

You make several posts on here about this being a bad idea but you have absolutely no data or citations to back it up.

Re:virtualization is the game now

[Anonymous Coward]

Or you could get one of these: EndRun Technologies CDMA Network Time Server
I bought 2 of these 7 years ago. They use time from local cell towers. Accurate to better than 10 microseconds. No cell phone account required. They are pretty much plug and play. I disabled network login (serial port login can be used). No exposure to hacking, the only thing it responds to are NTP requests. Options for TCXO, OCXO and Rubidium clock backup in case of loss of cell towers. Got a cell signal on 6th floor of 13 floor building inside interior computer room (no windows, at least 50 feet from nearest exterior wall).

Re:virtualization is the game now

[Anonymous Coward]

then reserve memory and CPU for the NTP server, and it will never be swapped.

Re:virtualization is the game now

[Anonymous Coward]

If only there was a way to manage memory in a hypervisor! Oh wait there is!

Pretty easy to avoid swap if you are capable of reading documentation.

Re:virtualization is the game now

[Reschekle]

Curious: have you actually tried virtualizing NTP or do you just think its a bad idea without any experience?

I virtualize ntp for UNIX (build cluster and lab farm) and domain controllers for Windows (which acts as a time server for our 5000 or so desktops at work). Both Microsoft and VMWare explicitly support doing timekeeping functions in their associated hypervisors so long as you follow their guides. No problems whatsoever.

only partially true

[Anonymous Coward]

There is also the cost implication.
many companies charge you licenses based upon the CPU Capacity (speed and cores). Running an application in a 2 CPU VM will often save you several hundred thousand $$$$ when compared to the 16core monster used in the underlying system.
IBM, Oracle, SAP etc all seem to use this pricing model.
Mind you waiting until your suppliers Q4 to make the purchase/renewal can also save you loads-a-money as well.

Re:virtualization is the game now

[profplump]

Yes. Cell towers are often much easier to use (better building penetration, more visible sources, etc.) and are at least as good a timing source as the average GPS receiver (stationary transmitters at near-field ranges). Even if you don't trust them to have the right time they're highly reliably oscillators; if they weren't it would be impossible to synchronize phones to them.

Shouldn't even want to virtualise everything.

[Let's All Be Chinese]

Virtualisation is great, but there are a few things that cause horrible chicken/egg problems if you virtualise them.

So I'd reserve at least two separate boxes to "do infrastructure". DNS, NTP, remote logging, trap receiving, bastion, and so on. You simply plunk a unix on them and put the individual services in jails or the local equivalent. Don't even need much in the way of performance, so any old 1U box will do fine. Heck, a soekris or an alix board will do. Those are short enough that you can stick'em in any old wiring closet too. Great for geographically dispersing.

If you're stumping up for infrastructure that can host hundreds of VMs, then of course that is enough capacity to also run "little boxes", but it'd be stupid to not also shell out the little extra to make your infrastructure robust, instead of risking hypervisor dependencies on not-yet booted VMs in your private cloud, or whatever you'd call it. "Seems to work" is not enough: Turn off the entire datacentre and then try and cold boot it, remotely. If it's fully virtualised including necessary basic supports, it'll take more time and trouble than if you don't virtualise the pillars on which you built up the rest.

If all I had was exactly two boxes, I'd still run NTP and local DNS next to the hypervisor, not under a guest. NTP in particular; I've had my fill of (windows) boxes claiming to be stratum two yet being off by two minutes because they only update once a week. Of course, on a virtualised unix it'll be much less, but I don't want to find out the hard way the VM distorted the timekeeping in unexpected ways later, so this is one thing that needs its own box. There are similar scenarios for the other basics, but I'll leave them as an exercise. The gains of virtualising, saving a bit on hardware and power, simply do not outweigh the trouble when you can least afford it.

Re:Shouldn't even want to virtualise everything.

[larppaxyz]

Heck, a soekris or an alix board will do.

I know this is not related, but failure rate for soekris boards are close to 100% in three years. I'm not sure if problem is with power unit that they use, but very soon everything starts to fail and magic smoke comes out of soekris.

Re:Shouldn't even want to virtualise everything.

[drwho]

Yes, you're pretty correct, there. I am not sure why these boards fail so quickly, or that they don't last over a few years, but they don't have the same build quality that real servers use. That's why I recommended something like an LP1000r in my previous post. Other than his choice of hardware, the poster of the grandfather of this message is correct: you shouldn't virtualize everything.

Careful

[wonkey_monkey]

The little boxes will make you angry.

Re:virtualization is the game now

[adolf]

If you care enough to put an antenna on the roof, then you should also care enough to pay attention to good grounding principals.

Grounded coaxial-fed antenna on roof == lightning rod. Period.

Without precaution and planning, the device responsible for dissipating that lighting (when, not if, it happens) will be your precious local Stratum-1 NTP source.

It's never hard to get grounding done right, but it's not always obvious, and it never happens by itself.

The hairs on a zulu warrior

[Hognoxious]

There's a black one,
And a white one,
And one with a bit of shite on.
But you can't put your muck in our dustbin by the ash grove.

Or something like that.

Don't use power hungry obsolete junk...

[sidetrack]

Oh yeah, also consider OpenVZ VMs (OK for ntp too), or libvirt/KVM VMs. The Dell R210-II will quite happily run a load of those...

Re:Why seperate boxes for tiny resource requiremen

[lamber45]

For bootstrapping and security, I imagine. If there's a cold outage, or an extended spike in network traffic, or a misconfiguration on a switch that blocks all network traffic for a few minutes, a few services will need to be working without depending on anything else when everything else is brought online. That might be master NTP, master DNS, master LDAP, or as stated monitoring (so you can see what actually went wrong in one place). And you could run all of them on one box, with two or three similar as backup, but the point of the question is that you don't need a 64-CPU SPARC box for those services even in a large datacenter; and even if you ran it on a 4-CPU x64 blade, that would be harder to find in the dark or with alarms going off than a standalone box.

Re:Just the same.

[hoboroadie]

If the people weren't made of ticky-tack, maybe they would have planted one or two fucking trees as prophylaxis against bleeding eyes.
Seriously, it's been 60 years, Millions of people look at that every day, fix it already.

Why think when you can Ask Slashdot?

[Zeromous]

This has to be the most obtuse Ask Slashdot in a long time.

Has OP been under a rock?

Re:virtualization is the game now

[Zeromous]

You know how I know you know fuck all about Virtualization in 2012?

boxes?

[nosilver4u]

I don't run anything close to a datacenter, so this probably is not applicable to anyone with more than 20 servers, but I run my ntp service on parts attached to a wooden 2x4 (just because I can). It's an arm 800 MHz cpu and keeps very accurate time. Most of our DNS and DHCP is on sheevaplug style 'boxes'. They don't keep accurate enough time for me...

Whatever's laying around

[Vrtigo1]

Last time I worked in a company that actually had a for real datacenter was about 13 years ago, and they just used whatever was lying around. Most of the support boxes were HP desktops or UltraSparc workstations that had been repurposed. All of their critical infrastructure stuff was running on those refrigerator-sized Sun boxes that I wasn't allowed to touch.

I don't want to be that guy, but...

[Laebshade]

Why not use a Raspberry Pi as an NTP server?

datacenter cruft

[drwho]

I've worked in many datacenters over the years. I want to follow up on my previous posts, where I recommend discrete, reliable legacy boxes for NTP and DNS. I want to make it clear that I don't think you should just pick up any old spare box, throw an OS on it, and be done in half an hour. That may work at home or in your hackshack, but in a professional environment, it isn't good enough. These services are part of the foundation upon which the datacenter is built. If the foundation is weak, no structure built on it will have a long life. These services and the machines that provide them are noticeable only in their absence, when all hell is breaking loose. Running a good datacenter requires proactive maintenance and planning. Just because things are running smoothly is no excuse for the operator to play endless games of solitaire waiting for a drive to fail so he can hot-swap it. Everything needs to be planned, documented, updated, and monitored. consumer-grade hardware just won't make the grade in critical infrastructure. Sure, that old PC MAY last twenty years, but that is not good enough. In a major data center, downtime is lost money, often lots of money. Spend money now to save later. Oh, I know, politicians these days want to cut down infrastructure spending so they can lower taxes and balance the budget deficit, but type of thinking leads to rash decisions that are penny-wise and pound-foolish.

Think of a fire station. Those guys just don't sit around playing cards, waiting for the next fire. They spend time at the station making sure their equipment is clean, in good shape, and exactly where it needs to be so when the call does come, they can perform. Run the datacenter in a similar way.

Re:virtualization is the game now

[green1]

GPS antennas don't tend to stick up any further than any of a number of other protrusions on the roof of a normal building. If your building doesn't normally get hit by lighting, then the GPS antenna will not change that in any way.

That said, proper grounding is always important and I would never argue against doing so.

Re:virtualization is the game now

[ls671]

Not everybody is time maniacs. I have seen many cell towers off 1000ms. They have highly accurate oscillator but the ntp time sources go out and nobody updates the cell tower ntp client for a year or more. They usually update it when it gets above the 1000ms threshold although because things start to screw up.

Are you guys are saying I could still use them as a tick source even if they are off? Sounds interesting...

Security Logging.

[attemptedgoalie]

Security logging.

If EventX happens on Box1 and EventY happens on Box2, I'd like to see which happened first, etc. I can correlate that with networks sniffs, firewall logs, etc. If all are on damn near the same millisecond, then I can walk the trail. If one is 3 seconds off, or a minute off, etc., it gets fuzzy.

If DoorA opens at Time1 and CameraX sees something at Time2...

If I have two GPS time boxes (with two weeks of time retention/accuracy in case of signal loss), I can have something that should stand up in court.

If I have a home built box, or hope that pool.ntp.org was working perfectly as well as my connection to it, during a time that an event happened that puts us in court, it might not stand up.

That's rare

[marcosdumay]

What an argument you came out with!

I can't say that never happens, because it does... Mutually exclusive librares are already rare, you needing two versions of them at the same time... I've seen it like once or twice, never on a server (I've installed Linux way more times on desktops than servers, so that's not unexpected), and can be mitigated by choosing a stable distro like Debian or just recompiling the one package that is giving you trouble.

In my experience, it's way more common that a service refuses to share a port than library incompatibilities.

Re:That's rare

[badkarmadayaccount]

OpenVZ

Re:virtualization is the game now

[adolf]

It doesn't matter how far it sticks up.

For that matter, it doesn't even matter if it's a direct hit: Inductance is a bitch who will fry the strangest things, given an opportunity to do so.

That you seem to think that just because it doesn't stick up further than other protrusions makes any meaningful difference in the context of the reliable systems that are the entire purpose of TFS means that you don't fully understand the concept.

Over here in the real world, things aren't so cut-and-dry. Lightning is not a completely rational phenomenon, and one must take extraordinary means in order to reliably survive its appearance -- especially with outside antennas.

Meanwhile, please don't hang a GPS (or any other) antenna on any building in which I have gear that I am responsible for until you learn another thing or three: While I'd love to have a local stratum-1 timesource, I don't want your shit breaking mine.

Re:virtualization is the game now

[green1]

All I can do is laugh... I agree proper grounding is important, as is surge protection. I also know for a fact that you are completely out to lunch with your paranoia.

Re:virtualization is the game now

[adolf]

No. I just work with antenna systems for a living. I have seen my share of gear that has been fried due to crude assumptions about lightning.

Please Motorola R56 a read before you accuse me of being paranoid.