Diskless Booting For the Modern Age

An anonymous reader writes "Ever wonder what happened to PXE? Intel's popular standard for diskless booting hasn't been updated since 1999, and has missed out on such revolutions as wireless Ethernet, cloud computing, and iSCSI. An open source project called Etherboot has been trying to drag PXE into the 21st century. One of their programmers explains how to set up diskless booting for your cloud, using copy-on-write to save space."

Comments

How is it slow?

[OrangeTide]

TFTP over UDP on a LAN, doesn't seem slow at all. It's stupid, but sufficient to bootstrap a small kernel to access the real meat of your OS. 1-10MB TFTP downloads over 100mbit is no big deal. You can't get good 1gbit performance (let alone 10gbit) out of the dumb drivers in a PXE boot ROM, but that's OK.

Re:How is it slow?

[emj]

Apparently PXE only allows a bootloader payload of 32KB, which makes it slow, and these days it wouldn't be a problem to fit a much more competent bootloader in flash on the mother board. So the question is why should I have to boot/download a 4MB Linux kernel to be able to use the 1Gbps drivers?

Re:How is it slow?

[improfane]

Is it possible to use multicast? The multicast could run continuously or on demand. Can you route multicast?

It would be pretty impressive if one could attach 3 nodes to the network and they all boot up.

It would be a Borg network. Everything you add to it would be immediately taken over!

NODE (DHCP): Hi I'm a Dell workstat-
BORG NODE #1 (x.x.3.43): We are borg
NODE: ACK
BORG NODE #2 (x.x.3.44): We wil add your likeness to our own
NODE: RST RST RST
BORG NODE #3 (x.x.3.40): Resistance is futile
NODE: Downloading borg....
BORG NODE #2: We will add your likeness to our own (as soon as you have downloaded. ...
BORG NODE #4 (x.x.3.45): We are borg.

Re:How is it slow?

[imsupporting]

Multicast would not work well for something like this. The problem with multicast is it doesn't care if you dont get all the packets. So if your network looses one packet, your whole boot will fail and will have to wait for the next multicast start.

Re:How is it slow?

[improfane]

What it be possible for a multicast to work with unicast corrections?

The receiving node must be aware when a packet is not received, if it is not at a packet level, perhaps at a more logical level like a block of data. Rather than wait for the next multicast, it explicitly requests a retransmission of that block to that node via unicast.

I am of course assuming that only a few packets would get lost.

Or perhaps another corrective multicast that streams corrected packets when enough people request them.

MULTICAST
degrades to multicast with corrective unicast
recovers to corrective multicast

I imagine this would be beneficial when a piece of hardware generally botches packets, such as a switch. If that ever happens in practice, I have no idea.

Re:How is it slow?

[Junta]

Well, one issue is boot payload is getting bigger and bigger. One distro has about 20MB of download that would be tftped in the default case. Windows uses tftp for a *lot* more.

tftp has the following issues:
-16-bit block indexes. Most firmware won't go above 1400 or so blocksize (with good reason, if they go higher and the network is set for jumbo frames, transfers will fail in many scenarios). This means a cap of about 98MB before you overflow the counter. Most tftp servers nowadays can deal with it in a unicast case, but it's not technically fixed in the spec.

-tftp exhibits a sort of half-duplex character with regards to transmits and block acknowledgement. Server sends 1.5 kilobytes, then does nothing, client receives the block, and only then does it request the other one. Compare to TCP windows in ftp and http and the differences are massive.

Re:How is it slow?

[Junta]

Issues:
-tftp multicast is inherently limited to smaller than 98MB images with sane MTU. The same block number wrapping in unicast can't work in multicast. When you want speedup the most, tftp multicast can't even work
-multicast only buys you something if a large number of clients are acquiring the same payload at the same time. In a large scale 'cloud 'configuration, things are generally heterogenous enough to negate any such hypothetical benefit.
-Most ethernet fabrics are either incapable or not configured for IGMP/MLDv2 snooping required to properly scope multicast resulting in all multicast traffic degrading to broadcast. This has very adverse results unless every entity on the network only cares about the transfer.

Re:How is it slow?

[guruevi]

Yes, Apple has been able to do it for a couple of years now (since the PowerPC era).

Basically a DHCP server says: I have a boot image
Client1 says: thanks - starts downloading 0-10%
Client2 comes on: starts downloading 10%-100%
Client1 continues downloading 10%-100%
Client1 boots up
Client2 requests 0%-10%
Client2 boots up

I think your example involves P2P but unless your client also has the boot image and a mechanism to give it to others after it has booted (which could be potentially a security risk) it doesn't work that way.

Re:How is it slow?

[Cylix]

Well that would be because PXE is pretty ancient stuff.

I have a gPXE deployment on my home network which screams like a banshee on roller blades.

Gigabit speeds, transfers over http and a good deal more configuration options.

With gPXE you can bootstrap a machine from Taiwan over the wan. The kicker is you still need a local agent to serve the tftpboot portion.

Beyond speeds the main benefit I find with gPXE is the ability to now load balance the kickstart servers. Even dhcpd has some features built in to assist with this which is very handy when you depend on those tftpboot services.

Now that said the documentation for gPXE is worse then PXE! The heavier weight rom can also be problematic for some cards, but I haven't had time to completely kill all of my personal bugs for a bit.

Re:How is it slow?

[Daengbo]

Back in the 90s, there used to be a proprietary-but-free scriptable boot system that used multicast (the name started with a "B"). You would boot each computer you wanted to do an installation on, set them to receive multicast, then once the computers were all connected to the server you had put DHCP, TFTP, and FTP on, the server would multicast an install image and a post-installation script.

I seem to remember that this was before Ghost has multicast.

Re:How is it slow?

[mawhin]

Are you referring to BPBATCH (Free as in beer)? That became Rembo (Not free, but we got a forever unlimited site licence for some few thousand pounds...)? That is now IBM Tivoli Deployment Manager (Massively not free)? We've used it for ten years, and it's excellent.

Re:How is it slow?

[ModernGeek]

So what you're telling us is that the solution is more XML?

Re:How is it slow?

[Anonymous Coward]

this is really not a problem, if you can convince
the "node" to press the "borg key" e.g "F12" key
or "F8" or whatever key that initiates the netboot process ...

Re:How is it slow?

[Anonymous Coward]

the real payload allowed is generally 350-400k. pxe could be improved quite a bit
by not being stuck in a 16-bit mindset.

Re:How is it slow?

[Daengbo]

Indeed. That was it. Wow. Thanks for that. I couldn't remember the name. Interesting that it went on from being a fairly small project to an IBM product.

I guess that KA-Tools became a popular replacement for BPBatch, but KA was still really immature back when I last used BPBatch. Now KA just seems abandoned, probably in favor of gPXE.

Re:How is it slow?

[larkost]

The mechanism you describe is only used when imaging a computer, not durring the netboot phase that is often used to deliver the new image. It is really nice if you can run broadcast ASR, but like most multicast has some real limiations when crossing subnets. It is also a bit of a bear to set all the variables so that you get good speed without taking down a network. But once you get the combination for your network it really works out well.

Re:How is it slow?

[Anonymous Coward]

most TFTPs support wrapping of the 16-bit counters. I've pushed 500MB over TFTP clients before.

half-duplex is perfectly fine on a LAN where latency is extremely low.

It's like people are trying to optimize the wrong thing. You aren't serving up enterprise websites over TFTP. You're just trying to bring a machine back up in your cluster so it can access the SAN/NAS again.

Re:How is it slow?

[evilviper]

Well, one issue is boot payload is getting bigger and bigger. One distro has about 20MB of download that would be tftped in the default case. Windows uses tftp for a *lot* more.

This is nonsense. Nobody in their right mind loads up the boot image in the first stage. The first stage is to download syslinux/grub/etc via TFTP. From there, you have all the features included in the boot loader of your choice, and can ignore PXE limitations. Want support for HTTPS? Good idea! Go include it in your boot loader.

The only reason PXE may need to be updated is because of IPv6 addressing.

Re:How is it slow?

[Junta]

As I mentioned above, most tftp clients go above and beyond the spec and for unicast transfers can outdo the spec (multicast a no-go).

half-duplex is not perfectly fine when there is a relatively accessible better technology.

http (or perhaps more appropriately ftp) are perfectly suitable protocols. The technology has progressed sufficiently far that a TCP stack is feasible in firmware, and there's no point in not doing it.

Re:How is it slow?

[Junta]

The thing is, syslinux/grub/etc all make PXE calls to retrieve files, which is TFTP. gPXE as a firmware (or as an undi driver delivered via PXE) gives syslinux http protocol support.

Since you mentioned IPv6, the DHCPv6 RFC is just about to come out, and specifies a url syntax for protocol flexibility, as well as recommending http as the protocol of choice (FTP kinda gets the shaft...).

PXE/DHCP/AoE

[Anonymous Coward]

I did a similar thing a while ago with an Ubuntu desktop image and ATA over Ethernet. Worked nicely. Didn't get round to COW though ...

    http://www.s-mart.net/aoe.txt

Re:PXE/DHCP/AoE

[MrEricSir]

There's always GOAT or even SOY.

Re:PXE/DHCP/AoE

[grantek]

And that gets to the point of answering the question "what ever happened to PXE?" - it got replaced by virtualised storage for booting virtual machines, running on a fat software hypervisor. It's still nifty though, and useful when you do it right.

Re:PXE/DHCP/AoE

[Anonymous Coward]

I booted a Windows 7 installation (over PXE) on a PC that had no storage devices once. I was so pissed off that I don't even remember what I did. I think, after I got the DOS network install image loaded (drivers were a PITA and took 1-2hrs), I realized that the Windows installer will not load under DOS anymore. So I threw a chair and created a Windows 7 PE image - I used to think the Debian installer was a little complex; compared to WinPE, it's actually enjoyable.

I still cry out in my sleep as I relive this moment. But now I wonder if ReactOS or similar could be used as the PXE image to make it easier.

Re:PXE/DHCP/AoE

[niteice]

If you have the resources to set up Windows Deployment Services it's stupid easy to get a network install of Windows (Vista/7) going...

Authentication

[Anonymous Coward]

The one thing missing from PXE is authentication: A PXE system will accept any DHCP address and with it any boot server configuration. Without cryptographic boot image authentication, network security is the Achilles' heel of PXE.

Re:Authentication

[rathaven]

It wasn't designed for it - PXE boots without authentication on the client so that the hardware gets the image thinly and then auth takes place when the OS is installed. It assumes control of the local LAN is in place and it is trusted. If you are looking for auth at this level you'd need to look at authentication to the switch or wireless on the network - pre-authentication using something like 802.1X. I'm not 100% clear but I believe gPXE has something that probably covers that in the docs as it has scripting capability pre-receiving DHCP addresses (at the level for wireless authentication and possibly 802.1X)...

Re:Authentication

[Anonymous Coward]

It's still an omission. If you use PXE for remote administration (instead of using it for completely diskless operation), then there is local data which can be compromised by a hostile PXE payload. How hard would it have been to verify a cryptographic signature against a public key stored in the BIOS configuration?

Re:Authentication

[Kjella]

So we're getting into cloak-and-dagger territory but wouldn't this be a simple way to have a malicious image installed on the client that'll relay logins back to you? If you're first redesigning it I'd make it so you can both verify cryptographic signatures and use SSL. It would be great if you can plug in your diskless machine into a presumed hostile network and know that it'll only connect to your designated boot server no matter what.

Re:Authentication

[Burning1]

If you permit rogue DHCP servers on your network, you have bigger problems than PXE.

The correct solution is to block DHCP response packets from unauthorized ports at the switch level, and make it a policy to smash the knees of anyone who connects a 'home router' to your corporate network.

Re:Authentication

[Anonymous Coward]

How hard would it have been to verify a cryptographic signature against a public key stored in the BIOS configuration?

In a 32k ram buffer? I'm not encryption expert, but that doesn't sound like much memory to store both a boot program AND encryption routines and data.

Re:Authentication

[rathaven]

It's still an omission. If you use PXE for remote administration (instead of using it for completely diskless operation), then there is local data which can be compromised by a hostile PXE payload. How hard would it have been to verify a cryptographic signature against a public key stored in the BIOS configuration?

Yes it was missed in PXE, however, based on the context of when PXE was developed I doubt we thought we needed it. The point of gPXE is that PXE hasn't been developed in line with changes in the computing arenas and that these omissions needed addressing.

I've checked further and even though there is a level of authentication built into the command line it doesn't yet have enough development to support 802.1X in gPXE, however, WEP, WPA and WPA2 are now supported so the remote boot from wireless can be undertaken securely. The things you mention are not exactly the point of the protocols like WPA2 and the encryption associated with them in the wireless realm and with 802.1X in both realms which enable authentication to the network. They won't stop a faked PXE image from a poisoned arp or a MIM attack on the http server or TFTP server but they are about securing your network so that a Hacker's device finds it a lot more difficult to get on and do those things you mention. The fact they are developing gPXE into something far more capable with support for HTTP, iSCSI, Wireless (encrypted) and others deserves a good round of support for trying to move things forward though its probably not where it should be yet...

Re:Authentication

[rathaven]

Yes it would and unfortunately its straightforward, however, there are lots of levels of mitigation in the infrastructure if network admins understand that it is there and use it - whether this is the case for the cloud is another matter.

What we are saying in a number of these posts is quite simply - "I don't trust core services on the network I'm connecting to". The response is, "Don't connect to that network or allow that network to connect to you".

Firstly - do you trust your router or DHCP server to supply DHCP without contamination as this gives the paths to the image source and a number of other details (including in some cases a level authentication to those sources)?
Secondly - do you trust that you can reach your sources for the disk images without contamination (is your TFTP, HTTP or iSCSI source really your source, is the image on them the one you put there or has it been changed/modified/rootkit installed/etc?
Thirdly - when you have the image installed, do you trust that you can access systems and services without contamination (malware, virus, etc)?

If you don't trust any of the above then you either need mitigation or need to not connect to the network. I can see how this technology is a great improvement to PXE in the private LAN, or secured network hence my comments, however, whether it is "cloud-ready" is another matter.

Re:Authentication

[evilviper]

The one thing missing from PXE is authentication: A PXE system will accept any DHCP address and with it any boot server configuration. Without cryptographic boot image authentication, network security is the Achilles' heel of PXE.

If untrusted machines are being attached to the same network segment you are, you've got a hell of a lot of problems, netbooting or not...

i dont wonder......

[Ruede]

i am using pxe often.

i have setup a few linux install "CDs" for network install, a few live CDs for an emergency OS. LTSP is using it too and a small intel atom box gets its kernel over tftp/pxe... the pxe provides the parameters for the nfsroot mount.
old win2k netinstall for ppl without a RIS uses that system too :)
the tftpd box that provides all that stuff is a small amd geode that is normally my router :)

i often thought about making a sourceforge project out of it.... :)

Re:i dont wonder......

[Anonymous Coward]

:) :)

Re:i dont wonder......

[icebraining]

I also use PXE. My home server (an old P3 laptop) runs from an USB drive, but since the laptop doesn't support USB booting, I use PXE to boot it.

The annoying part is that I can't reboot it remotely, as it won't have a TFTP server to feed the Linux image,

Re:i dont wonder......

[h00manist]

i often thought about making a sourceforge project out of it.... :)

I have often searched for how to set that up.

Re:i dont wonder......

[0100010001010011]

http://www.netboot.me/ + PXE = ultimate network config.

After setting up my own PXE setup, I put the settings in my DD-WRT. After completely hosing the only computer in my house (I left my laptop at work). I would have been SOL (no floppies, no CD's) Not so with Netboot.Me. Fired up a Debian recovery disk. The have recovery disks, minimal install disks, partition disks, FreeBSD, Linux.

I haven't been wondering...

[langelgjm]

I still use PXE to boot a diskless MythTV client. For a while I had the machine connected to a wireless router set up in bridge mode, so the machine effectively netbooted wirelessly.

Re:I haven't been wondering...

[Anonymous Coward]

Diskless netboot over a wireless connection? Frankly I don't see how that is possible. Please explain. You must have had a floppy, USB stick, or something to get the initial image started. I don't know of any way to do that over wireless.

Re:I haven't been wondering...

[Anonymous Coward]

Bridge Mode. RTFP.

Re:I haven't been wondering...

[langelgjm]

The trick is to use a wired card with a PXE ROM attached directly to a wireless router with firmware that supports a bridged mode. In my case, I used a WRT54G running DD-WRT.

You configure the router to bridge the wireless and wired networks, and enter the appropriate information for your wireless network. Then, you configure your netboot system normally (DHCP, TFTP, etc.). When the client boots up, the PXE ROM thinks it is directly attached to a wired network so everything works normally.

The downside is you need an extra router that must be configured as a bridge, but the upside is that it allows you to have the equivalent of a wireless, diskless machine.

Re:I haven't been wondering...

[Anonymous Coward]

The actual machine still talks over ethernet into a wireless bridge that handles the actual 802.11 link.

Re:I haven't been wondering...

[mrclisdue]

I'm guessing that the "machine *effectively* netbooted wirelessly" means that the machine was connected via ethernet cable to the wireless router (which was in bridge mode....)

Not really sure if my explanation serves to clarify....

cheers,

Re:I haven't been wondering...

[Xyde]

Obviously you've never used a Mac...

Re:I haven't been wondering...

[gig]

Macs since about 2007 can netboot wirelessly. Before that, you had to connect via Ethernet. But MacBook Air has no Ethernet, or optical drive. The way you install an OS is to boot from the DVD in another machine on the Wi-Fi network. Optionally, you can use a USB-to-Ethernet adapter to speed things up, or a USB DVD drive. But a Wi-Fi netboot is the default way.

The Mac firmware is savvy about all of the system's hardware. It can also connect to Bluetooth devices, so your keyboard and mouse continue to function whether the computer is booted into an OS or not.

My understanding is this all works with Windows and Linux as well, although I only use Mac OS so I don't know from experience. The firmware is EFI so in theory it should work. I did see a Linux disk with a cute penguin icon on the boot screen of a Mac once, but it may have been attached via USB or FireWire, and that may have been the old Mac firmware on a PowerPC Mac.

Netbooting on WiFi

[Anonymous Coward]

Am I the only one laughing at that?

Re:Netbooting on WiFi

[yupa]

could be great to propagate rootkited OS ;) Of course authentication is needed (but the code for doing that could be big and complex).

Cloud?

[plasticsquirrel]

Good grief, everything is a "cloud" now. Have some servers on a rack? Those aren't servers, that's a cloud! It's like some retards took a Cisco networking diagram, and went crazy when they realized that everything could be simplified into one of the "clouds".

Warning / Rant: The last 5 years of computing have been pretty lame. Concurrency and solutions to it using functional high-level languages are the future. That's where we should have been five years ago when it was so obvious that chips with large numbers of cores were the future. These cloud solutions are just a stupid name for the same old monolithic crap. It doesn't scale and isn't modular in a Unixy way. Modern applications just suck because they're so inflexible. Why can I do so many things from a little text terminal, but I can't easily script the behavior of my web browser without special add-ons? Why aren't modern applications flexible like this, with simple interfaces for communicating with other programs? Where is the equivalent of a shell pipe, in modern applications? It's like somebody threw away all the lessons of the past, and said "But this is the new way, we don't need the old way, because this is new." Fuck that, computing should be better than this. It should be better than these stupid clouds and old piece-of-shit reinvent-the-wheel C and C++ programs with buffer overflows and other ancient problems. Or the HTML / Javascript / whatever jerry-rigged "web applications" that run on some opaque "cloud" that a random company has. Why is it that languages like Smalltalk and Lisp have been around for so long, and nobody learns from them or uses them? It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.

Re:Cloud?

[fauxhemian]

It doesn't scale and isn't modular in a Unixy way. Modern applications just suck because they're so inflexible. Why can I do so many things from a little text terminal, but I can't easily script the behavior of my web browser without special add-ons?

http://en.wikipedia.org/wiki/Uzbl

Re:Cloud?

[sznupi]

It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.

I believe somebody said once "the amount of intelligence on Earth remains constant"...

Re:Cloud?

[warrior389]

My grandfather was telling me about how he used to walk inside of an old cray with 5 processors like 50 years ago. What happened to the people who used to know how to program those things efficiently? Apparently multi-core programming isn't so new.

Re:Cloud?

[antirelic]

There are two types of people who use the word "cloud". People who are in charge who have absolutely no idea how technology works, and those who make fun of those who use the word "cloud". Back in the day, we use to call the "cloud" the "network". Of course, the clueless now confuse the cloud with components of the cloud, such as clusters, SAN's, and mainframes

Re:Cloud?

[Anonymous Coward]

http://www.uzbl.org/faq.php

Cool, so I can install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering ... or I can install Firefox and actually get shit done. Sounds about right.

Re:Cloud?

[vbraga]

For sure it's not new. It's wide availability that's new. And algorithms and solutions using it for desktopish requirements were almost non existent.

Cray then also used vectored processing now available on desktops as GPUs.

Ways to develop for these technologies for sure exists but had another set of requirements in mind when it were made. A large number of programmers grew up without being able to play with it. It will take some time while people learn it.

Re:Cloud?

[h00manist]

I believe somebody said once "the amount of intelligence on Earth remains constant"...

I wager it goes down during wars, dictatorships, televised games, and elections. Orders are to destroy anyone who is nonconforming.

Re:Cloud?

[TheRaven64]

When you're paying a few million for a Cray, paying top salaries for programmers who can take advantage of it is the only sensible way of getting a return on that investment. When you're paying $500 for a multicore server, paying top salaries costs a lot more than just upgrading to a $5000 server.

Re:Cloud?

[TheRaven64]

Cloud is a useful filter word, like Beowulf Cluster. If someone uses it in a non-ironic way, you know that you can safely ignore anything else that they say on the subject.

Re:Cloud?

[fredrickleo]

I guess you've never heard of Applescript ;P

Re:Cloud?

[Anonymous Coward]

Warning / Rant: The last 5 years of computing have been pretty lame. Concurrency and solutions to it using functional high-level languages are the future. That's where we should have been five years ago

But five years later, that's not where we are; the world moved into a different direction. What are you going to do? You can either cling to your ideas and refuse to give them up, or accept reality and try to improve what we've got now, instead of what should have been but isn't.

Example: Plan 9 and Inferno. Much better than Unix, but didn't go anywhere. Pity? Yes, but it's futile to still hope for a takeover that'll never come.

Re:Cloud?

[icebraining]

Why can I do so many things from a little text terminal, but I can't easily script the behavior of my web browser without special add-ons?

Tried uzbl? Also, you don't need any addon to make a Firefox addon. Greasemonkey simplifies the process, but it's not required.

Re:Cloud?

[Anonymous Coward]

Someone was abused by a GUI, a high level language and a web application as a little boy, right?

Re:Cloud?

[Anonymous Coward]

Because the old paradigm doesn't work very well with today's "modern applications"?

Re:Cloud?

[Anonymous Coward]

> Example: Plan 9 and Inferno. Much better than Unix, but didn't go anywhere. Pity? Yes, but it's futile to still hope for a takeover that'll never come.

Every time someone comes up with something "better than Unix," some jerkwad joins the project and decides it should be backward-Unix-compatible for utility's sake. Then the project gets shot to hell...

(I keed, I keed!)

Re:Cloud?

[h00manist]

When you're paying a few million for a Cray, paying top salaries for programmers who can take advantage of it is the only sensible way of getting a return on that investment. When you're paying $500 for a multicore server, paying top salaries costs a lot more than just upgrading to a $5000 server.

$1200 for a multi core server with three Chinese indentured slave programmers, service for ten years. China software-factory sweat shop escape-and-rebellion insurance and translation services not included.

Re:Cloud?

[sea4ever]

It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.

I believe somebody said once "the amount of intelligence on Earth remains constant"...

while the amount of people it is spread between increases exponentially?

Re:Cloud?

[camperdave]

Oh... and Meteorologists.

Re:Cloud?

[Bearhouse]

http://www.uzbl.org/faq.php

Cool, so I can install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering ... or I can install Firefox and actually get shit done. Sounds about right.

Cool, so I could Firefox and actually get shit done, or I could install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering.

FTFY (this is /., after all...) *goes away to play with uzbl*

Re:Cloud?

[Nimey]

Unless that person happens to be your boss.

Re:Cloud?

[Nursie]

What the hell do you mean "Apparently"?

I was doing commercial multi-threaded programming a decade ago, and it wasn't new then. Multi-process across multiple cores has been with us for decades.

Parallel programming is not massively difficult, but here on Slashdot people seem to think it's some crazy new thing that's really hard and nobody's ever done it before.

No it isn't and yes they have.

Re:Cloud?

[syousef]

Good grief, everything is a "cloud" now. Have some servers on a rack? Those aren't servers, that's a cloud! It's like some retards took a Cisco networking diagram, and went crazy when they realized that everything could be simplified into one of the "clouds"..

Question: I tried to put my boot into the cloud but it just went through. What do I do?
Answer: You need our new solid cloud state technology at an additional cost of $3M a year.

Hard Drive is the Enemy. Eliminate.

[h00manist]

Network multi-booting is great to allow people to change OS's, but it's pretty complex to set up, and there is little knowledge of it.

redundant

[Lilo-x]

Im sorry, but with virtualisation and it's boot from SAN capability this is pretty redundant now, it's old technology and there are reasons it hasn't moved along,

changing drivers can break boot
changing the kernel means custom compiles
mpio support isn't straight forard
NFS is showing its age

and with virtualisation you can boot from the network, and you can use ethernet/fc/iscsi/nfs

Re:redundant

[Anonymous Coward]

with virtualisation and it's boot from SAN capability this is pretty redundant now

Because we all know, when you have a virtualisation platform you never, ever, need any physical hardware. Companies these days now run their entire infrastructure off of a single HP C7000 enclosure and some SAN.

Works pretty well and scales well too

[pdbaby]

We do this at work - we chain-load gPXE using PXE and then use that to iSCSI boot from a Linux SAN which uses LVM COW snapshots. It's pretty good - the etherboot project rocks! We've been doing it for a while but it always gives me a kick when I type something at the commandline which wakes up a machine using IPMI & then boots it off some SAN volume

Re:Works pretty well and scales well too

[ToasterMonkey]

iSCSI boot from a Linux SAN

If you call that a SAN, my calculator is a computer.

Re:Works pretty well and scales well too

[Lennie]

I took a look at the Dell Equallogic that we have at work, which had a nice pricetag, also runs Linux. So Linux as a SAN doesn't sound all that stupid. Now I wouldn't expect Dell Equallogic to be the top of the line, but they do have all the 'enterprise features'.

Re:Works pretty well and scales well too

[pdbaby]

We get a much better bang for our buck by using commodity hardware - we can build & maintain a few linux SANs for the price of an 'enterprise' one. We just build software which can deal with the underlying hardware failing or the network splitting.

xCAT

[Junta]

xCAT at its core focuses on network boot (and for x86 portions, includes gPXE). It does Windows, Redhat, SuSE, VMware. It takes a lot of the tedium out at scale of network install, statless boot, and iscsi boot.

Re:xCAT

[Daengbo]

gPXE (the Etherboot project) is used in Fedora's BFO project (based on BKO). If you have a local server, that is a cool project

I'm sorry that PXE and Etherboot get no love around here. I've been using Etherboot since ~2000, when it was the common boot method for LTSP (and K12LTSP). EB and gPXE are awesome projects that show how much you can do with so little space to work with.

When network booting several computers, I've found the boot to be much faster over the network than locally once the first computer is booted and the server has the files in memory. I wasn't even using GbE.

A few interesting things...

[Junta]

The DHCPv6 netboot standard about to come out recommends http as the protocol of choice where tftp would have been used, but uses URLs so the protocol is selectable.

The iSCSI portion of this is a wider standard, implemented by many firmware configurations out of the box.

Finally, I'm going to plug xCAT as a tool to wrap dhcp, dns, ntp, active directory, gPXE, iSCSI, PXE, bootp/tftp, ipmi, blades, vmware, kvm, xen, LPARs, and more to deploy vmware, windows, linux, and aix systems and do hardware management. It mostly pays off at larger scale, but it is a project that aims to understand how to best utilize those various technologies.

EEPROMs

[Sandman1971]

The only diskless OS booting I'm interested in and waiting for is booting from EEPROM. Near instant boot times. Why has this never really taken off?

Re:EEPROMs

[Anonymous Coward]

eeprom are slow to read and are too small to contain a modern OS. Better use flash (ssd).

Re:EEPROMs

[PhunkySchtuff]

My workstation boots off a 128GB chunk of solid-state storage - I don't know what you're talking about.

Isn't Fedora pushing hard for this as a standard?

[Anonymous Coward]

http://boot.fedoraproject.org/

I thought I read that their goal was to get rid of offering premade ISO's and just off one tiny network bootable image that does everything.

Here's what I can do, you can do it too...

[Anonymous Coward]

1. Remote KVM setup (requires KVM-over-IP)
2. Remote PXE unattended setup (requires BIOS configuration by KVM-over-IP or someone at the data center to make the changes for you)
3. Remote IPMI setup (requires BIOS config... blah blah blah)

You actually have to combine either 1 and 2 or 3 and 2 to install a system without putting a disc in the machine, and I've done both.

#3 came about because the machines we bought didn't have PS/2 connections to work with the KVM. Dell's IPMI implementation is "good enough" that you can install linux or freebsd over it provided you setup the BIOS to PXE boot.

By "good enough" I mean you don't need to pay for the additional "KVM-over-IP" feature, the IMPI serial-over-lan(SOL) works. You just need to know what you're doing.

It's not something I'd let non-IT staff touch though, it's kinda like dealing with BBS's back in the day.

But I still use PXE...

[jrronimo]

I have a laptop which cannot boot from USB and whose CD-ROM doesn't work. It's good enough for a Linux install, so I've taken to installing Ubuntu via PXE.

It works out quite well, when it works, and it's neat to do. I generally use this tutorial to set up the TFTP server on my Windows machine and just change whatever release they talk about to whatever release is current.

It's fun, frankly.

Fedora has OpenSharedRoot

[marciot]

I've been using OpenSharedRoot over NFS at my workplace on three different clusters. It works pretty well. Ultimately, I would love to be able to install gigabit switches everywhere and give everyone disk-less workstations that PXE boot. Unlike a thin-client solution, the engineers would still be able to use their desktop's CPU resources and local RAM, plus the local HD could be used for swap and data files, but not the OS itself.

Boot over the Interwebs

[MSDos-486]

I thought these were pretty interesting ideas:

http://boot.kernel.org/

http://boot.fedoraproject.org/

Although they are pretty slow at times, and I have had them freeze a few times (this may have been due to bad hardware though). It a pretty cool way to install a Linux distro.

Boot syncing your distro.

[kallistoteles]

Of topic, but instead of "updates available, download and update". Just reboot your system it syncs with the latest distro of choice. Should be nice, is there a solution already?

Not limited to small files

[feenberg]

I am not sure where the idea that PXE boot files are limited to 32KB comes from, but we are booting FreeBSD 8.0 with a 240KB boot file with PXE and tftp and have not had to do anything special. We also boot Linux (Fedora 11) with a 4MB initrd over tftp and that has not posed any difficulties either. Our FreeBSD experience is documented at http://www.nber.org/sys-admin/FreeBSD-diskless.html - it works quite well for us. I looked at gPXE and it doesn't really solve any problems we have had. Actually, we have had only one problem - sometimes the OS boot code doesn't support the motherboard ethernet, and we have to add a different ethernet card for post-boot LAN access.

Re:Not limited to small files

[OrangeTide]

I've PXE booted 9MB images for X terms. PXE grabs a tftp/UDP bootstrap program (only a few kB), and the rest is downloaded over TFTP. but it could have just as easily been grabbed over HTTP or NFS with a different bootstrap. (u-boot has some nice mini NFS client code capable of quickly grabbing files in a very small "kernel", easy to drop into other projects)

And we less computer-oriented nerds

[Wilson_6500]

naturally laugh at the mere suggestion of dickless booting.

USB Flash booting works fine

[mprinkey]

PXE is nice if you have an entire cluster to build. We use it and it can be great to get a disk image installed on a new cluster. But honestly, I've only seen two cases were PXE really works...small "toy" setups where somebody boots this system on the left side of the desk from an PXE/TFTP/NFS server on the right side of the desk. Or using it to clone/install large blocks of systems. (That is how ROCKS works.) But I don't know of anyone using PXE to boot a significant number of systems...workstations or cluster nodes.

Instead, I just installed OpenSUSE 11.2 to a 4GB flash drive. It worked perfectly. This is not a LiveCD or something like Slax that uses filesystem trickery to maintain a persistent system image. This is vanilla OpenSUSE installed to the flash just like a HDD or SSD. It recognized it as /dev/sda and installed. Of course, I didn't allow any swap space. For fun, I then reinstalled the system using two 4GB flash drives and created a raid0 for /. It also worked great and significantly faster than the single. So for $10 per system, you can boot from a flash drive locally...and then mount your user files from a NFS server. That is exactly what I'm doing. For future clusters that don't require scratch space, we might starting using a few internally mounted flash drives instead. All those drives need to do is handle bootup...most of the commonly used files come from NFS or would be cached in RAM.

the perfect attack vector

[mu22le]

Cloud related ranting apart, can you imagine a better attack vector than wireless booting? Just place a rouge access point in a crowded cafe or station, broadcast a boot image and infect all of the poor smucks that start-up their laptop in the area. You just need to patch their windows system and then start it, it will take 30 sec longer than the ususal 5min boot, they won't ever notice...

A fully automated PXE-based Cloud

[Anonymous Coward]

... you may want to take a look at openQRM - http://www.openqrm.com. It is a fully automated Datacenter Management and Cloud Computing Platform heavily based on PXE.
enjoy
Matt Rechenburg
Project Manager openQRM