Slashdot Mirror

HPV prevalence is about 70% in sexually active young people. HPV causes about 6100 deaths per year, and about 26900 cases of cancer per year in the US. Most deaths, however, occur in low-income countries - about http://www.who.int/mediacentre...">270,000 per year worldwide in 2012 for cervical cancer alone according to the World Health Organization.

So, I think it's reasonable to conclude HPV is indeed contagious, common and deadly. Also, in my experience (as a primary care physician), most people are enthusiastic about the idea of making Pap smears obsolete, which is a distinct possibility with widespread use of HPV vaccine.

When they started doing Energy Star ratings for LEDs, they tried really hard to avoid the CFL fiasco; see http://www.gizmag.com/energy-s... As a result, Energy Star rated LED bulbs are pretty reliable. I have about 60 (!) in my house, bought over the last 9 months. None of the Energy Star bulbs has failed yet. Two non-energy-star LED bulbs that were several years old (from vendors not around anymore?) did fail.

I'm now slowly converting the bulbs in the house I rent out to LEDs, with the tenant's cooperation. The only two bulbs she has liked so far are the Cree 40W TW http://www.creebulb.com/Produc... (for bathrooms only - it hums too much for living room) and the Phillips 40W A15 ( http://www.homedepot.com/p/Phi... ) for everywhere else. Her dimmers are old, and most LED bulbs flicker with them; I should get her newer dimmers. Haven't had that problem much at my house.

I'm quite happy with the LEDs so far, and am writing up my experiences at http://kegel.com/energy/lights... Your mileage may vary.

A working programmer and a computer scientist are two different things, but the computer scientist should be able to write a basic program:

A surprisingly large fraction of applicants, even those with masters' degrees and PhDs in computer science, fail during interviews when asked to carry out basic programming tasks.

For programmers, this is a basic test, but when a computer scientist can't do something this fundamental, it calls their higher-level qualifications into question; and even if it doesn't, it makes you worry that their architecture or design will consider real-world issues and implementability.

Me, too. I get mine at Home Depot (they have Cree, which I like better than the ones at Lowe's). Replacing frequently-used incandescents with LEDs pays for itself in about 9 months (at $10/LED, 4 hr/day, and $0.15/kwh; your milage may vary, see http://www.lektroninc.com/payb... )

Doing this (and a few other things) cut my power usage from 40KWH/day to 25KWH/day. Still too high... I need to dig some more to see where it's all going.

I'm blogging the experience at http://kegel.com/energy/lights...

The problem is that we are nearing the peak of what is possible with current technology in a single core

But aren't there still plenty of things that the hardware can do to make the software stuff easier?

Intel has actually started adding some stuff to help: http://en.wikipedia.org/wiki/Transactional_Synchronization_Extensions

So maybe Intel, AMD should interact more with the OS and compiler bunch and figure out how to use those billions of transistors in more useful ways.

There are things you can do in software to address the c10k problem (and similar problems): http://www.kegel.com/c10k.html
But I'm sure there are things you can do in hardware to make stuff even easier. It's not like the stuff that's being done is the best way of doing things. Furthermore the OS and apps people might be writing things in certain ways because certain things aren't done by the hardware or done fast.

I know that at least on the x86 platform checking the current time and also getting monotonic time is not as simple AND efficient as it could be. It was even worse before HPET, and now even HPET isn't that great. Monotonic system time (ticks) could be different from current human time, many programs need one or both.

Same goes for scheduling stuff, serializing stuff and getting stuff to trigger on arbitrary things all with minimal overhead. I suspect many things would be easier if you can create a way of having cluster wide consistent monotonic high-res time, and also fast low latency clusterwide locking.

On a related note many programming languages seem to like to push parameters onto a stack. That's fine but in many implementations they seem to push the data onto the code/call stack (which holds return addresses). This mixing of data and code stuff is unsafe. They should have separate data and code stacks. That way even if a hacker messes with the data, it's harder to execute arbitrary code- you'd just execute the same code but with mangled data, which is more likely to produce errors instead of arbitrary code execution.

If the reason for doing such insecure stuff is performance or convenience, then perhaps Intel etc can use those transistors to make it faster and easier to do things safer.

I have a page with links to other examples of, and discussion about, bluetooth traffic monitoring; see http://kegel.com/bluetooth-roads/

Another one of my original points (which I think among others was lost) was that NT had IOCP in 1995. According to a manpage and a quick google search, epoll() did not come about until 2002.

2001 actually.

As "all Unix programmers" will know (your term), poll() is a much less scalable interface, for reasons that do not apply to IOCP.

epoll (it's not a function) is a scalable implementation of interface with poll() semantics. It was added after scalability of the rest of the system caused poll() to become a bottleneck in a specific application of a single server thread with very large number of simultaneously connected clients. The history of this is recorded in http://www.kegel.com/c10k.html

Please note, at which number of concurrent connections the scalability problem manifested itself in the first place.

I'm still kind of skeptical that you know what the NT interfaces look like because in practice they are *extremely similar* to uses of epoll() and kqueue() [and not just in my programs]. A lot of writing out there about IOCP focuses unnecessarily on concurrency but if the last parameter to CreateIoCompletionPort is 1 and keep it on a single thread it's pretty similar to the Unix approach. And it existed years earlier.

No, then you still can't combine simultaneous waiting on unlimited number of I/O operations, you are forced to use concurrency for that. Unix design is very strict when it comes to concurrency in I/O model -- if user does not specifically want it, system does not impose it.

The main point is performance. Ryan Dahl wanted to write fast, scalable servers easily. We all know for years that threads don't scale but event loops do (see the second chart of memory consumption of apache vs nginx). Of course in order to have a highly concurrent evented server you can't use blocking system calls (which were a big mistake in my opinion to begin with - they are the only reason why you needed threads exposed at the application level for concurrency in the past). OK, so we want a portable, high performance, event-based, async-I/O, scalable, highly concurrent server. The obvious way to write such servers in a portable, OS-independent way was to write them in C using libraries like libev or libevent for event loops and libeio for non-blocking I/O. The result is great. But the problem is that it is not easy. C doesn't have lambdas, anonymous functions, closures or higher-order functions in a real sense, which all would make writing event handlers much easier. So Ryan was looking for a higher level language and found V8, the JavaScript virtual machine written by Google for Chrome. JavaScript has anonymous functions and closures. And V8 is fast. And also when you write JavaScript in the browser then you never use blocking function calls anyway, so people are already familiar with asynchronous I/O, events, callbacks, closures, futures and promises. Hell, you can even use Y combinators in JavaScript if you know your craft. Now, if only JavaScript had lazy evaluation and proper tail call optimization - maybe some day. Watch some talks by Ryan Dahl if you're interested and after 25 years in the field you should be. Oh, and Node doesn't have anything to do with the browser besides the V8 origins. It's all server-side. See the Wikipedia article on Node for more info and code examples. I'm glad that people who have been professionally programing for so many years are still willing to broaden their horizons. As I have written in the past it is not a universal property of programmers unfortunately. Have fun with new tools.

So when you're pushing data as fast as you can through a socket, the old read(byte[]) or write(byte[]) are faster? Wow, no kidding.

You do NOT use java.nio (like Jetty's SelectChannelConnector) for maximum throughput. You use it to handle persistent connections, like all those long polling requests via AJAX which return on an event or timeout after a minute. This article is like recommending Apache with its hard limits on how many requests it can serve concurrently over newer, asynchronous servers like Nginx for static media servers with keep alive enabled.

The slides even mentions the C10K problem, but what it doesn't do is mention when to use either technology - async IO for concurrency and endless scaling, and synchronous IO for pushing a 10G Ethernet link to the limits. No wait, the nio setup can do that too, 700MB/s or 5.6Gbit/sec per core on 2008 hardware should be enough to max out anything you can buy now. It's great that synchronous IO can hit 1GB/s, a whopping 30% faster, but useful? I'd say no.

For most users, you don't use either API. Lets be honest here, writing highly concurrent software is hard, why reinvent the wheel when you can get off the shelve software that can do it better? You use Jetty and choose between the SelectChannelConnector or SocketConnector, or choose between Apache or Lighttpd/Nginx depending on the traffic pattern. What you do write is the bit that accepts a whole HTTP request and returns a HTTP response, everything before and after is magic.

Unless you're a file server, each 50k sized HTTP response will require enough work to make sure you run out of CPU or Disk IO long before you hit even the 100Mb/s ceiling in most rack switches. Even if your app is fast, 16 cores x 100ms per request x 50K is only 62 Mbits. Not 5600.

But if you need to scale in concurrent client count, there's no way around async IO. The latest name to watch is Netty. In Plurk Comet: Handling 100,000+ Concurrent Connections with Netty, it scales up to 100000 concurrent connections on a quad core server with 20% CPU load.

Just stop worrying about sockets already, and start worrying about your SQL server suffering a meltdown. Even if you get manage to grow into the Facebook, it's not like using synchronous IO will save you from deploying 30000 servers, it's the application code that's slow. Zero copy, one copy, "string concatenation style twenty copies response building" socket writes don't matter at all, memcpy is cheap compared to a few lines of interpreted code, servers are cheap compared to developers, and never mind the cost of the programming gods giving these presentations.

Getting Students To Think At Internet Scale

After reading the headline, I thought this is an extension to http://www.kegel.com/c10k.html

The LA Times, for some reason, came out against
the proposal today. This fired me up, so
I spent some time putting together notes on the
subject at
http://kegel.com/energy/television/
in hopes that it'd help me write a killer
letter-to-the-editor explaining why they're wrong :-)

Depends on how much beer there is at that party.

That suggests a great marketing strategy for Linux: Throw Linux kegger parties next door to the W7 parties. Motto: Get smashed with Tux.

Here are a few notes I wrote a while ago on the subject: http://kegel.com/academy/opensource.html

http://kegel.com/wine/sweng/ might also be of some interest.

I'll be honest, I thought those links were to something else entirely.

And you clicked on them based on that rather than an interest in programming topics? Dude you're messed up... :P

Here are a few notes I wrote a while ago on the subject: http://kegel.com/academy/opensource.html

http://kegel.com/wine/sweng/ might also be of some interest.

I'll be honest, I thought those links were to something else entirely.

And you clicked on them based on that rather than an interest in programming topics? Dude you're messed up... :P

Here are a few notes I wrote a while ago on the subject:
http://kegel.com/academy/opensource.html

http://kegel.com/wine/sweng/ might also be of some interest.

I'll be honest, I thought those links were to something else entirely.

Here are a few notes I wrote a while ago on the subject:
http://kegel.com/academy/opensource.html

http://kegel.com/wine/sweng/ might also be of some interest.

I'll be honest, I thought those links were to something else entirely.

Here are a few notes I wrote a while ago on the subject:
http://kegel.com/academy/opensource.html

http://kegel.com/wine/sweng/ might also be of some interest.

Here are a few notes I wrote a while ago on the subject:
http://kegel.com/academy/opensource.html

http://kegel.com/wine/sweng/ might also be of some interest.

Another interesting read on sockets performance, for the most part specifically in Linux context, is this. It seems to be rather detailed, covering a lot of ground. However, it's last updated in 2006, so I wonder what had changed since then...

I hear your frustration. You'd like Wine to work really well, and you are impatient with the current rate of progress. But implementing Win16/win32/win64 is a shitwad of work, so no matter how fast we go, you'll probably always be impatient. And so will I.

The best way to look at the current experimental DIB engine implementation is as a prototype. If apps work well enough with it, it'll be easier to commit the resources to do a production-quality one.

If patches are rejected, it means there's something wrong with them; good, persistant developers who listen to the feedback and submit small, incremental patches can get them accepted. It's true that it can be frustrating at times, but that's not unique to Wine; gcc and the linux kernel are also frustrating for new contributors. And don't even get me started about gibc :-)

http://www.englishbreakfastnetwork.org/krazy looks great, but a lot of its checks are for C++ specific issues, which Wine doesn't have because it's based on C. Wine is well aware of the benefits of automated testing and code analysis. We do have an automated build and test system, see http://test.winehq.org/ and we're slowly working on getting all the tests green. And we pay attention to Coverity's scans of our source tree. (We also have a patchwatcher, but it's out of action right now because I'm lame.)

If you want to go help raise funds from governmens stuck on win32, please do! I tried to make a little pitch for this at the end of my CeBIT talk last week, see http://kegel.com/cebit But it's hard to get their attention. First, they aren't impressed unless it can run all their apps, and second, the ones that are really serious about getting away from Windows tend to focus on native applications anyway.

So: the grapes really aren't that sour. Dive in and help - and be patient and persistent.

• "Ubuntu Linux certified laptop" on the front page,
• not deeply buried behind tons of "XXX recommends Microsoft Vista" crap...
• at competitive price,
• and ships to New Zealand

I'd "click" on "buy" right now.

I've been keeping tabs on it since about 1999. See http://kegel.com/corporate_ethics.html

A surprisingly large fraction of applicants, even those with masters' degrees and PhDs in computer science, fail during interviews when asked to carry out basic programming tasks. For example, I've personally interviewed graduates who can't answer "Write a loop that counts from 1 to 10" or "What's the number after F in hexadecimal?" Less trivially, I've interviewed many candidates who can't use recursion to solve a real problem. These are basic skills; anyone who lacks them probably hasn't done much programming. Speaking on behalf of software engineers who have to interview prospective new hires, I can safely say that we're tired of talking to candidates who can't program their way out of a paper bag.

My pet peeve these days is diploma mills. A certain big, well-regarded university I know of seems to churn out unqualified masters students. I talked with the dean of CS there once about it, and he just said "We're already requiring so many courses, we can't require any more". Perhaps they need to be pickier about who they admit, but I've heard it speculated that the CS masters program is a profit center; and being pickier would hurt their bottom line.

Another useful data point is whether the person in question successfully gets lots of code into well-run open source projects like Wine or the Linux kernel. Handy tools to search for commits include http://ohloh.net/ http://marc.info/ and http://cia.vc/ . (And yes, despite being a wine advocate for years, I have fewer than 100 patches in. Lame! :-)

Dan Kegel of C10K fame has an excellent web page on this topic.

Dan Kegel of C10K fame has an excellent web page on this topic.

The C10K problem
http://kegel.com/c10k.html

High-Performance Server Architecture:
http://pl.atyp.us/content/tech/servers.html

(Note that this refutes http://ask.slashdot.org/comments.pl?sid=277739&cid =20331189 because this emphasizes that you should do as much work in one thread as possible in order to avoid redundant context switches. Thus don't separate threads based on role where one task must be picked up by many threads to complete -- instead, each thread should be able to execute any sequence through the state machine for any client.)

Maybe not a classic, but still nice (see its references):
Threads, Tasks, Coroutines, Processes, and Events:
http://shlang.com/writing/threads-tasks.html

This isn't really the same kind of component.
On the other hand, Linux has epoll, which fills the same role as kqueue.
In my experience, epoll is at least as good.
http://www.kegel.com/c10k.html#nb.epoll

Now MacOS X needs to fix their kqueue bugs, and the world will be a happy place.

Thats what we did:
1) build a tool chain using http://kegel.com/crosstool/. Note: this uses glibc instead of newlib/uClibc but there are patches to make it work.
2) Download and build the mainline kernel with needed modules compiled in
3) Place onto device.
4) Develop application
5) ???
6) Profit!

See The C10K problem

--jeffk++

Basically: program something you care about or enjoy.
The more time you spend designing/coding, the better you'll be at it.
See also my advice for CS students,
    http://www.kegel.com/academy/getting-hired.html

Hmm, would a PrisonLinux distro have everything inside a chroot jail?

I've updated http://kegel.com/c10k.html to make the "... for obvious reasons"
section clearer, and I now point to Cory's library, as
well as to an updated version of Polyakov's kevent and naio pages.

however, it's easier conceptually to write a threaded server, it's more natural to write, and you just launch a single thread per connection. unfortunately, currently, this doesn't scale (see Why Events Are A Bad Idea (for High-concurrency Servers) http://www.usenix.org/events/hotos03/tech/vonbehre n.html for an argument that thread implementations, and not their design, are the issue).

the former method can handle thousands of simultaneous connections with high throughput, even on a decent workstation; the latter cannot. threads simply have an inherent overhead that cannot be eliminated.

i've actually been working on writing a non-portable insanely fast httpd in my spare time (svn co svn://parseerror.dyndns.org/web/) over the past few weeks as a way to explore non-blocking I/O + epoll() and it performs very well (~600% faster conns/sec than a traditional fork()ing server (which i wrote first)).

for further discussion see The C10K Problem http://www.kegel.com/c10k.html which goes in-depth on these very subjects

Check out the C10K page for a very detailed discussion about this.

Check out the C10K page for a very detailed discussion about this.

1. You're gonna need multiple Linux flavors and versions from multiple sources that specialized in these platforms.
   
   • ALPHA
     
   • ARM
     
   • I686
     
   • IA64
     
   • M68K
     
   • PowerPC PenguiPPC Denx
     
   • MIPS
     
   • S390
     
   • SuperH
     
   • UltraSPARC
     
   • X86_64
     
   
   
2. To determine which versions of crosstool (compiler, linkers, debugger), check out The Matrix Guy (Dan Kegel), or more specifically THE MATRIX of workable gcc/g++/ld/gdb.
   
3. To ease your pain of figuring out the "./configure" options, definitely checkout PTXDist. Menuconfig is similar to Linux 'make menuconfig'. PTXDist also help to build a root file system in a jiffy, which in my book, is a PLUS!
   

1. You're gonna need multiple Linux flavors and versions from multiple sources that specialized in these platforms.
   
   • ALPHA
     
   • ARM
     
   • I686
     
   • IA64
     
   • M68K
     
   • PowerPC PenguiPPC Denx
     
   • MIPS
     
   • S390
     
   • SuperH
     
   • UltraSPARC
     
   • X86_64
     
   
   
2. To determine which versions of crosstool (compiler, linkers, debugger), check out The Matrix Guy (Dan Kegel), or more specifically THE MATRIX of workable gcc/g++/ld/gdb.
   
3. To ease your pain of figuring out the "./configure" options, definitely checkout PTXDist. Menuconfig is similar to Linux 'make menuconfig'. PTXDist also help to build a root file system in a jiffy, which in my book, is a PLUS!
   

Crosstool – haven't used it too extensively myself, but it did a great job when I had to cross-compile a Linux kernel to run on my AMD64 system. And if you're using ARM or similar architectures, QEMU might be helpful. I may be entirely missing the point here, but may as well offer my own limited knowledge anyway :-)

• emacs
• grep
• perl
• sed
• svn
• xml (manipulate XML from the command line)
• tar
• ssh (this one is fun: "ssh server tar -cf - directory | tar -xv")
• for (built-in bash command, one-line scripts from the command line are very useful)
• lsof (what processes have open network ports? why can't I unmount that disk?)
• wget
• ping
• telnet (test SMTP, HTTP, etc servers by hand)
• nmap

This page, while out of date, and referenced earlier during this discussion, needs re-emphasis. I hope it gets updated soon:

http://www.kegel.com/c10k.html

Very awesome paper. How do _you_ make a server that handles 10,000 connections?

--jeff++

In the same line - where is the discussion of different FD table polling mechanisms? select() versus poll(), and wheres the writeup about Linux's epoll(). I would have been interested in an epoll() article, especially how it compares to FreeBSD's kqueue().

For the overview, you want Dan Kegel's c10k page:

http://www.kegel.com/c10k.html

the real performance problem with socket IO has to do with using select and poll

Yeah, now that I have distcc and ccache up, things are much nicer for long compiles.

Right now I have a spare sparc and my laptop in the distcc cluster crosscompiling. I used Dan Kegel's crosstool to build the tool chains on sparc and cygwin.

Unfortunately, that doesn't help the java and install parts of OO (which seems to spend a lot of time generating what looks like NLS stuff).

There are lots of other reasons. I'm taking notes
at the OSDL desktop architects' meeting; see
    http://kegel.com/osdl/da05.html

Hello Dr. Thompson,

I appreciate your answering questions on the report, it takes some courage to face a hostile community.

Anyhow to the question, perhaps I should go back and read more, but what I would like to see are more specific details on the third party applications you were using, the issues they created, and how they were resolved.

I'm curious because it appears that some initial rules and choices that were made for the study were a recipe for disaster. Its like telling two teams they will be in a race to navigate a course as fast as possible and they must choose their vehicle without knowing what the course will be and they are stuck with whatever vehicle they chose. One team chooses an Formula 1 race car while the other picks a nice luxury yacht. The race course turns out to be from the Florida Keys to Jamaica and back. The Formula 1 guys are forced to make their car work as a boat because the rules say you already chose so your stuck with it.

Okay, so thats a bit extreme and perhaps I'm reading too much into your specifications for the model. For all I know the linux guys doomed themselves. But it sounds like the third party add-ons you were using are not properly supported on SuSE linux. If your results were typical of maintaining a linux e-commerce website then I doubt much of anyone would be using linux.

This scenario seems to be a common occurrence when windows and linux are benchmarked and reported in a Microsoft funded study, note the following url:

http://www.kegel.com/nt-linux-benchmarks.html

When the grueling details are scrutinized there are some real issues that need to be resolved and the comparisons and details provide a benefit to the linux community and to Microsoft. What is not beneficial is touting the superiority of one OS over another based on some finding which is sqewed by picking a weak point which could be easily overcome by picking the correct software, hardware, and configuration.

So how about some grueling details? ;)

burnin

Here we go...http://kegel.com/linux/comfort/, but was surprised to read that OpenOffice2.0 takes 5 minutes to save a 12MB file. This to me is unacceptable.

You don't need to buy MontaVista; you can compile everything using Kegel's crosstool. Also, all flash locations are unlockable at the OS-level by the hacker so, if you're confident, you can write your own u-boot (or whatever) to flash. Of course if you flash something bad you'll have a paperweight and will need to go in through JTAG.

Root access is available by hitting ctrl-shift-1 from within our GUI. Once you're there you can enable remote root login by setting a passwd and firing up sshd.

We also have three serial ports, one of which is pre-configured for console.

Actually, I would.

Quite simply, we've been too busy getting the device prepared for the consumer market to worry about advertising its hackability. At some point in the near future we'll be fully documenting the SDK and possibly an HDK but for now we're focusing on the consumer aspect.

Regarding crosstool support, we include libraries to run both apps compiled natively on the Pad, compiled using MontaVista's cross-compiler, and compiled using Kegel's crosstool. There's a set of dev environment RPMs sitting on the Pad just waiting to be installed.

For our initial release it's up to the hacker to find the stuff and figure out how to use it (simple once you find it) but, in the future, it will be fully documented once we have some spare cycles to burn.

But take some time tuning it, and your OS's TCP/IP stack, and you can easily outperform even Zeus.

You clearly have no idea what you are talking about. Apache uses a single thread / process per connection. Zeus multiplexes all connections in a single (or multiple, one per CPU) process. Zeus and any similarly architected server will always outperform Apache.

Apache works well for a lot of things, but speed isn't one of them. Even if the test was fair, I wouldn't be surprised if IIS is faster than Apache. A comparison to Zeus or lighttpd would be better.

A good starting place to learn about server architecture is The C10k problem page.

I hope Matt expands his work to include super-scalable I/O systems like I/O Completion Ports.

This has been in FreeBSD for about five years and is called kqueue. It exhibits O(1) scaling and goes like a rocket.

Check out these rather old benchmarks.

It's on OSX too, of course.

Dave