You can't just choose to rotate a drive platter 100 times faster. It may be remotely possible to make a 50,000 rpm drive in the near term, but 100x faster (and larger diameter) is WAY beyond the limits of known materials. It could also double as a dandy KE tank killer or space launch system if it could actually be built...
The limitation isn't the ability to read the data off the platter, it is the ability of the platter to not break into shrapnel.
I was elated when Apple announced the original Open Source Darwin initiative. I never would have guessed they would go for it, and I think it is a Very Good Thing.
Getting everything together for a public release is a very non-trivial task. I know the hassles we go through, and darwin is 100x the size of our codebase.
After all that work, including pressing CD's, it was met with a fairly resounding silence.
The darwin mailing lists were dead. It sometimes seemed like there were a grand total of a dozen people with darwin installed.
It was looking like this might go down as a large example of how going to the trouble of Open Source doesn't get you anything but hassle.
It didn't help that darwin was basically unusable by itself, because all you got was a single very slow text console with messed up key bindings. Not exactly a happy development environment.
(most of the active development work is done in the usable environment of OS-X server)
The general response that interested people gave as to why they weren't doing any development with darwin was that "everything is going to change in the next release" (the driver architecture was massively reworked).
Well, the new release is here now. There is still the problematic issue that you can't run ANY current gui on darwin 1.0. OS-X server and the developer seeds of OS-X client are both out of sync with the darwin codebase. All the excuses won't really go away until the next OS-X client release.
A couple months ago, I took on the porting of X windows to Darwin, so it could actually be considered halfway usable by itself.
I released the patches to get X windows running under MacOS-X server, which was basically the same core as the earlier darwin release.
I was then given the same excuse as other people -- why bother porting to the native darwin video and input drivers if everything is going to change soon?
As of now, I am actively feeling guilty about not finishing it. Everything is there for me now, I just need to find the time.
I had been spending my weekends on either GLX or darwin X server work after Q3 shipped, but my R&D "research" has shifted to "development" faster than I expected, and the past few weekends have been monopolized by new engine work. I'll get to it within the next month, but if someone wants to pick up first, feel free...
It may turn out that many of ESR's arguments just don't pan out for Apple, as far as having outsiders improve the core codebase. Even so, releasing the source will benefit Apple by giving application developers the "ultimate docs" on the OS.
I think Apple deserves a lot of credit for the step.
A GeForce should be able to run Q3 at 200 fps at 400x300 (r_mode 1) or possibly even 512x384 resolution if the cpu was fast enough. A dual willamette at the end of this year will probably do it.
We currently see 100+ fps timedemos at 640x480 with either a 1ghz processor or dual 800's, and that isn't completely fill rate limited. DDR GeForce cards are really, really fast.
Yes, it is almost completely pointless.
The only reasonable argument for super high framerates is to do multi frame composited motion blur, but it turns out that it isn't all that impressive.
I did a set of offline renderings of running Q3 at 1000 fps and blending down to 60 fps for display. Looked at individually, the screenshots were AWESOME, with characters blurring through their animations and gibs streaking off the screen, but when they were played at 60hz, nobody could tell the difference even side by side.
Motion blur is more important at 24hz movie speeds, but at higher monitor retrace rates it really doesn't matter much.
There are some poster-child cases for it, like a spinning wagon wheel, but for most aspects of a FPS, realistic motion blur isn't noticable.
Exagerated motion blur (light sabers, etc) is a separate issue, and doesn't require ultra-high framerates.
There are still plenty of things we can usefully burn faster cpu's on...
I have been working on the utah-glx project for about nine months now. I am proud of what we have accomplished, and I think it has been a good example of a working open source project. Matrox and ATI have been pleasantly surprised at how well things have worked out.
However, there are only a half dozen coders working part time on utah-glx, and we are split among three active chipset trees. Nvidia has more people than that working full time exclusively for their chips. We are pretty good. So are they. We can work from specs. They can go interrogate the designer of the hardware. It's a pretty simple equation - I expect their driver to be better than our drivers.
Nvidia is working to maintain a common source base between their windows drivers and their linux drivers. Bugs tracked down by the order of magnitude more windows users will be fixed automatically in the linux version.
DRI does not have all of its problems solved, and there are valid reasons for them to not use it. They might change their mind later.
It should be remembered that some people want to do 3D graphics on linux and don't care about open source principles. Most of the people coming from a technical workstation background just want a vendor to deliver a tool to help them get their work done. I also suspect that most game players will choose a faster driver, even if it is closed source.
The choice isn't between making their driver open source or closed source. They CAN'T open source it because of legal encumbrances on the code. The choice is between doing a closed source driver with their existing code, and doing a completely new driver. Not too many people get excited at the prospect of rewriting perfectly good code.
If you care about getting open source drivers, support Matrox, 3dfx, or ATI. They have released specs to the community, and put out cash for PI to develop and support DRI drivers.
If you just want good 3D, I think nvidia will satisfy you. As for not being done yet, it hasn't been that long since Xfree 4.0 shipped.
I mean that I never actually worked with low level register programming specs for the amiga, so I can't comment authoritatively. The reason is that when I was young and the Amiga looked interesting, I couldn't afford one. When I had the means, I no longer had the desire.
I certainly don't mean to imply that all Amiga users are fanatics, just that the advocates that made it to my mailbox were less well mannered than those for many other platforms. You are right, it did color my response.
So, to give you a somewhat better answer:
The Amiga's success was in demonstrating the large benefits of specialized graphics coprocessors for personal computers, and providing close to a workstation like environment while the PC was still struggling with segment registers in dos.
It wouldn't have been obvious at the time, but the Amiga was basically fated to go the way of a console generation, rather than evolve as the PC or mac did.
The reliance on low level hardware knowledge and programming provided the obvious visual superiority, but also locked it in to a very ungracefull evolution.
I was only into the Apple II/IIGS during the Amiga's strong times, so I never really got to give it a fair evaluation. My impression of the Amiga is mostly colored by later years of fanatics hounding me about supporting the "inherently superior amiga" when it was obviously well past its competative prime. John Carmack
>Actually, I believe you're referring to Carl Forhan of Songbird Productions
Heh, sorry... I just assumed all jaguar development was coming from a single crazy group.:-)
Even if the memory controller hadn't been broken, performance would still have sucked really bad without a cache.
The jaguar was definately significantly hampered by its technical flaws, which kept me from ever being too big of a jaguar booster. I was proud of my work on Wolf and DOOM (more so than just about any of the other console work Id has been involved in until just recently), but in the end, the better consoles won the war.
I actually dug up all my old jaguar development hardware to give to these guys a year or two ago.
Unfortunately, it turned out that I had lost the C compiler that I had retargeted to the jaguar RISC engines, so DOOM was no longer buildable.
There is something noble about developing on a dead platform -- it is so completely for the joy of the development, without any commercial motivation.
The quick recap on the jaguar:
The memory, bus, blitter and video processor were 64 bits wide, but the processors (68k and two custom risc processors) were 32 bit.
The blitter could do basic texture mapping of horizontal and vertical spans, but because there wasn't any caching involved, every pixel caused two ram page misses and only used 1/4 of the 64 bit bus. Two 64 bit buffers would have easily trippled texture mapping performance. Unfortunate.
It could make better use of the 64 bit bus with Z buffered, shaded triangles, but that didn't make for compelling games.
It offered a usefull color space option that allowed you to do lighting effects based on a single channel, isntead of RGB.
The video compositing engine was the most innovative part of the console. All of the characters in Wolf3D were done with just the back end scalar instead of blitting. Still, the experience with the limitations and hard failure cases of that gave me good amunition to rail against microsoft's (thankfully aborted) talisman project.
The little risc engined were decent processors. I was surprised that they didn't use off the shelf designs, but they basically worked ok. They had some design hazards (write after write) that didn't get fixed, but the only thing truly wrong with them was that they had scratchpad memory instead of caches, and couldn't execute code from main memory. I had to chunk the DOOM renderer into nine sequentially loaded overlays to get it working (with hindsight, I would have done it differently in about three...).
The 68k was slow. This was the primary problem of the system. You options were either taking it easy, running everything on the 68k, and going slow, or sweating over lots of overlayed parallel asm chunks to make something go fast on the risc processors.
That is why playstation kicked so much ass for development -- it was programmed like a single serial processor with a single fast accelerator.
If the jaguar had dumped the 68k and offered a dynamic cache on the risc processors and had a tiny bit of buffering on the blitter, it could have put up a reasonable fight against sony.
Now the LYNX, on the other hand, was very much The Right Thing from a programming standpoint. A fast little processor (for its niche), a good color bitmapped display, and a general purpose blitter.
Price and form factor weighed too heavily against it.
I have been following both Cray and Tera for many years now. I have been saddened watching the last of the supercomputer companies wither and die. Supercomputers were always so COOL, but for most things, they just aren't so "super" any more.
I have benchmarked several of my back end utilities on cray systems, and one of them on the early tera machine (the early compiler exploded on the others). None of the single processor runs were as fast as a pentium III, and this was quite some time ago.
Understand that this was often branchy and recursive code running with only 3D sized vectors, so it isn't the sweet spot for traditional supercomputers. If I was doing nothing but multiplying 1k by 1k matricies of doubles, even a five year old cray would kick the crap out of the latest athalon. Unfortunately, none of my code looks like that.
I even spent some time thinking how I could restructure calculations into a vectorizable form, which might make a cray J90 competative. I wanted to buy a Cray! Of course, this was silly. It took less effort to make the code SMP friendly, and the payoff was much larger.
We wound up with a 16 processor SGI origin-2000 system, which has been easy to develop for and predictable in performance. We just recently bought an 8 processor Xeon, which is actually faster than our old 16 processor SGI, but at exactly one tenth the price (the downside is that it is maxed out, while the SGI still has tons of growth potential).
I program all heavy workloads in a parallel fashion now as a matter of habit, but it is easy to overstate the benefits of parallel systems.
The common lunux advocate position of "beowulf makes supercomputers obsolete" isn't quite right. Even with code that is already written in a parallel manner, there is a large difference between developing for a shared memory system and a distributed cluster. Developing a compute (and especially data) intensive program for a cluster rather sucks.
If there was a single processor system that was really four times as fast as "consumer" machines, even if it cost fifty times as much, we would buy it. Unfortunately, there isn't. When the product release cycles are favorable, Alpha systems may be twice as good as x86 systems, but not much beyond that unless you are doing the 1k by 1k matrix type stuff.
It is often forgotten that the original Cray-1 was largely a success because it was the fastest SCALAR processor of the time. The vectorization was just a bonus. Now, vectorization is the only thing that gives them a reason for existance.
The tera architecture is very interesting, but for scalar code, it is VERY, VERY slow. I'm not sure if it will be competative with large processor count SGI origin-2000 systems even after it matures. It gains ease of programming from the lack of caches, but it gives away a lot of problem domains where it is going to look stupidly slow.
If a supercomputer company could make a scalar processor that ran many times faster than existing processors and had similar SMP capabilities, it would probably be a success. Even if it cost a million dollars, filled a room, and burned hundreds of kwatts.
The problem isn't really that supercomputers are bad, its just that we are so spoiled by how AMAZINGLY GOOD our cheap consumer hardware is.
I do still worry about the stiffling of innovation that comes from having so few architectural directions for systems, but in the end, wall clock performance is what really matters.
I had talked with Zoid about it a few times over the years, and I think he is making a pretty good decision.
It wasn't always clear from the outside, but Zoid was a remote contractor, not an employee. It was a low key relationship that worked out well for all of us. He stayed in canada and basically worked on whatever he liked, because I thought he had pretty good judgement. He had responsibility for the linux ports and the CTF code, but much of his time was his to allocate as he wanted.
With Loki now picking up the maintenance of the linux port (as well as my steadily increasing involvement with linux), and a new game design starting at Id, his choice was basically to either go develop a brand new Q3 mod by himself, or go work for one of the many gaming companies that had been trying to hire him.
We weren't interested in bringin on another core programmer at id, especially another one with immigration hassles (we have had enough issues with that for a small company). We would have been happy to continue the current arrangement indefinately, but he wanted to get out of the holding pattern.
Another thing he mentioned that I am sympathetic to is the desire to get a bit out of the community limelight. Being a public figure of some note isn't always all it is cracked up to be.
We are parting on the best of terms (leaving right AFTER a project completes is the considerate way to go). He is going to finish up the Quake2 linux updates (better X/GLX support), even if he has to complete the work from his new job.
There are valid, legal ways to provide a level of protection equal to closed source binaries (which is really only a level of obfuscation).
I realize that they (proxies / loaders / obfuscated modules) may be more of a hassle, but he doesn't get to choose to break the license to avoid a hassle. I traded several emails with Slade over the past month, and I still have a degree of sympathy for his position, but I can't just let him walk around the code license.
All the conspiracy theories about me wanting to destroy the Quake community are silly. I loved what happened with the DOOM source release, and I hoped that the Quake release would have similar effects.
An Nvidia GeForce/quadro kicks the crap out of a $100k+ SGI reality engine on just about any test you could devise.
The reality engine is quite a few years old, and its follow on, the infinite reality and IR2 still have some honest advantages. You can scale them with enough raster boards so that they have more fill rate, and they do have some other important features like configurable multisample anti-aliasing and 48 bit color.
Even now, lots of applications can be compiled on both platforms and just run much better on WinNT+nvidia than on the best sgi platform available. Having a 700mhz+ main processor helps, irrespective of the graphics card.
Some people still have a knee-jerk reaction that claims that "real applications" still run better on the high end hardware, and there are indeed some examples, but the argument bears a whole lot of resemblence to the ones put forth by the last defenders of ECL vector supercomputers.
Most people's applications don't show advantages on high end SGI hardware.
The big issue is the pace of progress -- SGI aimed for a new graphics family every three years with a speed bump in between. The PC vendors aim for a new generation every year with a speed bump at six months.
Lots of people were just waving their hands saying "Netreck solved this years ago" without being specific.
As you say, it isn't really a solution. If there were a couple million people playing netrek, there would be cheat proxies for it just like there are for Quake.
I think everyone can agree that it just isn't possible to solve the problem completely, you can only make it more difficult, which is exactly what the netrek solution is -- release binaries without the exact source and make it difficult to decypher.
First, the Quake architecture of (reletively) dumb clients conencted to an authoritative server prevents the egregious cheating possible in some games ("I say you are dead now!", "I say I have infinite ammo!").
For the most part, a cheating client can't make their character do anything that couldn't happen as a result of normal game interaction.
The cheating clients/proxies focus on two main areas -- giving the player more information than they should have, and performing actions more skillfully.
The "more information" part can take a number of forms. A reletively harmless one is adding timers for items and powerups. Good players will track a lot of that in their heads, but a simple program can "remind" players of it.
Media cheating provides more information. Changing all the other player skins to bright white and removing all the shadows from a level give players an advantage not within the spirit of the game. Some would say cranking your screen brightness and gamma way up is one step on that path.
More advanced clients can make available information that is not normally visible at all. The server sends over all of the entities in the potentially visible set, because the client can move around a fair amount between updates. This means that the client is often aware of the locations of players that are around corners. A proxy can display this information in a "scanner window". The server could be changed to only send over clients actually visible, but that would result in lots of players blinking in and out as you move around or turn rapidly.
The worst cheats are the aim bots. In addition to providing more information, they override the player's commands to aim and fire with very high accuracy. The player usually "drives" around the level, and the program aims and shoots for them. This is usually extremely devestating and does ruin the game for most people.
There are many possible countermeasures.
There are server-side countermeasures that look for sequences of moves that are likely to be bot-generated and not human-generated, but that is an arms race that will end with skilled human players eventually getting identified as subtle bots.
Media cheats can be protected by various checksums, as we do in Q3 with the sv_pure option. This is only effective if the network protocol is not compromised, because otherwise a proxy can tell the client that it's hacked media are actually ok.
If the network protocol is not known, then the extra-information cheats generally can't happen unless you can hack the client source.
Q3 performs various bits of encryption on the network protocol, but that is only relying on security through obscurity, and a sufficiently patient person with a disassembler can eventually backtrack what is happening. If only they would find something more usefull to spend their time on...
With an open source client, the network communication protocol is right there in the open, so any encryption would be futile.
Any attempt at having the client verify itself isn't going to work out, because a cheating client can just always tell you what you want to hear. People have mentioned nettreck several times, but I don't see how a completelty open source program can keep someone from just reporting what it is supposed to for a while (perhapse using a "real" copy to generate whatever digests are asked for), then switching to new methods. Anyone care to elaborate?
I think a reasonable plan is to modify QW so that to play in "competition mode", it would have to be launched by a separate closed-source program that does all sorts of encryption and verification of the environment. If it just verifies the client, it would prevent the trivial modified client scanners and aim bots. It could verify the media data to prevent media information cheating. To prevent proxy information cheating and aim bots, it would have to encrypt the entire data stream, not just the connection process. That might have negative consequences on latency unless the encrypter is somehow able to be in the same address space as the verified client or scheduling can be tweaked enough to force task switches right after sends.
In the end, it is just a matter of making it more difficult for the cheaters. If all it takes is editing and recompiling a file, lots of people will cheat. This is indeed a disadvantage of open source games. If they have to grovel over huge network dumps and disassemblies to hack a protocol, a smaller number of cheats will be available.
Even if the programs were completely guaranteed secure (I havem't been convinced that is possible even in theory), an aim bot could be implemented at the device driver level.
It would be a lot more work, but a program could be implemented that intercepts the video driver, the mouse driver, and the keyboard driver, and does bot calculations completely from that.
Kind of sucks, doesn't it?
John Carmack
Re:Just thought this was important to say
on
Quake 1 GPL'ed
·
· Score: 2
How about this:
Make a closed source program that acts as an exe loader / verifier / proxy for the open source main game.
Nope. We are the copyright holder of all works, and we can release any part of it under any license we choose.
Completely aside from that, I think it is still unclear exactly where the GPL wants the separation of code and content.
Few would argue that every document read by a GPL word processor would be covered by the GPL, and most would place maps entered by quake into that catagory, but things can quickly get murky.
Quake game mods are written in QC, but turned into data to be processed by the main code. I think the spirit of the GPL would want that code to be released, but it is only a small step from there to saying that every program loaded by a GPL operating system must be GPL, which is clearly not the case.
Heh. You don't know how much trouble it is to convince biz oriented people that this isn't just plain stupid.
While thinking in terms of money and profit are probably good ways of understanding the way most things work in the world, don't let yourself become so jaded or cynical to think that it is the ONLY way things work.
I do think The World Would Be A Better Place if all software companies released older code so users still interested could work with it or learn from it. (I'm not holding my breath, though)
John Carmack
Mac glquake should be pretty easy now
on
Quake 1 GPL'ed
·
· Score: 4
Producing a mac version of glquake or glquakeworld should be pretty easy with the existing code now that Apple has real OpenGL support.
Producing a version of the software renderer with decent performance would be VERY HARD. A huge amount of effort went into the assembly optimization for the PPC, and it still didn't quite measure up to the x86 code.
When the article first showed up, I thought "It IS documented in the release!". I went and looked, and unfortunately, that documentation from the previous release didn't make it into the latest release. Sigh. Our fuckup.
Apropriate quote: "Never attribute to malice what can be explained by incompetence".
I remain unconvinced that we have done something morally offensive.
Yes, we could have (should have, meant to) included a notice that it was going on in the EULA, but honestly, how many people carefully read and consider every line of all the EULA's they click through? How much of a difference would that have made to people?
I dislike lengthy legal verbiage, but it is reactions exactly like these that cause them to grow. Every time someone says "Sue 'em!" over something, a lawyer proposes another paragraph in a license document.
The most upstanding thing to do would be to have explicit UI that asks on installation if you don't mind sending your data when you play multiplayer games. I would consider that justified if we were sending a detailed system spec. That is something we may want to do in the future. Data like that is helpfull in making good development decisions.
But this is just a driver string riding along with your game version. It just seems silly, like requiring you to acknowledge before leaving your house that someone might see you. I would rather have fixed a bug somewhere.
I can see that it is a slipperly slope to be on, and I can easily project it to a scenario that I would be offended by, but I just can't convince myself that knowing the reletive distribution of different OpenGL implementations is violating people's rights.
The system was set up to allow us to notify people with a one-line message when their versions are out of date. I imagine some people are offended even by that, but I consider that a positive service to the community.
Including the renderer string was an afterthought to get some good unbiased data to help make future decisions on. Every once in a while we tally up the numbers, then dump all the logs. That's it.
This has been discussed before, and has been going on with the previous tests.
The message of the day server was intended as a half-assed auto update feature that could be cross platform.
We send a normal message most of the time, but if the version is out of date, we can send a message with telling you where to get the update.
I didn't want to deal with binary auto-updates on three platforms, and I worry a bit about security issues with that in any case.
You can disable it by setting "cl_motd 0" when the game starts up if you really don't want to send anything or see our message.
We added the result of glGetString( GL_RENDER ) to get some much needed information about the distribution of video cards and drivers.
We can see how many people aren't following directions and running glsetup. This is a big support issue.
We can see how many people are running minidrivers, which are going to make our lives a mess in the future.
We can see how many mac (steady 5%) and linux (5%at initial release, tailed off to 2%, probably due to dual booting) people are playing.
Getting this information has been usefull. We can compare the numbers of people playing with a given card with the amount of support emails we field, so we know which vendors (3DFX) we need to give more crap about their driver quality.
For back end rendering, they have a room full of MP sparc boxes. To my "SPARC? Why use the slowest of risc processors?" question, they replied that it isn't the speed of the individual processors that was important to them, but the speed PER CUBIC FOOT OF SPACE. Sun made quad pizza boxes, so it was computationally dense.
For modeling and development, they use a lot of SGI octanes. They also use linux + mesa for some internal tools.
When it debuted, it was a very good all around performar, and it had the highest fill rate of any intel based system.
Now, an Nvidia GeForce is just plain superior in almost every aspect. Higher fill rate, even in high res, 32 bit, trilinear modes. Faster, more capable geometry acceleration.
Any remaining areas of SGI superiority are probably due to driver optimization rather than hardware architecture. Nvidia hasn't had much call to optimize stippled lines, for instance.
The super-memory-system wasn't all it was touted to be. It worked well for sharing the load between the graphics and the cpu, but the cpu didn't actually see any better bandwidth than on a standard intel chipset. The cpu write bandwidth was actually about 10% LOWER than a consumer machine.
I have used a lot of intergraph and sgi machines, and the bottom line is that the consumer hardware has just outpaced the workstation hardware because they were on different growth curves. The workstations are better than they have ever been, but the consumer systems are just orders of magnitude better than they used to be.
I think SGI shot too low with the VW's graphics, somewhat out of fear of canibalizing their other workstations, and somewhat out of underestimating the consumer competition. Being quite a bit late didn't help, either.
Hey, >I didn't say "virtual reality"... I tend to agree with your assessment. VR is a term loaded with high-enthusiasm / low-results connotations. We have worked with a few VR companies in the past, and I have always found them to not have finishing ability. So much of the VR world (and much other academic style research) is high concept, but sketchy on the details. Most VR experiences are heavy on the "You are in a virtual world!!!!", but don't spend too much time on exactly what you are supposed to be doing there. Can you poke and prod to find interesting things? What happens when someone pushes you? Can you dodge something effectively? Are the controls linear, or integrated over time? etc. I think that one of my strengths is a blend of idealism and pragmatism that has resulted in good results over the years. In any case, of the half dozen things I listed, I am clearly not going to be able to do all of them, so it may be a moot point... John Carmack
There have been a few things that didn't have prior art that probably could have been gotten past a patent examiner -- constant Z perspective texturing in DOOM, surface caching in Q1, and the overbright gamma table stuff to trade range for precision in Q3, for example. The patent issue came up at Id a few times until a made it perfectly clear that if the company pursued any patents, I would leave. John Carmack
Slashdot Mirror
You can't just choose to rotate a drive platter 100 times faster. It may be remotely possible to make a 50,000 rpm drive in the near term, but 100x faster (and larger diameter) is WAY beyond the limits of known materials. It could also double as a dandy KE tank killer or space launch system if it could actually be built...
The limitation isn't the ability to read the data off the platter, it is the ability of the platter to not break into shrapnel.
John Carmack
I was elated when Apple announced the original Open Source Darwin initiative. I never would have guessed they would go for it, and I think it is a Very Good Thing.
Getting everything together for a public release is a very non-trivial task. I know the hassles we go through, and darwin is 100x the size of our codebase.
After all that work, including pressing CD's, it was met with a fairly resounding silence.
The darwin mailing lists were dead. It sometimes seemed like there were a grand total of a dozen people with darwin installed.
It was looking like this might go down as a large example of how going to the trouble of Open Source doesn't get you anything but hassle.
It didn't help that darwin was basically unusable by itself, because all you got was a single very slow text console with messed up key bindings. Not exactly a happy development environment.
(most of the active development work is done in the usable environment of OS-X server)
The general response that interested people gave as to why they weren't doing any development with darwin was that "everything is going to change in the next release" (the driver architecture was massively reworked).
Well, the new release is here now. There is still the problematic issue that you can't run ANY current gui on darwin 1.0. OS-X server and the developer seeds of OS-X client are both out of sync with the darwin codebase. All the excuses won't really go away until the next OS-X client release.
A couple months ago, I took on the porting of X windows to Darwin, so it could actually be considered halfway usable by itself.
I released the patches to get X windows running under MacOS-X server, which was basically the same core as the earlier darwin release.
I was then given the same excuse as other people -- why bother porting to the native darwin video and input drivers if everything is going to change soon?
As of now, I am actively feeling guilty about not finishing it. Everything is there for me now, I just need to find the time.
I had been spending my weekends on either GLX or darwin X server work after Q3 shipped, but my R&D "research" has shifted to "development" faster than I expected, and the past few weekends have been monopolized by new engine work. I'll get to it within the next month, but if someone wants to pick up first, feel free...
It may turn out that many of ESR's arguments just don't pan out for Apple, as far as having outsiders improve the core codebase. Even so, releasing the source will benefit Apple by giving application developers the "ultimate docs" on the OS.
I think Apple deserves a lot of credit for the step.
John Carmack
A GeForce should be able to run Q3 at 200 fps at 400x300 (r_mode 1) or possibly even 512x384 resolution if the cpu was fast enough. A dual willamette at the end of this year will probably do it.
We currently see 100+ fps timedemos at 640x480 with either a 1ghz processor or dual 800's, and that isn't completely fill rate limited. DDR GeForce cards are really, really fast.
Yes, it is almost completely pointless.
The only reasonable argument for super high framerates is to do multi frame composited motion blur, but it turns out that it isn't all that impressive.
I did a set of offline renderings of running Q3 at 1000 fps and blending down to 60 fps for display. Looked at individually, the screenshots were AWESOME, with characters blurring through their animations and gibs streaking off the screen, but when they were played at 60hz, nobody could tell the difference even side by side.
Motion blur is more important at 24hz movie speeds, but at higher monitor retrace rates it really doesn't matter much.
There are some poster-child cases for it, like a spinning wagon wheel, but for most aspects of a FPS, realistic motion blur isn't noticable.
Exagerated motion blur (light sabers, etc) is a separate issue, and doesn't require ultra-high framerates.
There are still plenty of things we can usefully burn faster cpu's on...
John Carmack
I have been working on the utah-glx project for about nine months now. I am proud of what we have accomplished, and I think it has been a good example of a working open source project. Matrox and ATI have been pleasantly surprised at how well things have worked out.
However, there are only a half dozen coders working part time on utah-glx, and we are split among three active chipset trees. Nvidia has more people than that working full time exclusively for their chips. We are pretty good. So are they. We can work from specs. They can go interrogate the designer of the hardware. It's a pretty simple equation - I expect their driver to be better than our drivers.
Nvidia is working to maintain a common source base between their windows drivers and their linux drivers. Bugs tracked down by the order of magnitude more windows users will be fixed automatically in the linux version.
DRI does not have all of its problems solved, and there are valid reasons for them to not use it. They might change their mind later.
It should be remembered that some people want to do 3D graphics on linux and don't care about open source principles. Most of the people coming from a technical workstation background just want a vendor to deliver a tool to help them get their work done. I also suspect that most game players will choose a faster driver, even if it is closed source.
The choice isn't between making their driver open source or closed source. They CAN'T open source it because of legal encumbrances on the code. The choice is between doing a closed source driver with their existing code, and doing a completely new driver. Not too many people get excited at the prospect of rewriting perfectly good code.
If you care about getting open source drivers, support Matrox, 3dfx, or ATI. They have released specs to the community, and put out cash for PI to develop and support DRI drivers.
If you just want good 3D, I think nvidia will satisfy you. As for not being done yet, it hasn't been that long since Xfree 4.0 shipped.
John Carmack
I mean that I never actually worked with low level register programming specs for the amiga, so I can't comment authoritatively. The reason is that when I was young and the Amiga looked interesting, I couldn't afford one. When I had the means, I no longer had the desire.
I certainly don't mean to imply that all Amiga users are fanatics, just that the advocates that made it to my mailbox were less well mannered than those for many other platforms. You are right, it did color my response.
So, to give you a somewhat better answer:
The Amiga's success was in demonstrating the large benefits of specialized graphics coprocessors for personal computers, and providing close to a workstation like environment while the PC was still struggling with segment registers in dos.
It wouldn't have been obvious at the time, but the Amiga was basically fated to go the way of a console generation, rather than evolve as the PC or mac did.
The reliance on low level hardware knowledge and programming provided the obvious visual superiority, but also locked it in to a very ungracefull evolution.
John Carmack
I was only into the Apple II/IIGS during the Amiga's strong times, so I never really got to give it a fair evaluation. My impression of the Amiga is mostly colored by later years of fanatics hounding me about supporting the "inherently superior amiga" when it was obviously well past its competative prime. John Carmack
>Actually, I believe you're referring to Carl Forhan of Songbird Productions
:-)
Heh, sorry... I just assumed all jaguar development was coming from a single crazy group.
Even if the memory controller hadn't been broken, performance would still have sucked really bad without a cache.
The jaguar was definately significantly hampered by its technical flaws, which kept me from ever being too big of a jaguar booster. I was proud of my work on Wolf and DOOM (more so than just about any of the other console work Id has been involved in until just recently), but in the end, the better consoles won the war.
John Carmack
I actually dug up all my old jaguar development hardware to give to these guys a year or two ago.
Unfortunately, it turned out that I had lost the C compiler that I had retargeted to the jaguar RISC engines, so DOOM was no longer buildable.
There is something noble about developing on a dead platform -- it is so completely for the joy of the development, without any commercial motivation.
The quick recap on the jaguar:
The memory, bus, blitter and video processor were 64 bits wide, but the processors (68k and two custom risc processors) were 32 bit.
The blitter could do basic texture mapping of horizontal and vertical spans, but because there wasn't any caching involved, every pixel caused two ram page misses and only used 1/4 of the 64 bit bus. Two 64 bit buffers would have easily trippled texture mapping performance. Unfortunate.
It could make better use of the 64 bit bus with Z buffered, shaded triangles, but that didn't make for compelling games.
It offered a usefull color space option that allowed you to do lighting effects based on a single channel, isntead of RGB.
The video compositing engine was the most innovative part of the console. All of the characters in Wolf3D were done with just the back end scalar instead of blitting. Still, the experience with the limitations and hard failure cases of that gave me good amunition to rail against microsoft's (thankfully aborted) talisman project.
The little risc engined were decent processors. I was surprised that they didn't use off the shelf designs, but they basically worked ok. They had some design hazards (write after write) that didn't get fixed, but the only thing truly wrong with them was that they had scratchpad memory instead of caches, and couldn't execute code from main memory. I had to chunk the DOOM renderer into nine sequentially loaded overlays to get it working (with hindsight, I would have done it differently in about three...).
The 68k was slow. This was the primary problem of the system. You options were either taking it easy, running everything on the 68k, and going slow, or sweating over lots of overlayed parallel asm chunks to make something go fast on the risc processors.
That is why playstation kicked so much ass for development -- it was programmed like a single serial processor with a single fast accelerator.
If the jaguar had dumped the 68k and offered a dynamic cache on the risc processors and had a tiny bit of buffering on the blitter, it could have put up a reasonable fight against sony.
Now the LYNX, on the other hand, was very much The Right Thing from a programming standpoint. A fast little processor (for its niche), a good color bitmapped display, and a general purpose blitter.
Price and form factor weighed too heavily against it.
John Carmack
I have been following both Cray and Tera for many years now. I have been saddened watching the last of the supercomputer companies wither and die. Supercomputers were always so COOL, but for most things, they just aren't so "super" any more.
I have benchmarked several of my back end utilities on cray systems, and one of them on the early tera machine (the early compiler exploded on the others). None of the single processor runs were as fast as a pentium III, and this was quite some time ago.
Understand that this was often branchy and recursive code running with only 3D sized vectors, so it isn't the sweet spot for traditional supercomputers. If I was doing nothing but multiplying 1k by 1k matricies of doubles, even a five year old cray would kick the crap out of the latest athalon. Unfortunately, none of my code looks like that.
I even spent some time thinking how I could restructure calculations into a vectorizable form, which might make a cray J90 competative. I wanted to buy a Cray! Of course, this was silly. It took less effort to make the code SMP friendly, and the payoff was much larger.
We wound up with a 16 processor SGI origin-2000 system, which has been easy to develop for and predictable in performance. We just recently bought an 8 processor Xeon, which is actually faster than our old 16 processor SGI, but at exactly one tenth the price (the downside is that it is maxed out, while the SGI still has tons of growth potential).
I program all heavy workloads in a parallel fashion now as a matter of habit, but it is easy to overstate the benefits of parallel systems.
The common lunux advocate position of "beowulf makes supercomputers obsolete" isn't quite right. Even with code that is already written in a parallel manner, there is a large difference between developing for a shared memory system and a distributed cluster. Developing a compute (and especially data) intensive program for a cluster rather sucks.
If there was a single processor system that was really four times as fast as "consumer" machines, even if it cost fifty times as much, we would buy it. Unfortunately, there isn't. When the product release cycles are favorable, Alpha systems may be twice as good as x86 systems, but not much beyond that unless you are doing the 1k by 1k matrix type stuff.
It is often forgotten that the original Cray-1 was largely a success because it was the fastest SCALAR processor of the time. The vectorization was just a bonus. Now, vectorization is the only thing that gives them a reason for existance.
The tera architecture is very interesting, but for scalar code, it is VERY, VERY slow. I'm not sure if it will be competative with large processor count SGI origin-2000 systems even after it matures. It gains ease of programming from the lack of caches, but it gives away a lot of problem domains where it is going to look stupidly slow.
If a supercomputer company could make a scalar processor that ran many times faster than existing processors and had similar SMP capabilities, it would probably be a success. Even if it cost a million dollars, filled a room, and burned hundreds of kwatts.
The problem isn't really that supercomputers are bad, its just that we are so spoiled by how AMAZINGLY GOOD our cheap consumer hardware is.
I do still worry about the stiffling of innovation that comes from having so few architectural directions for systems, but in the end, wall clock performance is what really matters.
John Carmack
> Static web serving is not problem (once you debug the code).
Nothing is a problem once you debug the code.
John Carmack
I had talked with Zoid about it a few times over the years, and I think he is making a pretty good decision.
It wasn't always clear from the outside, but Zoid was a remote contractor, not an employee. It was a low key relationship that worked out well for all of us. He stayed in canada and basically worked on whatever he liked, because I thought he had pretty good judgement. He had responsibility for the linux ports and the CTF code, but much of his time was his to allocate as he wanted.
With Loki now picking up the maintenance of the linux port (as well as my steadily increasing involvement with linux), and a new game design starting at Id, his choice was basically to either go develop a brand new Q3 mod by himself, or go work for one of the many gaming companies that had been trying to hire him.
We weren't interested in bringin on another core programmer at id, especially another one with immigration hassles (we have had enough issues with that for a small company). We would have been happy to continue the current arrangement indefinately, but he wanted to get out of the holding pattern.
Another thing he mentioned that I am sympathetic to is the desire to get a bit out of the community limelight. Being a public figure of some note isn't always all it is cracked up to be.
We are parting on the best of terms (leaving right AFTER a project completes is the considerate way to go). He is going to finish up the Quake2 linux updates (better X/GLX support), even if he has to complete the work from his new job.
John Carmack
There are valid, legal ways to provide a level of protection equal to closed source binaries (which is really only a level of obfuscation).
I realize that they (proxies / loaders / obfuscated modules) may be more of a hassle, but he doesn't get to choose to break the license to avoid a hassle. I traded several emails with Slade over the past month, and I still have a degree of sympathy for his position, but I can't just let him walk around the code license.
All the conspiracy theories about me wanting to destroy the Quake community are silly. I loved what happened with the DOOM source release, and I hoped that the Quake release would have similar effects.
John Carmack
An Nvidia GeForce/quadro kicks the crap out of a $100k+ SGI reality engine on just about any test you could devise.
The reality engine is quite a few years old, and its follow on, the infinite reality and IR2 still have some honest advantages. You can scale them with enough raster boards so that they have more fill rate, and they do have some other important features like configurable multisample anti-aliasing and 48 bit color.
Even now, lots of applications can be compiled on both platforms and just run much better on WinNT+nvidia than on the best sgi platform available. Having a 700mhz+ main processor helps, irrespective of the graphics card.
Some people still have a knee-jerk reaction that claims that "real applications" still run better on the high end hardware, and there are indeed some examples, but the argument bears a whole lot of resemblence to the ones put forth by the last defenders of ECL vector supercomputers.
Most people's applications don't show advantages on high end SGI hardware.
The big issue is the pace of progress -- SGI aimed for a new graphics family every three years with a speed bump in between. The PC vendors aim for a new generation every year with a speed bump at six months.
John Carmack
Thank you.
Lots of people were just waving their hands saying "Netreck solved this years ago" without being specific.
As you say, it isn't really a solution. If there were a couple million people playing netrek, there would be cheat proxies for it just like there are for Quake.
I think everyone can agree that it just isn't possible to solve the problem completely, you can only make it more difficult, which is exactly what the netrek solution is -- release binaries without the exact source and make it difficult to decypher.
John Carmack
First, the Quake architecture of (reletively) dumb clients conencted to an authoritative server prevents the egregious cheating possible in some games ("I say you are dead now!", "I say I have infinite ammo!").
For the most part, a cheating client can't make their character do anything that couldn't happen as a result of normal game interaction.
The cheating clients/proxies focus on two main areas -- giving the player more information than they should have, and performing actions more skillfully.
The "more information" part can take a number of forms. A reletively harmless one is adding timers for items and powerups. Good players will track a lot of that in their heads, but a simple program can "remind" players of it.
Media cheating provides more information. Changing all the other player skins to bright white and removing all the shadows from a level give players an advantage not within the spirit of the game. Some would say cranking your screen brightness and gamma way up is one step on that path.
More advanced clients can make available information that is not normally visible at all. The server sends over all of the entities in the potentially visible set, because the client can move around a fair amount between updates. This means that the client is often aware of the locations of players that are around corners. A proxy can display this information in a "scanner window". The server could be changed to only send over clients actually visible, but that would result in lots of players blinking in and out as you move around or turn rapidly.
The worst cheats are the aim bots. In addition to providing more information, they override the player's commands to aim and fire with very high accuracy. The player usually "drives" around the level, and the program aims and shoots for them. This is usually extremely devestating and does ruin the game for most people.
There are many possible countermeasures.
There are server-side countermeasures that look for sequences of moves that are likely to be bot-generated and not human-generated, but that is an arms race that will end with skilled human players eventually getting identified as subtle bots.
Media cheats can be protected by various checksums, as we do in Q3 with the sv_pure option. This is only effective if the network protocol is not compromised, because otherwise a proxy can tell the client that it's hacked media are actually ok.
If the network protocol is not known, then the extra-information cheats generally can't happen unless you can hack the client source.
Q3 performs various bits of encryption on the network protocol, but that is only relying on security through obscurity, and a sufficiently patient person with a disassembler can eventually backtrack what is happening. If only they would find something more usefull to spend their time on...
With an open source client, the network communication protocol is right there in the open, so any encryption would be futile.
Any attempt at having the client verify itself isn't going to work out, because a cheating client can just always tell you what you want to hear. People have mentioned nettreck several times, but I don't see how a completelty open source program can keep someone from just reporting what it is supposed to for a while (perhapse using a "real" copy to generate whatever digests are asked for), then switching to new methods. Anyone care to elaborate?
I think a reasonable plan is to modify QW so that to play in "competition mode", it would have to be launched by a separate closed-source program that does all sorts of encryption and verification of the environment. If it just verifies the client, it would prevent the trivial modified client scanners and aim bots. It could verify the media data to prevent media information cheating. To prevent proxy information cheating and aim bots, it would have to encrypt the entire data stream, not just the connection process. That might have negative consequences on latency unless the encrypter is somehow able to be in the same address space as the verified client or scheduling can be tweaked enough to force task switches right after sends.
In the end, it is just a matter of making it more difficult for the cheaters. If all it takes is editing and recompiling a file, lots of people will cheat. This is indeed a disadvantage of open source games. If they have to grovel over huge network dumps and disassemblies to hack a protocol, a smaller number of cheats will be available.
Even if the programs were completely guaranteed secure (I havem't been convinced that is possible even in theory), an aim bot could be implemented at the device driver level.
It would be a lot more work, but a program could be implemented that intercepts the video driver, the mouse driver, and the keyboard driver, and does bot calculations completely from that.
Kind of sucks, doesn't it?
John Carmack
How about this:
Make a closed source program that acts as an exe loader / verifier / proxy for the open source main game.
John Carmack
Nope. We are the copyright holder of all works, and we can release any part of it under any license we choose.
Completely aside from that, I think it is still unclear exactly where the GPL wants the separation of code and content.
Few would argue that every document read by a GPL word processor would be covered by the GPL, and most would place maps entered by quake into that catagory, but things can quickly get murky.
Quake game mods are written in QC, but turned into data to be processed by the main code. I think the spirit of the GPL would want that code to be released, but it is only a small step from there to saying that every program loaded by a GPL operating system must be GPL, which is clearly not the case.
John Carmack
Heh. You don't know how much trouble it is to convince biz oriented people that this isn't just plain stupid.
While thinking in terms of money and profit are probably good ways of understanding the way most things work in the world, don't let yourself become so jaded or cynical to think that it is the ONLY way things work.
I do think The World Would Be A Better Place if all software companies released older code so users still interested could work with it or learn from it. (I'm not holding my breath, though)
John Carmack
Producing a mac version of glquake or glquakeworld should be pretty easy with the existing code now that Apple has real OpenGL support.
Producing a version of the software renderer with decent performance would be VERY HARD. A huge amount of effort went into the assembly optimization for the PPC, and it still didn't quite measure up to the x86 code.
John Carmack
When the article first showed up, I thought "It IS documented in the release!". I went and looked, and unfortunately, that documentation from the previous release didn't make it into the latest release. Sigh. Our fuckup.
Apropriate quote: "Never attribute to malice what can be explained by incompetence".
I remain unconvinced that we have done something morally offensive.
Yes, we could have (should have, meant to) included a notice that it was going on in the EULA, but honestly, how many people carefully read and consider every line of all the EULA's they click through? How much of a difference would that have made to people?
I dislike lengthy legal verbiage, but it is reactions exactly like these that cause them to grow. Every time someone says "Sue 'em!" over something, a lawyer proposes another paragraph in a license document.
The most upstanding thing to do would be to have explicit UI that asks on installation if you don't mind sending your data when you play multiplayer games. I would consider that justified if we were sending a detailed system spec. That is something we may want to do in the future. Data like that is helpfull in making good development decisions.
But this is just a driver string riding along with your game version. It just seems silly, like requiring you to acknowledge before leaving your house that someone might see you. I would rather have fixed a bug somewhere.
I can see that it is a slipperly slope to be on, and I can easily project it to a scenario that I would be offended by, but I just can't convince myself that knowing the reletive distribution of different OpenGL implementations is violating people's rights.
The system was set up to allow us to notify people with a one-line message when their versions are out of date. I imagine some people are offended even by that, but I consider that a positive service to the community.
Including the renderer string was an afterthought to get some good unbiased data to help make future decisions on. Every once in a while we tally up the numbers, then dump all the logs. That's it.
John Carmack
This has been discussed before, and has been going on with the previous tests.
The message of the day server was intended as a half-assed auto update feature that could be cross platform.
We send a normal message most of the time, but if the version is out of date, we can send a message with telling you where to get the update.
I didn't want to deal with binary auto-updates on three platforms, and I worry a bit about security issues with that in any case.
You can disable it by setting "cl_motd 0" when the game starts up if you really don't want to send anything or see our message.
We added the result of glGetString( GL_RENDER ) to get some much needed information about the distribution of video cards and drivers.
We can see how many people aren't following directions and running glsetup. This is a big support issue.
We can see how many people are running minidrivers, which are going to make our lives a mess in the future.
We can see how many mac (steady 5%) and linux (5%at initial release, tailed off to 2%, probably due to dual booting) people are playing.
Getting this information has been usefull. We can compare the numbers of people playing with a given card with the amount of support emails we field, so we know which vendors (3DFX) we need to give more crap about their driver quality.
John Carmack
For back end rendering, they have a room full of MP sparc boxes. To my "SPARC? Why use the slowest of risc processors?" question, they replied that it isn't the speed of the individual processors that was important to them, but the speed PER CUBIC FOOT OF SPACE. Sun made quad pizza boxes, so it was computationally dense.
For modeling and development, they use a lot of SGI octanes. They also use linux + mesa for some internal tools.
John Carmack
I am typing this on a loaded SGI 320.
When it debuted, it was a very good all around performar, and it had the highest fill rate of any intel based system.
Now, an Nvidia GeForce is just plain superior in almost every aspect. Higher fill rate, even in high res, 32 bit, trilinear modes. Faster, more capable geometry acceleration.
Any remaining areas of SGI superiority are probably due to driver optimization rather than hardware architecture. Nvidia hasn't had much call to optimize stippled lines, for instance.
The super-memory-system wasn't all it was touted to be. It worked well for sharing the load between the graphics and the cpu, but the cpu didn't actually see any better bandwidth than on a standard intel chipset. The cpu write bandwidth was actually about 10% LOWER than a consumer machine.
I have used a lot of intergraph and sgi machines, and the bottom line is that the consumer hardware has just outpaced the workstation hardware because they were on different growth curves. The workstations are better than they have ever been, but the consumer systems are just orders of magnitude better than they used to be.
I think SGI shot too low with the VW's graphics, somewhat out of fear of canibalizing their other workstations, and somewhat out of underestimating the consumer competition. Being quite a bit late didn't help, either.
John Carmack
Hey, >I didn't say "virtual reality"... I tend to agree with your assessment. VR is a term loaded with high-enthusiasm / low-results connotations. We have worked with a few VR companies in the past, and I have always found them to not have finishing ability. So much of the VR world (and much other academic style research) is high concept, but sketchy on the details. Most VR experiences are heavy on the "You are in a virtual world!!!!", but don't spend too much time on exactly what you are supposed to be doing there. Can you poke and prod to find interesting things? What happens when someone pushes you? Can you dodge something effectively? Are the controls linear, or integrated over time? etc. I think that one of my strengths is a blend of idealism and pragmatism that has resulted in good results over the years. In any case, of the half dozen things I listed, I am clearly not going to be able to do all of them, so it may be a moot point... John Carmack
There have been a few things that didn't have prior art that probably could have been gotten past a patent examiner -- constant Z perspective texturing in DOOM, surface caching in Q1, and the overbright gamma table stuff to trade range for precision in Q3, for example. The patent issue came up at Id a few times until a made it perfectly clear that if the company pursued any patents, I would leave. John Carmack