Re:"cell" architecture is all about local memory
on
IBM Releases Cell SDK
·
· Score: 1
That 256k has to hold the program that the SPE is running, as well as all the data. For fast DMA, though, your data is probably double-buffered so divide your data space in half, and hey, you might want a little space for stack / other dynamic memory usage.
Suppose your program is 48k, you use 32k of memory dynamically, that leaves 172k for data, which is double-buffered, which means the program can only process 86k of data at a time.
DirectShow has been deprecated because damn, it is a crappy API. It'll probably be around for a little but but undeveloped and unsupported (see what happened to Direct3D Retained Mode).
Other more reasonable parts of DirectX (DirectInput, DirectSound, etc) will still be there...
Right now, a lot of high-end games using Direct3D spend a lot of time going through kernel traps because every time you call DrawPrimitive, well, there go a boatload of cycles. This is becoming a real bottleneck to how much can be rendered.
I don't know if that's the main motivator to them moving things to user-mode or not, but it seems so. I can imagine the drivers being built in a two-stage structure where the bulk of the driver is in user mode and a small back-end runs in kernel mode.
I am not sure what world Tranzig here lives in... Nvidia and ATI are totally prepped for the move to WGF.
Certainly, I know all about forward declarations and being careful about who includes what. My own codebase actually does a full recompile in 15 seconds on Visual Studio 6. But it requires a *lot* of discipline to maintain that, a lot more than it would in e.g. a business programming project, because of the extreme interdependency.
And the article was about the state of the industry, not what me or a couple of other people have managed to do re: build times. I've been on projects that had been going for a year, and stepped in to try and fix their build times, and it's *hard* at that point because everything is so calcified. It can still be done but it requires huge engineering effort.
My experience is that yes, there are some development houses that can't be bothered with good design, or whose idea of good design is totally out of whack. But then there are places that are not like that. And they still have problems with their build. It's just because game development involves such a crushingly huge amount of work that they really feel pressure to get visible results done. (Perhaps rightly so, because without those, how do you know you are proceeding toward the goal?) The importance of nitpicking your header files gets downplayed, if they ever believed it in the first place. And then you're rolling down that slope.
But it's a good deal that you saw optimizing the build as an important task. We need a lot more teams like that, that's for sure. Certainly there are schools of thought that that isn't important at all. (Ever try using the Unreal engine?)
And yeah, I won't argue that the PS2 tools suck. That doesn't mean, though, that less sucky tools aren't bad.
What exactly is there about game project "asset" control that is different from any other kind of software development asset control?
As one of the guys who replied to this thread mentioned, it's that we have huge truckloads of assets. In games currently in development, we tend to have a few gigabytes' worth of assets in the tree, and that is only getting bigger. (That's the size of one snapshot, not the size of the repository, of course). A lot of this is big binary files for 3D meshes, texture map, sound effects, etc etc.
Now imagine that you have 60 people pounding on this multi-gigabyte asset tree all day, every day. Problems arise. I'm not trying to say that they're insurmountable problems, because they're not... but, none of the asset control products have really risen to the challenge. (Though there are a few lately that are a lot better than what we've had in the past).
That seems insane. You have to jump all those hoops before committing a simple code change on a single file?
Yes. If you do not do this, the probability is high that you will break the build for everyone else in the company. That's a lot of lost time, both tangibly (number of minutes lost * number of workers) and intangibly (loss of momentum, loss of faith in tools working and just generally being able to get things done when you need to).
Full recompiles? In god's name, why? If you've changed a handful of files, all you should need to do is verify that those files compile+link+sort-of-work (before checking in to source control), surely...?
The point is that you often need to change a header file that is included widely, hence the full recompiles (or near-full recompiles). And you *need* to do this, because when you become afraid to change the global structure of your code, the code rots. Full recompiles are more necessary in games than other projects, because games have a higher connectedness factor -- there are a lot more dependencies than you have in typical business programming.
That said, full recompiles are definitely not a good thing and we generally try to avoid them. Most of the time we do get by with incremental builds. But if you only do full recompiles 10% of the time, but each of those compiles takes 30 minutes, that time still dominates.
If the tests (or the compile) fail, the job of the build master (usually by default the lead programmer) is to find out which developer(s) are responsible and get them to work out the problems.
The programmer's responsibilities are essentially just to verify that their code changes compile (and, preferably, do the right thing) before committing. But getting every programmer to test on all platforms, in both debug and release form, before committing any changes - that's just insane. So insane, in fact, that I'm really hoping I've completely misunderstood you and you don't really work like that... because it'd just lead to enormous time gaps between commits and massive pain when people have to routinely deal with significant code merging issues.
Please tell me I've misunderstood - or you were just exaggerating a bit for effect;-).
Catching build errors sometime during the next day is way too late... in the meantime you've screwed everyone who tried to sync during the checkin.
We do tend to have nightly-automated-compile-and-test processes like this, but they are more about producing definitive working builds for people like the artists and level designers to use, and also to act as a second-tier sanity check. But it doesn't do the whole job. If you check in something that breaks the build, you WILL have a lot of people annoyed at you, probably within 15 minutes of your checkin.
Yeah. This is a problem, but it's actually one that the industry is aware of.
Of course some of this is due to publishers just being imaginative and wanting to pump out the same old dreck as some other game that did pretty well. But really a lot of it has to do with games being hard to make. Often games have to cut tons of features/levels/testing in order to make it out the door only a year late. So usually the game you buy that is mostly fluff, that you're disappointed with, is not much like the game that the developers originally set out to create.
As we become more comfortable with basic technology (3D graphics and physics and stuff) it will probably become easier to get the basics done. At some point a lot of the risk will be mitigated, and you'll start seeing more creativity because we start developing with a higher baseline. Hopefully within the next few years!
I have done both business programming ("enterprise middleware development", etc etc) and game programming. And yes, there are commonalities between the two, but all I can say is, games are just a lot harder. Maybe you just have to have tried them both to really understand.
What I was trying to get at was not just ballooning complexity of application requirements, but also the inherent superconnectedness of subsystems in games. In a game, every subsystem wants to talk to every other one, and you have to work REALLY hard to prevent this from happening, and often you just can't. This changes a lot of things.
Still I do agree that there are some commonalities with other software development (in fact I say as much in the article; I divide it into two parts, one that's not so specific to games, and one that is...)
I think in the game industry the situation is actually the opposite of this. Most game companies, despite having been in business for years, still underestimate the difficulty of the task (because it keeps getting harder every year) and hire people who are underqualified (often because they just can't get anyone else).
Like, all the time I see job listings like "Lead programmer for massively multiplayer game, must have 3 years of C++ experience, must know Direct3D , Visual C++" and I just think "Wow, these guys don't have a chance -- if their stock was public I'd short it."
As the author of the article, I'll just say that I wasn't trying to impress anyone with how difficult my life is. I like games, that's why I work on them.
They are, however, one of the most challenging kinds of software engineering there is. And that's how I'm hoping the article is taken -- as a call to challenge for people who may be interested.
The problem is that games are just too big and cutting-edge for this kind of design approach to work. You can only do this for relatively simple problems that you completely understand. Games are the opposite of that. They have to be designed incrementally -- if you just sat down and tried to make a bunch of headers, without building the implementations, you would eventually find that your interfaces were completely wrong.
Interface classes can help a little, but only a little. The problem isn't so much having private data in a header file (though that is a problem) so much as the sheer interconnectedness of the dependencies in a project like this. That's the point of those diagrams on the first page of the article. Look at the one for a Massively Multiplayer Game and then think about what the header structure for that is like (considering that each box is not a file, but a cluster of files).
I don't dispute this... it is definitely easier to write a simple game than it used to be. That was sort of beyond the scope of the article, though -- a different subject.
There are independent developers right now who are targeting niche markets with simple games and doing okay at it (and some of them are doing great). But the game was focusing on the "high end" of game development -- if you go into a game store and buy something off the front shelf, what was required to make it today, versus 10 or 20 years ago.
As the author of this article I will throw a couple of cents in.
Content development (game design, story creation, etc) is harder than it should be. But that's mainly because the tools aren't very good. Why aren't the tools very good? Because it was all your programmers could to to scrape the basic game together; they hardly had any energy left to do more than cruddy tools.
I would have gone into this in detail, but the article was already over its length budget.
That said, even though content development is hard, it's still easier than programming. You have to have done them both in order to understand. Programming is always like building a fragile house of cards; content development isn't. That's the difference.
The hardest thing about content development in fact isn't making the content, but managing the content creation process; it's difficult for the producers and art leads to hold it all together.
Oh, by "we probably won't be doing all so much start from scratch", I don't mean to imply that I am in any way associated with Id Software. I'm not. By "we" I meant programmers of large game systems, in general.
Re:Why I Can't make a DOOM 3 clone
on
Razor Blade Games?
·
· Score: 2, Interesting
This is not as true as you think. There has been some code sharing in the Doom / Quake series, usually it goes in pairs (Doom -> Doom 2, Quake -> Quake 2, etc).
Quake was basically made from scratch, Doom code was not used. Quake 3 was basically made from scratch, Quake 2 code was not used (much). Doom 3 was basically made from scratch.
What you say is only now starting to be true (from here on, we probably won't be doing all so much "start from scratch" programming work). But it's not the reason why you can't make something of the quality level of Quake 3 or Doom 3.
The thing that *is* getting re-used, built and improved upon, is Carmack's (and other peoples') understanding of and experience with 3D. If you can't just sit down and write Quake 3, you just don't have enough programming experience yet. Quake 3 is actually not hard to write these days. The challenge in making a game that looks at least as good as Q3 is in getting people to do good art.
I'll agree that the Doom argument seems clearly false, since it ignores the priors -- we have evidence about where in human history we live, therefore we are not a random sample. Any time you make a probability argument, you need to fully account for the priors.
In the Simulation Argument, though, there are no priors, or at least no obvious ones (aside from "at least one universe exists"). So if it is false, it seems to me that it would be false in a different way. "It sounds ridiculous" is not enough reason to discount a logical argument.
No, the recent development is the probability part of the argument. There's a big difference between saying "wow man, the whole universe could just be in a compuuuter somewhere", and saying, "if you look at the probable branching factor of simulation creation, it's pretty clear that simulated universes vastly outnumber 'real' universes, unless we have a fundamental misunderstanding."
You don't need to upload the geometry again. You do, however, need to rasterize it again, and that's not free. But if your pixel shader is expensive compared to your vertex shader, multipass rendering is not necessarily super costly either.
The biggest issue is the programming work involved. Breaking things up into multiple passes is a pain, especially if you're trying to do something sophisticated. And if you want max performance, you'd have a broken-up version for the 9700, and a non-broken-up version for the NV30.
Also, making your code cope with the idea of multipass rendering is sometimes nontrivial. It will be a nice simplification in upcoming hardware if we can just make the assumption that things get rendered in a single pass. As 3D engines go, we need as much simplification as we can get.
This patent basically seems to claim ownership over all variants of least-squares fitting for image analysis (though admittedly I haven't read it too carefully, I might be missing something).
Wavelet compression, when you boil down the math, is solving a least-squares problem. In fact, least-squares problems like this are such an incredibly common tool in computer science that it's embarrassing the patent was granted.
Actually... the guy with the red shirt is me. And I'm not gay.
It's actually because I was working on computer vision apps, and needed to wear brightly colored clothing since that makes it easier for cheesy webcams to construct a good image.
You have to make sure your RGB values are in light-linear space (gamma = 1). The default, when you read them from a bitmap or a screen or something, is that they are not (gamma = 2.4 or thereabouts). So if you read colors out of a bitmap and put them through your 3x3 matrix, you won't actually get the right CIEXYZ colors. So then the final step (XYZ to LAB) is pretty meaningless.
So before you convert a pixel to XYZ, do this:
[1] Make sure each component is in the range from 0 to 1 (so if it's from an 8-bit-per-channel image source, divide each channel by 255).
[2] Raise each channel to the power GAMMA, where you define GAMMA to be something like 2.4.
[3] Now push the colors through the 3x3 matrix you came up with (which of course requires you to know what your illuminant and RGB phosphors are like... I use illuminant D65 and the standard phosphor responses and get good results).
I have source code I can send you. jon@bolt-action.com. Also, search on the web for Poynton's "FAQ about Color and Gamma".
Slashdot Mirror
That 256k has to hold the program that the SPE is running, as well as all the data. For fast DMA, though, your data is probably double-buffered so divide your data space in half, and hey, you might want a little space for stack / other dynamic memory usage.
Suppose your program is 48k, you use 32k of memory dynamically, that leaves 172k for data, which is double-buffered, which means the program can only process 86k of data at a time.
But it sure can do it fast.
DirectShow has been deprecated because damn, it is a crappy API. It'll probably be around for a little but but undeveloped and unsupported (see what happened to Direct3D Retained Mode).
Other more reasonable parts of DirectX (DirectInput, DirectSound, etc) will still be there...
Right now, a lot of high-end games using Direct3D spend a lot of time going through kernel traps because every time you call DrawPrimitive, well, there go a boatload of cycles. This is becoming a real bottleneck to how much can be rendered.
... Nvidia and ATI are totally prepped for the move to WGF.
I don't know if that's the main motivator to them moving things to user-mode or not, but it seems so. I can imagine the drivers being built in a two-stage structure where the bulk of the driver is in user mode and a small back-end runs in kernel mode.
I am not sure what world Tranzig here lives in
Certainly, I know all about forward declarations and being careful about who includes what. My own codebase actually does a full recompile in 15 seconds on Visual Studio 6. But it requires a *lot* of discipline to maintain that, a lot more than it would in e.g. a business programming project, because of the extreme interdependency.
And the article was about the state of the industry, not what me or a couple of other people have managed to do re: build times. I've been on projects that had been going for a year, and stepped in to try and fix their build times, and it's *hard* at that point because everything is so calcified. It can still be done but it requires huge engineering effort.
My experience is that yes, there are some development houses that can't be bothered with good design, or whose idea of good design is totally out of whack. But then there are places that are not like that. And they still have problems with their build. It's just because game development involves such a crushingly huge amount of work that they really feel pressure to get visible results done. (Perhaps rightly so, because without those, how do you know you are proceeding toward the goal?) The importance of nitpicking your header files gets downplayed, if they ever believed it in the first place. And then you're rolling down that slope.
But it's a good deal that you saw optimizing the build as an important task. We need a lot more teams like that, that's for sure. Certainly there are schools of thought that that isn't important at all. (Ever try using the Unreal engine?)
And yeah, I won't argue that the PS2 tools suck. That doesn't mean, though, that less sucky tools aren't bad.
As one of the guys who replied to this thread mentioned, it's that we have huge truckloads of assets. In games currently in development, we tend to have a few gigabytes' worth of assets in the tree, and that is only getting bigger. (That's the size of one snapshot, not the size of the repository, of course). A lot of this is big binary files for 3D meshes, texture map, sound effects, etc etc.
Now imagine that you have 60 people pounding on this multi-gigabyte asset tree all day, every day. Problems arise. I'm not trying to say that they're insurmountable problems, because they're not... but, none of the asset control products have really risen to the challenge. (Though there are a few lately that are a lot better than what we've had in the past).
Yes. If you do not do this, the probability is high that you will break the build for everyone else in the company. That's a lot of lost time, both tangibly (number of minutes lost * number of workers) and intangibly (loss of momentum, loss of faith in tools working and just generally being able to get things done when you need to).
The point is that you often need to change a header file that is included widely, hence the full recompiles (or near-full recompiles). And you *need* to do this, because when you become afraid to change the global structure of your code, the code rots. Full recompiles are more necessary in games than other projects, because games have a higher connectedness factor -- there are a lot more dependencies than you have in typical business programming.
That said, full recompiles are definitely not a good thing and we generally try to avoid them. Most of the time we do get by with incremental builds. But if you only do full recompiles 10% of the time, but each of those compiles takes 30 minutes, that time still dominates.
Catching build errors sometime during the next day is way too late... in the meantime you've screwed everyone who tried to sync during the checkin.
We do tend to have nightly-automated-compile-and-test processes like this, but they are more about producing definitive working builds for people like the artists and level designers to use, and also to act as a second-tier sanity check. But it doesn't do the whole job. If you check in something that breaks the build, you WILL have a lot of people annoyed at you, probably within 15 minutes of your checkin.
So no, it wasn't an exaggeration!
Yeah. This is a problem, but it's actually one that the industry is aware of.
Of course some of this is due to publishers just being imaginative and wanting to pump out the same old dreck as some other game that did pretty well. But really a lot of it has to do with games being hard to make. Often games have to cut tons of features/levels/testing in order to make it out the door only a year late. So usually the game you buy that is mostly fluff, that you're disappointed with, is not much like the game that the developers originally set out to create.
As we become more comfortable with basic technology (3D graphics and physics and stuff) it will probably become easier to get the basics done. At some point a lot of the risk will be mitigated, and you'll start seeing more creativity because we start developing with a higher baseline. Hopefully within the next few years!
(I wrote the article).
I have done both business programming ("enterprise middleware development", etc etc) and game programming. And yes, there are commonalities between the two, but all I can say is, games are just a lot harder. Maybe you just have to have tried them both to really understand.
What I was trying to get at was not just ballooning complexity of application requirements, but also the inherent superconnectedness of subsystems in games. In a game, every subsystem wants to talk to every other one, and you have to work REALLY hard to prevent this from happening, and often you just can't. This changes a lot of things.
Still I do agree that there are some commonalities with other software development (in fact I say as much in the article; I divide it into two parts, one that's not so specific to games, and one that is...)
(I wrote the article).
I think in the game industry the situation is actually the opposite of this. Most game companies, despite having been in business for years, still underestimate the difficulty of the task (because it keeps getting harder every year) and hire people who are underqualified (often because they just can't get anyone else).
Like, all the time I see job listings like "Lead programmer for massively multiplayer game, must have 3 years of C++ experience, must know Direct3D , Visual C++" and I just think "Wow, these guys don't have a chance -- if their stock was public I'd short it."
As the author of the article, I'll just say that I wasn't trying to impress anyone with how difficult my life is. I like games, that's why I work on them.
They are, however, one of the most challenging kinds of software engineering there is. And that's how I'm hoping the article is taken -- as a call to challenge for people who may be interested.
And there is no spoon.
(I wrote the article).
The problem is that games are just too big and cutting-edge for this kind of design approach to work. You can only do this for relatively simple problems that you completely understand. Games are the opposite of that. They have to be designed incrementally -- if you just sat down and tried to make a bunch of headers, without building the implementations, you would eventually find that your interfaces were completely wrong.
Interface classes can help a little, but only a little. The problem isn't so much having private data in a header file (though that is a problem) so much as the sheer interconnectedness of the dependencies in a project like this. That's the point of those diagrams on the first page of the article. Look at the one for a Massively Multiplayer Game and then think about what the header structure for that is like (considering that each box is not a file, but a cluster of files).
(I wrote the article).
I don't dispute this... it is definitely easier to write a simple game than it used to be. That was sort of beyond the scope of the article, though -- a different subject.
There are independent developers right now who are targeting niche markets with simple games and doing okay at it (and some of them are doing great). But the game was focusing on the "high end" of game development -- if you go into a game store and buy something off the front shelf, what was required to make it today, versus 10 or 20 years ago.
As the author of this article I will throw a couple of cents in.
Content development (game design, story creation, etc) is harder than it should be. But that's mainly because the tools aren't very good. Why aren't the tools very good? Because it was all your programmers could to to scrape the basic game together; they hardly had any energy left to do more than cruddy tools.
I would have gone into this in detail, but the article was already over its length budget.
That said, even though content development is hard, it's still easier than programming. You have to have done them both in order to understand. Programming is always like building a fragile house of cards; content development isn't. That's the difference.
The hardest thing about content development in fact isn't making the content, but managing the content creation process; it's difficult for the producers and art leads to hold it all together.
Mod parent down.
Not only was the game leaked a hell of a long time ago, the leaked copy was A DEBUG BUILD.
Either the poster is confused about the difference between Doom 3 and Half-Life 2, or this is a troll.
Oh, by "we probably won't be doing all so much start from scratch", I don't mean to imply that I am in any way associated with Id Software. I'm not. By "we" I meant programmers of large game systems, in general.
This is not as true as you think. There has been some code sharing in the Doom / Quake series, usually it goes in pairs (Doom -> Doom 2, Quake -> Quake 2, etc).
Quake was basically made from scratch, Doom code was not used. Quake 3 was basically made from scratch, Quake 2 code was not used (much). Doom 3 was basically made from scratch.
What you say is only now starting to be true (from here on, we probably won't be doing all so much "start from scratch" programming work). But it's not the reason why you can't make something of the quality level of Quake 3 or Doom 3.
The thing that *is* getting re-used, built and improved upon, is Carmack's (and other peoples') understanding of and experience with 3D. If you can't just sit down and write Quake 3, you just don't have enough programming experience yet. Quake 3 is actually not hard to write these days. The challenge in making a game that looks at least as good as Q3 is in getting people to do good art.
"Several thousand dollars"? You're on crack. Try several HUNDRED thousand dollars. No, I am not kidding.
It's August 27th... why isn't Slashdot showing the protest page, huh?
I'll agree that the Doom argument seems clearly false, since it ignores the priors -- we have evidence about where in human history we live, therefore we are not a random sample. Any time you make a probability argument, you need to fully account for the priors.
In the Simulation Argument, though, there are no priors, or at least no obvious ones (aside from "at least one universe exists"). So if it is false, it seems to me that it would be false in a different way. "It sounds ridiculous" is not enough reason to discount a logical argument.
No, the recent development is the probability part of the argument. There's a big difference between saying "wow man, the whole universe could just be in a compuuuter somewhere", and saying, "if you look at the probable branching factor of simulation creation, it's pretty clear that simulated universes vastly outnumber 'real' universes, unless we have a fundamental misunderstanding."
There are some exceptions to this. See this site.
You don't need to upload the geometry again. You do, however, need to rasterize it again, and that's not free. But if your pixel shader is expensive compared to your vertex shader, multipass rendering is not necessarily super costly either.
The biggest issue is the programming work involved. Breaking things up into multiple passes is a pain, especially if you're trying to do something sophisticated. And if you want max performance, you'd have a broken-up version for the 9700, and a non-broken-up version for the NV30.
Also, making your code cope with the idea of multipass rendering is sometimes nontrivial. It will be a nice simplification in upcoming hardware if we can just make the assumption that things get rendered in a single pass. As 3D engines go, we need as much simplification as we can get.
-J.
This is absolutely not true. At 120fps, you get half the latency that you do at 60fps, and that is very important (even in a single player game).
- A game developer who actually knows what he's talking about.
This patent basically seems to claim ownership over all variants of least-squares fitting for image analysis (though admittedly I haven't read it too carefully, I might be missing something).
Wavelet compression, when you boil down the math, is solving a least-squares problem. In fact, least-squares problems like this are such an incredibly common tool in computer science that it's embarrassing the patent was granted.
-N.
Actually... the guy with the red shirt is me. And I'm not gay.
It's actually because I was working on computer vision apps, and needed to wear brightly colored clothing since that makes it easier for cheesy webcams to construct a good image.
Usually... I wear a lot of black.
-J.
You have to make sure your RGB values are in light-linear space (gamma = 1). The default, when you read them from a bitmap or a screen or something, is that they are not (gamma = 2.4 or thereabouts). So if you read colors out of a bitmap and put them through your 3x3 matrix, you won't actually get the right CIEXYZ colors. So then the final step (XYZ to LAB) is pretty meaningless.
So before you convert a pixel to XYZ, do this:
[1] Make sure each component is in the range from 0 to 1 (so if it's from an 8-bit-per-channel image source, divide each channel by 255).
[2] Raise each channel to the power GAMMA, where you define GAMMA to be something like 2.4.
[3] Now push the colors through the 3x3 matrix you came up with (which of course requires you to know what your illuminant and RGB phosphors are like... I use illuminant D65 and the standard phosphor responses and get good results).
I have source code I can send you. jon@bolt-action.com. Also, search on the web for Poynton's "FAQ about Color and Gamma".
-N.