Slashdot Mirror
Why Xbox One Backward Compatibility Took So Long (ign.com)
A new report from IGN this morning explains why it took so long for backwards compatibility to be supported on the Xbox One. Microsoft veteran Kevin La Chapelle says the answer to the question can be found in 2015 -- the year that Phil Spencer announced backwards compatibility at Microsoft's Xbox E3 media briefing. From the report: The fan-first feature has evolved from an experiment conducted by two separate Microsoft Research teams into a service planned for Xbox One's launch -- complete with hardware hooks baked into the Durango silicon -- until the well-publicized changes to the Xbox One policies (namely, stripping out the always-online requirement for the console) forced it to be pushed to the back burner. It's obviously back for good now, and expanding into original Xbox compatibility of select titles on Xbox One (the first batch of which we announced today). Even the Xbox One X is getting involved, with a handful of Xbox 360 games getting Scorpio-powered enhancements like 10-bit color depth, anisotropic filtering, and up to 9x additional pixel counts displayed on screen. [...]
It was 2007. One of [the research] teams was working on PowerPC CPU emulation -- getting 32-bit code, which the 360 uses, to run on the 64-bit architecture that the third-generation Xbox would be using. The other team, out of Beijing, started writing a virtual GPU emulator based on the Xbox 360 GPU architecture. "These were like peanut butter and chocolate," Microsoft VP of Xbox software engineering Kareem Choudhry recalled. "[So we thought,] 'Why don't we put them both together?'" Choudhry did just that, and so the first steps to Xbox One backwards compatibility were taken, long before the console had a name or anything remotely resembling final specifications. As Durango crystallized, so too did plans for Xbox 360 compatibility on the new machine. "This was primarily a software exercise, but we enabled that by thinking ahead with hardware," Gammill explained. "We had to bake some of the backwards compatibility support into the [Xbox One] silicon." This was done back in 2011. Preliminary tests showed that support for key Xbox middleware XMA audio and texture formats was extremely taxing to do in software alone, with the former, Gammill noted, taking up two to three of the Xbox One's six CPU cores. But a SOC (system on chip) -- basically an Xbox 360 chip inside every Xbox One, similar to how Sony put PS2 hardware inside the launch-era PS3s -- would've not only been expensive, but it would've put a ceiling on what the compatibility team could do. "If we'd have gone with the 360 SOC, we likely would've landed at just parity," he said. "The goal was never just parity." So they built the XMA and texture formats into the Xbox One chipset...
It was 2007. One of [the research] teams was working on PowerPC CPU emulation -- getting 32-bit code, which the 360 uses, to run on the 64-bit architecture that the third-generation Xbox would be using. The other team, out of Beijing, started writing a virtual GPU emulator based on the Xbox 360 GPU architecture. "These were like peanut butter and chocolate," Microsoft VP of Xbox software engineering Kareem Choudhry recalled. "[So we thought,] 'Why don't we put them both together?'" Choudhry did just that, and so the first steps to Xbox One backwards compatibility were taken, long before the console had a name or anything remotely resembling final specifications. As Durango crystallized, so too did plans for Xbox 360 compatibility on the new machine. "This was primarily a software exercise, but we enabled that by thinking ahead with hardware," Gammill explained. "We had to bake some of the backwards compatibility support into the [Xbox One] silicon." This was done back in 2011. Preliminary tests showed that support for key Xbox middleware XMA audio and texture formats was extremely taxing to do in software alone, with the former, Gammill noted, taking up two to three of the Xbox One's six CPU cores. But a SOC (system on chip) -- basically an Xbox 360 chip inside every Xbox One, similar to how Sony put PS2 hardware inside the launch-era PS3s -- would've not only been expensive, but it would've put a ceiling on what the compatibility team could do. "If we'd have gone with the 360 SOC, we likely would've landed at just parity," he said. "The goal was never just parity." So they built the XMA and texture formats into the Xbox One chipset...
stupid outsourced chimps
It was developed by Microsoft. That is why.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
They took a simple idea and made it complicated.
Cool. But some actual technical details on how they're emulating the PowerPC cores on the x86 cores, or emulating the nVidia GPU on the AMD GPU would be nice.
Now that there's a BC emulator, I guess I won't be finding unwanted original Xbox games for pennies anymore. At least I amassed a good pile of games from back when I was ripping them into a chipped and juke-boxed Xbox. (Why pirate when the original disc is cheap?) I still have my two chipped Xboxes somewhere.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
As software goes, console games are architecturally horrible. This is mainly because of the legacy of 8- and 16-bit consoles where it was actually significant whether a program took a syscall (generally implemented as data-dependent branches), so optimizations like inlining are looked upon favourably even as they fix program to platform down to the hardware register. Those optimizations have been worthless since the race to half a gigahertz ended and RAM latency began to really get out of control, because since then syscall stubs (etc.) have been cacheable just like any hot-path thing, so doing a massive amount of them in a loop turns from an obstacle to effective utilization of hardware.
The lesson here is that one can always trust Microsoft to code like an obsessive twentysomething.
You have been watching too many Turboencabulator videos.
Does it run Mechassault yet? Because _that_ was the game of games. Near perfect mech control, near-perfect online gaming.
If not, going back to sleep.
More pixels don't mean shit if you don't have gameplay handled.
I've worked on original Xbox and Xbox 360 games, and I can't figure out what you're talking about. Microsoft's consoles are nothing like those in the 8 or 16-bit days, when game code interacted directly with the console hardware, programming "to the metal", as it were.
Rather, modern console development is actually pretty similar to programming on Windows (which shouldn't be surprising) where you write your code in C++, call DirectX and system API calls, and mostly rely on the compiler for low-level optimizations, while code architecture plays more of a part in high-level optimization. There are low-level intrinsics which you can use, but these are typically used quite sparingly in very performance-sensitive code, such as in your vector or matrix class operations, or for lockless thread-safe containers, etc.
Irony: Agile development has too much intertia to be abandoned now.
way to show complete and utter ignorance. The Xbox has no legacy in 8 and 16 bit architecture AT ALL! The reasons for the complexity are the completely different hardware architectures right down to the different architecture CPU's and GPU's PowerPC vs X86. Microsoft from their first console used a Cutdown version of DirectX for coders so you did not have that mess of calls even so the GPU and CPU architecture changes were massive and hence the complexity.
how so? they seem to have done what most others have failed to do, providing backwards compatibility across completely different hardware architectures (without including that hardware on the board) is no small achievement and is by no means a simple to achieve.
>Microsoft's consoles are nothing like those in the 8 or 16-bit days,
Yet you insist on reading me like I had said so!
To clarify just for you, I'm pointing out that inlining (among others) is a petty optimization such as those that were relevant twenty years ago, which Microsoft holds on to because of reasons undiscussed and effects unmeasured. This makes programs for their consoles _fucking awful_.
Do you know where the XBox got it's name? Microsoft wanted to make a box that runs DirectX and sits on your TV. This "DirectX Box" got shortened to "XBox", which stuck as the final product name.
The very core idea driving the project was that the system would run Windows & DirectX. You're coding against high level APIs on the Xbox, and can't opt out of it. It was a key design goal to have people coding against Microsoft's APIs instead of coding to the hardware.
As for the 8 and 16 bit consoles, syscalls were irrelevant. There was no operating system on those systems. You might have had some *really* simple, common features coded into the system rom, but otherwise *all* the code was contained within the individual game. These systems only had a bare minimal amount of storage in them - just enough to store any code needed to boot the system and hand off control to the game. Including storage in the system was too expensive to have any significant amount of code built in.
except that ISN'T the case with Microsoft consoles at all, they actually actively discourage inlining optimisations. The way Xbox One has been architected from a software point of view is actually pretty slick, their architecture has allowed them to easily virtualise consoles as they actively discourage inline optimisations that talk to hardware.
Syscall on an 8-bit system? There was no OS, no standard interrupts like in DOS, and no ROM firmware like old BIOS.
The parent poster seems to not understand old or new consoles.
I don't get what GP commenter is getting on about. The original XBox isn't musty old 8 or 16 bit hardware. It's got a Pentium 3 processor in it. Hardly the kind of processor that you do 'inline optimization' type coding on. I suspect there are lots of driver programmers who program Pentium 3 processors with ASM optimizations. I doubt very much that there were/are many console game coders touching ASM on a Pentium 3 processor.
Then they failed. Fucking utterly failed.
Anyone who's written DirectX shader code for the Xbox knows about the limitations of the hardware and the optimizations you had to make for it, let alone the custom texture compression scheme that was used on no other platform (DXT != S3 despite Shawn Hargreaves saying otherwise).
Then you had XACT, the "Cross Platform" Audio Creation Tool, creating collections of .xgs (Xbox Global Settings), .xsb (Xbox Sound Bank) and .xwb (Xbox Wave Bank) files with custom audio compression schemes just for the Xbox audio hardware. They gave excellent results, but it was in no way portable to Windows nor any other gaming platform.
We at mickeysoft thought that the xbox one would beat out the ps4 by its library alone but since it isn't we are working hard on backwards compatibility again so potential and existing xbox one and x owners have a bigger reason to buy/play a xbox one and x.
(1) They did include at least some backwards compatibility in the hardware (XMA audio and the textures format) which may or may not have been necessary. (2) The whole point the GP was making was precisely that they didn't go the simple route: include sufficient hardware to do backwards compatibility. Instead, they went the mostly emulation route which almost always results in issues. Look no further than XBox backwards compatibility on the XBox 360.
So, we can commend them on their efforts to do most of it in software? Sure. We can even be impressed, even though it sounds like they're going through and making special optimizations per game because a generic emulator won't do the job. It's still not as good as the real thing. All the claims of better? I doubt it. I used to be a really big fan of emulation. And at first, it seems quite amazing. Over time, though, it's very clear that emulation has many weaknesses and even doing per game optimizing and checks is never going to be enough.
*shrug* Then again, I don't have an XBox 360 or One.
Oh do check out an actual 8-bit system. You'll find ROM entry vectors everywhere: unless you figure the syscalls are somehow long enough to fit in three bytes...
Inlining is still a valuable optimization technique for when the overhead of a function call exceeds the execution cost of the function body, but obviously this has to be weighed against cache size considerations. This is not something videogame developers are unaware of. For the past several decades, the general rule of thumb is to avoid excessive inlining, because it works directly against cache coherency.
Generally speaking, when writing C++ code, you have some control of this via how code is organized, along with the inline keyword which acts as a compiler hint. Further hints are made to the MSVC compiler as to whether to optimize for speed or size, which is essentially a tradeoff of more or less aggressive inlining, etc.
But here's the thing... I don't understand how you think this has anything to do with Microsoft consoles in particular, or how it makes programs for them "awful". Every platform pretty much works the same way, and has the same essential trade-offs. You make these outlandish claims about how MS is two decades behind the times in this regard, but offer no evidence at all. What's more, the fact that many games are successfully developed on all Microsoft consoles over the years seems to demonstrate that it's not any sort of insurmountable problem in reality.
Irony: Agile development has too much intertia to be abandoned now.
Sony also have a software emulator, although the early PS3 hardware included a complete PS2 because the software emulation wasn't ready at the time.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
As opposed to not doing it at all, like Sony with the PS4? As opposed to doing it poorly, as Nintendo did with the WiiU, or not at all, with the Switch? I'm sorry, I know hating Microsoft is Slashdot tradition, but they're the only ones doing this properly this generation. Sony's farmed it all out to a game streaming service and Nintendo just has you buying Virtual Console games over and over again or, back in the day, they bundled the Wii in the WiiU with all the shitty graphics and poor resolution that implied.
Now that this has been solved, can we get some answers for why my XBOX 360 games (Forza, etc) don't have access to the downloadable content anymore? How hard is it to maintain data for two or three console generations?
Nobody writes direct to hardware anymore, surely. It's all DirectX library calls.
All the common 8-bit systems had ROMs with defined entrypoints for various functions. You could even get CP/M on option ROMs IIRC. The ZX Spectrum +3/+2A had a DOS (+3DOS) with defined entry points for all disc operations, plus several other ROMs. You placed the one you wanted into the address space by bank switching, then jumped to the appropriate point.
I don't know why you think virtualising a console to be "easy". The problem is not emulating the instruction set but getting the timing of both the CPU and GPU correct. The GPU is especially hard since the game is going to feed it broken shaders and use GL extensions that have to be emulated with the new GPU. And mapping the firmware calls (for audio, video, controls, achievments, copy protection, storage, chat / music, resolution negotiation) to their equivalents in the new console, or stubs. I expect also that some libraries such as Unreal, Havok, PhysX etc. are so prevalent in games that emulators will have special cases for when they encounter that code.
I do not get why most consoles are not backward compatible.. what do MS, Sony, Nintendo etc think... that we all want to have 100's of consoles standing around, just to be able to play both the old and new games?
I really fail to see how and why it helps ie MS to NOT give the newest whatever XBOX the option to play games made for older XBOX consoles... Do they think they can sell the games AGAIN for that particular console in some steam-like fashion?
no thanks... why would I buy anything that will be forever locked down to a single console?
atleast with steam games, I can still play them on whatever gaming pc I decide to buy in ie 10 years from now ...
Because they could have gotten parity at launch with an SOC, and that wouldn't have precluded them from working on a feature rich software solution going forward.
Join me and ascend to God hood for you are weak! And I am mighty!
http://saveie6.com/
Not all common 8-bit systems. Nintendo's Family Computer, Entertainment System and GameBoy didn't have any syscalls, and although they were latecomers those were overall the most popular 8-bit game consoles.
This is all just well and good, but in the end who cares? If this was a brand new console gen coming out that could run all these old games, that would be one great marketing differentiator. But it's not, and at this point you have to wonder why all this effort that isn't going to net Microsoft all that many console sales? Let alone give it a "win" in this console gen? It all feels rather pointless for the tiny number of Xbox One users who want to play old games on their new console.
Then they failed. Fucking utterly failed.
Anyone who's written DirectX shader code for the Xbox knows about the limitations of the hardware and the optimizations you had to make for it, let alone the custom texture compression scheme that was used on no other platform (DXT != S3 despite Shawn Hargreaves saying otherwise).
Then you had XACT, the "Cross Platform" Audio Creation Tool, creating collections of .xgs (Xbox Global Settings), .xsb (Xbox Sound Bank) and .xwb (Xbox Wave Bank) files with custom audio compression schemes just for the Xbox audio hardware. They gave excellent results, but it was in no way portable to Windows nor any other gaming platform.
Who said ANYTHING about portability being a design goal? Everything you described used layers and layers of Microsoft APIs no? I’m not making a sideways joke about Microsoft’s overall strategy, but you could take it that way. They nailed this design goal.
Then they failed. Fucking utterly failed.
I'd say the product was successful.
Only the State obtains its revenue by coercion. - Murray Rothbard
I do not get why most consoles are not backward compatible.. what do MS, Sony, Nintendo etc think... that we all want to have 100's of consoles standing around, just to be able to play both the old and new games?
The technical reason or the economic one?
Technically, all three companies switched CPU architectures and not just from one CPU generation to the next. MS (PowerPC -> x86), Sony (Cell -> x86), Nintendo (PowerPC -> ARM). On the GPU side, there were lesser changes MS (ATI -> AMD), Sony (Cell -> AMD), Nintendo (ATI -> NVidia) with MS having the easiest transition. Games can be emulated if the newest console has enough CPU/GPU processing power. However the trend for this generation is that the advancements were more on the GPU side than the CPU side. Emulation would be more difficult because of this balance. Then after that, optimization and tweaking would have to happen.
Economically there is not much of an advantage as it takes away from new game sales. All three companies could make the transition easier by making older titles available in some way. It would easier for the console makers to pay the title companies to recompile but there's little profit for the title companies to do so.
Well, there's spam egg sausage and spam, that's not got much spam in it.
You're a fucking retard.
Hugs and kisses,
Juan Epstein
32 bit instructions should run natively on 64 bit, so the real issue here was PowerPC emulation not bits.
GCS/MU/P d- s:- a-- C++++$ UL++ P+ L++ E+ W++ N o K- w--- O M+ V- PS+++ PE Y+ PGP t+ 5- X R++ tv+ b++ DI++ D++ G+ e++ h-
Yeah this is spot-on. I don't know what he's talking about either. You maybe do a few hand optimizations but its almost all taken care of by drivers / compilers / etc.
GCS/MU/P d- s:- a-- C++++$ UL++ P+ L++ E+ W++ N o K- w--- O M+ V- PS+++ PE Y+ PGP t+ 5- X R++ tv+ b++ DI++ D++ G+ e++ h-
Compilers these days will do this automatically with the right flags.
GCS/MU/P d- s:- a-- C++++$ UL++ P+ L++ E+ W++ N o K- w--- O M+ V- PS+++ PE Y+ PGP t+ 5- X R++ tv+ b++ DI++ D++ G+ e++ h-
Cycle counting died out once the 32-bit era started, pretty much. There was no need to.
So your old 8 and 16-bit consoles and computers require cycle-accurate emulation because everyone programming them was cycle counting in the first place. The Apple II is an extreme example of this, because of the way it operates and how the display DMA basically hijacked the data bus when the CPU wasn't using it between instructions. Plus, with 8 and 16 bit comptuers and consoles, you had a set number of clocks between events - usually between a horizontal retrace and the vertical retrace.
Pre-crash consoles and computers were particularly troublesome because they didn't have a framebuffer, and on something like the ATari 2600, a good 70% of the CPU is used for screen updates (the TIA only worked on a line by line basis, so it had to be reloaded with new line data every horizontal retrace).
But around the PS2/original Xbox era, it all changed. The addition of caches and other things to the processors completely changed how consoles were programmed. No longer did you have to cycle count (because it was all single-cycle these days), but even worse, the cache added timing variability to the whole thing. Even today, getting a simulator that is cycle accurate is actually quite expensive because they include cache modelling in order to retain cycle accuracy.
Granted, the Xbox360 and PS3 got rid of some caches in an attempt to be more deterministic, but by then no one really cared - they were powerful enough that the games were basically programmed like their PC counterparts.
As such, these days emulating an older console is rather easy - high level emulation will get you far. This is especially true for the Xbox, since it's Microsoft and the DirectX API has been pretty stable. Since every game would've been programmed to the DirectX API, accurate GPU timings are no longer necessary (and chances are, there was always going to be slight variability between consoles, but you never cared since you wrote your code against the API). And given the rest of the API is basically Windows APIs, all the high level API work is done. All you need to emulate is the CPU and possibly rewrite shader programs
Long and short of it, Microsoft is probably in a good spot w.r.t. emulating old consoles
I do have an Xbox One and from my experience the 360 emulation works pretty well now. When the first emulated games came out there were some that did perform worse than the originals, but it seems like they've got the emulation nailed down now as they release updates for games tweak them for better performance. The Gears of War games ran pretty terribly at first, but a few updates later they now run very well. I used to stick with the native 360 versions for playing those games online but now I play the emulated versions on the Xbox One.
It sounds like emulation will be even better on the Xbox One X (AKA Project Scorpio). I guess we'll see once it comes out in a few weeks how things run, but it seems like the Xbox emulation team is putting in some work on getting it right.
http://www.eurogamer.net/articles/digitalfoundry-2017-project-scorpio-back-compat-five-ways-your-existing-games-will-be-better
and a SOC would have cost them a fortune and raised the retail price or forced them to take a major profit hit for a feature that while popular is not used by the majority.
Well at least PPC emulation exists today. Not sure what possessed MS to go with a PPC chip on the X360 in the first place. Seems counter to their original goals to me. But hey, at least I can run AmigaOS 4.1 under UAE now.
Did the nvidia GPU has the old nvidia bug with compressed textures that makes the sky ugly in Quake 3? I don't remember well if it was still present on geforce 4ti, or fixed by then (it's on geforce, geforce 2, geforce 2MX). That was a problem with one profile misbehaving. I don't remember it being a problem in DirectX games but perhaps those tended not to use this compression profile. Specifically it was DXT1 that had a hardware bug. So you needed to use DXT5 instead.
Perhaps you're talking about something more subtle, I'm remembering this small NV10/NV20 bug specifically.
IIRC the PS3 included a completete PS2, then only half of it (what we may call the "GPU" - it had some crazy fillrate and bandwith), then nothing at all but also dropped the PS2 compatibility.
At Nintendo we did indeed include older hardware on our handheld platform for compatibility. The old hardware made for a great co-processor. Then there is this other way to do provide backward compatibility.
really bad example, emulating applications that do not have specific timings and framerates for video and that spend most of their time idle is not a challenge and is no real accomplishment. with a gaming console you have to get CPU and GPU timings nearly perfect or the end result is unplayable, can't be faster or slower and a game uses the FULL resources of a machine. An application emulation has no such limitations.
How well does that actually work? I've noticed a lot of games (1) copy substantial parts of them to different memory areas to get better performance and/or (2) modify their own code. Beyond that, the patent is clearly bullshit. The idea of decompiling and recompiling code to a new platform is itself an old idea. Unless the implementation is actually novel. The biggest stumbling block is precisely getting code to work, especially timing specific parts or not having too bad of performance regressions. So, yea, if you have relatively sane code and the time to tweak it, I can see it working once. Try doing that little trick 3 or 4 times, and not just to systems that are orders of magnitude faster. I'm fairly certain it will fail very badly.
And I'd love to be proven wrong.
Inlining does not mean using inline assembler. It is a technique to avoid branching (loops, function calls...)
comment so nice Gesture Lock Screen