Slashdot Mirror
Why Xbox One Backward Compatibility Took So Long (ign.com)
A new report from IGN this morning explains why it took so long for backwards compatibility to be supported on the Xbox One. Microsoft veteran Kevin La Chapelle says the answer to the question can be found in 2015 -- the year that Phil Spencer announced backwards compatibility at Microsoft's Xbox E3 media briefing. From the report: The fan-first feature has evolved from an experiment conducted by two separate Microsoft Research teams into a service planned for Xbox One's launch -- complete with hardware hooks baked into the Durango silicon -- until the well-publicized changes to the Xbox One policies (namely, stripping out the always-online requirement for the console) forced it to be pushed to the back burner. It's obviously back for good now, and expanding into original Xbox compatibility of select titles on Xbox One (the first batch of which we announced today). Even the Xbox One X is getting involved, with a handful of Xbox 360 games getting Scorpio-powered enhancements like 10-bit color depth, anisotropic filtering, and up to 9x additional pixel counts displayed on screen. [...]
It was 2007. One of [the research] teams was working on PowerPC CPU emulation -- getting 32-bit code, which the 360 uses, to run on the 64-bit architecture that the third-generation Xbox would be using. The other team, out of Beijing, started writing a virtual GPU emulator based on the Xbox 360 GPU architecture. "These were like peanut butter and chocolate," Microsoft VP of Xbox software engineering Kareem Choudhry recalled. "[So we thought,] 'Why don't we put them both together?'" Choudhry did just that, and so the first steps to Xbox One backwards compatibility were taken, long before the console had a name or anything remotely resembling final specifications. As Durango crystallized, so too did plans for Xbox 360 compatibility on the new machine. "This was primarily a software exercise, but we enabled that by thinking ahead with hardware," Gammill explained. "We had to bake some of the backwards compatibility support into the [Xbox One] silicon." This was done back in 2011. Preliminary tests showed that support for key Xbox middleware XMA audio and texture formats was extremely taxing to do in software alone, with the former, Gammill noted, taking up two to three of the Xbox One's six CPU cores. But a SOC (system on chip) -- basically an Xbox 360 chip inside every Xbox One, similar to how Sony put PS2 hardware inside the launch-era PS3s -- would've not only been expensive, but it would've put a ceiling on what the compatibility team could do. "If we'd have gone with the 360 SOC, we likely would've landed at just parity," he said. "The goal was never just parity." So they built the XMA and texture formats into the Xbox One chipset...
It was 2007. One of [the research] teams was working on PowerPC CPU emulation -- getting 32-bit code, which the 360 uses, to run on the 64-bit architecture that the third-generation Xbox would be using. The other team, out of Beijing, started writing a virtual GPU emulator based on the Xbox 360 GPU architecture. "These were like peanut butter and chocolate," Microsoft VP of Xbox software engineering Kareem Choudhry recalled. "[So we thought,] 'Why don't we put them both together?'" Choudhry did just that, and so the first steps to Xbox One backwards compatibility were taken, long before the console had a name or anything remotely resembling final specifications. As Durango crystallized, so too did plans for Xbox 360 compatibility on the new machine. "This was primarily a software exercise, but we enabled that by thinking ahead with hardware," Gammill explained. "We had to bake some of the backwards compatibility support into the [Xbox One] silicon." This was done back in 2011. Preliminary tests showed that support for key Xbox middleware XMA audio and texture formats was extremely taxing to do in software alone, with the former, Gammill noted, taking up two to three of the Xbox One's six CPU cores. But a SOC (system on chip) -- basically an Xbox 360 chip inside every Xbox One, similar to how Sony put PS2 hardware inside the launch-era PS3s -- would've not only been expensive, but it would've put a ceiling on what the compatibility team could do. "If we'd have gone with the 360 SOC, we likely would've landed at just parity," he said. "The goal was never just parity." So they built the XMA and texture formats into the Xbox One chipset...
It was developed by Microsoft. That is why.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
They took a simple idea and made it complicated.
Now that there's a BC emulator, I guess I won't be finding unwanted original Xbox games for pennies anymore. At least I amassed a good pile of games from back when I was ripping them into a chipped and juke-boxed Xbox. (Why pirate when the original disc is cheap?) I still have my two chipped Xboxes somewhere.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
As software goes, console games are architecturally horrible. This is mainly because of the legacy of 8- and 16-bit consoles where it was actually significant whether a program took a syscall (generally implemented as data-dependent branches), so optimizations like inlining are looked upon favourably even as they fix program to platform down to the hardware register. Those optimizations have been worthless since the race to half a gigahertz ended and RAM latency began to really get out of control, because since then syscall stubs (etc.) have been cacheable just like any hot-path thing, so doing a massive amount of them in a loop turns from an obstacle to effective utilization of hardware.
The lesson here is that one can always trust Microsoft to code like an obsessive twentysomething.
You have been watching too many Turboencabulator videos.
In particular, how they're emulating PowerPC LL/SC on x86 without heavyhanded methods such as virtualizing all memory accesses to LL'd pages with the MMU.
I've worked on original Xbox and Xbox 360 games, and I can't figure out what you're talking about. Microsoft's consoles are nothing like those in the 8 or 16-bit days, when game code interacted directly with the console hardware, programming "to the metal", as it were.
Rather, modern console development is actually pretty similar to programming on Windows (which shouldn't be surprising) where you write your code in C++, call DirectX and system API calls, and mostly rely on the compiler for low-level optimizations, while code architecture plays more of a part in high-level optimization. There are low-level intrinsics which you can use, but these are typically used quite sparingly in very performance-sensitive code, such as in your vector or matrix class operations, or for lockless thread-safe containers, etc.
Irony: Agile development has too much intertia to be abandoned now.
If Microsoft are smart, they would have implemented a dynamic recompiler similar to what Apple did with Rosetta or what MAME and Dolphin and other emulators are doing. Writing a dynamic recompiler for the PowerPC isn't exactly a new thing.
how so? they seem to have done what most others have failed to do, providing backwards compatibility across completely different hardware architectures (without including that hardware on the board) is no small achievement and is by no means a simple to achieve.
that method is only viable when you have vastly more processing power to play with. While the Xbox One is certainly a huge step up, the CPU is actually its weakest component a dynamic recompiler would produce highly inconsistent performance.
What's that got to do with emulating LL/SC? Look it up; the issue is a "little bit" more involved than emitting the right instruction.
Do you know where the XBox got it's name? Microsoft wanted to make a box that runs DirectX and sits on your TV. This "DirectX Box" got shortened to "XBox", which stuck as the final product name.
The very core idea driving the project was that the system would run Windows & DirectX. You're coding against high level APIs on the Xbox, and can't opt out of it. It was a key design goal to have people coding against Microsoft's APIs instead of coding to the hardware.
As for the 8 and 16 bit consoles, syscalls were irrelevant. There was no operating system on those systems. You might have had some *really* simple, common features coded into the system rom, but otherwise *all* the code was contained within the individual game. These systems only had a bare minimal amount of storage in them - just enough to store any code needed to boot the system and hand off control to the game. Including storage in the system was too expensive to have any significant amount of code built in.
I bet the XBOX One CPU has more power than the CPUs in the first Intel Mac systems (where Apple used a PPC Dynarec to support PPC apps on Intel Macs)
Syscall on an 8-bit system? There was no OS, no standard interrupts like in DOS, and no ROM firmware like old BIOS.
The parent poster seems to not understand old or new consoles.
I don't get what GP commenter is getting on about. The original XBox isn't musty old 8 or 16 bit hardware. It's got a Pentium 3 processor in it. Hardly the kind of processor that you do 'inline optimization' type coding on. I suspect there are lots of driver programmers who program Pentium 3 processors with ASM optimizations. I doubt very much that there were/are many console game coders touching ASM on a Pentium 3 processor.
Then they failed. Fucking utterly failed.
Anyone who's written DirectX shader code for the Xbox knows about the limitations of the hardware and the optimizations you had to make for it, let alone the custom texture compression scheme that was used on no other platform (DXT != S3 despite Shawn Hargreaves saying otherwise).
Then you had XACT, the "Cross Platform" Audio Creation Tool, creating collections of .xgs (Xbox Global Settings), .xsb (Xbox Sound Bank) and .xwb (Xbox Wave Bank) files with custom audio compression schemes just for the Xbox audio hardware. They gave excellent results, but it was in no way portable to Windows nor any other gaming platform.
(1) They did include at least some backwards compatibility in the hardware (XMA audio and the textures format) which may or may not have been necessary. (2) The whole point the GP was making was precisely that they didn't go the simple route: include sufficient hardware to do backwards compatibility. Instead, they went the mostly emulation route which almost always results in issues. Look no further than XBox backwards compatibility on the XBox 360.
So, we can commend them on their efforts to do most of it in software? Sure. We can even be impressed, even though it sounds like they're going through and making special optimizations per game because a generic emulator won't do the job. It's still not as good as the real thing. All the claims of better? I doubt it. I used to be a really big fan of emulation. And at first, it seems quite amazing. Over time, though, it's very clear that emulation has many weaknesses and even doing per game optimizing and checks is never going to be enough.
*shrug* Then again, I don't have an XBox 360 or One.
Oh do check out an actual 8-bit system. You'll find ROM entry vectors everywhere: unless you figure the syscalls are somehow long enough to fit in three bytes...
Inlining is still a valuable optimization technique for when the overhead of a function call exceeds the execution cost of the function body, but obviously this has to be weighed against cache size considerations. This is not something videogame developers are unaware of. For the past several decades, the general rule of thumb is to avoid excessive inlining, because it works directly against cache coherency.
Generally speaking, when writing C++ code, you have some control of this via how code is organized, along with the inline keyword which acts as a compiler hint. Further hints are made to the MSVC compiler as to whether to optimize for speed or size, which is essentially a tradeoff of more or less aggressive inlining, etc.
But here's the thing... I don't understand how you think this has anything to do with Microsoft consoles in particular, or how it makes programs for them "awful". Every platform pretty much works the same way, and has the same essential trade-offs. You make these outlandish claims about how MS is two decades behind the times in this regard, but offer no evidence at all. What's more, the fact that many games are successfully developed on all Microsoft consoles over the years seems to demonstrate that it's not any sort of insurmountable problem in reality.
Irony: Agile development has too much intertia to be abandoned now.
Sony also have a software emulator, although the early PS3 hardware included a complete PS2 because the software emulation wasn't ready at the time.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
As opposed to not doing it at all, like Sony with the PS4? As opposed to doing it poorly, as Nintendo did with the WiiU, or not at all, with the Switch? I'm sorry, I know hating Microsoft is Slashdot tradition, but they're the only ones doing this properly this generation. Sony's farmed it all out to a game streaming service and Nintendo just has you buying Virtual Console games over and over again or, back in the day, they bundled the Wii in the WiiU with all the shitty graphics and poor resolution that implied.
Nobody writes direct to hardware anymore, surely. It's all DirectX library calls.
All the common 8-bit systems had ROMs with defined entrypoints for various functions. You could even get CP/M on option ROMs IIRC. The ZX Spectrum +3/+2A had a DOS (+3DOS) with defined entry points for all disc operations, plus several other ROMs. You placed the one you wanted into the address space by bank switching, then jumped to the appropriate point.
I don't know why you think virtualising a console to be "easy". The problem is not emulating the instruction set but getting the timing of both the CPU and GPU correct. The GPU is especially hard since the game is going to feed it broken shaders and use GL extensions that have to be emulated with the new GPU. And mapping the firmware calls (for audio, video, controls, achievments, copy protection, storage, chat / music, resolution negotiation) to their equivalents in the new console, or stubs. I expect also that some libraries such as Unreal, Havok, PhysX etc. are so prevalent in games that emulators will have special cases for when they encounter that code.
Join me and ascend to God hood for you are weak! And I am mighty!
http://saveie6.com/
Then they failed. Fucking utterly failed.
Anyone who's written DirectX shader code for the Xbox knows about the limitations of the hardware and the optimizations you had to make for it, let alone the custom texture compression scheme that was used on no other platform (DXT != S3 despite Shawn Hargreaves saying otherwise).
Then you had XACT, the "Cross Platform" Audio Creation Tool, creating collections of .xgs (Xbox Global Settings), .xsb (Xbox Sound Bank) and .xwb (Xbox Wave Bank) files with custom audio compression schemes just for the Xbox audio hardware. They gave excellent results, but it was in no way portable to Windows nor any other gaming platform.
Who said ANYTHING about portability being a design goal? Everything you described used layers and layers of Microsoft APIs no? I’m not making a sideways joke about Microsoft’s overall strategy, but you could take it that way. They nailed this design goal.
Then they failed. Fucking utterly failed.
I'd say the product was successful.
Only the State obtains its revenue by coercion. - Murray Rothbard
I do not get why most consoles are not backward compatible.. what do MS, Sony, Nintendo etc think... that we all want to have 100's of consoles standing around, just to be able to play both the old and new games?
The technical reason or the economic one?
Technically, all three companies switched CPU architectures and not just from one CPU generation to the next. MS (PowerPC -> x86), Sony (Cell -> x86), Nintendo (PowerPC -> ARM). On the GPU side, there were lesser changes MS (ATI -> AMD), Sony (Cell -> AMD), Nintendo (ATI -> NVidia) with MS having the easiest transition. Games can be emulated if the newest console has enough CPU/GPU processing power. However the trend for this generation is that the advancements were more on the GPU side than the CPU side. Emulation would be more difficult because of this balance. Then after that, optimization and tweaking would have to happen.
Economically there is not much of an advantage as it takes away from new game sales. All three companies could make the transition easier by making older titles available in some way. It would easier for the console makers to pay the title companies to recompile but there's little profit for the title companies to do so.
Well, there's spam egg sausage and spam, that's not got much spam in it.
32 bit instructions should run natively on 64 bit, so the real issue here was PowerPC emulation not bits.
GCS/MU/P d- s:- a-- C++++$ UL++ P+ L++ E+ W++ N o K- w--- O M+ V- PS+++ PE Y+ PGP t+ 5- X R++ tv+ b++ DI++ D++ G+ e++ h-
Yeah this is spot-on. I don't know what he's talking about either. You maybe do a few hand optimizations but its almost all taken care of by drivers / compilers / etc.
GCS/MU/P d- s:- a-- C++++$ UL++ P+ L++ E+ W++ N o K- w--- O M+ V- PS+++ PE Y+ PGP t+ 5- X R++ tv+ b++ DI++ D++ G+ e++ h-
Compilers these days will do this automatically with the right flags.
GCS/MU/P d- s:- a-- C++++$ UL++ P+ L++ E+ W++ N o K- w--- O M+ V- PS+++ PE Y+ PGP t+ 5- X R++ tv+ b++ DI++ D++ G+ e++ h-
Cycle counting died out once the 32-bit era started, pretty much. There was no need to.
So your old 8 and 16-bit consoles and computers require cycle-accurate emulation because everyone programming them was cycle counting in the first place. The Apple II is an extreme example of this, because of the way it operates and how the display DMA basically hijacked the data bus when the CPU wasn't using it between instructions. Plus, with 8 and 16 bit comptuers and consoles, you had a set number of clocks between events - usually between a horizontal retrace and the vertical retrace.
Pre-crash consoles and computers were particularly troublesome because they didn't have a framebuffer, and on something like the ATari 2600, a good 70% of the CPU is used for screen updates (the TIA only worked on a line by line basis, so it had to be reloaded with new line data every horizontal retrace).
But around the PS2/original Xbox era, it all changed. The addition of caches and other things to the processors completely changed how consoles were programmed. No longer did you have to cycle count (because it was all single-cycle these days), but even worse, the cache added timing variability to the whole thing. Even today, getting a simulator that is cycle accurate is actually quite expensive because they include cache modelling in order to retain cycle accuracy.
Granted, the Xbox360 and PS3 got rid of some caches in an attempt to be more deterministic, but by then no one really cared - they were powerful enough that the games were basically programmed like their PC counterparts.
As such, these days emulating an older console is rather easy - high level emulation will get you far. This is especially true for the Xbox, since it's Microsoft and the DirectX API has been pretty stable. Since every game would've been programmed to the DirectX API, accurate GPU timings are no longer necessary (and chances are, there was always going to be slight variability between consoles, but you never cared since you wrote your code against the API). And given the rest of the API is basically Windows APIs, all the high level API work is done. All you need to emulate is the CPU and possibly rewrite shader programs
Long and short of it, Microsoft is probably in a good spot w.r.t. emulating old consoles
I do have an Xbox One and from my experience the 360 emulation works pretty well now. When the first emulated games came out there were some that did perform worse than the originals, but it seems like they've got the emulation nailed down now as they release updates for games tweak them for better performance. The Gears of War games ran pretty terribly at first, but a few updates later they now run very well. I used to stick with the native 360 versions for playing those games online but now I play the emulated versions on the Xbox One.
It sounds like emulation will be even better on the Xbox One X (AKA Project Scorpio). I guess we'll see once it comes out in a few weeks how things run, but it seems like the Xbox emulation team is putting in some work on getting it right.
http://www.eurogamer.net/articles/digitalfoundry-2017-project-scorpio-back-compat-five-ways-your-existing-games-will-be-better
That would be a dumb move. Games are not the same as computing applications which keep the CPU idle for the most part, waiting on user input. None of the high performance applications worked on Rosetta. Console games work the CPU GPU and memory hard, to extract maximum performance. The problem Apple was solving is not in the same domain or at the same scale. Perhaps it appears like it is to non-technical people such as yourself.
The only time I've seen games disappear from online stores is due to licensing issues, with music or some other IP license expired. Is this case could potentially be car licensing as well.
At Nintendo we did indeed include older hardware on our handheld platform for compatibility. The old hardware made for a great co-processor. Then there is this other way to do provide backward compatibility.
I think it probably is even more involved when it comes to emulating the AltiVec instructions. LL/SC would definitely pose a challenge since those are used for atomic operations on many RISC processors.* This is different than how Intel implemented atomic support with compare/exchange.
The next big thing going forward in terms of atomic support is transactional memory.
*Modern RISC processors are moving away from ll/sc to use dedicated atomic instructions like add, increment, swap, etc. due to scalability issues with ll/sc. ARMv8.2, for example, introduced a lot of atomic instructions. My employer made similar changes for MIPS to improve scalability with a lot of cores.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
comment so nice Gesture Lock Screen