Valve Bootstrapped Source 2 Engine On an Open-Source Vulkan Driver

An anonymous reader writes A new article out details how Valve bootstrapped their VULKAN back-end with the Source 2 Engine over a period of just four months thanks to relying on an open-source driver. With designing for the open-source Intel Vulkan Linux driver developed by LunarG, Valve developers were quickly able to resolve issues and progress the driver in a turn-key manner. This Intel Linux driver will be released as open-source once the Khronos VULKAN specification has been published.

Comments

Re:Wut?

[Anonymous Coward]

Do you work in the mail room or something?

Re:Wut?

[F34nor]

Hudsucker like a motherfucker.

Re:Wut?

[Anonymous Coward]

Slashdot 2015:

Will discuss, in depth, why intersectional feminism will save the world from toxic cis-het masculinity.

But if you mention important tech news, 2/3 of the posters will stare emptily at you like a dog trying to grasp quantum mechanics.

Universities have a lot to answer for.

Re:Wut?

[ArmoredDragon]

I believe what they're saying is that in this case they're able to troubleshoot issues from both the driver perspective and the game engine perspective. In the past you could only work with one or the other, and any problems found were harder to resolve because at the end of the day you had only a very narrow understanding of what the other is doing at the particular moment when the problem occurred.

Open Source Source 2

[Anonymous Coward]

Open source is neat, huh? Sure, would be nice if Source 2 were, you know, open source.

Re:Open Source Source 2

[Anonymous Coward]

Noooooooo! Open-source would undercut Valve's profits! Profits! Valve like MONEY!! Money money money money money money money money money!

Re:Open Source Source 2

[Anonymous Coward]

Self entitled much?

Why does someones work have to be open source for you?

You should be happy its free.

Re:Open Source Source 2

[jones_supa]

Open source is neat, huh? Sure, would be nice if Source 2 were, you know, open source.

Sure, and it would be nice if it was Christmas every day.

But...tadadadattataa! Tim Sweeney to the rescue! Unreal Engine 4 is fully open source if you really need an engine that you can modify all the way down to the hardware abstraction layers.

Re:Open Source Source 2

[Anonymous Coward]

Self righteous? I'm not advocating anything. Unlike you I don't give a fuck if someone is making bucket loads of money because I'm not a jealous prick.

If you have an issue with companies making profit from open source you can replace valve with google, microsoft, apple, facebook etc because valve does far more for open source than any of the others do.

Re:Open Source Source 2

[Anonymous Coward]

I've got news for you, there are a lot of people and businesses out there adding value and money to their lives by using free, open source software. And the thing is, many, if not most, of them don't use it to create more software. So even if you could force everyone who's ever touched open source software to release their own software as open source, there would be a lot of people benefiting from open source without returning anything.

If you want to make sure people pay their dues for benefiting from your software, you don't write open source, you write proprietary stuff that you charge money for. If you want people to collectively move on from paying money over and over again for the same basic software, or people rewriting code over and over again because they don't want to pay for libraries, so that people can collectively move on to doing putting resources toward new things, then you write open-source.

Re:Open Source Source 2

[beelsebob]

Uhhh, you realise that it was Valve who wrote the open source Vulcan driver, right?

They're making a shit ton of profit of code they wrote, and happened to be kind enough to open source for you to use.

That, and, if you open source something, you made a conscious decision to allow someone else to use that code, and to make a profit off it. At that point, they have no responsibility to you at all. It was your choice to take your code and open it. It was very nice of you to do that, but you are not buying some kind of "future favours" with it.

Re:Wut?

[Anonymous Coward]

But if you mention important tech news, 2/3 of the posters will stare emptily at you like a dog trying to grasp quantum mechanics.

Universities have a lot to answer for.

Universities can only do so much with the idiot-spawn they're given. Blame American mothers for spawning idiots.

Re:Wut?

[Anonymous Coward]

Khronos is the maintainer of the OpenGL specification, a 3D Graphics API. Vulkan was formerly known as GLNext and is an attempt to replace OpenGL with something that maps better to modern hardware and usage. In contrast to OpenGL 3/4 this is a rewrite from scratch that drops everything, no legacy compatibility profile, no outdated tutorials and no complex state management to bog down the drivers.

Re:Wut?

[Anonymous Coward]

Full disclosure makes bugs shallow. From TFA:

Valve developers part of their Linux cabal have previously talked of how it's nice having the open-source Intel/Radeon Mesa/Gallium3D drivers when porting games/engines to Linux as they're able to analyze the OpenGL driver code when debugging an issue or trying to workaround a performance shortcoming. In the early days, Valve also allowed Intel's open-source driver developers access to the code-bases of some of their games so they could look at what the Source renderer is doing, etc, in order to optimize the open-source driver performance. Open-source wins and makes the development process easier on both sides of the table when not having to deal with binary blobs in the equation. With the bringing up of Vulkan over the past few months, open-source was the champion once again.

Emphasis original.

Bootstrapped

[Anonymous Coward]

That word does not mean what you think it means.

Valve developers were quickly able to resolve issues and progress the driver in a turn-key manner.

Oh I see, we're playing bullshit bingo. In that case, as you were...

Re:Bootstrapped

[Anonymous Coward]

The article doesn't mention boots or straps anywhere. As usual, /. is adding value by adding utter shit.

Re:Bootstrapped

[Anonymous Coward]

Turn-key solution - it just goes out of the box, no effort required. Turn key, go.

In this context it means bugger all. In fact it's a nonsense; I can't make a meaningful statement out of that no matter how I try to mangle the term.

Re:Wut?

[Anonymous Coward]

Maybe slashdot has just gotten too popular for your taste, and is no longer only frequented by people who work in IT and programming.

Re:Wut?

[Anonymous Coward]

"Port of Valve's Source 2 game engine to new 3D API Vulkan was helped by availability of Open Source driver."

But if you put all the information in the headline, what are you going to put in the article? You have to progress the flow of information in a turn key manner somehow, so bootstrapping everything right in the headline is a no-go. Leverage expectations with jargon.

Re:Wut?

[Anonymous Coward]

Slashdot 2015:

Will discuss, in depth, why intersectional feminism will save the world from toxic cis-het masculinity.

But if you mention important tech news, 2/3 of the posters will stare emptily at you like a dog trying to grasp quantum mechanics.

Universities have a lot to answer for.

Thank you, for a few seconds there I was worried that the Male Justice Warriors of Slashdot wouldn't be able to turn also this discussion into being about feminism.

Re:Wut?

[Anonymous Coward]

Should sexist opensource developers have their projects censored or removed?

For the last cocksucking motherfucking time, YES, if the group providing press coverage, hosting, etc decides that they don't want to publish things from a known sexist. You want noticed? Then hide your opinions to make yourself more palatable to the sheep around you, like the rest of us do.

Buzzword much? Perhaps post should be deleted?

[Riventree]

"With designing for the open-source Intel Vulkan Linux driver developed by LunarG, Valve developers were quickly able to resolve issues and progress the driver in a turn-key manner." . . Leaving aside the broken English "With designing for" vs "Using"... "progress the driver in a turnkey manner" sounds (quite lilterally) like a buzzword generator. . . Looks more like an April-1 post to me.

Star Trek fan wonders...

[SeaFox]

Why is the Vukan-based driver have a spec named for the Klingon homeworld?

Re:Star Trek fan wonders...

[Anonymous Coward]

Shut up, pinkskin.

Re:Wut?

[gl4ss]

"there's an open source driver that's not compatible with opengl for intels new gpu so that the intel gpu can now run a game described as RETRO by the ceo of one of the companies".

something like that. who cares. intels gpu's still are so crap that you can make headlines of playing a 10 year old game on them, just like their gpu's were crappy 10 years ago in similar fashion.

Re:Wut?

[drinkypoo]

I work for one of the companies mentioned in the summary and I still have no clue what it means.

And I have to say, those floors are looking spotless.

Even if you've been under a rock, so you don't know what either Khronos or Vulkan are, you should still be able to use google and Wikipedia. If not, you are probably not part of the Slashdot audience.

No thanks

[loonycyborg]

600 lines to write a program that renders a triangle? Such apis are obsolete in 21th century period. Don't care what cowboy coders do on their game consoles. There's no need for line of code count maximization techniques in opensource.

Re:No thanks

[Anonymous Coward]

600 lines to write a program that renders a triangle? Such apis are obsolete in 21th century period.

Yes, but your program can now render those triangles 10-20% faster in terms of actual frame rates in practice, so that can be seen as a great improvement. It is also great for AMD (who started the "low level API" trend with Mantle), because work has been shifted from driver developers to application developers, and their hardware will not be held back by badly optimized drivers.

Re:No thanks

[loonycyborg]

It's really hard to prove that this 10-20% comes from api alone, if it exists at all. Sounds like random ass pull to me. (But a person who is paid per line of code will have no trouble proving this with a powerpoint presentation that'll absolutely convince anyone ignorant about graphics) Anyway right now we don't need more fps, but rather more stability. And having less code helps with that. Statistically bug count is proportional to line count and obviously shifting some code from drivers to be repeated by every application raises line count drastically.

Re:No thanks

[abies]

It doesn't meant that code to render million triangles will have 600 million lines. It is just a constant overhead to deal with various device trickery (multiple GPUs, multiple monitors, full screen versus window, etc etc). There will be 3rd party libraries immediately which will allow you to do 'render things on default screen, with default resolution, on default graphic card, not using SLI, using default malloc for memory allocation' with single function call, reducing line count drastically (but still probably considerably larger than 5-10 lines you would need in opengl). On the good side, as soon as you start creating parameterized shaders, instanced meshes etc, difference in line count will start to be neglible - it will be huge for both Vulkan and plain opengl/DirectX.
And it is still too much, then download Unity, Unreal or Cryengine. Especially Unreal should be good - you can program there with drag&drop (http://cybereality.com/wp-content/uploads/2015/02/UE4_Pong_BP.jpg)

Vulkan is API for game engine programmers, not for game developers. It is a lot closer to kernel API than to libc API if you want to compare with normal programming.

Re:No thanks

[TheRaven64]

How often do you render a single triangle? The goal of APIs like VULKAN is to provide complete control over every stage in the rendering pipeline (and a stable IR to make it easy to run code written in a custom DSL on the GPU). If you're just displaying a single triangle then this is not the API for you. It is intended to sit below APIs that provide scene graphs and so on for simple use.

Re:No thanks

[loonycyborg]

Then it's not replacement for opengl and shouldn't be considered one, but an api one level below. And I don't really think it's such a good idea. Like switching from C back to assembly.

Re:No thanks

[Anonymous Coward]

But a person who is paid per line of code

Who the hell gets paid per line of code? Maybe you should be blaming the problems on that instead...

At every development and consulting job I've been to, it was payment for time and results. If an extra day of coding could improve the speed of code ~10% for code that spent many cumulative days being waited for, that was a net win. Especially for code running on setups where 10% more hardware would cost more than a day's work. For code being run on a large number of client's computers, there can be less direct incentive, but 10% performance improvement can still be highly valued.

It's really hard to prove that this 10-20% comes from api alone,

10% change is really easy to demonstrate on any given setup. If you want to argue whether that difference is because the inherent API or because one driver is written better than the other, the difference doesn't matter much in the real world. Your fear shouldn't be from people paid per line of code (at any sane company at least), but from people trying to sell hardware. But it is easy enough to write a small test program in either case, considering ~600 lines of boiler plate is not that much to review, and a simple OpenGL shader program to just draw a couple triangles is also at least a couple hundred lines of code, more if you explicitly set up states instead of depending on defaults.

Also, if you have the same number of bugs per line of code for boiler plate stuff as the actual guts of the code, you need better programmers. And at least in Vulkan's case, some of the extra lines come from it being more explicit and verbose than OpenGL. OpenGL has a lot of functions that what they change depends on a state set many lines ago, and the programmer has to keep track of that or look back and forth a lot. Verbosity making each line of code clearer with what it does is a way to lower the total number of bugs even if volume of code increases.

Re:No thanks

[beelsebob]

No, they're not at all obsolete in the 21st Century.

When it comes to graphics programming, at the very low level (i.e. people who are writing the engines, not people who are using the engines), everything is about performance. These people (myself included) absolutely will jump through the hoops of writing 600 lines of setup code if it means I don't end up with half a CPU used up just recompiling shaders and verifying state changes all the time.

Ultimately, this code is written once, by the rendering engine developer, and then used an awful lot, so it being complex code means it's a large up front cost, but a very very low amortised cost. Meanwhile, it's a very very large performance gain.

Re:No thanks

[beelsebob]

No, it absolutely does come from designing the API differently.

The key conceptual difference is that in OpenGL, you change state, change state, change state, change state, render, change state, change state, change state, change state, change state, change state, render. You do this every render loop.

In {Vulcan | Direct3D 12 | Metal | Mantle}, you define two states at program launch, then each render loop, you do: bind existing state, render, bind existing state, render.

There's two gains here
1) A small one - you're calling fewer driver functions per frame. Many of these cross the kernel barrier, and as such are actually fairly large performance drags. When you're talking about doing a few thousand renders per frame, cutting out 2 kernel calls per render is a significant win. Cutting out 6 library calls per render is a less-significant, but still reasonable win.
2) The calls to change state can do much less. Like, unbelievable less. In OpenGL, the driver does not know when a render call is going to come, so it has two choices, either 1) it can do all the work for a state update every time a state change call comes in 2) it can cache all the state changes until a render call comes in. In the case of 1, this means it does a lot of duplicated work, in the car of 2, this means render calls lag a really long time. In {Vulcan | Direct3D 12 | Metal | Mantle}, instead, the driver can do all of the state verification and preparation work only once at application launch.

Why is state verification and preparation work so expensive I hear you ask. Well, a state change can have surprisingly large knock on effects. For example, on many graphics cards, blending is implemented as a frame buffer fetch, plus a few instructions at the end of the shader. That means that if you change the blending state, the driver actually has to re-compile the shader. Similarly, if you change the vertex format (e.g. to normalised vertices), again, that's implemented as a few instructions on the front of the vertex shader, so... gotta recompile and relink again.

Basically, a surprising amount of stuff requires a complete recompilation of the shader, and that's really costly. OpenGL makes the driver do this lots of times per frame. {Vulcan | Direct3D 12 | Metal | Mantle} do not.

Re:No thanks

[beelsebob]

You may not think it's a good idea, but every single game engine developer on the planet thinks it's a great idea ;).

Re:No thanks

[Anonymous Coward]

600 lines to write a program that renders a triangle?

Sounds fine to me. It gives me complete control over the entire pipeline, and I only have to write it once.

Re:No thanks

[loonycyborg]

Well, you're getting too excited about something that is evolutionary step backward and such hasty statements are a proof of this. I myself have some opengl experience and I can tell you getting to lower level is NOT what I would want. Yet I don't want to be locked into using an engine either with their often suspect design decisions. I consider opengl to be in sweet spot in this regard, though shaders are kinda pushing it.

Re:No thanks

[loonycyborg]

You don't make things clearer if you add more verbosity. It's the other way around. People will get lost in details without any job getting done. Copy/pasting boilerplate is boring and error prone(because you need to plug it into your code after all and may modify some bits). Better make api more powerful so there will be no need to have boilerplate in the first place.

Re:No thanks

[beelsebob]

Well, you're getting too excited about something that is evolutionary step backward and such hasty statements are a proof of this. I myself have some opengl experience and I can tell you getting to lower level is NOT what I would want. Yet I don't want to be locked into using an engine either with their often suspect design decisions. I consider opengl to be in sweet spot in this regard, though shaders are kinda pushing it.

So what you're saying is basically "I want I high level API for drawing graphics reasonably fast". That's fine, you should go grab an engine. There are plenty of very high quality ones out there, many of which are free (at least until you start making money). There are also plenty of mid-level ones that will not cause you to have significant lock in.

However, the person writing said engine for a AAA game, or an app that involves 3D rendering and that has to conserve battery life as much as possible while still hitting 60fps doesn't have the same set of constraints as you. People in that situation need to get as much performance as they possibly can, and for them, having OpenGL using up 20% of the CPU (on a desktop), or 50% of the CPU (on a mobile device) is completely unacceptable. The guy writing the AAA game could be using that CPU power to make the AI smarter, or simply to push more render items. Similarly, the guy writing the mobile app could get double the battery life if they didn't have OpenGL sat in the way.

For reference, the numbers above are reasonably accurate. Go try to push 200-300 draw calls on a reasonably high end Android or iOS device - you'll discover that an entire CPU core is used up by the OpenGL driver. When you compare that to Metal, where you can get more like 3000-4000 draw calls. That really makes the difference between a game that looks like a mobile game and a game that looks like a PS3 game.

Re:No thanks

[Bengie]

When your API is system call limited, then you reduce system calls by 20%-30% and you see a 20%-30% increase in performance, you might accidentally make a link.

Re:No thanks

[Bengie]

More like going from JavaScript to C. We need the performance. For certain parts of the rendering, Mantle tech demos were showing upwards of a 10x increase in performance. Some of the stuff I was reading was saying that while OpenGL and Direct3D made getting a minimum viable product out the door, Mantle was dramatically faster for getting a game that got more than 10fps. Once you hit the stage of game development where you need to optimize the code to make the game playable, Mantle was much faster for coding.

One example that they had is AMD took some expert game engine developers, the best in the industry, who had spent years optimizing their engine to make it faster, and with in 3 months of learning Mantle, they were able to have crappy unoptimized Mantle code out performing the DX11 code. Further time spent make Mantle much much faster with little effort.

The other big thing is graphic artists are easily able to try new techniques without putting months of optimizations behind them. They can quickly flesh out some code and get 20-30fps to see if it's even workable. It becomes the difference between trying new things and not.

Re:No thanks

[Anonymous Coward]

You don't make things clearer if you add more verbosity

When you are making it explicit what you are doing as opposed to requiring the programmer to be aware of various assumptions and previous states, it certainly does make it clearer. Instead of saying "Change the state (last thing loaded by function X but not Y)" to "Change the state of Z", where the part in parentheses isn't actually in the code, the latter is more verbose, but also much clearer.

Copy/pasting boilerplate is boring and error prone(because you need to plug it into your code after all and may modify some bits).

The large part of that boilerplate is just setting up the state of things. Whether it is explicitly written in code or hidden behind defaults, either way you have to know what to set things to. If you don't like the number of options, you can use certain defaults, or let some library make the choice for you.

Re:No thanks

[Anonymous Coward]

When you switch from using global or internal static variables to store what state you've selected and static arrays to hold them, to instead passing around actual references to the states and creating them, you're increasing the verbosity, but also greatly increasing the clarity of what your functions are actually acting upon. You don't have to worry about different control paths leaving the targets in different states, or remembering exactly which functions will update or change those.

Re:Wut?

[Anonymous Coward]

Yeah... lolwut.

Open source has it's places but good f'ing luck getting ahead of the RMT mofo's that plague and destroy online games if any part of the game engine's source is available. Instead of just having a "single static blob" that can be encrypted, you end up with a bazillion libraries, each with the weakness of being able to be substituted or wrapped to insert malicious/cheating code.

In the case of Source (which requires a C/C++ compiler to being with) and Unity (which can be decompiled from C#) the hackers have been having a fun time dismantling software made with these engines and posting the assets all over the web. Not that would stop anyone in the first place, since all you need to do on Windows is throw a OpenGL or Direct3D wrapper on top of a game to dump everything as it's rendered. 2D UI assets no problem. 3D assets, slightly harder.

I do hope that something more "unified" that can be used across everything without relying on Unity comes out but the only games today that really benefit from this are mobile games. Console games and Desktop games have significant overhead (moreso on the PC) that the simplest solution is to write a thin "OS" layer that that the system boots in a VM and runs the game in the "foreground" instead of the existing cruddy solution that involves running the game on top of a full featured OS that saps resources.

Re:Wut?

[Anonymous Coward]

Have you ever seriously considered not playing games with assholes?

Re:Wut?

[Anonymous Coward]

I'm pretty sure he knows that Khronos and Vulkan is... his comment was about the line...
"resolve issues and progress the driver in a turn-key manner. "

Which, lets be honest, is totally meaningless outside of marketing drivel.

Removing anuses from MMOs

[tepples]

good f'ing luck getting ahead of the RMT mofo's that plague and destroy online games

Have you ever seriously considered not playing games with assholes?

"RMT" (real money trading) is a term used in MMO (massively multiplayer online) games, which put thousands of players or more. So if an anus is playing at all, he is playing with you, in the same virtual economy even if not directly. How can anuses be removed efficiently from such an environment without violating some countries' implied warranty statutes? Or did you mean give up MMO games entirely?

Khitomer Accords

[tepples]

Aren't the Vulcans and Klingons working together by TNG anyway? The peace talks that led to a peace treaty between the United Federation of Planets and the Klingon Empire were in fact started by a half-Vulcan (Capt. Spock) and a Klingon (Chancellor Gorkon).

What's glibc?

[tepples]

Vulkan is API for game engine programmers, not for game developers. It is a lot closer to kernel API than to libc API if you want to compare with normal programming.

In this analogy that compares libc to a game engine, what's the counterpart to GNU C Library (glibc), a free implementation of libc?

Re:What's glibc?

[abies]

Nothing yet, because Vulkan is not yet public, so no 'free' implementations exist. But I'm quite sure that as soon as it appears, projects like Ogre3D or openscenegraph will provide their bindings over Vulkan.

This analogy is flawed in some respect - because glibc is a drop in replacement for libc. There is no such agreement on higher level API in this case, but rather competing APIs/engines.

Re:Wut?

[Anonymous Coward]

Actually, the standard has support of the holy trinity of graphics (AMD, nVidia, Intel), so the original post was bullshit.

Re:Wut?

[Half-pint HAL]

Should sexist opensource developers have their projects censored or removed?

Alternative question: when there are hundreds of opensource developers out there, what criteria do you suggest should be given publicity?

Re:Wut?

[Anonymous Coward]

1. code quality?
2. fit for purpose?

The politics of the developer are irrelevant.

So this is what they're doing instead of....

[afaiktoit]

working on half life 3? bastards.

Re:Wut?

[Anonymous Coward]

Popular? Slashdot is dying on its arse, in case you hadn't noticed.