Slashdot Mirror


DirectX 12 Performance Tested In Ashes of the Singularity

Vigile writes: The future of graphics APIs lies in DirectX 12 and Vulkan, both built to target GPU hardware at a lower level than previously available. The advantages are better performance, better efficiency on all hardware and more control for the developer that is willing to put in the time and effort to understand the hardware in question. Until today we have only heard or seen theoretical "peak" performance claims of DX12 compared to DX11. PC Perspective just posted an article that uses a pre-beta version of Ashes of the Singularity, an upcoming RTS utilizing the Oxide Games Nitrous engine, to evaluate and compare DX12's performance claims and gains against DX11. In the story we find five different processor platforms tested with two different GPUs and two different resolutions. Results are interesting and show that DX12 levels the playing field for AMD, with its R9 390X gaining enough ground in DX12 to overcome a significant performance deficit that exists using DX11 to the GTX 980.

47 of 96 comments (clear)

  1. Wait a min, Direct X 12 needs what? by Anonymous Coward · · Score: 5, Interesting

    The Developer now must know MORE about the underlying hardware to make the best use of Direct X 12?

    This is a total step in the WRONG direction. So now having Direct X 12 hardware doesn't mean your game now just works, oh no. If you want the full experience you now must have the HARDWARE that your game was written for, or forget all this compatible Direct X stuff. How's this different from the game developer just coding directly to the video hardware of choice? That's what they do now, especially when they are funded by the video hardware guys in an effort to sell more hardware..

    For this Direct X thing to really be useful, it needs to isolate the developer from the hardware implementation. You need to abstract away the vendor specifics and make the programming agnostic to what hardware it's running on... Otherwise this is all going to just going to be what it has always been, vendor lock in for specific games and drive us towards only ONE video hardware chip maker....

    1. Re:Wait a min, Direct X 12 needs what? by GigaplexNZ · · Score: 2

      The Developer now must know MORE about the underlying hardware to make the best use of Direct X 12?

      Uh, yeah, that's what a lower level API means.

    2. Re:Wait a min, Direct X 12 needs what? by exomondo · · Score: 5, Interesting

      I'm not quite sure why you are modded down, this is a perfectly valid and legitimate concern.

      Part of the problem of PC gaming has always been that the choice of hardware has meant we have had to have abstraction layers and these introduced inefficiencies which have reached significant levels nowadays. You remember seeing every time a new console generation is released that PC gamers proclaim it is terrible because their PC hardware is theoretically more powerful? Well that is true however being more powerful is pointless if that power can not be efficiently utilized, it just goes to waste. Heavy abstractions and generic implementations mean a lot of that increased power is wasted. So it is about bringing APIs up to scratch with modern hardware and removing some of the legacy cruft. It means more responsibility on the part of the application developer in the same way that it did when we went from the fixed function pipeline to the programmable pipeline many years ago.

      For this Direct X thing to really be useful, it needs to isolate the developer from the hardware implementation. You need to abstract away the vendor specifics and make the programming agnostic to what hardware it's running on.

      Well it still is relatively hardware agnostic, the difference is that we have had many advances in hardware that are not reflected in modern APIs. Take resource binding for example, currently resources are bound to "slots" when you define a shader pipeline and are fixed at draw time so if you want to change the resources that this shader pipeline uses you need to bind those new resources to those slots and draw again. This was a great general view of hardware at the time and is forward-compatible. But modern hardware has long had the capability to index a table of resources rather than just whatever is currently bound to the resource slots though the APIs are not architected to allow this. So this is generally implemented on an application-specific basis in the driver when the driver author (usually the hardware manufacturer) works with the application developer to understand what they are trying to accomplish that the API doesn't provide and then create a kind of munging layer in the driver that converts the application's "bind->draw bound, rebind->draw bound" workflow into a "create table->draw all" workflow. This is partly why you see significant performance differences in applications between driver versions but also why an application on similarly capable hardware can perform so vastly different between vendors.

      That's just one example, I hope it's somewhat clear. Yes there is less of an abstraction in some areas but at the API level it doesn't do things like expose an AMD R9 -specific feature or an nVidia GTX980 -specific feature.

    3. Re:Wait a min, Direct X 12 needs what? by Dutch+Gun · · Score: 3, Insightful

      No, don't worry. I haven't seen the DirectX 12 API yet myself (I'm still working in DX9-11 land), but I'm pretty sure all this is doing is making the abstraction layer more closely match the realities of the existing hardware designs. That is, it's not eliminating the abstraction altogether, but making it a much thinner layer, so as to avoid imposing unnecessary overhead.

      All the GPUs work in roughly the same manner, because they have to execute the same common shader micro-code. In order to be labeled as "DX11" or "DX12" compliant hardware, a GPU must be able to perform a minimum set of functionality. Moreover, the vast majority of this functionality is accessed via shader languages, and this doesn't change from GPU to GPU.

      I'd be surprised if there was any significant divergence at all between different types of GPUs in the code at all. DirectX 12 looks like it's going to be a very good thing, both for developers and for gamers.

      --
      Irony: Agile development has too much intertia to be abandoned now.
    4. Re:Wait a min, Direct X 12 needs what? by Anonymous Coward · · Score: 1

      The Developer now must know MORE about the underlying hardware to make the best use of Direct X 12?

      If you think that is bad, you should see the shit developers have to do for current APIs. If you're not a AAA game, technically you can only use the standard APIs, but these is incredibly slow making lots of system calls are inherently slow. Want to know why driver updates list increased performance in certain games? Because they driver devs look at how those games are using the APIs, then detect these games and do black magic behind the scenes that is all a bunch of bandaids to work around the horrible performance limitations of the current APIs

      Indy devs come along and want to make a game, they need to reverse engineer these undocumented behind-the-scenes bandaids that could break at any time because they're not part of the standard. DX12 is an attempt to remove all of this driver crap that has been used for decades. Maybe someday we won't have 300MiB graphics drivers.

    5. Re:Wait a min, Direct X 12 needs what? by Anonymous Coward · · Score: 3, Insightful

      I know slashdot usually does not read AC, but I reply none the less.

      Your post and grandparent is spot on, but I still do not think it is a problem. The ecosystem has changed a lot, "back then" basically every studio wrote their own engine, today we have three big engine makers (Epic with UE, Crytek with CryEngine, and Unity) on the market plus a few big in-house engines of the publishers that they do not license out.
      A few other larger projects make their own engines, but if we are down to half a dozen major ones and another half dozen big ones we are basically there.

      Small projects - and many AAA titles - just take one of the existing engines and do not bother.

      There also was a large change on the manufacturer side: there are not half a dozen big card makers anymore with a whole lineup of products, we are seeing two major makers for games + intel for everything else; their chips are also based on the very few actual designs and custom-building code for these two or five bases is possible - especially as we have big engine makers who do the heavy lifting.

      The heavy abstraction of old DirectX versions was the right thing to do - and the introduction of DirectX is actually one thing Microsoft did really right and we have to thank them for that. Does anyone remember the chaos in the DOS-days? - but now it is also the right thing to keep a common API but reduce the abstraction at the cost of big engine makers (putting 10s and 100s of millions in their products) and large publishers who in-house create an engine (Frostbite, for example) which fuels several AAA titles in the years to come.

  2. Driver Differences by nateman1352 · · Score: 2, Interesting

    I think what this benchmark really tells us is two things:

    1. nVidia has not optimized their driver stack for DX12 as much as AMD has optimized for DX12
    2. The performance difference between AMD and nVidia is likely a software issue, not a hardware issue (nVidia's driver has a more optimized DX11 implementation than AMD's). However, it is possible that nVidia's silicon architecture is designed to run DX11 workloads better than AMD's.

    Bullet #1 make sense, AMD has been developing Mantle for years now so they likely have a more mature stack for these low level APIs. Bullet #2 also makes sense, AMD/ATI's driver has been a known weak point for a long time now.

    1. Re:Driver Differences by Anonymous Coward · · Score: 1

      Not really.

      It has been known for a long time that AMD has poor driver overhead in Windows. This removes that. Its also known that in terms of brute power, AMD cards are in the vicinity of 15% faster overall. I fully expect AMD to catch up and perhaps overtake in DX12 benchmarking.

    2. Re:Driver Differences by exomondo · · Score: 1

      I think what this benchmark really tells us is two things:

      1. nVidia has not optimized their driver stack for DX12 as much as AMD has optimized for DX12

      Maybe but the whole idea is that should have little impact. These new APIs are about reducing driver overhead by re-designing the API such that it is a more accurate representation of the underlying hardware requiring a lot less work from the driver in converting between what the application thinks the hardware looks like and what the hardware actually looks like.

    3. Re:Driver Differences by Narishma · · Score: 1

      They aren't offloading to DX12, since DX12 is a much smaller API that does a lot less than the previous ones. They are offloading to the game engine developers.

      --
      Mada mada dane.
  3. If it means I have to take Windows 10? by hairyfeet · · Score: 5, Insightful

    I'll pass, even though I have all AMD hardware. Its just not worth giving up my privacy and having Big Brother MSFT snoop through everything I do just to have DX12. I love gaming but not enough to let MSFT stick a webcam over my shoulder, thx anyway.

    --
    ACs don't waste your time replying, your posts are never seen by me.
    1. Re:If it means I have to take Windows 10? by hairyfeet · · Score: 1

      Does everybody see NOW why ACs are worthless? they are apparently too fucking stupid to read headlines and certainly are too retarded to actually use a search engine because if they could do more than pick their own noses they would have seen that you can't turn off Windows 10 spying as the "privacy settings" are the equivalent of the button on a street light, something designed to make the user think he has control when its just bullshit.

      --
      ACs don't waste your time replying, your posts are never seen by me.
    2. Re:If it means I have to take Windows 10? by Anonymous Coward · · Score: 1

      If they didn't give a shit then why are they collecting data?

      Let me guess, you use the computer mommy and daddy bought for you purely for video games and Facebook, right? Of course *you* don't have privacy concerns, you have no responsibility nor any data worth harvesting.

    3. Re:If it means I have to take Windows 10? by Yunzil · · Score: 1

      Do you even know (or care) what it's sending back or did you just read the headlines and start freaking out about your privacy?

  4. "Computronium" by danomatika · · Score: 1

    I'm more interested in the fact that the game used for benchmarking has the following in it's backstory: "Computronium became the ultimate currency."

  5. DX12 levels the playing field for AMD by JohnStock · · Score: 2

    No.. This engine's implementation *may* level the playing field for AMD but that does not mean that API does.

  6. Re:Again, linux out of the equation. by GigaplexNZ · · Score: 1

    DirectX is a Microsoft API, so yes, it's pretty likely to be Windows only. For Linux, there's Vulkan.

  7. These results don't make much sense by Kjella · · Score: 2

    It is widely known that DX12 will reduce draw call overhead, making weaker CPUs perform better relative to stronger CPUs. This is of course good for AMD, since they don't have high-end CPUs anymore though it's bit of a "scorched earth" result where gamers don't need expensive CPUs at all. But if you look at "Ashes (Heavy) DX11 to DX12 Scaling - Radeon R9 390X" and look at an extremely powerful CPU like the Core i7-6700 you're seeing 50-100% gains. If you're that severely bottle-necked by a 4+ GHz quad core then this is not a typical DX11 game.

    We can compare the "typical" difference between a R9 390X and GTX980 in Anandtech's bench, though I have to substitute for a R290X "Uber" so the differences should actually be even smaller. Normally these cards are almost head to head, the question is not why DX12 is closing the gap but why there's such a huge DX11 gap to begin with. And the only reason I can come up with is because they're pushing way, way more draw calls than normal. Which may be DX12 enabling developers to do things they wanted to, but couldn't before or it could be to make someone look good/bad.

    --
    Live today, because you never know what tomorrow brings
    1. Re:These results don't make much sense by GigaplexNZ · · Score: 1

      If you're that severely bottle-necked by a 4+ GHz quad core then this is not a typical DX11 game.

      We already know this isn't a typical DX11 game. They're using way more draw calls than a typical DX11 game simply because DX12 allows them to make more. That's even stated in the article.

    2. Re:These results don't make much sense by loonycyborg · · Score: 1

      Anyway I have no idea what's up with this obsession with increasing the number of api calls. You don't need to do a lot of those if you design your rendering pipeline right. One glDrawElements would accomplish a lot of work, especially if you're using vertex and pixel shaders.

    3. Re:These results don't make much sense by Fragnet · · Score: 1

      It's not just the number of API calls, it's all the things that go on in the driver. Things like shader recompiles to match new hardware state, mutexes and blocks on resource use, resource use tracking to make sure the next call doesn't interfere with the previous one and so on. There's a huge amount of bloat in drivers at the moment and it all contributes to the relative lack of efficiency. There's a fantastic post here at gamedev on the subject

  8. Its the threading model by Anonymous Coward · · Score: 2, Interesting

    Direct3D 11 introduced the capability to render with multiple threads via deferred contexts. NVidia chose to support that feature, AMD did not. Direct3D 12 mandates multithreaded rendering.

    1. Re:Its the threading model by Fragnet · · Score: 1

      Gains from threaded rendering with D3D 11 were marginal to non-existent because of the way the driver worked. With D3D12 (and Vulkan), threading is a 1st class citizen. You'll be able to call into the driver without a blocking penalty, so it will genuinely be faster (all else being equal). D3D12 doesn't "mandate" MT rendering by the way. You can still do everything on a single thread if you want.

  9. What it actually shows by phizi0n · · Score: 1

    All that these results show is that AMD has higher draw call overhead than nVidia does on DX11 but DX11 and older games were not designed to make massive amounts of draw calls so it doesn't matter all that much when playing games designed for these older API's. DX12 was designed to minimize the API overhead to allow games to start drawing way more stuff and games that are designed to take advantage of this are going to suck on older API's when they support it. If developers were to write support for DX12 in their current games that don't draw much stuff then you wouldn't see nearly as much of a performance gain.

  10. Star Swarm style marketing continues by Luckyo · · Score: 4, Interesting

    Those who haven't clued in yet: this is the same engine that was used for "unreleased game turned DX 12 synthetic benchmark" star swarm. All same caveats apply:
    1. Unknown engine not available to public with unknown performance. We have no idea how DX11 implementation is made, or why DX12 is so much faster than anywhere else seen so far.
    2. Is in pre-alpha, meaning performance is all over the place and a complete black box, it could render faster in DX11 in next build for all we know.

    We've been there with mantle already. Specialized tech demos showing massive performance boost from using mantle over DX11. Then release, frostbite et al start supporting it and we see minimal to no performance boost outside really low end CPUs bundled with really high end GPUs.

    Show me this kinds of numbers on a known engine that has a polished DX11 implementation like unreal 4 engine, and I'll actually believe you. Until then, all I see is more marketing BS.

    1. Re:Star Swarm style marketing continues by Fragnet · · Score: 2
    2. Re:Star Swarm style marketing continues by Kaitiff · · Score: 1

      There's a problem with your argument there chief... this is a game that is about to release. I'ts not an alpha.. from the video I just watched the company is about to release 'Ashes' for purchase.

      DX11 is dead tree man. Might as well use that same argument a few years ago with DX9.. when M$ moves on with it's API, you either get onboard or you're left in the dust. I've been hoping for years and years that someone could make the push for OpenGL to become competitive again but that's not gonna happen. At least with DX12 being based off of Mantle, you will have the Vulkan equivalent that is almost code identical so games can work as well off of Win10 as on it. I want to see Linux and Mac games released at the same time as windows games and running just as fast. Thats what will open up the field to everyone finally.

      The other thing that is glaringly obvious is that the fastest AMD chips are suck for gaming with this API. I had hoped that Mantle would really bolster up the AMD cpu performance in gaming, but that was almost embarrassing to look at; the I3 is faster than AMD's flagship processor in this game. Ouch.

      --
      If I sound stupid, it's not me talking....
    3. Re:Star Swarm style marketing continues by drinkypoo · · Score: 1

      the I3 is faster than AMD's flagship processor in this game. Ouch.

      Right now for non-power users the sweet spot is Haswell i5, it is relatively cheap and is more powerful than anything AMD has to offer. However, it will cost you at least a couple hundred bucks more than the AMD solution when you factor in the motherboard as well, and it doesn't get you significantly better maximum frame rates. What it does get you is notable better minimum rates. What that suggests to me is that AMD's cores are just as fast as Intel's, but they can't shovel data into them as quickly. That's a bit ironic considering that was one of the big benefits of the K7, with its on-die memory controller... Everyone has one of those now, obviously.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    4. Re:Star Swarm style marketing continues by Fragnet · · Score: 1

      It doesn't say which version of DirectX it's comparing against. As he's using Windows 8.1 it certainly wasn't D3D 12.

  11. Re:DirectX is *NOT* the "future of graphics" by Anonymous Coward · · Score: 1

    Vulcan (sic), maybe. It's open platform.

    OpenGL has existed, and has been open, for longer than Direct3D itself has. Direct3D still exists. Direct3D isn't even the current most popular graphics API if you consider mobile devices, but if you're just looking at high-end PC games, it still hasn't been dethroned and doesn't show any particular signs of being dethroned anytime soon.

    DirectX 12, as if!!! Microsoft screwed up terribly tying DirectX to Vista where they denied DirectX upgrades to customers who wouldn't "upgrade" to Vista -- nearly everyone!

    Uptake of DX10/DX11 is currently over 97% amongst Steam users. As if!

    And their Windows 8 disaster chased developers off the Windows platform and onto mobile devices where OpenGL rules supreme.

    PC game sales did not take a noticeable dip after Windows 8. Go ahead, show me a some sort of supporting evidence of any sort and prove me wrong. You're making the claims here.

    Look: "Microsoft's Windows 10 privacy headache extends into gaming"

    Privacy shittyness is something to be concerned about with Windows 10, for sure. But, did you read the article you just cited, or were you taken in by its clickbaity title and stopped reading there? If you'd read it, you'd notice the entire article hinges on this EULA snippet that apparently applies to Skype and Xbox Live:

    We may automatically check your version of the software and download software updates or configuration changes, including those that prevent you from accessing the Services, playing counterfeit games, or using unauthorized hardware peripheral devices.

    Can you tell me the last time you bought a AAA game that doesn't have similar wording in its own EULA? Oh no, they're going to... possibly update the game's DRM to prevent you from pirating it or accessing XBL with a pirated copy? How terrible?

  12. Re:DirectX is *NOT* the "future of graphics" by Fragnet · · Score: 1

    Privacy is something to be concerned about not just on Windows but everywhere. I don't know why people focus on Microsoft when Apple and Google do similar things.

  13. Re:I'm not even torn any more. Windows 10 is a NOO by Fragnet · · Score: 1

    You can switch most if not all of Windows 10 "chatter" off. But if you're determined on this point just wait until 4th quarter or early next year for Vulkan, which will run on 7 (and XP) and obviously Linux. Valve have done a lot of work on this.

  14. Re:I'm not even torn any more. Windows 10 is a NOO by Blaskowicz · · Score: 1

    You shouldn't be in a hurry but upgrade to Windows 8.1 will be worth it eventually, with EOL in 2023 rather that 2020 and an update from WDDM 1.1 to WDDM 1.3.

  15. Re:Not terribly surprising by drinkypoo · · Score: 1

    NVIDIA is just as terrible for drivers. Just have to see the massive failure for the Windows 10 launch.

    A tiny minority of nVidia users had a crash bug, and a new driver was rolled out a day later. That doesn't sound like massive failure to me. Massive failure is when the GUI for your driver is larger, slower, and more bloated than Adobe Reader.

    --
    "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  16. For my new PC by Roodvlees · · Score: 1

    I'm hoping it'll
    - Make good use of DX12
    - Have a stable and performing Windows 10
    - Have the new Intel processor
    - Be super quiet

    --
    Thank you, Bradley Manning, Edward Snowden and so many others, for courageously defending humanity, my freedom and more!
  17. Re:Weird. by Fragnet · · Score: 1

    There are two factors to consider when it comes to vendor specific extensions or minimum sets of functionality. The first is that you want to design for the minimum if your goal is to reach the broadest possible audience. If you want to do something "cool" with extended functionality for specific classes of hardware or even a specific instance, then I don't see why a good API would prevent that. The second point is that being visible outside of a D3D, Vulkan or OpenGL API update cycle allows hardware innovation to proceed outside of it too.

    You can imagine a situation where hardware vendors invent feature X and then push for feature X to be part of the API. When the API is released nobody uses feature X. All it did was contribute to API bloat. If you allow the extension to be used outside of the API cycle, at least someone is going to try it out and see if it's of any value. Then you're in a better position to know whether it belongs in the next minimum set of functionality or not.

  18. Re:I'm not even torn any more. Windows 10 is a NOO by Fragnet · · Score: 1

    No. You didn't link this proof either.

  19. Re:I'm not even torn any more. Windows 10 is a NOO by JustNiz · · Score: 1

    I REALLY don't like the UI of Windows 8.1. I think its completely unworkable.
    I use Linux for anything other than gaming, so I also don't care about Win7 EOL since it doesn't seem to stop you actually using the OS, just no more annoying alerts about ambiguous security patches that ususally don't actually anything relevant/significant anyway.

    Would there be any noticeable benefit of WDDM1.3 over WDDM 1.1 if when just using Windows for gaming?

  20. Re:I'm not even torn any more. Windows 10 is a NOO by JustNiz · · Score: 1

    >> DX12 is all hype.

    Not according to this:
    https://www.youtube.com/watch?...

    The big issue seems to be whether the benchmark is synthetic/representative or not, but since it is really just an early version of a real game that will be released next year I tend towards believing it is legit.

  21. Re:I'm not even torn any more. Windows 10 is a NOO by JustNiz · · Score: 1

    I'm VERY interested in Vulkan but my fear is that all the AAA developers other than Valve will just keep assuming DirectX-only even for new games development, mostly through an incorrect belief that they're not loosing sales by failing to support Linux.

    I find it very frustrating that Bethesda especially keep plodding along on their own tired old windows-only engine instead of switching to, say UT4. Apart from the Linux-support-for-free that would bring, judging from what they showed at IGN the upcoming Fallout 4 would have looked graphically a whole lot better, instead of like a DLC pack for Fallout 3 (from 2008).

  22. Re:I'm not even torn any more. Windows 10 is a NOO by Fragnet · · Score: 1

    I'm guessing Bethesda have a lot of money invested in their engine and the bean counters just don't see the benefit of going cross-platform. This will be easier (cheaper) with Vulkan, assuming they're going to use a next-gen API with their next release. And I've got a sneaking suspicion that when developers look at the new API landscape, Vulkan will be a no-brainer. It won't be superior to D3D 12, it'll just be installed on more machines.

  23. Re:I'm not even torn any more. Windows 10 is a NOO by JustNiz · · Score: 1

    what about putting
    127.0.0.1 microsoft.com
    in your hosts file? Would that work?

  24. Re:Weird. by Dutch+Gun · · Score: 1

    No, it's not weird. You just seem to be confused by the description of "lower level hardware access".

    These aren't vendor-specific extensions. This is simply a redesigned API that eliminates a lot of the software bottlenecks, allowing developers to get a bit closer to the hardware in a GPU-independent manner. You're still programming to a common API. It's just that the API better reflects what a 2015-era GPU can do, rather than impose a lot of software overhead that isn't really needed.

    --
    Irony: Agile development has too much intertia to be abandoned now.
  25. Re:I'm not even torn any more. Windows 10 is a NOO by Blaskowicz · · Score: 1

    Except for the ugly window borders and other pet things.
    I don't even like Windows 7 that much either, it's mostly fine but I hate the file manager.

  26. Re:DirectX is *NOT* the "future of graphics" by BringMyShuttle · · Score: 1

    Safe to say the Troll(-1) mods downvoting this this aren't genuine but really Disagree(-1) or Shill(-1).

  27. Re:I'm not even torn any more. Windows 10 is a NOO by Blaskowicz · · Score: 1

    Windows 7 has the Home Basic version, install that and you only get the 2D desktop.

    Changing visual style? with Windows 8 they removed classic look, and with Windows 7 they removed the color schemes in clasic look.. I could use a very old third party tool but it didn't include the old styles only additional ugly ones.
    With Windows 8 you can get such train wreck of a theme http://kizo2703.deviantart.com...

    So with Windows the easy way I found is to install 7 Home Basic and leave it alone ("Aero Basic"). I don't put it on my PC though.

  28. Re:I'm not even torn any more. Windows 10 is a NOO by Blaskowicz · · Score: 1

    Some more efficient.. things.. the most easily understandable is better multitasking on the GPU.
    https://en.wikipedia.org/wiki/...
    And some.. things.. a wild guess is that it allows great performance in alt-tabbing out and in of a game. We all cordially hate Windows 8, but it is amazingly fast and smooth at showing you a Metro thing or the Charmed bar, whether you wanted it or not.