Keith Packard's Xfree86 Fork Officially Started

Reivec writes "I was having a discussion with Keith Packard on IRC about the current developments in the XFree86 Saga and politics already discussed here earlier, and I learned many interesting things. The project has a new website, xwin, and things are getting underway. 'We're in the process of building community, from that we can construct a government. It's a hard process to construct a representative system from what we have now, so it will take a bit of time. Weeks, not months. --Keith'" Read on for some more details. Update: 04/13 03:30 GMT by T : Reader Khalid points to this informative interview with Packard at Linux Weekly News, too. " The site is has only been up a day or so and there isn't a lot on it right now, but he would like to see a lot of community involvement on the site and many user submitted stories to get conversation rolling. A french site has already taken notice and posted some information on xwin as well. Since such a fork could make a large impact on many *NIX users, I felt the need to ask, 'assuming you had an active fork under development, how interchangable would you expect it to be with Xfree (assuming release builds). Do you think distros would be quick to change if it offered improvements? Or could they provide both and have the user choose upon installation?' Keith replied, 'Given that distros will have input into how it gets built, I expect they'd be interested in a version closer to what they need. And, given that RH and Debian maintainers are both actively encouraging changes, it's hard to see how they wouldn't want to follow. (or lead).' So if you have had any interest at all in the XFree86 development, this is definitely a community site you should take advantage of."

Comments

Re:What are their priorities?

[arkanes]

Some rebuttals to YOUR points:

1) This isn't about XFree being fast for you. And if it performs as well as (say) Windows 2k or XP on modern hardware, then you've spent alot of time tweaking X, and probably your kernel. X should be decent out of the box, and it isn't. "Works good enough" isn't something that I personally like settling for.

2) Standardization is absolutely a point of X. I don't know how you can think otherwise. One of the biggest objections to this port is the possible breaking of the X standards.

3) There is no reason whatsoever that XF86Config needs to be the monster that it is. A logical hierarchy of settings would be a good first step. Alot of the crap in XF86Config is handled by drivers using a standardized interface in Windows - this is a reasonable model to copy. That would help eliminate the need for every distro that's trying to be user-friendly to write it's own hardware detection program.

the usual misconceptions

[g4dget]

1. Performance. There needs to be some serious performance boosting. Rip out a whole lot of fluff. Honestly, how often do you need remote xwindows? Yes, there is a use for it, but that should be a seperate build altogether.

There is no "fluff" there. X11 runs as a separate user-mode process from applications. That means that commands to it need to go from the user process to the display process. X11 uses an asynchronous protocol and a mixture of shared memory and UNIX-domain sockets. And for games and other applications, there is DRI.

It happens to be the case that the X11 protocol and semantics are well-enough defined that the same protocol works over fast networks, but you don't pay anything for that.

Macintosh (as far a I can tell) works the same way: a display server, user mode applicatins, and some IPC mechanism connecting them. The only reason remote display for the Mac doesn't work like X11 is because it lacks some high-level primitives.

Windows used to start out as a frame buffer library, but it, too, works pretty much like X11 these days: asynchronous communications between user-mode processes and a display server running in a separate address space. The only thing NT/XP do differently is that the display server runs i the kernel. You could put an X11 server in the kernel, but it probably wouldn't make a big difference in performance (and it would be a headache).

When a particular X11 implementatin is slow, it's usually because of bad drivers or bad configuration. With comparable drivers, X11 performance is top-notch--usually better than Macintosh and comparable to Windows. And many X11 applications are slow or inefficient because their developers assumed they were programming a frame buffer--an assumption that is wrong on all major GUI platforms these days.

In short, this "X is slow because of network transparency" is wrong in multiple ways. First, X11 is not slow compared to other popular windowing systems. Second, nobody has ever been able to describe a way in which X11 could be made faster by choosing a different IPC mechanism. People who criticize X11 for using IPC usually assume incorrectly that other systems don't use IPC, but they do.

2. Standardization. Flexibility is nice, but having every damn program do things differently is annoying. It's also a very bad thing if you are trying to break into the mainstream.

X11 is standardized. What is not standardized is GUI environments and toolkits. But there is a reason for that: people are still figuring it out. It's software evolution in action. And it's not like Windows or Macintosh have figured that one out either: on Windows, people use dozens of different toolkits, several of which come from Microsoft Similarly for Macintosh. Gnome and KDE are making an effort to interoperate, and that's all you can ask for.

Also, there are plenty of programs that need to "o things differently". X11 is not just a desktop window system, it's used for scientific and engineering applications, customer terminals, ATMs, banking workstations, embedded systems, and lots of other applications. Those environments should not look like a regular desktop.

3. Easier configuration. It can be a real bitch to get xwindows running properly. Considering the huge amount of differing hardware in the wild, I'm not so sure it would be possible to simplify it too much. Oh, well.

I think people are doing as well as they can, given limited information from manufacturers.

But because X11 is standardized, you can always buy a commercially supported X11 server. Those usually run very well on the latest hardware. If you are using XFree86, you are using something that's both free and experimental.

As far as I can tell, "the split" is over none of these issues. Both branches will remain network transparent window systems, they will remain compatible, and they will continue not to force toolkits or desktop software on users. If they tried to, they would cease being X11 implementations. What Keith probably will do is accelerate bug fixing and bringing extensions into the X11 server. And that's what really matters.

Re:the usual misconceptions

[spitzak]

The horrid resizing behavior is due to the seperate window manager. There is absolutley no way to get a smooth update when two competing programs are resizing different parts of the display and there is no protocol method to say "don't draw this until this other thing happens".

The solution is to put the window manager into the toolkits like the buttons and everything else is. The result will be *better* than Windows, as Windows puts it in the server so there still is communication and it is impossible to do complex restrictions on the size of a window (such as a range of ratios or multiples of certain sizes) Windows also it relies of sending an event through the user program to synchronize the window changing size and the drawing, otherwise it would look as bad as X, but I don't recommend this route at all, just put the window borders into the user program.

The problem is that a thousand sheep here are going to bleat "but that will make it 'inconsistent' and it will 'confuse the user'". This naive response from so many is probably the most serious problem we are going to have in trying to fix X. Truth is: NOBODY is "confused" because the buttons are different colors, they are confused by crap interfaces! And the great Windows DOES NOT enforce lots of things (such as what shortcuts are used for menus) that people keep complaining about when they say that "windows is consistent and Linux is not". The truth is that "consistency" is the responsibility of the application programmers and trying to force it by making complex and slow interfaces so your favorite GUI method is forced on everybody is absolutely the WORST thing you can do.

Dragging windows (not resizing) does not have problems with the seperate window manager, so it is obvious that a lot of X programs do not respond to redraws very quickly. Some Windows programs have this problem too, but I would agree that not as many as Linux. Both X and GDI32 drawing engines suck almost equally badly so the amount of code in the app is about the same, and tests where lots of letters or rectangles are drawn indicate that GDI32 and Linux X are about equal speed. I suspect this is a combination of excessively complex toolkit programming and perhaps some basic failures of X such as an inability to deliver or respond to expose events quickly enough.

Re:the usual misconceptions

[be-fan]

Um, they were testing network communication here. They found out that network performance was less than optimal because of various reasons. It had nothing about local cases.

Re:the usual misconceptions

[be-fan]

You can't. There are problems somewhere in the chain, but it's not in X. Far more common is poorly optimized applications. For example, right now I'm using KDE. Certain applications, even very widget-heavy, complex applications like Qt designer, resize very smoothly. For example, on my GeForce4Go, copying a 500x500 pixmap to a window (x11perf -copypixwin500) can be done at 2300 fps, or about 575 megapixels per second. For a general purpose display API, that's really fast, about 1/3 the total memory bandwidth of the graphics accelerator. But when resizing KDE Kontrol Panel, the image, not much bigger than 500x500, redraws at maybe 4 or 5 fps. The problem is not the raw speed of X, but how the application responds to resize requests. A great many X apps have the problem that they handle resize events very poorly, so when users who are just trying to Linux for a short-while try to test speed (by doing the usual resize window-quickly "test") they get a very misleading impression. Meanwhile, users who use the system all day (and almost never resize windows in that manner) don't notice stuff like that and wonder why anyone things it's slow. Another example. When I open a menu in Konqueror, and start the settings panel, the menu will sometimes go away and the area underneath won't redraw until almost a second later. The redraw is the matter of a single bit-blit. We've already established that X can do those really quickly. The problem is not that X isn't fast enough, but the application is trying to load the settings panel (waiting on the disk usually) and isn't responding to redraw requests in the meantime. There are ways to fix these problems:

1) Add some synchronization between apps and the window manager, so they resize smoothly. OS X has something like this: the window frame won't resize faster than the window contents can redraw. Since Quartz in general is little slow (because all drawing, even in QE is done in software), this leads to very jerky behavior, but is much more "elegant."
2) Multithread apps. This is the big one. A properly multithreaded GUI can respond to high-priority user events without being blocked on low-priority I/O events. It was only recently that Linux got threadsafe GUI libraries (Qt 3.x and Gtk+ 2.0) and large parts of KDE still aren't threadsafe.
3) Optimize drawing routines. Take Konqueror vs Internet Explorer. Try doing the "resize" test on the two. Each frame of resize animation is very time consuming. Thanks to CSS and other modern HTML, the *entire* page has to be laid out each frame. The algorithms for doing this in IE are much more mature than the algorithms for doing the same in Konqueror.

Re:the usual misconceptions

[Anonymous Coward]

Yep, problem is with the kernel not x window, (they are using x window and xmms to test these changes because se the effect is immediately visible, audible). The realisation is that there are two types of prioreties that needs to be taken into account, niceing x, xmms, is just an old kludge as it's often not more processing power that is needed but faster access to it (to prevent lag, stutter).

Re:the usual misconceptions

[Anonymous Coward]

The solution is to put the window manager into the toolkits like the buttons and everything else is.

No, the solution is to put the window manager in the X server, and have XAA store a front buffer of each surface in video memory like every other sane display system does. Context switching is eliminated, and windows won't blow away their contents and force the client application to update them every time they're unobscured.

X sends unnecessary expose events to applications and, as a result, can't properly handle opaque move or resize. The 'BackingStore' option caches in shared memory, passing through system memory and the CPU. System memory to video memory copies are slow. Video memory to video memory copies are fast. In addition, the backing store is inconsistently applied - only to obscured windows, not offscreen windows.

Look at DirectFB for a sane design (though not yet bug-free implementation) of an accelerated framebuffer. The XFree people need to fork the display layer code off to some people who actually know how to write one, and concentrate on the X protocol layer and core.

Then again, I'm not managing them, so they're free to do whatever they want to ensure that XFree stays unresponsive on giant 64/128MB video cards.

Re:the usual misconceptions

[g4dget]

Latency in dragging windows can be handled by a call that attaches a window to the cursor so it is dragged around until the mouse is released.

We are not talking about opaque moves--current windowing systems, including X11, handle those just fine by bit-blitting.

I can't think of any good equivalent solutions for resizing,

That is because there isn't any. In the simplest case, which you are thinking about, the user decides how big the window is going to be, by using the mouse. The GUI communicates the new size to the application, the application sends back the updated appearance of the window. One round trip.

I think it should be left up to the program and there will be latency. However as the program can say "resize the window to this" and immediately follow it asynchronously with the new image for the resized window, there will be much less latency than the current version where the window is resized by the window manager and there has to be at least a round trip to the program to get the new drawing.

That doesn't help. The user determines how big the window is going to be and the window has to follow. The application can't just pick a size preemptively and send out redrawing commands for that--if you can see the latency now during redraws, you would see the latency then because the window size wouldn't follow your mouse.

Furthermore, it's a fundamental policy decision under X11 that applications don't get to say "resize to this". That has less to do with usability, and more with being able to use the same programs on a very wide variety of displays: programs simply don't know about all the possible displays and environments to be able to do window management correctly. Putting window management into clients is fundamentally broken, and it would be a huge step backwards for X11 to do that without helping with anything (in fact, I pretty much guarantee it's not going to happen).

Of course, you are free to try this under X11: if you like, your toolkit can resize its own window. You are even free to override the window manager altogether. So, the functionality is already all there, but thankfully, no major toolkit uses it. Just don't expect people to use your application a whole lot if you do that. The few applications that try this (xmms, etc.) are a big pain to use on displays that don't satisfy the assumptions that their author had in mind.

The drawing code definately should support display lists, though I don't think much of the complex structure is needed, because *some* communication is acceptable. A program that draws a hundred buttons of different sizes should be able to send a display list that can draw a button once (keeping it in the server much like an X pixmap) and then send on redraw it just has to send the rectangles and labels of all the buttons and the command to run the display list on each of them.

Well, as I was saying, X11 is getting support for that. That's nice for all sorts of reasons, but it will not solve the problem in general because once you have a round trip, you have latency, even if you only send a little data. What it will do is let the server generate a resized image that's pretty close for normal mouse movements, until the application gets around to redrawing things. It will give the illusion of smooth opaque resizing.

But, then, the server could probably already do that with bitmaps: you resize the window and the server does smooth bitmap scaling to fill in until the update comes from the client. For the 200ms that's going to be on the screen, it's going to look just fine.

But let's keep our perspective here: GUI toolkits are not written for letting animated widgets dance around the screen in real time, they are optimized for mostly static displays. That's a deliberate design tradeoff, and one that meets day-to-day needs well. The only place where that kind of animation occurs during normal usage is with opaque resizing. I don't think it's worth putting a lo

Re:What are their priorities?

[rsidd]

how often do you need remote xwindows

Every day.

Absolutely. People who don't need it everyday are people who only use one computer (eg, home users with only one machine) or people who never realized how easy it is to run a program on another machine and display it on your desktop. Remove this ability, and you remove a huge reason for using unix/linux on the desktop in the first place.

Re:So, what now?

[Anonymous Coward]

Sorry, let me correct that. That should read "There is no good reason to not open source drivers". I suppose being a total idiot is a reason, too.

Re:xwin- Quartz

[FuegoFuerte]

Remote X is not as widely used as it is endlessly hyped to be.

Excuse me? I use it all the time. And that's just at home. Using my laptop for something? Pop up a display from my main box on my laptop. Makes things like keeping email synced so much simpler. Just use the same installation of the same browser. Forward X over SSH. Do all sorts of crazy and wacky things windows users can only dream about. Yes, the networked aspect of X is important. VERY important I'd say. If it weren't, why would Microsoft be trying to catch up to it? (RDP, anyone?) Yes, X has some issues, but the remote feature is one thing that absolutely should NOT go away.

Re:xwin- Quartz

[be-fan]

Why the fuck does everyone thing that the remote capability adds overhead to X? X communicates via UNIX domain sockets (very fast in Linux) over a local connection. In Windows, GDI.dll communicates with the kernel via LPC (lightweight procedure calls, another form of IPC). For any of these mechanisms, shunting IPC communication to a remote connection is trivial and has no performance impact in the general case. Hell, even with COM, something much more low-level (which is based on C++ virtual function calls) network transparency has no performance hit in the local case.

Re:Effects on Gnome, KDE, Window Managers?

[edgecrusher]

actually when a toolkit creates a window it offers simple "hints" that the window manager picks up and manipulates.. there is very little inter-process communication between apps and the WM. So it would be rather pointless to force a TK to produce the window borders - which would also create problems with continuity between windows - GTK window decorations would probably look much different to KDE ones to any other TK out there.

As for rewriting... I don't know about QT, but GTK+ runs on many levels, at the bottom level is a graphics independat layer that can run on anything from Win32 to the Framebuffer to X to embedded drivers. All the widget elements on top don't notice, nor care.

X11 is built to be modular like this - although the current Xfree86 isn't quite (in fact it's actually very monolithic), which is why keithp want's to fork and make a more modular implementation of X11.

Re:Effects on Gnome, KDE, Window Managers?

[be-fan]

Check out their statement:

A sizeable group of developers from the two leading free software
projects developing desktops based on the X Window System, KDE and
GNOME, have been discussing the current situation among themselves
and decided to draft and release this document.

We acknowledge the dedication of the XFree86 project in providing us a
free and innovative implementation of the X11 industry standard,
something we benefit from on a daily basis. Therefore, we want to
share our joint point of view with the community.

1. XFree86's recent technical progress, culminating in the 4.3
release, brought significant advancements to the X desktop. Prior
X Window System implementations were lagging behind the needs of
modern desktop users.

Cursor theming, simplified font configuration, dynamic screen
resizing, and so on address long-overdue usability issues with X
desktops. XFree86's robust solutions in these areas have been
invaluable.

However, the work is not done. Our goal is to provide the
community with desktop systems far beyond what anyone offers
today. We are ready to take advantage of an X Window System
implementation that continues to innovate.

2. GNOME and KDE have two interests in X:

- We would like to have a single organization where X innovation
occurs. By innovation, we mean the definition of new APIs,
specifications, and features - new additions to the foundations
that KDE and GNOME rely on.

- We would like to have a frequently-released, robust, stable,
open source implementation of these APIs, specifications, and
features.

We are explicitly distinguishing innovation from implementation,
because standards should be adequate to allow multiple
fully-interoperable implementations.

Within the development organization responsible for defining and
crafting new features to be adopted as standards, innovation
should happen in the open, with all affected parties able to
participate early in the process.

3. We do not want to take sides on the recent political wrangling of
who did what when and who should be in charge. Our hope is that as
a community we can find a way to involve everyone in X's
development and move forward with solving technical challenges.

4. It makes sense to us if the organization responsible for X
innovation also develops the most widely used open source
reference implementation. This ensures an emphasis on working
code, and provides a pool of active technical expertise.

5. We would like to see this forum work toward a unified
organization, governed by active contributors, that implements,
deploys, and standardizes new X innovations.

We do not want to take an a priori position on how this
organization should be organized or governed - that is a
conversation we're trying to start, rather than one we're trying
to end. We trust and will support the X community as they work to
address this issue. :: signatures clipped ::

Other important news

[MrNop]

Carsten Haitzler (The Rasterman) from Enlightenment project started to work on XCB !
XCB seems to be the super fast replacement of Xlib keeping X protocol ...

More infos

Re:Remote CANNOT be an "option"

[Anonymous Coward]

it would still be silly, compressing images and sending them to render text on the other end, you really think that would be performance increase?

but like said, the current 'remote capability' is free (as in performance) and nothing to be bitching about.. problem with little knoweledge is that it is little and ends up in wrong conclusions, as is the 'knoweledge' that x is bloat because you see so much things loading.. would hiding those behind nice loading screen really help anything like in the ms world.

in windows people think the os is great if software handles redraw requests fast.. and when the software doesn't it's the softwares problem.

why people think that in x it's x's fault always?

you can have support for things without it being bloat.

Re:So, what now?

[Catiline]

The reality of doing business for these vendors dictates [closing driver source code], however.

Quoting from Eric S. Raymond's essay "The Magic Cauldron":

If you are a hardware vendor, you may fear that open-sourcing may reveal important things about how your hardware operates that competitiors could copy, thus gaining an unfair competitive advantage. Back in the days of three- to five-year product cycles this was a valid argument. Today, the time your competitor's engineers would need to spend copying and understanding is a damagingly large portion of the product cycle, time they are not spending innovating or differentiating their own product.

This is not a new insight. ...[skip two examples]... Acceleration to Internet time makes this effect bite harder. If you are really ahead of the game, plagarism is a trap you want you competitors to fall into!

In any case, these details don't stay hidden for long these days. Hardware drivers are not like operating systems or applications; they're small, easy to disassemble, and easy to clone. Even teenage novice programmers can do this -- and frequently do.

There is no practical business reason for closed source drivers: it is a false sense of security in protecting the innovations that really don't have (or need!) much protection nowadays. Furthermore, since you have already achieved the full value of R&D money spent (by developing your product), there is no "loss" (as you put it, "giving their R&D budget to the competition") to account for. Indeed, in this view releasing drivers under a "mandatory sharing" license such as the GPL multiplies the value of said research as it allows you to gain more free (zero cost) research as those who would copy your R&D must release their alterations to your code.