Microsoft's CoApp To Help OSS Development, Deployment

badpazzword writes "Microsoft employee Garrett Serack announces he has received the green light to work full time on CoApp, an .msi-based package management system aiming to bring a wholly native toolchain for OSS development and deployment. This will hopefully bring more open source software on Windows, which will bring OSS to more users, testers and developers. Serack is following the comments at Ars Technica, so he might also follow them here. The launchpad project is already up."

Comments

I'll follow them here too. :D

[His name cannot be s]

Ask me about CoApp, I'll tell ya everything ya wanna know.

Garrett Serack
CoApp Project Owner

Re:I'll follow them here too. :D

[Meshach]

In the same vein feel free to ask me about Linux.

Linus Torvalds
Linux Kernel Founder

Re:I'll follow them here too. :D

[tomhudson]

Ask us about nuclear disarmament.
We'll tell you everything WE want you to know.

Barack Obama
Dmitry Medvedev

Re:I'll follow them here too. :D

[His name cannot be s]

Well, considering that I spent several months hacking thru red tape to get VP approval, and the enthusiasm that I've been getting, I'm pretty damn confident that we're clear sailing.

And given the first three targets that on my radar are PHP, Apache and Python (and the 40 or so shared library dependencies), and that's what I took to the VP, I'm fairly confident that's not going to be an issue.

And, on top of that, MS doesn't own the project, I do. "Shutting it down" is not an option for them.

Re:I'll follow them here too. :D

[DoofusOfDeath]

Ask me about CoApp, I'll tell ya everything ya wanna know.

How do I know that MS won't file a software patent related to this work?

Re:I'll follow them here too. :D

[vux984]

You do know that the work agreement that you signed during orientation stated that Microsoft owns any software that you produce on your own time, as long as Microsoft may compete against said software at some point in the future?

Perhaps that was some of the red tape that needed to be cut. Guess what, you can run things past management, and get legal to sign off on something that amounts to an agreement between the employee and employer that a given project belongs soley to the the employee. I don't know about microsoft specifically, but lots of companies are amenable to this sort of thing.

Sometimes there are legitimate concerns that have to be resolved... often its just a matter of jumping through the required hoops.

Re:I'll follow them here too. :D

[causality]

Ask me about CoApp, I'll tell ya everything ya wanna know.

Garrett Serack CoApp Project Owner

I'll bite. Given Microsoft's track record, particularly its embrace-and-extend tactics, its questionable business practices, its status as a convicted monopolist, its use of vendor lock-in, its related use of proprietary file formats, and the Halloween e-mails from top management clearly defining Open Source as an enemy, I have just one question: why should we trust them?

Most (nearly all) of the upper management people who arranged everything I just listed are still working at Microsoft.

"Fool me once, shame on you; fool me twice, shame on me."

Re:I'll follow them here too. :D

[Cryacin]

I'm Santa.

Ask me about anything to do with snow, the north pole, midgets, flying reindeer and flying midgets.

Re:I'll follow them here too. :D

[sortius_nod]

It all feels too much like a dirty beat up van in a shopping mall parking lot with "free candy" painted on the side...

Re:I'll follow them here too. :D

[Anonymous Coward]

Curiously, given microsoft's recent wrist slappings by the EU, fostering the development of "Competing" products could help microsoft, rather than harm it;

Take for instance, the recent "Browser choice" screen. If Microsoft had been fostering a package downloader at that point in time, then they would have not needed to do anything to comply with the EU. Their OS would already have IE by default, and "Offer" a nice little package handler for those "Other Browsers".

If the EU were to press, and try to stick MS with the stigma of not actually wanting any other browsers to run on their OS, by making users use a round-about way of getting their browsers of choice, MS could point the finger right back at how much capitol they invested into the alternative software ecosystem, and how they leveraged their power to help bring FOSS and the package manager to their OS.

In short, creating a package manager like this is a good way for MS to be more two-faced than ever.

Not that I am gonna complain; ALL corporations are two-faced, and a well supported package manager, and better acceptance of the win32 platform (Not just windows, there are attempts at FOSS Win32 platforms.) by the FOSS community is a good thing all around.

I just dont think MS is overly concerned that it will compete with their software ecosystem at this point, and is more convinced that government regulators are the bigger threat.

Re:I'll follow them here too. :D

[grcumb]

Ask me about CoApp, I'll tell ya everything ya wanna know.

Garrett Serack CoApp Project Owner

Okay, serious questions:

Assuming that you've looked at APT and similar packaging tools, and given that you're still convinced that there's a 'Windows Way' (your term) to handle deployment that differs from Linux best practices, how do you plan to address:

• Package Repositories - This is one of the main strengths of Debian and related distros. Do you think it's even possible to replicate this level of community control in Windows? I know you've mentioned decentralisation, but have you considered the implications of such an approach? What is the cost of failure to affect consistent, formalised management of package builds?
• Dependancy Management - This issue is largely done and dusted on Linux, but remains a dog's breakfast on Windows (albeit not as frustrating today as it was in the mid-90s). In the absence of centralised repositories and the Unix toolchain philosophy, how do you propose to cope better with dependancies?
• File locations - How do you propose to manage the proper placement of libraries etc. when the conventions concerning where to put such files are not nearly as well defined on Windows? I'm suggesting here that you need cultural leverage rather than technical answers. You need to change perceptions, not toolkits.
• Security - Do you think it's even possible to replicate one of the main strengths of Linux package repositories: the ability to curtail security risks such as malware and flawed code?
• Scripting Interfaces - Say what you like about make and other command-line utilities, but as a busy sysadmin, I consider GUI package management a waste of my valuable time. If I'm going to deploy regular security updates, for example, I want to know that I can script every aspect of the operation. Even the tab-completion features in aptitude make it many times more efficient than a point-and-click interface. What is the potential for scripted deployment/management of packages under your system? Why?

I guess it's clear by now that I'm suggesting that what Windows needs is not another new way to do things. Package management in Debian, for example, is vastly more advanced and sophisticated than anything on Windows, and yet you feel the need to do things the 'Windows Way'. Don't you think you'd be better off learning from others who have been dealing successfully with package management for over a decade now?

These are all serious questions and I expect to be challenged by your replies. I applaud your courage in taking on this huge task. I also think that you're going to need to learn a lot more humility than you've demonstrated so far if you want to achieve something better than a new brand of anarchy in packaging.

Re:I'll follow them here too. :D

[martin-boundary]

I'm Batman.

Uh. That's it.

Re:I'll follow them here too. :D

[His name cannot be s]

That is precisely the red tape that I had cut.

Microsoft has given me a signed contract that says that whatever I produce for the CoApp project isn't owned by them. They do get a license to everything I make (fair deal), but they don't own it in the end.

That, and I've also chosen the BSD license for it's do-what-the-f*-you-want spirit.

Re:I'll follow them here too. :D

[His name cannot be s]

No.

My intent is to completely do away with the practice of everybody shipping every damn shared library. It's one of the things that piss me off the most. I've got a very workable solution that uses WinSxS to cleanly handle this.

It is extremely important that there is a unified method for sharing libraries between apps.

Re:I'll follow them here too. :D

[His name cannot be s]

Is this some kind of back-handed comment based on the general view at Microsoft about Open-source software, or the general view that MS would like to push out to userland? That people should use MS OSS because you need to be a developer to use it on other platforms?

No-no.. exactly the opposite

Have you tried to roll out some OSS apps on Windows?

On Linux it's two clicks, and BAM! Done.

On Windows, it's almost never that easy to setup OSS apps.

The problem I see is that it doesn't take a Developer on Linux to get Apache installed and configured. Why should it on Windows?

Re:I'll follow them here too. :D

[His name cannot be s]

If it does, so be it.

I've spent the last couple of years at Microsoft working to make PHP better on Windows, and validating PHP apps including CMS systems like Drupal on Windows. Seems to me they want some competition.

Re:I'll follow them here too. :D

[His name cannot be s]

Assuming that you've looked at APT and similar packaging tools, and given that you're still convinced that there's a 'Windows Way' (your term) to handle deployment that differs from Linux best practices, how do you plan to address:

Yes, I've worked with APT and RPM for a very very long time now. The reason I'm convinced there is a 'Windows way' is because it's a different system that Linux; yes, I've learned a lot about PMS from Linux, and I know how to apply that knowledge to Windows.

Package Repositories - This is one of the main strengths of Debian and related distros. Do you think it's even possible to replicate this level of community control in Windows? I know you've mentioned decentralisation, but have you considered the implications of such an approach? What is the cost of failure to affect consistent, formalised management of package builds?

I have a plan for allowing any publisher to publish packages in the CoApp ecosystem, provided they meet two qualifications:
- They must be able to host their repository meta-data on an SSL protected connection.
- All packages must be digitally signed with a certificate that chains back to to a commonly-accepted CA.

Dependancy Management - This issue is largely done and dusted on Linux, but remains a dog's breakfast on Windows (albeit not as frustrating today as it was in the mid-90s). In the absence of centralised repositories and the Unix toolchain philosophy, how do you propose to cope better with dependancies?

I'm working with the developer of WiX to ensure that we can trivially build chained MSI packages that have the necessary smarts to properly manage this. Kind-of mixing in something like ldconfig with the Windows SxS library management.

File locations - How do you propose to manage the proper placement of libraries etc. when the conventions concerning where to put such files are not nearly as well defined on Windows? I'm suggesting here that you need cultural leverage rather than technical answers. You need to change perceptions, not toolkits.

Yes. The change starts with PHP, Apache, and Python, and the 40+ packages needed to build them (community members from each are already on board) Half of the project is setting some intelligent standards, and then bootstrapping the ecosystem with packages to enable other software to follow.

Security - Do you think it's even possible to replicate one of the main strengths of Linux package repositories: the ability to curtail security risks such as malware and flawed code?

Yes. By requiring code-signing (and I've got a plan for opening that up without cost for smaller projects) we can replicate the benefits of MD5 and PGP signatures found in the Linux world.

Scripting Interfaces - Say what you like about make and other command-line utilities, but as a busy sysadmin, I consider GUI package management a waste of my valuable time. If I'm going to deploy regular security updates, for example, I want to know that I can script every aspect of the operation. Even the tab-completion features in aptitude make it many times more efficient than a point-and-click interface. What is the potential for scripted deployment/management of packages under your system? Why?

I agree 100%. Scripting interfaces are an absolute requirement, and will likely come well before the GUI.

Think of it as a clean adaptation of the same concepts to the model that will be attractive to Windows developers.

I also think that you're going to need to learn a lot more humility than you've demonstrated so far if you want to achieve something better than a new brand of anarchy in packaging.

I apologize if I'm coming off arrogant. Frankly it's taken an extremely long time to convince the powers-that-be at Microsoft that Linux's package management is stellar compared to Windows. It's also not near as hard or large as it sounds, I'm walking on the shoulders of giants here, both in the Linux and Windows worlds.

Re:I'll follow them here too. :D

[toastar]

At least you can diff a config file. Try that with a gui.

I do that that all the time, screenshot + OCR only takes me 30 secs to pop the data into excel.

office/windows is a pretty stable work platform, Linux I'm always spending 15 minutes getting a screenshot app working.

Re:I'll follow them here too. :D

[Compholio]

Ask me about Grim Fandango.

Re:I'll follow them here too. :D

[His name cannot be s]

As for the first five points, yes I'm aware of all of that, and I'm working to solve all of them. Some of them are not possible (mixing compilers has a lot of bad mojo) and some are solvable with some really good best practices.

1/ Microsoft are stopping using WinSxS assemblies for managing the C/C++ runtimes as it is complex to manage and get right;

Ah, Visual Studio is backing away from WinSxS. I read their justification. I didn't buy into it. I think it's a solvable issue.

2/ With XP, Microsoft were selling WinSxS as being able to deploy different versions of the binaries, but for Vista/Win7 they are now saying that WinSxS is for archival purposes (see the Engineering 7 blog)

Uh, what? I've been talking to the maintainer of the WinSxS system. He's fully supportive of my plans.

3/ It does not really work as intended in practice -- e.g. comctl32 version 6 is different in Vista/Win7 than in XP, yet the applications that reference the XP version use the Vista/7 version

It works just fine, as long as you use it correctly; if they didn't, it's not my fault. Some of the tools I'm building will make it easier not to screw up.

Re:I'll follow them here too. :D

[Cryacin]

FROM: Santa's Workshop helpdesk
TO: Virignia

We have finished the research on flying midgets. We are moving on to flying goats next, which should let us finally allow pigs to fly.

We thank you for your patience whilst we satisfy your mother's prerequisite requirements on when you can have a pony.

Re:I'll follow them here too. :D

[h4rr4r]

By killer you mean those that admin it wish to kill themselves right?

The thing even lacks the ability to use smtp auth.

Re:I'll follow them here too. :D

[Animaether]

My intent is to completely do away with the practice of everybody shipping every damn shared library.

If you only succeed in getting windows folks to learn this lesson you should be made a saint.

The major problem with this is that, as mentioned, Windows doesn't have a package manager, and Microsoft keeps telling developers that they cannot expect a user to have internet connectivity.

So when you compiled your application with Visual Studio 2008 SP1 with the ATL update installed - which means every user of your software will have to have the Visual C++ 2008 SP1 ATL runtime redistributable package installed as well, you're left with scant few options.

The most reasonable of which are:
A. If you're distributing something boxed, to include the redistributable package on the media (CD/DVD/USB stick/whatever).

B. If you're distributing something via downloads:
B.a. Include it because - again - you're not supposed to assume the user will have connectivity.
B.b. Don't include it, but detect whether the user has it installed and has internet access, and then offer to download it and install it (silently or otherwise).

Of course for option B.b., Microsoft further seems to suggest that you do not link to -their- download pages (after all, the URLs could change, etc.) but instead host the binaries yourself.

The only reason, thus, that Windows developers tend to include or download shared libraries at runtime, is simply because there -isn't- a package manager for Windows.

So don't blame the developers - blame the lack of a package manager. Which I fully welcomed the last time a topic hinting at a package manager popped up on /.
Unfortunately it seems like they would be two rather separate projects?
http://it.slashdot.org/story/10/03/24/189248/Microsoft-To-Distribute-Third-Party-Patches

Re:I'll follow them here too. :D

[TubeSteak]

I'm Alfred the Butler.
Would you like a cup of tea Sirs?

Re:I'll follow them here too. :D

[His name cannot be s]

think you had no choice to choose the BSD license instead of the GPL. Had you chosen GPL, it is likely the project would have been immediately rejected by Microsoft.

That's not true actually.

I didn't tell anyone what license I was going to use until a few days ago, by which time they'd already signed the agreement.

In addition to that; as a Microsoft employee for Microsoft, I've contributed code to GPL, LGPL, BSD, PHP and Apache licensed projects.

Re:I'll follow them here too. :D

[His name cannot be s]

I do have one question. Why, exactly, do you think that this sort of approach is likely to be easier than doing what Apple did and simply exposing a Posix API that is actually useful?

Because, even if we could get a great POSIX experience on Windows, it leaves out Windows developers.

One of my goals is to get Windows developers in the OSS game.

On top of that, there is a hell of a lot of non-POSIX open source software on Windows that needs fixing too.

Look at it this way: Would you respect someone who told you the best way to get FireFox running on Linux was to use some sort of Windows emulation layer... Like WINE? no, because FireFox *can* compile for Linux. Same thing with nearly all Open Source I encounter. I want to get the OSS quality and experience on Windows to exceed commercial developers... it needs the most love.

Like I tell people:
Working as an open source software developer at Microsoft is like being a preacher in Vegas. I figure I'm in the single most important place in the universe that I can be.

Re:I'll follow them here too. :D

[styrotech]

As an admin that maintains both Linux and Windows systems, this sounds really cool. Hopefully the guys writing the Tomcat AJP connectors for IIS will use it (that stuff can be a nightmare).

To me though the initial setup is never the main problem (except with AJP/IIS hehe), it's the ongoing maintenance and patching of 3rd party stuff that suffers the most on Windows.

Sure Windows Update / WSUS make all the MS stuff easy, but 3rd party Windows apps are a nightmare to keep up to date network wide. They all have their own separate update mechanisms that mostly require an admin being logged on to work.

I've love to see Windows Update and WSUS allow 3rd party repos (eg the equivalent of adding stuff to /etc/apt/sources.list) so that practically everything could be patched via Windows Update / WSUS without admin intervention on each machine.

I don't know if your work will end up tackling all that, or one day get incorporated by the existing patch mechanisms, but I can still dream :)

Best of luck anyway.

How Long Before ...

[WrongSizeGlass]

... MS pulls the plug on this and leaves OSS developers hanging high and dry? Or worse, pulls some slight of hand with licensing, copyrights or patents and forces OSS dev's to stop in their tracks waiting for MS's next move?

Re:How Long Before ...

[ChrisMounce]

Maybe they recognize that there's a ton of open source software that people really want to use, and that easy installs of OSS on Windows adds value to Windows.

Like how they contributed some Linux stuff a while back to make it easier to run Linux in a VM... with Windows as the host machine (I'm not clear on the details, so I'm probably getting the terminology wrong).

Re:How Long Before ...

[aBaldrich]

Do you know what was the first thing I thought when reading the article and the project's launchpad? Halloween Document II. Why does Microsoft need to support Free Software? I mean, they claim to have everything they need, their new shiny should 7 have it all.
This article's summary should be something along the lines of: Microsoft embraces OSS. How long before they extend their dirty tactics to OS? I don't trust them.
The jewels of OSS were built because we, developers, needed them. We needed an OS and made Gnu, and Linux. We needed a web server and made Apache. We needed a GUI and made GNOME, KDE and Compiz. We needed a web browser and made Firefox.
Now Microsoft needs package managing software, and they really believe we are going to build it for them. Laughable.

Re:wholly native toolchain

[Dragoniz3r]

Is there something wrong with that?

From TFA:

• Place binaries, libraries and header files in a logical and consistent location
• Facilitate sharing of components and allow multiple projects to easily both participate and consume them
• Allow for upgrades and patching of both libraries and applications
• Be Windows developer friendly. No forcing of building using ‘make’, but rather taking advantage of the nifty IDEs we already have

Clearly these are horrible goals and this should by no means be done, simply because he is planning to do it in Windows.

Re:Why only open source?

[His name cannot be s]

Why limit this to open source? It would be great if the users could update every program easily and painlessly, at least the ones that use this new system.

I'm Busted. It isn't really restricted to Open Source... but that's my mission. Commercial apps will be able to play just fine in this ecosystem.

I am assuming that this system will allow easy and painless upgrading like on most Linux systems. Is that true? Will it have automatic dependency handling and command line installation?

Yes. Painless and automatic dependency handling, and yes command line tools. You are singing the chorus to my theme song!

Re:Why only open source?

[His name cannot be s]

I second the question about limiting to open source. A good package management system that can could make using SxS painless would be awesome in an enterprise environment.

I agree. it ain't really limited to Open Source

Since this is open source and .msi based I assume you will be leveraging WiX somehow?

Yes indeed. The author of WiX is on the mailing list, and a personal friend. He's very excited about all this too.

I hope this isn't going to be a big collection merge modules with duplicated component guids..

Nope. I don't believe in merge modules. I believe in a system that works.

Re:wholly native toolchain

[His name cannot be s]

All but the last one are fine. I have some windows boxes I have to deal with and I sure as hell do not want to be stuck using some GUI IDE just to build the latest $foobar.

Use of the GUI ain't mandatory... it's just that in order to get Windows devs on board, it'll have to have one.

The core bits will all be able to be command-line driven.

Re:Microsoft has been surprising me lately

[phantomfive]

I can't help but think you are incredibly naive. Have you been paying attention to how Microsoft is suing third party controller makers? Or how they are subverting their own standards? Or if you are in Europe, how they are heavily lobbying your representatives to hurt open standards? If you live in the US, don't worry, they are working to influence your representatives too. Not to mention they stand firmly committed to helping out a truly evil empire (yeah, saying 'evil' is a bit much but a government that censors political speech and has secret trials for people they don't like isn't exactly nice).

I mean, this is just in 2010. You shouldn't have particular love for any company, but claiming that Microsoft has changed can only be done by ignoring the facts and reality. Don't do that.

Re:wholly native toolchain

[h4rr4r]

This is the sort of thing that makes the FSM smile upon you and shall earn you an afterlife spent near the beer volcano and stripper factory.

Let me right the first line of code for you.

[codepunk]

I am feeling generous this evening and decided I would donate the first line of code to this
fine project. I relinquish all copyrights on the following line of code, feel free to do with
it as you wish.

#include "ie6.h"

Re:"bring OSS more users"

[im_thatoneguy]

And more *windows* users, more windows license, more vendor lockin, and fewer alternative OS's...

Until Windows users realize that all their favorite apps run great on Linux as well as Windows.

If chrome runs on Windows and Linux and you just use Chrome most of the day then it becomes trivial to switch over to Linux since your app will look relatively familiar. The largest obstacle to Linux adoption besides its contempt for its users is the lack of applications people are familiar with. If someone got used to pidgen then they would be less likely to revolt when they tried using Linux.