OneGet has the notion of 'trusted' repositories. We're likely to expand this concept a bit in the future, but for now, that's what it is.
Built-in package sources from reputable sources may be marked as 'trusted' by default, but the majority of sources should be 'untrusted' until the user makes that change.
The real trick is getting package provider plugins to tell OneGet the truth if a repository is trusted or not.
I suspect that we're going to have to introduce a level of trust with the package providers too, and expose this to the user... somehow.
We're tossing around some notions about different factors that make a 'package' or 'repository' trustworthy.
I'm sure we can do some stuff with signed repositories and signed packages to detect when things 'change' and/or keep unsigned repositories 'untrusted'.
Really, our first target for this stuff is developers and admins, not my mom...
Actually, to be perfectly clear, OneGet isn't really a package manager.
It's a package-manager-manager -- It's a unified way of installing packages of software regardless of the how-it's-implemented-on-the-back-end.
The first real package provider plugin is a Chocolatey one. Why re-invent the wheel when the wheel already works?
The purpose here is to leverage all these different sources of software using a common set of commands and APIs.
Anything that can be represented as a 'source' of software can be plugged in on the back end. I'm aiming for plugins for NPM, Ruby Gems, Python, on top of the expected MSI, Chocolatey, NuGet, etc...
Plugins can be written by anyone, and I'm going to great lengths to make it as simple as possible -- it's about ~15 or so functions to implement and we can plug in virtually any package format or service into OneGet.
[FYI -- I'm @FearTheCowboy everywhere else, my/. id is so old that my name got trimmed from "His Name Cannot Be Spoken" 15ish years ago when they did a database adjustment... ]
I have had thoughts on how to do this; I suspect that while we may not set up a repo to do that, I may hack out the instructions on how that could be done easily if one wanted to maintain their own.
It really boils down to how much time I can throw at that.
Of course, we also want it to plug into WU and WSUS, but that'll be a bit more down the road.
I do have one question. Why, exactly, do you think that this sort of approach is likely to be easier than doing what Apple did and simply exposing a Posix API that is actually useful?
Because, even if we could get a great POSIX experience on Windows, it leaves out Windows developers.
One of my goals is to get Windows developers in the OSS game.
On top of that, there is a hell of a lot of non-POSIX open source software on Windows that needs fixing too.
Look at it this way: Would you respect someone who told you the best way to get FireFox running on Linux was to use some sort of Windows emulation layer... Like WINE? no, because FireFox *can* compile for Linux. Same thing with nearly all Open Source I encounter. I want to get the OSS quality and experience on Windows to exceed commercial developers... it needs the most love.
Like I tell people: Working as an open source software developer at Microsoft is like being a preacher in Vegas. I figure I'm in the single most important place in the universe that I can be.
think you had no choice to choose the BSD license instead of the GPL. Had you chosen GPL, it is likely the project would have been immediately rejected by Microsoft.
That's not true actually.
I didn't tell anyone what license I was going to use until a few days ago, by which time they'd already signed the agreement.
In addition to that; as a Microsoft employee for Microsoft, I've contributed code to GPL, LGPL, BSD, PHP and Apache licensed projects.
As for the first five points, yes I'm aware of all of that, and I'm working to solve all of them. Some of them are not possible (mixing compilers has a lot of bad mojo) and some are solvable with some really good best practices.
1/ Microsoft are stopping using WinSxS assemblies for managing the C/C++ runtimes as it is complex to manage and get right;
Ah, Visual Studio is backing away from WinSxS. I read their justification. I didn't buy into it. I think it's a solvable issue.
2/ With XP, Microsoft were selling WinSxS as being able to deploy different versions of the binaries, but for Vista/Win7 they are now saying that WinSxS is for archival purposes (see the Engineering 7 blog)
Uh, what? I've been talking to the maintainer of the WinSxS system. He's fully supportive of my plans.
3/ It does not really work as intended in practice -- e.g. comctl32 version 6 is different in Vista/Win7 than in XP, yet the applications that reference the XP version use the Vista/7 version
It works just fine, as long as you use it correctly; if they didn't, it's not my fault. Some of the tools I'm building will make it easier not to screw up.
Um, then what are you doing wasting your time here on/.? Shouldn't you be locked in a caffeine fueled coding frenzy, programming until your fingers are bleeding? Open source software won't write itself, you know;-)
I know!!!!
"His name cannot be s (16831)"
Is that a hint? Does that mean it could be one of the other 25 letters? Or maybe one of the 20 remaining consonants?
Well, ya see... with a five-digit slashdot-id I originally had "His name cannot be Spoken" as my name... then they did some database truncation about 12 or so years ago, and I lost some letters.
And ya can't change your name on Slashdot, and I didn't wanna give up my 5 digit ID.:D
All but the last one are fine. I have some windows boxes I have to deal with and I sure as hell do not want to be stuck using some GUI IDE just to build the latest $foobar.
Use of the GUI ain't mandatory... it's just that in order to get Windows devs on board, it'll have to have one.
The core bits will all be able to be command-line driven.
Assuming that you've looked at APT and similar packaging tools, and given that you're still convinced that there's a 'Windows Way' (your term) to handle deployment that differs from Linux best practices, how do you plan to address:
Yes, I've worked with APT and RPM for a very very long time now. The reason I'm convinced there is a 'Windows way' is because it's a different system that Linux; yes, I've learned a lot about PMS from Linux, and I know how to apply that knowledge to Windows.
Package Repositories - This is one of the main strengths of Debian and related distros. Do you think it's even possible to replicate this level of community control in Windows? I know you've mentioned decentralisation, but have you considered the implications of such an approach? What is the cost of failure to affect consistent, formalised management of package builds?
I have a plan for allowing any publisher to publish packages in the CoApp ecosystem, provided they meet two qualifications: - They must be able to host their repository meta-data on an SSL protected connection. - All packages must be digitally signed with a certificate that chains back to to a commonly-accepted CA.
Dependancy Management - This issue is largely done and dusted on Linux, but remains a dog's breakfast on Windows (albeit not as frustrating today as it was in the mid-90s). In the absence of centralised repositories and the Unix toolchain philosophy, how do you propose to cope better with dependancies?
I'm working with the developer of WiX to ensure that we can trivially build chained MSI packages that have the necessary smarts to properly manage this. Kind-of mixing in something like ldconfig with the Windows SxS library management.
File locations - How do you propose to manage the proper placement of libraries etc. when the conventions concerning where to put such files are not nearly as well defined on Windows? I'm suggesting here that you need cultural leverage rather than technical answers. You need to change perceptions, not toolkits.
Yes. The change starts with PHP, Apache, and Python, and the 40+ packages needed to build them (community members from each are already on board) Half of the project is setting some intelligent standards, and then bootstrapping the ecosystem with packages to enable other software to follow.
Security - Do you think it's even possible to replicate one of the main strengths of Linux package repositories: the ability to curtail security risks such as malware and flawed code?
Yes. By requiring code-signing (and I've got a plan for opening that up without cost for smaller projects) we can replicate the benefits of MD5 and PGP signatures found in the Linux world.
Scripting Interfaces - Say what you like about make and other command-line utilities, but as a busy sysadmin, I consider GUI package management a waste of my valuable time. If I'm going to deploy regular security updates, for example, I want to know that I can script every aspect of the operation. Even the tab-completion features in aptitude make it many times more efficient than a point-and-click interface. What is the potential for scripted deployment/management of packages under your system? Why?
I agree 100%. Scripting interfaces are an absolute requirement, and will likely come well before the GUI.
Think of it as a clean adaptation of the same concepts to the model that will be attractive to Windows developers.
I also think that you're going to need to learn a lot more humility than you've demonstrated so far if you want to achieve something better than a new brand of anarchy in packaging.
I apologize if I'm coming off arrogant. Frankly it's taken an extremely long time to convince the powers-that-be at Microsoft that Linux's package management is stellar compared to Windows. It's also not near as hard or large as it sounds, I'm walking on the shoulders of giants here, both in the Linux and Windows worlds.
I second the question about limiting to open source. A good package management system that can could make using SxS painless would be awesome in an enterprise environment.
I agree. it ain't really limited to Open Source
Since this is open source and.msi based I assume you will be leveraging WiX somehow?
Yes indeed. The author of WiX is on the mailing list, and a personal friend. He's very excited about all this too.
I hope this isn't going to be a big collection merge modules with duplicated component guids..
Nope. I don't believe in merge modules. I believe in a system that works.
I've spent the last couple of years at Microsoft working to make PHP better on Windows, and validating PHP apps including CMS systems like Drupal on Windows. Seems to me they want some competition.
Why limit this to open source? It would be great if the users could update every program easily and painlessly, at least the ones that use this new system.
I'm Busted. It isn't really restricted to Open Source... but that's my mission. Commercial apps will be able to play just fine in this ecosystem.
I am assuming that this system will allow easy and painless upgrading like on most Linux systems. Is that true? Will it have automatic dependency handling and command line installation?
Yes. Painless and automatic dependency handling, and yes command line tools. You are singing the chorus to my theme song!
Is this some kind of back-handed comment based on the general view at Microsoft about Open-source software, or the general view that MS would like to push out to userland? That people should use MS OSS because you need to be a developer to use it on other platforms?
No-no.. exactly the opposite
Have you tried to roll out some OSS apps on Windows?
On Linux it's two clicks, and BAM! Done.
On Windows, it's almost never that easy to setup OSS apps.
The problem I see is that it doesn't take a Developer on Linux to get Apache installed and configured. Why should it on Windows?
My intent is to completely do away with the practice of everybody shipping every damn shared library. It's one of the things that piss me off the most. I've got a very workable solution that uses WinSxS to cleanly handle this.
It is extremely important that there is a unified method for sharing libraries between apps.
Microsoft has given me a signed contract that says that whatever I produce for the CoApp project isn't owned by them. They do get a license to everything I make (fair deal), but they don't own it in the end.
That, and I've also chosen the BSD license for it's do-what-the-f*-you-want spirit.
Slashdot Mirror
Better get used to it.
It's still a long slide to the bottom.
'Approved' isn't the right word.
OneGet has the notion of 'trusted' repositories. We're likely to expand this concept a bit in the future, but for now, that's what it is.
Built-in package sources from reputable sources may be marked as 'trusted' by default, but the majority of sources should be 'untrusted' until the user makes that change.
The real trick is getting package provider plugins to tell OneGet the truth if a repository is trusted or not.
I suspect that we're going to have to introduce a level of trust with the package providers too, and expose this to the user ... somehow.
You've got a really good point.
We're tossing around some notions about different factors that make a 'package' or 'repository' trustworthy.
I'm sure we can do some stuff with signed repositories and signed packages to detect when things 'change' and/or keep unsigned repositories 'untrusted'.
Really, our first target for this stuff is developers and admins, not my mom...
Well, considering that the chocolatey provider for OneGet points to the community-controlled repository, I'll have to take that as a win :)
The concept of curated repositories is one that we're really trying to come up without screwing it up.
Regardless, with OneGet, the *user* maintains control. Which repositories they connect to, what software they install.
Actually, to be perfectly clear, OneGet isn't really a package manager.
It's a package-manager-manager -- It's a unified way of installing packages of software regardless of the how-it's-implemented-on-the-back-end.
The first real package provider plugin is a Chocolatey one. Why re-invent the wheel when the wheel already works?
The purpose here is to leverage all these different sources of software using a common set of commands and APIs.
Anything that can be represented as a 'source' of software can be plugged in on the back end. I'm aiming for plugins for NPM, Ruby Gems, Python, on top of the expected MSI, Chocolatey, NuGet, etc...
Plugins can be written by anyone, and I'm going to great lengths to make it as simple as possible -- it's about ~15 or so functions to implement and we can plug in virtually any package format or service into OneGet.
[FYI -- I'm @FearTheCowboy everywhere else, my /. id is so old that my name got trimmed from "His Name Cannot Be Spoken" 15ish years ago when they did a database adjustment... ]
I have had thoughts on how to do this; I suspect that while we may not set up a repo to do that, I may hack out the instructions on how that could be done easily if one wanted to maintain their own.
It really boils down to how much time I can throw at that.
Of course, we also want it to plug into WU and WSUS, but that'll be a bit more down the road.
âoeMr. Burns, Your Campaign Seems To Have the Momentum of a Runaway Freight Train. Why Are You So Popular?â
I do have one question. Why, exactly, do you think that this sort of approach is likely to be easier than doing what Apple did and simply exposing a Posix API that is actually useful?
Because, even if we could get a great POSIX experience on Windows, it leaves out Windows developers.
One of my goals is to get Windows developers in the OSS game.
On top of that, there is a hell of a lot of non-POSIX open source software on Windows that needs fixing too.
Look at it this way: Would you respect someone who told you the best way to get FireFox running on Linux was to use some sort of Windows emulation layer... Like WINE? no, because FireFox *can* compile for Linux. Same thing with nearly all Open Source I encounter. I want to get the OSS quality and experience on Windows to exceed commercial developers... it needs the most love.
Like I tell people:
Working as an open source software developer at Microsoft is like being a preacher in Vegas. I figure I'm in the single most important place in the universe that I can be.
That's the trouble with the six digit generation. Too Lazy.
Us 5-digit generation look down at you with disdain.
Now, get off my lawn!
It's 2010 and you are still doing *that*.
*sigh*
You know, that'd be funny if it was so damn sad. :)
think you had no choice to choose the BSD license instead of the GPL. Had you chosen GPL, it is likely the project would have been immediately rejected by Microsoft.
That's not true actually.
I didn't tell anyone what license I was going to use until a few days ago, by which time they'd already signed the agreement.
In addition to that; as a Microsoft employee for Microsoft, I've contributed code to GPL, LGPL, BSD, PHP and Apache licensed projects.
I'm busted.
I dunno man, I wrote that blog post a really long time ago, and then got stuck in red-tape. It's possible I never even proof-read it.
*sigh*
As for the first five points, yes I'm aware of all of that, and I'm working to solve all of them. Some of them are not possible (mixing compilers has a lot of bad mojo) and some are solvable with some really good best practices.
1/ Microsoft are stopping using WinSxS assemblies for managing the C/C++ runtimes as it is complex to manage and get right;
Ah, Visual Studio is backing away from WinSxS. I read their justification. I didn't buy into it. I think it's a solvable issue.
2/ With XP, Microsoft were selling WinSxS as being able to deploy different versions of the binaries, but for Vista/Win7 they are now saying that WinSxS is for archival purposes (see the Engineering 7 blog)
Uh, what? I've been talking to the maintainer of the WinSxS system. He's fully supportive of my plans.
3/ It does not really work as intended in practice -- e.g. comctl32 version 6 is different in Vista/Win7 than in XP, yet the applications that reference the XP version use the Vista/7 version
It works just fine, as long as you use it correctly; if they didn't, it's not my fault. Some of the tools I'm building will make it easier not to screw up.
Please, please, please, please make it easy to roll a python app into an MSI.
One of the first people on board with the project is Trent Nelson; he's all about Python.
I think we're gonna cover that.
Um, then what are you doing wasting your time here on /.? Shouldn't you be locked in a caffeine fueled coding frenzy, programming until your fingers are bleeding? Open source software won't write itself, you know ;-)
I know!!!!
"His name cannot be s (16831)"
Is that a hint? Does that mean it could be one of the other 25 letters? Or maybe one of the 20 remaining consonants?
Well, ya see... with a five-digit slashdot-id I originally had "His name cannot be Spoken" as my name... then they did some database truncation about 12 or so years ago, and I lost some letters.
And ya can't change your name on Slashdot, and I didn't wanna give up my 5 digit ID. :D
Dammit! I pooched some quote tags!
Sorry about that. Should have previewed. *sigh*
All but the last one are fine. I have some windows boxes I have to deal with and I sure as hell do not want to be stuck using some GUI IDE just to build the latest $foobar.
Use of the GUI ain't mandatory... it's just that in order to get Windows devs on board, it'll have to have one.
The core bits will all be able to be command-line driven.
Assuming that you've looked at APT and similar packaging tools, and given that you're still convinced that there's a 'Windows Way' (your term) to handle deployment that differs from Linux best practices, how do you plan to address:
Yes, I've worked with APT and RPM for a very very long time now. The reason I'm convinced there is a 'Windows way' is because it's a different system that Linux; yes, I've learned a lot about PMS from Linux, and I know how to apply that knowledge to Windows.
Package Repositories - This is one of the main strengths of Debian and related distros. Do you think it's even possible to replicate this level of community control in Windows? I know you've mentioned decentralisation, but have you considered the implications of such an approach? What is the cost of failure to affect consistent, formalised management of package builds?
I have a plan for allowing any publisher to publish packages in the CoApp ecosystem, provided they meet two qualifications:
- They must be able to host their repository meta-data on an SSL protected connection.
- All packages must be digitally signed with a certificate that chains back to to a commonly-accepted CA.
Dependancy Management - This issue is largely done and dusted on Linux, but remains a dog's breakfast on Windows (albeit not as frustrating today as it was in the mid-90s). In the absence of centralised repositories and the Unix toolchain philosophy, how do you propose to cope better with dependancies?
I'm working with the developer of WiX to ensure that we can trivially build chained MSI packages that have the necessary smarts to properly manage this. Kind-of mixing in something like ldconfig with the Windows SxS library management.
File locations - How do you propose to manage the proper placement of libraries etc. when the conventions concerning where to put such files are not nearly as well defined on Windows? I'm suggesting here that you need cultural leverage rather than technical answers. You need to change perceptions, not toolkits.
Yes. The change starts with PHP, Apache, and Python, and the 40+ packages needed to build them (community members from each are already on board) Half of the project is setting some intelligent standards, and then bootstrapping the ecosystem with packages to enable other software to follow.
Security - Do you think it's even possible to replicate one of the main strengths of Linux package repositories: the ability to curtail security risks such as malware and flawed code?
Yes. By requiring code-signing (and I've got a plan for opening that up without cost for smaller projects) we can replicate the benefits of MD5 and PGP signatures found in the Linux world.
Scripting Interfaces - Say what you like about make and other command-line utilities, but as a busy sysadmin, I consider GUI package management a waste of my valuable time. If I'm going to deploy regular security updates, for example, I want to know that I can script every aspect of the operation. Even the tab-completion features in aptitude make it many times more efficient than a point-and-click interface. What is the potential for scripted deployment/management of packages under your system? Why?
I agree 100%. Scripting interfaces are an absolute requirement, and will likely come well before the GUI.
Think of it as a clean adaptation of the same concepts to the model that will be attractive to Windows developers.
I also think that you're going to need to learn a lot more humility than you've demonstrated so far if you want to achieve something better than a new brand of anarchy in packaging.
I apologize if I'm coming off arrogant. Frankly it's taken an extremely long time to convince the powers-that-be at Microsoft that Linux's package management is stellar compared to Windows. It's also not near as hard or large as it sounds, I'm walking on the shoulders of giants here, both in the Linux and Windows worlds.
I second the question about limiting to open source. A good package management system that can could make using SxS painless would be awesome in an enterprise environment.
I agree. it ain't really limited to Open Source
Since this is open source and .msi based I assume you will be leveraging WiX somehow?
Yes indeed. The author of WiX is on the mailing list, and a personal friend. He's very excited about all this too.
I hope this isn't going to be a big collection merge modules with duplicated component guids..
Nope. I don't believe in merge modules. I believe in a system that works.
If it does, so be it.
I've spent the last couple of years at Microsoft working to make PHP better on Windows, and validating PHP apps including CMS systems like Drupal on Windows. Seems to me they want some competition.
Why limit this to open source? It would be great if the users could update every program easily and painlessly, at least the ones that use this new system.
I'm Busted. It isn't really restricted to Open Source... but that's my mission. Commercial apps will be able to play just fine in this ecosystem.
I am assuming that this system will allow easy and painless upgrading like on most Linux systems. Is that true? Will it have automatic dependency handling and command line installation?
Yes. Painless and automatic dependency handling, and yes command line tools. You are singing the chorus to my theme song!
Is this some kind of back-handed comment based on the general view at Microsoft about Open-source software, or the general view that MS would like to push out to userland? That people should use MS OSS because you need to be a developer to use it on other platforms?
No-no.. exactly the opposite
Have you tried to roll out some OSS apps on Windows?
On Linux it's two clicks, and BAM! Done.
On Windows, it's almost never that easy to setup OSS apps.
The problem I see is that it doesn't take a Developer on Linux to get Apache installed and configured. Why should it on Windows?
No.
My intent is to completely do away with the practice of everybody shipping every damn shared library. It's one of the things that piss me off the most. I've got a very workable solution that uses WinSxS to cleanly handle this.
It is extremely important that there is a unified method for sharing libraries between apps.
That is precisely the red tape that I had cut.
Microsoft has given me a signed contract that says that whatever I produce for the CoApp project isn't owned by them. They do get a license to everything I make (fair deal), but they don't own it in the end.
That, and I've also chosen the BSD license for it's do-what-the-f*-you-want spirit.
I use that all the time!