Canonical Drops CouchDB From Ubuntu One

rsk writes "Since the Ubuntu One desktop synchronization service was launched by Canonical it has always been powered by CouchDB, a popular document-oriented NoSQL data store with a powerful master-master replication architecture that runs in many different environments (servers, mobile devices, etc.). John Lenton, senior engineering manager at Canonical, announced that Canonical would be moving away from CouchDB due to a few unresolvable issues Canonical ran into in production with CouchDB and the scale/requirements of the Ubuntu One service. Instead, says Lenton, Canonical will be moving to a custom data storage abstraction layer (U1DB) that is platform agnostic as well as datastore agnostic; utilizing the native datastore on the host device (e.g. SQLite, MySQL, API layers, 'everything'). U1DB will be complete at some point after the 12.04 release."

Comments

Our sync service is not “powered by CouchDB&

[Chipaca]

Our structured data sync service is CouchDB, except for tomboy notes. Syncing files is a completely separate stack.

unresolvable issues

[Anonymous Coward]

dropped Ubuntu due to unresolvable issues with the way that they handle desktop environment migration. Liking Mint much better. Hope others are able to manage or migrate. Ubuntu is otherwise a very nice OS.

Specific Issues

[mysqlrocks]

It would be interesting to hear more from Canonical about what specific issues they ran into. They say that they worked with "the company behind CouchDB." While Couchbase is one company "behind" the project, CouchDB itself is an Apache project. Did they reach out to the Apache project itself? Also, why build something completely new rather than provide patches to existing software? I'm sure they had good reasons, but I'd like to hear some more details about what did and didn't work for them.

Re:Specific Issues

[Anonymous Coward]

The only "new thing" is a database abstraction layer that they should have already been using to begin with. Who in this day still writes their software heavily coupled to a single database rather than using a thin abstraction layer?

Re:Specific Issues

[Chipaca]

The only "new thing" is a database abstraction layer that they should have already been using to begin with. Who in this day still writes their software heavily coupled to a single database rather than using a thin abstraction layer?

we did, it's desktopcouch. Turned out to be too thin.

Re:Specific Issues

[tildeslash]

A lot of newer tech companies prefer to develop their own systems these days; there is a new culture of "dogfooding" i.e. building your tools from scratch and using your own product. There are good technical reasons for doing so when you are innovating, as existing systems will never quite meet your requirements. This is especially true of the cloud and "big data" (non-relational DBs), which are both still young and rapidly evolving.

As for specifically what went wrong: I suspect that comes under trade secrets. Building a cloud is hard.

Re:Specific Issues

[larry bagina]

meh, they should use an abstraction layer on top of other abstration layers to cover all the bases.

Re:Specific Issues

[bolthole]

You are confusing "Dogfooding" with "NIH syndrome"

Re:Specific Issues

[Medievalist]

meh, they should use an abstraction layer on top of other abstraction layers to cover all the bases.

Yeah, and XML-encapsulated SQL, definitely needs at least one layer of XML. Definitely.

Re:Specific Issues

[nschubach]

Just to be safe, you should store your data as XML elements in the fields of the same name:
select id from table
<id>1</id>
<id>2</id>

Re:Specific Issues

[BitZtream]

No, dogfooding is using the stuff you make yourself. It has nothing to do with WHY you made it, HOW you made it or anything else.

Its simply different than making a product to sell, and then using something entirely different internally. For instance, Microsoft selling Visual Source Safe ... but not actually using it internally for Windows because it sucked so hard. That is not dogfooding.

The term is 'eating your own dogfood', because you're eating what you made.

Linus dogfood's Linux for instance.

WTF is it with you kids fucking up phrases you don't understand. If you don't know what it means or where it came from will you fucking please stop god damn pretending you do. And stop calling all unpatched exploits zero day. They stopped being zero day within 24 hours of creation. After that they are just fucking exploits.

Original message from John Lenton (Canonical)

[Anonymous Coward]

From the first days of Ubuntu One, before we were even in Ubuntu, we've
had a structured data storage sync service based around CouchDB.

For the last three years we have worked with the company behind CouchDB
to make it scale in the particular ways we need it to scale in our
server environment. Our situation is rather unique, and we were unable
to resolve some of the issues we came across. We were thus unable to
make CouchDB scale up to the millions of users and databases we have in
our datacentres, and furthermore we were unable to make it scale down to
be a reasonable load on small client machines.

Because of this, we are turning off most of our CouchDB-related
efforts. The contacts, notes and playlists databases will continue to
exist on our servers to support the related services, but direct
external access to the underlying databases will be shut off. Any other
databases will be deleted from our servers entirely.

For these same three years we have created and maintained desktopcouch,
which is a desktop service (and related library) to access CouchDB more
conveniently. Because we are no longer going to pursue CouchDB, we will
no longer be developing desktopcouch; in fact, if anybody wants to take
over, we'll be happy to work with you to make that official. For the
upcoming 12.04 the Ubuntu One packages will not depend on desktopcouch
nor couchdb in any way, and we'd recommend the distribution seriously
consider whether they want to continue having the package in main,
especially if no maintainer shows up.

Because we still believe there is a lot of value to our users in the
service we wanted to offer based on CouchDB, we're building something
new, based on what we've learned. It's very small, merely a layer of
abstraction and the definition of an API that will allow us and others
to build what is needed ontop of existing tools. We're calling it U1DB
for now, until it comes of age. If you're interested and techincally
inclined you can follow our progress on lp:u1db; unfortunately our
timing and resources are such that we can only promise the reference
python implementation will be ready in time for 12.04, and thus 12.04
will ship without Ubuntu One having a solid story around synchronizing
arbitrary structured data.

Thank you for reading.

https://lists.ubuntu.com/archives/ubuntu-desktop/2011-November/003474.html

Re:Original message from John Lenton (Canonical)

[Anonymous Coward]

Imagine no CouchDB. It's easy if you try.
- John Lenton

Re:Original message from John Lenton (Canonical)

[Beuno]

Because the next release is an LTS, which will now be supported for 5 years. If couchdb is kept for the next release, it will need to be supported to another 5 years.

Re:Rolling your own

[JSBiff]

You did read the same thing I did, right? They *tried* using someone else's solution, and the solution did not fit their needs. If the existing solutions don't fit your needs, what else can you do other than roll your own? I guess you could drop the service/product altogether and just call it a day, but that doesn't sound like a great business model.

Re:Rolling your own

[Entrope]

If CouchDB doesn't work, there are at least six other competing solutions that will tell you how they are a better fit for your needs than all the others. Cassandra, Hadoop/HBase, MongoDB, and more. If one doesn't work for you, you can waste as much time as you like trying the others!

Why did syncing become so difficult?

[drx]

I wonder what became so difficult about syncing data that it has to be re-invented all the time?

I was happy using tools like rsync, diff and unison for a long time, until the moment when even Linux desktop software is too posh to store their data in files.

Now every software uses another database, at one time even Amarok used a MySQL backend. What is better about this than just putting the data in a file? Or at least making this file the Single Point Of Truth? If you need the database for speed, you can check if the file changed since the last time and then update the database from the file's contents. But simple files have been syncing and merging and everything perfectly for ages. Now it seems like every software needs its own syncing service.

Is there any reason for this, except brading the most simple things (like copying a file), or making money with cloud storage?

Re:Why did syncing become so difficult?

[plopez]

I would put my money on sloppy code that made the DB engine inefficient as well as ACID issues. Synching in real time in a distributed environment and staying ACID compliant is a *hard* problem. See this guy's research for some of the gory details: http://en.wikipedia.org/wiki/Jim_Gray_(computer_scientist)

Re:I find Mint very mint

[Rogerborg]

And according to distrowatch, Mint is now the #1 distro, ahead of Ubuntu. I applied the "my wife's box" test (fnar fnar), and yup, Mint 11 LXDE satisfied her completely, where Ubuntu 11.04 had left her dry and aching. I guess Shuttleworth's Reality Distortion Field needs a bit of work.

Re:Rolling your own

[funfail]

Lotus Notes does (maybe better). If you ignore the hate about its user interface, the server component (Lotus Domino) is very robust and scalable as a NoSQL provider.

Actually, Damien (author of CouchDB) is a former Lotus engineer and modeled his creation similar to Notes Storage File (NSF) structure.