Slashdot Mirror

Wrong.
http://www.marklogic.com/blog/can-you-pass-the-acid-test/

Allow me to rephrase:

After a project that I was biased against [1] failed, my bias was confirmed and I knew that my preferred solution would be much better.

I have a half-baked plan already, so surely the real thing can't be much more difficult.

Now I see evidence that contradicts my bias, so I'm completely surprised.

[1]: Why is a project I know little about using a database I know little about?

One of MarkLogic's strong points is that it uses that "example schema from a private insurance firm" as its starting point, keeping records arranged in the proper hierarchies for use in the healthcare industry. Yes, you could reproduce the constraints using another database, but why go to the extra work? Oh, right, there's that consistency point... but a quick search shows that MarkLogic is claiming ACID support.

So for a project in the healthcare sector is using a healthcare-oriented database. This doesn't seem to be a bad idea. The questionable part is how there are fewer MarkLogic experts than Oracle gurus, but that's not really a showstopper.

To be honest, I had never even heard of MarkLogic until this whole blowup happened, so I think I will have to look into what it can do.

We'd love to have you check us out. If you want to learn more about it, you can read more about us on our web site at http://www.marklogic.com/what-is-marklogic or in Gartner's recent operational DBMS Magic Quadrant: http://gtnr.it/Ieh0hq. You can also download and play with MarkLogic at http://developer.marklogic.com/products. Let me know if you'd like more info or help.

Best,
-- David

To be honest, I had never even heard of MarkLogic until this whole blowup happened, so I think I will have to look into what it can do.

We'd love to have you check us out. If you want to learn more about it, you can read more about us on our web site at http://www.marklogic.com/what-is-marklogic or in Gartner's recent operational DBMS Magic Quadrant: http://gtnr.it/Ieh0hq. You can also download and play with MarkLogic at http://developer.marklogic.com/products. Let me know if you'd like more info or help.

Best,
-- David

Hi Anonymous. As I mentioned in a previous post, I run Engineering at MarkLogic. I'm sorry to hear about the difficulties you had. Here's the link to our admin API, which allows you to script backups: http://docs.marklogic.com/admin/database. Hope this helps. If you have other questions, feel free to contact Support, who is there 24x7 to assist our customers.

It's also worth pointing out (since we're talking about backups), that in order to back up a database, you need to be able to get a consistent view of the data at a point in time and hold that consistency for the duration of the backup. For databases that don't provide strong consistency, you usually have to quiesce the database before backing it up. Another advantage of ACID transactions, even for applications that aren't OLTP-type applications.

-- David

Hello Ganjadude. I run Engineering at MarkLogic, so I'm quite familiar with the technology, and I'd be happy to answer any questions about what it is and how it works. Here's a start:

MarkLogic (the company) is a privately held company based in Silicon Valley, backed by Sequoia Capital, Tenaya Capital, and Northgate Capital. We've been around for over a decade, and have hundreds of customers in production. We like to think we were "NoSQL before NoSQL was cool."

MarkLogic (the product) is an Enterprise-class document-centric NoSQL database. Enterprise-class means that it doesn't throw out the enterprise functionality you need to run mission-critical applications. Things like ACID transactions, automatic failover, database replication, and item-level role-based security. Document-centric NoSQL means that it's optimized for storing complex data that contains a mix of traditional values (dates, names, etc.) and unstructured full text in denormalized (document) form. The document-centric model allows it to store hierarchical, sparse, or repeating data easily. MarkLogic is schema-agnostic, which means that entities (documents) in the database do not need to conform to a schema (although if they do, we can take advantage of it to do things like schema validation). This also means that a database can contain a mix of data from different sources in different formats that can all be queried together. This is a key reason why many customers use us. If you want to combine complex data from multiple sources (especially if those sources are changing over time) and query it in real-time with sub-second response time, MarkLogic lets you do that.

Like many NoSQL technologies, MarkLogic is built with a scale-out architecture that allows it to scale horizontally. It processes queries in parallel across nodes in the cluster, and it uses a sophisticated indexing scheme that mixes full-text and structured indexing together to provide sub-second response time to complex queries across huge amounts of data. MarkLogic uses MVCC to provide ACID transactions. For transactions that span nodes in a cluster, we use two-phase commit. Unlike many NoSQL technologies, we were designed for enterprise use, so for our customers, data consistency is important.

MarkLogic is not the only technology in use in the healthcare.gov architecture, but we are used in places where it makes sense to take advantage of our ability to integrate data from a variety of sources and query it quickly and consistently. In this particular application MarkLogic is performing well, responding in less than 1/4 second for 99.9% of queries.

MarkLogic is in use in hundreds of Global 1000 companies, and in many applications in public sector, civilian, military, and intelligence. We power the emergency operations network for the FAA. We powered the BBC's live coverage of the 2012 Olympics. We power the operational trade store for major investment banks. If you haven't heard of us, you should check us out. You can read more about us in Gartner's recent operational DBMS Magic Quadrant: http://gtnr.it/Ieh0hq. You can also download and play with MarkLogic at http://developer.marklogic.com/products.

-- David

FoundationDB actually did a pretty good job of separating out the truth from the hype recently in their post about NoSQL and ACID Compliance: https://foundationdb.com/acid-claims#!

Doesn't mention neo4j specifically - don't think the article authors considered graph/triple stores in their article - but it's a good read nonetheless. The article mentions MarkLogic too. Added my comments on it here: http://adamfowlerml.wordpress.com/2013/11/17/foundationdb-joins-acid-transaction-crusade/

FWIW MarkLogic 7 also is a triple store (not graph store), including support for the W3C Graph Store protocol and SPARQL. http://www.marklogic.com/what-is-marklogic/marklogic-semantics/

[Yes, I do work for MarkLogic - and proudly so!]

As someone who has done databases for a long time, I have very little respect for NoSQL, but that is mostly because everyone keeps trumpeting it as a killer of traditional databases. There are scenarios where NoSQL systems are an ideal fit. However, NONE of those scenarios require data to be very reliably stored in a guaranteed and predictable way.

If you don't get your tweets or your friends facebook posts as soon as they are posted, no one will really care. But for something as truly important as health insurance coverage? Are you f__king kidding me? And that's just from a reliability standpoint. Nevermind the fact that NoSQL is currently at the wild west stage where nobody is compatible with anybody else, there is nothing resembling a standard set of APIs between products, making it very difficult to develop expertise.

I certainly get where you're coming from with this, but a couple of points to address your concerns:

MarkLogic has been around since 2001, with features needed for enterprise deployments (ACID, HA/DR, security) baked in from the start; it's not just the new hawtness.

NoSQL systems aren't just for lots of data, they are for data with lots of variety. In those cases, a schemaless approach allows for much more rapid application development. And yes, many of those scenarios do require reliable storage. MarkLogic does ACID transactions. The MarkLogic customer page shows some of the customers who have found this helpful.

Regarding interoperability, MarkLogic has a REST API and a Java API, in addition to the ability to work with XQuery. Custom HTTP service endpoints can be built to fit into an existing environment. It still requires some learning -- a document store is different from relational, so that's unavoidable -- but with a little training developers can use a lot of their current skills.

MarkLogic has scheduled backups. You can also do a backup through the Admin UI, which is mentioned pretty early in the Administrator's Guide.

MarkLogic has scheduled backups. You can also do a backup through the Admin UI, which is mentioned pretty early in the Administrator's Guide.

MarkLogic has scheduled backups. You can also do a backup through the Admin UI, which is mentioned pretty early in the Administrator's Guide.

Why would anyone design such a product? Why would anyone buy such a product?

Because for many kinds of data, once I became familiar with MarkLogic I found I could build complex applications more rapidly than I could with relational technologies. It scales out, it's schema agnostic, and it's ACID compliant. It has HA/DR features and a robust security model. Customers buy it because it solves their problems. For anyone interested in learning a bit about MarkLogic, the Enterprise NoSQL page is a good place to start. Disclaimer: MarkLogic employee.

MarkLogic is ACID compliant. http://www.marklogic.com/blog/can-you-pass-the-acid-test/

http://www.marklogic.com/what-is-marklogic/marklogic-hadoop/

This seems like the relevant, interesting/crazy part. Whether it's the right choice or not, I leave to more seasons database professionals than myself.

Oh, but they claim to be ACID complaint: http://www.marklogic.com/blog/can-you-pass-the-acid-test/
lol

The XML databases and tools are just now becoming mature enough to compete on the same level as relational databases. I would say they won't completely replace them but they have the potential to augment them in some interesting verticals. Anywhere the problem can be thought of as document centric (or hierarchical) an XML DB has the potential to out preform a relational data-store.

For some examples of XML DBs you can refer to:
http://exist.sourceforge.net/
http://www.modis.ispras.ru/sedna/
http://www.marklogic.com/
http://www.x-hive.com/

Also take a look at http://www.saxonica.com/ (Michael Kay's company) for some insight on how XML DB's can be used.
Remember that it takes years for major process cycles to change. XML DB's may not be in the limelight yet but their time is approaching.

Shameless plug: http://marklogic.com/

There's a free download, good for 30 days and up to 1 GB.

That's the executive pay more than taken care of, now they can get nice fat "performance"-related bonuses too!

Ah, just kidding, they're doing good stuff.