Over 650 TB of Data Up For Grabs From Publicly Exposed MongoDB Database (csoonline.com)

itwbennett writes: A scan performed over the past few days by John Matherly, the creator of the Shodan search engine, has found that there are at least 35,000 publicly accessible and insecure MongoDB databases on the Internet, and their number appears to be growing. Combined they expose 684.8 terabytes of data to potential theft. Matherly originally sounded the alarm about this issue back in July, when he found nearly 30,000 unauthenticated MongoDB instances. He decided to revisit the issue after a security researcher named Chris Vickery recently found information exposed in such databases that was associated with 25 million user accounts from various apps and services, including 13 million users of the controversial OS X optimization program MacKeeper, as reported on Slashdot on Wednesday.

Comments

Clone available here

[Sir Holo]

I've posted a clone of this data-cache on my BBS. Just make sure you have at least a 54.6 baud modem, so ensure a speedy download.*

Long-distance telephone charges may apply.

Re:Web-scale breach

[vtcodger]

Doesn't matter if they forgot or tried, but their defenses were flawed by a misspelling, or misunderstanding, or a semicolon that should have been a colon. Truth is that trying to secure information on computers connected to the Internet of Horrors is roughly equivalent to stashing your wealth in a huge paper tent guarded by an elderly german shepard who has been sedated and two winos who have provided themselves with a liberal supply of cheap booze.

Folks, this internet thing really does have enormous potential. And it will always be useful for broadcasting and reference work and cat videos. But it's way too complex to provide a reliable vehicle for financial information, personal data, or command and control of most infrastructure. No amount of frantic patching, blaming someone else, and trying to administer remote computers with unknown configurations is going to fix that. This sucker just can't do what "they" want it to do. At least not safely.

What's the answer? I haven't a clue. But acknowledging that there is a problem is probably a good first step.

Re:Web-scale breach

[Anonymous Coward]

Read the documentation for the software which even comes with reasonable guides for Linux and Windows?

https://docs.mongodb.org/v3.0/core/security-network/

For best results and to minimize overall exposure, ensure that only traffic from trusted sources can reach mongod and mongos instances and that the mongod and mongos instances can only connect to trusted outputs.

Sorry but anybody working in this space should know better.

Yeah, BUT....

[JustAnotherOldGuy]

.....something something webscale.

If you're gonna leak data, go big or go home!

Gigabit fiber is arriving just in time!

[jeffb (2.718)]

Once AT&T hooks us up, I'll be able to download this volume of data in a mere two months.

Oh, wait. 1TB monthly data caps.

Okay, once AT&T hooks us up, I'll be able to download this volume of data in a mere 54 years.

Anyone who uses mongodb....

[Viol8]

... deserves to lose their data as a lesson not top use amateur hour software.

Field data longer than 8kb? Ooh, can't index that and it won't get returned in a query using that index.

Shard gets corrupted? Oh bad luck, thats some of your data gone - unless you've used also replication in which case you'll have spent 2 months trying to set it all up.

Lots of concurrent writes? Yeah, well, with monogdbs single monolithic write lock - good luck with that.

Want a DB that uses encrypted network transfers between shards and replica sets? Sorry.

Want a DB that uses a sane query language - ie not one thats a nightmare mashup between pure javascript and parameter passing using javascript to an underpowered underlying query engine? Don't use mongo.

Etc , the list goes on.

Re:Anyone who uses mongodb....

[PyramidOfDoom]

Want a DB that uses a sane query language - ie not one thats a nightmare mashup between pure javascript and parameter passing using javascript to an underpowered underlying query engine? Don't use mongo.

Etc , the list goes on.

Don't forget that you have different dynamic typing rules to battle with depending on whether you're doing a regular json query (BSON type rules) or whether or not you're doing something javascript-ey (like collection.mapReduce, where you get Javascript type rules).

Example (code is probably wrong; I'm working off memory and haven't touched Mongo in nearly a year):

db.collection.insert({foobar: "1"});

db.collection.find({foobar: 1}); //no results

db.collection.mapReduce(function(){
if (this.foobar == 1) emit(1, this.foobar); //works
}, /* rest of reduce function + crap here */);

For extra fun, throw PHP into the mix and marvel at how your integer properties get silently turned into strings whenever they pass into a html form and then get rewritten to the database!

NoSQL is amateur land.

[Anonymous Coward]

I'm not totally surprised. In my experience, those who use and advocate NoSQL "databases" tend to be on the amateurish side of the spectrum.

Except for a very small number of cases, if you have data to store you should be using one of the many existing relational database management systems.

It doesn't even matter which one you use. There are many you can pay for, and many free ones, too. PostgreSQL is the best free one. MySQL isn't very good, but it's a hell of a lot better than the NoSQL systems.

These databases can easily store key-value data. They can also easily store JSON, XML, and other non-relational formats.

Learn how to use indexes. It turns out that many of the performance problems these people use to justify switching to a NoSQL database only arise because they don't know how to index their relational database system. Some of these NoSQL supporters don't even know what an index is at all!

Learn how to use SQL. It ain't perfect, but it's a fuck of a lot better than writing queries in JavaScript, of all things!

Even the most basic book about relational databases that you can find at your local bookstore will clearly explain all of these concepts.

It's like most people who use NoSQL, especially when they're dealing with anything less than petabytes of data, try as hard as they can not to put in the small amount of effort needed to learn now to use relational databases properly.

Re:NoSQL is amateur land.

[Dcnjoe60]

The issue here really isn't SQL vs NoSQL. It's about securing the data and access. Lack of security is not inherent in NoSQL, it just occurs more often than SQL databases.

Re:NoSQL is amateur land.

[Ol Olsoc]

It's like most people who use NoSQL, especially when they're dealing with anything less than petabytes of data, try as hard as they can not to put in the small amount of effort needed to learn now to use relational databases properly.

I've had issues with getting people to even understand relational databases in the first place, including the sortakinda amusing case where under orders, I gave a co-worker one to work with and looking for a progress report a month later, he told us that the entire database didn't work and couldn't work and nothing could be done to make it work. Funny, because it had been in flawless use for years, and not funny because it put me a month behind.

It was also the last time my boss told me "Ol, you have to learn to delegate better".

Re:NoSQL is amateur land.

[anchovy_chekov]

The issue here really isn't SQL vs NoSQL. It's about securing the data and access. Lack of security is not inherent in NoSQL, it just occurs more often than SQL databases.

Well, I'd have to disagree here. If I install a MongoDB on some cloud VM using the default setup, I have an insecure database available on the internet. If I install Postgres, well... I can't even access the database remotely.

Much of the time NoSQL is a problem looking for a solution. I've had to deal with this a number of times before - and I'd concur what the AC above posted - the culture around NoSQL often seems to be "squeee! I have a NoSQL database.. look at my awesome speed" without investing time getting to know existing SQL properly. Or securing the systems.

Case in point: a while ago I replaced an Elasticsearch database with 40 lines of decent SQL. It wasn't rocket science (MySQL), but it was an order of magnitude faster and less complicated as it didn't require an additional server and the data pump between the SQL and Elasticsearch databases. Yes, I know Elasticsearch isn't a NoSQL db in the same sense as Mongo, but it was part of swag (4 in total) non-traditional databases in use papering over poor indexing / querying on the existing MySQL db. Oh.. and no security. That's the NoSQL culture I've experienced, and not just at one shop. All over the place.

We lost something when database administrators were given short shrift and we all became "full stack developers". So much knowledge has just gone out the window.

Re:NoSQL is amateur land.

[vel-ex-tech]

I think what AC is trying to say here is that there's a lot of woo surrounding NoSQL, and that people who buy into that woo don't have a solid grounding or understanding about how computers work, hence security isn't even an afterthought for them.

As far as I can tell, NoSQL is basically a buzzword that means either key-value pair or document storage. (I'm certain there must be some edge case where Mongo or others are the correct answer, at least I hope so.) In the case of key-value pairs, why not just use PostgreSQL? In the case of document storage, why not just use one of these things called a filesystem?

So, you're correct: lack of security is not inherent to NoSQL; lack of critical thinking, experience, and in-depth knowledge about tech is. This pretty much gets back to why our information infrastructure just sucks in general. Computers are sufficiently advanced technology and are thus indistinguishable from magic. Put another way, pretty much everybody, even some people in tech, thinks computers are magickal palantirs powered by waldos used to project some kind of digital self into the nether realm of cyberspace, a place not Sheol, which can only be tamed by wizards.

Their idea of a cyber-security is having a wizard on hand who can shout at the evil hackers, who have equally mystical powers drawn from the foul evils of the burning hells, "You cannot pass! I am a servant of the Secret Fire, wielder of the Flame of Anor. The dark fire will not avail you, Flame of Udun! Go back to the shadow. You shall not pass!"

Re:NoSQL is amateur land.

[anchovy_chekov]

Well, thanks to Rails and similar frameworks we've been treating the database as a dumb data repository for years. Not surprised that the kids have no idea how to make the things work properly.

Re:NoSQL is amateur land.

[anchovy_chekov]

I used to have a simple test in interviews. "Describe the difference between an inner and an outer join". In the end I gave up. Plus management didn't want to hire the only person to answer it correctly because they were clearly too experienced.

Re:NoSQL is amateur land.

[Dcnjoe60]

The problem is that NoSQL usually is set up by inexperienced people or amateurs. Relational databases tend to be the opposite. Regardless of whether NoSQL installs with total access and Postgress with no access doesn't matter. Both need to be configured to provide the access needed. I do agree that Postrgress' method is safer because nobody has access unless you grant them it. But, that doesn't change the fact that both need to be configured.

The reality is NoSQL is being implemented in cases where it is not the best solution. Same could be said for Relational databases. Often they are overkill for simple things. In the end, both are tools and selecting the right tool for the job is the best solutions.

Re:NoSQL is amateur land.

[unimacs]

Why use a relational database like postgres when you have no intention of using its relational features just because it can also store JSON data?

I'm a huge fan of postgres. I introduced it to the organization I work in probably 15 years ago and we continue to use it today. But there are some things it just doesn't do as well, - like scale, at least not as easily as a database like Cassandra can. Don't get me wrong, you give up a lot in many cases by not choosing an SQL database and it's not always the right choice. However, lack of critical thinking isn't inherent in using a NoSQL database. Lack of critical thinking is assuming a certain type of database is always the right choice for any future project, just because it's worked well in previous projects.

Re:NoSQL is amateur land.

[iamgnat]

The issue here really isn't SQL vs NoSQL. It's about securing the data and access. Lack of security is not inherent in NoSQL, it just occurs more often than SQL databases.

Well, I'd have to disagree here. If I install a MongoDB on some cloud VM using the default setup, I have an insecure database available on the internet. If I install Postgres, well... I can't even access the database remotely.

Ummm, unless something has changed recently the respective "createdb" tools for both MySQL and Postgres make it very simple to start up a new DB with an open root account and listening on an IP.

I agree with your general view on NoSQL and who tends to use it, but it's the ones using it that are the problem here and not the software itself. The only argument against NoSQL/Mongo here is that it is shocking in this day and age for any server to allow un-authed access. That can be applied to many server software packages though including OSS SQL DBs.

What this is really a symptom of is the people with no real experience that are just about "getting the job done" rather than understanding the actual problem and the potential risks. I don't know how many times I've seen cases in my career where the first sign of communication issues causes people to start tearing down the security barriers. Most of the time the real problem was a minor config issue (going over the wrong interface, missing host to host ACL, listening on the wrong port, etc..), but the damage done to "just get it working" is rarely ever fixed.

Re:NoSQL is amateur land.

[CastrTroy]

I'm not sure about Postges, but the default install of MySQL will only listen on localhost. So you have to change your config file to even get it to respond from a remote computer. Then there's the account you're actually accessing it from. By default, the root account in MySQL is also only accessible from localhost. So that's 2 things you have to change simply to even allow a connection from a remote machine in MySQL.

I agree that there''s a bigger problem with people just wanting to "get the job done". Even those two steps only take about 20 minutes to figure out how to get around them and leave your root account accessible from the internet, even if you have no idea what you are doing. But shipping the database with such lack security is right up there with default passwords on home routers and other such problems. Stuff like this just shouldn't happen. Even if the security can be easily disabled by a determined user, it shouldn't be insecure by default. That way, if they have an insecure setup, at least it's because of something they actively went out and did themselves.

Re:NoSQL is amateur land.

[CastrTroy]

That question is about 5 levels above the Fizz Buzz test, so I'm not surprised that you had so much trouble finding people who could pass your test. People who work with databases should definitely be able to answer that question, but it's also common enough to find people with computer science degrees who've never had to deal with a database in their life.

Most computer science degrees are only going to have a single course on databases, and it's probably easy enough that you can get through by kind of understanding how databases work just enough to pass the exam, and proceed to forget it all a couple weeks later. There are plenty of programming jobs out there that don't require databases. There's also plenty of programming jobs where everything has been blocked off, such that the only way to access the database is from a defined list of stored procedures. So it's completely believable that a very competent programmer might just have not had any exposure to databases and might struggle with a question like this. Just as some other people would have trouble answering questions dealing with something like how sorting algorithms work. Because the last time we had to do it was during some second year computer science course, and from that point on all we were concerned with was calling the correct function to get the job done.

Re: NoSQL is amateur land.

[Anonymous Coward]

No, lack of security is inherent in 20-something IT types who think that everything that's ever been dune in this profession is beneath them and not worth knowing. Those are the types who tend to use and recommend this garbage--then they set it up wrong because learning how stuff works is also beneath them.

Apologies to the 20-somethings who learn stuff and actually belong in this profession.

Are the users aware they use Mongodb?

[140Mandak262Jamuna]

It is very much possible many users of Mongodb are not even aware that they are Mongodb users. And also many users of these websites might not know that their web site is using Mongodb unbeknownst to even the web site owners.

There are so many people with very little internet skills own domains and basically hire someone to build their site. I recently registered a domain and I am being swamped by solicitations to "the new small business owner" about "making a great website to sell your product" with "search engine optimization" and "customer support data base" blah blah blah. Frankly even their email skills are fishy and I am sure they will jury-rig some solution. If they change the default installation to make it less secure, it definitely looks like someone managing something for someone else where convenience overrides security.

Re:Web-scale breach

[Antique Geekmeister]

> Security seems to be almost an afterthought.

By the time you get done activating security high availability, and integrity checking, you've often lowered the performance to below that of a more mature, well-tuned SQL database. There is a very amusing video that covers some of these issues in passing:

                      http://www.mongodb-is-web-scal...

Re:Web-scale breach

[anchovy_chekov]

I watched that vid probably a dozen times, just to keep mysql smiling in the crazy days of having Neo4j, InfluxDB, Redis and Elasticsearch all thrown in by devs into the one project (no.. I shit you not...)

For those without Flash, here's an another version: https://www.youtube.com/watch?...

Re:Web-scale breach

[anchovy_chekov]

Mysql smiling. WTF? s/mysql/myself. I'm frigging tired, going to bed. Have a good night peeps.

Re:Web-scale breach

[PlainWhiteTrash]

It's an issue as old as this industry.

NoSQL needs a boost in adopters to gain momentum and beat-out a lot of entrenched relational databases?

Easy. Just make it easy. Make it mind numbingly easy to make the database perform useful work. Simple package installation, no need to configure anything away from defaults. Example project can connect and do CRUD-y things in minutes. Explain away security and good practice in the documents no one will read, (because the manual as they see it is the blog post on "5 minutes to your first working NoSQL web project"). And.... Win. It's "easy", adoption skyrockets. It's hard to knock those results. Well, at least if you believe you have no moral obligation to help ensure your software doesn't do harm.

Meanwhile, back in the land of responsibility, an out of the box PostgreSQL instance can't even be connected to from another system. You're forced to confront the security configuration and access considerations. This takes more time and you actually have to read, but more importantly, you have to stop and define what should and shouldn't be allowed. You might have to understand the actual operating environment from a network access and authentication perspective.

This sort of press coverage will help to set things back to right. Sadly, it will do so at the cost of so many innocent(-ish) people's privacy.