Slashdot Mirror
Python-LMDB In a High-Performance Environment
lkcl writes: In an open letter to the core developers behind OpenLDAP (Howard Chu) and Python-LMDB (David Wilson) is a story of a successful creation of a high-performance task scheduling engine written (perplexingly) in Python. With only partial optimization allowing tasks to be executed in parallel at a phenomenal rate of 240,000 per second, the choice to use Python-LMDB for the per-task database store based on its benchmarks, as well as its well-researched design criteria, turned out to be the right decision. Part of the success was also due to earlier architectural advice gratefully received here on Slashdot. What is puzzling, though, is that LMDB on Wikipedia is being constantly deleted, despite its "notability" by way of being used in a seriously-long list of prominent software libre projects, which has been, in part, motivated by the Oracle-driven BerkeleyDB license change. It would appear that the original complaint about notability came from an Oracle employee as well.
time to get a reality check about that
"a high-performance task scheduling engine written (perplexingly) in Python"
guys, there is this thing, it's called "algorithm"....
... to have even the most scant indication of what the damn thing does?
I guess it's a Berkeley DB replacement? Similar API?
At some point there will be an article on Wikipedia, that only meets Wikipedia's notability requirements due to media spillover complaining about the notability requirements.
I believe the definition here is that the software hasnt become so notable as to pose an immediate threat to a certain large database corporation which would require it to bury it in cash, rebrand it, lock it down, and pedal what little innovative or remarkable features the application had into the ground while pretending that somehow forking projects of the original software arent making them look like a complete failure.
Good people go to bed earlier.
Never mind what projects use it; what have independent reliable sources written about LMDB?
If Wikipedia was a person I would smack it upside the head for shit like this. There is absolutely no reason not to have an article on LMDB, and deleting a perfectly good article for no reason is evidence of a mental disorder. It's not like they have to spend an extra penny for a piece of paper to hold the article, possibly making the book too thick. Wake up.
Yeah, I'm FAR from a Wikipedia hater, but when it pulls shit like this it reveals its stupidity.
Seriously, googling "python lmdb" shows up only documentation and lmdb blog updates on the first few pages, and also a link to this slashdot article. I don't see anybody interested in it or singing its praises.
It's okay that your pet project isn't wikipedia-noteworthy yet. Concentrate on your evangelism (like, get at least one person who isn't you to write about it), and then try submitting again once you have some sources to cite.
LMDB was deleted from Wikipedia, not Python-LMDB.
The bindings are not especially notable. The embedded database is.
If you look at the guys talk page -
http://en.wikipedia.org/wiki/User_talk:JohnCD
which has in a brightly colored box:
If you have come here about a page I deleted, you will probably find the explanation here; if that does not answer your question, click the link just above to leave me a message. Please mention the name of the page, and sign your post with four "tilde" characters ~~~~ so that I know who you are.
Further research (clicking the link) takes you to - http://en.wikipedia.org/wiki/User:JohnCD/Whydeleted
If you're reading this, it's probably because I deleted your page based on Wikipedia's speedy deletion policy while cleaning out speedy deletion nominations. Then, you came by my user page or talk page to leave me a message and to ask why I deleted your page. First let me assure you that I was not, in fact, the one who nominated your hard work to be filed in the digital dustbin--that honour belongs to the New Page Patroller who brought my attention to it. That user, unless also an admin, cannot undelete your page. Although I am able (and may be willing) to undelete your page, you need to continue reading below to find out why I agreed to delete it in the first place. ... lots more words ...
Maybe you should read this page, then open a dialog with JohnCD to argue your case.
Yep, and there are like 2 dozen python wikis where no one would mind a write-up, and links. Sometimes things just aren't really encyclopedia topics. And that's fine.
Wikipedia, in spite of all odds, somehow manages to hold onto a tiny reputation for informational quality. Part of that is not having more information than the users can reliably fact-check. And I've never held it against wikipedia if I look something up and it just doesn't have an article. I fall back to the whole rest of the internet right away.
OpenLDAP was originally using Berkeley DB, until recently. they'd worked with it for years, and got fed up with it. in order to minimise the amount of disruption to the code-base, LMDB was written as a near-drop-in replacement.
LMDB is - according to the web site and also the deleted wikipedia page - a key-value store. however its performance absolutely pisses over everything else around it, on pretty much every metric that can be measured, with very few exceptions.
basically howard's extensive experience combined with the intelligence to do thorough research (even to computing papers dating back to the 1960s) led him to make some absolutely critical but perfectly rational design choices, the ultimate combination of which is that LMDB outshines pretty much every key-value store ever written.
i mean, if you are running benchmark programs in *python* and getting sequential read access to records at a rate of 2,500,000 (2.5 MILLION) records per second... in a *scripted* programming language for goodness sake... then they have to be doing something right.
the random write speed of the python-based benchmarks showed 250,000 records written per second. the _sequential_ ones managed just over 900,000 per second!
there are several key differences between Berkeley DB's API and LMDB's API. the first is that LMDB can be put into "append" mode (as mentioned above). basically what you do is you *guarantee* that the key of new records is lexicographically greater than all other records. with this guarantee LMDB baiscally lets you put the new record _right_ at the end of its B+ Tree. this results in something like an astonishing 5x performance increase in writes.
the second key difference is that LMDB allows you to add duplicate values per key. in fact i think there's also a special mode (never used it) where if you do guaranteed fixed (identical) record sizes LMDB will let you store the values in a more space-efficient manner.
so it's pretty sophisticated.
from a technical perspective, there are two key differences between LMDB and *all* other key-value stores.
the first is: it uses "append-only" when adding new records. basically this has some guarantees that there can never be any corruption of existing data just because a new record is added.
the second is: it uses shared memory "copy-on-write" semantics. what that means is that the (one allowed) writer NEVER - and i mean never - blocks readers, whilst importantly being able to guarantee data integrity and transaction atomicity as well.
the way this is achieved is that because Copy-on-write is enabled, the "writer" may make as many writes it wants, knowing full well that all the readers will NOT be interfered with (because any write creates a COPY of the memory page being written to). then, finally, once everything is done, and the new top level parent B+ Tree is finished, the VERY last thing is a single simple LOCK, update-pointer-to-top-level, UNLOCK.
so as long as Reads do the exact same LOCK, get-pointer-to-top-level-of-B-Tree, UNLOCK, there is NO FURTHER NEED for any kind of locking AT ALL.
i am just simply amazed at the simplicity, and how this technique has just... never been deployed in any database engine before, until now. the reasons as howard makes clear are that the original research back in the 1960s was restricted to 32-bit memory spaces. now we have 64-bit so shared memory may refer to absolutely enormous files, so there is no problem deploying this technique, now.
all incredibly cool.
At some point there will be an article on Wikipedia, that only meets Wikipedia's notability requirements due to media spillover complaining about the notability requirements.
yaaay! :) works for me. wasn't there a journalist who published a blog and used that as the only notable reference to create a fake article? :)
Why weren't the people who participated in last year's deletion discussion able to find the sources you know about?
"a high-performance task scheduling engine written (perplexingly) in Python"
guys, there is this thing, it's called "algorithm"....
yeah.... except that algorithm took a staggering 3 months to develop. and it wasn't one algorithm, it was several, along with creating a networking IPC stack and having to create several unusual client-server design decisions. i can't go into the details because i was working in a secure environment, but basically even though i was the one that wrote the code i was taken aback that *python* - a scripted programming language - was capable of such extreme processing rates.
normally those kinds of speed rates would be associated with c for example.
but the key point of the article - leaving that speed aside - is that if something like PostgreSQL had been used as the back-end store, that rate would be somewhere around 30,000 tasks per second or possibly even less than that, over the long term, because of the overwhelming overhead associated with SQL (and NoSQL) databases maintaining transaction logs and making other guarantees in ways that are clearly *significantly* less efficient than the ways that LMDB do it, by way of those guarantees being integrated at a fundamental design level into LMDB.
I never understood the deletionist mentality on Wikipedia. But there's a whole group of people that want to remove information from the public view.
I semi-understand the idea that this "very important" encyclopedia is "too important" for such things as a page for each character from a game I never played. And somehow by culling these frivolous thing they somehow make wikipedia higher quality on the whole? Maybe? Kinda? I don't think these people understand how search works.
There are the obvious shills and PR people that want to sweep things under the rug. These are nefarious and to be found and fought.
There are fools who think it's expensive to store this information. As if an edit-war to remove it was cheaper.
I understand people don't want articles that are just free advertising. But I doubt anyone is going to delete the page for Monanto.
But fundamentally, I just don't get their worldview.
Never mind what projects use it; what have independent reliable sources written about LMDB?
i've written something and i'm pretty wubwubwubreliawibble oh look pretty coloured lights...
Don't know, don't care. The case against notability was stronger when it was first submitted, likely, but it is certainly hard to defend now. There are mature bindings for most languages, it underpins a number of higher level data stores (including OpenLDAP, of course, but also FineDB, Hustle), and is a supported back-end for a large number of projects (sometimes as a default component, as with CFengine and Zimbra).
Why don't they publish the algorithm on wikipedia instead? Putting a non-popular library on wikipedia seems a bit extreme. It may well include all github projects including mine...
Does any of those things you mentioned constitute an independent, reliable, published source?
I'll start with: LMDB is awesome, and I am super SUPER impressed with OpenLDAP's benchmarks over the last several years. I do not question LMDB's worth.
I'm just not really sure that this letter is evidence thereof. The author got poor performance from a SQL database with no indexing, which degraded as the number of records grew? You don't say! A database that has to do a full scan for reads performs poorly?
Surprise about load average seems equally naive. If you fork a bunch of processes that are doing IO, of COURSE the load increases. Load is a measure of the number of processes not sleeping. That's all it is. I don't understand his surprise that a system steadily doing a great deal of IO would show a lot of time spent in IO calls in profiling.
Reading that letter made me cringe. It didn't help that it sounds like another NSA project.
The author got poor performance from a SQL database with no indexing, which degraded as the number of records grew? You don't say! A database that has to do a full scan for reads performs poorly?
yes. it was that i had to do that analysis in a formal repeatable independent way, which i had never done before, and i was very surprised at the poor results. i was at least expecting a *consistent* and reliable rate of... well, i don't know: i was kinda expecting PostgreSQL to be top of the list and i was kinda expecting it to reach 100,000 or 200,000 records per second... and it just... couldn't. i was *completely* caught off-guard by the need to switch off all the safety checks, and by how dramatic the effect on performance of adding indexes really was.
so it was then by complete contrast that, for example, the py-lmdb benchmarks got an ORDER OF MAGNITUDE better sequential-read-speeds (2.5 million per second) than i was expecting that made me really sit up and take notice.
Surprise about load average seems equally naive. If you fork a bunch of processes that are doing IO, of COURSE the load increases. Load is a measure of the number of processes not sleeping. That's all it is. I don't understand his surprise that a system steadily doing a great deal of IO would show a lot of time spent in IO calls in profiling.
you've missed the point. it was that the exact same design using 20 (or so) shm file handles instead of 200 file handles opening to the exact same data (effectively) resulted in a reasonable loadavg, whereas having the 200 file handles open had a loadavg that ground the system completely to a halt.
so it's not the *actual* loadavg that is relevant but that the *relative* loadavg before and after that one simple change was so dramatically shifted from "completely unusable and in no way deployable in a live production environment" to a "this might actually fly, jim" level.
>wasn't there a journalist who published a blog and used that as the only notable reference to create a fake article? :)
I can recommend you a fascinating pair of books: The Secret History of the War on Cancer by Devra Davis and The Merchants of Doubt by Naomi Oreskes. There is a very long history of circular self-reference among dishonest journalists and scientists; for example Fred Singer would write a letter to the Wall Street Journal, then write an Op-Ed piece for a smaller outfit using the Wall Street Journal as a reference, then write an article for WSJ referencing the op-ed. In each case the claimed accuracy of the sources would be boosted - first in the letter it might make a bald claim like "tobacco is proven not harmful" or "global warming is beneficial" and the the op-ed would go on to state that "the wall street journal says tobacco is proven not harmful" then in the final piece "prominent scientists have repeatedly proven that tobacco is not harmful" (Singer really is a physicist or something like that). Eventually the final WSJ article would be cited in thousands of journals and papers funded by Singer's paymasters - this is still going on, the articles are still cited today. Read the books to find out more.
How would you download it or any software using it if it weren't?
You should pipe it to /dev/nul. That's webscale.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
You should pipe it to /dev/nul. That's webscale.
don't jest... please :) jokes about "you should just have a big LED on the box with a switch and a battery" _not_ appreciated :)
but, seriously: the complete lack of need in this application for joins (as well as any other features of SQL or NOSQL databases) was what led me to research key-value stores in the first place.
CPython is a compiler. It compiles Python source code to Python bytecode, and the Python runtime executes the compiled bytecode.
CPython has one major weakness, the GIL (global interpreter lock). I've seen the GIL harm high-throughput, multi-threaded event processing systems not dissimilar from the one you describe.
If you must insist on Python and want to avoid multi-threaded I/O bound weaknesses of the GIL, then use Jython.
-=/\- Jizzbug -/\=-
Facepalm. A source is a publication written on a particular subject, such as a book, magazine article, academic paper, etc.
Wikipedia editors aren't allowed to have opinions about a topic. The Neutral Point of View policy mandates that edits be deleted or re-written to present a reasonably neutral description of a topic. (And if needed, a neutral description of the sides in a controversial topic.)
Wikipedia editors aren't allowed to "know stuff" about a topic. The No Original Research policy mandates that facts and information must be Verifiable in published Reliable Sources. The sources need to exist, even if they aren't cited. Any information which is challenged, or is likely to be challenged, can be removed or tagged with {{citation needed}}.
Wikipedia editors aren't allowed to decide how "important" a topic is. This one causes the most confusion. Wikipedia's has a specific and somewhat unusual definition of Notability. Wikipedia Notability means that multiple independent Reliable Sources have published significant discussion of the subject. A musician who barely shows up at the #100 slot on a Billboard-top-100 list is Notable because The Wold has created the Billboard top-100 list to Take Note of musicians, and because a few paragraphs about the musician here and there in magazines give us Verifiable information from which to build an article. A Youtuber with more fans than the musician isn't Notable because (generally) books and magazines and the news don't publish any discussion of popular Youtubers. That means we have no independent sources from which to build an article.
So.... the reason this article was deleted rather than tagged "needs more verifiable sources" was that the number of independent usable sources was ZERO when it was nominated for deletion, and because everyone who participated in the deletion discussion did a search for more sources and came up with ZERO.
You can't built a valid Wikipedia article without verifiable sources, and you can't fix a broken article by adding sources to when the sources don't exist.
People can't write Wikipedia articles about themselves saying how awesome they are, or their company, or their pet project. (Well, they can write the article, but it will be deleted if it doesn't cite multiple independent published Reliable Sources discussing the subject).
It doesn't matter how awesome someone thinks their Python-LMDB project is. It doesn't matter how important someone thinks their Python-LMDB project is. If there's no magazines or books or news talking about it, then it's a dead-duck under Wikipedia Notability policy. We can't build an article based on just their own promotional materials, and editors can't just claim "personal knowledge" to make up stuff to write an article.
And no, this lame Slashdot story won't change that. ~~~~
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.
CPython is a compiler.
it's an interpreter which was [originally] based on a FORTH engine.
It compiles Python source code to Python bytecode,
there is a compiler which does that, yes.
and the Python runtime executes the compiled bytecode.
it interprets it.
CPython has one major weakness, the GIL (global interpreter lock).
*sigh* it does. the effect that this has on threading is to reduce threads to the role of a mutually-exclusive task-switching mechanism.
I've seen the GIL harm high-throughput, multi-threaded event processing systems not dissimilar from the one you describe.
yes. you are one of the people who will appreciate, given that the codebase could not be written in (or converted to) any other language, due to time-constraints, that using processes and custom-written IPC because threads (which you'd think would be perfect to get high-performance on event processing because there would be no overhead on passing data between threads) couldn't be used, means that the end-result is going to be... complicated.
If you must insist on Python and want to avoid multi-threaded I/O bound weaknesses of the GIL, then use Jython.
not a snowball in hell's chance of that happening :) not in a milllion years. not on this project, and not on any project i will actively and happily be involved in. and *especially* i cannot ever endorse the use of java for high performance reliable applications. i'm familiar with python's advantages and disadvantages, the way that the garbage collector works, and am familiar with the size of the actual python interpreter and am happy that it is implemented in c.
java on the other hand i just... i don't even want to begin describing why i don't want to be involved in its deployment - i'm sure there are many here on slashdot happy to explain why java is unsuitable.
there are many other ways in which the limitation of threads in python imposed by the GIL may be avoided. i chose to work around the problem by using processes and custom-writing an IPC infrastructure using edge-triggered epoll. it was... hard. others may choose to use stackless python. others may agree with the idea to use jython, but honestly if the application was required to be reasonably reliable as well as high-performance there would be absolutely no way that i could ever endorse such an idea. sorry :)
It doesn't matter how awesome someone thinks their Python-LMDB project is. It doesn't matter how important someone thinks their Python-LMDB project is.
the mistake you've made has been raised a number of times in the slashdot comments (3 so far). the wikipedia page that was deleted was about LMDB, not python-lmdb. python-lmdb is just bindings to LMDB and that is not notable in any significant way.
My theory is that they have a craving to exercise power over others, and Wikipedia deletion is as close as they can get to that goal. If they were intelligent enough and introspective enough to figure out their own motives, they'd be cops, teachers, politicians or drill sergeants. But since they either aren't smart enough or aren't self-aware enough to see this, and they get a righteousness buzz whenever they delete somebody else's work, they'll look for clever-sounding rationales to justify their behavior to themselves so they can continue getting that buzz.
But maybe I'm just cynical.
From an English language point of view? Yes.
Users? like CFEngine ended up here after enjoying instability in BerkelyDB (wedge it hon) and limited deployability with Tokyo Cabinet.
lmdb - also known as mdb at one time - http://symas.com/mdb/
replaces and is better than bdb.
Why don't you use English properly? It has rules like any computer language. And capitialisation is a rule of English.
... only hypothetically, via Pine Digital's business which prominently displays in wikileaks: https://www.wikileaks.org/spyf...
A couple new answers to that question have popped up in the intervening time.
https://www.usenix.org/confere...
https://www.usenix.org/confere...
-- *My* journal is more interesting than *yours*...
I personally know people who wrote articles about themselves or paid someone to write about them, there is no doubt about that. They did fulfill the requirements you've mentioned, yes, but the articles are longer and mention more publications than a usual article about a (relatively unknown) person of interest would contain.
Wikipedia editors aren't allowed to "know stuff" about a topic. The No Original Research policy mandates that facts and information must be Verifiable in published Reliable Sources.
Thank you. That seems to explain why Bjork's dress has it's own wikipedia article.
It is dangerous to be right when the government is wrong.
You mixed up the policies. No Original Research is unrelated to why Bjork's Academy Awards dress has it's own Wikipedia article. No Original Research is why the article doesn't contain any new ideas or opinions by the article-writers themselves. The article accurately describes what The World has to say about the dress. The article has 13 sources cited 18 times providing external documentation for almost every sentence in the article.
The policy you wanted was "Wikipedia editors aren't allowed to decide how 'important' a topic is... Wikipedia Notability means that multiple independent Reliable Sources have published significant discussion of the subject." The World decides what is and isn't Notable, not me. As a Wikipedia editor I'm not allowed the opinion that it's embarrassment to humanity that Academy-Awards-Dresses are considered newsworthy. (I can have the opinion, but I can't delete the article based on my opinion.)
The sources include: telegraph.co.uk, shine.yahoo.com, Filmology: A Movie-a-Day Guide to the Movies You Need to Know ISBN 978-1-4405-0753-3, All about Oscar: The History and Politics of the Academy Awards ISBN 978-0-8264-1452-6, Vanity Fair magazine, Spin magazine, New York magazine, Reel Winners: Movie Award Trivia ISBN 978-1-55002-574-3, BjÃrk: wow and flutter ISBN 978-1-55022-556-3, The Advocate magazine, today.msnbc.msn.com. And there is no doubt that there are countless other uncited sources that exist. The World has clearly decided that this topic is worthy of significant published coverage.
By the way, this particular article has been getting around 55 pageviews a day. That's a lot higher than many of our more serious minor topics. Apparently there are a fair number of people coming to Wikipedia searching for this article.
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.
Ah yes, because book and magazine authors are more reliable judges of veracity than the software developers whose projects form the very fabric of the internet.
I see, thank you.
It is dangerous to be right when the government is wrong.