Slashdot Mirror
A New Approach To Database-Aided Data Processing
An anonymous reader writes "The Parallel Universe blog has a post about parallel data processing. They start off by talking about how Moore's Law still holds, but the shift from clock frequency to multiple cores has stifled the rate at which hardware allows software to scale. (Basically, Amdahl's Law.) The simplest approach to dealing with this is sharding, but that introduces its own difficulties. The more you shard a data set, the more work you need to do to separate out the data elements that can't interact. Optimizing for 2n cores takes more than twice the work of optimizing for n cores. The article says, 'If we want to continue writing compellingly complex applications at an ever-increasing scale we must come to terms with the new Moore's law and build our software on top of solid infrastructure designed specifically for this new reality; sharding just won't cut it.' Their solution is to transfer some of the processing work to the database. 'This because the database is in a unique position to know which transactions may contend for the same data items, and how to schedule them with respect to one another for the best possible performance. The database can and should be smart.' They demonstrate how SpaceBase does this by simulating a 10,000-spaceship battle on different sets of hardware (code available here). Going from a dual-core system to a quad-core system at the same clock speed actually doubles performance without sharding."
Surely you mean EVE Online.
This is ludicrous. Paraphrasing: "We do databases, so we'll say that the solution to scaling parallel software resides in databases".
The applications for parallel processing are many and diverse. Databases are relevant to only some of them.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
We need to have greater code modularity along with the utilization of algorithms that can generate and optimize code en masse. That will allow programmers not to 'dirty their hands' with the details of how to implement their programs on ever more complex hardware, but rather to deal with the larger picture elements of interface design and general purpose design without worrying about Moore's or Amdahl's "Laws" (of which neither is).
-___-
*Squinty*
What the difference between threading an app and sharding it are? I'm kinda leaning towards writing this off as a bunch of theoretical BS, not the kind that makes sense either. Database servers are the highest load servers on most networks, distributing data process to them sounds idiotic at best.
Yep, it totally ignores cases where multiple threads can be chewing on the same piece of RAM without conflict. My domain is image processing, and as long as each thread can access its own sub-chunk of the image, parallelizing my code takes near-zero overhead. I don't have to split the data into chunks at all.
We can make a new language that can do processing in the database. That way we don't need to get all the rows we want to do operations on. It will look like this: "Select sum(`widgit_count`) from warehouses where state = 4 "
They charge for CPU usage of their DB software. More specifically they charge an arm and a leg for CPU usage of their DB software. Yes, moving more logic to the database sounds like a brilliant idea...
From my perspective it seems like things have been heading in the opposite direction with NoSQL solutions and simpler distributed data stores. Having core logic in both the app layer and the database layer adds quite a bit of complexity.
Also, when did spaceship battles become a database benchmark, and who would freely tightly integrate a performance critical real time application like a computer game with a database. If you have to use one, you can bet it will be your main bottleneck.
This submit is yet another example of how advertisement is disguised as Slashdot article
This "SpaceBase" thing is but a database product
This "parallel processing a-la database" thing is but part of an advertising campaign being pushed by the company "parallel universe" to advertise their "SpaceBase" database package
That's all
Muchas Gracias, Señor Edward Snowden !
Space is what keeps everything from being in the same place. If you can partition your problem spatially, it gets easier. You have to be able to handle interaction across boundaries, though. This is OK as long as you don't have interaction across multiple boundaries. Grid systems have trouble at corners where four grid squares meet.. (There's an argument for using hexes, because you never have more than three hexes meeting at a point.
Hard cases include fast moving objects, big objects, and groups of connected objects that cross multiple boundaries. Back when I did physics engines, I was talking to some people developing a planet-sized MMORPG, and said "Now let us all join hands across the world". Big groan.
Facebook hit this problem. They were college-based originally, and assumed that most interaction would be with people who were physically nearby. As Facebook scaled up, that was no longer true. They needed a lot more bandwidth between their data centers.
I could make more comments on this, but I'm going to go outside and ride my horse at the beach instead.
The problem is that it's hard to optimize parallel-ization for all useful factors/dimensions. Generally optimizing data for one grouping de-optimizes for another.
Replication may improve reading by copying and re-grouping the copies by the different dimensions (often on diff servers), but this makes writing more complex and slow because then the replication and reconstitution of copies for each dimension becomes a bottleneck.
The real problem is that we live in a 3-D universe. If we move to a 12-D universe, then our queries will scream (if we don't go above 11 factors). But they only allow celebrities like Elvis and Hoffa in there.
Table-ized A.I.
I could have sworn that said sharting.
By moving processing to the database, you're implicitly changing from scale-out (parallelism) to scale-up, aren't you? Unless you have a cluster with really, really fast row-level lock processing, the solution for faster DB is usually faster CPU and more memory for transaction buffers, not more computers. Larger buffers puts an additional overhead on memory-to-memory transfer (as well as more lock traffic & the delays that introduces) on a scale-out basis for databases, so the tendency is to scale up a database with fewer, but larger computers.
Innit?
(I'm using the almost archaic term "computers" here to indicate individual processing nodes. Saying "processors" has become ambiguous with the term "cores" I think.)
Do not mock my vision of impractical footwear
All this talk of sharting and spreading loads leaves a horrible taste in my mouth.
This doesn't sound at all ground-breaking. They've basically discovered what Hadoop already does -- if you shard your data, it makes sense to run the processing where the data is, to reduce communication overhead. And Hadoop didn't pioneer the idea, either. It's based on Google's MapReduce, and I'm pretty certain that the ideas go back much further than that.
Software sucks. Open Source sucks less.
How about actually writing efficient software instead of bloating it with useless crap and excess features?
Hardware has gotten better, the market is driving the bling and those improvements end up wasted on crap.
All that counts is explosion.png.
I'm a fan of databases, DSLs, query languages and parallelizing compilers. I think there are huge opportunities to punt problems to all manners of optimizers which dynamically figure out which resources are to be used to crunch a problem. It is in my view inevitable this is the future.
The problem is this only takes you so far. At some level you actually have to design a system that scales and you still have to get into the weeds to do it unless there is some serious human level AI involved.
There is a reason people pay big money for large single system image machines. Not everyone has the luxury of googles and facebooks problems.
Most databases, especially "large" ones wind up being I/O bound. A large database being defined, somewhat arbitrarily, as 3x or more larger than the amount of RAM available to store it.
Sharding is a solution designed to deal with horizontally or vertically partitioning a database. By splitting the database you can store the database on different mass storage systems. In so doing you achieve parallel I/O, which directly addresses the I/O binding.
All this ignores the effects of RAID, drive/controller caches, SSDs and so forth which help but are external to the OS.
Since processor cores are usually running at relatively low utilization rates anyway, adding more cores (or using the existing ones more efficiently) is rarely all that helpful. I'd never turn down an optimization, especially if it was meaningful and nondisruptive but this is perhaps not the place to devote lots of optimizing resources.
Having said all that, there is still something to what they are saying. One of the most scalable systems I know of (single image, commercially available, not a lab demo or v0.0001 pre-alpha whatever) is the IBM iSeries. It achieves scalability by tightly binding the database to the OS which allows the 2 to work better together. You still have to have the I/O channels to support additional processors/cores/whatever, but it's an integrated system and that is done at the time of system configuration.
The result is linear scaling over 2 orders of magnitude processor count increases. Few systems can do that.
However at some point I'd still expect to see scaling problems. Single image systems start to have severe locking problems at high process (and processor) counts. Supercomputers deal with this routinely. Eventually you have to migrate to shared-nothing designs. Which works but introduces significant communications overhead (that's why supercomputer designs prize high speed, low latency network communications).
Really? Wow. Who would have thought ;)
- I've got bad karma because I won't parrot everyone else's opinion
Wasn't that the rationale behind PICK?
"Under Gates, MS used to steal the best ideas; now they steal the bad ones."
Inside Microsoft, that is considered an improvement... in making people suffer. Still, however, Microsoft is forced to release bad operating system versions only every other release. MS doesn't have complete control yet. In the future, all releases will have new problems.
Everyone knows database systems designed for concurrency and parallel processing across threads, cores, servers, and data centers are best used as a bit bucket for a warehouse full of Java VMs.
Why, oh why, didn't I take the Blue Pill?
Distributed across MULTIPLE servers, especially with smaller data, will work well for performance!
This even works with simple text files, albeit even on a single machine (& is very effective with thread usage in combination with it on a multi-core machine)...
So - how do I KNOW this?
Couple ways, actually (in databasing first of all, but more recently, even with this):
Simple -> http://www.start64.com/index.php?option=com_content&id=5851:apk-hosts-file-engine-64bit-version&Itemid=74
That version of my application for host file mgt. has an "update" (version 6.0++) that I have been testing prior to re-release that is FASTER & uses something MUCH like what's being done here in fact...
The updated model takes the single, relatively LARGE custom hosts file data intake (between new data & merging the OLD data, deduplicating & normalizing it) as a single file, & essentially "shards" it too!
See - 1st, I pushed as far as I could go with it (does the job in under 7 minutes on an Intel I7 Core 3970/top-of-the-line afaik) & that was even using "tricks" like upping the CPU priority as well of the parent process thread doing that work...
Now - the new one's architecture varies on that & does the job, better/faster, so far:
I busted the data up as I process it & sent it to process simultaneously on multiple threads, 1 to a 'shard'...
(Threads for as many as the datasets I broke the single file into)
I cut the time down to THAT, albeit on a FAR lesser machine in testing here @ home!
(Intel I7 Core 920 - basically "middle of the road" in performance CPU's from INTEL nowadays, if that (it changes so far, more cache etc./et al on the better more costly chips - just like the one above, which I had a gamer I know who ALWAYS has the "latest/greatest" equipment test for me... same data I used, same all the way around, EXCEPT for my use of "sharding" (sort of))).
Thus - It shows you an algorithm/technique IS superior to "brute horsepower" which you should only use as a last resort...
APK
P.S.=> Now, above ALL else here - I've been doing database work in information systems since 1994 professionally, & anybody that's been doing it for awhile WILL tell you this - watch it with "big" JOINS: They're an example of where things get "too hairy" & SLOW...
So, by doing this 'sharding' technique, they're doing essentially what I noted above in my app that works on a single text file, & working across multiple smaller parts that make up a whole ordinarily & seeing speed/performance boosts!
That'd only get BETTER in a distributed computing scenario (just like how commonplace multiple hundreds of PC's can outdo supercomputers in a SINGLE rig)...
Breaking it into multiple parts for threaded concurrent processing IF NOT across multiple machines, will work...
Lastly: I even noted that here when I 1st built it, to a "naysayer" on HOW I was going to "eke more oomph" outta it using the SAME BASIC IDEA of busting the data up into smaller parts & using threads to get more outta it, AROUND THIS TIME LAST YEAR no less -> http://slashdot.org/comments.pl?sid=2019504&cid=35365340 ...
No question it works, and the 'variation' they're doing above will too, no questions asked! LOL, makes me wonder IF they "bit off my idea" (there's very little original thought though, but you never know, however I'd think that anyone with a bit of experience in DB work would know this or come to that realization independently, anyhow)...
... apk