Why does this make sense to you? Have you ever used ebay? The obvious solution is to adjust the transaction cost to $29 directly between the two buyers. ebay does this as the market maker, as would 99% of trading firms. No HPT needed. I'm sure there is a use-case where HPT gives value-add, but this isn't it.
The conversion of advertisement mail to spam completely ignores the fact that SMTP is a dying medium because of the signal-to-noise ratio problem. zombie servers on the internet are primarily spam-relays. The goal of the spam now is to look as legitimate and eye-drawing-worthy as computationally possible.
The thesis of TFA is that order stuffing is to effectively denial-of-service all your competitors. If the system can only handle 6,000 orders per second, and you issue 5,950 'fake' orders and 50 legitimate orders, then your computers only have to process 50 orders, but every other computer in the network has to run at max capacity processing 5950 [effectively] self-cancelling orders. Their ability to statistically analyze all 6000 orders will not be as accurate as you.
You realize that mmap is essentially programmatic swap right? That if you do random file-seeks on a large (hundred+ gig) file, that you're relying on the file-system cache 'swapping' (though not to a swap-file). And that both of these are very similar to explicit paginated disk-in / disk-out with the O_DIRECT flag.
I've programmed app-level caches, and am very aware of the memory/disk trade-offs, so I completely understand where he's coming from (even though I don't know the details of his tool).
What I think you're getting at is an app that assumes it can just use 300gig of RAM, and thus swap the whole OS to disk (e.g./dev/sda2 the swap partition). Maybe he did it and maybe he didn't, but I can certainly see how to apply his algorithm to explicitly swapped RAM (that doesn't hit swap). And I suspect that the performance characteristics would be similar (except for the responsiveness of other apps on the machine). Though for non O_DIRECT file-cache based super-sized apps, you'd have similar peer-app stalls.
Did you read the article? Not that it's terribly innovative or anything, but he's not saying anything about swap is slow.. He's saying when you need to page to disk (e.g. a database or disk-spilling cache), don't use traditional B-Tree's or B+Tree's which tend to only store a single level in a memory-page / disk-page. This causes Log(n) disk-lookups to find the leaf-node (where all B+Tree data lives, and half of B-Tree data lives). Instead store multiple levels of the tree in a contiguous mem/disk page. It's a relatively simple re-organization, and I don't think he's scaling out as well as he's suggesting (when you get to 1B + records - you're not going to get a large number of the 20 level lookups in a single vertical slice).
If I pay for electricity at home, why should I be forced to pay for it again at work.. Or at the mall. Or when I'm overseas.. That's not fair.. waaaaaaah.
Yeah, but they did this better in Object-C; called it message-passing. Pass a message to an object and see if it can handle it, if not, have some default symbolic handler. The big bonus is that you do your normal static stuff in C, then identify this quasi language stuff and put it in objects. Thus best of both worlds.
Perl, Python, Ruby all deal with function calls as hash-map lookups. Perl only has parent-namespace lookup tables. Not sure about Python/Ruby.
Guess what, you can do this in Java/C# as well with.. drum-roll-please... A HashMap!!! Sure it's not pretty looking, but it's a hell of a lot prettier than the equivalent parameterized passing in perl:
sub foo {
my %args = @_;
my $x = $args{x}; }
As for the 'message-passing' in java, You can do this with reflection (generic object proxies). That's how we do unit-testing actually.
Concieness aside, the question is whether this is really a good practice or not. You lose staticness with this model, and Java is all about provable code.
public <T extends SomeCallable> void foo(T t) {t.someCall(); }
We do it all the damn time, and it's a more reliable way to handle the code. I can trace every implementation of SomeCallable so I know if it's being used correctly.
What you're describing is a pseudo-reflective model, which is fine.. You can do that too in Java - though not as concisely, I agree. But reflection makes for coding errors due to the obligatory 'there be dragons here'.
"No they're not. Maybe unfortunately, but they don't even work with primitives in Java."
Think about what you're asking. From almost every other dynamic languages, you don't get true primitives - you have an object handle that contains the primitive bytes or an object like Integer (or worse, a polymorphic object which is very heavy weight and slow for things like ++).
So you get java collections gives you what you want from a logical perspective. The only thing you're missing is the memory efficiency and muteability (you can't just hit ++ on the stored hash value, well actually, yes you can, but via implicit new object instantiation).
At the end of the day, the generics don't change how you code, they just replace your references to Object with a token. Thus if your algorithm requires storage as per-object, you get NOTHING by storing the primitive. The only other possibility is to have the generic-library somehow produce different code for different types of primitives. That would be pretty sweet, BUT, bug-prone if not hard-to-read IMO.
If you want the memory compactness, there are primitive collection data-types out there - no per-item objects used. (try trove).
The single best case for using Java over most every other language is it's tendency to reduce syntactical and logic errors.. Yes, you still make them, but there are entire classes of common errors that go away with java.
Generics solve one problem very well in my opinion.. Nested collections. It is incredibly verbose, to say the least, but if you've ever had a map of a map (repeat ad infinitum), you know what I'm talking about.. You can very easily get your type-casting wrong, and if there is a rare use-case, you won't detect the type-cast exception until it's too late.
By adding strict binding to these generalized utilities, you dramatically increase the readibility and reliability of the code. It becomes a syntax error to insert or extract data of the wrong type. Now if everything is a string for you, this isn't going to help, but I typically deal with complex data-types, and it's a life-saver.
You are right that they are otherwise verbose and actually do decrease performance somewhat (because it forces an extra otherwise useless typecast java-op).
As for startup times.. what exactly are you dealing with where startup times are a problem? eclipse?? Try loading up MS word sometime without the boot-shifted offloading. That was C++ last I checked.. Complex code is slow to load in any language. Something like apache hides the complexity through lazy-loading, but this isn't always possible.
For most things you'd use Java for, there is a long lifetime of the process, so it's worth having a relatively slower startup period to identify JITable code - though it would be nice to cache the JIT metadata/code for future restarts. In fact, SUN Java6 DOES this for it's core libraries, not sure why it can't for dynamically loaded code (via SHA1 hashes or whatever). And of course, most java classes are deflated, thus you have the obligatory decompress stage to any startup process.. Again, SHA1 based metadata could avoid this entirely in theory.
You're about 3 years out of date here - annotations, aspect-oriented programming, and reflective-programming in general have replaced XML (outside of some copy-pastable bootstrap stage).
spring2, EJB3, hibernate3, servlet2.5, along with the proliferation of REST and JSON have obviated the need for XML almost entirely. I certainly have cheered on this post-XML movement.
As for 4 letter acrynyms, consider ruby-on-rails. There's nothing intrinsic about the language that makes this powerful, you can do it in perl/python and even java (it's already been done). The only difference is that it's the only one for Ruby, whereas people have tried it different ways in other languages, because one size doesn't always fit all.
.NET? How is being subservient to a historically abusive MicroSoft better than Oracle, who to my knowledge doesn't have a path that can force java development to be tied to Oracle licenses. Though architecturally equivalent in my opinion. There are as many java sub-languages as.NET languages.
PHP? Perl? Python? Certainly attractive to piercings, but I don't see this running on cell-phones. And my experience is that it isn't as attractive for larger (million-line) projects. Yes I'm aware of very large PHP projects, but I've not found them to be as easily packaged / extensible.
Ruby? I separate from the above 'scripted' languages, only because of the RAD popularity. People move away from this when they hit a performance ceiling. It just doesn't have anything fundamentally appealing to it, save maybe some syntax concieness.
erlang? Wonderfully efficient / highly available language.. Makes my eyes bleed when reading through.
lisp, scala? Less piercing, more snootiness. But plausible.
C, C++? Surprised these haven't adapted to the current web-services climate.. These are excellent languages - though they're just not fun for me anymore.
Objective-C? Need a registered fan-boy card to use, last I heard.
I'm sure I've missed some modern popular languages.
Wow - you still page to disk? Today? On a new machine? Seriously? Mem's like 10% of the machine cost. How much is an extra high perf writing SSD stick?
I totally love RDBMS's, don't get me wrong. You can manipulate schema on-the-fly (more or less). Introduce optimizations independently of the source-code. You don't have to think about fringe cases, or data-integrity. But when a project grows to a certain point, you have two decisions: Go to a hyper-expensive RDBMS solution ($100k.. $500k) (for a project that may only be worth $100k), or identify the key bottlenecks and try to re-architect the tables of interest.
I've often found that DRBD+NFS flat file solutions scale 100x that of a mysql solution and with 1/5th the hardware horsepower. 'Cursors are just a tad bit more efficient at the file-system level'. But obviously when doing this you're introducing massive volumes of potential bugs. But 3 hours of debugging relatively basic CIS practices to save you $96k of the $100k Oracle bill is well worth it in my opinion.
To me BigTable style solutions are the middle ground. (though the file-system cursoring is more akin to a Hadoop map-reduce than the explicit table architecture of HBase/Hypertable/Casandra).
[complaining that] "SQL being too hard"? Well, one can assume you can ignore this class of amateurs - there's no lack of free learning tools for SQL - and it's dirt simple.
"And yet a lot of them invent query languages/query languages similar to SQL. " - See, I think you're magically associating two classes of programmers. There are people, like myself that love the expressiveness of SQL over virtually any other language for data-set manipulation. Thus we would like as an optional to utilize SQL on even a simple key-value store. And there are tons of noSQL solutions that provide SQL front ends (hell, there are SQL front-ends to CSV stores). You typically lose efficiency at this point, but for rarely run reports, the lack of bugs makes it worth it.
" Hadoop takes a huge beating via the RDBMSes in performance" - By Hadoop, I assume you mean HBase which runs on top of several layers of technologies - the lowest of which is Hadoop. Naturally this layering produces inefficiencies. Consequently, things like HyperTable came about as functional equivalents of HBase without all the layers (and written in raw C I might add). When people say scales well, they typically mean runs slowly on a given node.. And thus something like HBase requires several dozen machines before it can overtake an optimized single-node (mysql/Oracle/what-have-you). Then when you jack up the performance of the single node-cluster (Oracle RAC), you need a lot more machines before you can overtake. This may not make sense for 90% of companies out there - having 1,000 machines just isn't practical and the maintenance costs will be killer in year-2. For Google it made absolute sense.. They simply can't make a single DB configuration go fast enough. And therein is the driving model that noSQL is trying to replicate.
"Now the funny thing is that Oracle was not included" - yeah funny how Oracle has a clause in their license that says you may NEVER publish performance results.. Guess why.. Makes it easier to suck suckers in to paying $100k, only to find that a mysql setup is faster on the same hardware. Yes, you can spend $200k on Oracle and have it faster than mysql will ever be, but you didn't budge for that when you were suckered in.
"1999 I worked at a place with terabytes of database space" - A peta-byte of archive data is not the same as a 100 gigabyte of actively manipulated data when you can only get 32Gig of RAM on the box (such that any random indexed lookup is almost guaranteed to hit the disk). It's somewhat easy to add disks to a virtual server cloud.. Use iSCSI via 100 mounted partitions into a tablespace that spans all 100 partitions (in linearlyly appended mode) - you can do this with mysql today. Not sure what the limits of LVM are, but you could do it that way too. Not too expensive either if you use cheap 2TB disks in sufficiently raided configurations. This use to be mainframe class (tons of IO with fully redundant hardware was their mantra). But my point is there are problem-spaces that make this not scale with RDBMS - unless you treat the offending table as a simple key-value store, such that you can shard it - thereby not properly utilizing the RDBMS.
"But it seems like an open source parallel database would do a lot to silence many nosql critics" - you're not going to silence people that think of data as simple key-value pairs, or highly specialized full-text-searching (which is related to but independent of RDBMS activity). Or even as log-file-processing (such as apache page-view reporting). These are things that RDBMS isn't the best solution for. It CAN do these things, which is why it's become the multi-use hammer. But, when batch processing a 10 million records a day, I have the choice of having a 30 minute load time in the RDBMS and a pretty heafty sustained load over several hours (due to the random-seeks that can't fit in memory). Or I can just store to a flat text file CSV, and maintain a cursor (I mean file-handle) to the last read item, and both load and process the entire
RDBMS's are optimized for READS, not writes. You can produce a 1,000 machine mysql-INNODB cluster that will be faster than memcached and be fully ACID complaint. But you'll only ever have 1 write node. You CAN do sharded masters with interleaved auto-incremented values, but then your foreign keys are totally out the window - as is your ACIDity. Oracle has clustered lock managers, but very quickly is going to max out it's scalability - especially if it's limited to a single SAN.
Relatively expensive 15,000 RPM disks are going to max out near 15,000 random seeks per second. RAID-10 or even RAID-50 (if you're sadistic) is only going to give you a small constant multiplier to this performance. And if you're maxing out said items, then the SCSI queueing and multi-gig RAID-controller memory cards will buy you mere seconds of peek-performance.. Utterly useless in sustained writes.
SSD's alleviate some of the disk-based limitations, EXCEPT that you are constrained by three factors.. 1) Disk size 2) SSD's like large block-sizes 3) SSD's can't write to one location too often. Thus the modern high performance SSDs do address remapping, which eventually degrades the overall performance. And ironically, while random-IO is faster on SSD's than disks, linear writes seem to be faster on disks than SSDs. Obviously SSD's are still in their infancy. The game may change any year now.
The two core write-scaling problems above are the inter-table dependencies (the foreign keys) and the random-IO necessary for diskhashtable or B+Tree backingstore layouts (factorial-layouts alleviate this somewhat). This also applies to read when you do large M-way joins of multi-giga-record tables. You're essentially requiring over a billion disk seeks to satisfy a single query - completely unmanageable. Yes there were ways to re-architect the product to mitigate this type of query (denormalization, externalized batch journaling, etc) - but your argument was that RDBMS's solve problems - this is a problem that requires hackery to avoid the intrinsic flaw in RDBMS's foreign key/join-key architecture.
Google's BigTable paradigm does away with the need for foreign keys by simply providing a 1-to-many relationship as a 3'rd dimension to the simple flat table. Yes this doesn't solve 4D problems, but MOST RDBMS's could be done away with by simply living in this 3D space.
You could achieve this pattern with existing RDBMS's simply by storing a hash-map in a blob column. But this would not be efficient at all. You'd have to lock the entire row and rewrite the entire blob to change a single value.
BigTable gives you MVCC on each 3D key-value pair with locking to the primary-key of the row. It's a column-oriented database (of which there are many in RDBMSs), but almost all the real meat is in this 3'rd dimension which is stored in a versioned, replicated, append-only manner.
The append-only immuteable data naturally fits a read scaling model.. Once saved to disk, you replicate the recordset to dozens, if not hundreds or thousands of machines (typically on a copy-on-cache-miss model when hitting one of a thousand servers). You then leverage the map-reduce model, to make sure you catch any writing nodes for the given column of interest, then on the reduce, you choose the newest version. Thus you have consistency (unlike some scalable approaches that do an 'eventually consistent' model).
Because of the map-reduced MVCC, you can then shard out writes to random machines.. It literally doesn't matter where the inserts/updates/deletes get written to, because on the next read, only the newest version will be passed to the client. There is some contention in centrally managing which nodes are doing writes, but at least you can spread writes to at least a dozen machines per column.. And with say a dozen columns, that means spreading writes across a hundred nodes. And with a dozen tables, you're over a thousand write nodes (across multiple data-centers or at least isolated networks) (though obviously you
Regional data has nothing to do with BigTable or RDBMS. Have you read the white-papers on BigTable? If google leverages any IP isolated network solutions, then it's at the networking/application level ABOVE BigTable.
BigTable itself leverages map-reduce to cascade the query to potentially thousands of machines, reducing their results back to a SINGLE requesting node.
Geo-location would pick one of several data-centers which house an isolated effective database. The upper layered code would act identically if it was Oracle or BigTable.
The consistent view comes from the fact that all columns have a version (the timestamp). The reduce phase guarantees that only the latest version of a column is returned.. Thus if an update is mid-flight in the replication stage, you'll still get the correct data. Now this is completely separate from multi column updates - though BigTable leverages Chubby which is a clustered locking system which presumably would facilitate multi-table consistent updates. But this falls into the category of problems RDBMS's have solved that you have to fend-for-yourself.
95% of the posts on this topic are on whether the topic should exist on slashdot.. wtf??? Wasting my time trying to actually learn something - weeding through the crap.
Now that I've paradoxically contributed.. I'll link to my various out-of-my non-physisist ass comments to regain karma:) http://slashdot.org/comments.pl?sid=1505312&cid=30719606 http://slashdot.org/comments.pl?sid=1505312&cid=30719560 http://slashdot.org/comments.pl?sid=1505312&cid=30719668
Sure, but the issue is how to do this in the vacuum of space between planets.
I'm still struggling to understand it, but I think the key hints are that "empty space" still has temperature, and temperature has a natural accelerating effect. Certainly planets exchange photons between one another, and thus can apply a forcing function based on the entropy-wells (a term I assume the author would promote overtop of gravity-well).
He talks about black-holes having a high temperature surface - so perhaps what he's describing is that a massive body shapes the ambient temperature around around it - and thus the energy that 'emerges' through it - including the temperature-halo of adjacent planets - thus being coupled.
He doesn't talk about the fundamental forces at all. And I believe that's important.
An electric force applies to any two particles, no matter how small, and no matter how far apart (though electric has both positive and negative forces which cancel out over large distances). I'm not as well versed at the strong-nuclear force, but it too apparently dies out at relatively short distances. The electro-magnetic force is so uniform, that layers upon layers of equations have been built that operate at incredible levels of resolution - down to individual photon exchanges - and upwards to star-sized regions.
The 'emergent' force of Gravity, on the other hand is like centripetal force, or elastic force, or the forcing function that can describe the diffusion of perfume. Basically a natural mathmatical mode / pattern which statistically directs atoms such that you can aggregate the motion by a simple Force equation.
The difference is that you can NOT apply the aggregate equation of an 'emergent' force to below a few million/thousand/hundred/dozen atoms (depending on the equation). You can with electro-magnetic equations. So there's a huge effective difference.
I believe you might be suggesting that even the electro-magnetic (fundamental) equations may ultimately be aggregating phenomena as well - and that would be sweet if it was ever discovered to be true. But the author has no basis to address that topic.
I HAVE however seen pseudo-scientists try and approach electro-magnetism as a mechanical result due to the existence of more fundamental particles (the collisions of plank-sized perfectly elastic balls that permeate as an oft-debunked aether).
"Gravity however, opposes entropy, since it pulls particles together, into a lower entropy state."
I'm not well versed on entropy / thermo-dynamics. The basis of making any of this intuitive is an elastic band - which apparently is a good description of a thermal entropy-chamber with an external force stretching a polymer chain. The temperature affects the restorative force which resists the stretching. In general, this force is not considered a fundamental force. It is the result of entropy - of individual atoms traveling random paths such that a lowest energy state is sought. The net migration of atoms - the diffusion can, on a macroscopic level, have a directional force measured. But this isn't a direct force (like weak-electro-magnetic or the strong nuclear force) - instead it's a net-force - aggregating all atomic paths given a particular orientation of matter at a given point in time.
The next critical piece of information is the mathematical representation of the universe as a 2D holographic surface. Or rather, looking at any particular event as a 2D surface that encloses any piece of information we wish to observe/describe. All the matter/energy/information within the surface is described mathmatically by the surface itself.. And thus the surface carriers information.. And consequently has an information density - the author describes a number of bits per unit area of information.
The author describes a maximum possible density - a minimum surface area that can hold a bit. And this is described as the event horizon of a black-hole.. Namely a 2D sphere with 100% information storage.. Any information that is absorbed by the black-hole corresponds to a growth of the sphere such that the total area has increased slightly, and thus can facilitate an extra bit of information.
Thus any region of space can be thought of as having an enclosed surface.. And if there is ANY energy there-in, there will be bits of information on that surface of a corresponding density.
For two surfaces enclosing different sizes of matter/energy, the density of the surfaces will be different.. Likewise, if two surfaces enclosing the same matter are of different sizes, the density will be different.
The final piece is describing a natural migration of this energy density. Namely, that energy/information that 'moves' from one surface to another will be traveling through different information-densities. Much like a gasious atom moving through a medium. The assymetries in the information-surfaces (like the assymetries in the atmosphere) will constrain the degrees of freedom of the energy. There net effect is equivalent - diffusion. Or more generally, that the laws of thermodynamics dictate the aggregate forcing functions used to describe the enclosed system.
The author then uses various equations to bring about entropy to the classical Neutonian F=ma (specifically F = Gm/r^2), and more impressively into red-shift equations for Einsteins relativity. Meaning he's able to relate the classical force of gravity into more-intuitive/tangible elasticity equations.
The end result is that he feels he can do away with action-at-a-distance, space-time, and gravity as a force. By saying that the attractive force of stellar bodies is really the diffusion of energy as defined by the laws of entropy. Whenever you have a 'gradient' between two adjacent arbitrary surfaces, you'll have a diffusion (as you'd naturally expect in fluids / gasses). This gradient typically has a complex measureable path between 2 or more massive bodies.. And thus any matter traveling along those paths will experience reduced degrees of freedom consistent with entropy/diffusion. The net motion can be measured as a forcing function equivalent to Neuton/Einstein. But the important thing to take away is that this is a NET motion.. NOT a natural force exerted on each particle - as a charged electric field would produce.
This is fundamentally why we have so much difficulty trying to incorporate gravity i
Right, but the problem is that, to my knowledge, the only new nuclear power plants being commissioned in the US are merely adding additional nodes to existing 2nd Gen plants. So where are the lessons learned?
Sure we can swap out switches / computers, etc.. But the state of the art is entirely passive based systems - theoretically fool proof.. We're no where ready to deploy those in the US due to politics.
Slashdot Mirror
Why does this make sense to you? Have you ever used ebay?
The obvious solution is to adjust the transaction cost to $29 directly between the two buyers. ebay does this as the market maker, as would 99% of trading firms. No HPT needed. I'm sure there is a use-case where HPT gives value-add, but this isn't it.
Your analysis is way off here.
The conversion of advertisement mail to spam completely ignores the fact that SMTP is a dying medium because of the signal-to-noise ratio problem. zombie servers on the internet are primarily spam-relays. The goal of the spam now is to look as legitimate and eye-drawing-worthy as computationally possible.
The thesis of TFA is that order stuffing is to effectively denial-of-service all your competitors.
If the system can only handle 6,000 orders per second, and you issue 5,950 'fake' orders and 50 legitimate orders, then your computers only have to process 50 orders, but every other computer in the network has to run at max capacity processing 5950 [effectively] self-cancelling orders. Their ability to statistically analyze all 6000 orders will not be as accurate as you.
You realize that mmap is essentially programmatic swap right? That if you do random file-seeks on a large (hundred+ gig) file, that you're relying on the file-system cache 'swapping' (though not to a swap-file). And that both of these are very similar to explicit paginated disk-in / disk-out with the O_DIRECT flag.
/dev/sda2 the swap partition). Maybe he did it and maybe he didn't, but I can certainly see how to apply his algorithm to explicitly swapped RAM (that doesn't hit swap). And I suspect that the performance characteristics would be similar (except for the responsiveness of other apps on the machine). Though for non O_DIRECT file-cache based super-sized apps, you'd have similar peer-app stalls.
I've programmed app-level caches, and am very aware of the memory/disk trade-offs, so I completely understand where he's coming from (even though I don't know the details of his tool).
What I think you're getting at is an app that assumes it can just use 300gig of RAM, and thus swap the whole OS to disk (e.g.
Did you read the article? Not that it's terribly innovative or anything, but he's not saying anything about swap is slow.. He's saying when you need to page to disk (e.g. a database or disk-spilling cache), don't use traditional B-Tree's or B+Tree's which tend to only store a single level in a memory-page / disk-page. This causes Log(n) disk-lookups to find the leaf-node (where all B+Tree data lives, and half of B-Tree data lives). Instead store multiple levels of the tree in a contiguous mem/disk page. It's a relatively simple re-organization, and I don't think he's scaling out as well as he's suggesting (when you get to 1B + records - you're not going to get a large number of the 20 level lookups in a single vertical slice).
If I pay for electricity at home, why should I be forced to pay for it again at work.. Or at the mall. Or when I'm overseas.. That's not fair.. waaaaaaah.
top
M
k
copy-paste ID
works pretty well for me in GUI's
I could figure out the killall / ps equivalent, but I'm too lazy.
Course I always have 30 terminal windows already open. Can't open a new one in swap-hell.
Yeah, but they did this better in Object-C; called it message-passing. Pass a message to an object and see if it can handle it, if not, have some default symbolic handler. The big bonus is that you do your normal static stuff in C, then identify this quasi language stuff and put it in objects. Thus best of both worlds.
Perl, Python, Ruby all deal with function calls as hash-map lookups. Perl only has parent-namespace lookup tables. Not sure about Python/Ruby.
Guess what, you can do this in Java/C# as well with.. drum-roll-please... A HashMap!!! Sure it's not pretty looking, but it's a hell of a lot prettier than the equivalent parameterized passing in perl:
sub foo {
my %args = @_;
my $x = $args{x};
}
As for the 'message-passing' in java, You can do this with reflection (generic object proxies). That's how we do unit-testing actually.
Concieness aside, the question is whether this is really a good practice or not. You lose staticness with this model, and Java is all about provable code.
Ok, no problem:
public <T extends SomeCallable> void foo(T t) {t.someCall(); }
We do it all the damn time, and it's a more reliable way to handle the code. I can trace every implementation of SomeCallable so I know if it's being used correctly.
What you're describing is a pseudo-reflective model, which is fine.. You can do that too in Java - though not as concisely, I agree. But reflection makes for coding errors due to the obligatory 'there be dragons here'.
"No they're not. Maybe unfortunately, but they don't even work with primitives in Java."
Think about what you're asking. From almost every other dynamic languages, you don't get true primitives - you have an object handle that contains the primitive bytes or an object like Integer (or worse, a polymorphic object which is very heavy weight and slow for things like ++).
So you get java collections gives you what you want from a logical perspective. The only thing you're missing is the memory efficiency and muteability (you can't just hit ++ on the stored hash value, well actually, yes you can, but via implicit new object instantiation).
At the end of the day, the generics don't change how you code, they just replace your references to Object with a token. Thus if your algorithm requires storage as per-object, you get NOTHING by storing the primitive. The only other possibility is to have the generic-library somehow produce different code for different types of primitives. That would be pretty sweet, BUT, bug-prone if not hard-to-read IMO.
If you want the memory compactness, there are primitive collection data-types out there - no per-item objects used. (try trove).
The single best case for using Java over most every other language is it's tendency to reduce syntactical and logic errors.. Yes, you still make them, but there are entire classes of common errors that go away with java.
Generics solve one problem very well in my opinion.. Nested collections. It is incredibly verbose, to say the least, but if you've ever had a map of a map (repeat ad infinitum), you know what I'm talking about.. You can very easily get your type-casting wrong, and if there is a rare use-case, you won't detect the type-cast exception until it's too late.
By adding strict binding to these generalized utilities, you dramatically increase the readibility and reliability of the code. It becomes a syntax error to insert or extract data of the wrong type. Now if everything is a string for you, this isn't going to help, but I typically deal with complex data-types, and it's a life-saver.
You are right that they are otherwise verbose and actually do decrease performance somewhat (because it forces an extra otherwise useless typecast java-op).
As for startup times.. what exactly are you dealing with where startup times are a problem? eclipse?? Try loading up MS word sometime without the boot-shifted offloading. That was C++ last I checked.. Complex code is slow to load in any language. Something like apache hides the complexity through lazy-loading, but this isn't always possible.
For most things you'd use Java for, there is a long lifetime of the process, so it's worth having a relatively slower startup period to identify JITable code - though it would be nice to cache the JIT metadata/code for future restarts. In fact, SUN Java6 DOES this for it's core libraries, not sure why it can't for dynamically loaded code (via SHA1 hashes or whatever). And of course, most java classes are deflated, thus you have the obligatory decompress stage to any startup process.. Again, SHA1 based metadata could avoid this entirely in theory.
Ok, good, so you've confirmed that Perl is 10x better than C.. Just checking.
You're about 3 years out of date here - annotations, aspect-oriented programming, and reflective-programming in general have replaced XML (outside of some copy-pastable bootstrap stage).
spring2, EJB3, hibernate3, servlet2.5, along with the proliferation of REST and JSON have obviated the need for XML almost entirely. I certainly have cheered on this post-XML movement.
As for 4 letter acrynyms, consider ruby-on-rails. There's nothing intrinsic about the language that makes this powerful, you can do it in perl/python and even java (it's already been done). The only difference is that it's the only one for Ruby, whereas people have tried it different ways in other languages, because one size doesn't always fit all.
.NET? How is being subservient to a historically abusive MicroSoft better than Oracle, who to my knowledge doesn't have a path that can force java development to be tied to Oracle licenses. Though architecturally equivalent in my opinion. There are as many java sub-languages as .NET languages.
PHP? Perl? Python? Certainly attractive to piercings, but I don't see this running on cell-phones. And my experience is that it isn't as attractive for larger (million-line) projects. Yes I'm aware of very large PHP projects, but I've not found them to be as easily packaged / extensible.
Ruby? I separate from the above 'scripted' languages, only because of the RAD popularity. People move away from this when they hit a performance ceiling. It just doesn't have anything fundamentally appealing to it, save maybe some syntax concieness.
erlang? Wonderfully efficient / highly available language.. Makes my eyes bleed when reading through.
lisp, scala? Less piercing, more snootiness. But plausible.
C, C++? Surprised these haven't adapted to the current web-services climate.. These are excellent languages - though they're just not fun for me anymore.
Objective-C? Need a registered fan-boy card to use, last I heard.
I'm sure I've missed some modern popular languages.
Wow - you still page to disk? Today? On a new machine? Seriously? Mem's like 10% of the machine cost. How much is an extra high perf writing SSD stick?
Did you read the article? The algorithm chosen could produce an infinite loop.
I totally love RDBMS's, don't get me wrong. You can manipulate schema on-the-fly (more or less). Introduce optimizations independently of the source-code. You don't have to think about fringe cases, or data-integrity. But when a project grows to a certain point, you have two decisions: Go to a hyper-expensive RDBMS solution ($100k .. $500k) (for a project that may only be worth $100k), or identify the key bottlenecks and try to re-architect the tables of interest.
I've often found that DRBD+NFS flat file solutions scale 100x that of a mysql solution and with 1/5th the hardware horsepower. 'Cursors are just a tad bit more efficient at the file-system level'. But obviously when doing this you're introducing massive volumes of potential bugs. But 3 hours of debugging relatively basic CIS practices to save you $96k of the $100k Oracle bill is well worth it in my opinion.
To me BigTable style solutions are the middle ground. (though the file-system cursoring is more akin to a Hadoop map-reduce than the explicit table architecture of HBase/Hypertable/Casandra).
[complaining that] "SQL being too hard"? Well, one can assume you can ignore this class of amateurs - there's no lack of free learning tools for SQL - and it's dirt simple.
"And yet a lot of them invent query languages/query languages similar to SQL. " - See, I think you're magically associating two classes of programmers. There are people, like myself that love the expressiveness of SQL over virtually any other language for data-set manipulation. Thus we would like as an optional to utilize SQL on even a simple key-value store. And there are tons of noSQL solutions that provide SQL front ends (hell, there are SQL front-ends to CSV stores). You typically lose efficiency at this point, but for rarely run reports, the lack of bugs makes it worth it.
" Hadoop takes a huge beating via the RDBMSes in performance" - By Hadoop, I assume you mean HBase which runs on top of several layers of technologies - the lowest of which is Hadoop. Naturally this layering produces inefficiencies. Consequently, things like HyperTable came about as functional equivalents of HBase without all the layers (and written in raw C I might add). When people say scales well, they typically mean runs slowly on a given node.. And thus something like HBase requires several dozen machines before it can overtake an optimized single-node (mysql/Oracle/what-have-you). Then when you jack up the performance of the single node-cluster (Oracle RAC), you need a lot more machines before you can overtake. This may not make sense for 90% of companies out there - having 1,000 machines just isn't practical and the maintenance costs will be killer in year-2. For Google it made absolute sense.. They simply can't make a single DB configuration go fast enough. And therein is the driving model that noSQL is trying to replicate.
"Now the funny thing is that Oracle was not included" - yeah funny how Oracle has a clause in their license that says you may NEVER publish performance results.. Guess why.. Makes it easier to suck suckers in to paying $100k, only to find that a mysql setup is faster on the same hardware. Yes, you can spend $200k on Oracle and have it faster than mysql will ever be, but you didn't budge for that when you were suckered in.
"1999 I worked at a place with terabytes of database space" - A peta-byte of archive data is not the same as a 100 gigabyte of actively manipulated data when you can only get 32Gig of RAM on the box (such that any random indexed lookup is almost guaranteed to hit the disk). It's somewhat easy to add disks to a virtual server cloud.. Use iSCSI via 100 mounted partitions into a tablespace that spans all 100 partitions (in linearlyly appended mode) - you can do this with mysql today. Not sure what the limits of LVM are, but you could do it that way too. Not too expensive either if you use cheap 2TB disks in sufficiently raided configurations. This use to be mainframe class (tons of IO with fully redundant hardware was their mantra). But my point is there are problem-spaces that make this not scale with RDBMS - unless you treat the offending table as a simple key-value store, such that you can shard it - thereby not properly utilizing the RDBMS.
"But it seems like an open source parallel database would do a lot to silence many nosql critics" - you're not going to silence people that think of data as simple key-value pairs, or highly specialized full-text-searching (which is related to but independent of RDBMS activity). Or even as log-file-processing (such as apache page-view reporting). These are things that RDBMS isn't the best solution for. It CAN do these things, which is why it's become the multi-use hammer. But, when batch processing a 10 million records a day, I have the choice of having a 30 minute load time in the RDBMS and a pretty heafty sustained load over several hours (due to the random-seeks that can't fit in memory). Or I can just store to a flat text file CSV, and maintain a cursor (I mean file-handle) to the last read item, and both load and process the entire
RDBMS's are optimized for READS, not writes. You can produce a 1,000 machine mysql-INNODB cluster that will be faster than memcached and be fully ACID complaint. But you'll only ever have 1 write node. You CAN do sharded masters with interleaved auto-incremented values, but then your foreign keys are totally out the window - as is your ACIDity. Oracle has clustered lock managers, but very quickly is going to max out it's scalability - especially if it's limited to a single SAN.
Relatively expensive 15,000 RPM disks are going to max out near 15,000 random seeks per second. RAID-10 or even RAID-50 (if you're sadistic) is only going to give you a small constant multiplier to this performance. And if you're maxing out said items, then the SCSI queueing and multi-gig RAID-controller memory cards will buy you mere seconds of peek-performance.. Utterly useless in sustained writes.
SSD's alleviate some of the disk-based limitations, EXCEPT that you are constrained by three factors.. 1) Disk size 2) SSD's like large block-sizes 3) SSD's can't write to one location too often. Thus the modern high performance SSDs do address remapping, which eventually degrades the overall performance. And ironically, while random-IO is faster on SSD's than disks, linear writes seem to be faster on disks than SSDs. Obviously SSD's are still in their infancy. The game may change any year now.
The two core write-scaling problems above are the inter-table dependencies (the foreign keys) and the random-IO necessary for diskhashtable or B+Tree backingstore layouts (factorial-layouts alleviate this somewhat). This also applies to read when you do large M-way joins of multi-giga-record tables. You're essentially requiring over a billion disk seeks to satisfy a single query - completely unmanageable. Yes there were ways to re-architect the product to mitigate this type of query (denormalization, externalized batch journaling, etc) - but your argument was that RDBMS's solve problems - this is a problem that requires hackery to avoid the intrinsic flaw in RDBMS's foreign key/join-key architecture.
Google's BigTable paradigm does away with the need for foreign keys by simply providing a 1-to-many relationship as a 3'rd dimension to the simple flat table. Yes this doesn't solve 4D problems, but MOST RDBMS's could be done away with by simply living in this 3D space.
You could achieve this pattern with existing RDBMS's simply by storing a hash-map in a blob column. But this would not be efficient at all. You'd have to lock the entire row and rewrite the entire blob to change a single value.
BigTable gives you MVCC on each 3D key-value pair with locking to the primary-key of the row. It's a column-oriented database (of which there are many in RDBMSs), but almost all the real meat is in this 3'rd dimension which is stored in a versioned, replicated, append-only manner.
The append-only immuteable data naturally fits a read scaling model.. Once saved to disk, you replicate the recordset to dozens, if not hundreds or thousands of machines (typically on a copy-on-cache-miss model when hitting one of a thousand servers). You then leverage the map-reduce model, to make sure you catch any writing nodes for the given column of interest, then on the reduce, you choose the newest version. Thus you have consistency (unlike some scalable approaches that do an 'eventually consistent' model).
Because of the map-reduced MVCC, you can then shard out writes to random machines.. It literally doesn't matter where the inserts/updates/deletes get written to, because on the next read, only the newest version will be passed to the client. There is some contention in centrally managing which nodes are doing writes, but at least you can spread writes to at least a dozen machines per column.. And with say a dozen columns, that means spreading writes across a hundred nodes. And with a dozen tables, you're over a thousand write nodes (across multiple data-centers or at least isolated networks) (though obviously you
Regional data has nothing to do with BigTable or RDBMS. Have you read the white-papers on BigTable? If google leverages any IP isolated network solutions, then it's at the networking/application level ABOVE BigTable.
BigTable itself leverages map-reduce to cascade the query to potentially thousands of machines, reducing their results back to a SINGLE requesting node.
Geo-location would pick one of several data-centers which house an isolated effective database. The upper layered code would act identically if it was Oracle or BigTable.
The consistent view comes from the fact that all columns have a version (the timestamp). The reduce phase guarantees that only the latest version of a column is returned.. Thus if an update is mid-flight in the replication stage, you'll still get the correct data. Now this is completely separate from multi column updates - though BigTable leverages Chubby which is a clustered locking system which presumably would facilitate multi-table consistent updates. But this falls into the category of problems RDBMS's have solved that you have to fend-for-yourself.
And in other news, Debeers bought NASA today...
95% of the posts on this topic are on whether the topic should exist on slashdot.. wtf??? Wasting my time trying to actually learn something - weeding through the crap.
:)
Now that I've paradoxically contributed.. I'll link to my various out-of-my non-physisist ass comments to regain karma
http://slashdot.org/comments.pl?sid=1505312&cid=30719606
http://slashdot.org/comments.pl?sid=1505312&cid=30719560
http://slashdot.org/comments.pl?sid=1505312&cid=30719668
Sure, but the issue is how to do this in the vacuum of space between planets.
I'm still struggling to understand it, but I think the key hints are that "empty space" still has temperature, and temperature has a natural accelerating effect. Certainly planets exchange photons between one another, and thus can apply a forcing function based on the entropy-wells (a term I assume the author would promote overtop of gravity-well).
He talks about black-holes having a high temperature surface - so perhaps what he's describing is that a massive body shapes the ambient temperature around around it - and thus the energy that 'emerges' through it - including the temperature-halo of adjacent planets - thus being coupled.
He doesn't talk about the fundamental forces at all. And I believe that's important.
An electric force applies to any two particles, no matter how small, and no matter how far apart (though electric has both positive and negative forces which cancel out over large distances). I'm not as well versed at the strong-nuclear force, but it too apparently dies out at relatively short distances. The electro-magnetic force is so uniform, that layers upon layers of equations have been built that operate at incredible levels of resolution - down to individual photon exchanges - and upwards to star-sized regions.
The 'emergent' force of Gravity, on the other hand is like centripetal force, or elastic force, or the forcing function that can describe the diffusion of perfume. Basically a natural mathmatical mode / pattern which statistically directs atoms such that you can aggregate the motion by a simple Force equation.
The difference is that you can NOT apply the aggregate equation of an 'emergent' force to below a few million/thousand/hundred/dozen atoms (depending on the equation). You can with electro-magnetic equations. So there's a huge effective difference.
I believe you might be suggesting that even the electro-magnetic (fundamental) equations may ultimately be aggregating phenomena as well - and that would be sweet if it was ever discovered to be true. But the author has no basis to address that topic.
I HAVE however seen pseudo-scientists try and approach electro-magnetism as a mechanical result due to the existence of more fundamental particles (the collisions of plank-sized perfectly elastic balls that permeate as an oft-debunked aether).
"Gravity however, opposes entropy, since it pulls particles together, into a lower entropy state."
I'm not well versed on entropy / thermo-dynamics. The basis of making any of this intuitive is an elastic band - which apparently is a good description of a thermal entropy-chamber with an external force stretching a polymer chain. The temperature affects the restorative force which resists the stretching. In general, this force is not considered a fundamental force. It is the result of entropy - of individual atoms traveling random paths such that a lowest energy state is sought. The net migration of atoms - the diffusion can, on a macroscopic level, have a directional force measured. But this isn't a direct force (like weak-electro-magnetic or the strong nuclear force) - instead it's a net-force - aggregating all atomic paths given a particular orientation of matter at a given point in time.
The next critical piece of information is the mathematical representation of the universe as a 2D holographic surface. Or rather, looking at any particular event as a 2D surface that encloses any piece of information we wish to observe/describe. All the matter/energy/information within the surface is described mathmatically by the surface itself.. And thus the surface carriers information.. And consequently has an information density - the author describes a number of bits per unit area of information.
The author describes a maximum possible density - a minimum surface area that can hold a bit. And this is described as the event horizon of a black-hole.. Namely a 2D sphere with 100% information storage.. Any information that is absorbed by the black-hole corresponds to a growth of the sphere such that the total area has increased slightly, and thus can facilitate an extra bit of information.
Thus any region of space can be thought of as having an enclosed surface.. And if there is ANY energy there-in, there will be bits of information on that surface of a corresponding density.
For two surfaces enclosing different sizes of matter/energy, the density of the surfaces will be different.. Likewise, if two surfaces enclosing the same matter are of different sizes, the density will be different.
The final piece is describing a natural migration of this energy density. Namely, that energy/information that 'moves' from one surface to another will be traveling through different information-densities. Much like a gasious atom moving through a medium. The assymetries in the information-surfaces (like the assymetries in the atmosphere) will constrain the degrees of freedom of the energy. There net effect is equivalent - diffusion. Or more generally, that the laws of thermodynamics dictate the aggregate forcing functions used to describe the enclosed system.
The author then uses various equations to bring about entropy to the classical Neutonian F=ma (specifically F = Gm/r^2), and more impressively into red-shift equations for Einsteins relativity. Meaning he's able to relate the classical force of gravity into more-intuitive/tangible elasticity equations.
The end result is that he feels he can do away with action-at-a-distance, space-time, and gravity as a force. By saying that the attractive force of stellar bodies is really the diffusion of energy as defined by the laws of entropy. Whenever you have a 'gradient' between two adjacent arbitrary surfaces, you'll have a diffusion (as you'd naturally expect in fluids / gasses). This gradient typically has a complex measureable path between 2 or more massive bodies.. And thus any matter traveling along those paths will experience reduced degrees of freedom consistent with entropy/diffusion. The net motion can be measured as a forcing function equivalent to Neuton/Einstein. But the important thing to take away is that this is a NET motion.. NOT a natural force exerted on each particle - as a charged electric field would produce.
This is fundamentally why we have so much difficulty trying to incorporate gravity i
Right, but the problem is that, to my knowledge, the only new nuclear power plants being commissioned in the US are merely adding additional nodes to existing 2nd Gen plants. So where are the lessons learned?
Sure we can swap out switches / computers, etc.. But the state of the art is entirely passive based systems - theoretically fool proof.. We're no where ready to deploy those in the US due to politics.