I think the single best way to find good programmers is to find enthusiasts.
The reason is that it's easier to determine how enthusiastic someone is than how good a product they develop:
- enthusiasts usually have side projects
- enthusiasts often create libraries of code that can be reused
- enthusiasts will have a variety of favorite tools - and can explain why they like them
- enthusiasts will likewise have a variety of favorite methods - and can explain why they like them
- enthusiasts read widely in their field
- enthusiasts know the names of those who have made impacts on their field
- enthusiasts often find themselves putting in too many hours - because they *enjoy* the work
- enthusiasts gravitate together - put them in a room together and you'll have a lively conversation
And there are technologies, methods and tools that attract enthusiasts. For example, I've found that even if python and ruby aren't the most marketable languages out there - they are great ways to find the enthusiasts.
Of course, this won't help a manager that lacks enthusiasts on his team. But a technical manager who is himself an enthusiast, and builds such a team should be able to easy find more. At least in my humble opinion.:-)
> Specialization increases productivity and, therefore, enriches the entire nation.
But there are hidden costs not included here: - In the US auto industry design was specialized - with separate teams for each part of the vehicle. And they found that the Japanese were dedicating a team to a new vehicle for 18-24 months to handle all design. By moving from specialization to generalization they got better quality, fewer mis-matched interfaces, and far better speed. - In meat-packing you don't just get boutique-level quality from a 'master butcher' - you reduce diseases such as salmonella and e coli, reduce worker injuries and reduce the need to overuse antibiotics. - In healthcare the interfaces between specialists are points in which information is lost - resulting in specialists often making incorrect decisions. And while there is more money to be made as a specialist than a generalist - there's been a movement to encourage more doctors to become generalists - family practitioners and a recognition that you need more than nurses and specialist doctors. - In IT again - an assembly-line approach generally results in poorer quality of service since servers and applications come in too many forms and are too customizable to allow complete standardization. Additionally, many issues like performance touch on too many pieces to be handled easily by specialized individuals. Lastly, when analyzing one company's time to create e-comm web sites using a j2ee platform years ago I found it to be slower and *far* more expensive per page than creation of cobol/cics applications. The cause was specialization - ejb developers vs jsp developers vs html developers vs dbas vs qa, etc, etc. - In business leaders generally want their employees to be decision-makers and aware of how their contributions map to corporate objectives.
I'm not saying that all specialized skills are bad, but specialized skills without a general knowledge foundation are bad. And I'm not saying that specialized roles are bad - but any process that only has specialist roles is brittle, isn't human-oriented and probably has a lot of hidden costs.
>>"introducing a new division of labor that eliminated the need for skilled butchers"
> That's just a good example of optimizing workforce utilization. >Same thing with skilled butchers not being used to do jobs that a person with no skills and no English can perform.
As Heinlein said, "specialization is for insects". What might appear to be job optimization ends up really just meaning:
- that the workers gets a miserable and repetitive job
- that workers don't get to see the 'whole picture' - and so have no ability to work around unanticipated problems
- that the end result is only better if you only measure cost
I've been in IT shops that insist that everyone do just one specialized thing - and a simple server problem results in a dozen 'matrixed' people on a call for ten hours failing to figure out what needs to happen. Previously, a single admin with a fuller set of responsibilities would have figured the problem out very quickly - or would never have let the circumstances result in the problem anyhow.
Likewise, in meat packing - you've got a really miserable and dangerous job that now nobody wants unless they're really desperate. Which mostly means non-english speaking people who aren't in a position to complain. Or to get worker's comp for the repetitive stress injuries they receive. And who aren't in a position to determine that an animal should be rejected due to disease. Etc. But that's all ok - since
- the company only very seldom has to pay for injured workers
- the us now has very few meat inspectors
- the cattle practically bathed in antibiotics to cope with bizarre and unsanitary conditions
But if a person is just looking at the cost for a pound of meat at the store and not concerned about overuse of antibiotics, chances of getting ecoli, what happens to the workers whose lives were destroyed at the plants. Then yeah, it's probably a good deal.
> Pardon my conspiracy theory, but hasn't the government been spying on us, well, forever?
yeah - to various levels of effectiveness and efficiency:
Data collection: 50 years ago it took a ton of paperwork and a massive staff to collect data on a lot of people. Typically, you'd just target a fraction of the population, or collect a little info on everyone.
Personal aggregation: this data could not be (for all practical purposes) aggregated by person
Profile aggregation: again, due to reasons of economy, the data by person could not be aggregated by profile
however, now:
You've got handy electronic records of everything you buy, everywhere you go, every web site you visit, everyone you call, and every message via sms and email. So, data collection is no longer extremely expensive - in fact it can be relatively trivial.
The challenge is in getting "common keys" between all these sources - so that they know that ken@acme.com is the same person as ken@hp.com who lives in the same household that visits salon.com. This part is still very expensive to do completely. So, aggregation is still a challenge - but the problem is being whittled away at by these departments and organizations.
The good news is that this is difficult to do well - especially given government bureaucracies.
> I say privacy is pretty much a thing of the past.
As we lose privacy we'll lose freedoms, so you better hope not. There will be little effective opposition to a majority government that controls this info - and can easily blackmail opposition figures over uncovered items. There will be enormous pressures to conform to perceived normalacy when deviations from it result in raised TerroristScore, lowered PatriotScore, or whatever.
Imagine for a second trying to get onto a plane past a security guard that notices your TerroristScore is a little high. Unknown to both you and him this is because you frequent a middle eastern falafal stand frequented by a few fundamentalist moslems that donate to Hamas. This guy isn't very good with numbers, isn't very good at understanding that your score is a little high - it's black & white with this guy. You're not getting on that plane, and you will be treated like a terrorist.
Or imagine getting denied a job or health insurance because you work out at a gym where there's a lot of gay men with aids. This has affected your HealthCostScore, even though you have no idea about this - the system will automatically reject your application.
Giving up on privacy in an age of information is like giving up on food. The only difference is that one will kill you faster than the other.
> unless SQL data representation grows up to modern non-fortran-like OO semantics MySQL will proliferate
You do realize that in activities like reporting sql and its set-based operations are far, far, far faster and easier to work with than oo implementations, right?
You can set up a typical star-schema and have certain tools (like microstrategy) immediately recognize it and generate queries for you. These queries will typically perform just fine and allow very powerful and fast drill-downs, drill-across, etc.
And oo approach involving the marshalling of millions of objects related to entries in the database would take forever to build to this kind of flexibility and would run slow as molasses. Might make the oo purists happy, but the customers would never use the product. At least against non-trivial amounts of data.
> Though, I must admit I can't see it running very fast compared to software DESIGNED for that type of platform.
Actually, it's probably the fastest solution out there for heavy reporting/analytical workloads on windows, linux or unix. Not sure about teradata and informix - haven't looked at them in quite a while.
Teradata, informix and db2 were doing the 'beowulf' thing years before anyone on slashdot asked what a beowulf cluster of these would be like. So, you can easily distribute data across a hundred db2 server blades to get very fast response times over terabytes of data. This can be far faster at the high-ends than the typical oracle scenario of range partitioning on a 32-way smp. And oracle grid is more failover oriented than performance oriented, so that doesn't really compete either, last I looked.
I've built and managed very large databases on oracle, db2 and sql server in the last five years. Of them all, db2 on unix was the easiest to build and manage. It did take a little extra time - since some of the tools are a little finicky, but the manageability and performance was far better than with either oracle or sql server. And with very large data - far, far better than with mysql or postgresql.
> MySQL, as others have pointed out, has better developer support and they know their target audience. They supply a fast, > easy to use database for those who don't need a whole lot. > Oracle supplies an enterprise level database that MySQL doesn't aspire to. > PostgreSQL doesn't know where to fit in.
This is an oversimplification. Each vendor sees itself in all markets:
- oracle/db2/sql server have free versions for tiny apps and very expensive versions for massive apps
- mysql says it doesn't want to do what oracle does, but also says that this is less than 1% of the market - and knows that plenty of smallish databases are on oracle
- postgresql like the others sees itself doing anything from very small databases to very large ones (often via Enterprise DB or other vendor extensions)
And using a single product for multiple sizes isn't illogical: if you have any very large databases (hundreds of gbytes or more) then you probably have a few dozen little ones as well. It's *far* easier to manage them all on oracle/db2/sql server - even with the small additional licensing costs - than to have a frankenstein collection of products to manage.
"Best tool for the job" is a good consideration when evaluating products (along with vendor viability, cost, etc, etc) - but once you've got a single tool in house to keep adding new products - each with their own licensing, support, patch, backup/recovery procedures, etc is a nightmare. Let alone actually federating your data - and having to test out how to virtualize or replicate data from oracle 10.x.x with mysql 5.y.y
> Performance is one aspect of the price tag, but it is certainly not the only factor. Very true - and for that reason Postgresql has more going for it than many alternatives, like:
- best licensing options - you don't need to pay a lawyer to go over your contract or license like you should if you use oracle or mysql commercially. And there's no fear that the vendor will change its license terms once you're locked in and start charging an arm and a leg.
- very good foundation - postgesql isn't built from duct tape and bailing wire. The functionality within it is well tested and robust.
- great support for standard database features - whether its subselects, stored procedures, triggers, etc - it's very simple to move from oracle to postgresql.
- great ansi sql support - again, very standard sql - no unnecessarily propretary language elements.
So, yeah - just because Postgresql is performing well on some benchmarks that doesn't mean you should immediately throw out oracle in favor of postgresql. On the other hand, you also shouldn't discard it because it is a good general purpose database solution.
No, read it years ago and thought it a waste of paper. Mostly mindless patriotism. Not worth rereading to double-check the industry/government cooperation that I thought I remembered. So, I'll cave first.:-)
Around 1850 engineers were looking to redesign London's sewer system and determined that Rome's Cloaca Maxima and Colosseum had the most modern sewer systems in the world - able to handle the waste disposal of 50,000 people about 2000 years ago.
So, in the 19th century London applied lessons learned from the colosseum 2000 years before. Between that and the various mechanisms for raising a cage out of the floor, or flooding the entire things for water fights or raising sails to shade the spectators - I'd call that a wonder.
this is why the internet isn't cool - there's nothing really to see
why computers aren't cool - you can't see the programs run
why antibiotics aren't cool - you can't see how they work
and so, if I've got a description of the great pyramids, colossus of rhoads, or great wall of china - the utter vastness of them, the timelessness of them and the unimaginable difficulties with which they were built...i'm not impressed. Because I can't see it in the flesh. Bah, I say.
well - intense nationalism, intense militarism and integration of the government and corporate concerns. I'd think that would be a good description of fascism, wouldn't it?
holy shit, original paperbacks going for $70. Note to self: must dig thru paperbacks to see if I still have this. Anyhow, this was a great parody of the fascist sci fi from heinlen...
> Who cares if a handful of morons have specific requirements for large processors with large I/O bandwidths?
The 'morons' you refer to include just about anyone analyzing large sets of data. Reporting systems, data mining, personalization, search engines, etc are all io-intensive activities that can work very well on mainframes. This covers a hell of a lot of very interesting ground - certainly more interesting than the typical CRUD app.
> Linux grid takes out firm's aging mainframe > By Jack Loftus, News Writer > 04 Jan 2007 | SearchOpenSource.com
So, this consulting firm sends out a marketing press release that describes how they updated a 20-year-old bad design managing 4.5 pedabytes of data. And only got a 70% performance increase.
I'd call that a total failure.
Sorry, but if the best claim that a white-washing consulting press release is 70% for moving an 80's mainframe to an intel grid consisting of 118 circa 2007 cpus - that's pathetic.
Not to say that I'd prefer development on a mainframe - unless you've got dedicated lpars the meetings to collaborate on changes are too much of a pain. I prefer dedicated unix or linux boxes. But this is just bullshit.
> Why bother with either. Just include sqlite and be done with it.
Sure, nice little database. Not much for multiple users and large data tho...
> Rails can take care of all the data integrity for you anyway.
ah, no - applications do a horrible job of managing data constraints over time. That is, you might test the bajesus out of your code this version and next - but are unlikely to test how code with this version handles data created with the code ten versions before. None of the data we're talking about here is wrong - it's just different valid values.
And if you do create a ridiculous version-compatibility layer in ruby then you'll end up without the ability to create a simple report using sql. Since the most common reporting tools all use sql (most powerful ones generate sql) you'll lose the ability to create reports. And that might sound fine when you're just starting your project - in hindsight later on you'll completely regret that decision.
And back down to earth - confirming that 100% of the rows in a table comply with a check constraint (simple boolean), foreign key, uniqness, etc - is a trivial one line declarative. The implementation of these trivial and fool-proof constraints in ruby is 3x-20x as much code, depending on whether or not active record can be used and the nature of the constraint. Given that it still doesn't even cover 100% of the data in the database, this shouldn't be a tough call...
> And? I never said those programmers don't exist. I just said that the person making the statement wasn't one of them. > You've never hired offshore workers. > Either run a study or shut your mouth. > It's embarrassing to watch you talk. > You don't know anywhere near as much about this as you think you do; > you don't know anyone who's hired enough people.
> I'd tell you to go to Egypt, but I have no doubt you'd talk like this to the natives, and end up dead in an alleyway. > So yeah, go to Egypt.
Within a single message:
- you have made a dozen assumptions about others without any basis
- you have unnecessarily insulted two people that you know nothing of
- you have made mental backflips to interpret their statements in ways they clearly didn't intend
- you attempted to refute common sense with a denial of statistical validity that nobody would accept
Go & Chess have each lasted for a thousand years or more. Still widey played, and there's no reason to believe they won't be widey played in another hundred years.
Besides these two there are countless other elegant and timeless board games games - backgammon, checkers, othello, etc. And quite a few great card games - poker, hearts, cribbge, etc. And some great dice games - such as farkle for instance. All of these games are in the public domain, can be had for pennies at a thrift store - or created yourself easily.
Compared to computer games that almost certainly won't exist in twenty years it seems difficult to see how these games wouldn't have made the top 10.
Wait, maybe someone meant Top *Computer* Games of the last 10 years?
In the Mythical Man Month (http://en.wikipedia.org/wiki/Mythical_man_month) Brooks referenced a study that found a 10:1 difference in productivity between best and mediocre programmers on a team.
I think Gates stated that there was a 100:1 difference today - due to greater technology leverage. And Google is certainly banking on that as well.
Given that there's a large correlation between experience and expertise, and given that it is difficult to find experience developers in India, China, etc - then it isn't unreasonable to say that a given experienced programmer in country X is 20x as productive as a team of junior-college grads in country Y. Hell, the country isn't the point - it's the experience.
And this is born out in my work with teams from Russia, India and China. Not so much with France, Germany and the UK. In all cases it was the experience of the programmers more than any other single factor.
Of course, ten years from now this will probably not be a factor. Unless they're starting to move programming jobs to Africa for cost savings. Then we'll just start the whole cycle over again.
I think good server hostnames in a large shop should indicate function, production vs test environment and location. It would be miserable to create a similar system without codes. For example:
- database servers - mythological gods
- test servers - trickster gods
- dev servers - evil gods
- prod servers - good gods
- atlanta servers - greek gods
- phoenix servers - norse gods
- denver servers - celtic gods So, shit - what am I going to do when I end up with four test servers in Phoenix? There was only 1 trickster god in norse mythology. Ok, so maybe the plus side is that you get to have a lot of interesting debates about the gods while at work. But I just don't think this works as well as a simple convention like: [organization]-[location]-[function]-[number][envi ronment], or act-den-dba-05p
I do miss that heady optimism tho. Like the way I thought I'd be retiring by the age of 50 back then and now I'm hoping to retire to working as a big box store greeter.
Re:Author needs to get out of the basement
on
The Internet Of Things
·
· Score: 4, Insightful
And regardless of the deployable outcomes of linking *too much* personal data...
Side note: notice that he talked about links our email addresses, but didn't talk about linking corporate identities? How about shedding a little light on the incestuous relationships between corporate boards & ceos?
But aside from that - the real world consists of things that are hard to classify:
where their boundaries are gradual:
- rivers (change over time)
- events (see example from TFA about coffee shop outside of event)
- times (new years eve party started exactly when? 8:00 pm? 11:00 pm?)
or where there are different opinions about what a thing is:
- table can be a chair if you sit on it (see Wittgenstein)
- a stick can be a tool, weapon, toy, etc
- what a computer is has changed over the past 90 years
or where there are arguments over which parts of a thing are the thing:
- is your mouse part of your computer? how about your monitor? your hard drive?
- does your home include the yard? odds are sometimes it does, sometimes it doesn't
And this doesn't even touch on concepts (the TFA stated that we should label concepts). That's truly rediculous. Imagine the above problems magnified a hundred times to deal with abstract concepts and language.
> in the PC world we're finally seeing this architecture recognized as new Intel chips tout their front-side bus and cache > more than sheer increase in speed. > This SUN machine is a bigger-scale example of the same.
No, not really - parallelism has of course been around forever. But its application for high performance computing has been constantly demonstrated on large servers for the past 15 years. This is back when parallelism on intel hardware might at most have meant two cpus. And that was rare.
MPP (Massively Parallel Processors) systems like Teradata and IBM's SP2 (aka DeepBlue - that defeated Kasparov at chess) successfully demonstrated great performance for the dollar back around 1994-1995 or so. These were originally designed around data mining and math computations - but found most of their sales in data warehousing. Meanwhile, CRAY was complaining that not all problems were good candidates for this kind of more cost-efficient hardware.
By 1998 you could put db2 or informix on a hundred-node SP2, each node consisting of an eight-way SMP, each with its own dedicated storage. Queries on that old system were lightning fast compared to most other options. I worked directly on SP2s and worked with a team that has a 128-node one. Oracle & Sun eventually ecliped these solutions with massive SMPs. But much of that was more due to Informix's financial issues than technical merit - since Informix and DB2 (and of course Teradata) on MPPSs easily out-scaled oracle on SMPs. The SMPs were easier to adapt to application design changes, but the MPPs were easier to grow indefinitely large.
These newer solutions are just more of the same thing - you've still got the same challenges in:
- tons of OS and application images that must be consistent
- node communication bandwidth (major selling feature of all these solutions are proprietary internal networks)
- failover (how do nodes failover, especially if they have any dedicated resources)
- scheduling (how do jobs get assigned to nodes)
So, they're much bigger and faster than 10-15 years ago - and I'm sure there's got to be some cool innovation going on under the hood. But nothing looks fundamentally different from then. And nothing here has been inherited from the pc world.
> In some respects I have always felt like this, but with the internet and my expanded access to information, > I simply cannot avoid the feeling that the media's portrayal of politics is a ridiculous charade.
I generally agree with your entire post - including the above line. However:
1. The mainstream media's establishment of a single point of view, sense of values, etc sometimes improves the welfare of man. For example: blacks, gays, atheists, etc - while they may not get fully equal and completely respectful treatment will be presented far better than they are in many communities in the US - gradually improving the attitude of some members of those communities towards these groups of people. Likewise, non-mainstream cultural, religious and social ideas get some of the same benefits.
Think about the civil rights struggle - how the civil war ended in 1865 yet one hundred years later protesters still had to struggle to get blacks in the south the vote and then forty years later one southern state (alabama i think) had a vote to remove the constitutional amendment that prohobited interacial marrage - and in some counties 80+% of the voters elected to keep that amendment. This kind of abnormal racism is gradually whittled away at by mainstream media in many ways (even if focusing on blacks and crimes results in one step backwards for every two forwards).
2. The internet is gradually becoming an extension of something worse than mainstream media - in which it can help enable audience members to reinforce even the most unsupportable views. Take racism in the south - the internet can help a group of racists plug into news interpretations only coming from fellow racists. They'll despise mainstream media as being too liberal and memorize all the arguments from the white supremists...
So, yeah - mainstream media in the US has generally supported a pro-business, status quo view that has generally insulated the population from reality. But it doesn't work perfectly for example, sometimes it pushes a very pro-administration view in which you aren't allowed to criticize the president (say, united states c.2004 re: bush) but at other times it pushes a very anti-administration view (say, united states c.1996 re: clinton).
>> Last time I checked, DB2 was more scalable than Oracle
> That depends entirely on the platforms you happen to run them. DB2 on NT (what used to be called "UDB") is a joke; DB2 on OS/390 > is pretty much what defines a "big-iron" database. Oracle on NT is nowhere near as good as it is on Solaris. But Sybase on NT is > actually quite good - almost as good as SQL Server on the same hardware. Sybase 12.x on HP-UX is also quite good.
DB2 UDB actually scales extremely well for warehousing, decision support, search engines, reporting, etc - far beyond anything except maybe Teradata. If you're talking about NT - you're talking about an OS that is two generations back-level. But aside from that personally, I'd rather run a database on unix anyway - much better scriptable and command-line environment.
> Not all database types are fit for all purposes. Relational databases for instance are bad for data mining/warehousing > due to poor query performance but good for data entry due to high transactional performance
Your information is incorrect: 1. relational databases have been used for warehousing and reporting (data marts) for 15 years - and are used for more than any other solution for this purpose. Ok, sure you've got a lot of OLAP out there, but there's probably almost as much Relational OLAP (ROLAP) via Microstrategy, etc as there is true OLAP.
2. Take db2 for example, it:
- support three different forms of range partitioning (union-all views, multi-dimensional clustering, and range partitioning)
- supports hash-partitioning of the data across many servers - think "beowulf cluster"
- given the two above you can spread your data across 100 4-way servers, each with fibre access to a heavily-cashed SAN.
- Now when you issue a query db2 will spin up all 100 servers - each hitting its own local piece of the data (not 100 copies of the whole data, but each server with 1% of the whole).
- Because it also supports range partitioning each server is probably only going to scan 10% of the total data in a typical query.
- Because it support query parallelism it'll split each query on each node into four separate pieces (getting near-linear performance speedups) - now you've got 400 cpus working.
- Because its optimizer is about the best one on the market - it isn't going to auger itself into the ground on your 100 line sql query.
- That should allow you to crunch down a billion rows to your 24 row output in couple of seconds at most.
- Of course, it's also smart enough to rewrite your query to automatically hit any summary table that could speed the query up. So, it may only have to scan 2400 rows - and may return the results in 0.001 seconds.
3. The point is that warehousing, reporting and analytics work very well in a relational environment. But you need to pick your products well. If you want to handle terabytes of data you can put it in MySQL, SQLite, MS Access, Foxpro, etc - if you really had to. But, life will be *far* easier if you put it into a product that can handle the volumes much better.
Slashdot Mirror
I think the single best way to find good programmers is to find enthusiasts.
:-)
The reason is that it's easier to determine how enthusiastic someone is than how good a product they develop:
- enthusiasts usually have side projects
- enthusiasts often create libraries of code that can be reused
- enthusiasts will have a variety of favorite tools - and can explain why they like them
- enthusiasts will likewise have a variety of favorite methods - and can explain why they like them
- enthusiasts read widely in their field
- enthusiasts know the names of those who have made impacts on their field
- enthusiasts often find themselves putting in too many hours - because they *enjoy* the work
- enthusiasts gravitate together - put them in a room together and you'll have a lively conversation
And there are technologies, methods and tools that attract enthusiasts. For example, I've found that even if python and ruby aren't the most marketable languages out there - they are great ways to find the enthusiasts.
Of course, this won't help a manager that lacks enthusiasts on his team. But a technical manager who is himself an enthusiast, and builds such a team should be able to easy find more. At least in my humble opinion.
> Specialization increases productivity and, therefore, enriches the entire nation.
But there are hidden costs not included here:
- In the US auto industry design was specialized - with separate teams for each part of the vehicle. And they found that the Japanese were dedicating a team to a new vehicle for 18-24 months to handle all design. By moving from specialization to generalization they got better quality, fewer mis-matched interfaces, and far better speed.
- In meat-packing you don't just get boutique-level quality from a 'master butcher' - you reduce diseases such as salmonella and e coli, reduce worker injuries and reduce the need to overuse antibiotics.
- In healthcare the interfaces between specialists are points in which information is lost - resulting in specialists often making incorrect decisions. And while there is more money to be made as a specialist than a generalist - there's been a movement to encourage more doctors to become generalists - family practitioners and a recognition that you need more than nurses and specialist doctors.
- In IT again - an assembly-line approach generally results in poorer quality of service since servers and applications come in too many forms and are too customizable to allow complete standardization. Additionally, many issues like performance touch on too many pieces to be handled easily by specialized individuals. Lastly, when analyzing one company's time to create e-comm web sites using a j2ee platform years ago I found it to be slower and *far* more expensive per page than creation of cobol/cics applications. The cause was specialization - ejb developers vs jsp developers vs html developers vs dbas vs qa, etc, etc.
- In business leaders generally want their employees to be decision-makers and aware of how their contributions map to corporate objectives.
I'm not saying that all specialized skills are bad, but specialized skills without a general knowledge foundation are bad. And I'm not saying that specialized roles are bad - but any process that only has specialist roles is brittle, isn't human-oriented and probably has a lot of hidden costs.
>>"introducing a new division of labor that eliminated the need for skilled butchers"
> That's just a good example of optimizing workforce utilization.
>Same thing with skilled butchers not being used to do jobs that a person with no skills and no English can perform.
As Heinlein said, "specialization is for insects". What might appear to be job optimization ends up really just meaning:
- that the workers gets a miserable and repetitive job
- that workers don't get to see the 'whole picture' - and so have no ability to work around unanticipated problems
- that the end result is only better if you only measure cost
I've been in IT shops that insist that everyone do just one specialized thing - and a simple server problem results in a dozen 'matrixed' people on a call for ten hours failing to figure out what needs to happen. Previously, a single admin with a fuller set of responsibilities would have figured the problem out very quickly - or would never have let the circumstances result in the problem anyhow.
Likewise, in meat packing - you've got a really miserable and dangerous job that now nobody wants unless they're really desperate. Which mostly means non-english speaking people who aren't in a position to complain. Or to get worker's comp for the repetitive stress injuries they receive. And who aren't in a position to determine that an animal should be rejected due to disease. Etc. But that's all ok - since
- the company only very seldom has to pay for injured workers
- the us now has very few meat inspectors
- the cattle practically bathed in antibiotics to cope with bizarre and unsanitary conditions
But if a person is just looking at the cost for a pound of meat at the store and not concerned about overuse of antibiotics, chances of getting ecoli, what happens to the workers whose lives were destroyed at the plants. Then yeah, it's probably a good deal.
> Pardon my conspiracy theory, but hasn't the government been spying on us, well, forever?
yeah - to various levels of effectiveness and efficiency:
Data collection: 50 years ago it took a ton of paperwork and a massive staff to collect data on a lot of people. Typically, you'd just target a fraction of the population, or collect a little info on everyone.
Personal aggregation: this data could not be (for all practical purposes) aggregated by person
Profile aggregation: again, due to reasons of economy, the data by person could not be aggregated by profile
however, now:
You've got handy electronic records of everything you buy, everywhere you go, every web site you visit, everyone you call, and every message via sms and email. So, data collection is no longer extremely expensive - in fact it can be relatively trivial.
The challenge is in getting "common keys" between all these sources - so that they know that ken@acme.com is the same person as ken@hp.com who lives in the same household that visits salon.com. This part is still very expensive to do completely. So, aggregation is still a challenge - but the problem is being whittled away at by these departments and organizations.
The good news is that this is difficult to do well - especially given government bureaucracies.
> I say privacy is pretty much a thing of the past.
As we lose privacy we'll lose freedoms, so you better hope not. There will be little effective opposition to a majority government that controls this info - and can easily blackmail opposition figures over uncovered items. There will be enormous pressures to conform to perceived normalacy when deviations from it result in raised TerroristScore, lowered PatriotScore, or whatever.
Imagine for a second trying to get onto a plane past a security guard that notices your TerroristScore is a little high. Unknown to both you and him this is because you frequent a middle eastern falafal stand frequented by a few fundamentalist moslems that donate to Hamas. This guy isn't very good with numbers, isn't very good at understanding that your score is a little high - it's black & white with this guy. You're not getting on that plane, and you will be treated like a terrorist.
Or imagine getting denied a job or health insurance because you work out at a gym where there's a lot of gay men with aids. This has affected your HealthCostScore, even though you have no idea about this - the system will automatically reject your application.
Giving up on privacy in an age of information is like giving up on food. The only difference is that one will kill you faster than the other.
> unless SQL data representation grows up to modern non-fortran-like OO semantics MySQL will proliferate
You do realize that in activities like reporting sql and its set-based operations are far, far, far faster and easier to work with than oo implementations, right?
You can set up a typical star-schema and have certain tools (like microstrategy) immediately recognize it and generate queries for you. These queries will typically perform just fine and allow very powerful and fast drill-downs, drill-across, etc.
And oo approach involving the marshalling of millions of objects related to entries in the database would take forever to build to this kind of flexibility and would run slow as molasses. Might make the oo purists happy, but the customers would never use the product. At least against non-trivial amounts of data.
> Though, I must admit I can't see it running very fast compared to software DESIGNED for that type of platform.
Actually, it's probably the fastest solution out there for heavy reporting/analytical workloads on windows, linux or unix. Not sure about teradata and informix - haven't looked at them in quite a while.
Teradata, informix and db2 were doing the 'beowulf' thing years before anyone on slashdot asked what a beowulf cluster of these would be like. So, you can easily distribute data across a hundred db2 server blades to get very fast response times over terabytes of data. This can be far faster at the high-ends than the typical oracle scenario of range partitioning on a 32-way smp. And oracle grid is more failover oriented than performance oriented, so that doesn't really compete either, last I looked.
I've built and managed very large databases on oracle, db2 and sql server in the last five years. Of them all, db2 on unix was the easiest to build and manage. It did take a little extra time - since some of the tools are a little finicky, but the manageability and performance was far better than with either oracle or sql server. And with very large data - far, far better than with mysql or postgresql.
> MySQL, as others have pointed out, has better developer support and they know their target audience. They supply a fast,
> easy to use database for those who don't need a whole lot.
> Oracle supplies an enterprise level database that MySQL doesn't aspire to.
> PostgreSQL doesn't know where to fit in.
This is an oversimplification. Each vendor sees itself in all markets:
- oracle/db2/sql server have free versions for tiny apps and very expensive versions for massive apps
- mysql says it doesn't want to do what oracle does, but also says that this is less than 1% of the market - and knows that plenty of smallish databases are on oracle
- postgresql like the others sees itself doing anything from very small databases to very large ones (often via Enterprise DB or other vendor extensions)
And using a single product for multiple sizes isn't illogical: if you have any very large databases (hundreds of gbytes or more) then you probably have a few dozen little ones as well. It's *far* easier to manage them all on oracle/db2/sql server - even with the small additional licensing costs - than to have a frankenstein collection of products to manage.
"Best tool for the job" is a good consideration when evaluating products (along with vendor viability, cost, etc, etc) - but once you've got a single tool in house to keep adding new products - each with their own licensing, support, patch, backup/recovery procedures, etc is a nightmare. Let alone actually federating your data - and having to test out how to virtualize or replicate data from oracle 10.x.x with mysql 5.y.y
> Performance is one aspect of the price tag, but it is certainly not the only factor.
Very true - and for that reason Postgresql has more going for it than many alternatives, like:
- best licensing options - you don't need to pay a lawyer to go over your contract or license like you should if you use oracle or mysql commercially. And there's no fear that the vendor will change its license terms once you're locked in and start charging an arm and a leg.
- very good foundation - postgesql isn't built from duct tape and bailing wire. The functionality within it is well tested and robust.
- great support for standard database features - whether its subselects, stored procedures, triggers, etc - it's very simple to move from oracle to postgresql.
- great ansi sql support - again, very standard sql - no unnecessarily propretary language elements.
So, yeah - just because Postgresql is performing well on some benchmarks that doesn't mean you should immediately throw out oracle in favor of postgresql. On the other hand, you also shouldn't discard it because it is a good general purpose database solution.
No, read it years ago and thought it a waste of paper. Mostly mindless patriotism. Not worth rereading to double-check the industry/government cooperation that I thought I remembered. So, I'll cave first. :-)
Around 1850 engineers were looking to redesign London's sewer system and determined that Rome's Cloaca Maxima and Colosseum had the most modern sewer systems in the world - able to handle the waste disposal of 50,000 people about 2000 years ago.
So, in the 19th century London applied lessons learned from the colosseum 2000 years before. Between that and the various mechanisms for raising a cage out of the floor, or flooding the entire things for water fights or raising sails to shade the spectators - I'd call that a wonder.
Before we start adding some piddling monuments like Mt. Rushmore, I think we need to consider some more of our overlooked wonders:
. html
. htm
world's largest ball of string:
http://www.roadsideamerica.com/attract/MNDARtwine
world's largest pecan:
http://www.worldslargestthings.com/missouri/pecan
world's largest buffalo:
http://www.wlra.us/wl/wlbuffalo.htm
world's largest pineapple:
http://www.wlra.us/wl/wlpineapple.htm
world's largest muskie:
http://www.wlra.us/wl/wlmuskie.htm
world's largest catsup bottle:
http://www.catsupbottle.com/
this is why the internet isn't cool - there's nothing really to see
why computers aren't cool - you can't see the programs run
why antibiotics aren't cool - you can't see how they work
and so, if I've got a description of the great pyramids, colossus of rhoads, or great wall of china - the utter vastness of them, the timelessness of them and the unimaginable difficulties with which they were built...i'm not impressed. Because I can't see it in the flesh. Bah, I say.
well - intense nationalism, intense militarism and integration of the government and corporate concerns. I'd think that would be a good description of fascism, wouldn't it?
then you need to read "Bill the Galactic Hero":e ro
r rison/dp/0743487079/ref=pd_bbs_1/002-7428068-27096 66?ie=UTF8&s=books&qid=1183845880&sr=1-1
http://en.wikipedia.org/wiki/Bill,_the_Galactic_H
and here it is on amazon: http://www.amazon.com/Bill-Galactic-Hero-Harry-Ha
holy shit, original paperbacks going for $70. Note to self: must dig thru paperbacks to see if I still have this. Anyhow, this was a great parody of the fascist sci fi from heinlen...
> The US government is almost certainly wiretapping people it would not be allowed to had it used the courts, otherwise it would use the courts.
Consider how much this is like the law & order mantra of "why do you care if the government monitors you if you've got nothing to hide"...
> Who cares if a handful of morons have specific requirements for large processors with large I/O bandwidths?
The 'morons' you refer to include just about anyone analyzing large sets of data. Reporting systems, data mining, personalization, search engines, etc are all io-intensive activities that can work very well on mainframes. This covers a hell of a lot of very interesting ground - certainly more interesting than the typical CRUD app.
> Linux grid takes out firm's aging mainframe
> By Jack Loftus, News Writer
> 04 Jan 2007 | SearchOpenSource.com
So, this consulting firm sends out a marketing press release that describes how they updated a 20-year-old bad design managing 4.5 pedabytes of data. And only got a 70% performance increase.
I'd call that a total failure.
Sorry, but if the best claim that a white-washing consulting press release is 70% for moving an 80's mainframe to an intel grid consisting of 118 circa 2007 cpus - that's pathetic.
Not to say that I'd prefer development on a mainframe - unless you've got dedicated lpars the meetings to collaborate on changes are too much of a pain. I prefer dedicated unix or linux boxes. But this is just bullshit.
> Why bother with either. Just include sqlite and be done with it.
Sure, nice little database. Not much for multiple users and large data tho...
> Rails can take care of all the data integrity for you anyway.
ah, no - applications do a horrible job of managing data constraints over time. That is, you might test the bajesus out of your code this version and next - but are unlikely to test how code with this version handles data created with the code ten versions before. None of the data we're talking about here is wrong - it's just different valid values.
And if you do create a ridiculous version-compatibility layer in ruby then you'll end up without the ability to create a simple report using sql. Since the most common reporting tools all use sql (most powerful ones generate sql) you'll lose the ability to create reports. And that might sound fine when you're just starting your project - in hindsight later on you'll completely regret that decision.
And back down to earth - confirming that 100% of the rows in a table comply with a check constraint (simple boolean), foreign key, uniqness, etc - is a trivial one line declarative. The implementation of these trivial and fool-proof constraints in ruby is 3x-20x as much code, depending on whether or not active record can be used and the nature of the constraint. Given that it still doesn't even cover 100% of the data in the database, this shouldn't be a tough call...
> And? I never said those programmers don't exist. I just said that the person making the statement wasn't one of them.
> You've never hired offshore workers.
> Either run a study or shut your mouth.
> It's embarrassing to watch you talk.
> You don't know anywhere near as much about this as you think you do;
> you don't know anyone who's hired enough people.
> I'd tell you to go to Egypt, but I have no doubt you'd talk like this to the natives, and end up dead in an alleyway.
> So yeah, go to Egypt.
Within a single message:
- you have made a dozen assumptions about others without any basis
- you have unnecessarily insulted two people that you know nothing of
- you have made mental backflips to interpret their statements in ways they clearly didn't intend
- you attempted to refute common sense with a denial of statistical validity that nobody would accept
In short you're a jack-ass.
Go & Chess have each lasted for a thousand years or more. Still widey played, and there's no reason to believe they won't be widey played in another hundred years.
Besides these two there are countless other elegant and timeless board games games - backgammon, checkers, othello, etc. And quite a few great card games - poker, hearts, cribbge, etc. And some great dice games - such as farkle for instance. All of these games are in the public domain, can be had for pennies at a thrift store - or created yourself easily.
Compared to computer games that almost certainly won't exist in twenty years it seems difficult to see how these games wouldn't have made the top 10.
Wait, maybe someone meant Top *Computer* Games of the last 10 years?
In the Mythical Man Month (http://en.wikipedia.org/wiki/Mythical_man_month) Brooks referenced a study that found a 10:1 difference in productivity between best and mediocre programmers on a team.
I think Gates stated that there was a 100:1 difference today - due to greater technology leverage. And Google is certainly banking on that as well.
Given that there's a large correlation between experience and expertise, and given that it is difficult to find experience developers in India, China, etc - then it isn't unreasonable to say that a given experienced programmer in country X is 20x as productive as a team of junior-college grads in country Y. Hell, the country isn't the point - it's the experience.
And this is born out in my work with teams from Russia, India and China. Not so much with France, Germany and the UK. In all cases it was the experience of the programmers more than any other single factor.
Of course, ten years from now this will probably not be a factor. Unless they're starting to move programming jobs to Africa for cost savings. Then we'll just start the whole cycle over again.
I think good server hostnames in a large shop should indicate function, production vs test environment and location. It would be miserable to create a similar system without codes. For example:i ronment], or act-den-dba-05p
- database servers - mythological gods
- test servers - trickster gods
- dev servers - evil gods
- prod servers - good gods
- atlanta servers - greek gods
- phoenix servers - norse gods
- denver servers - celtic gods
So, shit - what am I going to do when I end up with four test servers in Phoenix? There was only 1 trickster god in norse mythology. Ok, so maybe the plus side is that you get to have a lot of interesting debates about the gods while at work. But I just don't think this works as well as a simple convention like: [organization]-[location]-[function]-[number][env
I do miss that heady optimism tho. Like the way I thought I'd be retiring by the age of 50 back then and now I'm hoping to retire to working as a big box store greeter.
And regardless of the deployable outcomes of linking *too much* personal data...
Side note: notice that he talked about links our email addresses, but didn't talk about linking corporate identities? How about shedding a little light on the incestuous relationships between corporate boards & ceos?
But aside from that - the real world consists of things that are hard to classify:
where their boundaries are gradual:
- rivers (change over time)
- events (see example from TFA about coffee shop outside of event)
- times (new years eve party started exactly when? 8:00 pm? 11:00 pm?)
or where there are different opinions about what a thing is:
- table can be a chair if you sit on it (see Wittgenstein)
- a stick can be a tool, weapon, toy, etc
- what a computer is has changed over the past 90 years
or where there are arguments over which parts of a thing are the thing:
- is your mouse part of your computer? how about your monitor? your hard drive?
- does your home include the yard? odds are sometimes it does, sometimes it doesn't
And this doesn't even touch on concepts (the TFA stated that we should label concepts). That's truly rediculous. Imagine the above problems magnified a hundred times to deal with abstract concepts and language.
> in the PC world we're finally seeing this architecture recognized as new Intel chips tout their front-side bus and cache
> more than sheer increase in speed.
> This SUN machine is a bigger-scale example of the same.
No, not really - parallelism has of course been around forever. But its application for high performance computing has been constantly demonstrated on large servers for the past 15 years. This is back when parallelism on intel hardware might at most have meant two cpus. And that was rare.
MPP (Massively Parallel Processors) systems like Teradata and IBM's SP2 (aka DeepBlue - that defeated Kasparov at chess) successfully demonstrated great performance for the dollar back around 1994-1995 or so. These were originally designed around data mining and math computations - but found most of their sales in data warehousing. Meanwhile, CRAY was complaining that not all problems were good candidates for this kind of more cost-efficient hardware.
By 1998 you could put db2 or informix on a hundred-node SP2, each node consisting of an eight-way SMP, each with its own dedicated storage. Queries on that old system were lightning fast compared to most other options. I worked directly on SP2s and worked with a team that has a 128-node one. Oracle & Sun eventually ecliped these solutions with massive SMPs. But much of that was more due to Informix's financial issues than technical merit - since Informix and DB2 (and of course Teradata) on MPPSs easily out-scaled oracle on SMPs. The SMPs were easier to adapt to application design changes, but the MPPs were easier to grow indefinitely large.
These newer solutions are just more of the same thing - you've still got the same challenges in:
- tons of OS and application images that must be consistent
- node communication bandwidth (major selling feature of all these solutions are proprietary internal networks)
- failover (how do nodes failover, especially if they have any dedicated resources)
- scheduling (how do jobs get assigned to nodes)
So, they're much bigger and faster than 10-15 years ago - and I'm sure there's got to be some cool innovation going on under the hood. But nothing looks fundamentally different from then. And nothing here has been inherited from the pc world.
> In some respects I have always felt like this, but with the internet and my expanded access to information,
> I simply cannot avoid the feeling that the media's portrayal of politics is a ridiculous charade.
I generally agree with your entire post - including the above line. However:
1. The mainstream media's establishment of a single point of view, sense of values, etc sometimes improves the welfare of man. For example: blacks, gays, atheists, etc - while they may not get fully equal and completely respectful treatment will be presented far better than they are in many communities in the US - gradually improving the attitude of some members of those communities towards these groups of people. Likewise, non-mainstream cultural, religious and social ideas get some of the same benefits.
Think about the civil rights struggle - how the civil war ended in 1865 yet one hundred years later protesters still had to struggle to get blacks in the south the vote and then forty years later one southern state (alabama i think) had a vote to remove the constitutional amendment that prohobited interacial marrage - and in some counties 80+% of the voters elected to keep that amendment. This kind of abnormal racism is gradually whittled away at by mainstream media in many ways (even if focusing on blacks and crimes results in one step backwards for every two forwards).
2. The internet is gradually becoming an extension of something worse than mainstream media - in which it can help enable audience members to reinforce even the most unsupportable views. Take racism in the south - the internet can help a group of racists plug into news interpretations only coming from fellow racists. They'll despise mainstream media as being too liberal and memorize all the arguments from the white supremists...
So, yeah - mainstream media in the US has generally supported a pro-business, status quo view that has generally insulated the population from reality. But it doesn't work perfectly for example, sometimes it pushes a very pro-administration view in which you aren't allowed to criticize the president (say, united states c.2004 re: bush) but at other times it pushes a very anti-administration view (say, united states c.1996 re: clinton).
>> Last time I checked, DB2 was more scalable than Oracle
> That depends entirely on the platforms you happen to run them. DB2 on NT (what used to be called "UDB") is a joke; DB2 on OS/390
> is pretty much what defines a "big-iron" database. Oracle on NT is nowhere near as good as it is on Solaris. But Sybase on NT is
> actually quite good - almost as good as SQL Server on the same hardware. Sybase 12.x on HP-UX is also quite good.
DB2 UDB actually scales extremely well for warehousing, decision support, search engines, reporting, etc - far beyond anything except maybe Teradata. If you're talking about NT - you're talking about an OS that is two generations back-level. But aside from that personally, I'd rather run a database on unix anyway - much better scriptable and command-line environment.
> Not all database types are fit for all purposes. Relational databases for instance are bad for data mining/warehousing
> due to poor query performance but good for data entry due to high transactional performance
Your information is incorrect:
1. relational databases have been used for warehousing and reporting (data marts) for 15 years - and are used for more than any other solution for this purpose. Ok, sure you've got a lot of OLAP out there, but there's probably almost as much Relational OLAP (ROLAP) via Microstrategy, etc as there is true OLAP.
2. Take db2 for example, it:
- support three different forms of range partitioning (union-all views, multi-dimensional clustering, and range partitioning)
- supports hash-partitioning of the data across many servers - think "beowulf cluster"
- given the two above you can spread your data across 100 4-way servers, each with fibre access to a heavily-cashed SAN.
- Now when you issue a query db2 will spin up all 100 servers - each hitting its own local piece of the data (not 100 copies of the whole data, but each server with 1% of the whole).
- Because it also supports range partitioning each server is probably only going to scan 10% of the total data in a typical query.
- Because it support query parallelism it'll split each query on each node into four separate pieces (getting near-linear performance speedups) - now you've got 400 cpus working.
- Because its optimizer is about the best one on the market - it isn't going to auger itself into the ground on your 100 line sql query.
- That should allow you to crunch down a billion rows to your 24 row output in couple of seconds at most.
- Of course, it's also smart enough to rewrite your query to automatically hit any summary table that could speed the query up. So, it may only have to scan 2400 rows - and may return the results in 0.001 seconds.
3. The point is that warehousing, reporting and analytics work very well in a relational environment. But you need to pick your products well. If you want to handle terabytes of data you can put it in MySQL, SQLite, MS Access, Foxpro, etc - if you really had to. But, life will be *far* easier if you put it into a product that can handle the volumes much better.