While there was a bit of greedy butchery and mindless power-acquisition going on mostly what happened was story-telling, character development, working as a team, out-smarting opponents, building alliances, etc.
Grinding? In playing d&d, gurps, etc with a good group of people it *never* felt that way. We may have spent 2 years and never made it past 3rd level and struggled every step of the way - but it was a fun memorable time. Perhaps grinding happens more in computer-based "rpgs"? Or groups of kids that want nothing else than to have a 20th level character. Either way, in many years of playing I never really felt that.
Large databases use a combination of range partitioning AND indexing. If you're lucky you've also got hash partitioning - to distribute your database across N servers. Partitioning is far more general and forgiving for purposes like this than indexing.
And creating trends in your application layer then storing them in logs can work, but: 1. you still rely on figuring out in advance what you're going to need 2. if you don't load those logs back into the relational database then you can't effectively join it to all that data. And you then either must log redundant data or have useless logs. 3. grep & sed or custom reporting code against your logs is a poor substitute for any standard reporting tool or custom code against your database. 4. database features like partitioning, indexing, parallelism result in in-database aggregates being faster than logs 5. did i mention automatic query rewrite? where the database automatically converts eligible queries against the base table to actually run against the summary tables...
Note that most of these solutions come from the interwebs, social networks, etc. And it isn't so much anti-sql as it is anti-relational database (sql != rdb).
The basic premise is that we need different solutions that: can scale very high for very narrowly scoped reads & writes, don't need to perform ranged queries / reporting/etc, and don't need ACID compliance. And that may be the case. Sites like slashdot, facebook, reddit, digg, etc don't need the data quality that ebay needs.
On the other hand, ebay achieves scalability AND data quality with relational databases. And when I've worked with architectures that scale massively and avoid the relational trap for better solutions - they inevitably later regret the lack of data quality and complete inability to actually get trends and analysis of their data. It *always* goes like this:
Me: So, is this thing (msg type, etc) increasing?
Developer: No idea.
Me: Ok, so lets find out.
Developer: How?
Me: I don't know - typical approach - lets query the database.
Developer: It'll take four+ hours to write & test that query and then days to run. And when it's done we might find that we wrote the query wrong.
Me: What?!?
Developer: We had to do it this way, you can't report on 10TB databases anyhow
Me: What?!? Are you on crack? there are dozens of *100TB* relational databases out there that people are reporting on
Developer: well, we probably don't need to know what that trend is anyhow
Me: I'm outta here
Not really - you're thinking about stored procedure language. Which if you close one eye and squink with the other I guess could look a little COBOL-esque, that is without the column-orientation, section headings, etc.
SQL on the other hand looks like this:
SELECT dept.dept_name,
per.rank,
count(*) as personnel_rank_count FROM dept_table dept
INNER JOIN personnel_table per
ON dept.dept_id = per.dept_id WHERE dept.dept_type = 'internal' GROUP BY dept.dept_name,
per.rank ORDER BY 1,2
Even if you prefer to use a ORM like hibernate or active record at some point: - you'll have to modify an application that doesn't use one - you've got a bug in the ORM and need to compare its behavior against that of raw SQL - you'll need to diagnose problems quickly in a database - and will want to quickly write a 1-20 little queries to check everything out - you'll need to aggregate (pre-process) some data - and given SQL's set-orientation will be able to do it faster in the database than in your application. And it'll be running huge queries - maybe generating a dozen or more temp tables. And the ORM won't support your SQL. - etc, etc, etc
So, go ahead and use the ORM when it makes sense. Just get familiar with SQL as well - it'll save you a ton of time.
Also, back to the original thread: I see programming as building a house and writing sql as playing chess. It is completely different than writing most other code. And there are many times when I'm *very* glad that it is.
> I know all about the Mercedes Diesels from back then. They are reliable for sure but they are slow. > Especially when combined with 4 speed transmissions with not enough low gearing. They accelerate like a tourtise.
Must depend on the version. I had a '79 300SD with a turbo diesel that was fine. The only time it seemed slow was when climbing mountain passes in colorado. Other than that it was better than your average car. Not *fast* - but certainly more than anyone needs.
And I've driven VW turbo diesels - passats and jettas from the last ten years - and thought that they had very good acceleration and were very fun to drive. Really, at this point I don't think there's any noticeable difference between gas and diesel ownership for most people - except fuel economy.
> How could a VC firm not be intelligent enough to realize that mail-order perishables isn't going to work out well...
No - you need to think big. Really big. We're not talking merely about shipping milk & meat here, we're talking about milk & meat *portals*. In the meat portal for example, you'd find:
- forums where you can share thoughts with other meat enthusiasts - "The Meat Room"
- recreational activities for the meat-based lifestyle like http://www.hatsofmeat.com/
- meat-based recipes
- the "Meat Haus" chat room
- where you can buy or sell meat in any quantity - the "Meaty Money Pit"
I wouldn't give up on this. I know the Atkins diet has faded a bit, but perhaps it's just getting ready for a rebound? And we'll see a surge of excitement over meat again?
> Perhaps the traditional RDBMS experts will return when they can scale > their paradigms to datasets that are measured in the tens of terabytes > and stored on thousands of computers.
I'm not defending the authors of the article, but...
1. this article wasn't written by rdbms people, but rather by column database people. There's nothing traditional or relational about their background.
2. Every solution is a result of a variety of compromises, there is no such thing as the perfect solution outside of a context. And google's context is almost completely unique. So, it is somewhat meaningless that a solution works well for them. That doesn't mean it would work at all well for HP, Time-Warner, the State of Maine, the University of Colorado, or Ma Kettle's Fish Shack.
Until DR DOS 5 came out Microsoft was just willing to just let everyone suffer with DOS 4.11, 3.*, etc. Then as the number of DR DOS fans grew, Microsoft realized that they risked losing complete control over their platform and had to respond.
DOS 5 was pretty good, but it still wasn't as good as DR DOS 5, let alone DR DOS 6. And the only reason that they outsold DR DOS is that they dumped their product - by dropping their price from like $120 down to $29. DR DOS couldn't compete with microsoft deep pockets.
> I would love to see the 4th edition finally allow you to multiclass properly
Then you'll probably want to switch to a different game. Try something like GURPS - where the entire simple concept of classes was abandoned decades ago, and you can easily combine many different types of skills. Of course, how easily you master a skill will depend on your dexterity or intelligence and prerequisite skills - but still, you have total freedom.
> failing 65% of the time would be a quick way out of business
you would think so.
However, remember that you can meet requirements without meeting objectives. There are quite a few applications out there that met the requirements, but failed to meet the objectives of adaptability, usability, manageability, etc.
These apps are error-prone, frustrating, and inefficient to use. They are hated and despised by their users. But they may still be quoted as successes by IT ivory-tower folks specializing in project estimation, etc.
Lastly, I'm not talking only about custom software here. The above certainly applies to commercial solutions, to network/server/telecom architectures, etc.
> And unlike Python you can, you know, use indents to mark up the code instead of placating the language's wishes.
I was thinking about this - and couldn't come up with a scenario in which I wouldn't want my indentation to accurately reflect flow of control. This is what python enforces.
Ok, so aside from warehouse vs mart discussions - when we're talking about a data warehouse we're generally talking about a star-schema. And in this data model you've typically got: 1. many dimension tables
- columns: many wide character columns
- rows: few (generally less than 5,000)
- examples: time, location, product, organization
- bottom line - column-orientation would be great here - except these tables are typically only a few mbytes anyway and are easily cachable. 2. very few (maybe 1) central fact table
- columns: a couple dozen integer columns - mostly just keys to the dimensions
- rows: many (often billions)
- examples: product sales, security events, etc
- bottom line: column-orientation doesn't buy you much - since the columns are so small and are just integers. 3. a dozen or more summary tables (pre-aggregated subsets of the fact table)
- columns: less than a dozen
- rows: typically 1% or less than what's in the fact table
- bottom line: column-orientation doesn't buy you much here either - since the columns are small and the number of rows aren't that significant.
Now consider:
a. partitioning: data is often partitioned by time ranges since time-series or current-version queries are the most common. This works extremely well with row-orientation since all the rows can be put into their own virtual table. But it won't work with column orientation. So, for example when you need to get all data for march 2007 - you won't be able to just scan that 2% of the data, you'll have to scan 100% of the column-oriented data.
b. compression: in db2 you can expect to achieve 80% compression of your fact tables. All those integers are just simple recurring patterns. Column orientation would certainly improve on that. But by how much?
c. writing: data warehouses may be read-mostly, but more and more there's a desire to get close to real-time reporting. The warehouse that I support gets loaded every hour for one set of data, and every minute for another. And users expect one set of data to be available for reporting within 30 minutes of being created. That set comes in batches of 100,000 or more rows. If the write-delay was sufficient we wouldn't be able to meet that requirement.
So, I'd consider that there may be some advantages to column-orientation. However, if you've got a good star-schema data model, are using partitioning and compression - then the benefits shouldn't be that significant and the loss of fast writes and easy partitioning may completely offset those meagre benefits.
Well, mysql allows multiple storage engines. Each has its own proprietary file structures - anything from key-value pairs to isam.
DB2 used to support ISAM (and perhaps still does in its mainframe version?), but now mostly uses one main engine (which can store data in file system directories, in raw devices or raw volumes) but also has a completely separate xml storage engine that's pretty good. And just within the primary storage engine allows many different table types:
- mdc (multi-dimensional clustering) tables (like oracle - table is split into smaller tables on one server)
- compressed tables (gets you about 80% compression with old, nasty rows)
- typed tables (object tables)
- range partitioned tables (like oracle - table is split into smaller tables on one server)
- hash partitioned tables (partitions are split across many servers)
So, although I'm sure it would take work - I can't imagine they couldn't also offer a column-oriented table.
You're right - giving metrics to management is dangerous. And that reminds me - I was in a hurry and forgot to add a few others that address this issue. I've always resisted giving metrics that only measure part of a process - since management will always assume that the metrics cover the entire process. Giving metrics that only show efficiency encourages management to continually strip down your organization.
So, with that in mind I'd recommend also adding customer satisfaction metrics (mostly apply to paying customers that could go elsewhere), customer productivity metrics (mostly apply to internal customers), and absolutely - some kind of adaptability metrics (ability to lead or follow changes - like come up with metrics, adapt to new workflows, etc).
It's hard to do this well - but I'd recommend taking leadership over the metrics that will define your job. And if I was managing teams that insisted that refused to provide concise data on what they do - I wouldn't trust them, and might look for opportunities to reorganize them.
You really need to be using a request system to do the following, but really, if you've got more than just a couple of admins you probably should have one already.
So, having said that and just shooting from the hip, I'd look for:
- big cost-drivers:
- number of servers per administrators by os
- number of requests by day by request type by platform with average times, should cover:
- create/alter/remove a userid
- create/alter/remove a group
- assign/alter/remove user to a group
- recover a file
- os installs
- corporate policy & security stuff:
- number of os patches performed per month by os
- number of application patches performed per month by os
- number of infections caught
- general characteristics
- servers managed
- userids managed
- privileges managed
- gbytes of os & application code and data managed
Other issues like reliability are hard to meaningfully report on unless you've got a ton of servers over a lot of time.
1st edition was poorly designed, but at least it was relatively simple. 2nd edition tried to fix the design with a lot of poorly thought-out patches. 3rd edition tried to redesign the solution and managed to make it worse. 3.5 edition tried to fix the bad redesign with more bad patches. 4th edition will supposedly fix the prior bad designs.
However, this vendor has no credibility when it comes to go design - they've never really created an elegant gaming system and clearly enjoy changing things around to force players to buy $1000+ in books.
So, compare this to gurps (generic universal role playing game system at www.sjgames.com) for a second. Gurps came out around twenty years ago. They've had revisions - but the original game mechanics worked so well that the revisions are mere tweaks compared to the nonsense at wotc. This isn't to say that gurps is perfect - it could be more detailed than some groups would want - but it's a perfect example of how an elegant design ages well.
Given the options of buying the old 1st edition books used for $5 each, playing gurps (requires 1-2 books), or playing almost any other game I've got no idea why anyone would bother with d&d anymore.
> Actually, they dont leak. It is made out of concrete and polyurathane foam.
Right, that's the idea anyway: you inflate a huge bubble, go inside it, and spray a layer of polyurethane foam, then spray concrete over that. This gives you three layers from outside in:
- plastic bubble layer
- urethane foam layer
- concrete layer The plastic bubble layer is theoretically reusable, but generally isn't. The urethane foam layer provides insulation but is fragile. The concrete is strong.
Unfortunately, the two outer layers are far less durable than the concrete. So, the next step in the failed evolution of this design was to add a second layer of concrete on the outside. The result of this was that the two concrete layers reacted differently to temperature changes - and the result was cracks.
The next step in the evolution was to add rebar to the outside layer (chain link fences sections). That stopped the cracking problem. Or so the vendor said. We'll probably fine out in ten years that that caused other problems (not the least is cost) - but they won't talk about that until they have a fix ready to offer.
Personally, I think it's a good idea - and eventually we'll have a working solution. In the meantime I would never trust the "Monolithic Dome Institute" to be up front about its problems.
Yep, that would be great. But just like geodesic domes that preceded monolithic domes - there are unforeseen issues like:
- leakage - in the case of monolithic domes due cracking
- integration challenges - they're difficult to tie into other components
- windows - good quality windows don't come in arcs
- expense - they're not cheap to build (nor necessarily expensive)
A monolithic dome is at the very top of what I'd like to build to live in. Unfortunately, we just haven't yet worked out all the kinks. And worse, many of the kinks are brushed under the carpet by the evangelists behind them. Until years later when they admit that the prior design didn't work - but "the new design fixes that old problem that I always denied they had".
> Your examples count. My customer data has nothing to do with purchase data. The customer is an entity unto itself. > Inventory would count, but I rarely store inventory as numerical values -- I typically don't store inventory -- > I count(*) the purchase orders.
Normally, you update inventory counts from multiple sources - so you know the number on hand (based on what has arrived) and the number on order. The complexity of this depends on number of workflows that you want to automate (generally the size of the operation).
> It would be slower, but a query cache handles that perfectly.
The speed is one issue, but not the most critical - the most important point is that if one of your operations fails it could leave the application in an inconsistent state - with some application data reflecting the change and some not.
> Certainly, there are applications that I would write that would benefit from transactions -- I simply haven't > had the need; more specifically, the benefits haven't outweighed the costs.
I recommend that you pick a small candidate mini-project and try transactions out on that task. You will likely find that they normally simplify your work rather than add to it - while eliminating a number of data quality problems along the way.
> Transactions are based on "undoing" what was only partially done. Funny, but I never want to "undo" anything. > If it was done -- even partially -- I want the partial record of it. Instead, my applications utilize atomic > indicators to maintain database integrity.
You know - it's always fun to work out your own solution to complex problems. But...in areas that are extremely well-trodden - you really should do some research as well. And you might not find the books and literature that explain *why* we use transaction nearly as easy to find as those that explain *how* we use them.
But you still need to go there. Or spend some time discussing it with people that used to do what you're talking about twenty years ago. They could probably explain all the problems that transactions simplified, all the coding that was reduced, all the bugs that were eliminated, and all the new possible functionality enjoyed.
I'm not going to tell you that you need transactions for everything - you really don't. And in many of the batch-oriented (like data warehousing, analytic, reporting, etc) apps that I create I often don't need them. But - especially in something like e-comm where you've got multiple sets of data to change at one time:
- inventory levels
- purchase orders
- customer info
- billing records
- etc doing that without transactions is foolish. There's a reason that we left flat files, hierarchical databases, and application-level transactions in the 70s. No need to go back - merely because one of several free databases didn't offer transactions a few years ago. Especially since more functional, more standards-compliant, and less-license-impaired alternatives exist.
When you look at the licensing - and how unnecessarily complex it is, how open to interpretation free use is, etc.
Then when you look at the the history of mysql ab - how they used to tell developers that they didn't need referential integrity, didn't need transactions, etc - until they had them.
Then when you realize that this is a commercial organization - a for profit company. Then it should be apparent where this is all headed: They're growing a user base, and hope to tighten up the licensing and pull the profits in tomorrow.
I'm not going to use the right terminology here (since it changes quarterly) but...
this is just one pricing option: you can buy everything up front, or you can pay more to have them put in 'emergency' resources - that can be added later if you need it.
This later scenario can be good if you want to avoid overbuying but still have resources available in case you wildly underestimated what you'd need.
Slashdot Mirror
While there was a bit of greedy butchery and mindless power-acquisition going on mostly what happened was story-telling, character development, working as a team, out-smarting opponents, building alliances, etc.
Grinding? In playing d&d, gurps, etc with a good group of people it *never* felt that way. We may have spent 2 years and never made it past 3rd level and struggled every step of the way - but it was a fun memorable time. Perhaps grinding happens more in computer-based "rpgs"? Or groups of kids that want nothing else than to have a 20th level character. Either way, in many years of playing I never really felt that.
Large databases use a combination of range partitioning AND indexing. If you're lucky you've also got hash partitioning - to distribute your database across N servers. Partitioning is far more general and forgiving for purposes like this than indexing.
And creating trends in your application layer then storing them in logs can work, but:
1. you still rely on figuring out in advance what you're going to need
2. if you don't load those logs back into the relational database then you can't effectively join it to all that data. And you then either must log redundant data or have useless logs.
3. grep & sed or custom reporting code against your logs is a poor substitute for any standard reporting tool or custom code against your database.
4. database features like partitioning, indexing, parallelism result in in-database aggregates being faster than logs
5. did i mention automatic query rewrite? where the database automatically converts eligible queries against the base table to actually run against the summary tables...
Note that most of these solutions come from the interwebs, social networks, etc. And it isn't so much anti-sql as it is anti-relational database (sql != rdb).
The basic premise is that we need different solutions that: can scale very high for very narrowly scoped reads & writes, don't need to perform ranged queries / reporting /etc, and don't need ACID compliance. And that may be the case. Sites like slashdot, facebook, reddit, digg, etc don't need the data quality that ebay needs.
On the other hand, ebay achieves scalability AND data quality with relational databases. And when I've worked with architectures that scale massively and avoid the relational trap for better solutions - they inevitably later regret the lack of data quality and complete inability to actually get trends and analysis of their data. It *always* goes like this:
Me: So, is this thing (msg type, etc) increasing?
Developer: No idea.
Me: Ok, so lets find out.
Developer: How?
Me: I don't know - typical approach - lets query the database.
Developer: It'll take four+ hours to write & test that query and then days to run. And when it's done we might find that we wrote the query wrong.
Me: What?!?
Developer: We had to do it this way, you can't report on 10TB databases anyhow
Me: What?!? Are you on crack? there are dozens of *100TB* relational databases out there that people are reporting on
Developer: well, we probably don't need to know what that trend is anyhow
Me: I'm outta here
> If you knew COBOL it would look familiar :)
Not really - you're thinking about stored procedure language. Which if you close one eye and squink with the other I guess could look a little COBOL-esque, that is without the column-orientation, section headings, etc.
SQL on the other hand looks like this:
SELECT dept.dept_name,
per.rank,
count(*) as personnel_rank_count
FROM dept_table dept
INNER JOIN personnel_table per
ON dept.dept_id = per.dept_id
WHERE dept.dept_type = 'internal'
GROUP BY dept.dept_name,
per.rank
ORDER BY 1,2
> learn SQL if you haven't already
Yep
Even if you prefer to use a ORM like hibernate or active record at some point:
- you'll have to modify an application that doesn't use one
- you've got a bug in the ORM and need to compare its behavior against that of raw SQL
- you'll need to diagnose problems quickly in a database - and will want to quickly write a 1-20 little queries to check everything out
- you'll need to aggregate (pre-process) some data - and given SQL's set-orientation will be able to do it faster in the database than in your application. And it'll be running huge queries - maybe generating a dozen or more temp tables. And the ORM won't support your SQL.
- etc, etc, etc
So, go ahead and use the ORM when it makes sense. Just get familiar with SQL as well - it'll save you a ton of time.
Also, back to the original thread: I see programming as building a house and writing sql as playing chess. It is completely different than writing most other code. And there are many times when I'm *very* glad that it is.
Note that Japan has an extremely rigid social structure - and that asking for help or accepting assistance from others is often problematic.
In a culture like this robots are most likely less stressful assistants than in the west.
> I know all about the Mercedes Diesels from back then. They are reliable for sure but they are slow. > Especially when combined with 4 speed transmissions with not enough low gearing. They accelerate
like a tourtise.
Must depend on the version. I had a '79 300SD with a turbo diesel that was fine. The only time it seemed slow was when climbing mountain passes in colorado. Other than that it was better than your average car. Not *fast* - but certainly more than anyone needs.
And I've driven VW turbo diesels - passats and jettas from the last ten years - and thought that they had very good acceleration and were very fun to drive. Really, at this point I don't think there's any noticeable difference between gas and diesel ownership for most people - except fuel economy.
> How could a VC firm not be intelligent enough to realize that mail-order perishables isn't going to work out well...
No - you need to think big. Really big. We're not talking merely about shipping milk & meat here, we're talking about milk & meat *portals*. In the meat portal for example, you'd find:
- forums where you can share thoughts with other meat enthusiasts - "The Meat Room"
- recreational activities for the meat-based lifestyle like http://www.hatsofmeat.com/
- meat-based recipes
- the "Meat Haus" chat room
- where you can buy or sell meat in any quantity - the "Meaty Money Pit"
I wouldn't give up on this. I know the Atkins diet has faded a bit, but perhaps it's just getting ready for a rebound? And we'll see a surge of excitement over meat again?
> Perhaps the traditional RDBMS experts will return when they can scale
> their paradigms to datasets that are measured in the tens of terabytes
> and stored on thousands of computers.
I'm not defending the authors of the article, but...
1. this article wasn't written by rdbms people, but rather by column database people. There's nothing traditional or relational about their background.
2. Every solution is a result of a variety of compromises, there is no such thing as the perfect solution outside of a context. And google's context is almost completely unique. So, it is somewhat meaningless that a solution works well for them. That doesn't mean it would work at all well for HP, Time-Warner, the State of Maine, the University of Colorado, or Ma Kettle's Fish Shack.
It appears that there is no evidence that Churchill ever wrote or said that quote, and it has been attributed to a number of others before him.
Never the less, it may be less a bit of wisdom and more a self-serving quote to justify old stodgy conservatives.
Until DR DOS 5 came out Microsoft was just willing to just let everyone suffer with DOS 4.11, 3.*, etc. Then as the number of DR DOS fans grew, Microsoft realized that they risked losing complete control over their platform and had to respond.
DOS 5 was pretty good, but it still wasn't as good as DR DOS 5, let alone DR DOS 6. And the only reason that they outsold DR DOS is that they dumped their product - by dropping their price from like $120 down to $29. DR DOS couldn't compete with microsoft deep pockets.
> I would love to see the 4th edition finally allow you to multiclass properly
Then you'll probably want to switch to a different game. Try something like GURPS - where the entire simple concept of classes was abandoned decades ago, and you can easily combine many different types of skills. Of course, how easily you master a skill will depend on your dexterity or intelligence and prerequisite skills - but still, you have total freedom.
> failing 65% of the time would be a quick way out of business
you would think so.
However, remember that you can meet requirements without meeting objectives. There are quite a few applications out there that met the requirements, but failed to meet the objectives of adaptability, usability, manageability, etc.
These apps are error-prone, frustrating, and inefficient to use. They are hated and despised by their users. But they may still be quoted as successes by IT ivory-tower folks specializing in project estimation, etc.
Lastly, I'm not talking only about custom software here. The above certainly applies to commercial solutions, to network/server/telecom architectures, etc.
> And unlike Python you can, you know, use indents to mark up the code instead of placating the language's wishes.
I was thinking about this - and couldn't come up with a scenario in which I wouldn't want my indentation to accurately reflect flow of control. This is what python enforces.
Any scenarios you can think of?
Ok, so aside from warehouse vs mart discussions - when we're talking about a data warehouse we're generally talking about a star-schema. And in this data model you've typically got:
1. many dimension tables
- columns: many wide character columns
- rows: few (generally less than 5,000)
- examples: time, location, product, organization
- bottom line - column-orientation would be great here - except these tables are typically only a few mbytes anyway and are easily cachable.
2. very few (maybe 1) central fact table
- columns: a couple dozen integer columns - mostly just keys to the dimensions
- rows: many (often billions)
- examples: product sales, security events, etc
- bottom line: column-orientation doesn't buy you much - since the columns are so small and are just integers.
3. a dozen or more summary tables (pre-aggregated subsets of the fact table)
- columns: less than a dozen
- rows: typically 1% or less than what's in the fact table
- bottom line: column-orientation doesn't buy you much here either - since the columns are small and the number of rows aren't that significant.
Now consider:
a. partitioning: data is often partitioned by time ranges since time-series or current-version queries are the most common. This works extremely well with row-orientation since all the rows can be put into their own virtual table. But it won't work with column orientation. So, for example when you need to get all data for march 2007 - you won't be able to just scan that 2% of the data, you'll have to scan 100% of the column-oriented data.
b. compression: in db2 you can expect to achieve 80% compression of your fact tables. All those integers are just simple recurring patterns. Column orientation would certainly improve on that. But by how much?
c. writing: data warehouses may be read-mostly, but more and more there's a desire to get close to real-time reporting. The warehouse that I support gets loaded every hour for one set of data, and every minute for another. And users expect one set of data to be available for reporting within 30 minutes of being created. That set comes in batches of 100,000 or more rows. If the write-delay was sufficient we wouldn't be able to meet that requirement.
So, I'd consider that there may be some advantages to column-orientation. However, if you've got a good star-schema data model, are using partitioning and compression - then the benefits shouldn't be that significant and the loss of fast writes and easy partitioning may completely offset those meagre benefits.
Well, mysql allows multiple storage engines. Each has its own proprietary file structures - anything from key-value pairs to isam.
DB2 used to support ISAM (and perhaps still does in its mainframe version?), but now mostly uses one main engine (which can store data in file system directories, in raw devices or raw volumes) but also has a completely separate xml storage engine that's pretty good. And just within the primary storage engine allows many different table types:
- mdc (multi-dimensional clustering) tables (like oracle - table is split into smaller tables on one server)
- compressed tables (gets you about 80% compression with old, nasty rows)
- typed tables (object tables)
- range partitioned tables (like oracle - table is split into smaller tables on one server)
- hash partitioned tables (partitions are split across many servers)
So, although I'm sure it would take work - I can't imagine they couldn't also offer a column-oriented table.
You're right - giving metrics to management is dangerous. And that reminds me - I was in a hurry and forgot to add a few others that address this issue. I've always resisted giving metrics that only measure part of a process - since management will always assume that the metrics cover the entire process. Giving metrics that only show efficiency encourages management to continually strip down your organization.
So, with that in mind I'd recommend also adding customer satisfaction metrics (mostly apply to paying customers that could go elsewhere), customer productivity metrics (mostly apply to internal customers), and absolutely - some kind of adaptability metrics (ability to lead or follow changes - like come up with metrics, adapt to new workflows, etc).
It's hard to do this well - but I'd recommend taking leadership over the metrics that will define your job. And if I was managing teams that insisted that refused to provide concise data on what they do - I wouldn't trust them, and might look for opportunities to reorganize them.
You really need to be using a request system to do the following, but really, if you've got more than just a couple of admins you probably should have one already.
So, having said that and just shooting from the hip, I'd look for:
- big cost-drivers:
- number of servers per administrators by os
- number of requests by day by request type by platform with average times, should cover:
- create/alter/remove a userid
- create/alter/remove a group
- assign/alter/remove user to a group
- recover a file
- os installs
- corporate policy & security stuff:
- number of os patches performed per month by os
- number of application patches performed per month by os
- number of infections caught
- general characteristics
- servers managed
- userids managed
- privileges managed
- gbytes of os & application code and data managed
Other issues like reliability are hard to meaningfully report on unless you've got a ton of servers over a lot of time.
I used to play back in the 70s and 80s.
1st edition was poorly designed, but at least it was relatively simple.
2nd edition tried to fix the design with a lot of poorly thought-out patches.
3rd edition tried to redesign the solution and managed to make it worse.
3.5 edition tried to fix the bad redesign with more bad patches.
4th edition will supposedly fix the prior bad designs.
However, this vendor has no credibility when it comes to go design - they've never really created an elegant gaming system and clearly enjoy changing things around to force players to buy $1000+ in books.
So, compare this to gurps (generic universal role playing game system at www.sjgames.com) for a second. Gurps came out around twenty years ago. They've had revisions - but the original game mechanics worked so well that the revisions are mere tweaks compared to the nonsense at wotc. This isn't to say that gurps is perfect - it could be more detailed than some groups would want - but it's a perfect example of how an elegant design ages well.
Given the options of buying the old 1st edition books used for $5 each, playing gurps (requires 1-2 books), or playing almost any other game I've got no idea why anyone would bother with d&d anymore.
> Actually, they dont leak. It is made out of concrete and polyurathane foam.
Right, that's the idea anyway: you inflate a huge bubble, go inside it, and spray a layer of polyurethane foam, then spray concrete over that. This gives you three layers from outside in:
- plastic bubble layer
- urethane foam layer
- concrete layer
The plastic bubble layer is theoretically reusable, but generally isn't. The urethane foam layer provides insulation but is fragile. The concrete is strong.
Unfortunately, the two outer layers are far less durable than the concrete. So, the next step in the failed evolution of this design was to add a second layer of concrete on the outside. The result of this was that the two concrete layers reacted differently to temperature changes - and the result was cracks.
The next step in the evolution was to add rebar to the outside layer (chain link fences sections). That stopped the cracking problem. Or so the vendor said. We'll probably fine out in ten years that that caused other problems (not the least is cost) - but they won't talk about that until they have a fix ready to offer.
Personally, I think it's a good idea - and eventually we'll have a working solution. In the meantime I would never trust the "Monolithic Dome Institute" to be up front about its problems.
> The answer to that is easy. concrete dome.
Yep, that would be great. But just like geodesic domes that preceded monolithic domes - there are unforeseen issues like:
- leakage - in the case of monolithic domes due cracking
- integration challenges - they're difficult to tie into other components
- windows - good quality windows don't come in arcs
- expense - they're not cheap to build (nor necessarily expensive)
A monolithic dome is at the very top of what I'd like to build to live in. Unfortunately, we just haven't yet worked out all the kinks. And worse, many of the kinks are brushed under the carpet by the evangelists behind them. Until years later when they admit that the prior design didn't work - but "the new design fixes that old problem that I always denied they had".
> Your examples count. My customer data has nothing to do with purchase data. The customer is an entity unto itself.
> Inventory would count, but I rarely store inventory as numerical values -- I typically don't store inventory --
> I count(*) the purchase orders.
Normally, you update inventory counts from multiple sources - so you know the number on hand (based on what has arrived) and the number on order. The complexity of this depends on number of workflows that you want to automate (generally the size of the operation).
> It would be slower, but a query cache handles that perfectly.
The speed is one issue, but not the most critical - the most important point is that if one of your operations fails it could leave the application in an inconsistent state - with some application data reflecting the change and some not.
> Certainly, there are applications that I would write that would benefit from transactions -- I simply haven't
> had the need; more specifically, the benefits haven't outweighed the costs.
I recommend that you pick a small candidate mini-project and try transactions out on that task. You will likely find that they normally simplify your work rather than add to it - while eliminating a number of data quality problems along the way.
Good luck.
> Transactions are based on "undoing" what was only partially done. Funny, but I never want to "undo" anything.
> If it was done -- even partially -- I want the partial record of it. Instead, my applications utilize atomic
> indicators to maintain database integrity.
You know - it's always fun to work out your own solution to complex problems. But...in areas that are extremely well-trodden - you really should do some research as well. And you might not find the books and literature that explain *why* we use transaction nearly as easy to find as those that explain *how* we use them.
But you still need to go there. Or spend some time discussing it with people that used to do what you're talking about twenty years ago. They could probably explain all the problems that transactions simplified, all the coding that was reduced, all the bugs that were eliminated, and all the new possible functionality enjoyed.
I'm not going to tell you that you need transactions for everything - you really don't. And in many of the batch-oriented (like data warehousing, analytic, reporting, etc) apps that I create I often don't need them. But - especially in something like e-comm where you've got multiple sets of data to change at one time:
- inventory levels
- purchase orders
- customer info
- billing records
- etc
doing that without transactions is foolish. There's a reason that we left flat files, hierarchical databases, and application-level transactions in the 70s. No need to go back - merely because one of several free databases didn't offer transactions a few years ago. Especially since more functional, more standards-compliant, and less-license-impaired alternatives exist.
When you look at the licensing - and how unnecessarily complex it is, how open to interpretation free use is, etc.
Then when you look at the the history of mysql ab - how they used to tell developers that they didn't need referential integrity, didn't need transactions, etc - until they had them.
Then when you realize that this is a commercial organization - a for profit company. Then it should be apparent where this is all headed: They're growing a user base, and hope to tighten up the licensing and pull the profits in tomorrow.
I'm not going to use the right terminology here (since it changes quarterly) but...
this is just one pricing option: you can buy everything up front, or you can pay more to have them put in 'emergency' resources - that can be added later if you need it.
This later scenario can be good if you want to avoid overbuying but still have resources available in case you wildly underestimated what you'd need.