Skip lists have been in the Solaris VM subsystem for a while. Back to 2.5 at least. I came across them in the source, then went wandering (at the time, through AltaVista) looking for info on them, and liked what I saw. Neat tool.
How a company with no case at all and a marketcap of 237,5M USD can be so stupid as to sue a company with a market cap of 140B USD is beyond me.
I mean, there simply cannot be another reason than to inflate the stockprice.
The key part there is no case at all.
At the time it was filed, there were internal representations made to tech staff at the Santa Cruz offices of SCOX (the former actual "Santa Cruz Operation" part of the company) that there were large sections of non-header, stolen algorithm code in Linux from SysV Linux. Santa Cruz Operation people who had no inherent faith in the Caldera side of the business saw the info and believed it. That is probably accurate as to the execs beliefs at the time.
It has subsequently been shown that there is no such thing. I personally suspect that the original comparisons misunderstood the actual origins of large parts of the code, but I have no knowledge.
At some point in the process, it became clear to any reasonable person outside SCaldera that the copyright violations they'd sued over were not factually correct. It should have been clear to them and their attorneys as well. Instead of abandoning the effort, they have kept stretching and stretching, trying to find something else to keep the hope of the lawsuit alive.
It is arguable that their actions in launching the suit were entirely honorable and honest - There have been improper code releases in the past, and anyone who's been around for long enough has probably seen a few of them.
Whether their actions in continuing the suit once the source of the original violating code examples was shown to not be within SysV were moral, legal, or reasonable, is a completely different story.
They are probably going to lose the countersuits and be dissolved, and the full attorney/client communications from SCOX and its law firms will eventually become public. If that shows that anyone on the inside showed signs of knowing how bad the case had become and was talking about it, then everyone inside is likely going to get indicted and disbarred. It's remotely possible that none of them have realized how far in the hole they are, or at least not put it down on paper, in which case it will be a little harder to hold them responsible. But odds are high that they've got a paper trail.
What IBM should do next is file a counterclaim for barratry (abusive use of legal maneuvers) against SCOX and Boise Schillers, and get a protective order to keep anyone at SCOX or BS from destroying any of those internal documents... Elsewise guilty parties may just nuke their own records and walk away from this without the legal pummelling they so richly and properly deserve.
1. IT for companies that are doing stuff out in the field, such as field science, oil prospecting/drilling, etc.
2. Lateral move into sciences, doing fieldwork. Fieldwork engineer/scientists who can also do IT fluently, particularly programming, are often golden in some fields where the combination is hard to find.
3. Aerospace - new rocket companies (XCOR, SpaceX, etc). John Carmack isn't hiring right now I think, but that might change.
4. Systems engineering, with your background you might do aerospace or naval systems engineering.
Question: are you tied down by family, or relatively mobile?
As much as I like saving energy, there are some health issues. A fair number of people are very UV sensitive - my wife has a disease which renders her vulnerable to sunburn if she sits under normal fluorescent lights for any length of time (conveniently, the hospitals around only use these... making ER visits somewhat self-defeating). There are something like a million people with the same disease in the US, and there are several other diseases that cause similar photosensitivity.
We've tried CF bulbs before - they cause the sunburn.
The California proposed ban is actually thus a Americans with Disabilities Act (US federal law) violation.
The day it was announced, I called the California assemblyman who was proposing it's office and informed them of this and asked them what the health and safety exceptions were going to be. They said they'd get right back to me...
I have no problem moving away from incandescents. Unlike some, I don't mind fluorescent color temperatures, though even the good ones seem a little off compared to the incandescents I'm used to. I live with fluorescent at work. I would pay for good LED bulbs if I could - the lifetime and hassle avoidance seem good from an economics standpoint as well as environmental, and I just changed several incandescents that burned out over the weekend. But LED isn't on the market yet.
Anyone who lives in Australia - I urge you to make sure that your government takes photosensitive people's health into account and either defer an incandescent ban until LED bulbs are available or have them provide proper exceptions (and ensure market availability) for those who need to not use CF.
I haven't worked with it up close (browse around the website regularly, but don't run it), but all the docs I have seen say SQLite uses a B-tree, not a column store. Do you have an alternate reference to such?
SQLlite says not to use it for more than a few gb to tens of gb of data. Sybase IQ, for example, is routinely run with TB plus quantities of data. It's been tested to a trillion plus rows of data and 155 TB of input data (which autocompressed down to 55 TB of diskspace required to store it).
Vertica's headed thataways, I think, but also to be lighterweight and more general purpose than IQ.
A) Is your benchmark a data warehouse type app benchmark or transactional? Column oriented is slower for transactional typically but much faster for data warehouse. I don't care how many frames per second you measure if I'm buying a LAMP web server system.
B) Your benchmark data doesn't show that you've tried to run Sybase IQ or C-store column-oriented databases against the workload.
Are you really sure that you want to be so sure about this, given that you may not be testing the right thing, and haven't tested the comparable things? 8-)
A column oriented relational database? I'd like some more details on how that works.
Column oriented is easy. Imagine a database as a set of tables, each of which has rows of data records, in organized columns (column 1 = "User name", column 2 = "User ID", column 3 = "Favorite slashdot admin", etc).
Normal row-oriented databases store records which have a row of the data: "User name", "User ID", "Favorite slashdot admin" for user row #12345.
Column oriented databases store records which have a column of the data: "User name" for user rows 1-100,000; "User ID" for user rows 1-100,000; etc.
Updates are faster with row-oriented: you access the last record file and append something, or access an intermediate record file and update one "row" across.
Searches are faster with column-oriented: you access the record file for "Favorite slashdot admin" and look for entries which say "Phred", and then output the list of rows of data which match. Instead of going through the whole database top to bottom for the search, you just search on the one column. If you have 100 columns of data, then you look through 1/100th of the total data in the search. To pull data out, you then have to look at all the column files and index in the right number of records, but that goes relatively quickly.
Indexes are useful, but column-oriented is more efficient in some ways. You don't have to maintain the indexes, and can just automatically search any column without having indexed it, in a reasonably efficient manner.
Column-oriented also lets you compress the data on the fly efficiently: all the records are the same data type (string, integer, date, whatever) and lists of same data types compress well, and uncompress typically far faster than you can pull them off disk, so you can just automatically do it for all the data and save both speed and time...
Most of those RSC cards have a firmware upgrade which will do SSH. Many sites haven't deployed that, yet, though.
In the meantime: separate LAN or VLAN for it, not routed, bridged with a secure sysadmins-only bastion host, etc.
Having an exposed telnetd on any public network has been known to be bad for years, if nothing else because it exposes YOUR PASSWORD to the world in transit. Duh.
The second is a question about motivation, not feasibility. Whether we (or future humans) WOULD do it is different from COULD we do it.
The first is not mathematically all that serious; peak total fertility rate in the US was 3.8 children per female in the 1950s (see http://en.wikipedia.org/wiki/Fertility_rate ); that's a factor of around 1.7 to 1.8 increase in population per generation, including child mortality, and with an average generation length about 32 years gives a doubling in population every roughly 40 years. The original probe could be re-launched with a full crew, 80 years after arrival, leaving 3/4th of the then-colony population to keep growing, or 120 years later leaving 7/8th.
What's more significant is how long it will take for the new colony to populate up enough to economically produce new probes, which is so dependent on economics and engineering assumptions and the local resources where they colonize that I don't know if it's reasonably predictable ahead of time. We could go into scenarios, but I don't know what a typical case would look like.
You're assuming interstellar colonization as practical. While it's a sci-fi staple, load of fun to imagine, it's quite likely not.
The only question about its feasibility is whether we start using energy sources of the appropriate magnitude or not. All the rest is just engineering.
If Humanity stays on Earth and doesn't exploit extraterrestrial resources (space based solar, mining asteroids, the moon, etc) then we probably won't have that magnitude of energy.
If we do expand out, the energy required is pretty trivial to collect, not so trivial to focus and convert to usably driving interstellar spacecraft, but that's just an economics question.
1. The seas are important to us. 2. We will not allow anyone to deny us access to the seas. 3. We started using torpedos over 100 years ago, and had all sorts of nifty cannons before that, and before those people had catapults, bows, arrows, pointy bows for ramming, etc.
Nevertheless - anyone doing serious data warehousing who cares about read performance has been using Teradata (older apps) or column-oriented Sybase-IQ (newer apps). Oracle can store over a billion rows, sure; a terabyte's a lot of data, and people have had multi-terabyte databases for the better part of a decade, for some projects.
Why? Despite all the tuning, Sybase-IQ can still run through a general purpose query into its data around ten times faster than tuned Oracle.
It may not matter in the telephone company, but for people who actually have money on the line (financial companies), huge data processing uses appropriate tools. IQ and columns win.
This is a very good question. There's a very good, but not well known answer.
Mars has a lot of dust. Earlier missions got a good dusting on the landers and rover (Viking 1 and 2, Mars Pathfinder and the Sojurner rover). The more modern missions use solar cells for power, which are blocked slowly over time as dust builds up.
Dust accumulation on Mars solar cell arrays was a big problem within the early and mid-1990s Mars research community. Researcher Geoff Landis (http://en.wikipedia.org/wiki/Geoffrey_Landis) had an experiment on the Sojurner rover with a solar cell with a little movable cover glass on it, to see how much dust accumulated over time. Results from that were a prediction that solar arrays would lose most of their power over say four to six months.
Geoff had another experiment on the Mars Surveyor 2001 lander mission, which was supposed to try using static electricity to remove all the dust off a test cell, but the mission was cancelled after the Mars Polar Lander / Mars Climate Orbiter losses.
The two Mars Exploration Rovers were the next landers we sent. The expectation was that they'd last at least 3 months (90 days), and the hope was that nothing else would wear out until the solar arrays were too obscured for them to be able to power up properly anymore, perhaps six months or so into the mission.
What actually happened is one of those unexpected bonuses that the universe throws at you at random intervals. It turns out that the Mars winds at the height of the MER solar panels are just enough stronger than they are closer to the ground that the MER solar panels built up a moderate load of dust and stabilized there. There's plenty enough power remaining (except for mid-winter on Mars) for the rovers to keep operating, and it looks like the whole solar array dust problem just goes away if you put the arrays up off the ground.
There were some people who hoped that the arrays would be kept clean by the winds, but the best models we had before the MER rovers landed was that the winds weren't nearly strong enough. Pleasant suprise, and one that makes future missions a lot easier than we'd been afraid they were going to be. But not something which was taken into account in the MER designs to start with.
There was no expectation that the arrays would last more than about six months; designing anything else to last much longer than that, other than for safety's sake to make sure that nothing else failed before the solar cells dusted up, didn't seem to make any sense beforehand.
The next two Mars rovers are going to be powered by radioisotope thermal generators (RTGs) anyways, so that they can keep driving at night and in wintertime, now that we know that the basic MER design mechanisms will last for many years on the surface. Being able to turn on some headlights and keep driving at night should triple their effectiveness or better.
Apple Computer ran a number of other computer companies out of business in the 1980's for producing 'work-alike' Apple II copies. They weren't all 'copies' either. They were separately engineered and significantly different machines.
Slashdot Mirror
Skip lists have been in the Solaris VM subsystem for a while. Back to 2.5 at least. I came across them in the source, then went wandering (at the time, through AltaVista) looking for info on them, and liked what I saw. Neat tool.
At the time it was filed, there were internal representations made to tech staff at the Santa Cruz offices of SCOX (the former actual "Santa Cruz Operation" part of the company) that there were large sections of non-header, stolen algorithm code in Linux from SysV Linux. Santa Cruz Operation people who had no inherent faith in the Caldera side of the business saw the info and believed it. That is probably accurate as to the execs beliefs at the time.
It has subsequently been shown that there is no such thing. I personally suspect that the original comparisons misunderstood the actual origins of large parts of the code, but I have no knowledge.
At some point in the process, it became clear to any reasonable person outside SCaldera that the copyright violations they'd sued over were not factually correct. It should have been clear to them and their attorneys as well. Instead of abandoning the effort, they have kept stretching and stretching, trying to find something else to keep the hope of the lawsuit alive.
It is arguable that their actions in launching the suit were entirely honorable and honest - There have been improper code releases in the past, and anyone who's been around for long enough has probably seen a few of them.
Whether their actions in continuing the suit once the source of the original violating code examples was shown to not be within SysV were moral, legal, or reasonable, is a completely different story.
They are probably going to lose the countersuits and be dissolved, and the full attorney/client communications from SCOX and its law firms will eventually become public. If that shows that anyone on the inside showed signs of knowing how bad the case had become and was talking about it, then everyone inside is likely going to get indicted and disbarred. It's remotely possible that none of them have realized how far in the hole they are, or at least not put it down on paper, in which case it will be a little harder to hold them responsible. But odds are high that they've got a paper trail.
What IBM should do next is file a counterclaim for barratry (abusive use of legal maneuvers) against SCOX and Boise Schillers, and get a protective order to keep anyone at SCOX or BS from destroying any of those internal documents... Elsewise guilty parties may just nuke their own records and walk away from this without the legal pummelling they so richly and properly deserve.
Redundant Array of Irritating Discussions?
We did both this study and the Google study in the first couple of days after FAST was over. Completely redundant....
1. IT for companies that are doing stuff out in the field, such as field science, oil prospecting/drilling, etc.
2. Lateral move into sciences, doing fieldwork. Fieldwork engineer/scientists who can also do IT fluently, particularly programming, are often golden in some fields where the combination is hard to find.
3. Aerospace - new rocket companies (XCOR, SpaceX, etc). John Carmack isn't hiring right now I think, but that might change.
4. Systems engineering, with your background you might do aerospace or naval systems engineering.
Question: are you tied down by family, or relatively mobile?
As much as I like saving energy, there are some health issues. A fair number of people are very UV sensitive - my wife has a disease which renders her vulnerable to sunburn if she sits under normal fluorescent lights for any length of time (conveniently, the hospitals around only use these... making ER visits somewhat self-defeating). There are something like a million people with the same disease in the US, and there are several other diseases that cause similar photosensitivity.
We've tried CF bulbs before - they cause the sunburn.
The California proposed ban is actually thus a Americans with Disabilities Act (US federal law) violation.
The day it was announced, I called the California assemblyman who was proposing it's office and informed them of this and asked them what the health and safety exceptions were going to be. They said they'd get right back to me...
I have no problem moving away from incandescents. Unlike some, I don't mind fluorescent color temperatures, though even the good ones seem a little off compared to the incandescents I'm used to. I live with fluorescent at work. I would pay for good LED bulbs if I could - the lifetime and hassle avoidance seem good from an economics standpoint as well as environmental, and I just changed several incandescents that burned out over the weekend. But LED isn't on the market yet.
Anyone who lives in Australia - I urge you to make sure that your government takes photosensitive people's health into account and either defer an incandescent ban until LED bulbs are available or have them provide proper exceptions (and ensure market availability) for those who need to not use CF.
http://www.sqlite.org/whentouse.html
I haven't worked with it up close (browse around the website regularly, but don't run it), but all the docs I have seen say SQLite uses a B-tree, not a column store. Do you have an alternate reference to such?
SQLlite says not to use it for more than a few gb to tens of gb of data. Sybase IQ, for example, is routinely run with TB plus quantities of data. It's been tested to a trillion plus rows of data and 155 TB of input data (which autocompressed down to 55 TB of diskspace required to store it).
Vertica's headed thataways, I think, but also to be lighterweight and more general purpose than IQ.
It takes balls to say things like that about Michael Stonebraker in the database field... ...and lack of brains or historical clue...
A) Is your benchmark a data warehouse type app benchmark or transactional? Column oriented is slower for transactional typically but much faster for data warehouse. I don't care how many frames per second you measure if I'm buying a LAMP web server system.
B) Your benchmark data doesn't show that you've tried to run Sybase IQ or C-store column-oriented databases against the workload.
Are you really sure that you want to be so sure about this, given that you may not be testing the right thing, and haven't tested the comparable things? 8-)
Sybase IQ already shows that class of speedups on lots of datasets. Proof of concept is out there...
Column oriented is easy. Imagine a database as a set of tables, each of which has rows of data records, in organized columns (column 1 = "User name", column 2 = "User ID", column 3 = "Favorite slashdot admin", etc).
Normal row-oriented databases store records which have a row of the data: "User name", "User ID", "Favorite slashdot admin" for user row #12345.
Column oriented databases store records which have a column of the data: "User name" for user rows 1-100,000; "User ID" for user rows 1-100,000; etc.
Updates are faster with row-oriented: you access the last record file and append something, or access an intermediate record file and update one "row" across.
Searches are faster with column-oriented: you access the record file for "Favorite slashdot admin" and look for entries which say "Phred", and then output the list of rows of data which match. Instead of going through the whole database top to bottom for the search, you just search on the one column. If you have 100 columns of data, then you look through 1/100th of the total data in the search. To pull data out, you then have to look at all the column files and index in the right number of records, but that goes relatively quickly.
Indexes are useful, but column-oriented is more efficient in some ways. You don't have to maintain the indexes, and can just automatically search any column without having indexed it, in a reasonably efficient manner.
Column-oriented also lets you compress the data on the fly efficiently: all the records are the same data type (string, integer, date, whatever) and lists of same data types compress well, and uncompress typically far faster than you can pull them off disk, so you can just automatically do it for all the data and save both speed and time...
KX is primarily in-memory. The competing column-oriented product is primarily Sybase IQ, which has been on the market for a while now.
Vertica's website has had all the details about what they're doing for months. They've had a Wikipedia article for a long time.
This is some new Network World definition of "Stealthy", apparently...
Most of those RSC cards have a firmware upgrade which will do SSH. Many sites haven't deployed that, yet, though.
In the meantime: separate LAN or VLAN for it, not routed, bridged with a secure sysadmins-only bastion host, etc.
Having an exposed telnetd on any public network has been known to be bad for years, if nothing else because it exposes YOUR PASSWORD to the world in transit. Duh.
The standard Wikipedia term for anyone who contributes or modifies an article is "Editor"; the distinction you're claiming is nonexistent.
The second is a question about motivation, not feasibility. Whether we (or future humans) WOULD do it is different from COULD we do it.
The first is not mathematically all that serious; peak total fertility rate in the US was 3.8 children per female in the 1950s (see http://en.wikipedia.org/wiki/Fertility_rate ); that's a factor of around 1.7 to 1.8 increase in population per generation, including child mortality, and with an average generation length about 32 years gives a doubling in population every roughly 40 years. The original probe could be re-launched with a full crew, 80 years after arrival, leaving 3/4th of the then-colony population to keep growing, or 120 years later leaving 7/8th.
What's more significant is how long it will take for the new colony to populate up enough to economically produce new probes, which is so dependent on economics and engineering assumptions and the local resources where they colonize that I don't know if it's reasonably predictable ahead of time. We could go into scenarios, but I don't know what a typical case would look like.
The only question about its feasibility is whether we start using energy sources of the appropriate magnitude or not. All the rest is just engineering.
If Humanity stays on Earth and doesn't exploit extraterrestrial resources (space based solar, mining asteroids, the moon, etc) then we probably won't have that magnitude of energy.
If we do expand out, the energy required is pretty trivial to collect, not so trivial to focus and convert to usably driving interstellar spacecraft, but that's just an economics question.
1. The seas are important to us.
2. We will not allow anyone to deny us access to the seas.
3. We started using torpedos over 100 years ago, and had all sorts of nifty cannons before that, and before those people had catapults, bows, arrows, pointy bows for ramming, etc.
Nevertheless - anyone doing serious data warehousing who cares about read performance has been using Teradata (older apps) or column-oriented Sybase-IQ (newer apps). Oracle can store over a billion rows, sure; a terabyte's a lot of data, and people have had multi-terabyte databases for the better part of a decade, for some projects.
Why? Despite all the tuning, Sybase-IQ can still run through a general purpose query into its data around ten times faster than tuned Oracle.
It may not matter in the telephone company, but for people who actually have money on the line (financial companies), huge data processing uses appropriate tools. IQ and columns win.
"I'm sorry, Dave, I can't let you drive away."
"But I just filled up!"
This is a very good question. There's a very good, but not well known answer.
Mars has a lot of dust. Earlier missions got a good dusting on the landers and rover (Viking 1 and 2, Mars Pathfinder and the Sojurner rover). The more modern missions use solar cells for power, which are blocked slowly over time as dust builds up.
Dust accumulation on Mars solar cell arrays was a big problem within the early and mid-1990s Mars research community. Researcher Geoff Landis (http://en.wikipedia.org/wiki/Geoffrey_Landis) had an experiment on the Sojurner rover with a solar cell with a little movable cover glass on it, to see how much dust accumulated over time. Results from that were a prediction that solar arrays would lose most of their power over say four to six months.
Geoff had another experiment on the Mars Surveyor 2001 lander mission, which was supposed to try using static electricity to remove all the dust off a test cell, but the mission was cancelled after the Mars Polar Lander / Mars Climate Orbiter losses.
The two Mars Exploration Rovers were the next landers we sent. The expectation was that they'd last at least 3 months (90 days), and the hope was that nothing else would wear out until the solar arrays were too obscured for them to be able to power up properly anymore, perhaps six months or so into the mission.
What actually happened is one of those unexpected bonuses that the universe throws at you at random intervals. It turns out that the Mars winds at the height of the MER solar panels are just enough stronger than they are closer to the ground that the MER solar panels built up a moderate load of dust and stabilized there. There's plenty enough power remaining (except for mid-winter on Mars) for the rovers to keep operating, and it looks like the whole solar array dust problem just goes away if you put the arrays up off the ground.
There were some people who hoped that the arrays would be kept clean by the winds, but the best models we had before the MER rovers landed was that the winds weren't nearly strong enough. Pleasant suprise, and one that makes future missions a lot easier than we'd been afraid they were going to be. But not something which was taken into account in the MER designs to start with.
There was no expectation that the arrays would last more than about six months; designing anything else to last much longer than that, other than for safety's sake to make sure that nothing else failed before the solar cells dusted up, didn't seem to make any sense beforehand.
The next two Mars rovers are going to be powered by radioisotope thermal generators (RTGs) anyways, so that they can keep driving at night and in wintertime, now that we know that the basic MER design mechanisms will last for many years on the surface. Being able to turn on some headlights and keep driving at night should triple their effectiveness or better.
The MAN B&W 14K98MC7 has nearly 8% more power (116,875 HP vs 108,920 HP for this Wartsila-Sulzer) http://www.manbw.com/engines/TwoStrokeLowSpeedProp Engines.asp?model=K98MC7
Great fact-checking to start 2007 with...
Ever worked anywhere where security concerns meant that the UNIX Admins aren't supposed to have access to the database contents?
It gets much more fun.
See http://www.linuxdevices.com/ for probably the single best central resource for what you can buy off the shelf already.
Companies like VTech http://en.wikipedia.org/wiki/Laser_128 were able to stay in the market with clean reverse-engineered ROMs.