Pretty much any marketing BS can be published though the PR Newswire for a few hundred dollars per release. Publishing of grand but unverifiable claims through the PR is a tool to increase stock sales for PinkSheet companies, like this GTXC.PK. They are not even audited for crying out loud. Why does anyone have to take them seriously? Why should such crap be posted here?
all you have to do is edit out huge swaths of the article with the comment "No citation is given, and the facts are contradicted in sources A and B."
Unfortunately it does not work that way. You remore the text, then someone reverts your changes. A petty revert war starts over the subject. If the article has a lot of attention, you have a chance to win by merit of your edits. But if the article is obscure, the more stubborn one, or the one with more free time wins. Usually the educated and knowledgeable one has a full time job and not a lot of free time:-). At some point it just occurs to you that your time can be spent more productively than proving your valid point over and over again to every idiot who comes accross the article.
Wiki does have a problem. The "no authority" notion is Wiki's main driving force, and at the same time it's the greatest fallacy.
Google is in a unique position to implement HTTP 402 Payment Required. Once they roll out their payment system, they will need to give away a browser add-on, and a web server module.
First, it's a mistake to use the pin numbers in a set sequence. The customer should not know which pin is next. Second, "Please confirm blah blah by visiting blah blah" (the phisher makes just one unusual request before he can steal) and "Please confirm blah blah by visiting blah blah and then enter pin numbers" on a page that does not have the customer's name or the real account number or any other real details is a bit different. The victim must be more gullible.
Then this was just a single example. The banks have enough options to stop phishers. Like electronic pin generators, IP-verification, etc. A single bank will not do it because it's a burden for no gain. It can only be done only if mandated by law. Schnider is smart fellow.
There is a simple and cheap solution that banks can implement to stop phishers cold. They can use disposable pins for every outgoing transaction. When the customer opens an account, he gets a plastic card with pins. The card is either given in person, or sent by postal mail. Whenever the customer makes a payment, he is prompted by the bank to enter a pin. One pin - one transfer, the pin is never reused. The standrd credit-card sized card can hold about a hundred pins covered with scratch-off paint. The phishers can get the password and see the contents of the account, but they will not be able to transfer the money out of the account.
Why don't the banks do it? Becuse such system would seem like an unnecessary hassle to the majority of customers.
The issue exists because PostgreSQL supports user extensible aggregate functions, whereas most DB's don't.
Yes, I know. I think it was Tom Lane back in the days of 6.5 who gave me the same exactly reasons for not addressing the issue of slow aggregates. I understand your reasoning, I just disagree with it. How many users write their own aggregates compare to the number of users who just want to use COUNT/MIN/MAX? Your admittedly technologically cool solution is addressing needs of maybe 5% of users leaving the majority out in the cold.
Simple fact is, each and every RDBMS has their own distinct sets of issues, problems, caveats.
Yes. When I write a complex query I expect to tweak it or tweak the indexes for optimal performance. But when I write SELECT COUNT(*) FROM tbl I expect it to be snappy no matter how big the table is. I never expect to rewrite basic stuff like SELECT id, SUM(a) FROM tbl GROUP BY id because it's too slow. I don't exppect to run maintanence jobs (VACUUM) every hour during peak load times becuase otherwise the database does not perform.
If something as simple as "select count(*)" is a serious problem, then chances are, you shouldn't be using a database to begin with. While clearly more cumbersome, triggers are easily implemented which allow you to easily and rapidly store derived values, such as counts.
Re-read your words. What you are describing is exactly the steep learning curve: a simple thing requiring a complex solution. Is it not? What about porting issues? Just think about how much effort is wasted by reimplementing every simple query with triggers.
In fact, I would argue that it's certainly far less steep than Oracle or MSSQL
Less than Oracle (not "far" less) - yes. Less than MSSQL - no. But that could be just our shop since we have the people with the right experience. The query optimizer in MSSQL 7.0/2K is significantly better than the one in PgSQL 7.4 though. The PgSQL 7.4 optimizer is on par with MSSQL 6.5. Writing queries to MSSQL is easier than to PgSQL.
I would be interested to hear the details of your RDBMS, which seemingly could not be handled by PostgreSQL
The show stopper was a table with the market data. It had the bars with Open/High/Low/Close. It had about 20 columns, about 5-10 million records for a total 1-1.5GB. Not really that big. The new data was continuosly inserted, a lot of selects were issued. The majority of selects were issued for the recently inserted data, with values obviously not accounted for in the index statistics. Some selectes (about 10%) were issued for historical data.
The whole table was never kept in RAM, although the RAM was sufficient. The relatively rare queries for historic data trashed the cache, so selects for recent data had to refill it from disk. The continuous stream of inserts screwed index statistics slowing selects. Running vacuum analyze was taking a few minutes. During this time it slowed selects to an unacceptable level.
I don't claim I understand PgSQL internals well. That's just my take on the problem.
Just the same, version 8.0 does represent many improvements. Hopefully you and your guys will be willing to check it out again.
We will. As soon as we are sure that doing so won't get us in trouble with the accounting types:-). The last try of two weeks of paid time wasted was not pretty.
IMO, it has a "very steep" learning curve because so many people have
to unlearn all the bad habbits from MySQL
Read your next statement Sadly, select count(*) is still slow and don't imagine that changing any time soon. Is using count(*) a bad MySQL habbit? PgSQL is difficult because it has unusual problems, not expected from experience with Oracle/MSSQL/MySQQL.
As for aggregates, I know PgSQL functions can't use indexes by design. But addressing the most common cases of COUNT/MIN/MAX/SUM and maybe
AVG might be possible. For example, MIN/MAX queries can be rewritten with ORDER/LIMIT 1. It's really unexpected for someone new to PgSQL (but with a lot of experince with other databases) that COUNT(*) on large tables is unuseable AND there is no work around.
Assuming you had it configured properly, to correctly reuse all of your pages, you may still needed to run vacuum more than once per
day.
One quite capable DB admin worked on it full time for about two weeks, another one consulted. They tried everything imaginable. Still, running vacuum slowed selects by 5-10 times. That was not acceptable during the day, thus we could run it only at night. And that was not often enough.
Two experienced people spent two weeks and failed to configure it for sufficient performance. Yes, you can say they were not experienced enough. I can also say that the configuration was overly difficult.
I assume you mean as a non-dedicated server. Also, assuming you're running on Linux
I mean a dedicated server, as in nothing but PgSQL 7.4 on a FreeBSD 4.9. Say, the DB is 1GB, server has 2GB RAM. PgSQL should just suck everything in RAM and run read queries directly from there. Why can't it do it? Or maybe it can, but how to configure it in such a way is really non-obvious.
PgSQL had a very steep lerning curve even for people with experience in other databases. Various unusual stuff just popped up in places which could not be expected from previous experience elsewhere.
How about COUNT(*) and other aggregates? Is is still as slow in 8.0 as it used to be in 6.* and 7.*?
I guess 8.0 fixed a long standing problem of
select... where small_int_column=1 not using indexes. That was a serious problem for those just starting with PgSQL because it was totally unexpected.
MySQL is nearly maintanence-free. It just runs and runs. PgSQL needs VACUUM and friends. Once we tried to move a database (from MSSQL) with market data to PgSQL (I think it was 7.4). A lot of stuff was inserted, and a lot of selects performed. Running vaccum/vacuum analyze once in 24 hours was not enough! By the end of the day performance of selects was abysmal. Every time we ran VACUUM ANALYZE, it took longer, and longer, and longer.
What about the problem with memory management and cache? I see it's being addressed in 8.0. I have to test 8.0, but with 7.4 it was impossible to run PgSQL efficiently as a dedicated server - grab all available memory and cache everything aggressively. It was very frugal about memory even when it was not necessary.
And, of course, the query optimizer. But that's understandably difficult. I have to see if improvements in 8.0 are significant
Any computer model is as good as the underlying assuptions and the initial data. This research seems to be more political than scientific. Pro-Kyoto people must maintain a sufficient level of public fear to receive further funding.
Give me money and in a similar study I will prove that by 2050 up to 30% new species will be created due to global cooling.
Repeat after me People need to eat, and will do whatever it takes to ensure they get food and shelter.
Nice rant. I just get a suspicion that when you say "people" you actually mean "american people". Can you explain in a few simple words why you believe 300 million americans need food and shelter more than the other 5700 million people?
Anyone who needs to use COUNT() is completely out of luck. Can't use PgSQL. This problem has not been fixed in the latest release:
From http://www.postgresql.org/docs/7.4/static/function s-aggregate.html Users accustomed to working with other SQL database management systems may be surprised by the performance characteristics of certain aggregate functions in PostgreSQL when the aggregate is applied to the entire table (in other words, no WHERE clause is specified). In particular, a query like
SELECT min(col) FROM sometable;
will be executed by PostgreSQL using a sequential scan of the entire table.
Meaning horrendously slow for large tables. "Replication", "Enterprise features" when basic stuff like this is broken. This is a shame
1. The engine can be started when the airplane is at 5,000 meters in the air. They are not proposing to replace the conventional engine with the new one, but rather supplement it.
2. When the plane is over the Atlantic on the flight to London noise at the ground level is not an issue.
3. If the plane is supersonic, the sound transmission through the air is not a big problem for the pilot. Only the sound transmitted through the body of the craft has to be dealt with.
The article is actually quite short sighted. It's like someone from late 19th century predicting that by mid-20th steam-powered machinery will be so common, that it will ruin the horse industry. OMG! All people currently tending horses will become unemployed! What should we do?
I predict this: if a computer-based intelligence reaches the human level, then all todays concerns, including employment at McD will be absolutely irrelevant.
You seem to overestimate progress of artificial neural nets development.
Quite possible. I am very optimistic about the progress.
They show some impressive results in limited domains (mainly recognition and good old gradient search problems)
Show me a domain which is not limited:-). Actully look ap Wolfgang Maass and read his works. He has them online. It's a lot more than that.
And yes, I do know of liquid state machine, as of Boltzman machine, backpropagation nets, Hopfield networks etc.
Well, then you missed it. LSM is a very specific concept.
Re:Two unknowns dont make stuff work
on
AI Going Nowhere?
·
· Score: 1
> The algorithms however are lightyears from being applicable to real world problems.
Not light-years, just years:-). Maybe a decade or so.
> None of the "biologically inspired" algorithms today can compete with state of the art machine > learning techniques.
Well, Maass and Natschlager (sp?) show how their NNs outperform conventional neural nets by at least an order of magnitude. Maybe you mean expert systems? If so, it's like comparing apples to oranges. Or actually like comparing airplanes (expert systems) to rockets (NN) when you want to get to the moon (strong AI).
Another point. HNC (hnc.com) has been sold to Fairchild for tens of millions. I don't think it was paid for something which cannot compete. Check out Robert Heht-Nielsen's (spelling?) book in your favorite book shop.
Re:Will we ever have *real* AI?
on
AI Going Nowhere?
·
· Score: 5, Interesting
> You'll never have real, true intelligence
Define "real, true intelligence":-)
> You can try to simulate that, but so far > simulation consists of what amounts to a > gazillion 'if' tests
That's what tradiditonal AI school is doing. Yes, you are correct. It won't go anywhere. On the other hand spiking neural networks are very promising. Search google for "liquid state machine". These researches are making progress novadays, not Minsky.
Maybe the problem is Minsky himself?
on
AI Going Nowhere?
·
· Score: 5, Interesting
It's not the AI which is going nowhere. It's the traditional approaches to AI such as Minsky's symbolic logic which are not going anywhere. Seach google for Henry Markram, Maass, Tsodyks. Their research seems very promising.
Yes, SMTP is hopelessly broken, I can agree with that, but the rest of your comment is mistargeted.
Open relays are no longer the primary mechanism of spam delivery these days. Spam propagates through open proxies and tojaned hosts (jeem & friends).
Careless e-mail admins are not responsible for the bulk of spam. It's large DSL ISPs like rr.com and chello.nl. They run huge pools of home/small business DSLs. They absolutely do not care what happens on those hosts as long as customers pay. Probably as a matter of policy.
Slashdot Mirror
Pretty much any marketing BS can be published though the PR Newswire for a few hundred dollars per release. Publishing of grand but unverifiable claims through the PR is a tool to increase stock sales for PinkSheet companies, like this GTXC.PK. They are not even audited for crying out loud. Why does anyone have to take them seriously? Why should such crap be posted here?
Unfortunately it does not work that way. You remore the text, then someone reverts your changes. A petty revert war starts over the subject. If the article has a lot of attention, you have a chance to win by merit of your edits. But if the article is obscure, the more stubborn one, or the one with more free time wins. Usually the educated and knowledgeable one has a full time job and not a lot of free time :-). At some point it just occurs to you that your time can be spent more productively than proving your valid point over and over again to every idiot who comes accross the article.
Wiki does have a problem. The "no authority" notion is Wiki's main driving force, and at the same time it's the greatest fallacy.
Google is in a unique position to implement HTTP 402 Payment Required. Once they roll out their payment system, they will need to give away a browser add-on, and a web server module.
First, it's a mistake to use the pin numbers in a set sequence. The customer should not know which pin is next. Second, "Please confirm blah blah by visiting blah blah" (the phisher makes just one unusual request before he can steal) and "Please confirm blah blah by visiting blah blah and then enter pin numbers" on a page that does not have the customer's name or the real account number or any other real details is a bit different. The victim must be more gullible.
Then this was just a single example. The banks have enough options to stop phishers. Like electronic pin generators, IP-verification, etc. A single bank will not do it because it's a burden for no gain. It can only be done only if mandated by law. Schnider is smart fellow.
There is a simple and cheap solution that banks can implement to stop phishers cold. They can use disposable pins for every outgoing transaction. When the customer opens an account, he gets a plastic card with pins. The card is either given in person, or sent by postal mail. Whenever the customer makes a payment, he is prompted by the bank to enter a pin. One pin - one transfer, the pin is never reused. The standrd credit-card sized card can hold about a hundred pins covered with scratch-off paint. The phishers can get the password and see the contents of the account, but they will not be able to transfer the money out of the account.
Why don't the banks do it? Becuse such system would seem like an unnecessary hassle to the majority of customers.
The first civilian nuclear power plan was Soviet:
http://en.wikipedia.org/wiki/Nuclear_power
The site:
http://www.banryu.jp/index_e.html
Images:
http://www.banryu.jp/press/index_e.html
Here is another one with the same pictures:1 69765
http://www.chron.com/cs/CDA/ssistory.mpl/nation/3
For example, exceptionally poor performance of aggregates like COUNT(*), relatively weak optimizer.
See Wiki entry for more.
The issue exists because PostgreSQL supports user extensible aggregate functions, whereas most DB's don't.
Yes, I know. I think it was Tom Lane back in the days of 6.5 who gave me the same exactly reasons for not addressing the issue of slow aggregates. I understand your reasoning, I just disagree with it. How many users write their own aggregates compare to the number of users who just want to use COUNT/MIN/MAX? Your admittedly technologically cool solution is addressing needs of maybe 5% of users leaving the majority out in the cold.
Simple fact is, each and every RDBMS has their own distinct sets of issues, problems, caveats.
Yes. When I write a complex query I expect to tweak it or tweak the indexes for optimal performance. But when I write SELECT COUNT(*) FROM tbl I expect it to be snappy no matter how big the table is. I never expect to rewrite basic stuff like SELECT id, SUM(a) FROM tbl GROUP BY id because it's too slow. I don't exppect to run maintanence jobs (VACUUM) every hour during peak load times becuase otherwise the database does not perform.
If something as simple as "select count(*)" is a serious problem, then chances are, you shouldn't be using a database to begin with. While clearly more cumbersome, triggers are easily implemented which allow you to easily and rapidly store derived values, such as counts.
Re-read your words. What you are describing is exactly the steep learning curve: a simple thing requiring a complex solution. Is it not? What about porting issues? Just think about how much effort is wasted by reimplementing every simple query with triggers.
In fact, I would argue that it's certainly far less steep than Oracle or MSSQL
Less than Oracle (not "far" less) - yes. Less than MSSQL - no. But that could be just our shop since we have the people with the right experience. The query optimizer in MSSQL 7.0/2K is significantly better than the one in PgSQL 7.4 though. The PgSQL 7.4 optimizer is on par with MSSQL 6.5. Writing queries to MSSQL is easier than to PgSQL.
I would be interested to hear the details of your RDBMS, which seemingly could not be handled by PostgreSQL
The show stopper was a table with the market data. It had the bars with Open/High/Low/Close. It had about 20 columns, about 5-10 million records for a total 1-1.5GB. Not really that big. The new data was continuosly inserted, a lot of selects were issued. The majority of selects were issued for the recently inserted data, with values obviously not accounted for in the index statistics. Some selectes (about 10%) were issued for historical data.
The whole table was never kept in RAM, although the RAM was sufficient. The relatively rare queries for historic data trashed the cache, so selects for recent data had to refill it from disk. The continuous stream of inserts screwed index statistics slowing selects. Running vacuum analyze was taking a few minutes. During this time it slowed selects to an unacceptable level.
I don't claim I understand PgSQL internals well. That's just my take on the problem.
Just the same, version 8.0 does represent many improvements. Hopefully you and your guys will be willing to check it out again.
We will. As soon as we are sure that doing so won't get us in trouble with the accounting types :-). The last try of two weeks of paid time wasted was not pretty.
IMO, it has a "very steep" learning curve because so many people have to unlearn all the bad habbits from MySQL
Read your next statement Sadly, select count(*) is still slow and don't imagine that changing any time soon. Is using count(*) a bad MySQL habbit? PgSQL is difficult because it has unusual problems, not expected from experience with Oracle/MSSQL/MySQQL.
As for aggregates, I know PgSQL functions can't use indexes by design. But addressing the most common cases of COUNT/MIN/MAX/SUM and maybe AVG might be possible. For example, MIN/MAX queries can be rewritten with ORDER/LIMIT 1. It's really unexpected for someone new to PgSQL (but with a lot of experince with other databases) that COUNT(*) on large tables is unuseable AND there is no work around.
Assuming you had it configured properly, to correctly reuse all of your pages, you may still needed to run vacuum more than once per day.
One quite capable DB admin worked on it full time for about two weeks, another one consulted. They tried everything imaginable. Still, running vacuum slowed selects by 5-10 times. That was not acceptable during the day, thus we could run it only at night. And that was not often enough.
Two experienced people spent two weeks and failed to configure it for sufficient performance. Yes, you can say they were not experienced enough. I can also say that the configuration was overly difficult.
I assume you mean as a non-dedicated server. Also, assuming you're running on Linux
I mean a dedicated server, as in nothing but PgSQL 7.4 on a FreeBSD 4.9. Say, the DB is 1GB, server has 2GB RAM. PgSQL should just suck everything in RAM and run read queries directly from there. Why can't it do it? Or maybe it can, but how to configure it in such a way is really non-obvious.
PgSQL had a very steep lerning curve even for people with experience in other databases. Various unusual stuff just popped up in places which could not be expected from previous experience elsewhere.
How about COUNT(*) and other aggregates? Is is still as slow in 8.0 as it used to be in 6.* and 7.*?
I guess 8.0 fixed a long standing problem of select ... where small_int_column=1 not using indexes. That was a serious problem for those just starting with PgSQL because it was totally unexpected.
MySQL is nearly maintanence-free. It just runs and runs. PgSQL needs VACUUM and friends. Once we tried to move a database (from MSSQL) with market data to PgSQL (I think it was 7.4). A lot of stuff was inserted, and a lot of selects performed. Running vaccum/vacuum analyze once in 24 hours was not enough! By the end of the day performance of selects was abysmal. Every time we ran VACUUM ANALYZE, it took longer, and longer, and longer.
What about the problem with memory management and cache? I see it's being addressed in 8.0. I have to test 8.0, but with 7.4 it was impossible to run PgSQL efficiently as a dedicated server - grab all available memory and cache everything aggressively. It was very frugal about memory even when it was not necessary.
And, of course, the query optimizer. But that's understandably difficult. I have to see if improvements in 8.0 are significant
Any computer model is as good as the underlying assuptions and the initial data. This research seems to be more political than scientific. Pro-Kyoto people must maintain a sufficient level of public fear to receive further funding.
Give me money and in a similar study I will prove that by 2050 up to 30% new species will be created due to global cooling.
Read Crichton's recent lecture for a more elaborate discussion.
Repeat after me People need to eat, and will do whatever it takes to ensure they get food and shelter.
Nice rant. I just get a suspicion that when you say "people" you actually mean "american people". Can you explain in a few simple words why you believe 300 million americans need food and shelter more than the other 5700 million people?
So when will someone do Ruby?
Not soon. Ruby cannot be embedded in a threaded application without using a giant mutex. Only one thread at a time can call Ruby interpreter.
Anyone who needs to use COUNT() is completely out of luck. Can't use PgSQL. This problem has not been fixed in the latest release:
Meaning horrendously slow for large tables. "Replication", "Enterprise features" when basic stuff like this is broken. This is a shame
1. The engine can be started when the airplane is at 5,000 meters in the air. They are not proposing to replace the conventional engine with the new one, but rather supplement it.
2. When the plane is over the Atlantic on the flight to London noise at the ground level is not an issue.
3. If the plane is supersonic, the sound transmission through the air is not a big problem for the pilot. Only the sound transmitted through the body of the craft has to be dealt with.
Things are never clear cut black and white.
The article is actually quite short sighted. It's like someone from late 19th century predicting that by mid-20th steam-powered machinery will be so common, that it will ruin the horse industry. OMG! All people currently tending horses will become unemployed! What should we do?
I predict this: if a computer-based intelligence reaches the human level, then all todays concerns, including employment at McD will be absolutely irrelevant.
Here is a bit dated article which has A LOT more insight.
All of the spiking networks I've seen were nothing more than state machines that depend on numeric comparisons
All of human brains I've heard of are nothing more than water with some organics which depend on chemical reactions.
You seem to overestimate progress of artificial neural nets development.
Quite possible. I am very optimistic about the progress.
They show some impressive results in limited domains (mainly recognition and good old gradient search problems)
Show me a domain which is not limited :-). Actully look ap Wolfgang Maass and read his works. He has them online. It's a lot more than that.
And yes, I do know of liquid state machine, as of Boltzman machine, backpropagation nets, Hopfield networks etc.
Well, then you missed it. LSM is a very specific concept.
> The algorithms however are lightyears from being applicable to real world problems.
:-). Maybe a decade or so.
Not light-years, just years
> None of the "biologically inspired" algorithms today can compete with state of the art machine
> learning techniques.
Well, Maass and Natschlager (sp?) show how their NNs outperform conventional neural nets by at least an order of magnitude. Maybe you mean expert systems? If so, it's like comparing apples to oranges. Or actually like comparing airplanes (expert systems) to rockets (NN) when you want to get to the moon (strong AI).
Another point. HNC (hnc.com) has been sold to Fairchild for tens of millions. I don't think it was paid for something which cannot compete. Check out Robert Heht-Nielsen's (spelling?) book in your favorite book shop.
> You'll never have real, true intelligence
:-)
Define "real, true intelligence"
> You can try to simulate that, but so far
> simulation consists of what amounts to a
> gazillion 'if' tests
That's what tradiditonal AI school is doing. Yes, you are correct. It won't go anywhere. On the other hand spiking neural networks are very promising. Search google for "liquid state machine". These researches are making progress novadays, not Minsky.
It's not the AI which is going nowhere. It's the traditional approaches to AI such as Minsky's symbolic logic which are not going anywhere. Seach google for Henry Markram, Maass, Tsodyks. Their research seems very promising.
The correct formula is 27/log2(6) = 10.44 :-)
Yes, SMTP is hopelessly broken, I can agree with that, but the rest of your comment is mistargeted.
Open relays are no longer the primary mechanism of spam delivery these days. Spam propagates through open proxies and tojaned hosts (jeem & friends).
Careless e-mail admins are not responsible for the bulk of spam. It's large DSL ISPs like rr.com and chello.nl. They run huge pools of home/small business DSLs. They absolutely do not care what happens on those hosts as long as customers pay. Probably as a matter of policy.