...for ThreadRipper and i9 products to hit the market. I am building a new multi-threaded data management system that gets a lot faster as you add cores/threads so I want to benchmark it against the best CPUs from both Intel and AMD. It is not simply a system where you can run a bunch of different queries simultaneously (every server does that), but one that can also break a single query into pieces and run them in parallel. For the database functionality (just a small part of the system), if you have a query that says "SELECT name, address, zipcode FROM table WHERE name ILIKE '%Smith'" and the table has 50 million rows in it, it will run about 50% faster on a hex core than a quad core CPU with the same clock speed. Not every query can be broken into 36 pieces that are independent of each other so you will not see ever-increasing performance as you approach 36 threads, but most queries against big data will utilize as much horsepower as the hardware can throw at you.
I used to be a dedicated OS/2 user. It has some great features and I developed some applications and drivers for it. But I would not pay $99 just for the privilege of trying it out. I like open source, but I will still definitely pay for my own copy of good software (system or application). I think if we want good software, we have to support those who develop it financially. But you have to price it right if you want to build a base of loyal customers. $20 to $25 is what I call a fair price for stuff like this today. $10 is a no-brainer. If you sell a few million copies (instead of a few thousand), you will have all the cash you need. Just my 2 cents (that is what advice like this is worth these days).
I never said that SaaS or electric cars are bad. There are definite benefits to both but they are still just getting going and have a tiny fraction of total market share. I was just laughing at the assertion that all software can be in the cloud today and there is no valid reason to buy it for your computer anymore. Just like regular cars will still be around 50 years from now, desktop software will be as well.
It's like those criminals who do $100K damage to some expensive electrical equipment just so they can scrounge a few hundred dollars worth of copper. They simply don't care how much damage they do to other people as long as they get a few bucks in their pocket.
Some ideas are great in the context of where they apply, but are terrible if tried outside of that context. People tend to think that if X was a good idea for problem Y then it must also be a great idea for problem Z. Some ideas don't scale well. Some break down if you try to expand them to new applications. Solar panels might be the perfect solution to powering your cabin far from the grid. That doesn't automatically mean that they should power everything in the world. Maybe they should, but maybe they should not. It's when tons of money are thrown at something before it's time or to a poorly managed project (Solyndra for example) that they really break down.
Maybe the companies just decided that the extortionist was probably going to leak the videos anyway. Once the ransom is paid, there is absolutely nothing preventing him from releasing it. It's not like he has any morals whatsoever that might prevent him from double-crossing them. So it's just better to keep the money and not give an incentive for the next pirate hacker to try it too.
I will not buy any IoT device. The idea of smart devices is appealing, but I don't want any of them to be connected directly to the Internet. If they generate data, it needs to be stored on a device within my home that I control, not the manufacturer! If I want to send it a command from my phone, then it has to go through a secure device that once again, I control and can set the security settings on.
It looks to me like Intel/Micron wants to get their new product in the hands of early adopters first instead of trying to appeal to the mass market off the bat. This technology is radically different than what we are used to. They are looking for people who see the vision of what they are trying to accomplish and are willing to throw time, effort, and money at moving that technology forward. People who buy this first product are not the ones who think 'I am only interested if this is much better, cheaper, and faster than what I am now using!' but rather 'How can I help prove this technology by putting it through the paces because I like where this can lead to?' That is the mentality of an early adopter as opposed to the masses.
The holy grail for anyone selling anything, is the ability to charge the customer, not based on the value of the product or service, but rather on the customer's ability to pay. In the old days, this was accomplished by creating barriers for the movement of goods and services. A drug company could charge someone in rural Mexico a completely different price for a pill than they could charge someone in New York City because is was very hard for the more affluent customer to realize that it was available elsewhere for cheaper. Some goes for DVDs, books, software, and just about everything else. Now with the internet, anyone can order just about anything from anywhere. Those artificial market barriers have been broken down. This threatens the profit margins of many companies.
It doesn't help when lots of people with strong opinions (some I agree with, some I don't) take the stance of 'I believe in X and anyone who disagrees with me must be an idiot.' This is because so many people want to fight for their cause and somehow think you can attract more flies with vinegar instead of honey. They used to be just those people who would march in protest carrying some sign that called the other side stupid or evil. Now with social media, that hateful crowd has grown substantially and they don't go home and throw away the sign when it starts to rain. We see both sides of the political aisle take these kinds of approaches and even see it here on/. when people start flaming each other over what operating system or programming language they use.
I like FULL control...that's why I still program everything in ASSEMBLY. Can't have that fancy smancy compiler pushing an extra register on the stack that I know is not going to change within my function. That is a couple lost CPU cycles that I will never get back...now get off my lawn... Seriously, though I prefer C++ over most other languages (I've used about 20 over my career) because of its speed and control characteristics. But I do a lot of low-level programming, so what do I know?
The current trend in software is to distribute a problem to a bunch of commodity servers. That is a good strategy for extremely large data sets, but it can also be a crutch to compensate for bad algorithms and inefficient code. Rather than focus on getting the algorithm right and solve the problem using 80% fewer instructions; you just get more cheap computers (and electricity, and heat, and...) involved. I wonder how much this strategy has spilled over into the headcount equation. Don't get one great employee that can solve the problem by himself on a $150K salary...instead get 10 programmers that can try to do the same thing collectively at $15K each. It might look the same on the corporate balance sheet, but almost guaranteed that the solution by the one qualified engineer will be much better than the code cobbled together by the team of 10 mediocre or incompetent programmers.
...And believing that you can just raise rates and that will magically result in more government revenues is equally as silly. Like it or not, tax rates and regulations do effect people's behavior and the economy's growth. Government has to find a good balance between taxes that are too low and taxes that are too high. When you raise them too high, people will either stop putting in the same effort to earn it, or they will waste resources trying to get around them. If they are too low, you don't have enough money coming in to fund the government. It's tough to have civil dialog around the issue of fair tax rates when half the country thinks the other half is made up of either freeloaders or greedy exploiters.
I wonder if it will be any easier for Ballmer to figure out what the government spends, than it is for each of us to figure out what we actually owe in taxes? After weeks of filling out forms and finding receipts (and I use an accountant), I had to sign the form saying I agreed that the information was correct and complete. I am a programmer, but I honestly have no idea whatsoever if the check I wrote was the right amount. The only thing I am sure of is that the amount was much more than I thought it should be. I now have to pray that I didn't miss something and have the wrath of the IRS come down on me.
One doctor saying you have cancer while another says it is the flu is definitely 'different'. If both say you have lung cancer, but the second says it is a slight variation of the cancer diagnosed by the first doctor, does that meet the definition of 'different' for this study?
I wonder if these hackers would have more or less money if they put all that time coding their hacks into coding projects that make money? If you spend 100 hours to earn a $5000 prize, is that really any different than working for a real company at $50/hr?
Obviously, you have to read it from disk sometime to put it into memory. Now if you have enough memory to hold all your tables, then queries can be very fast (even table scans). But as you stated, you are dealing with databases that are much bigger than the RAM on your servers, even though your company spent a fortune to get 1.5 TB of RAM installed on them. I assume your databases have a bunch of tables that are only queried once in a while so they are kept mostly on disk. Wouldn't it be nice if you could query those tables much faster?
It is certainly impossible given the current database designs...which is precisely why I am re-inventing them. At my last company, we spent lots of money to have DBAs 'properly configure and proper index' our databases. Also, whenever the applications decided to query the tables in an entirely different way, the configuration and indexing needed to be changed. My system was still able to outperform those tables without any of the hassles. You might be right that a finely tuned Postgres database can come close to my performance, but as I stated already, I would rather have a system that doesn't need any of that work.
Thanks for your comments. That is exactly the point of my new system...being able to create fairly large tables; fill them with millions of rows; and then be able to do very fast queries (even ones that return a large percent of the data) on those tables without all the hassle of creating and maintaining indexes. Data read from disk only needs to be the minimal amount needed to satisfy the query. All queries give you the same kinds of performance advantages so you don't have to go and re-design the table just because your query pattern changes.
I take it you don't work with databases a lot. Yes, if your query (SELECT * FROM table WHERE...) only returns something like 100 rows then you only need to read in 100 pages from disk (assuming each row is in a separate page) if you have indexes on all the columns in your where clause (of course, you have to read in the indexes from disk first). If, on the other hand, your WHERE clause returns 10 million rows then you often end up reading in the whole table anyway.
How do you even read in a table with 200+ million rows from disk (even with a fast SSD) in less than a second? I tried the queries with all kinds of indexes in Postgres. Indexes work as designed if your query returns a tiny portion of the data. Once you hit something like 5% of the rows, the table scan is often faster. This of course does not even take into consideration all the performance tuning you have to do to create and maintain the indexes. Lots of indexes also slow down inserts and deletes (two copies of the data have to be changed). I would much rather have relational tables that work really fast without needing any indexes at all.
Great! The formatting screwed up my select statement. Second attempt without brackets: "Select Column1, Column5, Column19 FROM table WHERE Column32 ilike '%Apple%' OR Column48 > 1000;"
Slashdot Mirror
Old outdated technology is immune to the modern virus.
...for ThreadRipper and i9 products to hit the market. I am building a new multi-threaded data management system that gets a lot faster as you add cores/threads so I want to benchmark it against the best CPUs from both Intel and AMD. It is not simply a system where you can run a bunch of different queries simultaneously (every server does that), but one that can also break a single query into pieces and run them in parallel. For the database functionality (just a small part of the system), if you have a query that says "SELECT name, address, zipcode FROM table WHERE name ILIKE '%Smith'" and the table has 50 million rows in it, it will run about 50% faster on a hex core than a quad core CPU with the same clock speed. Not every query can be broken into 36 pieces that are independent of each other so you will not see ever-increasing performance as you approach 36 threads, but most queries against big data will utilize as much horsepower as the hardware can throw at you.
I used to be a dedicated OS/2 user. It has some great features and I developed some applications and drivers for it. But I would not pay $99 just for the privilege of trying it out. I like open source, but I will still definitely pay for my own copy of good software (system or application). I think if we want good software, we have to support those who develop it financially. But you have to price it right if you want to build a base of loyal customers. $20 to $25 is what I call a fair price for stuff like this today. $10 is a no-brainer. If you sell a few million copies (instead of a few thousand), you will have all the cash you need. Just my 2 cents (that is what advice like this is worth these days).
I never said that SaaS or electric cars are bad. There are definite benefits to both but they are still just getting going and have a tiny fraction of total market share. I was just laughing at the assertion that all software can be in the cloud today and there is no valid reason to buy it for your computer anymore. Just like regular cars will still be around 50 years from now, desktop software will be as well.
Just like everyone these days drives an electric car. Oh, wait...something like .001% do.
It's like those criminals who do $100K damage to some expensive electrical equipment just so they can scrounge a few hundred dollars worth of copper. They simply don't care how much damage they do to other people as long as they get a few bucks in their pocket.
Did anything good come out of the committee organized by Obama? or Bush? or Clinton?
Some ideas are great in the context of where they apply, but are terrible if tried outside of that context. People tend to think that if X was a good idea for problem Y then it must also be a great idea for problem Z. Some ideas don't scale well. Some break down if you try to expand them to new applications. Solar panels might be the perfect solution to powering your cabin far from the grid. That doesn't automatically mean that they should power everything in the world. Maybe they should, but maybe they should not. It's when tons of money are thrown at something before it's time or to a poorly managed project (Solyndra for example) that they really break down.
Maybe the companies just decided that the extortionist was probably going to leak the videos anyway. Once the ransom is paid, there is absolutely nothing preventing him from releasing it. It's not like he has any morals whatsoever that might prevent him from double-crossing them. So it's just better to keep the money and not give an incentive for the next pirate hacker to try it too.
I will not buy any IoT device. The idea of smart devices is appealing, but I don't want any of them to be connected directly to the Internet. If they generate data, it needs to be stored on a device within my home that I control, not the manufacturer! If I want to send it a command from my phone, then it has to go through a secure device that once again, I control and can set the security settings on.
It looks to me like Intel/Micron wants to get their new product in the hands of early adopters first instead of trying to appeal to the mass market off the bat. This technology is radically different than what we are used to. They are looking for people who see the vision of what they are trying to accomplish and are willing to throw time, effort, and money at moving that technology forward. People who buy this first product are not the ones who think 'I am only interested if this is much better, cheaper, and faster than what I am now using!' but rather 'How can I help prove this technology by putting it through the paces because I like where this can lead to?' That is the mentality of an early adopter as opposed to the masses.
The holy grail for anyone selling anything, is the ability to charge the customer, not based on the value of the product or service, but rather on the customer's ability to pay. In the old days, this was accomplished by creating barriers for the movement of goods and services. A drug company could charge someone in rural Mexico a completely different price for a pill than they could charge someone in New York City because is was very hard for the more affluent customer to realize that it was available elsewhere for cheaper. Some goes for DVDs, books, software, and just about everything else. Now with the internet, anyone can order just about anything from anywhere. Those artificial market barriers have been broken down. This threatens the profit margins of many companies.
It doesn't help when lots of people with strong opinions (some I agree with, some I don't) take the stance of 'I believe in X and anyone who disagrees with me must be an idiot.' This is because so many people want to fight for their cause and somehow think you can attract more flies with vinegar instead of honey. They used to be just those people who would march in protest carrying some sign that called the other side stupid or evil. Now with social media, that hateful crowd has grown substantially and they don't go home and throw away the sign when it starts to rain. We see both sides of the political aisle take these kinds of approaches and even see it here on /. when people start flaming each other over what operating system or programming language they use.
I like FULL control...that's why I still program everything in ASSEMBLY. Can't have that fancy smancy compiler pushing an extra register on the stack that I know is not going to change within my function. That is a couple lost CPU cycles that I will never get back...now get off my lawn... Seriously, though I prefer C++ over most other languages (I've used about 20 over my career) because of its speed and control characteristics. But I do a lot of low-level programming, so what do I know?
The current trend in software is to distribute a problem to a bunch of commodity servers. That is a good strategy for extremely large data sets, but it can also be a crutch to compensate for bad algorithms and inefficient code. Rather than focus on getting the algorithm right and solve the problem using 80% fewer instructions; you just get more cheap computers (and electricity, and heat, and...) involved. I wonder how much this strategy has spilled over into the headcount equation. Don't get one great employee that can solve the problem by himself on a $150K salary...instead get 10 programmers that can try to do the same thing collectively at $15K each. It might look the same on the corporate balance sheet, but almost guaranteed that the solution by the one qualified engineer will be much better than the code cobbled together by the team of 10 mediocre or incompetent programmers.
...And believing that you can just raise rates and that will magically result in more government revenues is equally as silly. Like it or not, tax rates and regulations do effect people's behavior and the economy's growth. Government has to find a good balance between taxes that are too low and taxes that are too high. When you raise them too high, people will either stop putting in the same effort to earn it, or they will waste resources trying to get around them. If they are too low, you don't have enough money coming in to fund the government. It's tough to have civil dialog around the issue of fair tax rates when half the country thinks the other half is made up of either freeloaders or greedy exploiters.
I wonder if it will be any easier for Ballmer to figure out what the government spends, than it is for each of us to figure out what we actually owe in taxes? After weeks of filling out forms and finding receipts (and I use an accountant), I had to sign the form saying I agreed that the information was correct and complete. I am a programmer, but I honestly have no idea whatsoever if the check I wrote was the right amount. The only thing I am sure of is that the amount was much more than I thought it should be. I now have to pray that I didn't miss something and have the wrath of the IRS come down on me.
One doctor saying you have cancer while another says it is the flu is definitely 'different'. If both say you have lung cancer, but the second says it is a slight variation of the cancer diagnosed by the first doctor, does that meet the definition of 'different' for this study?
I wonder if these hackers would have more or less money if they put all that time coding their hacks into coding projects that make money? If you spend 100 hours to earn a $5000 prize, is that really any different than working for a real company at $50/hr?
Obviously, you have to read it from disk sometime to put it into memory. Now if you have enough memory to hold all your tables, then queries can be very fast (even table scans). But as you stated, you are dealing with databases that are much bigger than the RAM on your servers, even though your company spent a fortune to get 1.5 TB of RAM installed on them. I assume your databases have a bunch of tables that are only queried once in a while so they are kept mostly on disk. Wouldn't it be nice if you could query those tables much faster?
It is certainly impossible given the current database designs...which is precisely why I am re-inventing them. At my last company, we spent lots of money to have DBAs 'properly configure and proper index' our databases. Also, whenever the applications decided to query the tables in an entirely different way, the configuration and indexing needed to be changed. My system was still able to outperform those tables without any of the hassles. You might be right that a finely tuned Postgres database can come close to my performance, but as I stated already, I would rather have a system that doesn't need any of that work.
Thanks for your comments. That is exactly the point of my new system...being able to create fairly large tables; fill them with millions of rows; and then be able to do very fast queries (even ones that return a large percent of the data) on those tables without all the hassle of creating and maintaining indexes. Data read from disk only needs to be the minimal amount needed to satisfy the query. All queries give you the same kinds of performance advantages so you don't have to go and re-design the table just because your query pattern changes.
I take it you don't work with databases a lot. Yes, if your query (SELECT * FROM table WHERE...) only returns something like 100 rows then you only need to read in 100 pages from disk (assuming each row is in a separate page) if you have indexes on all the columns in your where clause (of course, you have to read in the indexes from disk first). If, on the other hand, your WHERE clause returns 10 million rows then you often end up reading in the whole table anyway.
How do you even read in a table with 200+ million rows from disk (even with a fast SSD) in less than a second? I tried the queries with all kinds of indexes in Postgres. Indexes work as designed if your query returns a tiny portion of the data. Once you hit something like 5% of the rows, the table scan is often faster. This of course does not even take into consideration all the performance tuning you have to do to create and maintain the indexes. Lots of indexes also slow down inserts and deletes (two copies of the data have to be changed). I would much rather have relational tables that work really fast without needing any indexes at all.
Great! The formatting screwed up my select statement. Second attempt without brackets: "Select Column1, Column5, Column19 FROM table WHERE Column32 ilike '%Apple%' OR Column48 > 1000;"