The same exact thing happens in every other language I mentioned. In Ruby:
arg="'; rm foo; #" puts arg
I'm not sure why you're so passionately tied to the shell, but you're demonstrably wrong to make this claim here:
Shell is safe, all other languages are not.
Can you show me a language which isn't safe with that particular example?
If you insist I actually call/bin/echo, I can do that, too, as I demonstrated:
system 'echo', arg
Works, exactly, naturally, the way you'd expect -- and anyone who's been doing Ruby for very long would expect APIs not to require manual string concatenation or interpolation. This usage is right there in the docs, right exactly next to the docs on the "unsafe" version of system.
However, I don't think your 'echo' command above is safe:
arg='-e \n' echo $arg
Now, I don't know the correct escape codes offhand, but if I recall, it is possible to gain control of a terminal and execute arbitrary code that way. (If you doubt that, just 'cat' a typical binary file, chances are, you'll accidentally execute a bunch of gibberish commands.)
Shell is built to call external programs. Other languages are not.
You are making a claim about all other languages, yet you admit to not knowing one I've mentioned here. Are you really prepared to say that there is no language which is built to call external programs other than something compatible with/bin/sh?
The difference being that the user's input (you typing) is directly going to the prompt, whereas the user input in a shellscript is only able to enter via parameters.
Now you've changed the topic. I said "interactive" and "noninteractive", I didn't mention shell scripts.
It's a little hidden in the ruby manual.
Are you joking?
ri Kernel#system
On a recent (1.9) version of Ruby, I get this:
system([env,] cmd [, arg,...] [,options])
along with this description:
The arguments are processed in the same way as for +Kernel::exec+.
Hmm, let's try that:
ri Kernel#exec
And we find:
If optional arguments, sequence of +arg+, are not given, that argument is taken as a line that is subject to shell expansion before being executed. If one or more +arg+ given, they are passed as parameters to _command_ with no shell expansion.
That's very near the top of that page. It may be "buried" in the manual, but it's no deeper than 'man' in the shell.
There are thousands more bad programmers out there than good ones. And I have the feeling that the ratio is all the time going down.
And a bad programmer will find a way to write bad code in any language. The shell doesn't make this particularly easier -- how do I loop over a collection of strings which may contain spaces?
You can write COBOL in any language.
If languages would be more complex to do the bad system call then the good one, this would already improve general computer security.
By a small amount, perhaps. But it wouldn't change the basic fact that a bad programmer is going to introduce tons of security flaws regardless.
The solution is not to coddle the bad programmers. The solution is to make the right approach obvious and easy for the good ones.
The patent system is made up of scientists and engineers,
I have seen patents which would call that into serious doubt.
What I find most interesting is that its biggest proponents are people within the software industry itself, but usually not the real innovators.
And how do you judge whether or not they're "real innovators"? Let me guess, by how many patents they've filed?
Are you saying software simply can't be inventive?
Hardly. In fact, the problem seems to be largely that patents are holding back inventions. From the patents which currently exist, there is simply no way for me to avoid infringing on someone's patent and also write any software at all, without an army of patent lawyers to dig through all those patents and tell me what not to write -- a process which would significantly slow innovation, if indeed I could write anything new at all.
As it is, the way large corporations seem to handle this is to file their own patents as fast as they can, so that when (not if) they infringe on someone, there's hopefully enough mutual infringement that they can work out a licensing arrangement. It is, in other words, a sort of software MAD.
What I find truly disgusting is that software patents, unlike copyrights, can be vague enough that I could legitimately invent something, in a cleanroom, without being aware of the patent I'm infringing, and be hit with a patent lawsuit for some mathematical truth I end up using. That's right, it's not just invention, but underlying principles of the universe which are protected here.
And that's a worst-case scenario -- but what if I want to support something for the purposes of interoperability? Consider H.264, and tell me HTML5 video is not innovative. Tell me Firefox is not innovative. But because of the patent issues surrounding H.264, Firefox refuses to support it, and certainly, if they were to implement it themselves -- even if they wrote their own decoder from scratch, without looking at any of the code for the official H.264 decoders -- they'd still be in violation. It's not terribly hard to find similar examples where, only because there weren't sufficient patents (or because companies chose not to enforce them), we have interoperability -- would OpenOffice be where it is today if the Microsoft Office formats were patented?
Stop for a moment and consider what the world would look like if web standards were patented. If HTML, CSS, JavaScript, and so on could only be implemented by those willing to pay a licensing fee to a central authority. If any third-party re-implementation of TCP/IP would likely result in a lawsuit. Would you really want to live in such a world?
Well, you're a patent attorney, so don't answer that. But why should any sane person who is not a patent attorney want to see HTML patented?
So the inventors can be rewarded... really? Do you really think Tim Berners-Lee has received no rewards for his efforts?
That you can't possibly think of something in software that anyone else couldn't have thought of, even given the exact same problem set?
That's sometimes a good idea, but not always. Because Adobe either doesn't have patents on PDF, or doesn't use them, people can send me documents and expect me to be able to read them with any of a half-dozen PDF readers I have, and I can expect to be able to work with them -- chop them up, rearrange them, print them, and so on. Certainly, there are other ways of solving the same problem, such as PostScript, but that doesn't decrease the problem -- if some people can read and write PostScript, and others can read and write PDF, we can't communicate, no matter how "innovative" one of them might be.
Also: Why would you want to force people to reinvent the wheel? One of the largest problems in software today is NIH syndrome, and you want to increase that? Seriously?
doesn't slashdot have a quote feature? Could not find one except typing the <quote> tags. Awkward.
Not particularly. Also, there's <ecode> for sample code.
Because you call the command with an empty argument, then you call the rm. So the rm is executing in your interactive shell, not in the program you want to demonstrate to be exploitable.
In other words, I am exploiting the shell.
In other words, anywhere that has a "system"-type call, where a command can be assembled as a string (rather than an array of string arguments), is a potential avenue for abuse. What about my example implied the shell must be interactive?
You are mixing up typing in a command into an interactive shell and calling a shell with a parameter.
Again, no, this is not "typing in". Try the following in Ruby:
arg = "'; rm foo; #" system "echo '#{arg}'"
The same in Perl:
my $arg = "'; rm foo; #"; system "echo '$arg'";
The same in Python:
arg = "'; rm foo; #" import os os.system("echo '"+arg+"'")
I could continue, but I doubt you'd find PHP to be any better about this, and I don't want to touch it. Point is, blame the language or the shell, but either way, executing shell commands this way is a bad idea. Executing them the right way is a hassle, unless you're already in the shell, but unless I'm missing something, implementing a web service in the shell is a hassle.
And that's before we start examining commandline programs for security.
I hope you see the difference between the interactive shell and...
The only real difference between an interactive shell and a noninteractive shell is the way the prompt is displayed. That's why shell scripts are so cool to begin with -- if you know how to type it into an interactive shell, you probably know how to make a basic script, and as your scripting abilities improve, those translate into increased proficiency with the shell.
Yes, I understand the difference between a command executed in the context of a shell and commandline arguments passed to a shell script ($1, $2, etc).
yes, he did not use CPAN but system calls to run other unix utilities. Like calling mailx to send an email (using a user supplied email address, wow!) and he took 3 weeks to get 1 day worth of work done.
Ew.
Not knowing ruby I don't know how it's calling external commands. If it's doing an equivalent of "sh -c $command" you need to sanitize arguments.
That depends. The same method can be used to do either way:
system "echo #{myvar}" system 'echo', myvar
The first is vulnerable unless I trust myvar. The second is safe, assuming I trust ENV['PATH'], and if I don't, it accepts absolute paths as well.
Well not really a webservice, but what about a chatroom?
Either way.
By the way, I've seen games like tetris written in Bash, so I don't doubt it can be done. I just doubt it's sane -- it strikes me as similar to hanoimania.
This is the cool feature of shell, it does not in fact execute strings which look like they may break your system.
Neither does any other language I know.
As I said, it's the border between the two. I can combine strings as many ways as I like, and they won't jump out and eval themselves -- but if I 'eval' an untrusted string, I'm hosed. If I stay entirely within the shell, I can combine strings any way I like (though it seems slightly trickier), and they won't jump out and eval themselves -- but if I 'sh -c $foo', I'm hosed.
Everything that is a gamble and sometimes works and sometimes doesn't - does not. PC gaming or the Android store are good examples.
I have about similar rates of failure with PC games as I do with Xbox games, for various definitions of "failure".
The difference is, certain kinds of failure -- for example, cheating -- is something I can mitigate myself in the PC world (dedicated servers), but in the console world, we have to wait for a fix from on high.
Also, mods and various indie games. Lugaru is not a gamble, and as far as I know, is not available on any console. Ditto for World of Goo, Knytt Stories, etc.
Current rumours say that about a quarter of all apps in the Android store(s) are malware.
That is a serious problem, but not at all related. I don't care if Android stores have similar review processes -- I don't care if they even restrict languages. The point is, I'm not locked into any one Android store (unless I go with AT&T).
While I agree, they've got a lot farther to go to gain back the trust they've lost. Why would I commit to a platform where they could, on a whim, outlaw my entire fucking language?
The only way I'd consider iOS for anything serious is if they remove the approval barrier entirely, so that there's no longer a single schizophrenic point of failure for the entire software ecosystem.
Which also means, in other words, that this is nearly entirely useless -- at least, I wasn't having any issues whatsoever with the reverse-engineered drivers other than the lack of firmware.
Sorry, too much hassle to do quoting correctly here:-)
Quoting is precisely what's crucial to the disagreement.
test.sh '';rm -rf/#'
Have you tried this?
No, because I value my data. I've tested it with something similar, and I'll amend it to this:
test.sh '';rm -rf / #'
If spaces are allowed, this is not any harder to implement than the original (wrong) version.
This is not even passing the parameter to the test.sh and would barf your system as well if you use an equivalent test.pl
*facepalm*
You're right, it's not, which is the point. It doesn't care what parameter gets passed to test.sh -- this is someone exploiting a commandline I've constructed, and even if test.sh blows up, they don't care, they've managed to execute a 'nuke everything' command.
And invoking a separate Perl script is no better than invoking a separate shell script. The point is to avoid invoking a separate process at all.
So please try your claims before using them in arguments.
The fact that I've just tried it and the difference was a single space suggests that my argument was valid, just very, very slightly imprecise. Do you really care that much? If so, maybe I should fire up a VM and present screenshots proving that the command I've provided now will, in fact, nuke every file the current user has access to.
There is no real exec call in shell.
...what?
Even using the exec instruction explicitely, you won't execute parameters.
...what? What are you talking about? The above example doesn't rely on parameters being executed. Go ahead, try it, I dare you.
Maybe you you meant the reverse?
Nope, though "shell script" is irrelevant here; the vulnerability is the shell itself whenever I try to system() or exec() anything.
If you don't use CPAN, you will call the shell from perl. At least that's what most users do: system()
Right. And this presents all sorts of opportunities for abuse. Of course...
Anyway, in shell you call stuff which is not a shell from the shell. And this works wery well even when you're ignoring security concerns.
Your shell example using a cmdline variable is vulnerable, and it's not uncommon to want a variable like that.
I recently directed someone to use CPAN to get his work done, and the resulting perl was just an accumulation of various calls to other utilities.
"Other utilities" meaning CPAN modules, or non-Perl stuff?
And we paid for that guy!
How quickly did he get his stuff done? Would you rather pay someone who knows how to glue together the appropriate existing code (CPAN or otherwise), or someone who reinvents the wheel at every turn? Even if they took the same amount of time, I'd much rather maintain something which uses proven libraries than something which reinvents the wheel, badly.
Injections galore! Transforming the perl into simple shell (what it in effect is) and the injetions are gone.
Yes, if what you want is a shell script, I would agree that the shell is usually the best way to do it. It is possible to avoid trivial injections by using the multi-arg version of 'system' (at least in Ruby), but that's a hassle.
But why is everybody using system()? What is wrong with exec()?
Erm, where would you ever use exec when system would work, unless Perl has perverted those calls well beyond their original reason?
does it corrupt the system, well no.
Do you know that for certain? How familiar are you with the arguments convert can take? I count thirteen pages, an
Same issue as SQL injection. If you're going to use things like exec() directly, it then becomes an issue of the security of test.sh, and not of the shell itself.
Also arguing that convert may or may not have unexpected behaviour does not differ if you use perl or bash to call it. We discuss the problematic of perl ADDING insecurity compared to shell.
When was I ever suggesting that Perl should call 'convert' directly?
However, the shell requires multiple independent programs to be used at every step, each of which has its own quirks and calling conventions. CPAN, by contrast, is at least going to have similar behavior and calling conventions, and except where individual functions add "magic" (like that 'open' behavior), you get the added security of being able to treat data as data.
Compare that to any point at which you call a shell script from something which is not a shell script -- you have to be careful, or you're going to treat data as code.
Because if you have that `rm *` parameter in the perl program, you will actually execute it, but not in shell!
Erm, where in Perl would that parameter be executed where it wouldn't in the shell?
The equivalent perl construct written in shell is:
eval $command
You never do that in shell, as
$command
works as well. And then you are safe.
Erm... If $command is constructed insecurely, you have precisely the same problem in the shell as in Perl.
If you have read my previous post, I *did* execute it with spaces.
Maybe I missed something, but I actually took your example and attempted it with spaces.
Just convert did not know anything about the -r parameter.
In other words, you're still allowing the user to pass arbitrary parameters to convert, which is still a bug.
The list of problematic characters is long.
And the solution is trivial -- use sysopen, no sanitizing needed.
Never mind that I can't remember the last time I tried to open a filename provided by user data.
In shell, as well as in perl, you need to know convert's parameter list.
I'm sorry...
You seem to be misunderstanding my position, and what I thought this discussion was about.
I thought it was about the difference between, say, calling "convert" from Perl (or PHP, or Ruby, or Python, etc) and calling the ImageMagick bindings directly.
And I can assure you, the way I use ImageMagick, there's no option for the variable containing user data to ever be interpreted by libmagick as anything other than user data, no need to audit an impressive list of options to verify security.
Unless you are seriously considering using the shell to implement a web service (which is what TFA seemed to be referring to), that is the most likely use case here -- TFA is suggesting we call "convert" from whatever we wrote our web service in, while I'm suggesting we use native bindings.
Try executing that with arguments which have spaces. Now imagine a few which have dashes, or slashes, or...
I don't know convert's arguments by heart, but I bet I could find something exploitable in there, given free reign to pass any arguments I want, which is more or less what you've provided here.
With shell you don't need to read any manual or sanitize fancy characters.
I've just demonstrated that you do. Both. Without reading the manual, you can't know that 'convert' is safe to use in that way. Without sanitizing characters, you can't know that you can use $x the way you just did.
If perl would have been programmed with security in mind, the default (i.e. empty mode) would have been a forced '<' and if you want to call a program, you should need to make it '<|' or '>|' or anything funky like that.
How would that be any more secure? Wouldn't it make much more sense for these to be separate calls?
A random search skipping the links from perl.org brings up quickly a first instruction example how to read a file.
And covering buffer overruns in a first introduction to C wouldn't be great, either. But, I'm sorry, the manual is there.
And we don't talk about my programming business but the thousands of people who don't care or think about those problems. "Oh it works, then it must be ok".
Those people will generate security hazards no matter what language they use -- as you seem to have already done with the shell.
you can much more easily inject natsy things into perl or php than into shell.
Erm... how so?
Have you ever seen a shell executing data?
Uhm... yes? That's what they do. Ever heard of a shell script?
I would need to call "sh" or "eval" to make this work.
So imagine a hypothetical script in which you do something like this stupid little example:
my $cmdline = "convert $x.svg $x.png"; system($cmdline);
And in this case, $x is user input. Well, now you're screwed if anyone enters spaces, so you do this:
my $cmdline = "convert '$x.svg' $x.png'";
Now you've got a classic SQL-injection-style vulnerability. What happens if a user sets $x to "'; rm -rf/; #"? That could be more devastating than a traditional SQL injection vulnerability.
So how, exactly, is Perl more vulnerable than that?
On the other side, is perl which calls without you knowing "sh" on simple things as opening a file.
Sorry, I have to call bullshit on this. Here's the perldoc on 'open'. It tells you exactly under which circumstances the filename will be interpreted as a command. If you don't want to sanitize a pipe character from the beginning and end of your string, there's always sysopen, which is mapped directly to fopen -- if this is vulnerable, then every single program on your system is vulnerable.
If you can't be bothered to RTFM, what business do you have programming?
And again, how is this more vulnerable than the point at which you call the shell from a perl script?
Maybe I'm weird, but the only one of those I've done for close to 10h at a stretch is flying. Also: My couch has outlets, so does the waiting room at the doctor's office, and so do many airplanes.
Breaking down those usages:
Reading magazines and books,
An e-reader would be better and several times cheaper.
casual email
I like being able to touch-type when writing email, "casual" or otherwise, and a laptop is a lot more convenient than an iPad + keyboard. That pretty much leaves:
browsing the web
And here, unfortunately, I browse the web too actively. I bookmark things, and thanks to having a laptop, my bookmarks are always available. I write about them -- things like this Slashdot post do require a keyboard. I view content which wouldn't work on an iPad (Flash), and I do things to my web experience (user scripts, user stylesheets, full-blown extensions) which wouldn't be allowed on the iPad.
I can see it being somewhat nicer for one of the above, maybe. I certainly can't see it as what the ads describe as "Nothing short of a revolution," or what you describe as "Nothing like it!"
Aren't you a little confused about the difference between "a right" and "being right"?
I'm using a colloquial sense, here. How about this: Who would you view as having an informed enough opinion about the iPad to be taken seriously when they say it sucks? Would it have to be someone with enough Stockholm Syndrome to buy one anyway, and continue to use it, even though they hate it?
Do you understand the problem here? It really does seem like your opinion is that if anyone criticizes the iPad, they are automatically "pulling uninformed nonsense out of their ass."
E-ink is slightly better for novels (and I've read novels on the iPad, it's just fine), but it blows for anything with formatting / color / pictures.
Wait. What does "formatting" have to do with it? The e-ink screens I've seen have plenty of resolution to handle any kind of custom fonts you like.
So that pretty much leaves reading magazines. Cool, but not worth $500 for a magazine-reading appliance, which is all it ever will be to me until it gives me control.
If you are writing a program that touches more than two persistent data stores, it is too complicated.
Others have already mentioned cases where multiple datastores make sense. A trivial example: One database to handle user data, another to handle blobs (image conversions, etc) -- bonus if the second store can do its own conversions; a third to handle logging -- that's already three, and that's before we start considering things like RESTful services, which can function as intelligent datastores of their own...
If Linux can do it, you shouldn't.
Unless you're not on Linux. And, specifically:
Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job.
If you're doing something that truly works as a shell script, and isn't part of a larger app, I agree. However, PIL likely performs better, and it removes the shell as an issue -- if you thought SQL injection was bad, wait till you have people exploiting your shell commands. You can do it safely, but why would you bother, when you've got libraries that accept Python (or Perl, or Ruby) native arguments, rather than forcing you to deal with commandline arguments? Why do you want to check return values, when you can have these native libraries throw exceptions?
Parallelize When You Have To, Not When You Want To
If you don't at least think about parallelization in the planning stage, it's going to be painful later on. It's easy to build a shared-nothing, stateless architecture and run it in a single-threaded way. It's hard to build a stateful web service with huge, heavyweight sessions, and then make it run on even two application servers in the future. Possible, but awkward, to say the least.
For example, if you are doing web crawling, and you have not saturated the pipe to the internet, then it is not worth your time to use more servers.
...unless, maybe, it's CPU-bound? And this is odd to mention in a section about parallelization -- wouldn't slow servers be a prime candidate for some sort of parallelization, even on a single machine, even if it's evented?
If you have a process running and you want it to be restarted automatically if it crashes, use Upstart.
Cool, but it looks like Upstart is becoming a Maslow's Hammer for this guy. Tools like Nagios, Monit, and God exist for a reason -- one such reason is knowing when and why your processes are dying even if they're spread across a cluster.
NoSQL is NotWorthIt
People who have read my other posts likely know where I stand on this, but...
Redis, even though it's an in-memory database, has a virtual memory feature, where you can cap the amount of RAM it uses and have it spill the data over to disk. So, I threw 75GB of data at it, giving it a healthy amount of physical memory to keep hot keys in...
So you found out an in-memory database wasn't suitable when you have far more data than physical memory? Great test, there.
Redis was an unknown quantity...
Maybe so, but that wasn't terribly hard to guess.
Yes, maybe things could have been different if I used Cassandra or MongoDB...
So maybe you should've benchmarked a NoSQL database which is actually designed to solve the problem you're trying to solve? Just a thought.
especially if something like PostgreSQL can do the same job.
If PostgreSQL could do the same job, the current generation of NoSQL databases wouldn't have been invented. Unless something's changed, PostgreSQL can't scale beyond a single machine for writes, unless you deliberately shard at the application layer, which would violate his rule about multiple datastores, wouldn't it?
Emergency decisions could be controlled from the ground.
Because there'd never be any sort of interference between the plane and the ground...
Hijack this, bitch.
So now it can be hijacked with a cell phone, instead of box cutters. And now the TSA will start banning personal electronics on planes, making air travel even less pleasant, even though it would also be possible to do the same thing from the ground.
There are things you can do with a laptop that you can't do with an iPad. That's not really a surprise considering the iPad is not and was not meant to be a laptop.
Then what, exactly, is it meant to be?
It's too big to fit the "mobile device" category of things which fit in my pocket, yet it's barely smaller than a full-sized, clamshell laptop, never mind tablets, netbooks, and the like.
Similarly there are things you can do with a desktop PC that you can't do with a laptop.
What things, specifically? That gap, to the extent that it exists, is rapidly narrowing, largely because there are multiple competing manufacturers of laptops.
Anyway, don't buy one if you don't want one but don't go around claiming you know it sucks.
Wait, are you saying I don't have a right to say it sucks because I haven't bought one, and don't want one? Wouldn't that be exactly the behavior of someone who thinks it sucks?
Think about it. Who, in your opinion, has the right to criticize the iPad?
I tried reading books on a laptop -- it's not convenient.
Works well enough for me, but if I wanted a portable device for book-reading, I'd start by looking for something with e-ink, at least. If that's really the only application you can find, you overpaid by about 3-4 times for a worse screen than a Kindle.
the 10h battery life makes it very usable.
Yet laptops exist with similar battery lives. Laptops which can run Flash.
That was the first thing I noticed -- because the other things like it I've tried would, for instance, allow me to install Firefox. The built-in browser has serious limitations even for browsing traditional pages -- I can't adjust the font size or otherwise "zoom" the page without introducing horizontal scrolling. It's great at scrolling, but only the whole page -- any way of creating a scrollable subset of a page (frames, iframes, divs with overflow=scroll, anything) is impossible to scroll.
Yeah, not just awkward, but as far as I could figure out, impossible.
Never mind that I multitask. And I don't just mean being able to play music while doing something else, I mean being able to have multiple apps open side-by-side, tiled on the screen. Did I just never figure out how to do that on an iPad?
I could come up with a dozen other examples, and several of them are much less valid on an iPhone, but I really don't get the iPad. It's a giant iPhone that can't make phone calls.
10h in full use is what I get.
I got about 8h from a netbook, circa 2004. Yes, 2004, before it was called a netbook.
No. I hate Flash, always have. Use Flashblock. And before having an iPad I got an N900, and it proved without a doubt that Jobs is right, it's disastrous on mobile, esp. for battery life.
And did you use Flashblock on your N900?
And if you don't understand that battery life is one of the most important factors for mobile applications, it's no surprise you keep rehashing irrelevant nonsense.
Mostly because it's a matter of opinion. I'm never very far from some sort of power, and I never find battery life to be an issue. I'd love more battery life. I certainly enjoyed my 8h notebook above -- more like 9-12h if it wasn't at "full" use, and past that, I can't really justify more battery life, as when I get up to 12h or so, I've already been home for more than long enough to plug it in for the night.
By contrast, there's a large number of things I simply cannot do with the iPad, which I could do even with that circa-2004 laptop, without "jailbreaking" it, which was seen as illegal until very recently. Right now, jailbreaking is still likely to void my warranty, while even installing an alternate OS on that circa-2004 laptop would not.
And that's before we get into place where I actually do prefer the laptop form-factor. Touch is interesting, but I'm much faster with a real keyboard. An iPad would at best compliment a laptop, and if I have my laptop everywhere, why would I bother?
Against all these considerations, battery life is one of the least important factors when I purchase a mobile device.
Of course, once I have one, I'll certainly tweak it to improve battery life, and adjust my usage patterns to that. Even on laptops, I don't run Flash unless I have to. But I do run it from time to time -- far better to be able to burn a little battery and accomplish a task, than have way more battery life not being able to.
That's the point I don't think we'll ever agree on -- I just don't get why it's better to have more battery life that you can't use, and that's if I accept your premise.
With DRM Apple had 80% of the digital music download market as of 2008.
Which also means there was 20% of the market going elsewhere -- and is that counting piracy? It certainly isn't counting physical sales.
But they do have to be rooted (pre 2.2) to do something as simple as actually installing an app on the SD card and since most Android devices come with limited internal memory don't you think that's kind of silly?
Silly? Yes. But you pointed out that 2.2 added this capability, so you can hardly claim that this was a deliberate restriction, as opposed to the mere lack of a feature.
By contrast, Apple has made it clear that not only do they have no intention of allowing people to sell iPhone apps through any channels other than the App Store, they also have every intention of tightening the restrictions on said App Store when it suits them -- such as with the "original language" clause.
Well the anecdotal experience of "what your friends did" doesn't match up to real world statistics.
If you want to talk about evidence, you could provide some.
Of course people were pirating music. But in the context of purchased digital music, since iTunes came on-line its been by far the dominant digital music retailer.
And since it came on-line, certainly when it was DRM'd, it was a worse experience than piracy, money aside. Maybe my sample set is skewed, but I did find that people who bought from iTunes once or twice were not likely to repeat the experience once they understood the implications of that DRM. And yes, a few were bitten by it.
As opposed to Android devices that have to be "rooted" to install applications of your choice
Um, bullshit.
Yes, they have to be rooted to make some changes -- possibly including removing crapware. They do not have to be rooted simply to install applications of your choice.
Try installing any apps that are not on the market place on any Android phone sold by AT&T.
Ah, you mean this -- good to know, but these are far from the only android phones. Good thing I'm on Verizon.
Killing flash is a Good Thing, but the methods being used are despicable, and have too much collateral damage. It means, for instance, that I cannot develop iPhone apps in Perl, Python, Ruby, even C#, no matter how efficient or "native" their runtimes get -- even if they compile to raw Objective-C -- because they must be originally written in a very short list of languages. And even if these were acceptable languages, I'd have to be very careful not to implement any sort of auto-updating system, or ever use tools like eval, because that would be a potential way around Apple's censorship^Wapproval process.
I'm sorry, but that is not acceptable. I want Flash to die, but this is not the way.
To put it another way, I would very much like for religion to die. However, the right way to go about this is to promote critical thinking, and have lots of interesting and civil discussions. The wrong way to go about this is to try to ban religions -- Scientology has been banned from a few countries, and I believe France has banned the Burka, and as much as I dislike both Scientology and Islam, I cannot support either of these actions.
If Flash dies, great! It won't change my opinion about iStuff one iota -- Apple needs to die, if anything, more than Flash does.
the best you can do is write down the URL to view it on your computer later.
And a mobile device should provide you plenty of opportunities to build an electronic list of interesting URLs, then open them all in tabs as soon as you get home.
Well, there is simply no "later" in my ADHD based world:)
How many places could you buy mainstream music DRM free from the top 4 labels before 2007? You really think "many" people were buying FLAC encoded music? What is your definition of "many"?
Good questions, but I do know that those who weren't buying FLAC-encoded or otherwise DRM-free music were also largely shying away from buying music online at all. My more tech-savvy friends were pirating, my less tech-savvy friends were burned once or twice by iTunes and never bought from it again.
What software does Apple make that has any form of copy protection?
Let's see...
um...
I dunno...
MAC OS?
Yeah, let's start with that. And then we can talk about iOS, which still has to be fucking jailbroken before it'll let you install apps of your choice, not of Apple's choice.
even that is lenient.
Sorry, what? I can't hear you over Apple's rampant hypocrisy.
DRM is DRM, no matter how "lenient" it is or not. I like Steam's compromise, for instance, but I'd still call bullshit on Valve if they tried to claim they were anti-DRM.
But sometimes you have to step back from the computer, and realize that much of what you do requires extensive expertise.... Installing Flash block?
Sometimes, you have to take a bold step and realize that end-users can learn. However...
therefore, most people will get their battery drained by that POS Flash.
I think Android does something like Flashblock -- a click-to-activate plugin system -- by default.
And btw, I'm surprised at the number of Linux users here supporting this utmost POS Flash.
I'm not supporting Flash, I'm supporting choice. I hate when I'm forced to use Flash, much more so than when I'm forced to use a Java applet.
And before you point out I have an iPad, that's my only piece of proprietary equipment, and I bought it because there is no similar product from anyone.
Erm... Have you heard of tablet computers? That's like claiming there's no similar product to the iPod simply because they're all still called "mp3 players" and not "iPod killers".
until at least 6 months in the future.
Geez. So you're an early-adopter of hardware -- can't wait 6 months? Really? But in the same breath, you'll willingly disable even the option of a proprietary but de-facto and very old standard?
Again, it's not that I'm supporting flash, I just find your position more than a bit hypocritical. Examine your own motives -- I wouldn't be terribly surprised that you're justifying the lack of Flash furiously only as a kind of Stockholm Syndrome because you want to justify your truly, truly crippled tablet computer purchase.
The music industry wanted Apple to license it's DRM to competitors.
Right -- because that would be needed to have a DRM "standard" not entirely controlled by Apple.
Slashdot said that the only reason that Apple said that was because they knew that the industry would never allow them to sell DRM free music
Did it now? I never said any such thing.
What I do distinctly remember about this whole affair was that, while Amazon may have come after Apple and EMI, there have been other services offering DRM-free music well before iTunes and Amazon, services which didn't require special client software just to download a fucking song.
This is what Steve Jobs said in 2007 before anyone was selling DRM free mainstream music
And when piracy was in full swing, and many (including me) were sticking to indie flac files at best, mostly physical media or piracy.
Please note where Apple said that DRM "doesn't work".
Yet they use it on their own devices, for their own software. They have no problem saying it doesn't work for other people's content, but their own is not only going to be DRM'd, but they're not above pulling the DMCA to protect it.
Slashdot Mirror
The same exact thing happens in every other language I mentioned. In Ruby:
I'm not sure why you're so passionately tied to the shell, but you're demonstrably wrong to make this claim here:
Shell is safe, all other languages are not.
Can you show me a language which isn't safe with that particular example?
If you insist I actually call /bin/echo, I can do that, too, as I demonstrated:
Works, exactly, naturally, the way you'd expect -- and anyone who's been doing Ruby for very long would expect APIs not to require manual string concatenation or interpolation. This usage is right there in the docs, right exactly next to the docs on the "unsafe" version of system.
However, I don't think your 'echo' command above is safe:
Now, I don't know the correct escape codes offhand, but if I recall, it is possible to gain control of a terminal and execute arbitrary code that way. (If you doubt that, just 'cat' a typical binary file, chances are, you'll accidentally execute a bunch of gibberish commands.)
Shell is built to call external programs. Other languages are not.
You are making a claim about all other languages, yet you admit to not knowing one I've mentioned here. Are you really prepared to say that there is no language which is built to call external programs other than something compatible with /bin/sh?
The difference being that the user's input (you typing) is directly going to the prompt, whereas the user input in a shellscript is only able to enter via parameters.
Now you've changed the topic. I said "interactive" and "noninteractive", I didn't mention shell scripts.
It's a little hidden in the ruby manual.
Are you joking?
On a recent (1.9) version of Ruby, I get this:
along with this description:
The arguments are processed in the same way as for +Kernel::exec+.
Hmm, let's try that:
And we find:
If optional arguments, sequence of +arg+, are not given, that argument is taken as a line that is subject to shell expansion before being executed. If one or more +arg+ given, they are passed as parameters to _command_ with no shell expansion.
That's very near the top of that page. It may be "buried" in the manual, but it's no deeper than 'man' in the shell.
There are thousands more bad programmers out there than good ones. And I have the feeling that the ratio is all the time going down.
And a bad programmer will find a way to write bad code in any language. The shell doesn't make this particularly easier -- how do I loop over a collection of strings which may contain spaces?
You can write COBOL in any language.
If languages would be more complex to do the bad system call then the good one, this would already improve general computer security.
By a small amount, perhaps. But it wouldn't change the basic fact that a bad programmer is going to introduce tons of security flaws regardless.
The solution is not to coddle the bad programmers. The solution is to make the right approach obvious and easy for the good ones.
FYI, you can link directly to the time you want.
The patent system is made up of scientists and engineers,
I have seen patents which would call that into serious doubt.
What I find most interesting is that its biggest proponents are people within the software industry itself, but usually not the real innovators.
And how do you judge whether or not they're "real innovators"? Let me guess, by how many patents they've filed?
Are you saying software simply can't be inventive?
Hardly. In fact, the problem seems to be largely that patents are holding back inventions. From the patents which currently exist, there is simply no way for me to avoid infringing on someone's patent and also write any software at all, without an army of patent lawyers to dig through all those patents and tell me what not to write -- a process which would significantly slow innovation, if indeed I could write anything new at all.
As it is, the way large corporations seem to handle this is to file their own patents as fast as they can, so that when (not if) they infringe on someone, there's hopefully enough mutual infringement that they can work out a licensing arrangement. It is, in other words, a sort of software MAD.
What I find truly disgusting is that software patents, unlike copyrights, can be vague enough that I could legitimately invent something, in a cleanroom, without being aware of the patent I'm infringing, and be hit with a patent lawsuit for some mathematical truth I end up using. That's right, it's not just invention, but underlying principles of the universe which are protected here.
And that's a worst-case scenario -- but what if I want to support something for the purposes of interoperability? Consider H.264, and tell me HTML5 video is not innovative. Tell me Firefox is not innovative. But because of the patent issues surrounding H.264, Firefox refuses to support it, and certainly, if they were to implement it themselves -- even if they wrote their own decoder from scratch, without looking at any of the code for the official H.264 decoders -- they'd still be in violation. It's not terribly hard to find similar examples where, only because there weren't sufficient patents (or because companies chose not to enforce them), we have interoperability -- would OpenOffice be where it is today if the Microsoft Office formats were patented?
Stop for a moment and consider what the world would look like if web standards were patented. If HTML, CSS, JavaScript, and so on could only be implemented by those willing to pay a licensing fee to a central authority. If any third-party re-implementation of TCP/IP would likely result in a lawsuit. Would you really want to live in such a world?
Well, you're a patent attorney, so don't answer that. But why should any sane person who is not a patent attorney want to see HTML patented?
So the inventors can be rewarded... really? Do you really think Tim Berners-Lee has received no rewards for his efforts?
That you can't possibly think of something in software that anyone else couldn't have thought of, even given the exact same problem set?
That's sometimes a good idea, but not always. Because Adobe either doesn't have patents on PDF, or doesn't use them, people can send me documents and expect me to be able to read them with any of a half-dozen PDF readers I have, and I can expect to be able to work with them -- chop them up, rearrange them, print them, and so on. Certainly, there are other ways of solving the same problem, such as PostScript, but that doesn't decrease the problem -- if some people can read and write PostScript, and others can read and write PDF, we can't communicate, no matter how "innovative" one of them might be.
Also: Why would you want to force people to reinvent the wheel? One of the largest problems in software today is NIH syndrome, and you want to increase that? Seriously?
The reality is that we d
doesn't slashdot have a quote feature? Could not find one except typing the <quote> tags. Awkward.
Not particularly. Also, there's <ecode> for sample code.
Because you call the command with an empty argument, then you call the rm. So the rm is executing in your interactive shell, not in the program you want to demonstrate to be exploitable.
In other words, I am exploiting the shell.
In other words, anywhere that has a "system"-type call, where a command can be assembled as a string (rather than an array of string arguments), is a potential avenue for abuse. What about my example implied the shell must be interactive?
You are mixing up typing in a command into an interactive shell and calling a shell with a parameter.
Again, no, this is not "typing in". Try the following in Ruby:
The same in Perl:
The same in Python:
I could continue, but I doubt you'd find PHP to be any better about this, and I don't want to touch it. Point is, blame the language or the shell, but either way, executing shell commands this way is a bad idea. Executing them the right way is a hassle, unless you're already in the shell, but unless I'm missing something, implementing a web service in the shell is a hassle.
And that's before we start examining commandline programs for security.
I hope you see the difference between the interactive shell and...
The only real difference between an interactive shell and a noninteractive shell is the way the prompt is displayed. That's why shell scripts are so cool to begin with -- if you know how to type it into an interactive shell, you probably know how to make a basic script, and as your scripting abilities improve, those translate into increased proficiency with the shell.
Yes, I understand the difference between a command executed in the context of a shell and commandline arguments passed to a shell script ($1, $2, etc).
yes, he did not use CPAN but system calls to run other unix utilities. Like calling mailx to send an email (using a user supplied email address, wow!) and he took 3 weeks to get 1 day worth of work done.
Ew.
Not knowing ruby I don't know how it's calling external commands. If it's doing an equivalent of "sh -c $command" you need to sanitize arguments.
That depends. The same method can be used to do either way:
The first is vulnerable unless I trust myvar. The second is safe, assuming I trust ENV['PATH'], and if I don't, it accepts absolute paths as well.
Well not really a webservice, but what about a chatroom?
Either way.
By the way, I've seen games like tetris written in Bash, so I don't doubt it can be done. I just doubt it's sane -- it strikes me as similar to hanoimania.
This is the cool feature of shell, it does not in fact execute strings which look like they may break your system.
Neither does any other language I know.
As I said, it's the border between the two. I can combine strings as many ways as I like, and they won't jump out and eval themselves -- but if I 'eval' an untrusted string, I'm hosed. If I stay entirely within the shell, I can combine strings any way I like (though it seems slightly trickier), and they won't jump out and eval themselves -- but if I 'sh -c $foo', I'm hosed.
And of course, there are oth
Everything that is a gamble and sometimes works and sometimes doesn't - does not. PC gaming or the Android store are good examples.
I have about similar rates of failure with PC games as I do with Xbox games, for various definitions of "failure".
The difference is, certain kinds of failure -- for example, cheating -- is something I can mitigate myself in the PC world (dedicated servers), but in the console world, we have to wait for a fix from on high.
Also, mods and various indie games. Lugaru is not a gamble, and as far as I know, is not available on any console. Ditto for World of Goo, Knytt Stories, etc.
Current rumours say that about a quarter of all apps in the Android store(s) are malware.
That is a serious problem, but not at all related. I don't care if Android stores have similar review processes -- I don't care if they even restrict languages. The point is, I'm not locked into any one Android store (unless I go with AT&T).
Nope. It's that innovative.
While I agree, they've got a lot farther to go to gain back the trust they've lost. Why would I commit to a platform where they could, on a whim, outlaw my entire fucking language?
The only way I'd consider iOS for anything serious is if they remove the approval barrier entirely, so that there's no longer a single schizophrenic point of failure for the entire software ecosystem.
Which also means, in other words, that this is nearly entirely useless -- at least, I wasn't having any issues whatsoever with the reverse-engineered drivers other than the lack of firmware.
Sorry, too much hassle to do quoting correctly here :-)
Quoting is precisely what's crucial to the disagreement.
test.sh '';rm -rf /#'
Have you tried this?
No, because I value my data. I've tested it with something similar, and I'll amend it to this:
If spaces are allowed, this is not any harder to implement than the original (wrong) version.
This is not even passing the parameter to the test.sh and would barf your system as well if you use an equivalent test.pl
*facepalm*
You're right, it's not, which is the point. It doesn't care what parameter gets passed to test.sh -- this is someone exploiting a commandline I've constructed, and even if test.sh blows up, they don't care, they've managed to execute a 'nuke everything' command.
And invoking a separate Perl script is no better than invoking a separate shell script. The point is to avoid invoking a separate process at all.
So please try your claims before using them in arguments.
The fact that I've just tried it and the difference was a single space suggests that my argument was valid, just very, very slightly imprecise. Do you really care that much? If so, maybe I should fire up a VM and present screenshots proving that the command I've provided now will, in fact, nuke every file the current user has access to.
There is no real exec call in shell.
...what?
Even using the exec instruction explicitely, you won't execute parameters.
...what? What are you talking about? The above example doesn't rely on parameters being executed. Go ahead, try it, I dare you.
Maybe you you meant the reverse?
Nope, though "shell script" is irrelevant here; the vulnerability is the shell itself whenever I try to system() or exec() anything.
If you don't use CPAN, you will call the shell from perl. At least that's what most users do: system()
Right. And this presents all sorts of opportunities for abuse. Of course...
Anyway, in shell you call stuff which is not a shell from the shell. And this works wery well even when you're ignoring security concerns.
Your shell example using a cmdline variable is vulnerable, and it's not uncommon to want a variable like that.
I recently directed someone to use CPAN to get his work done, and the resulting perl was just an accumulation of various calls to other utilities.
"Other utilities" meaning CPAN modules, or non-Perl stuff?
And we paid for that guy!
How quickly did he get his stuff done? Would you rather pay someone who knows how to glue together the appropriate existing code (CPAN or otherwise), or someone who reinvents the wheel at every turn? Even if they took the same amount of time, I'd much rather maintain something which uses proven libraries than something which reinvents the wheel, badly.
Injections galore! Transforming the perl into simple shell (what it in effect is) and the injetions are gone.
Yes, if what you want is a shell script, I would agree that the shell is usually the best way to do it. It is possible to avoid trivial injections by using the multi-arg version of 'system' (at least in Ruby), but that's a hassle.
But why is everybody using system()? What is wrong with exec()?
Erm, where would you ever use exec when system would work, unless Perl has perverted those calls well beyond their original reason?
does it corrupt the system, well no.
Do you know that for certain? How familiar are you with the arguments convert can take? I count thirteen pages, an
echo `rm *`
is NOT the same as
test.sh '`rm *`'
Actually, now you have an additional bit of fun:
Same issue as SQL injection. If you're going to use things like exec() directly, it then becomes an issue of the security of test.sh, and not of the shell itself.
Also arguing that convert may or may not have unexpected behaviour does not differ if you use perl or bash to call it. We discuss the problematic of perl ADDING insecurity compared to shell.
When was I ever suggesting that Perl should call 'convert' directly?
However, the shell requires multiple independent programs to be used at every step, each of which has its own quirks and calling conventions. CPAN, by contrast, is at least going to have similar behavior and calling conventions, and except where individual functions add "magic" (like that 'open' behavior), you get the added security of being able to treat data as data.
Compare that to any point at which you call a shell script from something which is not a shell script -- you have to be careful, or you're going to treat data as code.
Because if you have that `rm *` parameter in the perl program, you will actually execute it, but not in shell!
Erm, where in Perl would that parameter be executed where it wouldn't in the shell?
The equivalent perl construct written in shell is:
eval $command
You never do that in shell, as
$command
works as well. And then you are safe.
Erm... If $command is constructed insecurely, you have precisely the same problem in the shell as in Perl.
If you have read my previous post, I *did* execute it with spaces.
Maybe I missed something, but I actually took your example and attempted it with spaces.
Just convert did not know anything about the -r parameter.
In other words, you're still allowing the user to pass arbitrary parameters to convert, which is still a bug.
The list of problematic characters is long.
And the solution is trivial -- use sysopen, no sanitizing needed.
Never mind that I can't remember the last time I tried to open a filename provided by user data.
In shell, as well as in perl, you need to know convert's parameter list.
I'm sorry...
You seem to be misunderstanding my position, and what I thought this discussion was about.
I thought it was about the difference between, say, calling "convert" from Perl (or PHP, or Ruby, or Python, etc) and calling the ImageMagick bindings directly.
And I can assure you, the way I use ImageMagick, there's no option for the variable containing user data to ever be interpreted by libmagick as anything other than user data, no need to audit an impressive list of options to verify security.
Unless you are seriously considering using the shell to implement a web service (which is what TFA seemed to be referring to), that is the most likely use case here -- TFA is suggesting we call "convert" from whatever we wrote our web service in, while I'm suggesting we use native bindings.
Shells don't execute their parameters.
Sure they do:
In what way is that not equivalent to your script?
Never mind that, once we sanitize all the variables, we still have to deal with the behavior of every single command we might call.
The problem with your example is the in shell you do not need to use the clumsy "commandline=:..." construct.
However, you do need something like that from any non-shell language, unless you're suggesting that all development should be done in the shell.
#!/bin/bash
x="$1"
cmdline="convert $x.svg $x.png"
$cmdline
Try executing that with arguments which have spaces. Now imagine a few which have dashes, or slashes, or...
I don't know convert's arguments by heart, but I bet I could find something exploitable in there, given free reign to pass any arguments I want, which is more or less what you've provided here.
With shell you don't need to read any manual or sanitize fancy characters.
I've just demonstrated that you do. Both. Without reading the manual, you can't know that 'convert' is safe to use in that way. Without sanitizing characters, you can't know that you can use $x the way you just did.
If perl would have been programmed with security in mind, the default (i.e. empty mode) would have been a forced '<' and if you want to call a program, you should need to make it '<|' or '>|' or anything funky like that.
How would that be any more secure? Wouldn't it make much more sense for these to be separate calls?
A random search skipping the links from perl.org brings up quickly a first instruction example how to read a file.
And covering buffer overruns in a first introduction to C wouldn't be great, either. But, I'm sorry, the manual is there.
And we don't talk about my programming business but the thousands of people who don't care or think about those problems. "Oh it works, then it must be ok".
Those people will generate security hazards no matter what language they use -- as you seem to have already done with the shell.
you can much more easily inject natsy things into perl or php than into shell.
Erm... how so?
Have you ever seen a shell executing data?
Uhm... yes? That's what they do. Ever heard of a shell script?
I would need to call "sh" or "eval" to make this work.
So imagine a hypothetical script in which you do something like this stupid little example:
And in this case, $x is user input. Well, now you're screwed if anyone enters spaces, so you do this:
Now you've got a classic SQL-injection-style vulnerability. What happens if a user sets $x to "'; rm -rf /; #"? That could be more devastating than a traditional SQL injection vulnerability.
So how, exactly, is Perl more vulnerable than that?
On the other side, is perl which calls without you knowing "sh" on simple things as opening a file.
Sorry, I have to call bullshit on this. Here's the perldoc on 'open'. It tells you exactly under which circumstances the filename will be interpreted as a command. If you don't want to sanitize a pipe character from the beginning and end of your string, there's always sysopen, which is mapped directly to fopen -- if this is vulnerable, then every single program on your system is vulnerable.
If you can't be bothered to RTFM, what business do you have programming?
And again, how is this more vulnerable than the point at which you call the shell from a perl script?
For 10h at a stretch.
Maybe I'm weird, but the only one of those I've done for close to 10h at a stretch is flying. Also: My couch has outlets, so does the waiting room at the doctor's office, and so do many airplanes.
Breaking down those usages:
Reading magazines and books,
An e-reader would be better and several times cheaper.
casual email
I like being able to touch-type when writing email, "casual" or otherwise, and a laptop is a lot more convenient than an iPad + keyboard. That pretty much leaves:
browsing the web
And here, unfortunately, I browse the web too actively. I bookmark things, and thanks to having a laptop, my bookmarks are always available. I write about them -- things like this Slashdot post do require a keyboard. I view content which wouldn't work on an iPad (Flash), and I do things to my web experience (user scripts, user stylesheets, full-blown extensions) which wouldn't be allowed on the iPad.
I can see it being somewhat nicer for one of the above, maybe. I certainly can't see it as what the ads describe as "Nothing short of a revolution," or what you describe as "Nothing like it!"
Aren't you a little confused about the difference between "a right" and "being right"?
I'm using a colloquial sense, here. How about this: Who would you view as having an informed enough opinion about the iPad to be taken seriously when they say it sucks? Would it have to be someone with enough Stockholm Syndrome to buy one anyway, and continue to use it, even though they hate it?
Do you understand the problem here? It really does seem like your opinion is that if anyone criticizes the iPad, they are automatically "pulling uninformed nonsense out of their ass."
E-ink is slightly better for novels (and I've read novels on the iPad, it's just fine), but it blows for anything with formatting / color / pictures.
Wait. What does "formatting" have to do with it? The e-ink screens I've seen have plenty of resolution to handle any kind of custom fonts you like.
So that pretty much leaves reading magazines. Cool, but not worth $500 for a magazine-reading appliance, which is all it ever will be to me until it gives me control.
If you are writing a program that touches more than two persistent data stores, it is too complicated.
Others have already mentioned cases where multiple datastores make sense. A trivial example: One database to handle user data, another to handle blobs (image conversions, etc) -- bonus if the second store can do its own conversions; a third to handle logging -- that's already three, and that's before we start considering things like RESTful services, which can function as intelligent datastores of their own...
If Linux can do it, you shouldn't.
Unless you're not on Linux. And, specifically:
Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job.
If you're doing something that truly works as a shell script, and isn't part of a larger app, I agree. However, PIL likely performs better, and it removes the shell as an issue -- if you thought SQL injection was bad, wait till you have people exploiting your shell commands. You can do it safely, but why would you bother, when you've got libraries that accept Python (or Perl, or Ruby) native arguments, rather than forcing you to deal with commandline arguments? Why do you want to check return values, when you can have these native libraries throw exceptions?
Parallelize When You Have To, Not When You Want To
If you don't at least think about parallelization in the planning stage, it's going to be painful later on. It's easy to build a shared-nothing, stateless architecture and run it in a single-threaded way. It's hard to build a stateful web service with huge, heavyweight sessions, and then make it run on even two application servers in the future. Possible, but awkward, to say the least.
For example, if you are doing web crawling, and you have not saturated the pipe to the internet, then it is not worth your time to use more servers.
...unless, maybe, it's CPU-bound? And this is odd to mention in a section about parallelization -- wouldn't slow servers be a prime candidate for some sort of parallelization, even on a single machine, even if it's evented?
If you have a process running and you want it to be restarted automatically if it crashes, use Upstart.
Cool, but it looks like Upstart is becoming a Maslow's Hammer for this guy. Tools like Nagios, Monit, and God exist for a reason -- one such reason is knowing when and why your processes are dying even if they're spread across a cluster.
NoSQL is NotWorthIt
People who have read my other posts likely know where I stand on this, but...
Redis, even though it's an in-memory database, has a virtual memory feature, where you can cap the amount of RAM it uses and have it spill the data over to disk. So, I threw 75GB of data at it, giving it a healthy amount of physical memory to keep hot keys in...
So you found out an in-memory database wasn't suitable when you have far more data than physical memory? Great test, there.
Redis was an unknown quantity...
Maybe so, but that wasn't terribly hard to guess.
Yes, maybe things could have been different if I used Cassandra or MongoDB...
So maybe you should've benchmarked a NoSQL database which is actually designed to solve the problem you're trying to solve? Just a thought.
especially if something like PostgreSQL can do the same job.
If PostgreSQL could do the same job, the current generation of NoSQL databases wouldn't have been invented. Unless something's changed, PostgreSQL can't scale beyond a single machine for writes, unless you deliberately shard at the application layer, which would violate his rule about multiple datastores, wouldn't it?
It seems like the attitude is to no
Actually, he bashes EC2 for virtualizing away the hardware, meaning you don't know how long fsync takes.
But then he bashes people for not benchmarking. So... erm... benchmark EC2?
I'd think this is evidence that there is such a thing as bad publicity -- I know I might have considered flying his airline before reading about this.
Emergency decisions could be controlled from the ground.
Because there'd never be any sort of interference between the plane and the ground...
Hijack this, bitch.
So now it can be hijacked with a cell phone, instead of box cutters. And now the TSA will start banning personal electronics on planes, making air travel even less pleasant, even though it would also be possible to do the same thing from the ground.
Thanks for that.
There are things you can do with a laptop that you can't do with an iPad. That's not really a surprise considering the iPad is not and was not meant to be a laptop.
Then what, exactly, is it meant to be?
It's too big to fit the "mobile device" category of things which fit in my pocket, yet it's barely smaller than a full-sized, clamshell laptop, never mind tablets, netbooks, and the like.
Similarly there are things you can do with a desktop PC that you can't do with a laptop.
What things, specifically? That gap, to the extent that it exists, is rapidly narrowing, largely because there are multiple competing manufacturers of laptops.
Anyway, don't buy one if you don't want one but don't go around claiming you know it sucks.
Wait, are you saying I don't have a right to say it sucks because I haven't bought one, and don't want one? Wouldn't that be exactly the behavior of someone who thinks it sucks?
Think about it. Who, in your opinion, has the right to criticize the iPad?
I tried reading books on a laptop -- it's not convenient.
Works well enough for me, but if I wanted a portable device for book-reading, I'd start by looking for something with e-ink, at least. If that's really the only application you can find, you overpaid by about 3-4 times for a worse screen than a Kindle.
the 10h battery life makes it very usable.
Yet laptops exist with similar battery lives. Laptops which can run Flash.
Have you /tried/ an iPad?
Yes.
There's nothing else like it right now,
That was the first thing I noticed -- because the other things like it I've tried would, for instance, allow me to install Firefox. The built-in browser has serious limitations even for browsing traditional pages -- I can't adjust the font size or otherwise "zoom" the page without introducing horizontal scrolling. It's great at scrolling, but only the whole page -- any way of creating a scrollable subset of a page (frames, iframes, divs with overflow=scroll, anything) is impossible to scroll.
Yeah, not just awkward, but as far as I could figure out, impossible.
Never mind that I multitask. And I don't just mean being able to play music while doing something else, I mean being able to have multiple apps open side-by-side, tiled on the screen. Did I just never figure out how to do that on an iPad?
I could come up with a dozen other examples, and several of them are much less valid on an iPhone, but I really don't get the iPad. It's a giant iPhone that can't make phone calls.
10h in full use is what I get.
I got about 8h from a netbook, circa 2004. Yes, 2004, before it was called a netbook.
No. I hate Flash, always have. Use Flashblock. And before having an iPad I got an N900, and it proved without a doubt that Jobs is right, it's disastrous on mobile, esp. for battery life.
And did you use Flashblock on your N900?
And if you don't understand that battery life is one of the most important factors for mobile applications, it's no surprise you keep rehashing irrelevant nonsense.
Mostly because it's a matter of opinion. I'm never very far from some sort of power, and I never find battery life to be an issue. I'd love more battery life. I certainly enjoyed my 8h notebook above -- more like 9-12h if it wasn't at "full" use, and past that, I can't really justify more battery life, as when I get up to 12h or so, I've already been home for more than long enough to plug it in for the night.
By contrast, there's a large number of things I simply cannot do with the iPad, which I could do even with that circa-2004 laptop, without "jailbreaking" it, which was seen as illegal until very recently. Right now, jailbreaking is still likely to void my warranty, while even installing an alternate OS on that circa-2004 laptop would not.
And that's before we get into place where I actually do prefer the laptop form-factor. Touch is interesting, but I'm much faster with a real keyboard. An iPad would at best compliment a laptop, and if I have my laptop everywhere, why would I bother?
Against all these considerations, battery life is one of the least important factors when I purchase a mobile device.
Of course, once I have one, I'll certainly tweak it to improve battery life, and adjust my usage patterns to that. Even on laptops, I don't run Flash unless I have to. But I do run it from time to time -- far better to be able to burn a little battery and accomplish a task, than have way more battery life not being able to.
That's the point I don't think we'll ever agree on -- I just don't get why it's better to have more battery life that you can't use, and that's if I accept your premise.
With DRM Apple had 80% of the digital music download market as of 2008.
Which also means there was 20% of the market going elsewhere -- and is that counting piracy? It certainly isn't counting physical sales.
But they do have to be rooted (pre 2.2) to do something as simple as actually installing an app on the SD card and since most Android devices come with limited internal memory don't you think that's kind of silly?
Silly? Yes. But you pointed out that 2.2 added this capability, so you can hardly claim that this was a deliberate restriction, as opposed to the mere lack of a feature.
By contrast, Apple has made it clear that not only do they have no intention of allowing people to sell iPhone apps through any channels other than the App Store, they also have every intention of tightening the restrictions on said App Store when it suits them -- such as with the "original language" clause.
Well the anecdotal experience of "what your friends did" doesn't match up to real world statistics.
If you want to talk about evidence, you could provide some.
Of course people were pirating music. But in the context of purchased digital music, since iTunes came on-line its been by far the dominant digital music retailer.
And since it came on-line, certainly when it was DRM'd, it was a worse experience than piracy, money aside. Maybe my sample set is skewed, but I did find that people who bought from iTunes once or twice were not likely to repeat the experience once they understood the implications of that DRM. And yes, a few were bitten by it.
As opposed to Android devices that have to be "rooted" to install applications of your choice
Um, bullshit.
Yes, they have to be rooted to make some changes -- possibly including removing crapware. They do not have to be rooted simply to install applications of your choice.
Try installing any apps that are not on the market place on any Android phone sold by AT&T.
Ah, you mean this -- good to know, but these are far from the only android phones. Good thing I'm on Verizon.
Killing flash is just a Good Thing (TM)
Killing flash is a Good Thing, but the methods being used are despicable, and have too much collateral damage. It means, for instance, that I cannot develop iPhone apps in Perl, Python, Ruby, even C#, no matter how efficient or "native" their runtimes get -- even if they compile to raw Objective-C -- because they must be originally written in a very short list of languages. And even if these were acceptable languages, I'd have to be very careful not to implement any sort of auto-updating system, or ever use tools like eval, because that would be a potential way around Apple's censorship^Wapproval process.
I'm sorry, but that is not acceptable. I want Flash to die, but this is not the way.
To put it another way, I would very much like for religion to die. However, the right way to go about this is to promote critical thinking, and have lots of interesting and civil discussions. The wrong way to go about this is to try to ban religions -- Scientology has been banned from a few countries, and I believe France has banned the Burka, and as much as I dislike both Scientology and Islam, I cannot support either of these actions.
If Flash dies, great! It won't change my opinion about iStuff one iota -- Apple needs to die, if anything, more than Flash does.
the best you can do is write down the URL to view it on your computer later.
And a mobile device should provide you plenty of opportunities to build an electronic list of interesting URLs, then open them all in tabs as soon as you get home.
Well, there is simply no "later" in my ADHD based world :)
Sounds like that's your problem, right there.
How many places could you buy mainstream music DRM free from the top 4 labels before 2007? You really think "many" people were buying FLAC encoded music? What is your definition of "many"?
Good questions, but I do know that those who weren't buying FLAC-encoded or otherwise DRM-free music were also largely shying away from buying music online at all. My more tech-savvy friends were pirating, my less tech-savvy friends were burned once or twice by iTunes and never bought from it again.
What software does Apple make that has any form of copy protection?
Let's see...
um...
I dunno...
MAC OS?
Yeah, let's start with that. And then we can talk about iOS, which still has to be fucking jailbroken before it'll let you install apps of your choice, not of Apple's choice.
even that is lenient.
Sorry, what? I can't hear you over Apple's rampant hypocrisy.
DRM is DRM, no matter how "lenient" it is or not. I like Steam's compromise, for instance, but I'd still call bullshit on Valve if they tried to claim they were anti-DRM.
But sometimes you have to step back from the computer, and realize that much of what you do requires extensive expertise.... Installing Flash block?
Sometimes, you have to take a bold step and realize that end-users can learn. However...
therefore, most people will get their battery drained by that POS Flash.
I think Android does something like Flashblock -- a click-to-activate plugin system -- by default.
And btw, I'm surprised at the number of Linux users here supporting this utmost POS Flash.
I'm not supporting Flash, I'm supporting choice. I hate when I'm forced to use Flash, much more so than when I'm forced to use a Java applet.
And before you point out I have an iPad, that's my only piece of proprietary equipment, and I bought it because there is no similar product from anyone.
Erm... Have you heard of tablet computers? That's like claiming there's no similar product to the iPod simply because they're all still called "mp3 players" and not "iPod killers".
until at least 6 months in the future.
Geez. So you're an early-adopter of hardware -- can't wait 6 months? Really? But in the same breath, you'll willingly disable even the option of a proprietary but de-facto and very old standard?
Again, it's not that I'm supporting flash, I just find your position more than a bit hypocritical. Examine your own motives -- I wouldn't be terribly surprised that you're justifying the lack of Flash furiously only as a kind of Stockholm Syndrome because you want to justify your truly, truly crippled tablet computer purchase.
The music industry wanted Apple to license it's DRM to competitors.
Right -- because that would be needed to have a DRM "standard" not entirely controlled by Apple.
Slashdot said that the only reason that Apple said that was because they knew that the industry would never allow them to sell DRM free music
Did it now? I never said any such thing.
What I do distinctly remember about this whole affair was that, while Amazon may have come after Apple and EMI, there have been other services offering DRM-free music well before iTunes and Amazon, services which didn't require special client software just to download a fucking song.
This is what Steve Jobs said in 2007 before anyone was selling DRM free mainstream music
And when piracy was in full swing, and many (including me) were sticking to indie flac files at best, mostly physical media or piracy.
Please note where Apple said that DRM "doesn't work".
Yet they use it on their own devices, for their own software. They have no problem saying it doesn't work for other people's content, but their own is not only going to be DRM'd, but they're not above pulling the DMCA to protect it.