RAID isn't really necessary for an application like that. RAID means when you change something on the drive, you update another drive simultaneously, in real time. For your bit torrent example it's enough to just use ordinary backups, which just means rsync'ing one drive to another every so often (nightly or whatever). That is fairly fast since it only copies the new or updated files, keeping the drives in sync. That may even help reliability, since the second drive isn't spinning except when you do a backup.
Use a hardware encrypted disk. They are becoming a lot more affordable and easier to find. You don't have to do a single thing to the software.
Visit newegg.com and type "fde" (for "full disk encryption") into the search form, and you will find a bunch.
These drives encrypt the disk in ECB mode, which leaves some kinds of data patterns detectable, but it's a huge improvement over leaving the disk unencrypted. It should stop the most scary types of personal information disclosure in this application, even though it leaks vaguer sorts of information that make it unsuitable for a general purpose cryptographic solution. There are some ways to compensate for this with special software, but you're trying to avoid using any of that.
It sounds like you're just trying to save some server resources, not run an ultra high security operation, and that you probably DO want to let legitimate users access the site even when they're travelling. The simplest thing, it sounds like, is to just ask people to enter their zip code when they register. Explain that it's a site for a certain locality and they shouldn't register if they're not from there. If they enter an out-of-area zip code, give them an error page explaining the same thing again. Don't tell them the "valid" zip codes. Yes they could look them up if they're motivated enough, but in that case you probably want to let them in anyway, i.e. maybe they used to live there and still have connections to the area, that sort of thing.
All the stuff about IP geolocation seems like a waste of resources in this context.
to these damn companies anyway? Facebook, Myspace, Livejournal and all the rest of them. The whole thing gives me the willies. Much better to get plain old web hosting and pay for it and control it yourself. Anyone remember Facebook's "Beacon" program? It's one insidious scheme after another. After this TOS stuff, it will be something else.
There is a very obvious anonymous payment system run by the US Treasury and its counterpart organizations in other countries. At registration time ask for the serial number of a one dollar bill, and require that the bill be sent by snail mail to confirm that it is real, and to help with site expenses. No names or return addresses are required and no spammer will go anywhere near that.
Oh I see what you mean. The following approach may be worth experimenting with, but no promises:
1) Configure one browser the way you like it, using the method described above.
2) Look in your.mozilla/firefox/(profile name) directory for a file called cert8.db
3) Push that file out to other desktops in your installation.
Edit -> Preferences and select the "advanced" (gear) icon. From there, select the "encryption" tab. Under that tab, click "view certificates" and then "authorities". Select any CA that you want to remove and click "delete".
and I think we know better than "no one will ever want more than 640K" or 4GB as the case may be. The latest trendy accessory is ultra expensive ($25/GB) Intel X25-E flash drives and a lot of the motivation for buying them is inadequate ram capacity in the host computer (since the flash disk costs more per GB than RAM which is 100x faster, though volatile).
Yeah a lot of people are still running 32 bit OS's, but almost all desktop hardware now being shipped is 64 bit-- we're in something like the tail end of the Windows 3.x era. I think most serious users will run 64 bit OS's pretty soon. The Mac Pro uses FB-DIMM and has 8 sockets (wish it had 16) and for a big class of data crunching tasks, what matters most is the amount of ram you can throw at it. The recent collapse in ram prices has been amazing. If enough sockets were available we could fit out $5000 boxes (think of a fully loaded Dell Precision or Mac Pro, not exactly a mass market consumer pc, but not a high end Sun server either) with 128GB or maybe even 256GB. That really extends the range of problems you can attack. But, the bottleneck even in server boards seems to always be ram sockets.
The Nehalem/i7 uses DDR3 which is a lot more expensive per GB compared to DDR2 and not available in as high capacity. It has more bandwidth but its latency (which matters more) is about the same. The usual desktop mobo is limited to 2 dram modules per channel. DDR2 boards usually have 2 channels (4 sockets max) while DDR3 boards have 3 channels (6 sockets). But 4GB DDR2 modules are around $100 (link) while DDR3 currently maxes out at 2GB. So you can populate a Phenom or Core 2 mobo with 16gb of ram for $400 but you can't put that much on a normal consumer i7 board for any amount of money. 2GB DDR2 parts are a lot cheaper still, you can put on 8gb (4x 2gb) at around $15/gb, $120 total. Right now a 2gb DDR3 part is $50-ish, 3x as expensive (link). It helps that you can put 6 of them on a board (12gb total, $300) but you have to take the cost difference per GB into account with 2GB parts, and comparing with 4GB DDR2 parts there is $/GB parity but lower total capacity (4x4gb vs 6x2gb). And of course when 4gb ddr3 does come out, it will bring a welcome increase to 24gb total capacity, but it will be WAY expensive for quite a while (the 4gb ddr2 modules that are $100 now were $500+ for most of this year).
I just don't understand why there aren't more consumer boards with a lot more sockets, using FB-DIMM or registered DDR. You have to go to server boards for that ($$$).
In this case I think it's the cops who are exploiting a weakness (that most cell phone users are identifiable unless they take special precautions), not that anything is wrong with cops using what they can under the circumstances. But, as a general matter, private communications are a GOOD thing. If we have a situation where a criminal wore gloves to avoid leaving fingerprints, we normally wouldn't say they exploited a weakness of the fingerprint system that needs to be plugged by outlawing gloves.
> Also I think the strysand effect may well be helping the sales skyrocket.
The version of the record with that picture has been out of print for many years. They switched it over to a picture of the (fully clothed) band. The record itself may well be out of print by now.
I think most would consider Haskell to be a purely functional language even though it has some types that are implemented with mutable data for efficiency under the surface. For example, the ST monad is implemented with a mutable cell, unlike the State monad which is purely functional. In reality they do the same thing, you could implement ST's semantics using State, it would just be slower. Same thing with STArray and so forth. I actually haven't used ST so far, but I'm still a relative Haskell newbie.
I have the impression that Ocaml is easier than Haskell for imperative programmers to get accustomed to, but I haven't tried it. Haskell certainly has its notorious learning curve. But, it seems to be the focus of quite a bit more interest and development activity than Ocaml these days.
1. In fact everyday programmers do prove things about their programs all the time. Not every last detail of the semantics (that is very difficult) but they rely very heavily on type safety, which is enforced either by a static type system (like in Java or Ada) or by automatic runtime checks (languages like Python). Languages like C, which have leaky type systems AND no runtime checks, are generally full of bugs that end up getting exploited by viruses. Almost every security bug we hear about, and the spam botnet that arises from the exploit, is caused by some type error (usually unchecked array overflow) in a C or C++ program. Functional languages (not exactly inherently, but more as a cultural matter among the designers) treat type systems as lightweight theorem proving schemes. So Haskell's type system is FAR more advanced than what you'll find in something like Java.
Let's say you have a few thousand (name, address) pairs and you want to be able to quickly look up a name to get the corresponding address, to add new names, etc. In imperative programming you'd probably use one of the mainstay data structures of CS 101, the good old hash table. To add a new name, you hash it and go and poke that address in the table to record the entry.
Well remember that stuff about values in functional programming being immutable? Right, no hash tables in functional programming. You'd instead use something like an AVL tree or red-black tree, that let you create a completely new structure that shares most of its content with the old one, except that the new one has this extra node. Of course FP language libraries come with modules for making those structures, and in practice you can use them at the API level sort like how you used to use hash tables, but they are completely different underneath, and if you want to program them yourself you are going to have to learn a lot of very basic techniques from scratch all over again. Chris Okasaki's book "Purely Functional Data Structures" is a good place to learn about this stuff in detail.
Even more basic: the good old "for" loop, which updates an index variable each time through. Whoops! You can't update the index in a functional language, so there's no "for" loop. You instead use recursion, or a "higher order function" (function that operates on other functions). So instead of
for (i = 0; i < n; i++) xs[i] = f(ys[i])
You'd write something like
ys = map f xs
("map" takes a function f and a list of values xs, applies the function to each item in the list, and gives you back a new list). There is also a "list comprehension" syntax that you might know from Python:
ys = [f(x) | x <- xs]
but for complicated functions you end up having to use higher order functions and recursion explicitly. You really have to think a lot harder to program 20 lines of Haskell than 20 lines of C. But those 20 lines can do an order of magnitude more.
(Aside:) In case you were wondering, yes, you can implement traditional hash tables and other mutable structures in functional languages, and there are times when it's necessary, but it's comparatively a pain in the ass and you give up some of the advantages that had you programming functionally in the first place. Here is an article about someone's experiences switching from a mutable structure to a functional structure in a large program, and the headaches the functional structure solved:
IO in Haskell is done with values called "actions" which can be viewed as state transformers on the real world (in fact their datatype is a function taking a "RealWorld" value and returning another one). So the value (print "hello world") is a function into which you (figuratively) enter a state of the world in which you're looking at a blank screen, and the function gives back a state where your screen says "hello world".
So what happens if that function tries to do something sneaky, like save a copy of the input state and let you re-use it to "go back" in time? The answer is that the data types surrounding I/O actions are organized in a way that prevent you from getting access to that "value", sort of like private instance variables in Java. Monads are simply a class of data types that make it convenient to express that kind of abstraction, and also to write what looks and acts like imperative code when you want to, even though it's still functional under the skin. People get confused by thinking monads somehow implement the I/O. Rather, they just describe datatypes used by the runtime system, that implement those I/O state transformer actions.
Since as long as you're running on the same hardware, you use the same instruction set regardless of what higher level language you program in, yeah, there will probably always be optimizations available in assembler that aren't available in HLL's. The question is how much extra work are you willing to do to get at them. Functional programming languages (FPL's) like Haskell are simply higher level languages than imperative languages like C or Java.
Here is a Haskell program to print the sum of the squares of the first million integers, using multicore parallelism:
print $ psum [| x*x | x http://www.cse.unsw.edu.au/~dons/blog/2007/11/29
Again, yeah, you could do that with Posix threads but it would be hellish by comparison. And we haven't even talked about the STM (Software Transactional Memory) library, which lets you share data between threads without messing around with locks (the compiler's type system automatically makes sure the sharing is thread safe).
Check out the book http://book.realworldhaskell.org/ (text is online) for more info about this. Note it doesn't include the DPH stuff which is very new.
Python has some features inspired by FP languages including Haskell, but is not anything like real functional programming. Haskell is far more powerful and serious, but also a heck of a lot more difficult to use. Python has a "practicality beats purity" mantra; you could think of Haskell as "purity strikes back".
Stuff Haskell gets you:
Serious native-code compiler whose output can beat compiled C code for some programs (and does darn well on average, see the Alioth shootout)
Ability to use multi-core parallelism, with a library module that treats shared memory as a transactional database, allowing use of shared data between threads while getting rid of all the lock management headaches of languages like Java. This can work because Haskell's functional purity guarantees that the threads won't clobber each other except under circumstances precisely controlled by the library.
Data parallelism allowing computations of list comprehensions to automatically be done in parallel on multiple CPU's
Rigorous type system (nothing like the broken ones like in C or Java that you might be used to) lets you express complex invariants in your datatypes so that errors can be caught by the compiler. This greatly decreases the amount of runtime debugging required.
Haskell has a very steep learning curve compared with other languages (or "unlearning curve", as some put it, since you have to forget everything you knew), but learning it (a still ongoing process for me) is one of the most interesting and mind-expanding things I've ever done as a programmer.
which is written as if it were just like any other compute task, to be scheduled as the OS sees fit (maybe with bumped up priority, but that's not enough to make sure the right thing happens). It instead has to be scheduled as a realtime task with guaranteed bounds on the response time for low-overhead user operations, which means locking stuff in RAM even at the cost of more swapping on less interactive tasks. That also means turning on realtime kernel support in systems that don't already have it active.
I've thought for a while that the Linux window system should be rewritten by game programmers, who tend to have some clue about how to identify the parts of an interactive program have to be responsive, and to make those parts actually be responsive.
PC-6400 ram is around 15 dollars a GB now, and the 6400 stands for MB/sec, i.e. ram is over 20x faster than this flash drive and has no write wear issues or slowness of random writing. The only thing wrong with it is volatility, but in an enterprise environment you can use a UPS and/or maintain a serial update log on a fast hard disk or RAID (since the log is serial, the flash drive's ultrafast seek advantage doesn't apply). There is just a narrow window where this $21/gb 32gb flash drive is really cost effective.
Google disclosure an "option" since Mozilla's revenue comes from Google payments for the traffic. They are not going to shut that source of revenue down.
I think the answer for Ubuntu is to reject the EULA and compile a browser that makes the Google disclosure opt-in, and is called something other than Firefox.
Coca-Cola is one of the most heavily enforced trademarks in the world. That doesn't mean I have to accept a EULA every time I open a can of it. I don't see why Firefox is different from that.
Slashdot Mirror
RAID isn't really necessary for an application like that. RAID means when you change something on the drive, you update another drive simultaneously, in real time. For your bit torrent example it's enough to just use ordinary backups, which just means rsync'ing one drive to another every so often (nightly or whatever). That is fairly fast since it only copies the new or updated files, keeping the drives in sync. That may even help reliability, since the second drive isn't spinning except when you do a backup.
Use a hardware encrypted disk. They are becoming a lot more affordable and easier to find. You don't have to do a single thing to the software. Visit newegg.com and type "fde" (for "full disk encryption") into the search form, and you will find a bunch. These drives encrypt the disk in ECB mode, which leaves some kinds of data patterns detectable, but it's a huge improvement over leaving the disk unencrypted. It should stop the most scary types of personal information disclosure in this application, even though it leaks vaguer sorts of information that make it unsuitable for a general purpose cryptographic solution. There are some ways to compensate for this with special software, but you're trying to avoid using any of that.
Take a look at decapod-project.org for a complete system. Note that software dewarping is quite a hard problem, but it is part of decapod.
Modern web frameworks like Cobol on Cogs already do burned-in CRT emulation and various other effects too. ;-).
Er, what do you think Sony's "core business" is? Hint: it's not TV sets.
It sounds like you're just trying to save some server resources, not run an ultra high security operation, and that you probably DO want to let legitimate users access the site even when they're travelling. The simplest thing, it sounds like, is to just ask people to enter their zip code when they register. Explain that it's a site for a certain locality and they shouldn't register if they're not from there. If they enter an out-of-area zip code, give them an error page explaining the same thing again. Don't tell them the "valid" zip codes. Yes they could look them up if they're motivated enough, but in that case you probably want to let them in anyway, i.e. maybe they used to live there and still have connections to the area, that sort of thing.
All the stuff about IP geolocation seems like a waste of resources in this context.
to these damn companies anyway? Facebook, Myspace, Livejournal and all the rest of them. The whole thing gives me the willies. Much better to get plain old web hosting and pay for it and control it yourself. Anyone remember Facebook's "Beacon" program? It's one insidious scheme after another. After this TOS stuff, it will be something else.
There is a very obvious anonymous payment system run by the US Treasury and its counterpart organizations in other countries. At registration time ask for the serial number of a one dollar bill, and require that the bill be sent by snail mail to confirm that it is real, and to help with site expenses. No names or return addresses are required and no spammer will go anywhere near that.
Oh I see what you mean. The following approach may be worth experimenting with, but no promises: 1) Configure one browser the way you like it, using the method described above. 2) Look in your .mozilla/firefox/(profile name) directory for a file called cert8.db
3) Push that file out to other desktops in your installation.
In firefox 3.0.5:
Edit -> Preferences and select the "advanced" (gear) icon. From there, select the "encryption" tab. Under that tab, click "view certificates" and then "authorities". Select any CA that you want to remove and click "delete".
Yeah a lot of people are still running 32 bit OS's, but almost all desktop hardware now being shipped is 64 bit-- we're in something like the tail end of the Windows 3.x era. I think most serious users will run 64 bit OS's pretty soon. The Mac Pro uses FB-DIMM and has 8 sockets (wish it had 16) and for a big class of data crunching tasks, what matters most is the amount of ram you can throw at it. The recent collapse in ram prices has been amazing. If enough sockets were available we could fit out $5000 boxes (think of a fully loaded Dell Precision or Mac Pro, not exactly a mass market consumer pc, but not a high end Sun server either) with 128GB or maybe even 256GB. That really extends the range of problems you can attack. But, the bottleneck even in server boards seems to always be ram sockets.
I just don't understand why there aren't more consumer boards with a lot more sockets, using FB-DIMM or registered DDR. You have to go to server boards for that ($$$).
In this case I think it's the cops who are exploiting a weakness (that most cell phone users are identifiable unless they take special precautions), not that anything is wrong with cops using what they can under the circumstances. But, as a general matter, private communications are a GOOD thing. If we have a situation where a criminal wore gloves to avoid leaving fingerprints, we normally wouldn't say they exploited a weakness of the fingerprint system that needs to be plugged by outlawing gloves.
> Also I think the strysand effect may well be helping the sales skyrocket.
The version of the record with that picture has been out of print for many years. They switched it over to a picture of the (fully clothed) band. The record itself may well be out of print by now.
I think most would consider Haskell to be a purely functional language even though it has some types that are implemented with mutable data for efficiency under the surface. For example, the ST monad is implemented with a mutable cell, unlike the State monad which is purely functional. In reality they do the same thing, you could implement ST's semantics using State, it would just be slower. Same thing with STArray and so forth. I actually haven't used ST so far, but I'm still a relative Haskell newbie. I have the impression that Ocaml is easier than Haskell for imperative programmers to get accustomed to, but I haven't tried it. Haskell certainly has its notorious learning curve. But, it seems to be the focus of quite a bit more interest and development activity than Ocaml these days.
1. In fact everyday programmers do prove things about their programs all the time. Not every last detail of the semantics (that is very difficult) but they rely very heavily on type safety, which is enforced either by a static type system (like in Java or Ada) or by automatic runtime checks (languages like Python). Languages like C, which have leaky type systems AND no runtime checks, are generally full of bugs that end up getting exploited by viruses. Almost every security bug we hear about, and the spam botnet that arises from the exploit, is caused by some type error (usually unchecked array overflow) in a C or C++ program. Functional languages (not exactly inherently, but more as a cultural matter among the designers) treat type systems as lightweight theorem proving schemes. So Haskell's type system is FAR more advanced than what you'll find in something like Java.
Let's say you have a few thousand (name, address) pairs and you want to be able to quickly look up a name to get the corresponding address, to add new names, etc. In imperative programming you'd probably use one of the mainstay data structures of CS 101, the good old hash table. To add a new name, you hash it and go and poke that address in the table to record the entry.
Well remember that stuff about values in functional programming being immutable? Right, no hash tables in functional programming. You'd instead use something like an AVL tree or red-black tree, that let you create a completely new structure that shares most of its content with the old one, except that the new one has this extra node. Of course FP language libraries come with modules for making those structures, and in practice you can use them at the API level sort like how you used to use hash tables, but they are completely different underneath, and if you want to program them yourself you are going to have to learn a lot of very basic techniques from scratch all over again. Chris Okasaki's book "Purely Functional Data Structures" is a good place to learn about this stuff in detail.
Even more basic: the good old "for" loop, which updates an index variable each time through. Whoops! You can't update the index in a functional language, so there's no "for" loop. You instead use recursion, or a "higher order function" (function that operates on other functions). So instead of
for (i = 0; i < n; i++) xs[i] = f(ys[i])
You'd write something like
ys = map f xs
("map" takes a function f and a list of values xs, applies the function to each item in the list, and gives you back a new list). There is also a "list comprehension" syntax that you might know from Python:
ys = [f(x) | x <- xs]
but for complicated functions you end up having to use higher order functions and recursion explicitly. You really have to think a lot harder to program 20 lines of Haskell than 20 lines of C. But those 20 lines can do an order of magnitude more.
(Aside:) In case you were wondering, yes, you can implement traditional hash tables and other mutable structures in functional languages, and there are times when it's necessary, but it's comparatively a pain in the ass and you give up some of the advantages that had you programming functionally in the first place. Here is an article about someone's experiences switching from a mutable structure to a functional structure in a large program, and the headaches the functional structure solved:
http://www.cs.tufts.edu/~nr/pubs/zipcfg-abstract.html
IO in Haskell is done with values called "actions" which can be viewed as state transformers on the real world (in fact their datatype is a function taking a "RealWorld" value and returning another one). So the value (print "hello world") is a function into which you (figuratively) enter a state of the world in which you're looking at a blank screen, and the function gives back a state where your screen says "hello world".
So what happens if that function tries to do something sneaky, like save a copy of the input state and let you re-use it to "go back" in time? The answer is that the data types surrounding I/O actions are organized in a way that prevent you from getting access to that "value", sort of like private instance variables in Java. Monads are simply a class of data types that make it convenient to express that kind of abstraction, and also to write what looks and acts like imperative code when you want to, even though it's still functional under the skin. People get confused by thinking monads somehow implement the I/O. Rather, they just describe datatypes used by the runtime system, that implement those I/O state transformer actions.
Since as long as you're running on the same hardware, you use the same instruction set regardless of what higher level language you program in, yeah, there will probably always be optimizations available in assembler that aren't available in HLL's. The question is how much extra work are you willing to do to get at them. Functional programming languages (FPL's) like Haskell are simply higher level languages than imperative languages like C or Java. Here is a Haskell program to print the sum of the squares of the first million integers, using multicore parallelism: print $ psum [| x*x | x http://www.cse.unsw.edu.au/~dons/blog/2007/11/29 Again, yeah, you could do that with Posix threads but it would be hellish by comparison. And we haven't even talked about the STM (Software Transactional Memory) library, which lets you share data between threads without messing around with locks (the compiler's type system automatically makes sure the sharing is thread safe). Check out the book http://book.realworldhaskell.org/ (text is online) for more info about this. Note it doesn't include the DPH stuff which is very new.
Python has some features inspired by FP languages including Haskell, but is not anything like real functional programming. Haskell is far more powerful and serious, but also a heck of a lot more difficult to use. Python has a "practicality beats purity" mantra; you could think of Haskell as "purity strikes back".
Stuff Haskell gets you:
Serious native-code compiler whose output can beat compiled C code for some programs (and does darn well on average, see the Alioth shootout)
Ability to use multi-core parallelism, with a library module that treats shared memory as a transactional database, allowing use of shared data between threads while getting rid of all the lock management headaches of languages like Java. This can work because Haskell's functional purity guarantees that the threads won't clobber each other except under circumstances precisely controlled by the library.
Data parallelism allowing computations of list comprehensions to automatically be done in parallel on multiple CPU's
Rigorous type system (nothing like the broken ones like in C or Java that you might be used to) lets you express complex invariants in your datatypes so that errors can be caught by the compiler. This greatly decreases the amount of runtime debugging required.
I could go on, but you get the idea.
Good tutorial: http://learnyouahaskell.com/
More detailed book (full text online): http://book.realworldhaskell.org/
Haskell has a very steep learning curve compared with other languages (or "unlearning curve", as some put it, since you have to forget everything you knew), but learning it (a still ongoing process for me) is one of the most interesting and mind-expanding things I've ever done as a programmer.
I am runnning swappiness=0 right now, and some stuff got swapped out when I copied a few dozen GB of files from one disk to another.
which is written as if it were just like any other compute task, to be scheduled as the OS sees fit (maybe with bumped up priority, but that's not enough to make sure the right thing happens). It instead has to be scheduled as a realtime task with guaranteed bounds on the response time for low-overhead user operations, which means locking stuff in RAM even at the cost of more swapping on less interactive tasks. That also means turning on realtime kernel support in systems that don't already have it active. I've thought for a while that the Linux window system should be rewritten by game programmers, who tend to have some clue about how to identify the parts of an interactive program have to be responsive, and to make those parts actually be responsive.
PC-6400 ram is around 15 dollars a GB now, and the 6400 stands for MB/sec, i.e. ram is over 20x faster than this flash drive and has no write wear issues or slowness of random writing. The only thing wrong with it is volatility, but in an enterprise environment you can use a UPS and/or maintain a serial update log on a fast hard disk or RAID (since the log is serial, the flash drive's ultrafast seek advantage doesn't apply). There is just a narrow window where this $21/gb 32gb flash drive is really cost effective.
I think the answer for Ubuntu is to reject the EULA and compile a browser that makes the Google disclosure opt-in, and is called something other than Firefox.
Coca-Cola is one of the most heavily enforced trademarks in the world. That doesn't mean I have to accept a EULA every time I open a can of it. I don't see why Firefox is different from that.