Re:Encryption and compression make a lot of sense.
on
PKWare Zips to Growth
·
· Score: 1
That is a good idea, I wonder if any of the big guys use compression to get better performance. I think they just tell people to buy more RAM and do more caching instead.
One problem is that compression is always a lot slower than decompression so a database would want to store blocks uncompressed initially and then have some sort of background-idle process that compresses blocks that have not been written to in X amount of time. Depending on your applicaiton, 99% of database access is read-only so you could see a significant performance increase for large databases. Basically you get 2X or more speedup of your harddrives at the expensive of CPU. I imagine that most databases would compress by much better than normal data.
Depending on the application, Databases are IO limited, not CPU limited so trading some CPU speed for better IO is a good tradeoff.
And the database could keep two levels of cache in memory - one for compressed data and one for decompressed data. This would allow you to store a much larger cache in memory while eliminating decompressions of frequently used data.
Re:Encryption and compression make a lot of sense.
on
PKWare Zips to Growth
·
· Score: 2
I've not really had a problem with this. I compress blocks of 64k at a time so random access to large files is about this same or faster than normal. If an application had a long spell of jumping around in the file, reading only one byte at a time then performance would suffer - but it's likely to suffer worse without compression because there is a higher possibility of a disk seek, which is much slower than decompression.
While the block size might be configurable in the future, beyond 64k compression doesn't get much better and there is the additional memory overhead from having to store an entire block in memory before passing it to the application. But for your example, a divx movies is unlikely to compress much - if any, no matter what the block size it.
Re:Encryption and compression make a lot of sense.
on
PKWare Zips to Growth
·
· Score: 2
Yeah, As far as I know, there isn't anything else like this. There are some system-level things you can do if you install device drivers, but what user is going to put up with that for every other application or data file they receive?
Thinstall is all user-mode code and doesn't install any drivers. Everything needed for the additional functionality is built into the EXE. So it will run directly from CDROM/floppy on any Windows computer you walk up to. The overhead size it adds to the EXE is about 75k, which is much smaller than InstallShield would take, and I think smaller than WinZip's self-extracting EXEs.
Re:Encryption and compression make a lot of sense.
on
PKWare Zips to Growth
·
· Score: 5, Interesting
Since a Zip has to be decompressed anyway
While until just recently, this was true - now you can create a "ZIP" file that doesn't decompress. The idea is instead of decompressing the files to disk, a tiny user-mode OS is inserted between the application that needs to use the data and the compressed data. The new OS does transparent decompression/decryption and to the application it appears the files reside on the hard drive. The OS provides streaming decompression so only small blocks are decompressed at a time and the memory requirements are very low. Yes, the data is present in memory in unencrypted form at some point so it is possible to hack - but it provides a pretty good level of data security.
The cool thing is that the archive size is usually the same size as a ZIP, but it runs directly with no install and no decompression time. Usually applications load 2x faster in this state.
This is something I've spent the last year working on. Checkout here
I noticed the two interfaces, but neither one works for me. The old interface tells me to use the new interface, and the new interface tells me I don't have any domains!
I registered several of domains through joker.com and now their website doesn't show that I own them. Yesterday it showed I owned them, but then I'd go to change one and it would say I had none. I guess they are working on the interface - but at least post a notice on the website! I mailed support and haven't heard back... so think twice about joker.com. Still, I'm never going to use NetSol again unless I'm forced to... talk about horrible service, horrible interface, and scummy spam-mail campains.
This is acutally the "Press DOS attack." You get some security expert to claim that a worm is spreading all over the internet and will attack X site at 5pm. Then everyone who reads the story will go see if the site is down at 5pm. And of course since everyone is hitting reload to see when it is down, the site gets flooded and goes down while the virus/worm never exsisted!
I was just thinking that one authorization code would work for all VMWare Virtual machines, but that isn't necessarily the case. I don't know anything about VMWare's code internals but something are likely to be different:
- Volume serial number (based on creation date)
- Processor serial number (if enabled, VMWare probably allows these instructions to run natively)
- Amount of RAM (possibly - but is not very unique)
The rest of the IDs should look the same because they are emulated hardware devices.
But, since this is 3 things, you couldn't necessarily use someone else's code (unless you run the cracked copy).
What about the opposite, wood to bits?
on
Books on Demand
·
· Score: 2
Does anyone know of a machine that will do the exact opposite? If you are like me, then you have way too many bookshelves filled with technical reference books, many of them 10 years old. But you can't bare to throw away that ancient PC Interrupts book least someday you have an arcane question about the PC BOIS call while working on a Virtual PC emulator. I agree that reading paper copies is preferable most of the time, but books that I'm going to access once every 5 years would be better off as digital. These books take of a ton of space and space is expensive in San Francisco.
I want a machine that can remove the bindings from books and scan in both sides of the pages as a big stack of papers. Then I'm left with a digital copy, which fits into one millionth of the space, and some a lot of firewood. Anyone have a suggestion on the best way to remove book bindings? Are there service companies out there that will do this? It's not going to be worthwhile if it takes a lot of time or money per book, but I'd be willing to pay a dollar or two.
Do you have any recommendation on where to look for dot-com auctions? Local newspaper? Is there an online site for such listings (particularly in the SF/Bay Area).
Good idea. This is how I "encrypt" my email address to hide it from spam robots when posting on the web. To bad GIF and JPEG can't do 2d-block compression or the size could be kept pretty small.
However without that the size would be very large and everyone would hate you for filling up their mailbox/harddrive.
To save them the space, you could run an external server that keeps your message in text form (it would be encrypted in transit or on a LAN) and passes them on to an external GIF image. When their mail client request the GIF, the external server generates the corresponding GIF for them. This way you save them space, and you can do evil things like track how often and when they read the message. Also you can prevent them from copying it to some degree because they aren't likely to retype the message if they can't cut and past. You can tell if they forwarded the message to someone else by looking at IP logs for the image access. So maybe Bill Gates really will send the 500th person to disney land! haha.
Any decent 128bit encryption would require a 1000 pentiums to work for... oh say.. the lifetime of the planet earth (at which point the power would fail). If it only takes 10 years then that smacks of weakened security mandated by the government.
From the article it almost sounds like they are using frequency hopping with a 128bit random function. I'm not sure but that might be legal in more countries because the data is not actually encrypted or decrypted - just the frequency path followed.
The hard part, identifing the runner's and their time was done with ankle radio trackers - I'm assuming. So really all this did was given a list of times and 5 hours of frames, mpeg encode 180 frames per runner. Doesn't seem like that difficult of a task that 45 machines would be needed. If they used re-used information then the encoding could probably be done in less then 12hrs on 1 server... but it sounds like it was just a brute force shell script that brute forced the videos as seperate operations. Whatever works!
Oh well, I guess the enemy of my enemy is my friend. That one always works. Just look how it's helped the U.S. Gov't in the last half a century.
Sounds like you are trying to be sarcastic here. But you probably wouldn't be here if the Allies didn't form against a common enemy. About the only thing we had in common with Russia is a hatred of Hitler and Germans. Without Russia, it is very questionable if we would have won the war. Not tring to imply that carries over to AMD and Transmeta though. And yes, that is slightly more than a half century ago.
It's not "unplugged from the Internet", it's "unplugged". As in unpowered.
Even then, you are not totally safe. The contents of your RAM are often valid for several seconds to several minutes after you power off. With lower temperatures this can be up to hours. This must be taken into account with high security applications where physical access is possible. For example, tamper-detection circuits must erase RAM as well as EEPROMs when intrusion is detected.
Since PCs don't clear out memory before they power off, your passwords and encryption keys could possibly be stolen from RAM even with the best security precautions taken. Mind you, I haven't heard of anyone actually using this technique, just that it's a possibility.
executables have stripped relocation tables, you can only load them to address 0x400000
That's not true. Most exe's have relocation information. You can strip it out, but it's pretty rare that EXEs have this information removed. Even if they did it just means the process that loads them needs to be loaded at a different base address. There is nothing special about 0x400000, many exes have a base address other than that. Notepad.exe uses 0x1000000 for example (under w2k). Use the program "rebase" that comes with W2k to change the base address to whatever you like. You can load an EXE without relocation information as DLL. Also DLLs don't have to have relocation information assuming they are not going to conflict.
even if the relocation table would be present, the OS doesn't permit loading & dynamically linking against an executable. It was originally possible, but the changed because it can cause security problem.
Changing variable names won't substantially change the binary code the code compiles into.
Changing variable names won't change the binary code the code compiles into by a single bit. Compilers don't care what variables are named as long as you follow proper syntax.
So what's the difference between linking with a DLL and forking a processes that is GPLed - you can achieve the same levels of functionality. In Windows, the DLL and EXE file format are exactly the same. So renaming a.EXE to.DLL would be a violation of GPL? Taking this argument to it's logical extreme, any program that can create a pipe to another process could be in violation of GPL.
I was kind of surprised to see the climbers using this machine didn't have on a saftey line - so they must trust it pretty well.
If you realize you are about to run out of air pressure then you get out your mountain climbing equipement and do it the old fashion way. I.e. hammer a bolt into the surface you are climbing and hook a line into your harness. If you are climbing a building, it *might* be hard to hammer into steel - but there is almost always some thing to grab onto in bulding structures - window ledges, etc.
Don't forget that the 1/3rd paid for legal fees is tax-deductible. That means you could recover an a total of 10% more through legal means. Probably not worth it for other matters:
- Legal procedures could take months - they might go bankrupt or shut down before you can collect.
- They aren't likely to keep you around as a client if you sue them. If you don't have any better options out there then you might need to keep collecting you 50% to stay in business.
Re:Everybody's not above average!
on
IT Unions?
·
· Score: 1
And before US States started using standardized testing, all States reported their students as "Above Average."
Lisp programs are not programs in the interpreter. They are data. You can call (eval) on the data, but it's still just data that conforms to a standard. That is what makes lisp cool is that data=code. You can do the same with XML - that is create a stand where XML data can be evalulated. For example :
In lisp this would be :
(print (+ 4 5))
After the parser has gone through both cases the data structures are basically the same. LISP can be used as a programming language and compiled to executable format, but it can also be used as a data-definition language. A lisp parser doesn't need the LISP execution environment - i.e. garbage collection and predefined functions. But, by allowing predefined functions such as defclass you can do everything you can do in XML in LISP.
Slashdot Mirror
That is a good idea, I wonder if any of the big guys use compression to get better performance. I think they just tell people to buy more RAM and do more caching instead.
One problem is that compression is always a lot slower than decompression so a database would want to store blocks uncompressed initially and then have some sort of background-idle process that compresses blocks that have not been written to in X amount of time. Depending on your applicaiton, 99% of database access is read-only so you could see a significant performance increase for large databases. Basically you get 2X or more speedup of your harddrives at the expensive of CPU. I imagine that most databases would compress by much better than normal data.
Depending on the application, Databases are IO limited, not CPU limited so trading some CPU speed for better IO is a good tradeoff.
And the database could keep two levels of cache in memory - one for compressed data and one for decompressed data. This would allow you to store a much larger cache in memory while eliminating decompressions of frequently used data.
I've not really had a problem with this. I compress blocks of 64k at a time so random access to large files is about this same or faster than normal. If an application had a long spell of jumping around in the file, reading only one byte at a time then performance would suffer - but it's likely to suffer worse without compression because there is a higher possibility of a disk seek, which is much slower than decompression.
While the block size might be configurable in the future, beyond 64k compression doesn't get much better and there is the additional memory overhead from having to store an entire block in memory before passing it to the application. But for your example, a divx movies is unlikely to compress much - if any, no matter what the block size it.
Yeah, As far as I know, there isn't anything else like this. There are some system-level things you can do if you install device drivers, but what user is going to put up with that for every other application or data file they receive?
Thinstall is all user-mode code and doesn't install any drivers. Everything needed for the additional functionality is built into the EXE. So it will run directly from CDROM/floppy on any Windows computer you walk up to. The overhead size it adds to the EXE is about 75k, which is much smaller than InstallShield would take, and I think smaller than WinZip's self-extracting EXEs.
Since a Zip has to be decompressed anyway
While until just recently, this was true - now you can create a "ZIP" file that doesn't decompress. The idea is instead of decompressing the files to disk, a tiny user-mode OS is inserted between the application that needs to use the data and the compressed data. The new OS does transparent decompression/decryption and to the application it appears the files reside on the hard drive. The OS provides streaming decompression so only small blocks are decompressed at a time and the memory requirements are very low. Yes, the data is present in memory in unencrypted form at some point so it is possible to hack - but it provides a pretty good level of data security.
The cool thing is that the archive size is usually the same size as a ZIP, but it runs directly with no install and no decompression time. Usually applications load 2x faster in this state.
This is something I've spent the last year working on. Checkout here
I noticed the two interfaces, but neither one works for me. The old interface tells me to use the new interface, and the new interface tells me I don't have any domains!
I registered several of domains through joker.com and now their website doesn't show that I own them. Yesterday it showed I owned them, but then I'd go to change one and it would say I had none. I guess they are working on the interface - but at least post a notice on the website! I mailed support and haven't heard back... so think twice about joker.com. Still, I'm never going to use NetSol again unless I'm forced to... talk about horrible service, horrible interface, and scummy spam-mail campains.
This is acutally the "Press DOS attack." You get some security expert to claim that a worm is spreading all over the internet and will attack X site at 5pm. Then everyone who reads the story will go see if the site is down at 5pm. And of course since everyone is hitting reload to see when it is down, the site gets flooded and goes down while the virus/worm never exsisted!
I was just thinking that one authorization code would work for all VMWare Virtual machines, but that isn't necessarily the case. I don't know anything about VMWare's code internals but something are likely to be different:
- Volume serial number (based on creation date)
- Processor serial number (if enabled, VMWare probably allows these instructions to run natively)
- Amount of RAM (possibly - but is not very unique)
The rest of the IDs should look the same because they are emulated hardware devices.
But, since this is 3 things, you couldn't necessarily use someone else's code (unless you run the cracked copy).
Does anyone know of a machine that will do the exact opposite? If you are like me, then you have way too many bookshelves filled with technical reference books, many of them 10 years old. But you can't bare to throw away that ancient PC Interrupts book least someday you have an arcane question about the PC BOIS call while working on a Virtual PC emulator. I agree that reading paper copies is preferable most of the time, but books that I'm going to access once every 5 years would be better off as digital. These books take of a ton of space and space is expensive in San Francisco.
I want a machine that can remove the bindings from books and scan in both sides of the pages as a big stack of papers. Then I'm left with a digital copy, which fits into one millionth of the space, and some a lot of firewood. Anyone have a suggestion on the best way to remove book bindings? Are there service companies out there that will do this? It's not going to be worthwhile if it takes a lot of time or money per book, but I'd be willing to pay a dollar or two.
Do you have any recommendation on where to look for dot-com auctions? Local newspaper? Is there an online site for such listings (particularly in the SF/Bay Area).
Not getting mail from people who solely browse using lynx is an added bonus. :)
Good idea. This is how I "encrypt" my email address to hide it from spam robots when posting on the web. To bad GIF and JPEG can't do 2d-block compression or the size could be kept pretty small.
However without that the size would be very large and everyone would hate you for filling up their mailbox/harddrive.
To save them the space, you could run an external server that keeps your message in text form (it would be encrypted in transit or on a LAN) and passes them on to an external GIF image. When their mail client request the GIF, the external server generates the corresponding GIF for them. This way you save them space, and you can do evil things like track how often and when they read the message. Also you can prevent them from copying it to some degree because they aren't likely to retype the message if they can't cut and past. You can tell if they forwarded the message to someone else by looking at IP logs for the image access. So maybe Bill Gates really will send the 500th person to disney land! haha.
Any decent 128bit encryption would require a 1000 pentiums to work for... oh say.. the lifetime of the planet earth (at which point the power would fail). If it only takes 10 years then that smacks of weakened security mandated by the government.
From the article it almost sounds like they are using frequency hopping with a 128bit random function. I'm not sure but that might be legal in more countries because the data is not actually encrypted or decrypted - just the frequency path followed.
The hard part, identifing the runner's and their time was done with ankle radio trackers - I'm assuming. So really all this did was given a list of times and 5 hours of frames, mpeg encode 180 frames per runner. Doesn't seem like that difficult of a task that 45 machines would be needed. If they used re-used information then the encoding could probably be done in less then 12hrs on 1 server... but it sounds like it was just a brute force shell script that brute forced the videos as seperate operations. Whatever works!
Oh well, I guess the enemy of my enemy is my friend. That one always works. Just look how it's helped the U.S. Gov't in the last half a century.
Sounds like you are trying to be sarcastic here. But you probably wouldn't be here if the Allies didn't form against a common enemy. About the only thing we had in common with Russia is a hatred of Hitler and Germans. Without Russia, it is very questionable if we would have won the war. Not tring to imply that carries over to AMD and Transmeta though. And yes, that is slightly more than a half century ago.
It's not "unplugged from the Internet", it's "unplugged". As in unpowered.
Even then, you are not totally safe. The contents of your RAM are often valid for several seconds to several minutes after you power off. With lower temperatures this can be up to hours. This must be taken into account with high security applications where physical access is possible. For example, tamper-detection circuits must erase RAM as well as EEPROMs when intrusion is detected.
Since PCs don't clear out memory before they power off, your passwords and encryption keys could possibly be stolen from RAM even with the best security precautions taken. Mind you, I haven't heard of anyone actually using this technique, just that it's a possibility.
Ohh.. I can't resist a troll...
;
:)
executables have stripped relocation tables, you can only load them to address 0x400000
That's not true. Most exe's have relocation information. You can strip it out, but it's pretty rare that EXEs have this information removed. Even if they did it just means the process that loads them needs to be loaded at a different base address. There is nothing special about 0x400000, many exes have a base address other than that. Notepad.exe uses 0x1000000 for example (under w2k). Use the program "rebase" that comes with W2k to change the base address to whatever you like. You can load an EXE without relocation information as DLL. Also DLLs don't have to have relocation information assuming they are not going to conflict.
even if the relocation table would be present, the OS doesn't permit loading & dynamically linking against an executable. It was originally possible, but the changed because it can cause security problem.
Try it yourself :
HMODULE m=LoadLibrary("c:\\winnt\\system32\\notepad.exe")
Amazing!!
Changing variable names won't substantially change the binary code the code compiles into.
Changing variable names won't change the binary code the code compiles into by a single bit. Compilers don't care what variables are named as long as you follow proper syntax.
So what's the difference between linking with a DLL and forking a processes that is GPLed - you can achieve the same levels of functionality. In Windows, the DLL and EXE file format are exactly the same. So renaming a .EXE to .DLL would be a violation of GPL? Taking this argument to it's logical extreme, any program that can create a pipe to another process could be in violation of GPL.
I was kind of surprised to see the climbers using this machine didn't have on a saftey line - so they must trust it pretty well.
If you realize you are about to run out of air pressure then you get out your mountain climbing equipement and do it the old fashion way. I.e. hammer a bolt into the surface you are climbing and hook a line into your harness. If you are climbing a building, it *might* be hard to hammer into steel - but there is almost always some thing to grab onto in bulding structures - window ledges, etc.
Don't forget that the 1/3rd paid for legal fees is tax-deductible. That means you could recover an a total of 10% more through legal means. Probably not worth it for other matters:
- Legal procedures could take months - they might go bankrupt or shut down before you can collect.
- They aren't likely to keep you around as a client if you sue them. If you don't have any better options out there then you might need to keep collecting you 50% to stay in business.
And before US States started using standardized testing, all States reported their students as "Above Average."
Some points well made, I don't have time to write back right now - but just wanted acknowdege your comments were read. :)
slashdot doesn't like XML... try again with ( replacing
XML:
(print)(plus arg1=4 arg2=5/)(/print)
LISP is a programming language. XML is a syntax.
Lisp programs are not programs in the interpreter. They are data. You can call (eval) on the data, but it's still just data that conforms to a standard. That is what makes lisp cool is that data=code. You can do the same with XML - that is create a stand where XML data can be evalulated. For example :
In lisp this would be :
(print (+ 4 5))
After the parser has gone through both cases the data structures are basically the same. LISP can be used as a programming language and compiled to executable format, but it can also be used as a data-definition language. A lisp parser doesn't need the LISP execution environment - i.e. garbage collection and predefined functions. But, by allowing predefined functions such as defclass you can do everything you can do in XML in LISP.