Slashdot Mirror

Sparrowvsrevolution writes "With much of the web upset over about Google's latest privacy policy changes, it's helpful to remember it could be much worse: A search engine called Skipity offers the world's worst privacy policy (undoubtedly tongue-in-cheek), filled with lines like this: 'You may think of using any of our programs or services as the privacy equivalent of living in a webcam fitted glass house under the unblinking eye of Big Brother: you have no privacy with us. If we can use any of your details to legally make a profit, we probably will.' The policy gives the company the right to sell any of your data that it wants to any and all corporate customers, send you limitless spam, track your movements via GPS if possible, watch you through your webcam, and implant a chip in your body that is subject to reinstallation whenever the company chooses."

brothke writes "In the classic poem Inferno, Dante passes through the gates of Hell, which has the inscription abandon all hope, ye who enter here above the entrance. After reading The Tangled Web: A Guide to Securing Modern Web Applications, one gets the feeling the writing secure web code is akin to Dante's experience." Read below for Ben's review. The Tangled Web: A Guide to Securing Modern Web Applications author Michal Zalewski pages 320 publisher No Starch Press rating 10/10 reviewer Ben Rothke ISBN 1593273886 summary Incredibly good and highly technical book on browser security coding In this incredibly good and highly technical book, author Michal Zalewski writes that modern web applications are built on a tangled mesh of technologies that have been developed over time and then haphazardly pieced together. Every piece of the web application stack, from HTTP requests to browser-side scripts, comes with important yet subtle security consequences. In the book, Zalewski dissects those subtle security consequences to show what their dangers are, and how developers can take it to heart and write secure code for browsers.

The Tangled Web: A Guide to Securing Modern Web Applications is written in the same style as Zalewski's last book - Silence on the Wire: A Field Guide to Passive Reconnaissance and Indirect Attacks, which is another highly technical and dense book on the topic. This book tackles the issues surrounding insecure web browsers. Since the browser is the portal of choice for so many users; its inherent secure flaws leaves the user at a significant risk. The book details what developers can do to mitigate those risks.

This book starts out with the observation that while the field of information security seems to be a mature and well-defined discipline, there is not even a rudimentary usable framework for understanding and assessing the security of modern software.

In chapter 1, the book provides a brief overview of the development of the web and how so many security issues have cropped in. Zalewski writes that perhaps the most striking and nontechnical property of web browsers is that most people who use them are overwhelmingly unskilled. And given the fact that most users simply do not know enough to use the web in a safe manner, which leads to the predicament we are in now.

Zalewski then spends the remainder of the book detailing specific problems, how they are exploited, and details the manner in which they can be fixed.

In chapter 2, the book details that something as elementary as how the resolution of relative URL's is done isn't a trivial exercise. The book details how misunderstandings occur between application level URL filters and the browser when handling these types of relative references can lead to security problems.

For those that want a feel for the book, chapter 3 on the topic of HTTP is available here.

Chapter 4 deals with HTML and the book notes that HTML is the subject of a fascinating conceptual struggle with a clash between the ideology and the reality of the on-line world. Tim Berners-Lee had the vision of a semantic web;namely a common framework that allows data to be shared and reused across applications, companies and the entire web. The notion though of a semantic web has not really caught on.

Chapter 4 continues with a detailed overview of how to understand HTML parser behavior. The author writes that HTML parsers will second-guess the intent of the page developer which can leads to security problems.

In chapter 12, the book deals with third-party cookies and notes that since their inception, HTTP cookies have been misunderstood as the tool that enables online advertisers to violate users privacy. Zalewski observes that the public's fixation on cookies is deeply misguided. He writes there is no doubt that some sites use cookies as a mechanism for malicious use. But that there is nothing that makes it uniquely suited for this task, as there are many other equivalent ways to sore unique identifiers on visitor's computes, such as cache-based tags.

Chapter 14 details the issue of rogue scripts and how to manage them. In the chapter, the author goes slightly off-topic and asks the question if the current model of web scripting is fundamentally incompatible with the way human beings works. Which leads to the question of it if is possible for a script to consistently outsmart victims simply due to the inherent limits of human cognition.

Part 3 of the book takes up the last 35 pages and is a glimpse of things to come. Zalewski optimistically writes that many of the battles being fought in today's browser war is around security, which is a good thing for everyone.

Chapter 16 deals with new and upcoming security features of browsers and details many compelling security features such as security model extension frameworks and security model restriction frameworks.

The chapter deals with one of the more powerful frameworks is the Content Security Policy (CSP) from Mozilla. CSP is meant to fix a large class of web application vulnerabilities, including cross site scripting, cross site request forgery and more. The book notes that as powerful as CSP is, one of its main problems is not a security one, in that it requires a webmaster to move all incline scripts on a web page to a separately requested document. Given that many web pages have hundreds of short scripts; this can be an overwhelmingly onerous task.

The chapter concludes with other developments such as in-browser HTML sanitizers, XSS filtering and more.

Each chapter also concludes with a security engineering cheat sheetthat details the core themes of the chapter.

For anyone involved in programming web pages, The Tangled Web: A Guide to Securing Modern Web Applications should be considered required reading to ensure they write secure web code. The book takes a deep look at the core problems with various web protocols, and offers effective methods in which to mitigate those vulnerabilities.

Michal Zalewski brings his extremely deep technical understanding to the book and combines it with a most readable style. The book is an invaluable resource and provides a significant amount of information needed to write secure code for browsers. There is a huge amount of really good advice in this book, and for those that are building web applications, this is a book they should read.

Ben Rothke is the author of Computer Security: 20 Things Every Employee Should Know.

You can purchase The Tangled Web: A Guide to Securing Modern Web Applications from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

brothke writes "In the classic poem Inferno, Dante passes through the gates of Hell, which has the inscription abandon all hope, ye who enter here above the entrance. After reading The Tangled Web: A Guide to Securing Modern Web Applications, one gets the feeling the writing secure web code is akin to Dante's experience." Read below for Ben's review. The Tangled Web: A Guide to Securing Modern Web Applications author Michal Zalewski pages 320 publisher No Starch Press rating 10/10 reviewer Ben Rothke ISBN 1593273886 summary Incredibly good and highly technical book on browser security coding In this incredibly good and highly technical book, author Michal Zalewski writes that modern web applications are built on a tangled mesh of technologies that have been developed over time and then haphazardly pieced together. Every piece of the web application stack, from HTTP requests to browser-side scripts, comes with important yet subtle security consequences. In the book, Zalewski dissects those subtle security consequences to show what their dangers are, and how developers can take it to heart and write secure code for browsers.

The Tangled Web: A Guide to Securing Modern Web Applications is written in the same style as Zalewski's last book - Silence on the Wire: A Field Guide to Passive Reconnaissance and Indirect Attacks, which is another highly technical and dense book on the topic. This book tackles the issues surrounding insecure web browsers. Since the browser is the portal of choice for so many users; its inherent secure flaws leaves the user at a significant risk. The book details what developers can do to mitigate those risks.

This book starts out with the observation that while the field of information security seems to be a mature and well-defined discipline, there is not even a rudimentary usable framework for understanding and assessing the security of modern software.

In chapter 1, the book provides a brief overview of the development of the web and how so many security issues have cropped in. Zalewski writes that perhaps the most striking and nontechnical property of web browsers is that most people who use them are overwhelmingly unskilled. And given the fact that most users simply do not know enough to use the web in a safe manner, which leads to the predicament we are in now.

Zalewski then spends the remainder of the book detailing specific problems, how they are exploited, and details the manner in which they can be fixed.

In chapter 2, the book details that something as elementary as how the resolution of relative URL's is done isn't a trivial exercise. The book details how misunderstandings occur between application level URL filters and the browser when handling these types of relative references can lead to security problems.

For those that want a feel for the book, chapter 3 on the topic of HTTP is available here.

Chapter 4 deals with HTML and the book notes that HTML is the subject of a fascinating conceptual struggle with a clash between the ideology and the reality of the on-line world. Tim Berners-Lee had the vision of a semantic web;namely a common framework that allows data to be shared and reused across applications, companies and the entire web. The notion though of a semantic web has not really caught on.

Chapter 4 continues with a detailed overview of how to understand HTML parser behavior. The author writes that HTML parsers will second-guess the intent of the page developer which can leads to security problems.

In chapter 12, the book deals with third-party cookies and notes that since their inception, HTTP cookies have been misunderstood as the tool that enables online advertisers to violate users privacy. Zalewski observes that the public's fixation on cookies is deeply misguided. He writes there is no doubt that some sites use cookies as a mechanism for malicious use. But that there is nothing that makes it uniquely suited for this task, as there are many other equivalent ways to sore unique identifiers on visitor's computes, such as cache-based tags.

Chapter 14 details the issue of rogue scripts and how to manage them. In the chapter, the author goes slightly off-topic and asks the question if the current model of web scripting is fundamentally incompatible with the way human beings works. Which leads to the question of it if is possible for a script to consistently outsmart victims simply due to the inherent limits of human cognition.

Part 3 of the book takes up the last 35 pages and is a glimpse of things to come. Zalewski optimistically writes that many of the battles being fought in today's browser war is around security, which is a good thing for everyone.

Chapter 16 deals with new and upcoming security features of browsers and details many compelling security features such as security model extension frameworks and security model restriction frameworks.

The chapter deals with one of the more powerful frameworks is the Content Security Policy (CSP) from Mozilla. CSP is meant to fix a large class of web application vulnerabilities, including cross site scripting, cross site request forgery and more. The book notes that as powerful as CSP is, one of its main problems is not a security one, in that it requires a webmaster to move all incline scripts on a web page to a separately requested document. Given that many web pages have hundreds of short scripts; this can be an overwhelmingly onerous task.

The chapter concludes with other developments such as in-browser HTML sanitizers, XSS filtering and more.

Each chapter also concludes with a security engineering cheat sheetthat details the core themes of the chapter.

For anyone involved in programming web pages, The Tangled Web: A Guide to Securing Modern Web Applications should be considered required reading to ensure they write secure web code. The book takes a deep look at the core problems with various web protocols, and offers effective methods in which to mitigate those vulnerabilities.

Michal Zalewski brings his extremely deep technical understanding to the book and combines it with a most readable style. The book is an invaluable resource and provides a significant amount of information needed to write secure code for browsers. There is a huge amount of really good advice in this book, and for those that are building web applications, this is a book they should read.

Ben Rothke is the author of Computer Security: 20 Things Every Employee Should Know.

You can purchase The Tangled Web: A Guide to Securing Modern Web Applications from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

jrepin writes "The KDE community has released version 4.8 of their Free and open source software bundle. The new version provides many new features, improved stability, and increased performance. Highlights for Plasma Workspaces include window manager optimizations, the redesign of power management, and integration with Activities. The first Qt Quick-based Plasma widgets have entered the default installation of Plasma Desktop, with more to follow in future releases. KDE applications released today include Dolphin file manager with its new display engine, ..., and KDE Telepathy reaching its first beta milestone. New features for Marble virtual globe keep arriving, among these are: Elevation Profile, satellite tracking, and Krunner integration. The KDE Platform provides the foundation for KDE software. KDE software is more stable than ever before. In addition to stability improvements and bugfixes, Platform 4.8 provides better tools for building fluid and touch-friendly user interfaces, integrates with other systems' password saving mechanisms and lays the base for more powerful interaction with other people using the new KDE Telepathy framework."

alphadogg writes with an excerpt from a Network World article: "The 2012 Japan Prize, one of the world's most prestigious science and technology honors, has been awarded to three American medical experts who are fighting cancer as well as to a Japanese inventor whose magnet technology has implications for energy conservation. In years past, the prize (which comes with a $650K purse) has honored computing accomplishments, including last year, when Unix inventors Dennis Ritchie and Ken Thompson were honored."

New submitter Novin writes with exciting news from the Cinnamon project. Quoting the release announcements: "Cinnamon 1.2 is out! All APIs and the desktop itself are now fully stable! I hope you'll enjoy the many new features, the desktop effect, desktop layouts, the new configuration tool, the applets, changes, bug fixes, and improvements that went into this release. This is a huge step forward for Cinnamon." The release reintroduces desktop effects, fixes a slew of bugs, and introduces a new applet API (fixing a number of issues intrinsic to shell extensions).

New submitter Novin writes with exciting news from the Cinnamon project. Quoting the release announcements: "Cinnamon 1.2 is out! All APIs and the desktop itself are now fully stable! I hope you'll enjoy the many new features, the desktop effect, desktop layouts, the new configuration tool, the applets, changes, bug fixes, and improvements that went into this release. This is a huge step forward for Cinnamon." The release reintroduces desktop effects, fixes a slew of bugs, and introduces a new applet API (fixing a number of issues intrinsic to shell extensions).

parallel_prankster writes "The Washington Post reported Tuesday that Google will require users to allow the company to follow their activities across e-mail, search, YouTube, and other services; a radical shift in strategy that is expected to invite greater scrutiny of its privacy and competitive practices. The information will enable Google to develop a fuller picture of how people use its growing empire of Web sites. Consumers will have no choice but to accept the changes. The policy will take effect March 1 and will also impact Android mobile phone users. 'If you're signed in, we may combine information you've provided from one service with information from other services,' Alma Whitten, Google's director of privacy, product, and engineering, wrote in a blog post." The angle of the Washington Post article is a bit negative; Google sees this as consolidating an absurd number of privacy policies for its various services into a single, unified document. Reader McGruber adds: "Donald E. Graham, the Washington Post's chairman and CEO, joined Facebook's Board of Directors in January 2009. Curiously, the Washington Post article neglects to disclose that."

Back in 2009 G.M. and Segway talked about the P.U.M.A., or Personal Urban Mobility and Accessibility vehicle. Now it's the EN-V, which stands for Electric Network Vehicle. G.M. (along with partner Shanghai Automotive Industry Corporation) debuted the thing in Shanghai in 2010, then displayed it at the Las Vegas Consumer Electronics Show in 2011, and now they're showing it off at auto shows, no doubt hoping to get a lot of buzz going for this two-wheeled wonder, which is supposed to be so loaded with navigation and collision avoidance electronics that you can sleep in it on your way to work. (Please wake us up when we get there, okay?)

bs0d3 writes "Since the news was released that Poland will sign ACTA later this month, activists have taken to the streets in protest. Also, Anonymous has aimed their DDoS cannons at Polish websites. A government minister admitted the government had failed to fully consult the public on the issue. Piratbyran Co-Founder Marcin de Kaminski has been following the issue on ACTA in Poland, and agrees with activists that Anonymous' DDoS is hurting the situation. Now the Polish government is trying to speed up the signatory process, making a statement of not giving in to 'cyber terrorists.'"

An anonymous reader writes "The petition on 'We the People' website petitioning the administration to investigate Chris Dodd for corruption has reached the required 25,000 votes in two days: now the government has to officially respond to the petition. The petition ... stemmed from Chris Dodd's statement that tried to portray campaign donations as quid-pro-quos for SOPA/PIPA votes."

wiredmikey writes with an update on Microsoft's takedown of the Kelihos botnet. From the article: "Microsoft is not just taking down botnets; it is taking them down and naming names. In an amended complaint [PDF] filed Monday in U.S. District Court for the Eastern District of Virginia, Microsoft named a man from St. Petersburg, Russia, as the alleged head of the notorious Kelihos botnet. Naming names can be a risky business. Previously, Microsoft alleged Dominique Alexander Piatti, dotFREE Group SRO and several unnamed 'John Does' owned a domain cz.cc and used cz.cc to register other subdomains used to operate and control the Kelihos botnet. However, the company later absolved Piatti of responsibility when investigators found neither he nor his business was controlling the subdomains used to host Kelihos. Whether naming Sabelnikov – who, according to Krebs on Security, once worked as a senior system developer and project manager for Russian antivirus vendor Agnitum, will have the same effect as naming the Koobface gang remains to be seen. Though Kelihos has remained defunct since the takedown last year, the malware is still on thousands of computers."

wiredmikey writes with an update on Microsoft's takedown of the Kelihos botnet. From the article: "Microsoft is not just taking down botnets; it is taking them down and naming names. In an amended complaint [PDF] filed Monday in U.S. District Court for the Eastern District of Virginia, Microsoft named a man from St. Petersburg, Russia, as the alleged head of the notorious Kelihos botnet. Naming names can be a risky business. Previously, Microsoft alleged Dominique Alexander Piatti, dotFREE Group SRO and several unnamed 'John Does' owned a domain cz.cc and used cz.cc to register other subdomains used to operate and control the Kelihos botnet. However, the company later absolved Piatti of responsibility when investigators found neither he nor his business was controlling the subdomains used to host Kelihos. Whether naming Sabelnikov – who, according to Krebs on Security, once worked as a senior system developer and project manager for Russian antivirus vendor Agnitum, will have the same effect as naming the Koobface gang remains to be seen. Though Kelihos has remained defunct since the takedown last year, the malware is still on thousands of computers."

For the first few years of its existence, it would have been fair to say that Canonical was essentially polishing, packaging and publishing Debian Linux (and Gnome) to create the base Ubuntu desktop, to great acclaim. For the past few years, though, the company has pushed new looks and new applications (cf. Unity and Ubuntu TV), and refused to stick with prettifying existing interfaces. Now, Barence writes with this excerpt from PC Pro: "Ubuntu is set to replace the 30-year-old computer menu system with a 'Head-Up Display' that allows users to simply type or speak menu commands. Instead of hunting through drop-down menus to find application commands, Ubuntu's Head-Up Display lets users type what they want to do into a search box. The system suggests possible commands as the user begins typing – entering 'Rad' would bring up the Radial blur command in the GIMP art package, for example. HUD also uses fuzzy matching and learns from past searches to ensure the correct commands are offered to users. Canonical's Mark Shuttleworth told PC Pro the HUD will make it easier for people to learn new software packages, and migrate from Windows to Linux software without having to relearn menus. The HUD will first appear in Ubuntu 12.04."

For the first few years of its existence, it would have been fair to say that Canonical was essentially polishing, packaging and publishing Debian Linux (and Gnome) to create the base Ubuntu desktop, to great acclaim. For the past few years, though, the company has pushed new looks and new applications (cf. Unity and Ubuntu TV), and refused to stick with prettifying existing interfaces. Now, Barence writes with this excerpt from PC Pro: "Ubuntu is set to replace the 30-year-old computer menu system with a 'Head-Up Display' that allows users to simply type or speak menu commands. Instead of hunting through drop-down menus to find application commands, Ubuntu's Head-Up Display lets users type what they want to do into a search box. The system suggests possible commands as the user begins typing – entering 'Rad' would bring up the Radial blur command in the GIMP art package, for example. HUD also uses fuzzy matching and learns from past searches to ensure the correct commands are offered to users. Canonical's Mark Shuttleworth told PC Pro the HUD will make it easier for people to learn new software packages, and migrate from Windows to Linux software without having to relearn menus. The HUD will first appear in Ubuntu 12.04."

While he was at the International Auto Show in Detroit, Timothy Lord looked at the Tesla Model S, then the CODA electric car. Now he shows us the latest Smart (brand) gas and electric vehicles, including a "concept car" he doesn't think will make it into production.

While he was at the International Auto Show in Detroit, Timothy Lord looked at the Tesla Model S, then the CODA electric car. Now he shows us the latest Smart (brand) gas and electric vehicles, including a "concept car" he doesn't think will make it into production.

jrepin writes with this quote from a post at the European Commission's JoinUp site: "The administration of Spain's autonomous region of Extremadura is moving to a complete open source desktop, replacing the current proprietary desktop platform, confirms the region's CIO, Teodomiro Cayetano López. The IT department started a project to install the Debian distribution on all 40,000 desktop PCs. 'The project is really advanced and we hope to start the deployment the next spring, finishing it in December.' The project makes it Europe's second largest open source desktop migration, between the French Gendarmerie (90,000 desktops) and the German city of Munich (14,000 desktops)."

darthcamaro writes "In any given year, Slashdot always has stories about how a researcher hacked a browser in only a few minutes at the Pwn2own hacking challenge. This year the rules are a bit different, and instead of hackers winning for just one vulnerability, the rules allow for multiple vulnerabilities to be presented. The winner isn't the first one to hack a browser, but is the one that can hack the browser the most. 'In the past, due to the way the competition was architected, we had lots of sensationalist headlines, things like "Mac hacked in three seconds,"' said Aaron Portnoy, Manager of the Security Research Team at HP TippingPoint. 'We don't think that type of sensationalism was representative of all the research that was going on.'"

A Commentor writes "Perhaps to balance the good news with the Supreme Court ruling on GPS, a judge in Colorado has ordered a defendant to decrypt her hard drive. The government doesn't have the capability to break the PGP encryption, and 'the Fifth Amendment is not implicated by requiring production of the unencrypted contents' of the defendant's computer."

redletterdave writes "On Jan. 19, Apple introduced iBooks 2, its digital solution to the physical textbook. In the first three days of release, users have downloaded more than 350,000 e-textbooks from the new platform, and more than 90,000 users have downloaded the authoring tool to make those e-textbooks, called iBooks Author. It makes sense that Apple's iBooks 2 platform is taking off in such a short period of time; there is very little merit to the physical textbook, and the education industry has been waiting for a viable solution like this for some time. Physical textbooks lack portability, durability, accessibility, consistent quality, interactivity and searchability, and they're not environmentally friendly."

An anonymous reader writes "Over the weekend, Blake Ross, Facebook's product director and co-founder of Firefox, worked with Facebook engineers Tom Occhino and Marshall Roch to demonstrate how evil they think Google's newly launched Search plus Your World (SPYW) feature really is, and created a 'proof of concept' showing how it should really work. His team got some help from Twitter engineers and Myspace engineers, and consulted other social networks as well to really make sure the message hits home: SPYW should surface results from all social networks, not just Google+. By leveraging Google's own algorithms, the group built a bookmarklet called 'don't be evil' (a jab at Google's informal motto) and released it on a new website named Focus on the User."

An anonymous reader writes "In 2009 we discussed the Star Wars Uncut Project, a crowd-sourced effort to re-create the entirety of Episode IV as series of 15-second clips filmed and submitted by fans. Well, it's done at last, and CmdrTaco has written his review of the two-hour film. He says, 'A great number of shots demonstrate technology well beyond what was available when the original Star Wars was made. Other shots are stylishly animated in CG. Others are rotoscoped in mediums from chalk to pencil to marker. There’s also a plethora of stop motion, claymation, and even some puppetry to go around. [A] few shots are only slightly modified from the originals, but those are the rarity, and generally done to pull off a specific joke. ... It's a glorious achievement.'"

You asked Carl Malamud about his experiences and hopes in the gargantuan project he's undertaken to prod the U.S. government into scanning archived documents, and to make public access (rather than availability only through special dispensation) the default for newly created, timely government data. (Malamud points out that if you have comments on what the government should be focusing on preserving, and how they should go about it, the National Archives would like to read them.) Below find answers with a mix of heartening and disheartening information about how the vast project is progressing.

LoC?
by an Anonymous Reader

So how many GB/TB is a Library of Congress? :)

Or, more seriously, how big are you estimating? Are you using raw scans or some sort of compression (JPG, PNG, etc)? What resolution are you using? Do you vary the resolution depending on the document?

What sort of meta data are you putting in?

CM: The reason John Podesta and I suggested a Federal Scanning Commission in our letter at YesWeScan.Org is we really don't know how big the holdings of the government are. I can tell you that the Library of Congress is about 32 million cataloged books (a significant increase from the 6,487 books Thomas Jefferson donated to get them started). But, this is about more than books, it is about paper records, microfilmed technical papers, video, audio, photographs, and much more.

The scale is fairly vast. The Smithsonian has 137 million objects, including about 13 million images. David Ferriero, the Archivist of the United States estimates he has over 10 billion pages of text documents, 7.2 million maps, and 40 million photographs including everything from past census records to presidential dinner menus, and that includes about 7.5 million motion pictures and sound recordings. The Government Printing Office distributes their documents to the Federal Depository Library Program, and that includes over 60 million pages of collections including the Official Journals of Government such as the Federal Register. That's just scratching the surface, and we recommended a Federal Scanning Commission to begin the process of understanding what we have (and what is worth digitizing).

As to standards? There are lots of pretty good standards on how to digitize. NARA, Library of Congress, GPO all spec out document scans at 400 dpi, for example. For photographs, moving images, and other objects, there are some pretty good and pretty detailed standards at www.digitizationguidelines.gov. I know Brewster Kahle's operation and my own tend to work off those specifications (in fact Brewster does quite a bit of scanning for the government).

As to compression? Well, I've found people tend to overcompress things. That said, sometimes the initial quality isn't that great, so a 600 dpi uncompressed scan would be silly in some cases. But, for photographs I try very hard to keep the TIFF images around and not rely on JPEG. Likewise, for audio it is really nice to keep a nice 48 khz version of your file around if you can simply because if you screw up the compression maybe somebody else can do a better job in a few years. Disk space is relatively cheap, so that isn't the barrier it used to be. For video, I rip MPEG2 at whatever it is on a DVD, when I'm actually digitizing I try to get the video bitrate up to 8-10 mbps when ripping a Betacam or Umatic. Some people think that is overkill, but I'd rather be safe than sorry.

Metadata? Well, you got to have it or you're not going to get very far when it comes to access. Many librarians have made perfect the enemy of the good when it comes to metadata and have resisted any attempt at digitization because we don't have the very best metadata we might have. I'm more in the camp of scan what you have and get as much of the metadata as you can into it. For example, we have 3,200 1000-page volumes of briefs from the 9th Circuit of the U.S. Court of Appeals. We didn't have good metadata, but we had the Internet Archive scan them anyway. Then, after we got our PDF files, I shipped those off to a double-key team in India and they broke the briefs up into individual documents and typed the metadata into a spreadsheet for me, which we hope to release soon.

My point is that sometimes you can shoehorn the metadata in after the fact or you can use a variety of techniques to pull the metadata out of the documents (e.g., smart OCR). In theory, you can use crowdsourcing to get the metadata, but so far I've not had a lot of luck persuading thousands of people to spend their time doing that kind of work. A captcha is a quick thing to do and is between you and something you want, whereas entering metadata in for videos or documents is one of those civic duty things that everybody thinks everybody else should be doing.

Total size? Brewster says a book is about 400 Mbytes (though he's very quick to point out that you could put the words in all the books in the library into a terabyte and if you're distributing PDFs, you can easily throw 130,000 full-color, searchable PDFs onto a 4 TB drive). But, you were probably asking about raw data. Here's some raw numbers:

32 million books at 400 Mbytes each is 12.8 petabytes 50 million photos at 150 Mbytes each is 7.5 petabytes 10 billion pieces of paper ("records") at 100 Kbytes each is 1 petabyte 20 years of video at 8 mbps is only 630 Tbytes.

(Somebody check my math?)

If you're talking a decade-long federal digitization initiative, we're looking at well south of 50 petabytes, which seems pretty doable in this day and age!



Can the rare books collections be digitized?
by autophile

Three closely related questions about the rare books collections at the Library of Congress:

1. I know there is some kind of effort going on to digitize the rare books collections, but can it be sped up? There are many high-quality low-cost archival book scanners out there (such as the ones developed at diybookscanner.org).

2. It gets really annoying to have to receive paper copies of books when copies are requested. Why not DVDs of high-quality images?

3. Why is there no outreach by the LoC to smaller, cheaper book scanning efforts? The Internet Archive, DIYBookscanner.org, and Decapod all come to mind.

CM: In reverse order. I don't know why we aren't distributing and decentralizing our scanning efforts. The Internet Archive is a heavy-duty production shop and they do an amazing job, as do folks like Google Books and the folks digitizing things the Mormon Church. But, there are a bunch of DIY solutions and it would be really nice if we could get more people pitching in. The biggest problem on distributing the digitization efforts is quality control. I know when it comes to ripping video, I can easily teach other people how to grab an MPEG2 off a DVD, but when it comes to things like digitizing a Betacam, that takes some training. But, we're all trainable and I wish we could all do more.

Getting back paper copies of books and papers when they're doing a copy anyway is just plain dumb. Likewise with things like FOIA results. John Podesta testified before the Senate about FOIA and said if an agency answers a FOIA request, they should also post their result online so others can see it. That seems pretty obvious.

As far as digitizing rare book collections, there are some amazing pockets throughout the government but there is no real coordination and there certainly is no effort to scan at scale or to come up with a realistic national digitization strategy. That is why we called on the White House to lead the effort. Within the Library of Congress there are some amazing collections, but if you look around to places like the National Agricultural Library or the National Library of Medicine or the libraries in the service academies you'll find lots more. Some have argued that digitizing rare books is silly because the audience is just a few academics, but I can tell you from my own experience helping host the network site for the Archimedes Palimpsest that when you make this kind of information available, there is an amazing long tail.

If you scan it, they will come. And, to answer your question, if we all scan it, they will come much sooner.



Real time legislation drafting
by kerskine

Would it be possible to implement a system that would allow real-time and continuous review of legislation while it's being drafted? Much has been made over the past three years about legislation being available for review before voting by the House or Senate. The final draft for review usually is huge PDF that makes it near impossible for citizens, interest groups, and the media to thoroughly analysis in time.

CM: You want to see the sausage being made not just buy the hot dog! I'll comment on the U.S. Congress since that's the system I know best. Thomas is a pretty good system if you happen to be stuck in 1994. It does have all the amendments and the actions and the various stages that legislation go through. But, it isn't real time, more like "pretty quick." As Van Jacobson once quipped, "Same day service in a nanosecond world." And, Thomas isn't really machine processable, it is final form, usually formatted ASCII text (shades of NROFF!). People like Josh Tauberer who built GovTrack.US have spent considerable time crawling those systems and trying to get the data into regularized formats and make it available to others to reuse via APIs, but that isn't the same as exposing the inner working of the sausage factory.

Majority Leader Cantor's staff has been pushing a system to make the raw data all available in XML from the Clerk's office and I think that is a very promising initiative which hopefully will bear fruit. (They're having a February 2 conference to discuss their plans if you are interested. I have no idea if it will be streamed for those of who aren't Inside the Beltway and I don't know their schedule for moving past conferences and into production.)

Congress is a pretty complicated beast. I know some folks like Sean McGrath have had better luck with some of the state legislatures. The problem is you need to dig deep into the inner working of a legislature. In the Congress, that means you're changing things like authoring tools that are used in the Clerk's office and by all the staff members, so you have to be careful or you get a bunch of really angry Congressman yelling at you because their staff can't crank out the flavor-of-the-week in the form of a bill or amendment.

There's also a bit of an issue of will. My work with the Congress to put hearings on-line showed that you could take the official transcripts of a hearing and use those to generate closed captions on the video. All you need is the official transcript of the hearing, but in order to get those I had to execute a special Memorandum of Understanding with the House Oversight Committee. Other committees guard their transcripts jealously and won't let them out for several when. When I started processing a bunch of historical videos we purchased from C-SPAN, I went to the Government Printing Office and found that many committees never deliver their transcripts, even a decade after the fact!



How to keep track of legislative activity about open access?
by oneiros27

Recently in the federal register, there were two calls for comments about access to data and research from federally funded research:

http://federalregister.gov/a/2011-28623 [federalregister.gov] http://federalregister.gov/a/2011-28621 [federalregister.gov]

I didn't hear about these until ~4 weeks after the original announcement, and with the holidays, it was too late to try to get the societies I'm involved with to prepare and vote on official statements. Are there any places where people can get/post notices of these sorts of things so that we can stay informed and try to help influence policies?

CM: The Federal Register is getting a lot better now that it is a much more open system. The idea of "Federal Register 2.0" was a paper I wrote for the Obama transition, so it is an issue I've tracked pretty closely and frankly, I've been amazed at how much better it is now. What they did is instead of selling the raw data feed for the Federal Register for $17,000/year, they went from SGML to XML and then released the data in bulk for free. A few guys out in San Francisco were looking for something to do to enter a contest and they took that bulk data and dreamed up GovPulse.US. That was such a better version of the Federal Register that the Office of the Federal Register switched the official site over to their open source platform. My point is the tools are there to do better notification mechanisms, and I'm sure the government would welcome somebody grabbing the GovPulse.US code out of Github and making it even better.

That's the technical answer. But, the substantive answer is that there is a huge boatload of stuff in the Federal Register and it is pretty hard to figure out what to pay attention to. I also missed that particular call for comment, and I've even missed several Requests for Information coming out of places I try and pay attention to, like the White House's Office of Science and Technology Policy. And, I do this stuff full-time! Perhaps better targeted notification mechanisms are the answer. Maybe it is a social media solution, where you pay attention to things your friends are paying attention to. I hope the answer is not that the only way to pay attention is to be employed with a beltway bandit which can afford hundreds of minions that do nothing but pay attention to Washington. Indeed, there are some very fancy for-pay services from folks like Congressional Quarterly and Bloomberg that cost an arm and a leg, but I can't help but think there has to be a better way that is also open.



What do you think of corporate partnerships?
by mhh5

I'd like to know what you think about corporate partnerships in the process to get public data released. (I'm not sure if Google Patents existed before the USPTO released its databases.) Do corporations that get involved in the process tend to make the process better without question, or are there tradeoffs in some areas because the corporations always want to help but then try to retain a proprietary version of the data for themselves?

CM: The theory is that the government gets some kind of valuable service (like digitization) that the government wouldn't get otherwise so it is a "win-win." But, the reality is all too often the government gets snookered and what we do is give some corporation exclusive access to some pot of data and the government doesn't get much of anything. The deal between Amazon and the National Archives was a good example of that kind of a private fence around the public domain. With a help from Boing Boing, I started systematically purchasing those public domain videos and re-releasing them in the wild. I have no problem with Amazon selling public domain video, I just hate it when they get a de facto or a contractual exclusive. (My testimony before Congress on this subject is here.)

There are lots of other examples of government getting snookered. For example, the Government Accountability Office let Thomson West get access to 60 million or so pages of federal legislative histories. At great cost to the government, they were all packed up and dispatched to West which digitized them all and then sent them back to the government. West now sells access to his amazing database. What did the government get for it's trouble? A few logins for GAO staffers. Even members of Congress need to pay to access the database! (We have an interesting paper trail on this issue.)

I'm glad you brought up the Google Patent system because I was personally involved in making that happen and I can tell you that this one is totally legit. Jon Orwant is the lead developer on this for Google and I played a small part in helping convince the White House and the Patent Office they ought to give Jon access to their data (the heavy lifting on that deal was by Beth Noveck who was the Deputy CTO at the time). Google makes all the data they got from the Patent Office available for bulk access with no strings attached. I can vouch for that because I did a mirror of their system. Last I heard Google was sending out anywhere from 1 to 10 terabytes of data PER DAY to external sources and even normally very critical folks who work in this arena have been really happy.

The big problem in the Patent Office is their computing infrastructure is a real catastrophe. Their power plant is over 95% capacity (e.g., plug in a computer, bring the building down!) and even though the Under Secretary knew that selling DVD subscriptions was silly, he wasn't able to switch over to an FTP service. He cut the deal with Google Patent and it worked out well for the government, for Google, and for everybody else.

What's the difference between the Google deal and the Amazon deal? In the case of the Amazon and GAO/West deals, the government lawyers did all the negotiating and they were totally outsmarted by some sharks in industry. But, when government has people like Under Secretary Kappos and Beth Noveck doing the negotiating, these things can work out just fine. The key is government should partner with people who want to do public service, not people who want to service the public.

Encouraging Governments?
by theNAM666

In a city such as Nashville, things as basic as business ownership and property records are not available online. In states such as New Jersey, public records such as basic corporate filings (officers, operating address/address for service of process) are accessible only for a fee.

What concrete actions can citizens confronting such situations, take to encourage accessibility and accountability?

CM: I find you need a carrot and a stick to make this stuff happen, especially at the local level. Folks like Everyblock.Com and CodeForAmerica.Org have done great working prying some of these databases loose, but there is still lots to do.

The first thing you should do is pick up the phone (or pick up your email client) and write/call the people who run the system. Ask them if you can have access to the data. Sometimes, it is as simple as that.

Other times, though, it isn't quite as simple since they want the money (or they want the control or they think this should be done by "private industry" by which they mean some buddy who is a contractor). The nice thing about any government system is somebody usually has oversight responsibilities. So, the next step is to find a city council member of state legislator who has oversight on the agency in question and ask them.

Again, life isn't usually that simple, but sometimes you win! If you can't get anywhere that way, what I usually end up doing is basically competing with the government system. Build a proxy system like RECAPtheLaw.Org did to recycle paid documents. Or, get a sponsor and buy a reasonable number of docs and build a web site that looks like it is going to be a real production system.

Then, go back again and ask. Maybe if you have eyeballs or at least have a nice web site, that is enough to get the government moving. But, if that doesn't do the job, you may have no choice but to compete with them for real, which of course requires a big commitment in time and energy and not everybody can do that. I know in the case of the Patent Office, I started pestering them in 1993, including several times when I spent 6-figure sums purchasing their data, and it still took until 2011 to crack that nut.

The real trick is focus/obsession. Pick one thing you really care about and just keep pestering them until you crack it open. If you're surfing from one opengov problem to another, showing up for a 1-day hackathon then moving on to something else, you're not going to get anywhere. Pick something real and make it your thing.



Privately Owned, Copyrighted Law
by AdamnSelene

I think I have read that the law itself cannot be copyrighted and it should be possible to make it available available to everyone. But as a techie who drafts standards and specifications, I was wondering about how far this goes--especially since Congress recently proposed enacting some of our standards into law. (They decided not to, but they read some parts into the committee records as they debated.) Can you still accomplish your project if a governmental body adopts (or considers adopting) a privately owned, copyrighted technical reference manual or set of safety standards as administrative law (or regulations that carry the force of law)? Or would such obstacles keep you from being able to digitize all of the government's laws (and archives of proposed laws)?

CM: The idea that the law has no copyright is a fundamental part of the American system of government. That applies to states and municipalities as well. The basic decision is Wheaton v. Peters from 1834 but that decision has been reaffirmed over and over. The law is sacred in the American system. You can't have equal protection under the law or due process under the law if there is a poll tax on access to justice.

When we get to a privately developed standards however, it turns into a very interesting issue. The basic mechanism is called Incorporation by Reference. The government will take some external document (such as a model building code) and incorporate the entire text to make it the law of the land. A guy named Peter Veeck was responsible for a landmark decision in 2002 when he published the Texas Building Code which was an incorporation of a privately-developed and very expensive model code. The court ruled that while the model code had copyright, the law of the land did not.

Based on the Veeck decision, my group went and posted many of the public safety codes enacted by the states. We started by purchasing model codes, finding the incorporating legislation, and concatenating the two pieces together and posting the resulting PDFs. More recently, we've done some extensive reworking of the California public safety codes, known as Title 24, converting the entire text into valid XHTML, recoding the graphics as SVG graphics, the formulas as MathML, and regenerating the PDF documents as nicely typeset documents instead of low-quality scans. You can see this work on the web but it is also available as Google Code project.

The federal government also uses this mechanism intensively, with over 2,000 standards incorporated into the Code of Federal Regulations. This is non-trivial stuff, things like all the OSHA safety regulations. The issue was recently considered by a federal group called the Administrative Conference of the U.S. which basically rolled over and endorsed the idea that it is ok for important parts of the law to cost money. (Read EFF's protest letter if you want a good critique of what they did.)

I'm not necessarily saying that government should be able to appropriate any privately-developed standard and make it available. And, I'm not necessarily saying you want OSHA bureaucrats drafting the standards. But, I do think the big standards establishment and the government regulators have cut a deal that results in the law not being available and the costs forked off on private citizens and small business with extortionate monopoly prices. I just paid $847 for a 48-page safety standard from Underwriters Labs and $60 for 2-page safety standard from the Society of Automotive Engineers, both of which are mandated by law in the CFR. They do need money to run their operations, but let me just point out that in 2009 the 501(c)(3) nonprofit Underwriters Labs paid their CEO $2,138,984 and the nonprofit SAE paid their CEO $412,578.

Ancestry.com
by An Anonymous Reader

What is your opinion about websites like Ancestry.com which make use of public records and charge a subscription fee for access? What is the incentive for the government to migrate old documents into digital form when services like these exist? Do you think Ancestry.com should be a 501(c)(3)?

CM: I'm not a big fan of for-profit corporations that have a business model of monetizing the public domain. I'm fine if they exist and fine if they make billions of dollars, but if they are the only game in town they've taken something that belongs to all of us and and turned it into their private property.

The government got snookered on the Ancestry.Com deal. They could have insisted that the raw data be available in bulk for anybody else to use. The folks that approach the government to cut these sweetheart deals argue that is unreasonable because they need a "return on investment" and the argue that if they don't get the return on investment they won't do the deal (and by extension nobody else will do the deal).

But, government can argue much harder! For example, instead of negotiating some exclusive thing with Ancestry.Com, how come they didn't ask the Internet Archive to grab the data? Or put together something creative with a couple of foundations that would pay for the digitization in return for the kind of payback the foundations like to see (e.g., good press, photo opportunity with the President, or other tools of the trade)?

You asked if Ancestry.Com should be a 501(c)(3)? Not all nonprofits do something that I think which should be an essential part of their mission, which is allow others to compete with them. I believe providing open access to all data ought to be a precondition to getting nonprofit status (an idea that Gil Elbaz has been pushing for quite some time). A good example of a nonprofit that builds walls is Guidestar which wants to be the place where you go for all your nonprofit information. The IRS should be making all Form 990 returns of nonprofits available in bulk for anybody to use, which would knock the bottom out of Guidestar's attempts to build walls and force them to stay innovative and provide value.

Pacer Problems
by onyxruby

How much difficulty do you anticipate in getting and publishing records in Pacer? If there's one system that should be free it the decisions that our courts make and yet you are charged by the page just to view the results. Are you concerned about a court taking an unkind view on your archiving what is in Pacer?

CM: PACER is an abomination. Do they take a dim view of our efforts? Well, the Administrative Office of the U.S. Courts reacted so strongly to our efforts to make their data available that they called the FBI on Aaron Swartz and cancelled the only meaningful public access system they had, which consisted of one terminal in each of 17 public libraries around the country. In this era of rapidly decreasing costs, they just boosted their access charges from 8 cents a page to 10 cents a page, arguing that this is a bargain compared to 25 cents a page for a copy machine.

What I find so disturbing about PACER is that when we did get 20 million pages of docs, we were able to conduct a comprehensive analysis of privacy violations in the courts, an analysis that led to a nice thank-you letter from the Judicial Conference and changes in their privacy rules. In other words, only when public interest groups got access to the data did we begin to address privacy issues. Public access is not just about pro se prisoners defending themselves from a jail cell, which is the view of many in the Administrative Office of the Courts. Public access is about attempts like ours (and many other folks) to make our system of justice function better. When we say we are "an empire of laws not a nation of men" that means we write down what we are doing in our courts so that it is no longer the arbitrary decisions of individuals. The paper trail is there so we can make sure the system is functioning properly. When you limit that access to those that only have a Gold Card, you pervert democracy and you pervert justice.

This principle that access to justice shouldn't hide behind a cash register goes back to the Greeks. Theseus in Euripedes' Suppliants said "when there are no public laws, one man holds power by keeping the law all for himself, and there is no more equality. But when the laws are written, the weak man and the rich man have equal justice." The PACER system is justice for the rich man.

Steve Schultze and the team at Princeton did a lot of the heavy lifting on this issue, including the very nice RECAPtheLaw.Org system they built. They've also done a lot of financial analysis that shows that the courts are not only recovering their costs for operating the expensive PACER system, they're making a huge profit (to the tune of $100 million/year) and using their excess profits to do things like buy big-screen TVs in direct violation of the E-Government act.

The basic problem on PACER is the Judicial Conference has delegated the issue to a few techie judges who think what they've built is something great. But, PACER is a hairball of bad PERL code and the result has not served the judges, the bar, or the American people very well. My only hope is that eventually, the Judicial Conference will see that their information technology is 30 years behind the rest of the Internet and feel ashamed at the travesty they have wrought. Until then, we have RECAP.

If you're interested in the issue, a couple of resources to look at are the PACER paper trail and a bit of a rant that I delivered at the Gov 2.0 summit.



How to visualize opened data?
by hardwarejunkie9

The amount of information you're trying to free is entirely staggering and consists, largely, of tables of numbers. These numbers are incredibly significant, but people generally can't see them.

After you free all of this information and make it available to the public (as it should be), then what? What do you expect for the public to do with these numbers? Tables of information are not nearly as useful as graphs. This data needs to be seen, but, more importantly, it needs to be understood.

Do you have any ideas for how to disseminate this information? Perhaps a team-up with someone like gapminder.org's Hans Rosling might be particularly valuable for all of us.

CM: Actually, most of the data I'm looking at is not tables of numbers, it is video, images, textual documents, technical papers, maps, and books.

But, I definitely get what you're saying and there are a lot of numbers. For example, the IRS Form 990s should be structured data instead of PDF documents, so extracting the data from the mass of paper is the initial challenge. There are lots of other examples of this kind of initial extraction, getting what were printed paper docs into structured data. There are some interesting tools, such as OCRopus which does layout analysis, but there needs to be much more. One of the reason we called for a Federal Scanning Commission is that we think there is a lot of directed R&D that could not only scale up mass digitization but could also work on the important value-added of extraction of structured data and handling some of the tricky issues like detecting the presence of Social Security Numbers.

Once you have the data, as you say, then what? I'm a big fan of the idea that the government starts by providing bulk data, then they provide an API, and then maybe they also build web sites and apps and other things along with everybody else out there. That's a 3-part hierarchy that Ed Felten and some of his students developed and it should be a law that applies to all government information systems that are externally facing.

The issue here is that all too often people look at a problem like "digitize all government information" and they want to see the whole stack of the solution from one place. But, I think you can do a layered approach and count on the fact that there is always somebody smarter out there and our job is to reduce the barriers to entry. So, how would I visualize the data? I have no idea, but I'd make damned sure that folks like Martin Wattenberg at Many Eyes and Hans Rosling at Gapminder knew the data was out there and then I'd sit back and be amazed at whatever they come up with. How's that for pushing the problem downstream?



Why is data access so hard?
by CanHasDIY

Can you provide any explanation as to why it is so difficult and cost-prohibitive to obtain records from the government, especially considering the abundance of laws requiring government compliance with requests for information (AKA "Sunshine Laws")?

Is it simply a matter of government employee ineptitude, or have you found evidence of a more nefarious rationale?

CM: I get that question a lot. Why would a member of Congress take deliberate steps to stop public hearings from being available? Why would a court administrator deliberately restrict access to public court documents? Usually the answer is, as Heinlein said, "you have attributed conditions to villainy that simply result from stupidity." When I'm explaining why something is so broken on a big government system, my usual answer is that there are a lot of people still stuck in the 1970s and 1980s, when information dissemination was really, really hard and it took men in white lab coats and computers the size of freight trains to process data. In other words, the problem with a lot of folks who are government gatekeepers is they just don't get the Internet and they don't get computers. In fact, usually when some senior bureaucrat is throwing stones at me, you can find younger staffers working for them rolling their eyes.

That's an optimistic view, and if I'm right things will get better. But, I'm often wrong on my predictions of the future. (I was the guy who saw TimBL demo the web in 1992 and thought to myself "interesting, but it won't scale.")

But, there is also some more nefarious stuff happening, often the accumulation of power by being able to cut exclusive deals with contractor buddies. If your life in government consists of receiving emissaries from Lockheed Martin, maybe you think you're making everybody happy by letting them build you a $1 billion computer system. Often, you think your problems are so unique that the $1 billion solution is the only answer.

And, in some cases, as we've seen from numerous GAO reports, Inspector General reports, Congressional hearings, and newspaper articles, there are some really evil people out there who think the public domain and the government is their personal business opportunity. Looting the federal government is the kind of civic crime that ranks right up there in my book with stealing cookies from Girl Scouts and selling fake medicines to sick people.



Who is the worst?
by TheBrez

Which government agency is the worst to get information from?

CM: I don't know who the worst are (there's a lot of competition for that slot), but the ones that piss me off the most are the ones that should know better.

Public.Resource.Org is a really small operation. I'm the only staff member. My part-time sysadmin is @mdkail who is pretty busy with his day job as CIO at NetFlix. My ISP is Jim Martin and his team at ISC who are kind of busy running the F-Root. My office net is supported by the amazing systems team at O'Reilly which rents me office space at below-market rates.

I'll grant you government would have a tough time getting that kind of help. But, I'm a one-man shop and we run the 4th most popular U.S. government video channel on YouTube, we're the source for a lot of the on-line presence of the U.S. Court of Appeals, and we've supported efforts for the U.S. Congress, the White House, and the National Archives. If we can do this out of Northern California, couldn't the vast resources of the federal government in Washington, D.C. do a whole lot better than they're doing now?

For me, my current bete noir is the U.S. Congress. We got half-way through processing their archives of video from congressional hearings, publishing about 31 terabytes of data. Then, a couple of staffers decided this was a bad idea and pulled the rug out from under us. They actually decided it was a bad idea to publish video from public congressional hearings.

Like any agency, Congress is a mixed bag. We had tons of support from Darrell Issa, for example, and ran a very successful pilot project for him for a year. We talked to all sorts of people on committees and in the various agencies that support the Congress. But, at the end of the day, a couple of staff members were able to decide that the public archive shouldn't be public and they terminated our project. (If you have some time, you might like to read our rather surreal paper trail.)

So, rather than the worst, I think we need to look for the most shameful, the ones that have the privilege and the power and could easily do better. I know it is in vogue to throw stones at government in general and Washington in particular, but there are times when government can be so useful and so awe inspiring it takes your breath away. Government can be that shining city on the hill but we all have to take an active part in our government to keep those lights shining bright.

New submitter libertyernie writes "Although FileSonic has disabled sharing and Uploaded.to has blocked access to the U.S., the CEO of Texas-based MediaFire is not concerned about government action against his company. 'We don't have a business built on copyright infringement,' says Derek Labian. 'Like many other cloud-based sharing services like Box.net and Dropbox, we're a legitimate business targeting professionals.'"

snydeq writes "IT fight club, dirty dev data, meatball sandwiches — InfoWorld offers nine more tales of brain fail beyond belief. 'You'd think we'd run out of them, but technology simply hasn't advanced enough to take boneheaded users out of the daily equation that is the IT admin's life. Whether it's clueless users, evil admins, or just completely bad luck, Mr. Murphy has the IT department pinned in his sights — and there's no escaping the heartache, headaches, hassles, and hilarity of cluelessness run amok.'"

Last week Timothy Lord looked at the Tesla Model S. He also took a quick look at the CODA electric car. Like Tesla, CODA is based in California. Like Tesla, CODA is building purely electric, "plug-in" cars. But unlike Tesla, CODA is making a bland but practical sedan that can go up to 150 miles on a charge and costs about $37,000. That's not exactly a Kia-competitive price, even though Tim says it looks kind of like a Kia. But it's 100% electric and costs less than a Tesla -- really, hardly more than a Nissan Leaf. And it has a fully-usable back seat and a decent-sized trunk. And unlike the Nissan Leaf, it's made right here in the good old USA.

bs0d3 writes "Not so long ago, a legal video was taken down by repetitive DMCA requests to YouTube. In response, Megaupload filed a lawsuit against Universal Music. This past week, Megaupload was raided by U.S. authorities and forced offline, which is costing Megaupload millions of dollars in damage. Today; while employees are in U.S. custody, Megaupload has mysteriously dropped their lawsuit against Universal Music."

bs0d3 writes "Not so long ago, a legal video was taken down by repetitive DMCA requests to YouTube. In response, Megaupload filed a lawsuit against Universal Music. This past week, Megaupload was raided by U.S. authorities and forced offline, which is costing Megaupload millions of dollars in damage. Today; while employees are in U.S. custody, Megaupload has mysteriously dropped their lawsuit against Universal Music."

Hugh Pickens writes "Not long ago, Apple boasted that its products were made in America. Today, almost all of the 70 million iPhones, 30 million iPads and 59 million other products Apple sold last year are manufactured overseas. 'It isn't just that workers are cheaper abroad,' write Charles Duhig and Keith Bradsher. 'Rather, Apple's executives believe the vast scale of overseas factories as well as the flexibility, diligence and industrial skills of foreign workers have outpaced their American counterparts so much that "Made in the U.S.A." is no longer a viable option for most Apple products.' Apple executives say that going overseas, at this point, is their only option and recount the time Apple redesigned the iPhone's screen at the last minute, forcing an assembly line overhaul. A foreman immediately roused 8,000 workers inside the company's dormitories, and then each employee was given a biscuit and a cup of tea, guided to a workstation and within half an hour started a 12-hour shift fitting glass screens into beveled frames. Within 96 hours, the plant was producing over 10,000 iPhones a day. 'The speed and flexibility is breathtaking,' says one Apple executive. 'There's no American plant that can match that.' Apple's success has benefited the U.S. economy by empowering entrepreneurs and creating jobs at companies like cellular providers and businesses shipping Apple products. But ultimately, Apple executives say curing unemployment is not Apple's job. 'We don't have an obligation to solve America's problems. Our only obligation is making the best product possible.'"

An anonymous reader writes "A few months ago, reports of battery fires from crash-tested Chevy Volts caused the National Highway Traffic Safety Administration to open an investigation into the type of batteries used in the Volt and other EVs. That investigation has now concluded, and the NHTSA says the cars are safe. 'The agency and General Motors Co. know of no fires in real-world crashes. GM and federal safety officials say they believe the fires were caused by coolant leaking from damaged plastic casing around the batteries after side-impact collisions. The coolant caused an electrical short, which sparked battery fires seven days to three weeks after the crashes. GM announced earlier this month that it will add steel plates to about 12,000 existing Volts to protect the batteries in the event of a crash.'"

Trailrunner7 writes "At the S4 security conference this week, 'Project Basecamp,' a volunteer-led security audit of leading programmable logic controllers (PLCs), performed by a team of top researchers found that decrepit hardware, buggy software and pitiful or nonexistent security features make thousands of PLCs vulnerable to trivial attacks by external hackers that could cause PLC devices to crash or run malicious code. 'We were looking for a Firesheep moment in PLC security,' Peterson told the audience of ICS security experts. They got one. 'It's a blood bath mostly,' said Wightman of Digital Bond. 'Many of these devices lack basic security features.' While the results of analysis of the various PLCs varied, the researchers found significant security issues with every system they tested, with some PLCs too brittle and insecure to even tolerate security scans and probing."

An anonymous reader sends in an article discussing whether other commonly used file storage sites are in danger of being shut down now that Megaupload has been closed. Quoting: "In the wake of the crackdown on the file-sharing website Megaupload, sites offering free content-sharing, file linking and digital locker services, such as RapidShare, SoundCloud and Dropbox, could be next in the crosshair of anti-piracy authorities. ... RapidShare and MediaFire are two of the biggest services left after Megaupload's exit. However, these sites have undergone a revamp, and now ... no longer host pirated content that could lead to a permanent ban. Others in the line of fire are DropBox, iCloud and Amazon S3, which support hosting any file a user uploads. Though their intention of supporting open file-sharing is legitimate, there is really no control over the type of content being uploaded."

An anonymous reader writes "Next month at FOSDEM there will be an announcement of a fully open-source and reverse-engineered ARM Mali graphics driver for Android / Linux. This driver, according to Phoronix, is said to support OpenGL ES and other functionality from reverse engineering the official ARM Linux driver. Will this mark a change for open-source graphics drivers on ARM, just as the Radeon did for x86 Linux?"

smitty777 writes "Highlighting another unique way to use cutting edge DNA technology, the U.S. Department of Defense has a new weapon in its efforts to combat counterfeit parts: plant DNA. This article at Wired discusses how plant DNA can be used to make an almost unique code (1 in 1 trillion) for parts identification. A graphic shows some of the ways this could be done: bolts with DNA-marked coating, invisible bar codes, and fluorescing inks are some of the possible applications. In a similar but unrelated project, World Micro has a different solution to detect counterfeit items in the military that have been 'blacktopped,' where items have been re-surfaced to allow remarking."

smitty777 writes "Highlighting another unique way to use cutting edge DNA technology, the U.S. Department of Defense has a new weapon in its efforts to combat counterfeit parts: plant DNA. This article at Wired discusses how plant DNA can be used to make an almost unique code (1 in 1 trillion) for parts identification. A graphic shows some of the ways this could be done: bolts with DNA-marked coating, invisible bar codes, and fluorescing inks are some of the possible applications. In a similar but unrelated project, World Micro has a different solution to detect counterfeit items in the military that have been 'blacktopped,' where items have been re-surfaced to allow remarking."

asgard4 writes "In recent years GPUs have become powerful computing devices whose power is not only used to generate pretty graphics on screen but also to perform heavy computation jobs that were exclusively reserved for high performance super computers in the past. Considering the vast diversity and rapid development cycle of GPUs from different vendors, it is not surprising that the ecosystem of programming environments has flourished fairly quickly as well, with multiple vendors, such as NVIDIA, AMD, and Microsoft, all coming up with their own solutions on how to program GPUs for more general purpose computing (also abbreviated GPGPU) applications. With OpenCL (short for Open Computing Language) the Khronos Group provides an industry standard for programming heavily parallel, heterogeneous systems with a language to write so-called kernels in a C-like language. The OpenCL Programming Guide gives you all the necessary knowledge to get started developing high-performing, parallel applications for such systems with OpenCL 1.1." Keep reading for the rest of asgard4's review. OpenCL Programming Guide author Aaftab Munshi, Benedict R. Gaster, Timothy G. Mattson, James Fung, Dan Ginsbur pages 603 publisher Addison-Wesley Pearson Educatio rating 9/10 reviewer asgard4 ISBN 0321749642 summary A solid introduction to programming with OpenCL. The authors of the book certainly know what they are talking about. Most of them have been involved in the standardization effort that went into OpenCL. Munshi, for example, is the editor of the OpenCL specification. So all the information in the book is first-hand knowledge from experts in OpenCL. The reader is expected to be familiar with the C programming language and basic programming concepts. Some experience in parallelizing problems is a benefit but not a requirement.

The book consist of two major parts. The first part is a detailed description of the OpenCL C language and the API used by the host to control the execution of programs written in that language. The second part is comprised of various case studies that show OpenCL in action.
The authors get straight to the point in the introduction, discussing the conceptual foundations of OpenCL in detail. They explain what kernels are (basically functions that are scheduled for execution on a compute device), how the kernel execution model works, how the host manages the command queues that schedule memory transfers or kernel execution on compute devices, and the memory model.

While this first chapter is all prose, the second chapter dives right in with some code and a first HelloWorld example. The following chapters introduce more and more of the OpenCL language and API step-by-step. All API functions are described in somewhat of a reference style with a lot of detail, including possible error codes. However, the text is not a reference. There is always a good explanation with examples or short code listings, the only notable exception being chapter three, which presents the OpenCL C language. A few more examples would have made the text less dry in this chapter.

An important chapter is chapter nine on events and synchronization between multiple compute devices and the host. This chapter is important because — as any experienced parallel programmer knows — getting synchronization right is often tricky but obviously essential for correct execution of a parallel program.

An interesting feature in OpenCL is the built-in interoperability with OpenGL and, surprisingly, Direct3D. Various functions in the OpenCL API allow creating buffers from OpenGL/Direct3D objects, such as textures or vertex buffers, that can be used by an OpenCL kernel. This opens up interesting possibilities for doing a lot more work on the GPU in graphics applications, such as running a fluid simulation on the GPU in OpenCL, which directly writes its results into vertex buffers or textures to be used directly for rendering without the host CPU having to intervene.

Before delving into the case studies the book briefly discusses the embedded profile that is available for OpenCL and the standardized C++ API that the Khronos Group provides in addition to the regular OpenCL API (which is defined exclusively as C functions). The C++ API makes using some of the OpenCL objects a little bit easier and somewhat nicer.

The second part of the book contains various interesting case studies that show off what OpenCL can be used for, such as computing a sobel filter or a histogram for an image, computing FFTs, doing cloth simulation, or multiplying dense and sparse matrices. The choice and variety of case studies is definitely interesting and most will be immediately applicable to the reader when going forward developing applications using OpenCL. All the code for the examples and the case studies in the book are available for download on the book's website.

Overall, the OpenCL Programming Guide succeeds in being a great introduction to OpenCL 1.1. The book covers all of the specification and more, has an easy to read writing style and yet provides all the necessary details to be an all-encompassing guide to OpenCL. The good selection of case studies makes the book even more appealing and demonstrates what can be done with real-life OpenCL code (and also how it needs to be optimized to get the best performance out of current OpenCL platforms, such as GPUs).

Martin Ecker has been involved in real-time graphics programming for more than 15 years and works as a professional game developer for Sony Computer Entertainment America in sunny San Diego, California.

You can purchase OpenCL Programming Guide from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

asgard4 writes "In recent years GPUs have become powerful computing devices whose power is not only used to generate pretty graphics on screen but also to perform heavy computation jobs that were exclusively reserved for high performance super computers in the past. Considering the vast diversity and rapid development cycle of GPUs from different vendors, it is not surprising that the ecosystem of programming environments has flourished fairly quickly as well, with multiple vendors, such as NVIDIA, AMD, and Microsoft, all coming up with their own solutions on how to program GPUs for more general purpose computing (also abbreviated GPGPU) applications. With OpenCL (short for Open Computing Language) the Khronos Group provides an industry standard for programming heavily parallel, heterogeneous systems with a language to write so-called kernels in a C-like language. The OpenCL Programming Guide gives you all the necessary knowledge to get started developing high-performing, parallel applications for such systems with OpenCL 1.1." Keep reading for the rest of asgard4's review. OpenCL Programming Guide author Aaftab Munshi, Benedict R. Gaster, Timothy G. Mattson, James Fung, Dan Ginsbur pages 603 publisher Addison-Wesley Pearson Educatio rating 9/10 reviewer asgard4 ISBN 0321749642 summary A solid introduction to programming with OpenCL. The authors of the book certainly know what they are talking about. Most of them have been involved in the standardization effort that went into OpenCL. Munshi, for example, is the editor of the OpenCL specification. So all the information in the book is first-hand knowledge from experts in OpenCL. The reader is expected to be familiar with the C programming language and basic programming concepts. Some experience in parallelizing problems is a benefit but not a requirement.

The book consist of two major parts. The first part is a detailed description of the OpenCL C language and the API used by the host to control the execution of programs written in that language. The second part is comprised of various case studies that show OpenCL in action.
The authors get straight to the point in the introduction, discussing the conceptual foundations of OpenCL in detail. They explain what kernels are (basically functions that are scheduled for execution on a compute device), how the kernel execution model works, how the host manages the command queues that schedule memory transfers or kernel execution on compute devices, and the memory model.

While this first chapter is all prose, the second chapter dives right in with some code and a first HelloWorld example. The following chapters introduce more and more of the OpenCL language and API step-by-step. All API functions are described in somewhat of a reference style with a lot of detail, including possible error codes. However, the text is not a reference. There is always a good explanation with examples or short code listings, the only notable exception being chapter three, which presents the OpenCL C language. A few more examples would have made the text less dry in this chapter.

An important chapter is chapter nine on events and synchronization between multiple compute devices and the host. This chapter is important because — as any experienced parallel programmer knows — getting synchronization right is often tricky but obviously essential for correct execution of a parallel program.

An interesting feature in OpenCL is the built-in interoperability with OpenGL and, surprisingly, Direct3D. Various functions in the OpenCL API allow creating buffers from OpenGL/Direct3D objects, such as textures or vertex buffers, that can be used by an OpenCL kernel. This opens up interesting possibilities for doing a lot more work on the GPU in graphics applications, such as running a fluid simulation on the GPU in OpenCL, which directly writes its results into vertex buffers or textures to be used directly for rendering without the host CPU having to intervene.

Before delving into the case studies the book briefly discusses the embedded profile that is available for OpenCL and the standardized C++ API that the Khronos Group provides in addition to the regular OpenCL API (which is defined exclusively as C functions). The C++ API makes using some of the OpenCL objects a little bit easier and somewhat nicer.

The second part of the book contains various interesting case studies that show off what OpenCL can be used for, such as computing a sobel filter or a histogram for an image, computing FFTs, doing cloth simulation, or multiplying dense and sparse matrices. The choice and variety of case studies is definitely interesting and most will be immediately applicable to the reader when going forward developing applications using OpenCL. All the code for the examples and the case studies in the book are available for download on the book's website.

Overall, the OpenCL Programming Guide succeeds in being a great introduction to OpenCL 1.1. The book covers all of the specification and more, has an easy to read writing style and yet provides all the necessary details to be an all-encompassing guide to OpenCL. The good selection of case studies makes the book even more appealing and demonstrates what can be done with real-life OpenCL code (and also how it needs to be optimized to get the best performance out of current OpenCL platforms, such as GPUs).

Martin Ecker has been involved in real-time graphics programming for more than 15 years and works as a professional game developer for Sony Computer Entertainment America in sunny San Diego, California.

You can purchase OpenCL Programming Guide from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

New submitter scibri writes "Researchers working on highly pathogenic H5N1 avian influenza have said they will stop work on the virus for 60 days, to allow them to explain the importance of their work to politicians and the public. Quoting: 'Despite the positive public-health benefits these studies sought to provide, a perceived fear that the ferret-transmissible H5 HA viruses may escape from the laboratories has generated intense public debate in the media on the benefits and potential harm of this type of research. We would like to assure the public that these experiments have been conducted with appropriate regulatory oversight in secure containment facilities by highly trained and responsible personnel to minimize any risk of accidental release.'" Reader Harperdog sends in a related article arguing that we shouldn't be having a debate about the censorship of research, but rather a debate over whether the research should have been allowed in the first place.

MrSeb writes "Megaupload's shutdown poses an interesting question: What happens to all the files that were stored on the servers? XDA-Developers, for example, has more than 200,000 links to Megaupload — and this morning, they're all broken, with very little hope of them returning. What happens if a similar service, like Dropbox, gets shut down — either through bankruptcy, or federal take-down? Will you be given a chance to download your files, or helped to migrate them to another similar service? What about data stored on enterprise services like Azure or AWS — are they more safe?" And if you're interested, the full indictment against Megaupload is now available.

CSHARP123 writes "The Department of Justice launched an investigation into the 'No Poaching' agreement between Apple and Google in 2010, but details of the case were only made public for the first time yesterday. TechCrunch was the first to sift through the documents, and has uncovered some ostensibly incriminating evidence against not only Google and Apple, but Pixar, Lucasfilm, Adobe, Intel, and Intuit, as well. According to the filings from the U.S. District Court for the Northern District of California in San Jose, these companies did indeed enter 'no poach' agreements with each other, and agreed to refrain from soliciting employees. The documents also indicate they collectively sought to limit their employees' power to negotiate for higher salaries."

rysiek writes "At a meeting of Polish Government officials with Polish NGOs and business representatives it was confirmed that the European Union is poised to sign the Anti-Counterfeiting Trade Agreement as soon as January 26th. But all is not lost. The Treaty still needs to be ratified by the Euro Parliament and member states individually. The ratification vote is important, as it is an either-or vote — if not ratified there, ACTA gets rejected in its entirety. The Ministry of Administration and Digitization is not amused and has asked the Prime Minister (who promised this May to hold ACTA adoption until the kinks are worked out) to cancel the signing authorization for the time being."

First time accepted submitter Rozine writes "Ever wanted to launch your own satellite into space? Thanks to a project at the Cornell Space Science Lab, now you can. In the words of the grad student leading the project, Zac Manchester, 'What better way of showing off your uber-geek credentials than having your own spacecraft?' Zac hopes that by shrinking the size of each spacecraft and using advancements in computer and solar cell technology, satellites can follow the path of the personal computer revolution, opening up space for the masses. For small donations you will receive mementos, but for $300 and up you will get your very own satellite to be launched into space. Perfect for slashdotters and school projects everywhere!" We covered this project in its infancy back in July. I'm glad to see it gained traction.

First time accepted submitter EW87 writes "Shortly after a federal raid today brought down the file sharing service Megaupload, hackers aligned with the online collective Anonymous have shut down sites for the Department of Justice, Universal Music Group and the RIAA. 'It was in retaliation for Megaupload, as was the concurrent attack on Justice.org,' Anonymous operative Barrett Brown tells RT on Thursday afternoon."

Orome1 writes "After having first claimed that the source code leaked by Indian hacking group Dharmaraja was not stolen through a breach of its networks, but possibly by compromising the networks of a third-party entity, Symantec backpedalled and announced that the code seems to have exfiltrated during a 2006 breach of its systems. Symantec spokesman Cris Paden has confirmed that unknown hackers have managed to get their hands on the source code to the following Symantec solutions: Norton Antivirus Corporate Edition, Norton Internet Security, Norton Utilities, Norton GoBack and pcAnywhere."

itwbennett writes "Would-be cellular carrier LightSquared claims that the company's LTE network was set up to fail in GPS interference tests. 'Makers of GPS (Global Positioning System) equipment put old and incomplete GPS receivers in the test so the results would show interference, under the cover of non-disclosure agreements that prevented the public and third parties from analyzing the process,' LightSquared executives said on a conference call with reporters Wednesday morning."

Snirt writes "Following up on a story previously discussed here, it now appears Eastman Kodak, the company that invented the hand-held camera, has filed for bankruptcy protection. The move, according to Kodak's news release, gives the company time to reorganize itself without facing its creditors, and Kodak said it would mean business as normal for customers. The company has recently moved away from cameras, focusing on making printers to stem falling profits."

Hugh Pickens writes "Stargazing skies all over the world are disappearing, as the sky above New York City is Class 9 on the Bortle ranking and American suburban skies are typically Class 5, 6, or 7. But some places are making an effort to preserve their skywatching heritage as Exmoor National Park was granted International Dark-Sky Reserve status in November and people in the Exmoor town of Dulverton were challenged to switch off their lights as part of the BBC's Stargazing Live, demonstrating that you don't need special equipment to see the stars more clearly, if you have a decent pair of binoculars. 'The whole idea is to show that even a small town, which is still quite dark, can give off quite a lot of light,' says astronomer Mark Thompson. The event in Dulverton gained a lot of support from local residents and businesses. 'It needed a bit of organization to get everyone to say yes,' says town mayor Chris Nelder. 'We want people to just enjoy the night sky, to treasure the fact we have them and to look after them,' adds Claire O'Connor from Exmoor National Park Authority."

alphadogg writes "When Google announced last spring that Kansas City, Kan., had landed the tech company's much-pursued super-speed Internet project, the company gushed about the local utility poles. Now it turns out that differences over where and how to hang wires on those poles, and what fees or installation costs may be required, have created a troublesome bump in plans to launch the project."

langelgjm writes "While much of the web is focused on the SOPA and PIPA blackout, supporters of the public domain today quietly lost a protracted struggle that began back in 2001. The Supreme Court, in a 6-2 decision, rejected the argument that Congress did not have the power to convey copyright upon works that were already in the public domain. The suit was originally filed to challenge provisions that the U.S. adopted when signing the TRIPs agreement. Justices Breyer and Alito dissented, arguing that conveyed copyright on already existing works defied the logic of copyright law. Justice Kagan recused herself. The text of the opinions is available here (PDF)."