Slashdot Mirror

itwbennett writes "DARPA (the U.S. Defense Advanced Research Projects Agency) has awarded $3 million to software provider Continuum Analytics to help fund the development of Python's data processing and visualization capabilities for big data jobs. The money will go toward developing new techniques for data analysis and for visually portraying large, multi-dimensional data sets. The work aims to extend beyond the capabilities offered by the NumPy and SciPy Python libraries, which are widely used by programmers for mathematical and scientific calculations, respectively. The work is part of DARPA's XData research program, a four-year, $100 million effort to give the Defense Department and other U.S. government agencies tools to work with large amounts of sensor data and other forms of big data."

concealment writes in with an interview with a creator of the (fairly) new language Julia designed for number crunching. Quoting Infoworld: "InfoWorld: When you say technical computing, to what type of applications are you specifically referring? Karpinski: It's a broad category, but it's pretty much anything that involves a lot of number-crunching. In my own background, I've done a lot of linear algebra but a fair amount of statistics as well. The tool of choice for linear algebra tends to be Matlab. The tool of choice for statistics tends to be R, and I've used both of those a great deal. But they're not really interchangeable. If you want to do statistics in Matlab, it's frustrating. If you want to do linear algebra in R, it's frustrating. InfoWorld: So you developed Julia with the intent to make it easier to build technical applications? Karpinski: Yes. The idea is that it should be extremely high productivity. To that end, it's a dynamic language, so it's relatively easy to program, and it's got a very simple programming model. But it has extremely high performance, which cuts out [the need for] a third language [C], which is often [used] to get performance in any of these other languages. I should also mention NumPy, which is a contender for these areas. For Matlab, R, and NumPy, for all of these options, you need to at some point drop down into C to get performance. One of our goals explicitly is to have sufficiently good performance in Julia that you'd never have to drop down into C." The language implementation is licensed under the GPL. Lambda the Ultimate has a bit of commentary on the language, and an R programmer gives his two cents on the language.

aceydacey writes "Sometimes a picture is worth a thousand words. Beginning Python Visualization: Creating Visual Transformation Scripts, published in February 2009 by Apress, shows how Python and its related tools can be used to easily and effectively turn raw data into visual representations that communicate effectively. The author is Shai Vaingast, a professional engineer and engineering manager who needed to train scientists and engineers to do this kind of programming work. He was looking for a tutorial and reference work, and unable to find a suitable text, wound up writing his first book. He writes in the easy and clear style of someone comfortable and engaged with the subject matter." Keep reading for the rest of aceydacey's review. Beginning Python Visualization: Crafting Visual Transformation Scripts author Shai Vaingast pages 363 publisher Apress rating 9/10 reviewer aceydacey ISBN 1430218436 summary learn how to process, organize, and visualize data from various sources using the Python language The book uses several very specific examples that illustrate general principles.

The first example is using GPS data. By using Python one can extract data from GPS receivers and enter it into the computer and manipulate it to do what one wants including creating graphs and charts. In this section he shows how to use CSV, comma separated values, as a most useful file format. He shows show to extract data from real world GPS devices and import it via serial ports and the PySerial module. It would be easy for the reader to duplicate and extend this project.

The heart of the book is coverage of useful examples utilizing MatPlotLib, NumPy and SciPy. These related tools are easy to use and fully integrated with Python. MatPlotLib is for plotting data and graphs, including interactive graphs and image files. NumPy is a powerful math library comparable to commercial tools like MatLab, and SciPy extends NumPy to for the sciences. Examples are numerous and include signal analysis using Fourier transforms.

There is also a section on Image Processing using PIL, the Python Imaging Library. This is used for relatively simple image cropping and sizing and also for bit by bit image processing. Interpolation and curve fitting are also well covered. For anyone wanting an introduction to graphical analysis of statistical data, this would be an excellent resource.

The author is obviously a professional in this field. He has a knack for good organizational style and a pragmatic approach to the work. In the book he says "Most of the time, research is organized chaos. The emphasis, however, should be on organized, not chaos." A real value I got from the book is a better understanding of data files, format, and organization as well as methods and guidelines for selecting file formats and storing and organizing data to enable fast and efficient data processing. It is obvious that this book was written by a practicing engineer.

The theme of the book is that Python can be an all purpose environment for data manipulation and visualization, using nothing but free and open source tools that are easily integrated and scriptable without using multiple programming languages. The book should be an invaluable tool for scientists and engineers but it is also easily accessible to anyone interested in math and data analysis. There is no need for an advanced math background. While, as a matter of full disclosure, I have undergraduate degrees in Math and Physics, I feel the book should be easily accessible to anyone with a solid high school math background who is seriously interested in the subject. The book contains a short introductory tutorial on the basics of Python so anyone familiar with programming in any language should be fine.

The book is an easy read from front to back, and I am sure it will also be a good reference resource for the future. The writing style is very clear and unforced and I found surprisingly few errors. While the Python world has a surplus of introductory and general books, books covering this kind of specific domain are especially welcome, and we could use more on other topics by competent authors.

At 363 pages the book is a surprisingly fast read. Its methodology is to use specific, short code examples to make all the key points. Most of the code samples are well selected, short and written in clear, concise Python. This is not the kind of book that overwhelms you with massive amounts of code. Either the book was well edited or else it was written by an exceptionally lucid thinker, or both.

So, if you want to learn how to process, organize, and visualize data from various sources using the Python language, I recommend this book to you. I have also posted a podcast of an interview with the author at Python411

You can purchase Beginning Python Visualization: Crafting Visual Transformation Scripts from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

aceydacey writes "Sometimes a picture is worth a thousand words. Beginning Python Visualization: Creating Visual Transformation Scripts, published in February 2009 by Apress, shows how Python and its related tools can be used to easily and effectively turn raw data into visual representations that communicate effectively. The author is Shai Vaingast, a professional engineer and engineering manager who needed to train scientists and engineers to do this kind of programming work. He was looking for a tutorial and reference work, and unable to find a suitable text, wound up writing his first book. He writes in the easy and clear style of someone comfortable and engaged with the subject matter." Keep reading for the rest of aceydacey's review. Beginning Python Visualization: Crafting Visual Transformation Scripts author Shai Vaingast pages 363 publisher Apress rating 9/10 reviewer aceydacey ISBN 1430218436 summary learn how to process, organize, and visualize data from various sources using the Python language The book uses several very specific examples that illustrate general principles.

The first example is using GPS data. By using Python one can extract data from GPS receivers and enter it into the computer and manipulate it to do what one wants including creating graphs and charts. In this section he shows how to use CSV, comma separated values, as a most useful file format. He shows show to extract data from real world GPS devices and import it via serial ports and the PySerial module. It would be easy for the reader to duplicate and extend this project.

The heart of the book is coverage of useful examples utilizing MatPlotLib, NumPy and SciPy. These related tools are easy to use and fully integrated with Python. MatPlotLib is for plotting data and graphs, including interactive graphs and image files. NumPy is a powerful math library comparable to commercial tools like MatLab, and SciPy extends NumPy to for the sciences. Examples are numerous and include signal analysis using Fourier transforms.

There is also a section on Image Processing using PIL, the Python Imaging Library. This is used for relatively simple image cropping and sizing and also for bit by bit image processing. Interpolation and curve fitting are also well covered. For anyone wanting an introduction to graphical analysis of statistical data, this would be an excellent resource.

The author is obviously a professional in this field. He has a knack for good organizational style and a pragmatic approach to the work. In the book he says "Most of the time, research is organized chaos. The emphasis, however, should be on organized, not chaos." A real value I got from the book is a better understanding of data files, format, and organization as well as methods and guidelines for selecting file formats and storing and organizing data to enable fast and efficient data processing. It is obvious that this book was written by a practicing engineer.

The theme of the book is that Python can be an all purpose environment for data manipulation and visualization, using nothing but free and open source tools that are easily integrated and scriptable without using multiple programming languages. The book should be an invaluable tool for scientists and engineers but it is also easily accessible to anyone interested in math and data analysis. There is no need for an advanced math background. While, as a matter of full disclosure, I have undergraduate degrees in Math and Physics, I feel the book should be easily accessible to anyone with a solid high school math background who is seriously interested in the subject. The book contains a short introductory tutorial on the basics of Python so anyone familiar with programming in any language should be fine.

The book is an easy read from front to back, and I am sure it will also be a good reference resource for the future. The writing style is very clear and unforced and I found surprisingly few errors. While the Python world has a surplus of introductory and general books, books covering this kind of specific domain are especially welcome, and we could use more on other topics by competent authors.

At 363 pages the book is a surprisingly fast read. Its methodology is to use specific, short code examples to make all the key points. Most of the code samples are well selected, short and written in clear, concise Python. This is not the kind of book that overwhelms you with massive amounts of code. Either the book was well edited or else it was written by an exceptionally lucid thinker, or both.

So, if you want to learn how to process, organize, and visualize data from various sources using the Python language, I recommend this book to you. I have also posted a podcast of an interview with the author at Python411

You can purchase Beginning Python Visualization: Crafting Visual Transformation Scripts from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

leighklotz writes "Jeff Breidenbach and 200 lines of Python code have brought us the Glyphsaw Puzzle solver. Hold a puzzle piece up to a webcam, and the display sgiws exactly where in the puzzle the piece belongs. The solver uses the Python Imaging Library (PIL), Numerical Python, and the PARC DataGlyph Toolkit. By the way, you can make your own DataGlyphs."