Slashdot Mirror

and due in no small part to pandas. link

if used correctly (due to its NumPy foundation) with vectorised operations it rocks

Also, how in god's name do you consider that easier to read and simpler than https://numba.pydata.org/?

R is definitely still ahead for data modeling, but Python has some advantages too. With a bigger set of modules (libraries) to choose from and high popularity in the financial sector, there are big improvements all the time. For the purposes of this discussion, the most important Python modules are:
IPython: powerful interactive shell
numpy and scipy: numerical, matrix, and scientific functions (matlab-ish)
pandas: R-like data structures and data analysis tools (analysis mostly limited to regression)
statsmodels: statistical analysis, complements pandas
sk-learn: machine learning

So can Python do everything that R can? No. Or, at least, not as easily. But it is improving in that direction quite quickly, and if Python's data analysis capability meets your needs, then you can likely do everything in one language instead of calling R routines from another.

Python and R are sort-of converging via Pandas. I'm partial to Python, but Pandas really starts to blur the lines conceptually.

I've gotten a lot of mileage out of Python for cleaning and pre-processing CSV and JSON datasets, using the obviously named "csv" and "json" modules. ... However, if you are doing very much manipulation of tabular data, I'd recommend learning a bit of SQL too.

You may want to look into pandas as a middle ground. It's great for sucking in tabular or csv data and then applying statistical analysis tools to it. It has a native "dataframe" object which is similar to database tables, and has efficient merge, join, and groupby semantics. If you have a ton of data then a database and SQL is the right answer, but for a decent range of use cases in between pandas is extremely powerful and effective.

Come on, you haven't looked at Python for this kind of work have you?

R as a lot more libraries, that cover specific needs, but the basics (and more) are all covered in Python, and are extremely easy to use.

You need: Python + Numpy (or Scipy) + statsmodels + Panda.
Or get a pre-packaged distribution that has all that, like Anaconda or Enthougt (haven't used them).

As for your question, you need Pandas, which is similar to R's data tables: http://pandas.pydata.org/

import pandas as pd
df = pd.read_csv('foo.csv', header=[0], sep=',')

Pretty similar.

I think your friend is mistaken.

Why? When you use SciPy to do number crunching, the heavy lifting is being done by compiled FORTRAN and C code. That's not slow.

The genetics guys might have their own special libraries that do things they need, and again Python can wrap those.

We aren't talking about starting over from scratch and writing your own scientific library in pure Python. That would be glacially slow (unless, maybe, you used PyPy to run it... PyPy is amazing). Plus, why throw away good libraries that have already been debugged?

Perl and Python are used more for glue code,

Right: SciPy is a bunch of FORTRAN and C code glued together in Python.

R is used heavily for statistics

I'm not a stats guy or a science guy, but from what I have heard, Python with the Pandas library is a good replacement for R. I hear that R is very good at its particular problem domain, but not so good for general-purpose stuff; while Python with Pandas is pretty good at the same things R is good at, while still being a very usable all-around language.

http://pandas.pydata.org/

Since you mention VBA, I suspect that your data is in Excel spreadsheets? If you want to try to speed this up with minimum effort, then consider using Python with Pyvot to access the data, and then numpy/scipy/pandas to do whatever processing you need. This should give you a significant perf boost without the need to significantly rearchitecture everything or change your workflow much.

In addition, using Python this way gives you the ability to use IPython to work with your data in interactive mode - it's kinda like a scientific Python REPL, with graphing etc.

If you want an IDE that can connect all these together, try Python Tools for Visual Studio. This will give you a good general IDE experience (editing with code completion, debugging, profiling etc), and also comes with an integrated IPython console. This way you can write your code in the full-fledged code editor, and then quickly send select pieces of it to the REPL for evaluation, to test it as you write it.

(Full disclosure: I am a developer on the PTVS team)

Install these 2 and you'll be good to go
http://ipython.org/notebook.html
http://pandas.pydata.org/

It's not exactly a spreadsheet, but Pandas is totally awesome and is useful for many tasks for which you might think of using a spreadsheet.

pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. It is already well on its way toward this goal.

http://pandas.pydata.org/index.html

IPython Notebook is sort of like a combination of the normal ipython shell and an IDE. You interact via your browser but it connects to a normal python process on your local (or remote?) system.

http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html

I've used these tools together for many tasks for which I might otherwise have used a spreadsheet, particularly for "pivot tables" and time series analysis. Again, even combined they do not a spreadsheet make, but they are in many ways superior. They can handle very large data sets, and best of all you are doing it all in Python.

Luckily all of the XData funded work is required to be open source. This looks like the numpy work: http://blaze.pydata.org/

Wow, I hope not. As much as I am actually a Ruby fan at heart; and as much as I appreciate the R community and everything R has done, it always seems much easier to write slow and/or memory-intensive R code than in Python. Perhaps I never quite spent enough time with it but there are many corners to the language which seem unnecessarily tedious. And no references - variables are all copied around the place, which is expensive. I know, I know... worrying about pass-by-value and efficiency of assignment statements (well, R doesn't really have statements; everything-is-an-expression) means I'm doing it wrong, but most code I debug is written by someone else who is also doing it wrong..

Then there's pandas and the rest of the SciPy stack, which is the only reason I used Python over Ruby (I had also considered Perl+Moose) in my last project. pandas is extremely fast, and I was able to write some quite advanced data processing stuff which would normally have needed far more effort in Ruby or Perl.

Pandas http://pandas.pydata.org/ is another great tool for data analysis. It use numpy and is highly optimized with critical code paths which is written in C.