Slashdot Mirror
Finance, Scientific Users Get ActivePython Updates
jcasman sends along this clip from PCWorld: "ActiveState has added three open source mathematics libraries to its ActivePython Python distribution that might interest financial and scientific computing markets, the company announced Thursday. The packages are being added, in part, to anticipate the demand that may arise from new proposed rules for the US financial community brought about by the US Securities and Exchange Commission. ... In April, the government agency posted a set of proposed rules for handling asset-backed securities that called for financial firms to disclose, along with their prospectus filings, the source code of the programs that generated the filings, as rendered in Python. The government agency will be accepting input about the proposed rule until August 2. The three libraries that are being added to the ActivePython package are NumPy, SciPy, and matplotlib."
These are great and free tools for making publication-quality plots as well as the analysis of the data.
Okay so I'm the lead Microsoft certified developer at a Fortune 500 company. I have to get this crappy software integrated with our project that is running as a Win32 application by the end of the month and so I downloaded the packages and dragged and dropped them into Microsoft Visual Studio. Then I created a VB file that basically calls SciPy.getSECReport(somedata) and nothing happens. I get some "Error Method or data member not found" even though the stuff I downloaded was unzipped and dragged right into the rest of my libraries in my WIN32 directory.
... as well as dragging and dropping the excel files onto the packages (I'm not an idiot, I've tried everything). I even went to SourceForge to find documentation on this crap and there is nothing. Nothing! Can someone help me figure this crap out? It's impossible to use, the open source nuts have made it so it isn't streamlined and integrated with Windows. I mean, I'm a pretty talented hacker (MSCD and everything) and this just goes to show how crappy open source can be.
Oh and I also tried double clicking the packages and nothing happens
Will someone please rewrite this in Visual Basic so I can do my job?
That link is unrelated to this post. The referenced scientific/numeric libraries for python are implemented in C as native modules. They are not just as fast as the equivalent code written in C, they *ARE* the equivalent code written in C, merely interfaced to with python. You might lose a meaningless tiny fraction of time preparing your vectors for processing in python, but you'll save several orders of magnitude more time not worrying so much about malloc corruption.
I read the script, and I think it would help my character's motivation if he was on fire. -Bender
For about an average of half the lines of code they might use in C, scientists ...
Scientists?! You're probably joking but I've been over to accounting and they're using Excel and *shudder* Access for all their heavy data lifting. Sometimes they need help and if they're kind enough I don't lie about how much I know about those ancient products. Most importantly they're not scientists. They're accountants and business people ... they don't care if they have to wait five minutes for Excel to open a worksheet containing the entire set of order histories of our company.
...
I don't think these packages are intended for NASA space mission flight certified calculations. Just something to really help you out if you want to comply with the SEC. Side bonus, I'll bet that when you submit this code, you're going to achieve compliance a lot faster when the SEC only has to check half the lines of code and not analyze your memory management
Software development is about trade-offs. Why aren't you complaining that it's not in some targeted processor specific machine language?
My work here is dung.
Oops - that's embarassing. I just realized that the Python program must be those used to produce the actual filings. So the programs' fitness to purpose must have been already established. Presumably they don't loop forever, or at least only do so after producing the filings.
I know bonafide scientists who use Excel for analysis. Scary.
I also know quite a few who use numpy, scipy, matplotlib, and python for real science. In fact, I think the astro community specifically was an early supporter of some of these packages. I myself have been using them in science for 5 or 6 years.
From http://www.sec.gov/rules/proposed/2010/33-9117.pdf (p 205-206)
This proposed requirement is designed to make it easier for an investor to conduct a thorough investment analysis of the ABS offering at the time of its initial investment decision. In addition, an investor may monitor ongoing performance of purchased ABS by updating its investment analysis from time to time to reflect updated asset performance.338 In this way, market participants would be able to conduct their own evaluations of ABS and may be less dependent on the analysis of third parties such as credit rating agencies.
The waterfall is a critical component of an ABS. Currently investors receive only a textual description of this information in the prospectus, which may make it difficult for them to perform a rigorous quantitative analysis of the ABS.339
Actually, NASA JPL does most of their mission design work using Python (combining compiled C modules of course, just like NumPy and SciPy).
Those packages are fantastic and really 90% of what I use in python are in those packages. I have been using enthought edition python rather than active-state (many reasons), and this tips the scales a bit more toward recommending active-state to others.
FYI: Matplotlib makes 2D and 3D presentation quality plots of data (even an absurd quantity of data). Numpy and scipy provide scientific and matrix functions that pretty much cut matlab off at the knees unless you are a simulink user. Matlab is many thousands of dollars, python is free, and they are both remarkable similar, except matlab chokes on large data sets where python doesn't.
Sheldon
It is only irrelevant if all the computations you are performing are done by the libraries. I tried using python for data processing tasks, and it was unbearably slow despite the use of scipy. I think it was due in large part to poor I/O and bit-twiddling performance while reading/writing data files but I'm not sure. Anyway with the amount of time I spent optimizing code, I could have just written the damn thing in C to begin with. I just don't understand python's poor performance. All it's high level language features were implemented in LISP and ML decades ago with good performance, but python just can't seem to get it right
I don't think you understand. These things are used as any other python code. The magic is that C is the backend that actually crunches the numbers. You don't even have to know or care that C is used. All you know is you call the foo method in the bar module.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Speaking as someone who uses both R and Python all the time, I'd say that while R is very very good at a lot of what it does, it's just not as good as a general-purpose language as Python. I find myself doing as much preprocessing in Python as possible, then saving the results in DB tables and having R finish the analysis. And yes, I know about RPy, but the programming overhead of representing data structures in both languages, and making sure that they're talking to each other correctly, can be considerable; so is the runtime overhead of passing really big data sets back and forth. (Note that it's been a while since I used RPy for anything big, and a lot has been improved in that time, so it may be time for me to give it another shot.) Python code is just cleaner and easier to write for most tasks. I like both languages for their strengths, but overall, if you can do a particular analysis in Python then that's usually the easier choice.
The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
TFA and TFS fail to mention that SciPy, Numpy and Matplotlib have been added only to the Business, Enterprise, and OEM Editions of ActivePython. The Community Edition (the only one that's free) doesn't contain these libraries.
http://www.activestate.com/activepython
Or the loop was part of the design.
I leave my machine on overnight and lots of things are looping "endlessly" - and that's not a problem.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Mod parent up!
I also use both, quite heavily. Rpy2 is a real improvement over RPy, but I share your preference for allocating tasks to the two languages according to their strengths, and the rpy/rpy2 data structure representations are much trickier than I like.
I find myself gravitating to R when the builtins are useful (robust estimation, smoothing kernels), where I need to split and analyze data subsets (using by() and its cousins) or where I want to plot things. Python is the ticket for interfacing with C, talking to the outside world, parsing, and expression of simple mathematical models.
You know, it is still possible to write bad code in Python... I use these packages on a daily basis, and while I wouldn't use them for production-level numerical weather prediction system, I do use it for data processing and such. There have been times when I wrote the code poorly and the code ran slowly, but as I learned Python and embraced the language (weaning myself away from C), I found myself writing faster programs. Which high-level language features do you think are lacking? I particularly like list comprehensions and the function argument syntax (positional and keyword-based).
but while I was programming an important app, I accidently hit the space bar just before tabbing. Since this error wasn't visible on printouts or screen views, [...]
What, the glaring "IndentationError" exception that gets thrown as soon as you import the file didn't tip you off?
The situation you describe can never happen silently. I call bull$!7. Theoretically, it is possible to construct a situation where you would get a silent inconsistency, but this isn't it. I've programmed extensively in Python since '97 and never experienced problems due to indentation. In real life, this just isn't a problem.
But you can't blame them, I mean what sort of idiot language has whitespace signify blocks of code?
Well.... all of them do. How else would you find the blocks in a program? I know that many languages also use special tokens like { and } or "begin" and "end", but programmers still use the indentation to identify blocks even when it contradicts the tokens. The canonical example is something like:
if (some_test())
....i = foo();
....bar(i);
baz();
Perfectly valid code, so no compiler errors etc., but most programmers will read it as a an if-statement with a true-block containing two statements... (I had to use dots to get indentation since pre-tags do not seem to be understood...?!)