Slashdot Mirror
Ask Slashdot: Statistical Analysis Packages For Libraries?
HolyLime writes "I'm a librarian in a small academic library. Increasingly the administration is asking our department to collect data on various aspects of our activities, class taught, students helped, circulation, collection development, and so on. This is generating a large stream of data that is making it difficult, and time consuming, to qualitatively analyze. For anything complicated, I currently use excel, or an analogous spreadsheet program. I am aware of statistical analysis programs, like SPSS or SAS. Can anyone give me recommendations for statistical analysis programs? I also place emphasis on anything that is open source and easy to implement since it will allow me to bypass the convoluted purchase approval process."
R is my personal favorite but you're going to have to get down and dirty with some high level programming (scripting). Check out the data import package (you would probably export your spreadsheets to flat txt files and import although the functionality is ever increasing). There's no user interface in this suggestion ... what there is, however, is a massive collection of packages for statistical analysis. Very well maintained, constantly updated and ever expanding.
The other suggestion has a better GUI but is really heavyweight. WEKA has helped me time and time again perform advanced statistical calculations on data sets and it's in Java so runs on just about anything. Their interface occasionally improves too, they now have an explorer that I use to prep data and remove outliers/null data (don't worry, this isn't climate data). It's well documented.
These (probably) require an intermediate data transformation step but are open source and extensively supported. Any examples of what you wanted to do? Simple stuff like standard deviation or complex stuff like principle component analysis (PCA)? I guess if it was just simple stuff, that'd be built into Excel, right? Maybe your problems are simple enough to just need a good macro writer to tackle? Whatever happens, good luck!
My work here is dung.
I find that libraries carry a lot of common information and not so much uncommon information. This sort of muckery seems to encourage concentration of information into a smaller and smaller realm, constantly sorting out first the never-used, then the minimally-used, to maximize volume of return but minimize the use of the library as a haven for obscure and long-forgotten knowledge. Effectively, like burning some books while not burning other books--removes knowledge.
As with all things, there must be balance. A library where you don't increase holding of more useful texts is less immediately useful; although if you removed all the most used texts, you would have an interesting outcome... the obscure and oft-overlooked need retention, too.
Support my political activism on Patreon.
Try this giant list
From personal experience, I can recommend WINKS. It's ridiculously easy to use.
Required reading for internet skeptics
Sage (formerly SAGE?) is an open source mathematical package that includes statistical functions. I wanted to add that to the usual mentions of R, etc.
However, are you sure this is what you want? It sounds to me like your real problem is that you have too much data to store. If you're currently using Excel to process your data, and it has been working except that you are running out of space, perhaps what you really need is a database, like Access. If you want OSS, you can probably try LibreOffice, or engage a local student to design a web based system based on MySQL.
"Here Lies Philip J. Fry, named for his uncle, to carry on his spirit"
SAS is a great package but is probably prohibitively expensive. An open source version like R is probably more appropriate.
Hear me out. We deal with about 3 million data-producing elements and track in real-time to near-real-time. We ingest everything into MySQL (via macros, scripts, tools, etc.) and normalize the data on the way in. For analysis we simply query. Those queries may have their outcome displayed in a simple report generator, or (more often than not) via HTML5 Canvas graphs/charts, Cacti graphs, etc. What we're doing doesn't lend itself well to a SAS type solution. If you could use SAS for what you're doing, this probably wouldn't work for you.
Look at the free SPSS work-alike PSPP. http://www.gnu.org/software/pspp/ Sounds like R might be a bit much for your needs.
Depending on the type of "analysis" you might be better off with something like PowerPivot. There's alot that you can probably gleen from your data without doing sophisticated statistics, but instead using PowerPivot to slice/dice/summarize/chart your data in different ways. It is easiest to use if you structure your data in a data warehouse/star schema fashion.
Blue skying the toolset is not gonna work. What output do they want, then figure out what tools can generate that output.
If the most important thing is inserting pretty graphs into newsletters, thats one thing.
If the most important thing is hard core data warehousing analysis (for a library?) thats another thing.
The other thing is what answer do they want? They're just looking for data to back up an unpopular decision or glorify themselves demonstrating their amazing management talents. So figure out what that is (by asking them?) and help them get the data they want. Don't give them a graph of declining circulation if they're trying to emphasize their brilliant leadership. Don't give them a graph of increasing student help, if they're trying to justify downsizing.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Seriously, stick with Excel. You and anyone who comes after you would need to learn whatever statistical package you introduce. That is either overkill for the kind of data you're collecting and analysing, or it's a full time job requiring specialist knowledge for which they should be hiring someone else.
Excel has a few bugs but for the most part it's very capable. Ensure you run the service packs and can install the addons that come with it (analysis pack). Get them to send you on advanced short courses for Excel and Statistics. If there isn't that kind of commitment there's no room for any statistical package.
Almost all ask slashdot stories that are work related can be answered the same way - bad idea: you're already out of your depth and if you can't be bothered to google for the information the project is doomed.
These posts express my own personal views, not those of my employer
I have grown accustomed to doing statistical analysis using Python and R using http://rpy.sourceforge.net/rpy2
I also place emphasis on anything that is open source and easy to implement since it will allow me to bypass the convoluted purchase approval process.
Sorry to burst your bubble, but if you want good support and easy implementation, you have to look for normal paid-for solutions. Besides, open source is not synonym for free. This is especially true with specialized software or something you want good support for. Open source just means you get the code aswell, so you can implement your own additions (without use of plugins) or change it.
Your point may be valid. But what would really help your validity is mentioning some proprietary products that beat R and WEKA at their own game. Sure, I've used Matlab and it can't be beat in some respects and is heavily supported. But to suggest that just because it effortlessly interfaces with Excel spreadsheets when the person could get by with a simple export in Excel to run their R script on the resulting files? Not worth the cash, in my opinion. I don't go out and buy every piece of software to evaluate it, though. I'm aware of Matlab and Mathematica and have used them quite a bit ... but I still prefer R and WEKA. So, CmdrPony, go ahead and list all the proprietary point-and-click-omg-it-just-works software for our friend here. We're all waiting.
But unless you get an product from a company that is spending money to develop it, you never get good software and good support.
Say, friendo, have you ever heard of Linux? Eclipse? Audacity? PostGRES? VLC?
No one can make both because everything in this world costs money, and developers have to live too. Open source and free software model works well for the likes of Google and Firefox because the developments get paid by money made with advertising. Statistical analysis software, and other specialized software is a different matter.
Can you tell me what advertising model is employed to funnel money through Firefox into Google? I mean, Google makes a competing product called Chrome -- the rendering engines are even different! What in the world are you free basing?
My work here is dung.
Check out PDL (Perl Data Language). It may not be the most convenient solution but it's free and has a great, informed and responsive user group.
"player 4 hit player 1 with 0 stroms"
Anyone with decent recommendations, aside from R's own website, where to do a quickstart when you're a SAS geek?
This blog explains some of the stuff you do in R and as he does it, he compares it to SAS.
Example:
Unlike SAS, which has DATA and PROC steps, R has data structures (vectors, matrices, arrays, dataframes) that you can operate on through functions that perform statistical analyses and create graphs. In this way, R is similar to PROC IML.
And here's an entire book on the topic (although may be difficult to find)!
My work here is dung.
It almost seems like you are not doing statistics as much as creating reports from data.
Maybe you should be using a database instead of a spreadsheet or a statistics program.
The Uber geek way would be to set up a LAMP server and create a webased system.
The more convent way would be something like Access.
You can then use Excel to manipulate the data as needed or the database program.
In the end if you know excel you may want to stick with it. I see people use Excel for databases all the time. Drives me a bit nuts but sometimes what ever works is just fine.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
R
There is nothing that beats it on any platform. Some links:
http://www.sr.bham.ac.uk/~ajrs/R/r-gallery.html
http://addictedtor.free.fr/graphiques/index.php
http://opencpu.org/
https://r-forge.r-project.org/
http://hlplab.wordpress.com/
http://rseek.org/
http://www.r-bloggers.com/
Excel and other spreadsheets suck at stats:
* Burns, P. (2005). Spreadsheet Addiction.
* Cryer, J. (2001). Problems with using Microsoft Excel for StatisticsPDF.
* Pottel, H. (n.d.). Statistical flaws in Excel. PDF
* Practical Stats (n.d.), Is Microsoft Excel an Adequate Statistics Package?
* Heiser, D. (2008). Errors, faults and fixes for Excel statistical functions and routines
For a more comprehensive and technical discussion, see the papers by Yu (2008); Yalta (2008); and McCullough & Heiser in Computational Statistics and Data Analysis 52(10).
I suggest you post your question to the code4lib mailing list. It's going to get you much more informed and practical advice. You might even find some people who already have a good workflow who will share their tools.
-Esme
Depending on how large your dataset is, you may have luck using Matlab (or the opensource gnu octave). These programs will let you do *whatever* you want with the data (plotting, correlation, fft, etc).
With at least Matlab, there are some MySQL plugins available that will let you get data out of your database and into arrays rather quickly. And of course, both matlab and gnu octave let you import csv and plaintext datafiles.
Here is the matlab plugin I have used very successfully (and it's open source. No idea if it would work with octave):
http://www.cims.nyu.edu/~almgren/mysql/
You will need some background with math, statistics, and programming to effectively do this. If you don't have the skills, learn them or pay up for some overpriced commercial product...
I love R, but if you want something that looks more like SPSS, you could try the free SPSS clone PSPP:
http://www.gnu.org/software/pspp/
http://cran.r-project.org/web/packages/RMySQL/
if a full stats package is a bit heavy, try python + http://www.scipy.org/
below is using the ipython shell
In [1]: import scipy
In [2]: x = [1,3,6,8,9,4,9,0,5,3,6,8,6,8]
In [3]: scipy.mean(x)
Out[3]: 5.4285714285714288
In [4]: scipy.std(x)
Out[4]: 2.7957693986829897
and if you need more than that you can really delve into its stats submodule http://www.scipy.org/doc/api_docs/SciPy.stats.html.
What is your ILS? Depending on what it is, you may already have access to just about all of what you need there along with Excel. Atriuum from Booksys has wonderful features like you are asking about, record tracking, and it exports to Excel very well. Voyager from Ex Libris had wonderful integration with Access and my boss could pull out some amazing statistics with it.
If you don't have an ILS then seriously look at Atriuum as they are great for the smaller libraries.
lordjim AT gmail DOT com
OK, this is a horribly shameless self-plug, but hey, it's directly relevant. I started two projects aimed at tracking reference statistics: Libstats, which is PHP-based and open-source. I'm also one of the founders of Gimlet, which is hosted and closed-source, but provides a similar workfow.
If you're looking to spend some time delving in code, Libstats is looking for maintainers -- I'm no longer working in libraries, so it's largely orphaned.
What you think is large might be trivial even for OpenOffice/LibeOffice.
Also, the real solution might be to automate data collection and storage in a database. Manipulation would then sort itself out.
If you're at a University, then you should go to the Math Dept and talk to some Statistics grad student or maybe even an econometrics grad student in the College of Business. Heck, there's probably Comp Sci undergrads looking for a project to add to their resume.
"I don't know, therefore Aliens" Wafflebox1
Have gnu, will travel.
I use and like JMP from SAS. They offer a free 30 day demo and I think it does a good job at data visualization and statistical modeling, or as they call it, discovery. It will interface with SAS, R, Excel along with various database packages for additional capability that may not exist in the core product. I found it pretty easy to pick up with a fairly active user base to help get started.
Their product list is here. In particular, I think you would be interested in RapidMiner and RapidAnalytics. WIkipedia has a good overview of RapidMiner.
Video tutorials for both RapidMiner and RapidAnalytics are available on their website. Those videos are a great way to get a good sense of what the product line is capable of. Searching on YouTube will find plenty more that focus on specific use cases and more advanced functionality.
All of their software is dual licensed with a GPL version and closed source license available. GPLed versions of their software also has support contracts available for everything from basic troubleshooting support to full implementation. That includes both Rapid-I itself as well as partnerships with contracting companies in the U.S. and elsewhere. In addition, Rapid-I hosts a community forum that is well run and has active developer input.
I've been using RapidMiner myself for 3 years for smaller projects. I have had occasion to use all of the free resources that I mention above. I have found them all to be very solid. The developers in particular have proven themselves to be knowledgeable and very polite. (IME, that's only to be expected of co-founders who happen to be German. :-) )
I will echo the support for the open-source statistics package R. R is incredibly powerful, and in the natural sciences it is fast becoming the standard statistics software.
I will also echo the sentiment that, by itself, R is fairly low-level and typically requires at least some simple programming to get what you want.
However, there is a very nice graphical front end for R called RKWard (http://rkward.sourceforge.net/). With RKWard, importing and exporting data, running basic analyses on it (descriptive statistics, linear regression, t-tests, etc.), and producing basic graphs is very straightforward and does not require detailed knowledge of the R language. Plus, RKWard is also a nice development environment for writing R code, so if you want to take your project further, you can easily do so. So, I'd recommend giving RKWard + R a look.
It seems to me that all you need is descriptive statistics (change from last month, mean, min, max, etc and probably graphing). Using a general spreadsheet application like Excel or Calc will do the job just fine. Remember that Excel is designed to support business calculations and what you are asked to provide is exactly that! Using a dedicated statistics software for this task (in your environment) is a waste of resources. Full stop.
However, the solution may not be straight-forward to solve in Excel or any other program. In my experience there are two main reasons:
1. The request for data is unclear.
Why do they "increasingly want data on various aspects of our activities"? It could be that the data you have provided so far has not provided support to decisions. Are the questions they really want answered possible to support with the data you can provide? Meet up with the actual decision makers or at least someone who knows what the statistics are actually used for and ask them WHY they need it. Is it used to support resourcing? Is it used to describe changes? Not even a university administration creates statistics for no reason. Most likely, what they really want to know is a handful of numbers like "change from last month", "overall sum", "hours spent on teaching vs information searches".
Do this with an open mind. You will probably learn that many of the imperfections you see in the details are less important to them. When you know their true needs, suggest a package of data, graphs, free-text report or whatever is suitable. If some parts are easy to provide, be clear about that. If something is more difficult to produce, tell them that it is is possible but time-consuming and costly. Get their buy-in before you spend time on producing the output.
2. The raw data is not optimally formatted for the calculations
First of all, if raw data quality can be improved, do that first. Update forms used for feedback, ask for output in a specific format etc. Then arrange the data and calculations in Excel to make it flexible and easy to read and troubleshoot. The trick is to use structure your data and calculations in Excel in a way that is easy to follow visually and logically. In my experience it is very useful to use different tabs for data entry, data analysis and presentation.
It seems from your examples that your input will come from a variety of sources, both manually entered and output from other systems. To get it into Excel, create separate source data tabs where you can enter or paste your raw data. For each source data tab, create a "clean up and calculate" tab where you rearrange source data and make most of the calculations. If raw data is very far from optimal or calculations are complex you may want to use several tabs or even several workbooks for this. Then create presentation tabs where you present the results from calculations in a useful format.
I'm convinced you are suffering from both these problems. Attack them in numeric order and you are well on your way. And by all means, sign up for a course in advanced Excel that is suitable for your application. Best of luck!
http://www.sofastatistics.com/
I've used them all and in terms of engineering and academia, MATLAB seems to be where most theoretical prototyping is done. The license costs for academic/student use are reasonable but it's about $2K for a commercial single seat license. Octave is the MATLAB open source alternative and for most basic functions it does well however it doesn't have the extension packages available that MATLAB does.
My favorite and one I use all the time is "R" because it does have great open source community support and there's not a lot it can't do.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
Agreed ... odds are, they're not running homebrewed circulation software and someone in the library community has tried to extract metrics from whatever they're using.
Build it, and they will come^Hplain.
As others have said, if you're mainly doing reports, stick with Excel or a database solution. Excel lets you look at your data from a variety of angles (pivot tables, etc), and has usable graphs. As usual, Microsoft has numerical issues, so you may get wrong answers under certain conditions, but hey, it's Excel.
What is it that "anything complicated" means? Fancy graphs? Fancy partitioning/aggregation of data? Modeling and forecasting? Summary statistics? Graphs that aren't fancy, but Excel doesn't provide?
An open source option that I haven't seen mentioned is gretl. It has a reasonable GUI and can make nice graphs (though not terribly customizable), give summary statistics, sample data in various ways, and do basic modeling. (It comes from an econometric world, so has quite a few time series capabilities.) If you need to do some things with time series, it would be helpful. (Though if you don't know what you're doing, it simply makes it easy to shoot yourself in the foot.)
I also work for a (relatively) small academic library, but our campus has free licenses for SAS and JMP. I had to go through hoops to get it (bureaucracy being what it is) but I use SAS all the time for inventory and usage data. It helps that I was a SAS programmer once upon a time, but I love it for its abilities to clean data as much as its statistical chops. Check around campus if you haven't done so already. You may find access to one or both of these to be easier than you think.
If you do go with R, be sure to check out Rstudio (rstudio.org), which is a very nice front-end for R.
In response to the posters who tell you that R is low quality because it's open source, I can tell you that's nonsense. I have Stata, Matlab, and R on my machine, and access to SAS on a research server. There are times to use each, but all else equal I use R. It's not trivial to learn, but it's a powerful high-quality piece of software, widely used in the statistics community. Whether it's appropriate for your use depends on you and the task. But it's great software.
Sounds like R might be a bit much for your needs.
Agreed. Another good alternative is MYSTAT, the free "student" version of SYSTAT. Note also that many academic institutions negotiate site licenses for SYSTAT, so you might already have the full version available to you.
Not open-source but very easy to generate reports off relation data sources. http://www.yellowfin.bi/
Nice response I totally agree... however I totally agree with others that Access is a pretty poor database but it works on the small-ish scale. (Although I do know of an IT company that uses it for their ticketing system... I know, I know you would think they know better, but they are IT pros not programmers and it works for them)
If you have the skills I like the suggestions by others of: MySQL and/or a LAMP (Linux/Apache/MySQL/PHP) server for a web-based database program(although it seems a lot more complex then what you already have in place).
I also like your basic aggregation questions, if these are accurate questions you need answered then Access can do them as Anon said above.
As a Perl guy, Python(x,y) has a complete scientific computing package. While Perl and Ruby can do these things, Python(x,y) does it in a slick way.
It is a Windows only package as far as I can tell.
Perl, Python and Ruby can deal with Excel and R but Python(x,y) provides a nice interface for everything.
PSPP is a nice idea, but lacks functionality. SPSS is ridculously priced, even with IBM's "discount" for non-profits.
DeduceR and R commander give you access to the full power of R under R GUI. DeduceR gives you spreadsheet
like data entry and basic stats, and then you can load R Commander for a menu driven interface more advanced
functions.
Were that I say, pancakes?
Data mining with Rattle and R .... http://rattle.togaware.com/
Most librarians were probably not math majors, and are unlikely to be expert in statistics. But if you can work your way through the book, you may get enough insight into your data to ask good questions from a local Math department. No doubt some graduate student(s) can get a paper out of it, or at least some applied class project credit.
But if you don't understand what it is you are looking for, you probably won't coax them into figuring out what questions you ought to be asking. So start with the book.
While a free version is on the site, support the work by buying a hardcopy for the library ;>
Businesses use Customer relationship management systems. These tools also provide statistics.
I have been using SAS for about 3 years and it is not suitable based on the posters requirements:
1. "I also place emphasis on anything that is open source and easy to implement": SAS is not open source and has a tremendously steep learning curve.
2. "allow me to bypass the convoluted purchase approval process.": It is licensed directly from SAS, you won't find it at CDWG or JourneyEd etc.
SAS requires a lot of effort, but will produce tremendous results. Since this is a relatively small part of the poster's job, it doesn't look like it would be time or cost effective approach.
You might want to take a look at LibAnalytics (full disclosure: I work for Springshare). If you're actually trying to do statistical analysis, then I think others' recommendations will server you better - but if you're looking for a way to track many different sorts of data and generate reports, then I think LibAnalytics would serve you very well.
Subject line fills in the banks omitted in other responses.
Leslie Satenstein Montreal Quebec Canada
Here's something that might give you a direction.
DataMart-Tool.pdf
For the kind of constant, ad hoc, data nagging your talking about the above would be a good start. Using something like MSSQL Analysis Services with Excel on top.... Or (get ready to feel really dirty) MS Access pivot tables....
I haven't pointed someone toward the Microsoft in a while but if you don't have serious programming chops you're best bet is cubes in Access and they will take you quite far. They are extremely similar to Excel Pivot Tables. If you haven't explored Pivot Tables in Excel, there's a possibility that half your battles will be won there on the data as you currently have it.
A more static but open source solution at the scale of Excel Pivot Tables is OpenOffice data pivot functionality. In fact, you might be able to put something together where OpenOffice Calc sits on top of MySQL! That get's you database, spreadsheet and pivot. Any super serious stats can be handled in R....
Every rule has more than one consequence.
This is not the answer you're looking for... Could you post a reply here with whatever you chose to do?