Slashdot Mirror
Jon Udell on the Nerd's Spreadsheet
rcs1000 writes "Jon Udell has a interesting article on a new type of spreadsheet: one targeted specifically at techies. The skinny is that any spreadsheet is actually a computer program, only in Resolver One, the product profiled in Udell's piece, this is explicit rather than implicit. And the code is IronPython rather than VBA. There are some other cool things it does — allowing cells to contain objects, and allowing spreadsheets to back-end websites." Udell's screencast gives a good demo, though the presenters are a bit hard to hear due to the phone connection. Resolver's own screencast is an alternative.
Multiply 850*77.1 correctly?
while(1) attack(People.Sandy);
Oh jeez, this reminds me of the last company I worked at, where they tried to do *everything* in spreadsheets, even documents that would have been much better in word processors or databases.
SQL databases have become much lighter and more efficient these days. Why should I use this store data over a lightweight SQL database?
We all know what to do, but we don't know how to get re-elected once we have done it
In a Resolver spreadsheet, these objects are visually persistent. I haven't yet got my hands on Resolver, but here's an example of what I think that will mean. Suppose that I have a data set I want to transform, against which I'm testing five different versions of a transformation function. I'd put the data in cell A1, the functions in cells B1..B5, and the results in C1..C5. Now I'll see everything at a glance.
That . . . sounds just like a normal spreadsheet to me.
Solution looking for a problem?
Your hair look like poop, Bob! - Wanker.
Aside from having a console window for custom formulae and a debug output window, it is not that much different from a typical spreadsheet program. This can easily be built on top of existing office suites, if they haven't include them already.
Organising code on a spreadsheet... I guess it will resemble Befunge
and allowing spreadsheets to back-end websites
munge them?
hack them?
copulate with them?
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
Everything looks like a nail when all you have is a hammer.
Spreadsheets are so useful today that they can do many tasks that are better done with other tools... If you know the other tools.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
A good portion of spreadsheets actually should be database tables of some kind. People end up manually grouping and other stuff that report-writers can do automatically. What is needed is a kind of "dynamic" RDBMS tool that has open-ended columns and column widths. A "spreadbase"? The Oracle clones are all too rigid.
As far as spreadsheets for programming, I've experimented a lot with data dictionaries to simplify column management and column sub-sets for regular ol' edit-and-report screens. So far it is tricky because one often wants to tweak something for a particular context and one-size-fits-all hits a wall. The trick is finding a good, clean way to "override" specifics from the table when needed or just make alternative entries of a given column and select them via set notation when needed; but I've yet to find a clean, simple convention. It ends up fairly messy such that regular copy-and-paste is unfortunately the cleaner solution much of the time. Maybe if the toolset and the language was geared toward nimble data dictionaries, these approaches would be smoother. Forcing a non-data-oriented language to act data-oriented is like trying to keep a toddler in line.
Table-ized A.I.
Wow... that screencast is perfect for me to sit facing the screen with my eyes closed, and anyone that walks past my cube will think i'm doing some spreadsheet wizardry....
nice. Now i'll go someway toward meeting the quota for those that sleep at work.
And the code is IronPython rather than VBA.
IronPython is not a language!
Is this a spreadsheet, or a poor man's database?
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
Seems like a fairly on-topic question. Anyone know the answer?
"Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
http://common-lisp.net/project/cells/ is a dataflow extension that provides Common Lisp with classes where instances of the class have properties (slots in lisp parlance) that can be calculated like spreadsheet values.
"even documents that would have been much better in word processors or databases." Well, that's true - but once your data is in a database you still need to analyse it and present it. Spreadsheets are used for three things: storing, analysing and presenting data. Most complex spreadsheets are actually *applications* for analysing and reporting on data. The things you get with databases are: a single authoratitive data source, transactions and queries. Most people producing reports don't want to do their analysis by writing SQL queries though! Part of the idea of Resolver is that it is really an application platform, with a familiar interface. Your data can be stored in a database (this is a good thing) and tables (worksheets) can be populated from database queries. You can then analyse (with real code if it is warranted) and present that data. You can also add user interface elements to push stuff back to the database if you want. It is massively cheaper than writing a custom application and a darn sight easier than doing the same thing with Excel.
Importing and manipulating hex would please me no end.
H.
This is a really interesting idea. If you think about it, every spreadsheet is a computer program. Excel is probably the most widely used development environment in the world. And it sucks. Big time. Watching the screencast, it looks like this is a great halfway house between bad, ad hoc, spreadsheet development, and a traditional IDE. The choice of Python as the language is a great idea - it's much, much easier for someone than VBA, and it's much more semantically powerful. I'd be interested in discovering whether it works existing Python and .NET libraries.
http://common-lisp.net/project/cells/
[quote]I haven't yet got my hands on Resolver, but here's an example of what I think that will mean.[/quote]
I said "WTF?" and stopped reading right there.
Yo dawg, I heard you like the Ackermann function, so OH GOD OH GOD OH GOD
is that this is the first time i am seeing a demo developed in silverlight. Yeah, i know, there are lots of demos on MS site. But this is the first time i am seeing somebody using SilverLight for some real demo on their own site. is this the sign of things to come? Time will surely tell
I had an interview with these people about 18 months ago (didn't get the job) but I thought there product was pretty cool. Its basically a simple spreadsheet that allowed you run an IronPython script in each cell. So you can run your spreadsheet with all the power the Python and .NET can give you ... say you wanted your spreadsheet of investments to update against a stock exchange web site every 30 seconds
while(e) { Kyoatie(); }
I never, ever want to see A1..A10 again.
Give me an item dispenser, the program should name things sensibly as they're created and all formulas should read as plain text like: profits = sales - expenses.
Please.
William
Sphinx of black quartz, judge my vow.
this is just another VC whore I reckon
There may be some exceptions, but I haven't found any... Once a bloger, journalist, programmer, and I'm sure lawyer, goes to work for "the Borg" they lose all sense of objectivity.
Why in this day and age would someone want to lock themselves into yet another Windows-only application?
I would avoid this thing like the plague. Even if Linux hasn't taken over the desktop, there are certainly enough Apple users these days that OS portability should be BUILT-IN to the design process from day one.
The whole point of an OS is to isolate the application from the hardware. But what good does that do us when so many new applications are designed to work with only a single OS, and that single OS is designed to work with only one type of hardware? Has our industry gone mad?
"Spreadsheets with First-Class Functions and Objects"
This is just a spreadsheet with the ability to put functions and other objects into cells. It's a good idea, but there is no need to dress this up in marketspeak. We understand techno-babel just fine here.
Not reading TFA, it reminds me a bit of my custom db/ui solution that I use for generalized information storage: links, books and movies I watch, developer notes, etc... it some web-based Perl CGI and is, in effect, a big old flat table database, the columns defined by text files, and it auto-generates a convenient form with the usual spattering of HTML input types...
SO YOU'RE GOING TO DIE: The Comic for Dealing with Death
If I wanted one that'd take functions or objects for most given languages, I could probably write the basics in a weekend. This is like the MS powershell thing that allows users to pipe objects; a solution in search of a problem.
This is absolutely an amazing product. It can do store objects and back a website!!! It can do all the same things as my well supported database product!!!
There was a Spreadsheet made for the CIA by Xerox that would easily be the Nerds Nerd of spreadsheets as it enabled full access to the underlying programming language, Smalltalk.
Here are some links to the old version and newer developments.
http://wiki.cs.uiuc.edu/VisualWorks/The+Analyst
http://www.mojowire.com/TravelsWithSmalltalk/DaveThomas-TravelsWithSmalltalk.htm
http://www.sunless-sea.net/wiki/SmallTalk
http://www.google.ca/search?num=100&hl=en&newwindow=1&q=%22the+analyst%22+xerox+smalltalk&btnG=Search&meta=
These people need a reality resolver instead of a spreadsheet replacement.
"Follow me" the wise man said, but he walked behind.
Really, besides the laugh we had the other day, Excel does not suck. It's does a lot of things very very well. Just don't try to use it beyond what it's designed for.
I used to work in the finance department for a very large company, and I was inundated with Excel. People used it for everything and loved it.
The only hurdles come when someone tries to do something that's grossly inappropriate. That notion comes from finance people who love it and try to do everything with it. Sometimes they succede, sometimes it was ridicules. I don't fault Excel for that.
I Just think about basic command line stuff I use everyday. With bash you have sort, awk, grep, sed etc.. You can do very much the same things with Excel, only sometimes much more easily and with instant visual results.
Cripes I'm a Unix admin and I pine for Excel on Linux. OOCalc is missing ohh.. I don't know decent graphing, pivot functions etc..
I still find myself doing complex sorts, replaces, etc.. with excel ^H^H^H^H^H(ehh. OOCalc) because it's just easier.
That and the data is in a portable compatible format (not just a static text file) that anyone can pick up and continue to use, look at my formulas etc.. without much trouble.
And does it poorly. And insecurely.
You can already access spreadsheet content from Visual Basic, and include VB script in spreadsheets. The same scripting ability which allows the "wow" features in spreadsheets also creates the potential for abuse - remember macro viruses? Suddenly, documents which formerly contained only data now contained executable code, and it gave rise to a security nightmare.
Yes, today, with VBA, you can do what the article mentions. In fact, it's been possible for years. Problem is that:
- Very few people use it, and
- Those who do use it tend to use it poorly.
Yes, you can back end a website with your spreadsheet. But why would you? A spreadsheet is a horrible way to manage data; there's no referential integrity checks, no versioning, no security, and doesn't scale well. Furthermore, your crucial data is tied to a particular application, rather than a database.Just like VBA, it's a nice nerd's toy, but the wise system programmer recognizes that it has limitations.
The society for a thought-free internet welcomes you.
And for Gantt charts, there's a program which lets you express each task in python code (including whatever calculations, remote data, or whatever that it needs to get data from)
http://faces.homeip.net/
and here is why, even moderately smart programmers are more comfortable with a spreadsheet than a database or full blown application. Example which I believe still applies today:
I was a member of a "club" of techies who were investing in the stock market in the 1996 timeframe. We cycled the clubs officer positions through the club membership so everyone knew what others were doing and how. The 2nd year, I got the Treasurers position and my job was to collect "investments" at each meeting and provide a overall club report and reports for every member's position/investment in the club. I was handed a spreadsheet with tons of data entry cells sheets to fill in data. I could have cleaned this up but hey, this was the age of the internet right. So I build a SQL database to handle the accounting, wrote code to drive a CGI interface to a web server and wrote scripting to trigger updates on stock valuation and triggers to update the database on changes. It was all accessible from a web browser with a login. And very repeatable. It worked great after the 3rd months meeting and for the remainder of the year. Because it was cross platform, it would run on anybodies computer and did not need to be on the internet.
Here's the kicker( finally ). The next Treasurer was the guy who started the club, owned his own business, was a Windows software engineer who's product he built over close to 10 years and was a signal processing kit(PC and software appliance ). He knew what I was saying when I described the package I'd built to do the books but he wanted none of it. He wanted the latest report and then he write a spreadsheet to handle the task. And for every meeting from then on until the end of the club 2 years later, we were constantly correcting data in the spreadsheet and sometimes, corrections required pulling out paperwork from previous months meetings.
There is a fools familiarity with spreadsheets which people find comforting. They will go to great extremes to try and get the spreadsheet to work for what they are trying to do. Great extremes. And with the plethora of Neanderthal-ish computer users out there who think they are computer guru's because they can do "=(A2+B2)" in MS Excel, it is a long long road to get movement outside of the spreadsheet.
I just found out a buddy has taken on a 2nd job and is trying to build a business out of it. He's under charging to get experience in more aspects of the field and to construct a spreadsheet to "automate" the task(s). He's thinking that the business will grow around the "application"/spreadsheet.
So I think the title should be, "Jon Udell on the Geek's Spreadsheet" because the nerd is really not that 'into' the technology to really know or want to work outside of a standard spreadsheet. IMO.
LoB
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
I've always felt that MS Access is a really shitty implementation of a really good idea. Yes, it was backed by an SQL database, but you can do spreadsheet stuff, too, and behind the scenes you could tie it all together and make it look nice. Another nice feature is that managers, with no programming experience and just a bit of SQL knowledge, can create and generate their own reports without having to bother a developer. Alas, the SQL engine has many fatal flaws, the scripting language is junk, and the GUI is just too quirky and weird. The report stuff is pretty nice, though.
Unfortunately nobody else has ever tried to do it right. The pieces are all out there, but they're not integrated.
I may be a diehard Red Sox fan, but I still cheer when a Yankee makes a good play.
Python scripting was possible for a long time for Openoffice users ...
Like other posters said: half the time they're busted, denormalized lame-ass substitutes for databases. The other half of the time they lead to convoluted algorithms that would be expressed more simply in code. Spreadsheets are the shits-- logic and presentation layer tightly coupled. The only value they give is in sortable and filterable views, and grid controls on a webpage will do that for you nowadays.
Get your teeth into a small slice: the cake of liberty
I am wondering if we have forgotten this cute little app... Thwe webpage says
2000-12-07, but I think I've played with it long before that. And yes, it had
database connectivity, could serve data over HTTP and, of course, the extension
(and half of implementation, I'd guess) language was Scheme.
http://siag.nu/siag/
Paul B.
(shameless plug)
I'm a nerd as well, and built an online calculator/spreadsheet for myself. It's a new take to an old problem: instant answers, inline units (MB, GB), and shareable results. Yes, it handles 850 * 77.1 and more.
Thought you might find it useful.
But you are talking about a half-ass query language. I think that "true nerds" wouldn't even use spreadsheets, but a somewhat normalized database. It is easier to process data when it is "row-atized" such that you wouldn't have a different column for each region or product or month instance like one often does in spreadsheets, but rather a single product category/location/month code/indicator as part of a row. (dBASE and relatives were great at ad-hoc data chomping, by the way. SQL-based DB's assume too much formality in my opinion for ad-hoc work.)
Spreadsheets are an electronic version of accounting grid paper and meant to emulate the paper world. But true geeks leave paper representations behind when it suits the problem better, using a "relativity engine" to deliver the needed view via math-like transformations. Spreadsheets are not very transcendental, even if you do improve the formula propogation.
Thus, "nerd spreadsheet" is possibly a contradiction.
Table-ized A.I.
Wolfram called; they want their concept back.
SIERRA TANGO FOXTROT UNIFORM
I'm not sure why using IronPython instead of VBA (well, anything's better than Visual Basic, true...) and "cool" things like embedding objects is an improvement to the traditional spreadsheet/Excel app, but I've often thought that somebody out there ought to redesign the spreadsheet. Excel is great for a lot of things, even for science applications, even though it's often just a cheap, quick-and-dirty alternative for better software. In many cases, it's easier to make a spreadsheet than to write a program or take out a loan to buy a package like MatLab. But the assumption that all data needs to be represented by a single fixed length table some 65536 high by 26^4 wide leaves a little to be desired. There ought to be a mostly GUI interface that can represent large blocks of numerical data and records in a more intelligent way, seems to me. And it isn't Labview. Assuming the implementation wouldn't suck (ie written partially in TCL/TK, Perl, with smatterings of Java), the open source world could introduce yet another killer app.
Cells is a mature, stable extension to CLOS that allows you to create classes, the instances of which have slots whose values are determined by a formula. Think of the slots as cells in a spreadsheet (get it?), and you've got the right idea. You can use any arbitrary Common Lisp expression to specify the value of a cell. The Cells system takes care of tracking dependencies among cells, and propagating values.
Resolver seems to take this idea a step further. It looks like you can write nice reporting tools with this. There is no need to bash Resolver because you don't like Access or Excel.
Here's a similar proposal for distributed, extensible, language-neutral spreadsheets! Someone at Google, please implement this? Cheers!
-- Subvert the dominant paradigm. Repeat as desired. http://ownlifeful.com/
Back before we first got involved with Mathematica 1.0, I was already looking for something that would put real calculations in table cells but Wolfram became wedded to the notebook format that was arguably the only practical solution to being cross platform in the 1980s.
On the Mac more was often promised but rarely delivered. There was a very early "spreadsheet" called Trapeze which allowed you to lay out a multiplicity of typically smaller grids which always appealed to me more than the idea of forcing unrelated data structures into a single sheet. For a moment I hoped that Numbers, the new spreadsheet component of iWork, might have finally revived that concept but on the minimal playing around I've done to date, it too misses the point.
Meanwhile there has been one product from Germany, originally Mac only but nowadays also available for Windoze which is the only moderately expensive software that I keep going back to when I really need to get something right. RagTime is positioned mostly as a layout program but it does have full-feature spreadsheet and word processing components (able to import Orifice files) and a very strong component model, despite the developers having been badly burned by Apple's misadventures with object systems in the 1990s.
Even earlier I had been able to use it to automate the weekly random positioning of display ads in a printed newsletter. And even in 2007, I retain a sneaking wish that it would at least gain a developer API so we might be able to add Perl and symbolic math to make a true Swiss Army knife.
-- Our systemic servants do not good masters make.
Every time a user changes a Resolver spreadsheet, the software generates Python code expressing their desires;
Oh yeah? How does Python express hot, naked babes and a sixpack?
One thing that is different is that it auto generates python code that fully describes the spreadsheet (structure, display properties, and cell values!) - and which apparently is then executed to actually draw the spreadsheet itself (kinda a weird bootstrapping thing going on there).
.Net app as is, and you have a powerful python-powered spreadsheet engine inside your custom app. This means you can give your users a familiar spreadsheet UI as part of a larger app where appropriate, with all your business logic, without the unconstrained craziness of an excel VB App. Nice!
Anyway, I wonder how well modifications you make to the generated code are treated by changes made to the spreadsheet itself...
Anyone who has tried to hack auto-generated code - e.g. from UI code generators (VC++?) knows how ugly this can get. Maybe with the closed loop (from generating code, back to displaying the spreadsheet) they have managed to deal with this, but the screencasts only show such trivial examples of editing so you have to wonder.
The thing that seems REALLY cool about this tool (and the real answer to your question I think), is that you can take this auto-generated code + your custom code, and drop it into a
At some point, an organization realizes that they either don't have the training to be able to keep working from a spreadsheet, or they need something more powerful anyway.
So they buy FileMaker Pro.
Which means, they essentially have a bunch of different, interesting views, with some (very) limited scripting, all ultimately centered around one flat table.
(Yes, I know there are later versions which actually function as relational databases, but most FileMaker stuff I've seen is around version 5 or 6. Most people don't want to upgrade past that, because it'd require upgrading every single computer at once. I know all about relations and "portals", etc -- almost unusable for linking more than two tables in the same query, and I hear reports of corruption, etc.)
From what I understand of spreadsheets, some make sense as spreadsheets, and some don't. Of the ones that make sense as spreadsheets, they could still use a more powerful spreadsheet program, and I don't mean Python.
Suppose, for instance, you just want to calculate a table of values -- compound interest, say. Some initial investment, an annual interest rate, and an additional annual value. You could do something like this:
=A4*(A$3/100+1)
And you could drag it down and across. But it's not necessarily an obvious formula, and if you ever have to make a change to it -- for example, extend it so it moves in increments of 5 years, or of n years -- you have to remember to drag the change (to the formula) over all the cells to update them. If you want the table to be bigger, you have to drag it some more, and fill in some more values, or write formulas for them.
But you don't really need much in the spreadsheet other than your table of values. You don't need it for a chart, and if you did, it might not be sufficient -- for instance, if you're modeling a mathematical function, you'd want a much bigger table, and you wouldn't be interested in the values so much as the graph.
Still, all of these could be modeled by a program very like a spreadsheet.
The other common use is to manage information, and to analyze and present it. Spreadsheets are really bad at this, Filemaker is somewhat better at analyzing and presenting it, but not much better at storing it -- the biggest improvement, really, is indexing. But the kind of information you store in a spreadsheet would probably map really well to a relational database.
So the trick, then, would be a GUI that's as easy as Filemaker, almost easier than a spreadsheet, but which maps to a real relational database. I'm not the first to think of this; there are several other projects, but most of them have considerable limitations, and it seems like a good target for a web app, anyway. But I've never found the time to do it right.
Considering that FileMaker is often used as a replacement for things like Act, QuickBooks, or other, similar software, and that other such replacements have been made online (but specific to the domain of the app), I think a generic GUI for building online, database-driven apps would be a good project.
But I haven't had the time to do it, much less do it right. Maybe soon, though.
Don't thank God, thank a doctor!
Firstly, there are other major commercial players. FileMaker, for instance -- which I have to work with occasionally.
Second, FileMaker (and Access) are object lessons in why you shouldn't do it this way. Like you said, the SQL engine has flaws, the scripting language is junk, and the GUI is quirky and weird, but you like the report stuff.
Were this done right, you could keep the reports, and replace everything else. This is how KOffice seems to be implementing this, or trying to -- Postgres or MySQL can be used, and maybe SQLite for some of these. You can then run Kexi to manage it and Kougar to print reports. But any of these can be run independently, and while I believe Kexi will let you plug Python into it, you can script it with all kinds of other things. Plus, there are about 5 other, similar GUIs which will all use Postgres as a backend (oddly, none seem to be MySQL-specific, but one is Postgres-specific).
I mean, you're right, it's almost a good idea, which is why there are so many clones. But all of them suck. I propose we do it in AJAX, but then we have to fight about what language to use on the server side, and thus still end up with five or so implementations -- still, I'm going to do that, at some point.
Don't thank God, thank a doctor!
http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html People who are Smart but don't Get Things Done often have PhDs and work in big companies where nobody listens to them because they are completely impractical. [...] For example, they will say, "Spreadsheets are really just a special case of programming language," and then go off for a week and write a thrilling, brilliant whitepaper about the theoretical computational linguistic attributes of a spreadsheet as a programming language. Smart, but not useful."
For every expert, there is an equal and opposite expert. - Arthur C. Clarke
Programming in a spreadsheet? It's called Flash.
Moderators, the author of the parent message appears to work for the company, and his messages ought to be given consideration for mod points. See the messages from 'voidspace' for moderation consideration.
As someone who wrote exactly this kind of thing for Java, I can attest to the idea that it can be useful and not a security issue. The trick is to think of its uses in a completely different way than what we tend to associate with Excel. The way I use Bean Sheet (my programmer's spreadsheet) is in the following ways: - To hold and manipulate small amounts (usually sub-100,000 rows) of formatted and structured data. One thing I've never seen anybody do in Excel is script your own sorting algorithms for example. - To have a format compatible with a version control system so as to see diff's in simple, but formatted, lists of data. - A more visual way of modeling/testing/debugging API and components. - To allow programmatic access to the spreadsheets. This is what this guy was talking about as a good opportunity to seed quickly changing business rules into an otherwise static system. Suppose you have a piece of code that needs to perform some business-rules driven operations on a data set. You can encapsulate those business rules in such a spreadsheet containing no data, test it, and deploy it with your application. The app then loads the data into the spreadsheet (the spreadsheet might even contain information on where to put the data set into it, so the interaction can be quite generic) and the app then queries the spreadsheet's "result" cells. And since these kinds of spreadsheets can contain all manners of data types, integration can be really simple. Then, if need be, pop open the spreadsheet, tweak the business rules, and deploy it back into your app -- perfect component-oriented programming model implementation .
Alexey