Hmmm... I never thought this little blog entry would get Slashdotted. Really I hadn't intended to criticize the decisions MIT made. The project met its goals on time and under budget. The selection of 100 percent Microsoft tools was apparently a smart choice. Had it been my project I would have perhaps added some goals, e.g., more of an online community aspect for the front-end and easy to package up all the software behind the service to give away to other schools. These goals might have led to some different decisions on tools or perhaps not. Actually one nice thing about Microsoft tools is that you are guaranteed that most people will be willing to adopt them.
One of the things that we try to teach in the class (textbook is online at http://philip.greenspun.com/internet-application-w orkbook/ if you're curious to see what the students suffer through) is that being a good code monkey/CS nerd isn't sufficient to function well as an engineer. We try to give the students some experience with taking vague client specs and turning them into precise requirements, with presenting their work clearly, with constructively criticizing others' work in meetings, with conducting and learning from user testing, etc. The rationale for this is laid out in http://philip.greenspun.com/teaching/one-term-web
So it was actually very gratifying that our guest speakers came in and demonstrated that state-of-the-art American IT development projects no longer involve plain-old-programmers in America. Our students need to learn this early so that they can plan their careers and further education accordingly.
Thanks, guys, for giving the Sun E450 (philip.greenspun.com shares that 5-year-old machine with photo.net) and Harvard's little blog server a workout.
Don't assume from my posting that I'm in any personal hurry to learn VB and PHP! But having so many bright young people in 6.171 gives one a fun opportunity to take a high-level look at the programming tools of the moment.
What would I personally prefer to use? The same thing that I would have wanted to use 10 years ago: Common Lisp, CLOS, plus an ML-like type inferencing compiler/error checker. I find this preference, shameful, however, and try to keep it concealed from young people.
Just yesterday I ran into a friend. She's a 23-year-old graduate student in computer science at Harvard. Conversation rolled around to programming tools. Unprompted she said "What I think would be best is Common Lisp Object Systemw with a modern type system". I was stunned. I thought it was only dinosaurs like me that clung to Lisp.
I had a second ephiphany for the week... Believing that Lisp circa 1982 plus some mid-1980s ML tricks thrown in is better than all of the new programming tools (C#, Java) that have been built since then is sort of like being a Holocaust denier.
Thank you for pointing out the fact that my last day of influence at ArsDigita was in April 2000 (nearly two years ago). Here I am, a quiet retired schoolteacher, and people are still beating me up for opinions and attitudes that they think I might have had at some time in the preceding decade. I try to leave the old writings up on my Web site so as to be a good Internet citizen and not break links. But people take things that I wrote in the early 1990s and assume that I still hold these opinions. To take a non-controversial example: I struggled to get Windows NT 3.51 working and couldn't. I vented my rage against Bill Gates in some site. But now I love Microsoft. About half of my students use the.NET tools and are extremely productive. I'm running Windows XP on two computers at home and haven't spent more than 15 minutes on sysadmin. The interesting research in online communities is all being done at... you guessed it, Microsoft Research.
I'll come back home after walking the dog or get out of the bathtub where I've been reading New Yorker magazine. I amble over to the computer and there is email from someone yelling at me to "Get a life and stop picking on Bill Gates you miserable envious poverty-stricken grad student."
Online communities only work if someone puts forth a strong opinion against which others can react. This is why photo.net works so well. We write "Nikon autofocus sucks" and that yields a huge pile of interesting comments from Nikon users about how they've been able to get great photos with the Nikon AF system. But when you have people with long memories and/or Google to dredge up these ancient opinions, which may never have been held strongly to begin with, life gets kind of confusing.
Mea culpa. I shouldn't have advised you to get into the Internet craze. I started loving the Internet because it lets people learn from other people. That was back in 1976 when I began using ARPAnet. And what is amazing is that I can still wonder in 2002 at how useful Internet is. For example, I've learned so much lately from other pilots whom I've met on the Internet and could never have connected with as a practical matter otherwise. One of my interests is in getting a Stemme S-10 motorglider. They build 18 per year for sale worldwide. None are based in the Boston area. But I found the owners and their impressions of the aircraft on rec.aviation.* groups (thanks to dejanews (Google Groups now), of course). Then I emailed them and they were very helpful to me, despite my status as a novice pilot.
At the end of the 1990s this wonder turned into something more akin to fever. And, along with the pressures that come from managing a company that was growing 1000%/year, that didn't bring out the most attractive parts of my personality. Bottom line: I'm sorry for giving you unsolicited career advice. If you come to the US and you don't value your life too much, I'll take you up on a sightseeing flight! (with Alex in the back seat, of course)
[And don't sell Internet apps short just yet. With some improvements in infrastructure, such as ubiquitous high-speed wireless IP connectivity and reasonable conversational speech recognition software, a lot of interesting new applications should be enabled. Of course, it might be 5 or 10 years before we have this infrastructure in place, which is why I'm working with young people at MIT instead of trying to build the stuff myself.]
don't blame me for any photo.net sysadmin stuff
on
ArsDigita Shut Down
·
· Score: 2
It is gratifying to me that people still like photo.net and learn on the site (up to more than 1 million user-visits per month). But I also find it alarming that people blame me when the site is dead or slow. I was never an expert Unix sysadmin or Oracle dba. Being an MIT Lisp Machine programmer, I always hated Unix. I learned as little as possible about. As for Oracle, it is a wonderful tool with a great abstraction barrier (SQL) but I don't know of anyone who would say "I had a lot of fun last night administering my Oracle installation."
If you thought the site was a pig in 1997 you can blame me because I was personally involved with some of the sysadmin/dbadmin stuff back then. But I'm retired now! I do the stuff that (a) is fun, and (b) that only I can do. So I write articles for photo.net and critique learning photographers' submissions to the image critique forum but I don't try to beat Unix and Oracle into submission. I drove to Nova Scotia for the entire month of September. I spent the rest of the fall in art museums, Civil War battlefields, and National Parks between Boston and Texas. When I got back from three solid months on the road, what did I find? Email from people blaming me for something that they didn't like on some Internet server somewhere.
Rajeev and his merry band will eventually slay the sysadmin dragon and photo.net will be responsive once more. But when it happens you should thank him and not me!
I never said the Slashdot code sucked!
on
ArsDigita Shut Down
·
· Score: 4, Interesting
Your recollection of my comment about Slashdot is ALMOST correct (I'm glad someone in the room was listening anyway). I mentioned that the guys who wrote Slashdot weren't professional programmers but that they built a great site by carefully tailoring the software to the evolving users' needs. I mentioned that some people have thrown rocks at the way it was coded (but not me since I've actually never looked at it; I have enough trouble with my own source code!) but that it really doesn't matter since data model plus page flow are the only things that affect the user experience.
You must have caught one of my last one-day courses. I've stopped doing them because I think that enough people have heard what I have to say. The ideas are out there for anyone who needs them, either in a document I've written or in the heads of former students, users of photo.net, embodied in products such as Microsoft.NET or openacs.org, etc. It is much more relaxing to limit myself to teaching one semester per year at MIT (6.171; Software Engineering for Internet Applications -- to avoid the kinds of tool flame wars that one sees on Slashdot we don't mandate the use of any particular tools except an ACID-compliant RDBMS (practically speaking that means Postgres, Oracle, or Microsoft SQL Server)).
I spend the rest of my time taking flying lessons (doing my primary training now, airplane arrives on March 1, starting instrument training in March, leaving for a 3-month Alaska trip on June 1). There is nothing quite so humbling as learning to fly. You show up at the airport and, with 60 hours of experience, are almost surely the most incompetent person on the property. If your instructor is "old-school" (mine is) there is an intermittent shouting in your headset ("more right rudder"; "what do you think you're doing?"; "don't you see where the nose is pointed?"). Everything is happening way too fast.
Anyway, flying sure makes arguments about the proper way to code up a Web script (which after all is simply merging the results of an SQL query with an HTML template) seem irrelevant. Every time you bring the airplane down in a crosswind you're putting your life and your passengers' lives on the line. Screw up a bit and you've got yourself a $228,000 pile of vaguely airplane-shaped scrap. Screw up more than a bit and you've dug yourself an airplane-shaped grave right on the runway.
Anyway I guess I should close by thanking the Slashdot guys once again for their great achievement in building this site. I often use it as an example to students of how one does not need or want too many collaboration areas on a site. Users need to be able to find the discussion. What I like about Slashdot is that one need only check the front page in order to see what is being discussed.
Yes, well photo.net was formerly run off smaller machines. But we didn't have 100,000 registered users. Or the photo critique forum. Or "hot or not". You can't serve the entire world with 4 CPUs that have a combined power roughly equivalent to a 10-year-old's brand-new Quake-playing machine. I think you'll find very few sites on the Internet that are as popular as photo.net (700,000 unique visitors per month) served with just 1600 MHz of total crunching power for RDBMS + scripts + HTTP server. This is just barely adequate on a normal day. But on a Slashdot day, it is hopeless. I think we could handle the load if we had another 4 CPUs dedicated to scripts + HTTP service and left the E450 to do just Oracle.
The truly horrible thing is that my personal server runs from the same physical box as photo.net. This is a squirrel-powered Sun E450 (4 400 MHz processors) serving something like 4 million hits/day all of those hits resulting in script execution and many requiring Oracle queries. So you guys have shut down philip.greenspun.com and also photo.net. Load average was up to 26! Now it is down to 20....
Oh yes, for those of you who are tools-obsessed, you'll be pleased to know that the new curriculum is tools-neutral. When we started out, none of our students came in with experience building Web apps and none had used RDBMSes. But now a student might have had a summer job using PostgreSQL/mod_perl/Apache. I don't want to look at his Perl code but I'm not going to tell him he can't use his familiar tools to complete 6.916. If another student is jazzed up about Microsoft.NET and she wants to use C#, VB, and ASP.NET, more power to her. We'll look at her data model, page flow, and what the application accomplishes.
This is sort of the same progression as has been followed by ACS. When we started packaging up our apps we said "Wouldn't it be nice to have three versions, one for Oracle/AOLserver, one with a Microsoft Active Server Pages presentation layer, and one with a JSP presentation layer". But we were only five part-time programmers. So we never got around to doing the other versions and figured someone else might. Early in the summer of 2000, when I was still at aD, I twisted Jin's arm into leading a small team to do a 100% Java version. And that opened up all kinds of possibilities for new and different execution environments (Tomcat, built-into-Oracle, etc., etc.).
But guess what? Nobody cares. You might think C sucks but you still run your spreadsheet. When I give talks to people in big organizations they often are most impressed by WimpyPoint (see http://www.arsdigita.com/wp/). They've never seen anything like it before and think it is amazing that you can view, at a glance, the most recent work of a whole bunch of people in one orgazation. Having seen it, they ask how to get it. But nobody has ever asked what computer languages were used to build it.
I've written a small article on how we've refined our software engineering for Internet applications course over the years. It includes some discussion of what has been effective about ArsDigita University. See http://philip.greenspun.com/teaching/teaching-soft ware-engineering
(still in draft form so please email if you have comments/corrections).
Unix is a good example of a transition/hybrid system. It only took two guys to develop the initial prototype. For distribution, it required cooperation and coordination from the staff of the world's largest and most valuable company (AT&T). It was about as close to open source as anything was in its day (late 1970s and early 1980s). It got distributed largely along the lines of the Internet as it then existed (ARPAnet among the universities). And it only got distributed so cheaply because it had been done in Bell Labs rather than an AT&T product division. Let me quote from http://www.arsdigita.com/asj/professionalism:
"What were a programmer's options, then, [in the 1980s] if in fact craftsmanship proved to be an unsatisfying career goal? The only escape from the strictures of closed-source and secrecy was the university. A programmer could join a computer science research lab at a university where, very likely, he or she would be permitted to teach others via publication, source code release, and face-to-face instruction of students. However, by going into a university, where the required team of 50 would never be assembled to deliver a software product to market, the programmer was giving up the opportunity to work at the state of the art as well as innovate and teach."]
Bell Labs, where Unix was developed, is in many ways analogous to a university research lab. Engineers there have to give up on the idea of getting applications into the hands of end-users, though of course occasionally it has happened.
If you read http://www.arsdigita.com/asj/professionalism carefully you'll see that the article does not posit large engineering groups either today or any time in the past. It says that the closed-source packaged software business strategy ends up requiring large groups of people for physical packaging, traditional marketing, and fulfillment to retailers.
I'll give you a personal example. I developed a computer-aided engineering system back in the 1980s with one other programmer. This automated the design of large steel structures. We built it on a Symbolics Lisp Machine and therefore our productivity as programmers was extremely high. It took us about a year to get the thing working properly on a test problem of doing one person-year of engineering for an air-cooled heat exchanger (like a car radiator but the size of a house with about 20,000 parts). It probably took another two years before the 100th end-user was looking at his or her design taking shape on the screen. And we needed several full-time business people to convince people to buy the software and a Lisp Machine to run it (more Symbolics machines were sold to run this app than for any other purpose; then it got converted to run on Suns).
Contrast that with adding a feature to photo.net. If I were to write a new service on photo.net, it would take less than one hour to reach the 100-user mark (photo.net attracts more than 30,000 visitors per day across all services on the site).
At least for me, the world has changed a lot since I started programming (1976).
It was the job of every engineering school to churn out drones for cubicle farms... until open source and the Web came along. See http://www.arsdigita.com/asj/professionalism for how the options of a software engineer back in, say, 1985 were limited to working within a largeish product group. Only recently has it been possible for a couple of guys to do something huge like slashdot (for example).
What keeps me excited about teaching computer science is the fact that someone with a good CS background is in a great position to touch a lot of human lives. That wasn't true 20 years ago when computers were only in the back rooms of big companies. If someone with a first-rate CS education instead chooses to become a drone, that is sad but we don't live in Roman times (where a parent could kill his adult child if he didn't like the way he or she turned out; not sure if teachers were accorded the same privilege).
(Oh yes, and please don't sell Oracle employees short. They've got enough PhD computer scientists in their core RDBMS server programming team to equip a medium-size university. That said, I wouldn't want work in their huge bay-side office towers (no dogs allowed))
Actually we posted the content online in hopes primarily of helping other universities (both established ones and start-up one-year post-bacc programs like ADU). Also we figured it would be nice for individual learners. Finally we posted the content online because we post everything online! That's what the Internet is for!
Our aim was NOT to be "MIT without overhead". ADU is/was a post-baccalaureate program. Only college graduates could apply. Whereas for MIT undergrad CS, people who already have bachelor's degrees are excluded from even applying. The idea was to give people with bachelor's in other fields as much of the CS undergrad education as practical within 9 or 10 months.
MIT remains great for what it is (undergrad education for people done with high school; grad education for people done with college) and we never claimed that we would be able to do better than MIT at MIT's chosen mission.
See http://philip.greenspun.com/teaching/teaching-soft ware-engineering for some analysis of what worked well so far at ADU.
I've learned a bunch of things from this discussion. Let me say that I did not intend "Managing Software Engineers" to be the last word on the subject. That's why there is an "add a comment" link at the bottom of the page. My experience is limited to the handful of commercial and open-source software products that I've built plus some dozens of Web-based systems built on top of ACS in which I've been peripherally involved. My thoughts are intended to be useful for people who find themselves similarly situated and to be a magnet for contributed thoughts from people with better expertise than my own (see Chapter 1 of _Philip and Alex's Guide_).
The other day a Sloan MBA student asked me if being an entrepreneur was worth it. By that he meant the long hours, the risk, the pain of seeing one's baby disfigured by new management, venture capitalists, etc. I reflected for a moment and said that "Yes, it is still worth it. Being an entrepreneur means that you get to pick the people with whom you work."
So I guess that's what it boils down to for me. I never enjoyed working inside organizations built by others, even if I got to go out to the opera every week and spend long evenings at friends' houses. But it has always been fun for me to work at my own companies because I love the people that were pulled in (in the case of ArsDigita, there are my co-founders Jin Choi, Tracy Adams, Eve Andersson (also my girlfriend!), Bruce Keilin, and Aure Prochazka, plus a lot of really great people that joined just after the protoplasmic stage). Did it ever upset me to spend long hours with these guys? No! We were getting a lot done and being positive reinforced by the reaction from programmers worldwide, end users, and by our own growth in skills. Would we have been happier in the long run if we'd gone to work for Ciitbank in the IT department, married and had kids? Probably! I wrote about that in Travels with Samantha.
Anyway, I'm glad that my article spurred this lively debate! I don't want to be remembered for advocating a long work week. There is a lot more to the article and I certainly wouldn't advocate long hours to anyone who didn't love his or her job and wasn't learning every day.
To answer your question... we've never hired contractors, unless you count the sysadmin geniuses at techsquare.com who got us started stably. And they are probably cheaper per hour than our Ferrari-driving, Cape house lounging, dual-LCD monitored employees.
I don't think it is stock options that motivated our people. We worked harder at the beginning and we didn't have any stock options or dreams of getting rich. We just wanted to get our software out the door and into folks' hands (as of today about 10,000 programmers have downloaded ACS). We wanted to make our clients and end-users happy (the CEO of Siemens claims that the knowledge management system we built for them helped them win a $1 billion contract).
Now that we have venture capital and stock options and dreams of an IPO, people are working less hard on average!
Microsoft is actually a good example of small teams, each person individually strong and working hard. For example, Netscape had nearly unlimited resources in terms of bodies. Microsoft crushed them and built a much better browser (IE) with a team of only about 30 developers.
I'm glad that you love your wife and kid but it isn't the case that 99% of the world's great innovations come from people whose families are their #1 priority. Very few MIT students, for example, are married. It would probably be possible to construct a research university specifically for those who want to work 9-5 and put family first. But as most of the people attracted to research universities aren't married, it makes more sense to figure out how to get everyone charged up and cranking. At MIT we DO care about our students. But we don't demonstrate it by encouraging them to knock off work at 5:00 pm and go home to the families that they don't have!
A family doctor lives next door to us on the Cape (Chatham). I think he is around 65 years old and is considered one of the best diagnosticians in the area. He gets into work every morning at 0615 and can stay until 5 in the afternoon. He also works evenings and weekends as necessary for his patients. I don't think that you'll find too many doctors or lawyers who are considered to have made important contributions to their communities who have worked 9-5 their entire life.
Almost all doctors work extremely hard during the career-building years about which I was writing (age 22-30).
It seems okay to me. I've been using the arsdigita.com site all day. Anyway, there isn't much hardware behind arsdigita.com (normally gets no more than 100,000 hits/day). We never said that ACS was going to handle more traffic per CPU than a non-personalized system that doesn't hit the RDBMS. As for AOLserver and Tcl, you don't have to use either. ACS Java is released and available from http://www.arsdigita.com/download/ It contains not a single line of Tcl code (same old SQL and PL/SQL and Java-in-the-database core plus a 100% Java presentation layer). It should work with any Java application server and any Web server (or no Web server at all; you can run the whole thing from within Oracle 8.1.7 and its built-in Web server).
Actually ArsDigita pays most entry-level workers $100K (these are fresh MIT CS grads). The reason cost of living is high in cities like Cambridge is that Cambridge is a great place to live. You can learn from almost anyone that you run into on the street. They are likely to have an Ivy League education. They are likely to have done something interesting recently and be willing to tell you about it. If your idea of the great life is a big-screen television, 100 channels of Cable, and a big house with blank walls to hold the TV, you're right the cost of living is lower in Peoria. But if your idea of the great life is finding interesting people to talk to, lectures and performances by the world's best and brightest, a city like Manhattan might be a better choice (despite the expense).
I'm not trying to force anyone to go on a death march! But the programmers with whom I hang out long to do something innovative and creative, to solve a problem that nobody has solved before. It is tough under those circumstances to rely on a project manager to map out everything that must be done. It is also tough to keep people from pounding away at a problem until it is solved.
Most of the people at ArsDigita are young. They have no families. They have no personal reputation. Find me a 35-year-old who has accomplished a lot IN ANY FIELD, who has changed the world in some positive way, and who has never worked long hours. The articles I put on my various Web sites are not intended to help people who just want to live a quiet comfortable life (I'm not an expert on this). They are intended to help young people turn into Linus Torvalds or Richard Stallman or Dan Bricklin and Bob Frankston (Visicalc).
I don't even put "PhD" or "Dr." on my business card (I admit to having a photo of little Alex on the back, though). You're right that a PhD per se = useless. But at ArsDigita we actually have managed to get a lot of work out of people with PhDs. Much of the edge comes from the screening process. Someone with a PhD from Caltech is going to be very smart and very persistent. He or she will have managed to plan, produce, and publish a long-ish document (the engineers with the most influence are usually those who are willing to write). This is why companies like McKinsey recruit PhDs from top schools. But just like McKinsey, the first thing that we have to do at ArsDigita is deprogram the PhD. We say "here you ship your product every two months, not every two years" and "here you write up your results over a weekend, not over a season" and "here we do look for the grand elegant complete solution but we try to get to it via incremental releases rather than wait quietly for 10 years".
Thanks, Bob. Your point was sort of an unstated assumption of mine. At ArsDigita we do tend to get fairly young people who are very bright. They want to do something that will impress their classmates from MIT or UCLA or Caltech or wherever. The key to successful management is to provide an inspiring goal that these guys and gals can buy into and then a working environment that lets them achieve the goal. It does result in some long hours but they have 5 weeks/year to recover. If they get sick of it they can always join a slacker company and work 40 hours/week.
Peopleware has some useful advice but it is fundamentally about corporate IT projects where there isn't any competition. In the real world (cf. Silicon Valley), a few months of slippage can very well mean death. The competition may not necessarily be devoted to the 40-hour week. I find it interesting that so many people on Slashdot advocate a short working week because their family time is so precious. These are the really focussed workers who get everything done in 40 hours and then go home to accolades from their grateful coworkers. How exactly do they maintain their productivity at work? Apparently by surfing Slashdot:-)
Slashdot Mirror
One of the things that we try to teach in the class (textbook is online at http://philip.greenspun.com/internet-application-w orkbook/ if you're curious to see what the students suffer through) is that being a good code monkey/CS nerd isn't sufficient to function well as an engineer. We try to give the students some experience with taking vague client specs and turning them into precise requirements, with presenting their work clearly, with constructively criticizing others' work in meetings, with conducting and learning from user testing, etc. The rationale for this is laid out in http://philip.greenspun.com/teaching/one-term-web
So it was actually very gratifying that our guest speakers came in and demonstrated that state-of-the-art American IT development projects no longer involve plain-old-programmers in America. Our students need to learn this early so that they can plan their careers and further education accordingly.
Thanks, guys, for giving the Sun E450 (philip.greenspun.com shares that 5-year-old machine with photo.net) and Harvard's little blog server a workout.
Don't assume from my posting that I'm in any personal hurry to learn VB and PHP! But having so many bright young people in 6.171 gives one a fun opportunity to take a high-level look at the programming tools of the moment.
What would I personally prefer to use? The same thing that I would have wanted to use 10 years ago: Common Lisp, CLOS, plus an ML-like type inferencing compiler/error checker. I find this preference, shameful, however, and try to keep it concealed from young people.
Just yesterday I ran into a friend. She's a 23-year-old graduate student in computer science at Harvard. Conversation rolled around to programming tools. Unprompted she said "What I think would be best is Common Lisp Object Systemw with a modern type system". I was stunned. I thought it was only dinosaurs like me that clung to Lisp.
I had a second ephiphany for the week... Believing that Lisp circa 1982 plus some mid-1980s ML tricks thrown in is better than all of the new programming tools (C#, Java) that have been built since then is sort of like being a Holocaust denier.
Thank you for pointing out the fact that my last day of influence at ArsDigita was in April 2000 (nearly two years ago). Here I am, a quiet retired schoolteacher, and people are still beating me up for opinions and attitudes that they think I might have had at some time in the preceding decade. I try to leave the old writings up on my Web site so as to be a good Internet citizen and not break links. But people take things that I wrote in the early 1990s and assume that I still hold these opinions. To take a non-controversial example: I struggled to get Windows NT 3.51 working and couldn't. I vented my rage against Bill Gates in some site. But now I love Microsoft. About half of my students use the .NET tools and are extremely productive. I'm running Windows XP on two computers at home and haven't spent more than 15 minutes on sysadmin. The interesting research in online communities is all being done at ... you guessed it, Microsoft Research.
I'll come back home after walking the dog or get out of the bathtub where I've been reading New Yorker magazine. I amble over to the computer and there is email from someone yelling at me to "Get a life and stop picking on Bill Gates you miserable envious poverty-stricken grad student."
Online communities only work if someone puts forth a strong opinion against which others can react. This is why photo.net works so well. We write "Nikon autofocus sucks" and that yields a huge pile of interesting comments from Nikon users about how they've been able to get great photos with the Nikon AF system. But when you have people with long memories and/or Google to dredge up these ancient opinions, which may never have been held strongly to begin with, life gets kind of confusing.
Mea culpa. I shouldn't have advised you to get into the Internet craze. I started loving the Internet because it lets people learn from other people. That was back in 1976 when I began using ARPAnet. And what is amazing is that I can still wonder in 2002 at how useful Internet is. For example, I've learned so much lately from other pilots whom I've met on the Internet and could never have connected with as a practical matter otherwise. One of my interests is in getting a Stemme S-10 motorglider. They build 18 per year for sale worldwide. None are based in the Boston area. But I found the owners and their impressions of the aircraft on rec.aviation.* groups (thanks to dejanews (Google Groups now), of course). Then I emailed them and they were very helpful to me, despite my status as a novice pilot.
At the end of the 1990s this wonder turned into something more akin to fever. And, along with the pressures that come from managing a company that was growing 1000%/year, that didn't bring out the most attractive parts of my personality. Bottom line: I'm sorry for giving you unsolicited career advice. If you come to the US and you don't value your life too much, I'll take you up on a sightseeing flight! (with Alex in the back seat, of course)
[And don't sell Internet apps short just yet. With some improvements in infrastructure, such as ubiquitous high-speed wireless IP connectivity and reasonable conversational speech recognition software, a lot of interesting new applications should be enabled. Of course, it might be 5 or 10 years before we have this infrastructure in place, which is why I'm working with young people at MIT instead of trying to build the stuff myself.]
It is gratifying to me that people still like photo.net and learn on the site (up to more than 1 million user-visits per month). But I also find it alarming that people blame me when the site is dead or slow. I was never an expert Unix sysadmin or Oracle dba. Being an MIT Lisp Machine programmer, I always hated Unix. I learned as little as possible about. As for Oracle, it is a wonderful tool with a great abstraction barrier (SQL) but I don't know of anyone who would say "I had a lot of fun last night administering my Oracle installation."
If you thought the site was a pig in 1997 you can blame me because I was personally involved with some of the sysadmin/dbadmin stuff back then. But I'm retired now! I do the stuff that (a) is fun, and (b) that only I can do. So I write articles for photo.net and critique learning photographers' submissions to the image critique forum but I don't try to beat Unix and Oracle into submission. I drove to Nova Scotia for the entire month of September. I spent the rest of the fall in art museums, Civil War battlefields, and National Parks between Boston and Texas. When I got back from three solid months on the road, what did I find? Email from people blaming me for something that they didn't like on some Internet server somewhere.
Rajeev and his merry band will eventually slay the sysadmin dragon and photo.net will be responsive once more. But when it happens you should thank him and not me!
Your recollection of my comment about Slashdot is ALMOST correct (I'm glad someone in the room was listening anyway). I mentioned that the guys who wrote Slashdot weren't professional programmers but that they built a great site by carefully tailoring the software to the evolving users' needs. I mentioned that some people have thrown rocks at the way it was coded (but not me since I've actually never looked at it; I have enough trouble with my own source code!) but that it really doesn't matter since data model plus page flow are the only things that affect the user experience.
.NET or openacs.org, etc. It is much more relaxing to limit myself to teaching one semester per year at MIT (6.171; Software Engineering for Internet Applications -- to avoid the kinds of tool flame wars that one sees on Slashdot we don't mandate the use of any particular tools except an ACID-compliant RDBMS (practically speaking that means Postgres, Oracle, or Microsoft SQL Server)).
You must have caught one of my last one-day courses. I've stopped doing them because I think that enough people have heard what I have to say. The ideas are out there for anyone who needs them, either in a document I've written or in the heads of former students, users of photo.net, embodied in products such as Microsoft
I spend the rest of my time taking flying lessons (doing my primary training now, airplane arrives on March 1, starting instrument training in March, leaving for a 3-month Alaska trip on June 1). There is nothing quite so humbling as learning to fly. You show up at the airport and, with 60 hours of experience, are almost surely the most incompetent person on the property. If your instructor is "old-school" (mine is) there is an intermittent shouting in your headset ("more right rudder"; "what do you think you're doing?"; "don't you see where the nose is pointed?"). Everything is happening way too fast.
Anyway, flying sure makes arguments about the proper way to code up a Web script (which after all is simply merging the results of an SQL query with an HTML template) seem irrelevant. Every time you bring the airplane down in a crosswind you're putting your life and your passengers' lives on the line. Screw up a bit and you've got yourself a $228,000 pile of vaguely airplane-shaped scrap. Screw up more than a bit and you've dug yourself an airplane-shaped grave right on the runway.
Anyway I guess I should close by thanking the Slashdot guys once again for their great achievement in building this site. I often use it as an example to students of how one does not need or want too many collaboration areas on a site. Users need to be able to find the discussion. What I like about Slashdot is that one need only check the front page in order to see what is being discussed.
Yes, well photo.net was formerly run off smaller machines. But we didn't have 100,000 registered users. Or the photo critique forum. Or "hot or not". You can't serve the entire world with 4 CPUs that have a combined power roughly equivalent to a 10-year-old's brand-new Quake-playing machine. I think you'll find very few sites on the Internet that are as popular as photo.net (700,000 unique visitors per month) served with just 1600 MHz of total crunching power for RDBMS + scripts + HTTP server. This is just barely adequate on a normal day. But on a Slashdot day, it is hopeless. I think we could handle the load if we had another 4 CPUs dedicated to scripts + HTTP service and left the E450 to do just Oracle.
The truly horrible thing is that my personal server runs from the same physical box as photo.net. This is a squirrel-powered Sun E450 (4 400 MHz processors) serving something like 4 million hits/day all of those hits resulting in script execution and many requiring Oracle queries. So you guys have shut down philip.greenspun.com and also photo.net. Load average was up to 26! Now it is down to 20....
Oh yes, for those of you who are tools-obsessed, you'll be pleased to know that the new curriculum is tools-neutral. When we started out, none of our students came in with experience building Web apps and none had used RDBMSes. But now a student might have had a summer job using PostgreSQL/mod_perl/Apache. I don't want to look at his Perl code but I'm not going to tell him he can't use his familiar tools to complete 6.916. If another student is jazzed up about Microsoft .NET and she wants to use C#, VB, and ASP.NET, more power to her. We'll look at her data model, page flow, and what the application accomplishes.
This is sort of the same progression as has been followed by ACS. When we started packaging up our apps we said "Wouldn't it be nice to have three versions, one for Oracle/AOLserver, one with a Microsoft Active Server Pages presentation layer, and one with a JSP presentation layer". But we were only five part-time programmers. So we never got around to doing the other versions and figured someone else might. Early in the summer of 2000, when I was still at aD, I twisted Jin's arm into leading a small team to do a 100% Java version. And that opened up all kinds of possibilities for new and different execution environments (Tomcat, built-into-Oracle, etc., etc.).
But guess what? Nobody cares. You might think C sucks but you still run your spreadsheet. When I give talks to people in big organizations they often are most impressed by WimpyPoint (see http://www.arsdigita.com/wp/). They've never seen anything like it before and think it is amazing that you can view, at a glance, the most recent work of a whole bunch of people in one orgazation. Having seen it, they ask how to get it. But nobody has ever asked what computer languages were used to build it.
I've written a small article on how we've refined our software engineering for Internet applications course over the years. It includes some discussion of what has been effective about ArsDigita University. See http://philip.greenspun.com/teaching/teaching-soft ware-engineering
(still in draft form so please email if you have comments/corrections).
Unix is a good example of a transition/hybrid system. It only took two guys to develop the initial prototype. For distribution, it required cooperation and coordination from the staff of the world's largest and most valuable company (AT&T). It was about as close to open source as anything was in its day (late 1970s and early 1980s). It got distributed largely along the lines of the Internet as it then existed (ARPAnet among the universities). And it only got distributed so cheaply because it had been done in Bell Labs rather than an AT&T product division. Let me quote from http://www.arsdigita.com/asj/professionalism:
"What were a programmer's options, then, [in the 1980s] if in fact craftsmanship proved to be an unsatisfying career goal? The only escape from the strictures of closed-source and secrecy was the university. A programmer could join a computer science research lab at a university where, very likely, he or she would be permitted to teach others via publication, source code release, and face-to-face instruction of students. However, by going into a university, where the required team of 50 would never be assembled to deliver a software product to market, the programmer was giving up the opportunity to work at the state of the art as well as innovate and teach."]
Bell Labs, where Unix was developed, is in many ways analogous to a university research lab. Engineers there have to give up on the idea of getting applications into the hands of end-users, though of course occasionally it has happened.
If you read http://www.arsdigita.com/asj/professionalism carefully you'll see that the article does not posit large engineering groups either today or any time in the past. It says that the closed-source packaged software business strategy ends up requiring large groups of people for physical packaging, traditional marketing, and fulfillment to retailers.
I'll give you a personal example. I developed a computer-aided engineering system back in the 1980s with one other programmer. This automated the design of large steel structures. We built it on a Symbolics Lisp Machine and therefore our productivity as programmers was extremely high. It took us about a year to get the thing working properly on a test problem of doing one person-year of engineering for an air-cooled heat exchanger (like a car radiator but the size of a house with about 20,000 parts). It probably took another two years before the 100th end-user was looking at his or her design taking shape on the screen. And we needed several full-time business people to convince people to buy the software and a Lisp Machine to run it (more Symbolics machines were sold to run this app than for any other purpose; then it got converted to run on Suns).
Contrast that with adding a feature to photo.net. If I were to write a new service on photo.net, it would take less than one hour to reach the 100-user mark (photo.net attracts more than 30,000 visitors per day across all services on the site).
At least for me, the world has changed a lot since I started programming (1976).
It was the job of every engineering school to churn out drones for cubicle farms... until open source and the Web came along. See http://www.arsdigita.com/asj/professionalism for how the options of a software engineer back in, say, 1985 were limited to working within a largeish product group. Only recently has it been possible for a couple of guys to do something huge like slashdot (for example).
What keeps me excited about teaching computer science is the fact that someone with a good CS background is in a great position to touch a lot of human lives. That wasn't true 20 years ago when computers were only in the back rooms of big companies. If someone with a first-rate CS education instead chooses to become a drone, that is sad but we don't live in Roman times (where a parent could kill his adult child if he didn't like the way he or she turned out; not sure if teachers were accorded the same privilege).
(Oh yes, and please don't sell Oracle employees short. They've got enough PhD computer scientists in their core RDBMS server programming team to equip a medium-size university. That said, I wouldn't want work in their huge bay-side office towers (no dogs allowed))
Actually we posted the content online in hopes primarily of helping other universities (both established ones and start-up one-year post-bacc programs like ADU). Also we figured it would be nice for individual learners. Finally we posted the content online because we post everything online! That's what the Internet is for!
Our aim was NOT to be "MIT without overhead". ADU is/was a post-baccalaureate program. Only college graduates could apply. Whereas for MIT undergrad CS, people who already have bachelor's degrees are excluded from even applying. The idea was to give people with bachelor's in other fields as much of the CS undergrad education as practical within 9 or 10 months.
t ware-engineering for some analysis of what worked well so far at ADU.
MIT remains great for what it is (undergrad education for people done with high school; grad education for people done with college) and we never claimed that we would be able to do better than MIT at MIT's chosen mission.
See http://philip.greenspun.com/teaching/teaching-sof
I've learned a bunch of things from this discussion. Let me say that I did not intend "Managing Software Engineers" to be the last word on the subject. That's why there is an "add a comment" link at the bottom of the page. My experience is limited to the handful of commercial and open-source software products that I've built plus some dozens of Web-based systems built on top of ACS in which I've been peripherally involved. My thoughts are intended to be useful for people who find themselves similarly situated and to be a magnet for contributed thoughts from people with better expertise than my own (see Chapter 1 of _Philip and Alex's Guide_).
The other day a Sloan MBA student asked me if being an entrepreneur was worth it. By that he meant the long hours, the risk, the pain of seeing one's baby disfigured by new management, venture capitalists, etc. I reflected for a moment and said that "Yes, it is still worth it. Being an entrepreneur means that you get to pick the people with whom you work."
So I guess that's what it boils down to for me. I never enjoyed working inside organizations built by others, even if I got to go out to the opera every week and spend long evenings at friends' houses. But it has always been fun for me to work at my own companies because I love the people that were pulled in (in the case of ArsDigita, there are my co-founders Jin Choi, Tracy Adams, Eve Andersson (also my girlfriend!), Bruce Keilin, and Aure Prochazka, plus a lot of really great people that joined just after the protoplasmic stage). Did it ever upset me to spend long hours with these guys? No! We were getting a lot done and being positive reinforced by the reaction from programmers worldwide, end users, and by our own growth in skills. Would we have been happier in the long run if we'd gone to work for Ciitbank in the IT department, married and had kids? Probably! I wrote about that in Travels with Samantha.
Anyway, I'm glad that my article spurred this lively debate! I don't want to be remembered for advocating a long work week. There is a lot more to the article and I certainly wouldn't advocate long hours to anyone who didn't love his or her job and wasn't learning every day.
To answer your question... we've never hired contractors, unless you count the sysadmin geniuses at techsquare.com who got us started stably. And they are probably cheaper per hour than our Ferrari-driving, Cape house lounging, dual-LCD monitored employees.
I don't think it is stock options that motivated our people. We worked harder at the beginning and we didn't have any stock options or dreams of getting rich. We just wanted to get our software out the door and into folks' hands (as of today about 10,000 programmers have downloaded ACS). We wanted to make our clients and end-users happy (the CEO of Siemens claims that the knowledge management system we built for them helped them win a $1 billion contract).
Now that we have venture capital and stock options and dreams of an IPO, people are working less hard on average!
Microsoft is actually a good example of small teams, each person individually strong and working hard. For example, Netscape had nearly unlimited resources in terms of bodies. Microsoft crushed them and built a much better browser (IE) with a team of only about 30 developers.
I'm glad that you love your wife and kid but it isn't the case that 99% of the world's great innovations come from people whose families are their #1 priority. Very few MIT students, for example, are married. It would probably be possible to construct a research university specifically for those who want to work 9-5 and put family first. But as most of the people attracted to research universities aren't married, it makes more sense to figure out how to get everyone charged up and cranking. At MIT we DO care about our students. But we don't demonstrate it by encouraging them to knock off work at 5:00 pm and go home to the families that they don't have!
A family doctor lives next door to us on the Cape (Chatham). I think he is around 65 years old and is considered one of the best diagnosticians in the area. He gets into work every morning at 0615 and can stay until 5 in the afternoon. He also works evenings and weekends as necessary for his patients. I don't think that you'll find too many doctors or lawyers who are considered to have made important contributions to their communities who have worked 9-5 their entire life.
Almost all doctors work extremely hard during the career-building years about which I was writing (age 22-30).
It seems okay to me. I've been using the arsdigita.com site all day. Anyway, there isn't much hardware behind arsdigita.com (normally gets no more than 100,000 hits/day). We never said that ACS was going to handle more traffic per CPU than a non-personalized system that doesn't hit the RDBMS. As for AOLserver and Tcl, you don't have to use either. ACS Java is released and available from http://www.arsdigita.com/download/ It contains not a single line of Tcl code (same old SQL and PL/SQL and Java-in-the-database core plus a 100% Java presentation layer). It should work with any Java application server and any Web server (or no Web server at all; you can run the whole thing from within Oracle 8.1.7 and its built-in Web server).
Actually ArsDigita pays most entry-level workers $100K (these are fresh MIT CS grads). The reason cost of living is high in cities like Cambridge is that Cambridge is a great place to live. You can learn from almost anyone that you run into on the street. They are likely to have an Ivy League education. They are likely to have done something interesting recently and be willing to tell you about it. If your idea of the great life is a big-screen television, 100 channels of Cable, and a big house with blank walls to hold the TV, you're right the cost of living is lower in Peoria. But if your idea of the great life is finding interesting people to talk to, lectures and performances by the world's best and brightest, a city like Manhattan might be a better choice (despite the expense).
I'm not trying to force anyone to go on a death march! But the programmers with whom I hang out long to do something innovative and creative, to solve a problem that nobody has solved before. It is tough under those circumstances to rely on a project manager to map out everything that must be done. It is also tough to keep people from pounding away at a problem until it is solved.
Most of the people at ArsDigita are young. They have no families. They have no personal reputation. Find me a 35-year-old who has accomplished a lot IN ANY FIELD, who has changed the world in some positive way, and who has never worked long hours. The articles I put on my various Web sites are not intended to help people who just want to live a quiet comfortable life (I'm not an expert on this). They are intended to help young people turn into Linus Torvalds or Richard Stallman or Dan Bricklin and Bob Frankston (Visicalc).
I don't even put "PhD" or "Dr." on my business card (I admit to having a photo of little Alex on the back, though). You're right that a PhD per se = useless. But at ArsDigita we actually have managed to get a lot of work out of people with PhDs. Much of the edge comes from the screening process. Someone with a PhD from Caltech is going to be very smart and very persistent. He or she will have managed to plan, produce, and publish a long-ish document (the engineers with the most influence are usually those who are willing to write). This is why companies like McKinsey recruit PhDs from top schools. But just like McKinsey, the first thing that we have to do at ArsDigita is deprogram the PhD. We say "here you ship your product every two months, not every two years" and "here you write up your results over a weekend, not over a season" and "here we do look for the grand elegant complete solution but we try to get to it via incremental releases rather than wait quietly for 10 years".
Thanks, Bob. Your point was sort of an unstated assumption of mine. At ArsDigita we do tend to get fairly young people who are very bright. They want to do something that will impress their classmates from MIT or UCLA or Caltech or wherever. The key to successful management is to provide an inspiring goal that these guys and gals can buy into and then a working environment that lets them achieve the goal. It does result in some long hours but they have 5 weeks/year to recover. If they get sick of it they can always join a slacker company and work 40 hours/week.
Peopleware has some useful advice but it is fundamentally about corporate IT projects where there isn't any competition. In the real world (cf. Silicon Valley), a few months of slippage can very well mean death. The competition may not necessarily be devoted to the 40-hour week. I find it interesting that so many people on Slashdot advocate a short working week because their family time is so precious. These are the really focussed workers who get everything done in 40 hours and then go home to accolades from their grateful coworkers. How exactly do they maintain their productivity at work? Apparently by surfing Slashdot :-)