Slashdot Mirror
Being Enron's SysAdmin
CowboyRobot writes "FreeBSD's Kirk McKusick has a long interview with Enron's former SysAdmin, Jarod Jenson, where he describes the nuts and bolts of working in and managing such a large-scale operation." From the article: "EnronOnline was a Web-based trading application. We had several hundred, even thousands of commodities that we would price in realtime, the same way that equities are priced. We were trying to push realtime pricing information out to clients who could do instantaneous transactions on them. People who are familiar with financial markets--the commodity markets--would recognize EnronOnline as sort of the same thing. We had a lot of the same issues that the markets had trying to push out realtime data--not only within our local network but also to the customers--as quickly as we could globally, and trying to make sure that what every trader saw on the screen matched what every company in the world had on theirs."
I wonder if EnronOnline had realtime showing of the stockholders assets flying out the window.
There is a large collection of email from Enron vailable online. It has been usefull for research in natural language processing, text classification, and data mining. Check it out here: http://www-2.cs.cmu.edu/~enron/.http://www-2.cs.cm u.edu/~enron/
man shred(1)
Karma: Chameleon - mostly influenced by bad '80s New Wave music
"rm -rf /" on everyone's machines.
Very interesting discussion about the relationship between developers and admins in TFA. My own take is that it's basically the only useful thing that team leads or managers can do anything about by setting the incentive structure for both to be somewhat similar.
Essentially, if a developer's job relies on the same thing that an admin's job relies on (that is, stable, secure and reliable operations) then you have the foundation for harmony. If a developer's job relies on features and new functionality at the expense of stability, security and reliability, you have a recipe for hostility.
You can tell the priorities at a company by how cranky its admins are.
On the other hand, admins need to be open and available to developers, offering advice on OS, hardware, infrastructure, etc. and be able to clearly define the requirements for SSR so that any new designs or requirements can be supported from day 1.
Oh, and a great way to get documentation from developers: give all their cell#s to the admins, so when something breaks at 3am on a sunday and there's no documentation, the admin has a little company. A few calls like that and developers can write some pretty handy documentation!
IT Job Offer From Enron:In the first I was offered a trading systems developer position. They used PowerBuilder on Windows for their trading systems - nothing wrong with that. But the IT Director kept talking about how quickly the laws changed that concerned their energy trading. I enquired as to how they maintained their business rules. He replied roughly that "it's all in code; when the laws change we rewrite the code." I asked what they would do if there were an audit say, 6 months later after they'd gone through 4 versions of code. He replied that they would set up a standalone network, load the old software and restore snapshots of the database and rerun the old data.
Naturally I asked if he had heard of rule-based systems and hinted that, if they had a dynamic rulebase architecture, they could avoid the recoding and versioning. Instead they could have a much simpler system and better controls. He said he had never heard of such a thing and couldn't see how it could be done in their environment. But he was interested.
Seeing that their viewpoint was extremely short-term and unenlightened, I turned down Enron's offer. Many of my cohorts at the time said I was a fool, but the IT situation at Enron seemed to be impoverished in thought although rich in resources. In time I realized my hunches were correct.
Encounter With Enron Broadband:Flash forward to 2 years later. I've gone completely over to WWW development and to FOSS. I'm attending a Microsoft event on IIS mostly out of curiosity. Next to me is an outspoken individual who works for Enron Broadband. He speaks endlessly of the benefits of IE and ActiveX. I ask if Enron uses ActiveX much; he replies that they are tied to IE and that ActiveX is a necessary part of their architecture. I ask about Apache, Netscape, and FOSS; he replies that they have no capability in those areas to his knowledge. I once again decide that Enron has somehow missed the boat.
Enron never carried significant loads on their web servers. Their limit was the number of energy traders in the U.S., which is a relatively small number. Today, small sites do much more than Enron did with fewere resources. There are few useful lessons to be learned from Enron's IT group or their BroadBand division fiascoes.
Frankly, all of my contact with Enron has shaped my opinion as such that if I had worked at Enron, I'd be reluctant to put it on my resume. I'd rather have a sabattical in Tibet, stay in rehab, or time in the peace core on there instead. Much of the criminal activity at Enron was IT-based. When your accouting system is an application, your IT had to know that the books were being cooked. The fact of the matter is that they chose not to report it. Instead,a lowly bean counter blew the whistle on all of them. Now that I know that this man has come out and proudly announced that he was part of such an organization, I would seriously recommend that he not be hired for any kind of a position of trust. I don't see a lot here to be proud of, ethically or morally.
Anyone who's been an SA for any length of time knows that being an SA carries an ethical and moral burden. Just because you accidentally read an email while trying to fix something on the mail server doesn't mean you can go gossip about it. If you happen to see a file that has private contents on someone's desktop, you don't go gossip about it. If you happen to find kiddie porn, you inform the FBI. There are rules to this business. Annoucing that you've violated the most important ones doesn't exactly make you a desirable candidate.
How can anyone, who claims to be providing web services for "the public", tie themselves to an IE/Windows/Active X architecture? I work for a university and, while our web traffic is quite atypical, IE accounts for only 1/2 of our traffic and has for some time. Windows only accounts for about 60% of our operating systems. Since we seem to "lead the curve" on rest of the net, I would suspect that in the next year or two most sites, at least ones that take international traffic, will start seeing a similar shift in their traffic.
The whole paradigm of web services is that it's supposed to work on any OS, any browser, etc. without the need for any specific client software. If you're web application isn't browser/OS agnostic, you've totally missed the boat. Frankly that's not anything to be bragging about technically either.
So, to sum it all up. He's bragging about being part of an ethically corrput and technically deficient company and wondering why he's not got a job.
2 cents,
Queen B
HDGary secures my bank
This was modded insightful? Bah. You statement shows a complete lack of understanding of the impact of the Enron collapse on many of its employees. The fact that he got another job is a simplistic, and quite frankly juvenile view of a major financial event in many peoples lives. Almost everyone who worked for Enron got another job. The Houston job market is pretty good, and particularly good for people with solid IT credentials. There was very little stigma associated with having Enron on your resume unless you were in the inner circle of Lay/Skilling/Fastauw.
The real impact was that many people at Enron had the vast majority of their personal savings in Enron stock or other Enron securities. The company management strongly pushed employees to do that, and there was significant corporate cultural pressure to invest ALL of your 401K in Enron stock. When the company tanked, people who had worked for years for Enron or one of the companies Enron had acquired suddenly went from having accumulated enough wealth to be close to retirement to having to start over. Plenty of stories of folks who at age 50 suddenly found themselves going from being worth millions on paper to having no life savings and no fiscal security. This happened to folks in a matter of weeks, and while it was happening compnay management was encouraging those employees to stay the course - hold the stock - as it would come back. It changed the lives of tens of thousands of people. I have one acquaintance who went from having 85,000 in a 401K, who got a settlement check for 43.00 from the bankruptcy court. Fourtunately he had other monies and the Enron investments really didn't change much for him long term, but I mention it to give you a sense of the scale of the collapse for folks.
The 'he got a job' comment may resonate with you folks who are young, have no long term obligations like a family, and are living paycheck to paycheck with no view of your future beyond how fast can I save up for a new Athlon dual core, but for those of us whose lives are a bit more established, it stinks.
Disclaimer - don't work for Enron, never did, no one in my family or close circle of friends did/does. But I do live in Houston and have seen what its done to good, honest people who did nothing wrong but believe the propaganda delivered by the people for which they worked.
"Would you, could you, with a goat?" Dr Seuss
I also worked at Enron, and we were trying to get money and hardware resources pumped into several open source projects, namely the OpenBSD project, because we were one of the companies trying to use it in our production environment. Unfortunately my layoff came before we could get anything signed, at which point there was nothing left to give.
Enron and its subsidiaries had a lot of great people working for them, and it was one of the few places where bright minds/tech people could get promoted for being great at what they did, and it didn't matter if you were some fresh-faced college graduate recruited for our analyst program or a guy who dropped out of high school but was brilliant at programming. It also left a lot of those bright people in a financial and professional lurch. What was worse than the sudden loss of employment was the loss of professional stature. I had prospective employers, even those in the energy trading business, actually deny me employment because I worked at Enron, as if I had something to do with their downfall as an upper level tech person. A lot of people thought we were part of some vast conspiracy, when many of us were the ones who got screwed in the process of Enron's downfall just like the other stock shareholders. The only difference between us and them is it was significantly more perseonal.
Remember the Alamo, and God Bless Texas...
These can't be real names ;-)
No. That's their porn names.
Is anybody else seeing the humor in an application called "EOL"? (end-of-life) ?
His points about 'taking ownership' of the problem appear to be spot-on right, when people take ownership of a problem it provides a better chance to solve the problem. This comes straight out of the book In Search of Excellence as a method of building a better company. Difficulties and loss of effort happen when the general environment is one of "it's not my problem." When one takes ownership of problems in order to fix them no matter where the actual problem is you can produce excellent results. But again, if you're not looking in the right place to find the solution, chances are you won't find it.
The lessons of history teach us - if they teach us anything - that nobody learns the lessons that history teaches us.
99% full: we have scripts thank you. We have tried many different policies to clean up once we are alerted: asking nicely (they ignore us), asking harshly (they complain), being draconian (removing older, bigger files) people get sarcastic at you (here, I give you 100 bucks, buy a new disk), imposing limits (everybody has a good reason to need more than their colleagues), doing careful audits (discovering all the porn, movies, pictures, unauthorized software, etc., people, kid you not, complain about their privacy being violated.)
So after a while we just turn the damn 99% script off. Once the filesystem gets full we send an email to all the parties involved to sort it out. Say what you may, this works better than all the approaches above. We could try group quotas or smaller volumes assigned to smaller teams, but live is too short and I am sure people would still find problems with this or any other solution.
Disasters at 3am: you know, shit really happens, I know Mr Rumsfeld has discredited this very true notion, but shit does happen. You can have full redundancy, careful considerations to avoid downtime even people working to cover on site 24x7. Shit still will happen. So get over yourself and be grateful that there are people out there interested on this kind of job, which we can see form your post, get no recognition at all in spite of the users being stupid asses.
IANAL but write like a drunk one.