Slashdot Mirror
Being Enron's SysAdmin
CowboyRobot writes "FreeBSD's Kirk McKusick has a long interview with Enron's former SysAdmin, Jarod Jenson, where he describes the nuts and bolts of working in and managing such a large-scale operation." From the article: "EnronOnline was a Web-based trading application. We had several hundred, even thousands of commodities that we would price in realtime, the same way that equities are priced. We were trying to push realtime pricing information out to clients who could do instantaneous transactions on them. People who are familiar with financial markets--the commodity markets--would recognize EnronOnline as sort of the same thing. We had a lot of the same issues that the markets had trying to push out realtime data--not only within our local network but also to the customers--as quickly as we could globally, and trying to make sure that what every trader saw on the screen matched what every company in the world had on theirs."
Must have been a big headache to manage something that big.
I wonder if EnronOnline had realtime showing of the stockholders assets flying out the window.
I think the Sysadmin had allot on his plate setting a goal like that. But it is a possible and probable goal to attempt and well worth it.
... whos "printer freindly" webpage ends as -
KM
In terms of performance mistakes, the operations guy says it's the developer's fault, the developer says it's the operations guy's fault. Whose fault is it?
JJ
whereas the non-"printer freindly" version continues with the interview, this is why I don't join the ACM, they can't even take 9 seperate pages and format them to be printed, I now re-christen the ACM - ABM, Association of Broken Minds.
Any sufficiently advanced man is indistinguishable from God
Enron the company was doing lots of good and innovative things in the marketplace as TFA shows. It was certain groups of people that drove the company into the ground.
[Fuck Beta]
o0t!
>and trying to make sure that what every trader saw on the screen matched what every company in the world had on theirs
Keeping it honest is indeed very important!
News about the Kettle Open Source project: on my blog
Many new technologies to come out of his IT shop including:
- File cabinets that perform on-demand document shredding and disposal
- Scripted generation of plausible deniability when you log off from your terminal
(Such as search and replace sed script that removes your name from all financial documents and replaces it with that of a random employee name, and a reverse script that puts your name back in when you log back on)
There is a large collection of email from Enron vailable online. It has been usefull for research in natural language processing, text classification, and data mining. Check it out here: http://www-2.cs.cmu.edu/~enron/.http://www-2.cs.cm u.edu/~enron/
... ...
;-)
trying to make sure that what every trader saw on the screen matched what every company in the world had on theirs
It helps when your company falsely manipulates the prices
Bye karma...
Wtf?
I don't know about that. You should watch an excellent documentary by the title 'Enron: The Smartest Guys in the Room'. The impression I get is that almost everyone at Enron knew that something shady was going on. For example, a lot of the power outages in California were due to traders calling up power plants and asking them to shut down in order to drive the price of energy up.
The entire operation was extremely corrupt and a lot of innocent people not involved with the commodity trading (employees of utility companies that Enron bought, custodians, etc..) lost their life savings. Not to mention that they dabbled a bit too much in politics (they were partially responsible for Arnold becoming governor of California, only so he could keep Energy prices deregulated). The only reason this scam went on so long is because Enron the company was doing lots of evil things in the marketplace. Enron, as a corporation, was able to garner special status with the SEC and this allowed them to use a different kind of book keeping which essentially let them claim profits on monies not yet received.
Are you high? That sandwhich does sound tasty but seriously, who thinks about sandwhiches when reading about Enron?
Could you post a vegetarian alternative please?
"You know you don't act like a scientist, you're more like a game show host." Dana Barret
man shred(1)
Karma: Chameleon - mostly influenced by bad '80s New Wave music
"rm -rf /" on everyone's machines.
Very interesting discussion about the relationship between developers and admins in TFA. My own take is that it's basically the only useful thing that team leads or managers can do anything about by setting the incentive structure for both to be somewhat similar.
Essentially, if a developer's job relies on the same thing that an admin's job relies on (that is, stable, secure and reliable operations) then you have the foundation for harmony. If a developer's job relies on features and new functionality at the expense of stability, security and reliability, you have a recipe for hostility.
You can tell the priorities at a company by how cranky its admins are.
On the other hand, admins need to be open and available to developers, offering advice on OS, hardware, infrastructure, etc. and be able to clearly define the requirements for SSR so that any new designs or requirements can be supported from day 1.
Oh, and a great way to get documentation from developers: give all their cell#s to the admins, so when something breaks at 3am on a sunday and there's no documentation, the admin has a little company. A few calls like that and developers can write some pretty handy documentation!
IT Job Offer From Enron:In the first I was offered a trading systems developer position. They used PowerBuilder on Windows for their trading systems - nothing wrong with that. But the IT Director kept talking about how quickly the laws changed that concerned their energy trading. I enquired as to how they maintained their business rules. He replied roughly that "it's all in code; when the laws change we rewrite the code." I asked what they would do if there were an audit say, 6 months later after they'd gone through 4 versions of code. He replied that they would set up a standalone network, load the old software and restore snapshots of the database and rerun the old data.
Naturally I asked if he had heard of rule-based systems and hinted that, if they had a dynamic rulebase architecture, they could avoid the recoding and versioning. Instead they could have a much simpler system and better controls. He said he had never heard of such a thing and couldn't see how it could be done in their environment. But he was interested.
Seeing that their viewpoint was extremely short-term and unenlightened, I turned down Enron's offer. Many of my cohorts at the time said I was a fool, but the IT situation at Enron seemed to be impoverished in thought although rich in resources. In time I realized my hunches were correct.
Encounter With Enron Broadband:Flash forward to 2 years later. I've gone completely over to WWW development and to FOSS. I'm attending a Microsoft event on IIS mostly out of curiosity. Next to me is an outspoken individual who works for Enron Broadband. He speaks endlessly of the benefits of IE and ActiveX. I ask if Enron uses ActiveX much; he replies that they are tied to IE and that ActiveX is a necessary part of their architecture. I ask about Apache, Netscape, and FOSS; he replies that they have no capability in those areas to his knowledge. I once again decide that Enron has somehow missed the boat.
Enron never carried significant loads on their web servers. Their limit was the number of energy traders in the U.S., which is a relatively small number. Today, small sites do much more than Enron did with fewere resources. There are few useful lessons to be learned from Enron's IT group or their BroadBand division fiascoes.
Would you like me to make you some sandwiches?
/Bad Santa
No. Posting a vegetarian alternative would mean that the social pariahs win.
Problem with your statement is that a company is only as good as its people. A company isnt a free-willed creature.
---- Booth was a patriot ----
The article sounds like a consultant who is in dire need of work and does the ultimate desperate action, getting free ad time on /.
His story sounds like a fairy tale. Not like the life of a system administrator.
A tadpole is a pollywog
On a TV show I saw a year or so ago, one of the former traders they interviewed said they faked most their trading activity: Whenever a visitor would come through they'd gather up secretaries and anyone else they could find and sit them at a terminal & have them act like they were trading. Doesn't sound like a huge system load to me.
Frankly, all of my contact with Enron has shaped my opinion as such that if I had worked at Enron, I'd be reluctant to put it on my resume. I'd rather have a sabattical in Tibet, stay in rehab, or time in the peace core on there instead. Much of the criminal activity at Enron was IT-based. When your accouting system is an application, your IT had to know that the books were being cooked. The fact of the matter is that they chose not to report it. Instead,a lowly bean counter blew the whistle on all of them. Now that I know that this man has come out and proudly announced that he was part of such an organization, I would seriously recommend that he not be hired for any kind of a position of trust. I don't see a lot here to be proud of, ethically or morally.
Anyone who's been an SA for any length of time knows that being an SA carries an ethical and moral burden. Just because you accidentally read an email while trying to fix something on the mail server doesn't mean you can go gossip about it. If you happen to see a file that has private contents on someone's desktop, you don't go gossip about it. If you happen to find kiddie porn, you inform the FBI. There are rules to this business. Annoucing that you've violated the most important ones doesn't exactly make you a desirable candidate.
How can anyone, who claims to be providing web services for "the public", tie themselves to an IE/Windows/Active X architecture? I work for a university and, while our web traffic is quite atypical, IE accounts for only 1/2 of our traffic and has for some time. Windows only accounts for about 60% of our operating systems. Since we seem to "lead the curve" on rest of the net, I would suspect that in the next year or two most sites, at least ones that take international traffic, will start seeing a similar shift in their traffic.
The whole paradigm of web services is that it's supposed to work on any OS, any browser, etc. without the need for any specific client software. If you're web application isn't browser/OS agnostic, you've totally missed the boat. Frankly that's not anything to be bragging about technically either.
So, to sum it all up. He's bragging about being part of an ethically corrput and technically deficient company and wondering why he's not got a job.
2 cents,
Queen B
HDGary secures my bank
This was modded insightful? Bah. You statement shows a complete lack of understanding of the impact of the Enron collapse on many of its employees. The fact that he got another job is a simplistic, and quite frankly juvenile view of a major financial event in many peoples lives. Almost everyone who worked for Enron got another job. The Houston job market is pretty good, and particularly good for people with solid IT credentials. There was very little stigma associated with having Enron on your resume unless you were in the inner circle of Lay/Skilling/Fastauw.
The real impact was that many people at Enron had the vast majority of their personal savings in Enron stock or other Enron securities. The company management strongly pushed employees to do that, and there was significant corporate cultural pressure to invest ALL of your 401K in Enron stock. When the company tanked, people who had worked for years for Enron or one of the companies Enron had acquired suddenly went from having accumulated enough wealth to be close to retirement to having to start over. Plenty of stories of folks who at age 50 suddenly found themselves going from being worth millions on paper to having no life savings and no fiscal security. This happened to folks in a matter of weeks, and while it was happening compnay management was encouraging those employees to stay the course - hold the stock - as it would come back. It changed the lives of tens of thousands of people. I have one acquaintance who went from having 85,000 in a 401K, who got a settlement check for 43.00 from the bankruptcy court. Fourtunately he had other monies and the Enron investments really didn't change much for him long term, but I mention it to give you a sense of the scale of the collapse for folks.
The 'he got a job' comment may resonate with you folks who are young, have no long term obligations like a family, and are living paycheck to paycheck with no view of your future beyond how fast can I save up for a new Athlon dual core, but for those of us whose lives are a bit more established, it stinks.
Disclaimer - don't work for Enron, never did, no one in my family or close circle of friends did/does. But I do live in Houston and have seen what its done to good, honest people who did nothing wrong but believe the propaganda delivered by the people for which they worked.
"Would you, could you, with a goat?" Dr Seuss
Jarod Jenson is real, I worked with him at Baylor. Quite a talented person, as his achivements show.
For taking such good care of your servers! We picked up a bunch of them when the company went bust, and they're still running fine today. Hell... We even left the Enron asset tags on them for kicks.
Fuck the greedy bastards. They can always flip burgers.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
I can't believe how little effort some people put into writing grammatically correct prose.
>>> You can tell the priorities at a company by how cranky its admins are.
Is cranky good or is cranky bad? My guess is cranky is bad.
The best developers I've worked with know how to create both features and stability. What makes sysadmins cranky is, to take an example, getting into an argument with a developer who has just noticed that all the soft links have permissions 777 and thinks it's a security problem.
Actually the bigger the organization the less variation in crankiness. It may seem overly bureaucratic, but having that bureaucracy keeps people from rolling out bad code thirty minutes before the market opens. I think TFA was a pretty typical description of work at a large company (except this company turned out to be crooked.)
The big point TFA missed (this being a late 90s kind of project) was how these days, Java makes programmers lazy or stupid by abstracting system internals. Why should a developer worry about mmap() vs malloc() when he can blame all performance problems on the sysadmins' inability to twiddle JVM and system parameters properly? Again, in the largest organizations, there actually are some Java programmers who understand performance, because they actually are more likely to have a training budget, as opposed to smaller organizations where training is random.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
Stupid ass admin's who do shit like mount application log files on the root filesystem... and stupid ass admins who don't have scripts to alert them when a file system is 99% full... or stupid ass admins who wait until a disaster strikes at 3am in the morning to do anything about it!
You said: "This was modded insightful? Bah. You statement shows a complete lack of understanding of the impact of the Enron collapse on many of its employees. The fact that he got another job is a simplistic, and quite frankly juvenile view of a major financial event in many peoples lives."
I say to you: "This was modded as 2? Bah. Your statement shows a complete lack of understanding of personal finance strategies most people with a higher education consider to be common sense. The fact that your example tells of people succumbing to pressure of others on how to invest their money is a simplistic, and quite frankly juvenile view of a major financial event in many peoples lives."
These can't be real names ;-)
No. That's their porn names.
Just goes to show how intelligent people will still consider stock in some database exactly the same as cash in hand.
One difference: Enron's employees willingly dumped their pension plan funds into Enron stock. The management or anyone from management NEVER asked them to do that.
Read "Conspiracy of Fools".
Besides that i agree with all others.
Jeff Skilling and Andy Fastow's asses should be lubed before being sent to high-security prison for Blacks in California.
Wish they would show their screams and travails as Realtime TV.
Man imagine the rush to watch This !!!
Would surely beat the socks of Grammy, American Idol, and BSG combined.
Heck, people would PAY money to watch these two assholes get their lubed asses pounded by 15" coc*s for 12 hours.
"Doing what i can, with what i have." ~ Burt Gummer
Quoting the article...
The other cases were real applications. One was an XML parser We found a very easy change that he could make in his code that yielded a performance improvement in the 60 percent range. We did that one in about 10 minutes. It was kind of stressful, but it was fun.
KM
What tools did you have at your disposal there?
JJ
The number one tool was DTrace. We had just recently created a Java agent for DTrace, called DVM, that allowed us to correlate events in native code and on the system. If it weren't for DTrace, there is no way we could have done this challenge. What I've been doing for the past two years now is helping mostly Sun customers with performance problems, and DTrace is my tool of choice. Of the customers we've helped, we have been able to get them all an improvement, ranging from 12 percent to almost 5,000 percent performance gains. The observability with DTrace is really unmatched. When I used to schedule work to go solve performance problems, the standard length of time was two weeks to a month. I can literally schedule three customers a day now. That is because with DTrace, we have the tool that we need.
I wonder if this is him...?
Il n'y a pas de Planet B.
Is anybody else seeing the humor in an application called "EOL"? (end-of-life) ?
His points about 'taking ownership' of the problem appear to be spot-on right, when people take ownership of a problem it provides a better chance to solve the problem. This comes straight out of the book In Search of Excellence as a method of building a better company. Difficulties and loss of effort happen when the general environment is one of "it's not my problem." When one takes ownership of problems in order to fix them no matter where the actual problem is you can produce excellent results. But again, if you're not looking in the right place to find the solution, chances are you won't find it.
The lessons of history teach us - if they teach us anything - that nobody learns the lessons that history teaches us.
99% full: we have scripts thank you. We have tried many different policies to clean up once we are alerted: asking nicely (they ignore us), asking harshly (they complain), being draconian (removing older, bigger files) people get sarcastic at you (here, I give you 100 bucks, buy a new disk), imposing limits (everybody has a good reason to need more than their colleagues), doing careful audits (discovering all the porn, movies, pictures, unauthorized software, etc., people, kid you not, complain about their privacy being violated.)
So after a while we just turn the damn 99% script off. Once the filesystem gets full we send an email to all the parties involved to sort it out. Say what you may, this works better than all the approaches above. We could try group quotas or smaller volumes assigned to smaller teams, but live is too short and I am sure people would still find problems with this or any other solution.
Disasters at 3am: you know, shit really happens, I know Mr Rumsfeld has discredited this very true notion, but shit does happen. You can have full redundancy, careful considerations to avoid downtime even people working to cover on site 24x7. Shit still will happen. So get over yourself and be grateful that there are people out there interested on this kind of job, which we can see form your post, get no recognition at all in spite of the users being stupid asses.
IANAL but write like a drunk one.
If people worth millions (paper money or not) can't be arsed to buy "Investment 101" or "Savings for Dummies" they are setting themselves for failure.
And don't bring the "they were forced to buy Enron" bullshit. I am pretty sure that nobody put a gun to the head of anybody to force them to risk all their saving eggs in one basket only.
The bad luck of a fool is always regrettable and certainly one can feel a degree of pity for the fool, but the fool has to take responsibility for his own actions if he or she is a normal functioning grown up adult.
IANAL but write like a drunk one.
I read the article, and it has some useful ideas, not necessarily new. Jared is a skilled performance tuner and its a good discussion to read. What is disappointing is the flambait on this slashdot discussion board.
In reality, even the folks who support and maintain an IT system usually have no idea how it works internally.
(The average Apache user can admin, manage and maintain their server, but do they have any idea how it works
internally?)
Most sysadmins and systems folks support infrastructure whose data is not transparent to them.
A rough example would be, say, the sysadmin for a large webhosting joint.
Said sysadmin is only managing the disks, directories and databases that contain all the web
content for all customers, not viewing each and every single web page, script or atom of info
within.
In much the same way, the systems support staff for some large ERP or accounting system is only managing
the backend for that system. He/she will only be dealing with how to keep it up and running, available,
and resilient against disaster. Said support person or staff is NOT going be perusing each and
every atom of information that the end user entered. (Usually, the data stored in such a system isn't
transparent via the backend, but usually only comprehensive when viewed through its frontend.)
It is the job of the end users (the accountants and management) of said tool to ensure that the data
being entered is correct, not the sysadmin or the systems support staff. Your electric utility provides
you with electricity, but if you use it to power a chainsaw to attack some poor soul, it isn't their fault.
Their job is to make sure you get your energy. Not to tell you how to use it. Like it or not, believe it
or not, most large scale systems support operates in much the same fashion.
Your comment:
On a day to day basis in most places, IT usually isn't even aware of the company's accounting or financial state. So how are you expecting them to know what are the correct numbers, and what aren't?