Slashdot Mirror
The Rise and Rise of IT Administrators
maffstephens writes "Have you noticed how difficult it's become to develop software? Not because software is more complex, but because there seems to be an army of administrators standing in your way - sys admins, network admins, database admins, runtime admins - the list is endless. They should be there to help us, to make our lives easier, but the reality is often very different. This thought-provoking article from Software Reality is all about the emerging culture of spiteful, dog-in-the-manger prevention amongst corporate IT administrators. Software development has become so inefficient as a result, it's no wonder so many companies are outsourcing."
Developers, Developers, Developers, Developers...
(That's all I hear these days, thank you Steve Ballmer)
As a sysadmin, the Devs need to learn how to play nice and keep the system stable. As a developer, I want total access to everything.
Solution? Developer network off the main network. If they blow it up, it's their fault and they fix it. Sounds good in theory. I think programs like Ghost will play a big role in this type of setup.
I work for a government agency in the U.S., and as you can imagine, it's saturated with sysadmins who watch over security, resource allocations, etc. Our solution was to build our own network infrastructure. We purchased two servers, cross-trained about six of us to work as admins on those servers, and completely bypassed the regular admins. The result is that we're one of the most productive organizations in our industry, because we were willing to put in a little extra effort to get around the problem.
It's good to use your head, but not as a battering ram.
I'm not defending all the admins, because there ARE a lot of clueless ones out there, but have you ever stopped to think that we're here as subject matter experts?
I'm a UNIX system administrator. My responsibility is to ensure my systems perform well. This includes actual performance statistics (I/O, CPU, memory), security, reliability, scalability.
It also means I need to scale up the hardware as applications grow. I need to keep tabs on what my systems are doing, and why.
I'm the "guy who gets in your way" because my responsibility is to the system, not to you.
I don't work for you. I work for the systems. They are my "customers" if you will.
Sure, I slow you down when I tell you "No, your app can't run as root."
I slow you down when I make you diagram your database so we can lay out the I/O correctly.
I slow you down when I make you tell me what you're doing with shared memory so I can tune my kernel.
I slow you down when I ask for projections over the next year so I can plan the hardware and scale appropriately.
I slow you down when I shut off telnet, ftp, r services, and every other plaintext protocol. You b*tch and moan because your expect script from 1994 needs to be rewritten, but too bad.
I slow you down when I ask for a detailed list of which ports your application uses, who they communicate with, and what IP blocks I need to permit access from.
Yep, I'm in your way.
That's my job.
And if you don't like it, well, too bad. I *DON'T* ask you why you're using C instead of Java. That's not my business.
I'm a systems subject matter expert. I don't pretend to be a code expert.
Your a coding expert. Don't pretend to be a systems expert.
Let me do my job, and I'll let you do yours. We need to work *together* and understand the interactions between your code and my systems.
Systems are NOT simple. They're very complex; you need to understand all the interactions here, from the kernel through the disk management (whether it be VxVM, LVM, or whathaveyou), through the network drives, through the firmware, through the HBA drivers...
Let me do my job. Yep, it'll "slow you down" a bit, but in the end, we'll actually have a complete SYSTEM that functions. Code, OS, hardware.
So you can't roll things out in an hour anymore. At least it works now.
I disagree. We need to focus on the quality, not the quantity. We need better planning, not just less of it.
I've been involved with companies who spend forever planning and twice as long coding, and they still produce crap. Why? Because the design is always done by committee, so no really good ideas get out there, and the design always ends up as a preoptimized mess with a few "management-approved" ideas thrown in.
I seriously think a small tag-team (2 or 3 people) should be responsible for projects, and they should take in all of the input and recommendations, and produce a solid spec by themselves.. rather than the typical '10 departments sit around a table for 20 meetings and produce a piece of shit' method.
mogorific carpentry experiments
I my office, the IT personnel grant or deny software installation requests (among many other IT-related tasks, but software installation draws a nice example). People usually want software because it would make doing their job easier. Does this mean IT should allow any tool to be installed? If a software request is denied, the requestor will sometimes complain loudly along the lines of "Your job is to make sure I can do mine, not regulate it," to which IT will retort "but if we allow any software, it will result in incompatabilities between departments ultimately reducing productivity and increasing maitenance costs." Both are right; this sometimes results in a bitter relationship, but lets face it, they're keeping checks on each other. A successful development company needs to find a balance between the two.
Sounds like how they wrote Windows ME.
For too long developers have been held up as the ultimate in computing knowledge, while administration has been seen as some monkeyboy sitting in a computer room swapping tapes out.
As a result, nearly every end user of a developed system is given attention before system administrators and operators. The secondary result is SA's and operators are left with big piles of innefficient crap to wade through, and much of the pressure of making said piece of crap work. How many folks here have had to work in huge, bloated teams of SA's all to support an ill-concieved and poorly developed (but gee whiz does it look greeeat!) product, getting paged and phoned all night to come in an slap more duct tape? How many people here have had to manage a bunch of boot-camp MCSE's trying to do 400 manual processes an hour because "that's the way it was developed"? How many people here have had to explain to a customer that some piece of code written by a fresh off the MIS degree train VB developer isn't RFC compliant and therefore 45 percent of the people in the world won't be able to interface with it? How many SAs here have had to tune the crap out of boxes and networks because a login page makes 75+ ODBC database calls? How many security consultants have had to go in and basically tell a company that they'll have to repartition and reinstall every server because someone found SQL injection in an app that required superuser privileges?
The list goes on and on. Administrators aren't there to make life easier for developers, they are there to make things work--and make them work better, more reliably, and more securely. I'd suggest that this whiny ivory tower developer wake up and realize that coporations have gotten smart to the crap he's been turning out and further realized that the people who run the stuff are just as important as the people who write it and use it.
In short, he needs to learn how to work on a team.
Developers are smart, but they aren't the top of the computing pyramid. There are many other groups of people that are just as smart in different areas.
ditto on that.
:)
i just got word that my ex-school district is purchasing PDAs for every student enrolled in middle school and high school. when i was in 6th grade, i could barely keep track of my lunch money, nonetheless a PDA. id hate to see the rate of these things get broken/stolen/lost.
in addition, the IT admins for our 2000+ high school didnt know what puTTY was and kept removing it from my personal storage folder out of fear of what it was. not to mention they stored their win2k domain password as one of the usernames (in the format "adminPASSWORD") in case they happened to forget it somehow.
on the bright side, if im ever desparate for a job, i know one place i can go for sure.
This pushes me to take responsibility for having an overall understanding of how the application fits into a larger security context, and that the application works in the real world/under load.
Only then is the app dumped onto the larger network. I think all developers should do some real-life system adminstration, and system administrators should do some development.
Nice mindset there; you're a real team player. The reason we are there(network/sysadmin here) is to HELP you.
However, we're also there to make sure you don't do stupid things. While you say "these IT people get in our way", I point to a laundry list of really, really, REALLY stupid things developers have done at every company I've ever worked for; they just don't THINK about anything besides code, and they get Great Ideas without thinking through the consequences, either technology or business-wise. Some of it is just sheer laziness, and I've been faced with developers who act liked goddamn 5 year old spoiled rotten BRATS- this was particularly bad a few years ago when anyone who knew what "printf()" meant, got a 75k+ job.
Prime examples of stupid things I've seen: logging into machines using the root account because you're too lazy to use su. Or not allowing you to ssh directly into a system from your home PC without a VPN. Or yelling at you when you use temp tablespace for permanent data. Or not letting you move production functionality to some desktop system underneath your desk. Or using the database admin account for your application, instead of a seperate account?Or not implementing your latest code changes until you're willing to put down on paper that you actually did your job properly and TESTED the damn changes(do you know how many times I've seen developers just push code out without testing? Guess who gets blamed first. Guess who gets PAGED first, at 3am when it crashes. Management doesn't distinguish between a misnamed variable and a "Internal Server Error 500"; they're both production problems, and you're not in charge of production).
We're part of the team, and we're here to stay. You can either work with us, and clearly communicate to OUR supervisors(not just us) what your needs are...or you can make us the enemy, always try to do things half-assed, and get nothing done. Your choice- but management usually sides with safety, and we're the ones saying "that's not safe", and even if management doesn't side with us- when things blow up, we simply point to the emails we sent saying "that wasn't safe", and let you sweat it out while we restore from backups and clean up your mess.
Sometimes it's simply not our choice; it's "do it this way, tell Development that". You have no idea how frustrating it can be sometimes for even us- I once worked at a place where root passwords were changed on us sysadmins, and we were told "use sudo". The incompetent assholes a few levels up didn't realize that gee, guess what, if the machine crashes and fsck fails, you need the root password.
Please help metamoderate.
Well, there you go.
My developers tend to want to run their web servers on port 80. I won't let them.
Why not? Because then they have to have root privs to start/stop the app.
No dice.
What's my solution?
Run the webserver on a high port (I tend to use 8000, but that's arbitrary)
Put the systems (Yes, each app has to have at least two for redundancy) behind a pair of load balancers. Let the load balancer do the work. While we're at it, make sure the load balancers have SSL accelerators too, so we can offload that from the CPUs...
Much saner architecture than letting a developer download Apache from Sunfreeware and running it on port 80.
And then people wonder why we have sysadmins?
What I often see is the people who least understand the big picture when it comes to technology are the ones who feel held back.
...
..."
The people I see getting mad just don't understand the impact or implications their "simple requests" may have on others.
"Can't you just open up ports 135-139 in the firewall for everybody"?
"It works fine on my system, something must be wrong with the server"
and my all time favorite when people don't have a clue why their system isn't working
"It must be the network"
They really don't understand how their system works.
As an admin (LAN, WAN, firewall, server, email, etc... you get the idea) for a med size (3000 users) organization I often have to learn other peoples jobs just to figure out what the heck they are really trying to accomplish. It usually goes something like....
Customer: "We need
Me: "Why?"
Customer: Pick one:
1) Vendor says so
2) We tried everything else
3) Thats what someone else said
4) ?
Me: "What are you really trying to do?"
Customer: "What do you mean?"
Me: "Don't tell me what you think you need, tell me what you are trying to do?"
Once I understand what someone is trying to accomplish then I can often work somethign out for them.
Keep the Classic Slashdot.
No truer words have been said.
They need to wake up and understand that us developers are the true brains behind the enterprise. We walk on water. We are GODS I tell you. I can't count the number of times I've had to yell at my sysadmins for making the coffee too strong, not popping grapes in my mouth fast enough, or moving the hand-fans too slowly. The fuckers. It's as if they don't understand that their purpose in life is to serve me. That the entire company exists not as a profit generating entity, but as my personal support system. Heaven forbid I do something smart like suggest or create a decent PROJECT LIFE CYCLE to avoid conflicts with other departments. I'd much rather whine on slashdot. Now I have to go. My 3 o'clock rubdown is coming up and I need at least another 2 hours of slashdot reading time before that. I mean christ, what do they pay me for.
For every annoying gentoo user, are three even more annoying anti-gentoo crybabies. Take Yosh from #Gimp for example.
I agree that marching off to program with no planning would be silly. But I am a big believer in pathfinding programming, where you spend no more than a day building just enough of an application to illustrate the underlying design and/or interface.
:)
Then, come back and demontrate your idea to the larger group, with the expectation that more than likley you will throw the whole thing away.
After a basic model has been developed that makes sense, only then sit down in meeting to flesh out the spec.
And that's what I meant by prototypes, yes they're very useful, I just wrote one yesterday. I wrote a small proof of concept about some enhancements to Psort and on Monday I'll sit down and do it right - determining how to write the code without jamming it in with a shoe horn.
And prototypes should be thrown away, most likely they're done with very poor quality. I recall one of my old profs when teaching us this made us write out prototype in a different language from what he wanted the final product in to "force us to not reuse it." Perhaps that's a bit extreme, but it illustrated the point.
That there is a reason why alot of admins are paranoid about giving anyone , not just developers, control of their box. The case in the article was an extreme example, but I couldn't help but wonder "What did some developer do years ago that completlty hosed everything?"
The back up situation at the place in the article sounded outrageous. The author had every right to be angry about that.
As far as the firewalls go..if there is a security breach, the developers would not get sacked and new abused ports are discovered. Users find ways of clogging everythign up with Yahoo! IM going through port 80, outside KaZAa users from Brazil suddenly thing that you have LTR Return of The King hidden somewhere on your network or script kiddies from Korea sudenly decide to scan port 1021 all day long...In other words, there are lots of reasons to change the configuration of a firewall daily (unless disconnected from the outside completlty..but no users want that).
Like NetNinja said, cleaning up after them is a nightmare, plus the admins are liable for the mess , not the developers. Communication between groups is the key.
He has decided that his problems are due to administrators, who are all clueless, and that he would be so much better off if his world was run by developers, that are all knowing.
The reality is that the authors problems are due to inept individuals and the corporate bureaucracy that keeps these inept individuals in place. The problems are not simply admins vs. developers. This is no different than any other profession.
There are countless bad administrators out there. Many/most do not deserve the title of Administrator. But, at the same time, there are just as many developers out there that should not be allowed near a keyboard and yet they are forcing new "applications" down end user's throats on a daily basis, "applications" that reduce productivity due to bad design and processing inefficiency, buggy and untested code, and a total lack of understanding of the business process.
There are far too many inept individuals on both sides of the fence. It is not about admins vs. developers.
One more thing, the author seems to understand that J2EE is a bad idea so, why does he continue to develop with it?
If idiots could fly, you'd be an ass-tro-nut!!!
I've seen both the best and the worst of having admins involved, and in not having them involved. About three years ago, my firm rolled out a web-based customer service app. My comrades in UNIX, NT and my network team were only told that they needed to provide servers and connections, not that there was a major application rollout. The first day the app was in use, I had the operations VP screaming in my ear that his agents at our remote site were unable to work because the app was so slow. We found that, because of the undocumented and untested requirements of the new application, the WAN usage at our remote sites went from under 200k to maxing out two T1 circuits. It took two years to finally get that situation stabilized by increasing our bandwidth several times over (increasing our costs) and spending several man-years to correct the application.
After a change at the CIO level, we now have multidisciplinary teams - programmers, admins, DBA's - working together to prevent such expensive oversights. The problem with the article is that it romanticizes the past. How many of us have had to live through DOS programs whose programmer assumed their program was the only one running? Today, more than in the past, we cannot afford to have walls built between the various groups in IT. The costs of failure are too great.
I think, therefore I am - Rene Descartes; I yam what I yam, an' that's what I yam - Popeye
Many of the posters who disagree with the author, wrap themselves in the flag of "looking after the company's interests" - well who the hell do you think the developers are working for? - they aren't just making shit up on their own - Its the managerial idiots in the company who want you to roll out "Project I Pulled Out of My Ass So I Can Feel Important # 15" and no I would give you business requirement because this project is too overdue already just back fill them from the tech spec you guys make up and oh, yeah its important you follow all the processes and work nice with the poor "we're only trying to do our job" Operations Dept. And by the way if its late or wrong (because you read my mind incorrectly) then its your ass, not anyone else's.
:) I think they (IT management) think that things like wget is an example of what they would term "shareware".
He is right, admins have too much power and too little responsibility for being on the line for projects getting rolled out.
Here are some tidbits from one of my jobs at a Fortune 100 Co:
When I first started working at CoX there were no UNIX tools on any of the UNIX servers prod or dev. I had to compile gzip, top, wget, perl and all the other tools needed for a normal system. Why didn't they need top or ntop, because if there were problems on the system they would throw up their hands and say it was because of the developers processes and called them/us.
The network admins would refuse to participate in troubleshooting and no one else was allowed to use the sniffers. They would also do network work including taking switches and routers down during the nightly batch processing without notifying the "developers" who then got called at 4 AM to troubleshoot why "their" overnight processing failed.
The Oracle DBA said that it was not possible for the same query to take different lengths of time to run(at different times).
PC admins - no FTP GUI clients were on the list of approved software since the business users didn't need that type of product. No "shareware" allowed. They were starting to talk about no "shareware" for the UNIX servers around the time I was leaving
The security admins ruled that the r* commands are a "security risk" [period, blanket, no appeals] and the developers were give three weeks to change all the production processes - never mind that getting approval for a change request (from the tribunal of these idiots that run the change control "process") takes longer than that and all the code needs to be changed and tested before submitting the change request (into the IIS/VB million dollar change management system that could keep even the CIA from pulling any usable information from it). You will need to be prepared to justify any and all aspects of your project before the tribunal, even though they are the ones who are forcing you to make the changes.
The list goes on and on. My experience across many jobs (20 years) being both an admin and a developers, is that generally admins are less competent and more useless than developers. The order in terms of least knowledgeable and most "preventative":
1. Project Managers (completely and utterly useless)
2. Security Admin (most seem obsessed with think that make the least difference for true security - ie patching iPlanet so that it doesn't do HTTP TRACE) Their job usually also involves the slimy, salacious task of monitoring people's email and looking through http server logs for who's downloading porn)
2. (tied with security) Network Admins, won't help troubleshoot; nothing is wrong with the network; I can ping that machine from this one so its not the network; no you can't have any performance data about the net/router/switches its "confidential"; no you can't have the snmp password for the machines that you end up having to support because all the admins are useless, its "confidential"; no you can't use the sniffer, but its not the network so you don't need the sniffer anyway;
3. DBAs (The