Slashdot Mirror
Ask Slashdot: Advice On Enterprise Architect Position
dave562 writes: I could use some advice from the community. I have almost 20 years of IT experience, 5 of it with the company I am currently working for. In my current position, the infrastructure and applications that I am responsible for account for nearly 80% of the entire IT infrastructure of the company. In broad strokes our footprint is roughly 60 physical hosts that run close to 1500 VMs and a SAN that hosts almost 4PB of data. The organization is a moderate sized (~3000 employees), publicly traded company with a nearly $1 billion market value (recent fluctuations not withstanding).
I have been involved in a constant struggle with the core IT group over how to best run the operations. They are a traditional, internal facing IT shop. They have stumbled through a private cloud initiative that is only about 30% realized. I have had to drag them kicking and screaming into the world of automated provisioning, IaaS, application performance monitoring, and all of the other IT "must haves" that a reasonable person would expect from a company of our size. All the while, I have never had full access to the infrastructure. I do not have access to the storage. I do not have access to the virtualization layer. I do not have Domain Admin rights. I cannot see the network.
The entire organization has been ham strung by an "enterprise architect" who relies on consultants to get the job done, but does not have the capability to properly scope the projects. This has resulted in failure after failure and a broken trail of partially implemented projects. (VMware without SRM enabled. EMC storage hardware without automated tiering enabled. Numerous proof of concept systems that never make it into production because they were not scoped properly.)
After 5 years of succeeding in the face of all of these challenges, the organization has offered me the Enterprise Architect position. However they do not think that the position should have full access to the environment. It is an "architecture" position and not a "sysadmin" position is how they explained it to me. That seems insane. It is like asking someone to draw a map, without being able to actually visit the place that needs to be mapped.
For those of you in the community who have similar positions, what is your experience? Do you have unfettered access to the environment? Are purely architectural / advisory roles the norm at this level?
I have been involved in a constant struggle with the core IT group over how to best run the operations. They are a traditional, internal facing IT shop. They have stumbled through a private cloud initiative that is only about 30% realized. I have had to drag them kicking and screaming into the world of automated provisioning, IaaS, application performance monitoring, and all of the other IT "must haves" that a reasonable person would expect from a company of our size. All the while, I have never had full access to the infrastructure. I do not have access to the storage. I do not have access to the virtualization layer. I do not have Domain Admin rights. I cannot see the network.
The entire organization has been ham strung by an "enterprise architect" who relies on consultants to get the job done, but does not have the capability to properly scope the projects. This has resulted in failure after failure and a broken trail of partially implemented projects. (VMware without SRM enabled. EMC storage hardware without automated tiering enabled. Numerous proof of concept systems that never make it into production because they were not scoped properly.)
After 5 years of succeeding in the face of all of these challenges, the organization has offered me the Enterprise Architect position. However they do not think that the position should have full access to the environment. It is an "architecture" position and not a "sysadmin" position is how they explained it to me. That seems insane. It is like asking someone to draw a map, without being able to actually visit the place that needs to be mapped.
For those of you in the community who have similar positions, what is your experience? Do you have unfettered access to the environment? Are purely architectural / advisory roles the norm at this level?
What they're offering isn't out of the norm, though I might negotiate with them and ask for read-only access (non-root for servers) at least. I've been a network architect for a few years, and one of the things that comes with: loss of enable access to the routers and switches. Mind you, I was a data center network engineer for a whole bunch of years so I know my way around them. But the organizations would rather I "look, but don't touch". The great thing about it is: I can't be called for an on-call issue because there's nothing I can do to fix it. :-)
Welcome to needing to think strategically. Take what they're offering as a compliment and run with it!
Jason Van Patten
A company with 3000 employees is small-to-moderate size. The market cap is impressive for a company that size, but the company itself isn't in any way considered "big".
Jason Van Patten
An architect is someone who designs, implements and oversees the day to day progress of large(r) scale projects. You get to define who/what to buy and how to realize your vision. But no, you don't need access to the systems but you do need an overview of the entire infrastructure, you're an architect, not a builder/maintainer/owner, you get to see the site, you define future upgrades but you don't maintain the system(s), you surround yourself with others that do that job.
If you can't get a full picture of the network and systems without full admin access, your underlings are doing something wrong and it's time for you to kick them out or go on a major discovery/documentation project. If I were an architect, I'd make sure I have plans, diagrams and documentation on the entire picture first before embarking on a next project.
Custom electronics and digital signage for your business: www.evcircuits.com
With 60 hosts and 1500 VMs I would certainly expect separate roles for enterprise architecture and system provision/admin. If you were talking about a a dozen hosts and 100 or fewer VMs then a sys admin with architectural responsibilities would be quite common. The main thing here is what can you do? My guess is with your experience they could use you overseeing a system admin team and doing EA, moving more to the latter as the team's experience grows.
I'm one of several that play the EA role for an 8,000+ employee firm and for the most part I have no special rights to the systems, domain and network. I can see read rights to everything your describing being reasonable, but just like we don't give developers rights to the production app, the architects shouldn't have the ability to change production either. The person designing and advocating for the solution always thinks they are right. But having to put it through a good change control process helps reveal assumptions and ill considered ideas. That isn't to say you should be able to see into production either directly or be able to request any data you need to design better solutions. And you're not powerless, sounds like the last guy had enough power to screw things up, use the same bully pulpit to start guiding the changes that make it better.
I work for a similar sized company, and our architecture team was built taking senior people from several different admin groups. Separation of duties, especially when you are covered by PCI, Sox, etc, is the norm.
As long as you have access to complete information as to the design, implementation, and how systems are running and working, you probably would not have unfettered access. It should not prevent you from doing your job. But you absolutely have to have a good working relationship with the people who do have that access.
The role of an enterprise architect is to work with stakeholders, gather requirements, create time lines and then hand their work over to another team to implement and continue to provide governance. At best you might be lucky to get access to some sort of test environment. I am TOGAF certified and like you before I started didn't understand what it was before I started. The trainer I had described it as creating cartoons for executives. I still got the cert but realized it really wasn't for me. I will say that I think the role is very important and as an implementor is designed to answer the questions I often have when building something like number of users, availability requirements etc.
What you want are metrics. So while you cant get access to the systems ... you can (probably) dictate everything you need to measure and monitor. After you have lots of data points, then you can report the crap out of it to start making i better.
The architect is the "big picture guy", he should be able to design it and explain it. But he sure as hell shouldn't be running it.
The architect is most decidedly not the sysadmin, he's there for strategic and long term planning, but not day to day stuff.
If you want to be both the architect and the admin, you'll do a piss poor job of both, and likely cause more problems than you realize. I've met a few architects who thought they should still keep their fingers on the switch, so to speak ... and as they generally made a hash of things because they didn't have the time to be good admins, and though they knew everything at all times.
An architect who thinks he's ad admin is someone who has delusions of being able to do everything, and ends up doing everything badly.
Your management is right, it's an either or thing.
If your organization is small enough you can do both, you're not really an architect. If it's large enough to need an architect, it also needs a sysadmin. It doesn't need some guy who thinks he can do both.
Lost at C:>. Found at C.
I would say that the best enterprise architects are the ones influencing the business, listening to their pain points, identifying the risks with the security manager. Set the boundaries or constraints of your organisation. No point wanting to use public cloud if you hold TS information. Do you suffer more pain for not technologically being able to spin up things automatically? Or do senior management have drivers to meet compliance this year because they now got to a new $ value and have more auditing responsibilities. They will be the ones with the money to let you execute your roadmap, so you need to show the global view of how IT enables them. Don't be precious around opensource vs Windows, Oracle vs PostGres. Only if you want to analyse the costs and direction of the company as a whole, might you say that you have an organisational direction to set. Expect possible resistance from the person who takes your job. Negotiate, document, communicate, review and sign-off.
Develop roadmaps, ISSP's and ways to get there that are easy for people to follow. Use the operational staff from your new level to derive the information on your behalf. Do your work on it, and then communicate how you want to tackle problems. If you have not had clear leadership in the organisations technology aspirations space, then take advantage of being the EA/CA and the position it generally enjoys closer to the senior management. For all sides though, you need to bring ways forward, budgets expected to be consumed, tech sets to be used, tech sets to be retired. Read up on TOGAF or other architecture methodologies. See how an EA can slot into the surrounding processes such as programme management, business planning. And fully expect those people and processes to be needing help too.
I can say as a 6+ year infrastructure architect, I've only rarely hopped onto a read-only view of a vCentre, or into an Azure/AWS config. Moreso for my own training than anything. Same goes for spinning up VM's, playing with controllers - only for training and keeping across what the market has to offer. You will find very quickly it is hard to stay abreast of all of that, especially with the ton of work mentioned above. Embrace that, do a good job and try not to upset the alphas on either side ($$ and priorities on management / business side, and tech arguments on IT side can bring out worst in some meetings).
I don't want leadership, management or anyone with any kind of oversight responsibility making changes on the live gear. That's the entire point of having an operations staff.
However, I see absolutely nothing wrong with read only access. The ability to change things - Not Good. The ability to gather information, that I would deem to be necessary for someone who's going to handle the care and feeding of the system going forward.
It also sounds like you need to clean house if your ops staff is pushing back at designed changes, however. Putting in a competent staff that will follow your dictates and provide you with the information you need would go along way to making access to the actual gear unnecessary
Comment removed based on user account deletion
I'm an Architect. Also with a long technical background. Similar size organisations. It's not normal to have admin access. Largely because that level of detail can overwhelm you. It's also easy to get dragged back into your old job if you can be dragged back. In one org I worked in where the Architects did have access (before I was one...) one of our vendors develops the habit of finger pointing when mysterious issues occurred that looked like unauthorised change. We stopped that with some config monitoring software that notified us of any settings change - but I mention it to show what can happen.
One of the hard concepts to grasp is what is Architecturally significant. Mostly that's big block level stuff, but sometimes certain details can be significant too. Working out which without looking at every detail is where your experience comes in.
Most of the time the team members doing the design and implementation work can show you the detail when you need to see it - and by asking them you can discuss what you're looking for and why. This builds up trust that your solutions aren't just ivory tower creations from some distant figure but things they're connected with.
If you must have some ability to see every little detail you could always try asking for read-only access. It might be a reasonable compromise.
This has been a bit of a rambling post, but I hope it has something useful....
NX-01 (Archer)
NCC-1701 (Kirk 1)
NCC-1701A (Kirk 2)
NCC-1701B (Harriman)
NCC-1701C (Garrett)
NCC-1701D (Picard/Riker)
NCC-1701E (Picard/Riker)
I'll forget alternate timelines and also the boats as they've already been "architected."
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
Weather or not your overseeing people you are overseeing projects and larger scale strategic strategy, which makes you a manager.
You don't have the luxury anymore of being able to do things yourself. Context switching from strategic to tactical mode and back has a huge cost. Humans suck at multitasking. That's the reason that in most human endeavors over the millennia we've settled on the idea that organizations work most effectively when you have a few people overseeing the larger scale picture and many people managing the day to day tactical situation and reporting the information that the leaders need in order to make decisions up.
You can no more be successful if you're in systems typing ps to discover what's going on as a general can be if they had to write every field report by hand. At best you'll burn yourself out and then everyone will be in trouble.
Instead my advice would be to take this as a coaching opportunity. "Hey, I'd like to take a peek at this config file. Mind bringing it up for me? Ah, see, that's where the problem is, you how that quotation mark is missing? The next line is being included int he string. Thanks, I'll raise a change to fix that." You've just taught someone something. Next time they will remember to check their string terminators. It's a win-win.
And I know this because I was in EXACTLY the same spot and mindset as you about 10 years ago. It's time to shift your mental viewpoint. It's not easy, but the fact that you were given this responsible suggests your fellow leaders believe you're up for it.
Min
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
I worked a number of years ago as Chief Architect reporting to the CIO of a similar-sized organization. To answer your question directly: I didn't normally have admin access to systems. I could get it easily if I needed it. Mostly, what I had was access to the configuration management system which was a reflection of everything else. More importantly, what I had was unfettered access to any _person_ in the organization with a role in technology. For the complexity of the systems I was dealing with, it wasn't really possible for me to know (or want to know) all the details. Detail was, certainly important, but I trusted most other people to get that stuff right. The situation you are in, seems like it would require a lot of clean-up. I was in a similar situation. In my case, the clean-up was necessary because many systems had been custom-built by offshore providers who had low levels of technical skill. The best tool I had going for me was to use Scrum as a way to do incremental cleanup of large systems. Scrum (or other Agile methods) are an enterprise architect's best friend! Build an internal team of people that you really trust to get things right, get them to work in short increments 2 or 3 weeks long, give them the vision of cleaning everything up, but doing it incrementally, and help them prioritize the work. You will be surprised at the amazing things you can do without direct access to the details. (FWIW, I love your analogy about map-drawing, but I don't think it applies.)
Helping with organizational effectiveness is our job.
If you are logging on to boxes, you are getting too close to operations and too far away from architecture. You get the admins to pull reports and logs you need, but you don't really need logins to the entire infrastructure. What on earth would you do with it that you can't get from the admins? I'm an IT architect for a DR outsourcing company; I wouldn't even have the least clue HOW to login to the gear I'm buying by the truckload (much less do anything useful with it), so obviously I don't have the ability to do so either.
An architect need not have admin rights or even the knowledge an admin must know. (And likewise, it's not important for an admin to know things an architect must.)
P.S. Errr, 1500 VM's for 3000 employees? I sense that a lot of these (and whatever massive amounts of stale data they are attached to) sit utterly disused.
P.S.S. And your historical analogy isn't even valid. Cartographers are generally not surveyors.
An architect (and one that is trying to be forward thinking and implement all sorts of fascinating new gear) is wasting time learning the admin interface for every box he/she specifies.
And if an architect is having trouble getting away from daily ops, not having any access to the boxes at all will help with that transition. (Not to mention that the architect will inevitably get pulled into ops problems, leaving less time to do the actual job.)
They could be doing a Citrix-like thing where everyone is logging on to server-housed remote instances. If each instance was one VMWare VM, then 3,000 employees and 1500 VMWare instances makes sense.
They could also be a company where each corporate customer, or groups of individual customers, require their own virtual server. For example: an SaaS accounting firm similar to FreshBooks may have a separate virtual server for every corporate customer, or Blizzard where groups of users have their own dungeon.
Otherwise, even for a IT company, they have the IT infrastructure from hell. I can't imagine 1500 different server applications in a company of 3,000 people.
It would be funny if your comment had some context.
With 60 hosts and 1500 VMs I would certainly expect separate roles for enterprise architecture and system provision/admin..
This statement is quite right. Apparently the OP doesn't deal with auditors at all in his job. Lucky him. I do in mine and I have something like a Linux system admin job. For the product I work on, and I work for a Fortune 500 company that sells a lot of software products and services, I am the main contact person every year for auditors. Since the OP works for a publicly traded company, he should know that audits are required by US law. Every year I have to answer the same questions from the auditors about separation of responsibilities on the product I support. Honestly, I don't know how the OP doesn't know that getting that kind of access for an architect is going to raise all kinds of red flags in an audit that have to be explained. If I remember correctly, we have exactly 4 people who have root access to our servers who don't work on my team. They're software developers who've worked on the product for years and need that access in an emergency if we have a software related disaster that impacts customers. We have to jump through a lot of hoops to justify this on the audit. In fact, we've actually had our access restricted from some activities we used to do that fall outside of traditional system admin tasks just because it's easier for auditing purposes for us to not be able to do it anymore. In my job my group also doesn't have access to the storage, network or virtualization layers except as users/clients and all changes have to be done by others. Sometimes it's a pain, but at auditing time it makes my life easier as I can tell the auditors "We don't have the ability to change that, so you have to talk to group X on that one".
I think you may be overlooking one thing...eSOX or equivalent policies.
Company assets have to be partitioned. You can't have people that are not trained and/or not accountable for data/hardware messing with stuff. Auditors for the Government and various Standards organizations (ISO this or that) look for these things. For instance, as an "Administrator" for our Manufacturing software, I can change master controls and permissions. But I cannot actually use the software to do anything like create POs, print Invoices, etc. and that is it should be.
I am also in the approval chain for granting access to shares...but I ultimately do not have access to any of the shares (except mine). What's in them is none of my business and outside the scope of my duties, so eSOX and similar policies say I should be locked out.
If you have someone who can do anything they want to anything they want, you are setting yourself up for a disaster.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
It's clear you do not understand the position. You need to read about what the job entails, and then decide if you want to accept the offer or not. It's a substantial change in role from your current position. In this case, the Wikipedia article is pretty accurate:
https://en.wikipedia.org/wiki/...
As others have stated, your primary responsibility is specification. To do this, you meet with stakeholders, and do projective business IT strategic planning.
While you can relatively easily negotiate for read only access as a demand from "on high", you should not personally use it. It should be used by your staff, temporary or permanent, for the performance of detailed specification compliance audits and spot checks. This is adequate justification to get management buy-in for this type of access. This type of access is for a role, not for a person. This is one of the reasons it should not be you.
For day to day operations, what you need are dashboards, which measure the degree of compliance with the detailed specification in an ongoing basis. The main purpose of the dashboards are to give you information you can summarize periodically to the executives, and as feedback into your strategy decisions going forward (particularly decisions surround capacity planning and technology adoption).
The purpose of the ability to audit is to ensure that the dashboards are not giving you fudged numbers based on what you want to hear, vs. what IT whats to implement (or what they can implement; you may be asking for the impossible, as a dictator, when you should be viewing the detailed specifications as a negotiation). Audits can also provide progress reporting on deployment of specification changes, based on what IT is reporting vs. actual. Since you appear to be planning a lot of churn for them, I suspect you will need one FTE staff member to perform rolling audits to ensure that things are on schedule, and if they aren't, you can negotiate either a schedule change, or a working emphasis (this is a prioritization list: other things will suffer if you have insufficient staff in IT for the demands being made).
Good examples of what you can dictate are things you've complained about: Automated Provisioning, use of SRM in VMWare installations, enabling automated tiering in EMC storage hardware, and so on. Things which will bite the enterprise on the ass eventually, if they are not done.
Your initial dashboards should be based on displaying progress on this (e.g. "percentage of VMWare installations with SRM enabled", etc.).
Note that before any of your shit starts running down hill, you need to make sure you are not downhill from them. To do that, you are likely going to have to have meetings, a couple times a week (usually something like Tuesday/Friday), to collect requirements for the business, and then mash it into part of the requirements document that you will need to prepare before you start defining strategy and dictating conformance/performance). Otherwise you will find yourself buried in crap, because your goals will not be clearly derived from the enterprise goals.
Your ability to take new input from the early in the week meeting and report it in the late in the week meeting with the stakeholding execs is going to be your main performance metric until you go into the design, then implementation phases. Your goal is to get to an ongoing maintenance/change phase. Your metrics will be different in each phase. You will use these in your performance reviews to justify yourself.
If you have other things you care about, they need to be in the specifications -- and they probably need a dashboard, and they need to have a schedule.
For example, if you care about automated provisioning, then you need to have a scratch machine that is identical to the production machines, and you need to have a metric of "how long from a zeroed state does it take to provision the m
Thank you everyone who took the time to respond to my question. Reading the responses has been very insightful and a bit humbling.
I appreciate those of you who called out my tone, pointed out that I'm a whiner and even insinuated that I am not qualified for the position. What would an "Ask Slashdot" post be without one or two snide comments along the lines of, "If you have to ask slashdot, you're obviously an idiot."?
I came to the community as humbly as I could because I realized that my own ego was likely getting in the way, my understanding of what the position is might be skewed, and needing a reality check. I got it.
There were way too many questions and comments along the way to address them all individually. (tl;dr feel free to skip the rest) I will try to respond to most of them here. I hope that by providing some background about my professional experiences and how I got to where I am, others who are on a similar path will gain some insights.
A lot of people had questions about the company itself, its size, the VM to user ratios, infrastructure and other questions. Without spending all day writing about it, the company is included in the Russell 2000 Index. That makes it "medium" sized here in the States. It is a consulting company and we frequently bid (and occasionally win) jobs for the same organizations that KPMG, Deloitte and PriceWaterhouseCoopers go after. My five years at the company have been spent working in the legal technology segment. We provide electronic discovery services to some of the largest organizations in the world. Most of the VMs are application / processing VMs that churn through large batch jobs. (Think producing TIFF files of tens of millions of emails, Office documents, etc. from a large corporation involved in a dispute. Think Enron. Getting caught rigging LIBOR. Creating MBS products that send the economy into a recession...). We also have a number of SaaS solutions for that market.
The IT organization has an ITIL compliant change management process. I deal with auditors frequently. Due to the nature of litigations we are holding onto reams of personally identifiable information, confidential information, privileged information. We deal with large financial sector clients who are subjected to all of the regulations. We deal with health care clients who are subjected to all of the regulations. As irksome as auditors are, I have found that they truly do help us elevate our operations and we have been able to use audits to get capital for systems that we otherwise would have never been able to justify on our own.
When I say that the IT group was traditionally internally facing, they were. They deploy laptops, manage remote offices, keep Exchange running. Their customers are internal to the business. The prior CIO (who was moved out a few years ago) failed to properly size the "cloud" (kill me now for even using that term). Our operations completely outstripped the resources available and required millions of dollars of additional investments in storage (primarily) and compute resources. It was such a large investment that there were even rumors of the business divesting itself of the practice entirely rather than spending the money.
Before I got to my current company, I was a consultant in the (truly) small to medium sized business (SMB) market. (1-250 employees) In that life I was the primary IT resource for small companies where I did everything from design to deployment to operational support. I worked with everyone from architectural firms, to city governments, to waste management companies, 501c3 non-profits, air freight shippers, restaurants, manufacturers (things are still made in America?!?) ... a very diverse client base. I have been working with IT systems professionally since 1996 and using and building my own computers since the early 90s. (The first computer I built myself was a 486DX2/66. I am not as grey bearded as some here, but old enough to have used a 2400 baud modem and