Slashdot Mirror
Preventing Vendors From Playing The Blame Game?
Johan asks: "In choosing an enterprise architecture, one of the considerations I have is support. IBM provides a good database (DB2), a bad Enterprise JavaBeans Container (Websphere) and an excellent OS (AIX). If I tolerate the delinquent EJB container I will have the advantage of one phone number to call for support and a single "point of responsibility". However, I dont want to have a mediocre architecture by getting all three components from the same vendor (IBM). I would like to get the database, EJB Container and OS from different companies (Oracle, BEA and IBM). How do I prevent (or at least minimize) them blaming each other when support issues come up? Anyone have a solution for this?"
It's easy... when you have a problem, don't bother ringing support - just Ask Slashdot
:-)
I've done tech. support and required tech. support for products like these, and I can only recommend that you not compromise by single-sourcing your products when there are better alternatives for individual components.
If you understand exactly how you are using each of these components, it will generally be self evident which one is to blame for any particular problem. Convincing the vendor can be another problem, but if you come armed with clear facts about the situation, it shouldn't be that bad.
Know your system, and how it interacts with each product. Convincing the correct support department that it's their issue isn't as bad as people would have you believe, as long as you are insistent and persistent.
--
--
We have fought the AC's, and they have won.
Easy: get a "blame consultant" or "blame vendor". This is: get a guy or company (an external one, an internal employee will NOT do the trick) who/which, per contract, HAS TO solve your system problems. The idea is to set up things in such a way that if Mr. Externalguy says "Its a database problem" your answer has to be: "Ok then, call Oracle and have them fix it; if they claim it's an OS problem, then YOU call IBM and coordinate them with Oracle and whoever else to get the thing solved, Mr. Externalguy! That's the job I'm paying you for!" In short, the idea is to pay someone to serve as a "no-excuses" focal point and have him deal with as many vendors you may need.
Strength, balance, courage and reason. If you know what's this about, contact me!
My company did the same thing last year. We ended up going with AIX, DB2, and Websphere. _BAD_ choice. We've since switched a lot of what we'd hoped to use the RS/6000 for to a different platform.
The AIX has been OK, but IBM doesn't seem to know their own hardware/software very well. We've tried to order some performance monitoring software and were sent the wrong CD (three times and counting). When we discussed our performance problems (definitely not IBM-related) the senior technical person had 'never heard' of a configuration like ours (internal SSA disks). When trying to buy an extra hard drive we were promised Monday delivery. On Thursday we get a call asking for more detailed information so they can ship the right drive (even though there is only one possible drive to ship for that model/capacity).
DB2 is not a bad database, but good luck finding A: any software that _really_ supports it, and B: any DBA that can support it -- there are lots of Mainframe guys out there, but mainframe DB2 is a bit different than DB2 'UDB'. Enjoy the 1-2 books out there.
For my part I would heartily recommend looking _really_hard_ at Oracle, MS SQL server, Sybase, DB2, et. al. and verify you can: get a DBA, can find developers familiar with the database (DB2 is different enought you don't just develop and pretend it's 'any' database), and that any future software you might use supports the database.
As far as 'no finger pointing' -- getting everything from IBM does nothing to prevent finger pointing. Instead of Oracle blaming Sun who blames BEA, you'll have the websphere group blaming the DB2 group who blames the AIX group within IBM. Unless you're a really big IBM customer (read: have a mainframe or two) forget about getting it resolved instantly.
"But actually trying to use m4 as a general-purpose langage would be deeply perverse" --ESR
The answer is that these days there is no problem. All the old "big iron" companies have re-invented themselves as service supplies. Consider, for example, Compaq who bought out DEC not for their hardware or software but for their large portfolio of nice juicy service customers.
;o)
IBM is no different. All you have to do is tell your IBM rep what 3rd party components you want to run and that you want IBM to support the whole shebang. They'll be glad to do it - revenue is revenue. The one thing you'll *never* hear them say is "Er...I don't know how to do that"
It shouldn't be more expensive than paying for three separate support contracts either, in fact you'll very likely be able to negotiate a discount.
Looking beyond the obvious, you could in principle get a comprehensive support contract from just about any major service supplier, eg Compaq or ICL. But IBM are one of the best these days. They had to get that way to survive (remember how they were bleeding money a few years ago).
Consciousness is not what it thinks it is
Thought exists only as an abstraction
Since you said the magic words "enterprise architecture" I'm assuming you're already planning to spend a lot of money. That's usually a good way to start, although you have to make sure that its a lot of money relative to the amounts typically spent through your VAR, ISV or the local sales office of your company. Leverage with the local sales office is CRITICAL; these people count on your money and keeping them fat and rich will keep them loyal to you and will enable them to lean on the organization for support in ways that you couldn't. Even if you split your purchases among multiple businesses like you'd like to, you always have the leverage with each one (except IBM, since neither Oracle or BEA sell hardware) to drop its product and move to something sold by the other one. Often referred to as the "divide and conquer" startegy...
Being vocal in support groups (online, real-world, etc) that give you an option to make support statements in front of other users is really helpful. Its one thing for the vendor to lie to you in a private conversation, "We think AIX sucks and Oracle should really be run on Sun.." It's quite another for you to say in front of Oracle and other customers "Why does Oracle have these bugs on AIX?" If your claims are true, they'll have to address the problem in a more constructive manner, especially if you've brought up this problem publicly before. They're much less adept at finger pointing in public.
In a slightly simimlar manner, I had a problem with my home DSL service. My ISP for DSL was also my business ISP. The ISP kept giving me a rash of crap and foot dragging about their inability to deliver advertised bandwidth. Finally I sent a message to the technical suport mailing list for corporate customers asking why DSL bandwidth was so lame and why engineering was too lazy to look at it (not those exact words, but that exact meaning). Within six hours I had an email from a senior engineer and two phone calls from the director of operations and the director of engineering. The point is that public humiliation is a powerful tool..
The downside to buying everything from one shop is that they might tend to look at you as a bit easy. If you were willing to spend all your money with them, why won't you believe everything they say?
You buy single vendor support, usually from one of the big systems integration houses:
The above is obviously a sale blurb (and not directly relevant to your problem at that) but you get the idea: Pay somebody lots of $$$ to make them own the problem. I'm sure there is a growing market for this type of services.
I can't really recommend anybody for your particular problem.
---
"Where do you come from?"
Hi!
I work for IBM (full disclosure :) supporting a major telco. This telco has Sun, HP and IBM servers with the whole lot supported by IBM. I personally support the HP servers and know next to nothing about IBM servers (ironic I think).
So referring back to your question, simply select the bits and pieces that you want to run with and then get a major service supplier to bid on supporting the lot. You'll find that all of them will be more than happy to support whatever you have.
Just choose IBM since they are the best! (wearing my big blue IBM underwear on the outside :-)
If you have a bunch of lousy developers, yuo will be in trouble. Good tech support or not. Chances are that you might get better developers on board if you go for solution X. Once a crap system is build, no tech support in the world can save you.
Even with the best selection of vendors, you cannot escape a certain amount of "not my problem; it's him" finger pointing. The reason is simple; when something breaks, you are not dealing with a hive-mentality supplier where everyone in that company knows everything about the company's products. You will be dealing with individuals. And usually, the first person dispached on a problem call is the lowest-ranked, -paid, and -trained. And a large percentage of such, if they can't resolve the problem, will resort to finger-pointing instead of calling their next level support.
As suggested elsewhere, you (or someone in your shop) will need to be learn enough about the entire installation that you can tell when someone is pouring lemonade in your ear and saying it's rainwater. Don't just give some surplus manager the title of "Architect" and let it go. Find a techie who knows how to back off and observe.
Other suggestions:
Getting reps from all suppliers in the same room is a good idea, but not generally possible until you have been down much longer than you want.
Don't be afraid to "escalate" the problem. If the first response body can't / won't solve the problem, ask for a manager. Managers don't like talking to customers, but the good ones like even less having unhappy customers because of untrained employees or employee attitude. Odds are that the manager may not even know about your problem (other than "there was a ticket but we closed it").
If you find a support rep who really knows her stuff, saying "thank you" never hurts. (Lots of customers think it does.) For a really good job, a note to the salesman (to be forwarded to the support boss) will be appreciated.
Sometimes, it is possible to request that a particular support rep be assigned to your account. If you find a good one, do so.
Bottom line: support reps from the suppliers are not your employees and so do not care quite as much about your company's problems as you do. Since you cannot completely avoid getting involved in problem resolution, get trained. Get multiple people trained.
One good way of minimizing vendors blaming each other is making sure your product works with components from at least two vendors. Then you can always substitute the faulty component of the package, and tell the vendor whose software faulted that "when I replace this with , the problem does not occur". Sometimes it's enough that you support several versions of the same vendor's product, but for best results, you need to support different vendors' products.
For some reason, it seems that when you tell a vendor that you are readily able to replace their software with someone else's, they respond much better to requests for support.
The same works other way around too. If you have vendors blaming each other, you just need to replace one of the components with something else, and you are able to say the problem was or was not corrected by replacing that component.
If none of this is possible for your product, then the only real alternative is to prove your point by producing a test driver that only uses one of the products [again replacing other products with something else], but is still able to reproduce the problem.
-- Esa Pulkkinen
This isn't Dell supporting your new PC with Win98 - you won't have 20-year-old college drop-outs answering the phone. Enterprise Unix support is a big part of the revenue for IBM, Compaq, Sun, HP, Oracle, etc. The people who will be supporting you are professionals, and if they know what's good for them (and their company) they won't play "the blame game" - it just doesn't make business sense at this level. If they try to give you the run around on some critical problem, you will eventually find out (or else your machine will remain down), at which point you'll probably walk to another vendor, taking your fat support contract with you.
Finally! A halfway decent "ask slashdot" question.
--
http://gammatron.weblogger.com
It would be simple, if you (or your staff), knew ALL the ins and outs of ALL these parts. In the real world, this is not likely to happen. But, if you can isolate a problem down to being NOT in two out of those three, then it looks pretty likely that it would have to lie in what's left. Hence "know what you don't know." That is, know what areas you are weak, and what areas you are strong. Use that to your advantage in trying to sort things out.
Often, in my experience, have I found myself going down the debugging path in frustration until I step back and realize the problem is not where I thought it had to be. That meant those areas which were not even a consideration, before, now come back for another look. I had again stumbled upon a case of:
But, I suspect the real desire of this post is not improved debugging skills, but to avoid the need for debugging inter-vendor problems in the first place.
Good luck! It's the nature of the beast that bugs tend to occur in the interfaces.
But, there are some steps that can be taken to lessen the challenge... find someone else who has been down this path before and can point out the landmarks and land mines along the way. I commend you for doing so here in /., but there is more that can be done:
When you get that working and stable, take what you've learned and then add capabilities. Build not your house upon the sand.
Consider, for example, your database vendor. Contact IBM and ask for references of customers who have used their DB2 database in a similar situation. Contact those references and find out what THEY learned from their experience. Do the same with Oracle and MicroSoft. Build up a table of strengths and weaknesses.
Yes, it is a lot of work, up front, but it has been well said:
Yeah, and wait 18 mos for it to happen. In the mean time all your competitors have crushed you in the speed to market game.
Unfortunately, there is no way of avoiding finger-pointing. As others have pointed out, if you go and get a single big vendor, they will keep bouncing you between project groups, who will keep remarking on what a fascinating problem you have or what a strange/nonstandard setup you have.
The best ways to get a problem fixed in a hurry is to keep escalating it. You don't want to be a 'problem customer', but it sounds like you are pumping plenty of moolah into this, so they won't be able to just brush you off.
Ultimately, picking a single vendor will better avoid the fingerpointing between unrelated service techs because you can always find someone who is superior to both of them. But if you pick an inferior system, you'll have to make more service calls.
If it is onsite service, throw a 'tea party'. Get all the techs from each software package which has been blamed, and say that none of them can leave until the problem is fixed. I've had three weeks of back and forth end in two hours that way-- usually the tech isn't going through his whole routine and so is missing something. Once he is in that environment, he will have to do everything, even the stuff he thinks he doesn't have to do.
I guess my answer to your question is to thoroughly explore interoperability before you buy, but then get the best stuff with the best service-- without trying to find the one company for everything. Make sure interoperability is explored before you buy, so the sales reps won't weasel out of it later.
Most of the vendors we have worked with are really good about integration, but one in particular seemed to really stand out. Have you checked out Allaire's JRun 3.0? We tried them out against BEA and really felt like they kicked WebLogic's butt. Here's Allaire's website.
Scrap the expensive and vendor lock stuff. Use FreeBSD (or Linux) use Perl, use DBI and any Database you want. Forget VB, which you will be left stranded and locked into an obsolete path when MS starts pushing C#. Open Source solutions will have significantly more longevity than most closed solutions and whatever closed propriety closed solution you choose you will find yourself in a continous upgrade for $$$ cycle. Oh yes, and Perl is Rapid Development compared to even VB.
I have just gotten out of tech support, so let me shed a little light from that perspective.
What you have to do to avoid finger pointing is ISOLATE the problem BEFORE you call.
For example, you have a problem with the tape drive on your RS\6000, using some non-IBM tape backup software. BEFORE you pick up the phone, try tar. Now you know who to call. Since it is IBM's tape, in IBM's system, with IBM's tar it is IBM's problem if it doesn't work. OTOH, if it works, when you call your backup (or DB) vendor and tell them their software is goofy. If (when) they try to point at IBM you tell them, "Hmm, works with tar."
See?
Again, from a tech support guys point of view, this is YOUR JOB. Using a single vendor makes your job easier, because you don't have to figure out who to call. But it is not the tech support guys fault if you just call the vendor who's product is exhibiting the SYMPTOM, if it is not the component that actually has the PROBLEM. You will be the victim of finger pointing as long as you don't believe it is your responsibility to isolate the problem before you pick up the phone.
Please don't interpret that last paragraph as "finger pointing is the customers fault." It is not. The vendor should say "It is possible, or even likely that the problem is in fooware. Here is how we can make sure our product, barsoft, is working correctly." Bad support techs don't do this. (Actually, this is a management problem, but that's another thread.)
The point is sometimes you will get bad tech support. Doing your homework is how you avoid finger pointing.
-Peter
By going to a good VAR.
If you can find a Value Added Reseller that supports the combination of products you want to use, of course.
But, be very picky about choosing your VAR, as many are little more than fly-by-night operations.
Of course, most VARs will just try to convince you that what you REALLY need is what they already offer, 'trust us, we know this stuff!'. Don't beleive it unless they can show you.
A truly good VAR will listen to your needs, and try to support your choice of products with a comprehensive support contract. That is, of course, what 'Value Added' means.
--
The real Webmaven is user ID 27463. I don't rate an imposter, because my ID is such a lame-ass high number.
Fast forward a month, when it comes time to pay the bill, i get not one, but two. The first dealership just dripped with incompetance, and forgot to cancel the initial account. Ok, so i call Roger's tech support to get this corrected... only to find out that want me, personally, to go back into the incompetent dealership to make them fix the problem, despite the fact i had no contract with Rogers on that number, and that it was an internal screwup.
The point of the story... yes, while they kept putting the onus on me to fix the problem their incompetence created, I kept insisting that it was their fault, I had no contract for that number, and therefore was under no obligation to fix it. I kept increasing the asshole factor (asking to talk to the supervisor, etc.), until eventually they caved in, and the supervisor personally called the incompetent dealer that messed up, getting the problem fixed.
The lesson to be learned is that... well, put enough pressure on them when it is obviously their responcibility, and wonderful things will happen. =^)
-legolas
i've looked at love from both sides now. from win and lose, and still somehow...
(Commercial) support like you say is just one of the arguments upon which the enterprise design should be based. I think there are some other points that are more important.
1) Human resources.
How many administrators and developers can you bring together with good knowledge of those IBM components (AIX, Websphere, DB2)? Here Linux or FreeBSD clearly beat AIX. For webapplication developers with Java and RDMS skills the scale (for now) is probably a bit more even, due to healthy competition between the different webapplication frameworks.
2) Product Continuity.
The basic architecture you lay down now will probably be around for a few years, maybe even more, so having your products evolve in sync with new standards is very important. OSS projects, by their very nature, nearly always tend to more aware of interoperabillity than their closed source equivalents.
3) Security, Robustness, Scalability, Performance.
I think the the OSS models, in general create products that score quite good on each of these points, especially when they have been activly developed on during more than a few years.
4) Product functionality.
The XML and Java features of the Java Apache Project or the entreprise ready Enhydra server (with commercial support from Lutris) are already more advanced than those of the IBM Websphere modules. Combine that with an excellent RDMS (with full JDBC driver support) like PostgreSQL and I think you got an excellent webapplication platform that is able to quickly evolve with future technology extensions.
- just my Euro 0.02c
In a sentance, nothing beats having skilled employees that know there stuff!
Ands it really does not matter what OS/ DB/ EJB's you use, it matters more how well the people that you are getting to do the job, know what they are doing.
send flames > /dev/null
Only 'flamers' flame!
Not that this is purely an IBM thing. I've had support people at several major corporations try to give me the run around, pass the buck or blow me off. Having been on their side of the phone, I can generally spot this and I don't put up with it. If I have to call support, I'm having a problem their frontline has never seen before and I'm not about to put up with some $8 an hour phone monkey jerking me around. I tell them that too, and ask for their manager. I tried to provide the level of support I'd want on the front lines when I was working the OS/2 support desk at IBM Boca and my manager told me that if I didn't get my numbers up, they'd fire me. Most of the frontline drones are idiots, because the smart ones either get promoted out rapidly, quit in disgust or get fired because some fucking bean counter would prefer quality to quantity. I ended up getting promoted out of phone support at Boca, in record time.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
I can't help but jump in on this...
Tech's notes on avoiding the run around with commercial vendors:
* Document everything. Every configuration from the Prom to the OS to
the details of your custom application must be in hardcopy.
* Don't call anyone until you have every relevant piece of information
you can imagine (and there will be some that you can't imagine).
* Keep daily tabs on the products you use- Subscribe to the newsgroups,
announce lists, etc., check one of the Usenet searches like deja weekly,
join a users group.
* Train at least two people in your organization on each product
and make them responsible for it. Get backing for a policy that no
closed-source, proprietary, or commercial product can be used unless
there is in-house knowledge.
* Call the sales reps and pre-sales engineers monthly and ask them about
patches, notices, updates, etc.
* Never buy a product that only runs on one platform or with one
back-end. Best-of-breed also means best chance of survival, so don't get
locked in.
* Fight against using proprietary or non-standard anything when
possible. Stay as flexible as possible but don't think for a second that
open-source will save you from a knowledge deficit.
* Always have more than one product of a given type under testing.
* Negotiate with multiple vendors but try not to let on with which ones
you are in contact.
* Get a support contract from the vendor and from a third party. Use the
third-party to hassle with the vendor for you.
* When selecting a product try to get a written statement that the
product works as described and what the vendor's responsibilities are.
Run everything past your legal people.
We work vendors for a large variety of products, from databases to CORBA OBR's, etc.
;)
The key to resolving these conflicts, as far as we have determined, is knowing EXACTLY what the problem is, what is causing it, and then contacting the appropriate vendor.
Every vendor product has a well defined (at least, in theory) interface and functionality for their product. For example, an Oracle database will have an API for the embedded SQL library, and an ORB vendor will state their CORBA compliance.
If something is not working correctly between components, it is either user error, or a flaw in the component. (or unsupported functionality, but you should have verified this before purchase right?
At this point, you know exactly where the problem lies, and which vendor owns the problem.
The problem with going with a single vendor is are you really getting a single vendor. Outsouring is a major problem in the industry. The quality level of the tech support is pretty much related to the turn-around rate at the centre you get connected to. So even if you were to go with IBM, that doesn't mean you won't get sent over to a support centre run by Stream or some other outsourcer of ill repute.
And even when you do get the actual vendor will they even care. Will they speak f*ck'n english? A former tech support person for a major database company once told me that it was standard practice to blame it on the OS first. If the customer was actually able to prove the problem was with the DB then they'd submit a bug. But the point was it was up to the customer to do all the leg work.
I think we as techs are to blame though. It's our way of doing things in IT. You've seen the SNL skit with the bastard tech support people. Sure they couldn't get the terms right if their life depended on it, but the point is we have huge ego's and little personal responcibility. Even played the blame game inside the company when someone left?
BOSS: "Why isn't this project done?!?!"
TECH: "Ahh..yeah..Habib was supposed to do that before he left the company."
BOSS: "Damn that Habib!"
At any rate, it's been said before, quality people. Also, if you have the money, go with a vendor that will offer "internal" classes for the product. HP for example has special "Internal Use" classes they will let customers go to. It might cost a little more. Hell, you may have to send your people to the UK for them. But it's well worth it to know what's going on under the hood.
Although this sounded like a good idea, practically it was hopeless. We had to farm all hardware support out to a 3rd party, and finding a competant 3rd party that was aware of our OS was impossible. We also had to learn all the applications and other tools used by the customer, not all of which we had supplied.
One customer was a nightmare. They used us for everything. They'd call us for support on a DOS application that was over 8 years old, wondering why it couldn't handle PentiumMMX processors. They'd call us to download printer drivers for their 5 year old printers. They'd even get us to call hardware engineers to clean a mouse. Yet they wouldn't upgrade their network to even a remotely current release!
Eventually we stopped doing hardware, as we were loosing money.
I've actually seen IBM fix something. Fast. Once.
Heh, I had a good experience with their storage group in Austin when I was working for their Internet Division in White Plains.. Had to get a 7135 RAIDiant array working with AIX 3.2.5, but the controller and drive microcode apparently had issues with the most recent PTF sets (thank you fixdist!) loaded on the box, so I ended up being escalated to an engineer who helped design and build the thing, and he FTP'd me the latest microcode fixes that day, had everything working fine the next..
The next best thing to opensource!
Your Working Boy,
How do I prevent (or at least minimize) them blaming each other when support issues come up? Anyone have a solution for this?
Before you buy, make sure that your prospective vendors are members of TSANet. Then, if one of your vendors points the finger at another, tell them to use TSANet to open a ticket with the other vendor on your behalf. As a support engineer at Tivoli Systems, I've used TSANet to open tickets with other vendors in order to resolve customer problems.