Slashdot Mirror
How Would You Make a Distributed Office System?
Necrotica writes "I work for a financial company which went through a server consolidation project approximately six years ago, thanks to a wonderful suggestion by our outsourcing partner. Although originally hailed as an excellent cost cutting measure, management has finally realized that martyring the network performance of 1000+ employees in 100 remote field offices wasn't such a great idea afterall. We're now looking at various solutions to help optimize WAN performance. Dedicated servers for each field office is out of the question, due to the price gouging of our outsourcing partner. Wide area file services (WAFS) look like a good solution, but they don't address other problems, such as authenticating over a WAN, print queues, etc. 'Branch office in a box' appliances look ideal, but they don't implement WAFS. So what have your companies done to move the data and network services closer to the users, while keeping costs down to a minimum?"
Or, in other words, how do i put servers in branch offices without putting servers in branch offices?
If you solve that one let me know...it's been bothering me a while too...
Such as OpenAFS.
Something like coda might be nicer but progress on global filesystems seems to have pretty much stalled.
Deleted
Financial. Liability.
There is no good and cheap solution to this one.
You can try the application accelerators that are out there now from Cisco. They basically use smoke and mirrors to keep traffic off the WAN and act as local proxies for different services.
Otherwise, your choices are limited. Citrix servers would be good for some apps, but get god-awful expensive fast. And an organization too cheap to build out a decent system to begin with isn't likely to make the investment in writing efficient apps.
If you're running on slow lines, bump them to at least fractional T3.
It sounds like the system was designed to serve 5 gallons of water through a swizzle stick. Ain't gonna work unless something is radically changed.
Or better....
Fire the outsourcing partner and the management that buys their bull, and build out a proper distributed archetecture.
Financial companies, at least in my State, have very specific requirements for storing and transmitting data. Without knowing what your specific needs are, I have no answer other than "Define your problem".
The reality is other companies, such as yourself, exist and function probably better. If that indeed is the case, perhaps a friendly lunch with another IT staff member might help you.
I've consolidated offices and I've also pushed out servers to remote offices. It all depends on the need of the client. Examples
1. Client wanted 99.999% uptime and the only way I could get that was to have their servers in a data center. We moved them and uptime has been great.
2. Client wanted fast file access. We setup DFS with WIndows 2003 over a WAN link (T1) the client has never been happier.
So, to answer your question, it depends on your needs.
Find a new partner.
So basically, -1 troll/offtopic is really slashdots way of saying "I hate that you thought of something before me."
Some basic truths.
IT costs money. I'm sorry that your outsourcer had some bad ideas. But your management must understand that IT services aren't free, and the health of your company depends on it's infrastructure.
Without knowing the specifics, the only low cost suggestion I can provide is converting desktop PC's into Linux servers, thus providing you with the distributed server network you need. Of course, the boxes will be underpowered and fall over all the time (yay desktop hardware), but if you really want to cut costs, there you have it. For backups, put in extra hard disk and backup to disk, it beats nothing at all.
I am government man, come from the government. The government has sent me. -- G.I.R.
Think happy thoughts, and sprinkle some pixie dust over your IT infrastructure, and all your problems will be solved.
But whatever, you do, don't fire your incompetent outsourcing partner or actually invest in beefing up your IT resources. Both of those paths are DOOMED, DOOOOOOMED, I say!
Moderator hint: a comment is neither "Flamebait" nor "Troll" if it is true.
Dedicated servers for each field office is out of the question ... such as authenticating over a WAN, print queues, etc
Print queues over WAN is taking the consolidation thing a little to the extreme, isn't it? Login authentications and print jobs really want to be local. Sorry about your predicament but you're going to get a lot of comments telling you to switch outsourcers or bite the bullet on their prices. What is the other traffic (as if that isn't bad enough): one assumes email, but are there big apps hosted on remote servers with lots of data traffic to db servers and the like? Simple document file sharing shouldn't be that much of a problem, or is it? You're going to get a lot of guesses without knowing the exact needs of your remote traffic. Good luck!
and eating it too? Is it just me, or is this one of those situations where upper management makes a design decision from something they glanced over in some IT mag, then decided to implement without consulting anyone with any IT background?
I don't see how you can create an insanely diffuse network, then turn around and expect it to perform like a network that has a centralized "HQ" with file services etc and a fat WAN connection.
Of course, you could just ask the execs to spring for ~100 WAN accelerators... =)
I suggest you pay more attention to the data itself. Do an comprehensive and brutaly unbiased audit of what data/resources are needed by whom. You would be amazed at how much of your infrastructure is either superfulous or capricious. Once you do this then you at least have a smaller mountain to climb.
"This message was sent from an Apple
The only responsible answer to this question is to get someone in that has a track record of fixing problems like this. Don't expect to get a reasonable answer from a sketchy problem definition in a place like slashdot.
Engineering is the art of compromise.
Either consolidate your servers, or don't.
Exactly what costs were you thinking of saving by consolidating? If it's just the cost of building and maintaining those physical servers, then here is the cold, hard truth: You are paying less for less service. Put servers at each branch office if you'd rather pay more for more service.
You get what you pay for.
Now, if it's other problems that are keeping you from setting up those dedicated boxes, realize that these are other problems. Identify them, and bring them back to Ask Slashdot. We're Slashdot, we're not psychic.
If it's your outsourcing partner gouging prices, dump them for an outsourcing partner which doesn't gouge prices, or do it in-house.
If it's the inability to manage all those servers, get them to talk to each other, etc, that's a more interesting technical problem that Slashdot might be able to help solve.
There are a few exceptions -- you might be able to get away with something like Coda or AFS, though I don't know how well that scales to crappy bandwidth. But if so, that would imply that your only problem is managing strictly filesystem data -- it doesn't help at all if the problem is access to, say, an intranet webapp. So again, we need details, if we are to find the clever exceptions.
Otherwise, upgrade your bandwidth, and/or outsource your actual application servers to someone who can scale. If it's just web/email/docs, Google can do that. Otherwise, find someone who specializes in what you're doing (our SVN is run by cvsdude.com), or bite the bullet and buy some virtual servers.
Don't thank God, thank a doctor!
Checkout Riverbed, Cisco, and many others. Basically they do caching, compress traffic, do TCP/IP traffic control the way it should be done (with the hindsight of 30+ years experience) and some application specific round-trip optimization (some even do voodoo optimization :).
Not cheap - but easy.
WSUS servers out at all locations is fairly costly as it requires a decent server and Win2K3. That could be a lot of extra hardware and licenses to buy/support. Unless your company needs to run full bandwidth 24/7, just schedule your updates for the middle of the night and it doesn't matter there is only one server pushing it out. I currently do this for my company that has 30 branches, half overseas, and all on slower connections than I would like. Windows Updates are the lowest bandwidth concern of mine now, because they happen once a month and when no one is even around to notice.
... seems to be that your oursourcing partner has you on the Merry-Go-Round. They work it like this...
1. Propose a WAN-based solution.
2. When that slows to a crawl, propose a branch server solution.
3. When that proves to be too expensive to administer, propose a centralized solution.
4. When that proves to be difficult, unproductive, or slow, propose a branch office solution with accelerators, DFS, and all the goodies.
5. When that proves too expensive to administer, propose a thin client/remote app solution.
6. Repeat steps 2-5 as needed, substituting current technology for at least three iterations.
7. If you still have this client, you may now feel free to propose ANYTHING, including cans and string, or gerbils. They will buy it. Change your technical onsite staff every 6 months, rotating in fresh and untrained candidates. Rotate out those who show promise to be re-deployed at newer clients who are at step 4 or earlier in the process.
It's kinda sad. Consulting outfits can rarely make a living by doing right for a large client. Sooner or later, they either get replaced when the client starts 'analysing' the operation, or get replaced when some other outfit has a stronger line of bull to offer management.
Of course, there's incompetence, but my former boss isn't involved. He's busy screwing people in a different business, when he's not busy screwing his employees.
deleting the extra space after periods so i can stay relevant, yeah.
I've done a light evaluation of riverbed's steelhead appliances in the past (less from the efficiency stand point and more for manageability). To call it a dell server with centos is an understatement since there's a lot of software intelligence intercepting various protocols and caching the data that may be transmitted. Handling file locking, multiple email recipients of the same large attachment, and be transparent to the network, aren't easy problems to solve at the protocol level, so I'd say they deserve a few kudos. They weren't a simple WAFS, multiple protocols were included, it would simulate the reply from the remote server when possible, and all traffic to another data center or office with a steelhead would be compressed regardless of protocol (it's been a few years, so feel free to double check those facts). I believe they also included some physical bypass hardware so if the box completely died or needed to be rebooted, you wouldn't lose your network. All in all, I thought it was a nice solution. And no, I have no affiliation with the company.
I used to work in the high tech industry with companies that made lots and lots of money. These companys had the fastest bandwidth, and the most creative people coming up with cool solutions to solve problems. But basicly the point was, everyone made lots of money, so if IT infastructure was a problem, they threw money at the problem, and it was solved...period. Since that time, I have seen general compression of the $$ side of things, the bright people go somewhere else, and the people outsource the smart clever IT folks that worked at the the tech company to some outsourcing firm...
and all the call centers are shipped off to India.
So... I think... where is all the money now, and clever people?
Google.
Just ask Google to host your IT applications, they already index the rest of the damn web anyway.
This would beat Googgle to their next big thing anyway... why not just host the world at Google?
Storing your sensitive financial information will be just a spec of content compared to the rest of the web. Then buy some good fiber connections from Verizon. (I'm spoiled with my FIOS service at home...better than the DSL at my companies remote office)... and viola, problem solved. Besides, then anyone can get to your data from anywhere.... the security issue is a myth... who has time to look up all this financial information anyway... most people are reading Dilbert cartoons about how your company outsourced the network.
Plus, you can tell all your clients to buy Google stock, prior to handing over all the data.
-- R
Ross Youngblood
Just have everyone telecommute to the central office. Problem solved!
Tsunami -- You can't bring a good wave down!
Of a good a idea that worked well in one area but is not ready for full adoption. Wide Are Network has too much latency to simply turn local office systems global.
Your company is trying to cheat their development model. Rather than setup a distributed IT application they have simply tried to distribute a small office network worldwide. If you look back to the tried and true OSI model. 7 layers. The 7 layer model doesn't speak of Network File Sharing, it speaks of Hardware and Application. TCP/IP (which we have taken quite for granted) is around/below the application level. If you have an application that runs at the TCP/IP level you are good to go.
I have setup distributed systems for several ISPs in the late 90's. We didn't think about what we were doing or why it worked. It looked like we could long haul anything we wanted. A little lag in sending mail or a few extra milliseconds to authenticate LDAP is no big thang. The Internet is distributed by nature. Sometimes DNS was a little slow but that was acceptable for 56k modems and DSL customers. But we spent 2 years working on a central web based administration/billing/customer support application with 1 SQL base in the center. We didn't distribute the application and have it write to the SQL base directly or move files around.
But you can't distribute the file layer. SANs in a local building have had some of the same problems. Any lag affects all applications and you solve it by throwing a big fat fiber backbone in the local building, but it break downs when you try to long haul over WAN links.
If your company is thinking it can sneak around coming up with a decent workflow model, and then implementing that in an application by simply given MS Office and Exchange (or whatever they have employed) to everybody they are sadly mistaken.
But worry not. You are not alone. Many business execs scratch their heads as to why the simply can't share out MS Project and their Excel Spreadsheets to 25 plus people teams and it will work fine. You still need to do the leg work of figuring out the work flow and reducing that to a transaction based system centrally located. That's it. All we've done in the last 20 years is replaced printouts with emails and spreadsheets, and the night operator (a job I used ta do) with scripts (or procedures) that dynamically update or run every 10-15 minutes. You still need a central system and then distribute parts of it, or have slim down interface that everyone can use remotely. Look at how a bank does it, just good ole dumb terminals.
No magic bullets yet. We need faster broadband and much lower latency before you can share out at the file layer using a network stack meant for transaction based appilications.
Let yourself off the hook. No mortal IT person can turn this tide....
You need local servers to reduce the latency. You need some decent thought on the application, not the OS and Office Suite. Good luck!
"Don't fear death... fear not living..." -me