Slashdot Mirror
How Would You Make a Distributed Office System?
Necrotica writes "I work for a financial company which went through a server consolidation project approximately six years ago, thanks to a wonderful suggestion by our outsourcing partner. Although originally hailed as an excellent cost cutting measure, management has finally realized that martyring the network performance of 1000+ employees in 100 remote field offices wasn't such a great idea afterall. We're now looking at various solutions to help optimize WAN performance. Dedicated servers for each field office is out of the question, due to the price gouging of our outsourcing partner. Wide area file services (WAFS) look like a good solution, but they don't address other problems, such as authenticating over a WAN, print queues, etc. 'Branch office in a box' appliances look ideal, but they don't implement WAFS. So what have your companies done to move the data and network services closer to the users, while keeping costs down to a minimum?"
Or, in other words, how do i put servers in branch offices without putting servers in branch offices?
If you solve that one let me know...it's been bothering me a while too...
Such as OpenAFS.
Something like coda might be nicer but progress on global filesystems seems to have pretty much stalled.
Deleted
Financial. Liability.
There is no good and cheap solution to this one.
You can try the application accelerators that are out there now from Cisco. They basically use smoke and mirrors to keep traffic off the WAN and act as local proxies for different services.
Otherwise, your choices are limited. Citrix servers would be good for some apps, but get god-awful expensive fast. And an organization too cheap to build out a decent system to begin with isn't likely to make the investment in writing efficient apps.
If you're running on slow lines, bump them to at least fractional T3.
It sounds like the system was designed to serve 5 gallons of water through a swizzle stick. Ain't gonna work unless something is radically changed.
Or better....
Fire the outsourcing partner and the management that buys their bull, and build out a proper distributed archetecture.
Financial companies, at least in my State, have very specific requirements for storing and transmitting data. Without knowing what your specific needs are, I have no answer other than "Define your problem".
The reality is other companies, such as yourself, exist and function probably better. If that indeed is the case, perhaps a friendly lunch with another IT staff member might help you.
I've consolidated offices and I've also pushed out servers to remote offices. It all depends on the need of the client. Examples
1. Client wanted 99.999% uptime and the only way I could get that was to have their servers in a data center. We moved them and uptime has been great.
2. Client wanted fast file access. We setup DFS with WIndows 2003 over a WAN link (T1) the client has never been happier.
So, to answer your question, it depends on your needs.
Find a new partner.
So basically, -1 troll/offtopic is really slashdots way of saying "I hate that you thought of something before me."
www.simdesk.com
They are currently in ASP mode but they are working to package their solution for installation into a company datacenter.
Put WSUS severs at the offices to keep update bandwidth down.
Some basic truths.
IT costs money. I'm sorry that your outsourcer had some bad ideas. But your management must understand that IT services aren't free, and the health of your company depends on it's infrastructure.
Without knowing the specifics, the only low cost suggestion I can provide is converting desktop PC's into Linux servers, thus providing you with the distributed server network you need. Of course, the boxes will be underpowered and fall over all the time (yay desktop hardware), but if you really want to cut costs, there you have it. For backups, put in extra hard disk and backup to disk, it beats nothing at all.
I am government man, come from the government. The government has sent me. -- G.I.R.
I use the cisco waas boxes with some success.
They're not perfect but I clocked CIFS going about 30% faster.
Just run good thin clients in the remote office. Such as the Sun Ray.
Think happy thoughts, and sprinkle some pixie dust over your IT infrastructure, and all your problems will be solved.
But whatever, you do, don't fire your incompetent outsourcing partner or actually invest in beefing up your IT resources. Both of those paths are DOOMED, DOOOOOOMED, I say!
Moderator hint: a comment is neither "Flamebait" nor "Troll" if it is true.
Dedicated servers for each field office is out of the question ... such as authenticating over a WAN, print queues, etc
Print queues over WAN is taking the consolidation thing a little to the extreme, isn't it? Login authentications and print jobs really want to be local. Sorry about your predicament but you're going to get a lot of comments telling you to switch outsourcers or bite the bullet on their prices. What is the other traffic (as if that isn't bad enough): one assumes email, but are there big apps hosted on remote servers with lots of data traffic to db servers and the like? Simple document file sharing shouldn't be that much of a problem, or is it? You're going to get a lot of guesses without knowing the exact needs of your remote traffic. Good luck!
and eating it too? Is it just me, or is this one of those situations where upper management makes a design decision from something they glanced over in some IT mag, then decided to implement without consulting anyone with any IT background?
I don't see how you can create an insanely diffuse network, then turn around and expect it to perform like a network that has a centralized "HQ" with file services etc and a fat WAN connection.
Of course, you could just ask the execs to spring for ~100 WAN accelerators... =)
I suggest you pay more attention to the data itself. Do an comprehensive and brutaly unbiased audit of what data/resources are needed by whom. You would be amazed at how much of your infrastructure is either superfulous or capricious. Once you do this then you at least have a smaller mountain to climb.
"This message was sent from an Apple
Riverbed Steelhead appliances or similar products?
The only responsible answer to this question is to get someone in that has a track record of fixing problems like this. Don't expect to get a reasonable answer from a sketchy problem definition in a place like slashdot.
Engineering is the art of compromise.
Any application that won't run in a Firefox window is unneeded and merely distracts from the company's core mission. You won't believe how much of a performance boost you will get when you shut down those apps.
Laws affecting technology will always be bad until enough techies become lawyers.
Whatr you are looking for is keeping central and local files synchronous, allow for dodgy connections +/- disconnects, be fast locally and yet have everything centrally.
Is this not a case for CodaFS?
We don't know which country you're in (and hence which set of regulations you have to adhere to).
We don't know how much data needs to be made available to each office - is it everything? Or is it just a different subset of the total in each office?
We don't know if you're talking about megabytes, gigabytes or terabytes of data. We also don't know how much that data changes on a daily basis.
We don't know if there are any existing factors to consider - be they political or technical (eg. "management almost certainly won't contemplate anything without Microsoft or Cisco plastered all over it").
If it helps, I can tell you what I've done - but I only have two branch offices I need to worry about, no financial regulation and my manager is more interested in saving money on server and client access licenses than buying whatever Microsoft deem to be the Next Big Thing . Each branch office has its own server running Debian Etch as a VMWare host and a number of virtual machines - including a fileserver, DNS and LDAP slaved from head office for authentication. About the only thing that needs backup is the fileserver, and that is done by nightly rsync to head office, and thence to tape. Provided the data doesn't change too drastically (at a rough guess, I can probably handle up to 2-3GB of changes per day while remaining within the backup window) I should be OK. You could probably achieve a similar net effect with Active Directory and DFS.
This is one of those questions where the only real answer is "it depends"
Start by assessing what services and applications are accessing the network or putting an undue load on it. Once you have the information from that assessment you can start looking at how to reduce that load.
Can you get decent performance by setting up a few remote servers at your larger offices, while keeping your smaller offices on the existing system?
Will adding database replication servers to some offices reduce the WAN load?
Will adding bandwidth to sites 22 and 44 make the performance in those offices acceptable? Does this take enough traffic off the central system to make the existing system usable?
If you add a database replication server to site 66, could you then have the dedicated lines from sites 88 and 55 changed over to link to site 66, and access that replication server?
If you don't have the expertise to do this, hire someone that does.
It's a no go, OpenAFS and kerberos is a very nice idea, but it doesn't work, the client software for most platforms is very bad.
Either consolidate your servers, or don't.
Exactly what costs were you thinking of saving by consolidating? If it's just the cost of building and maintaining those physical servers, then here is the cold, hard truth: You are paying less for less service. Put servers at each branch office if you'd rather pay more for more service.
You get what you pay for.
Now, if it's other problems that are keeping you from setting up those dedicated boxes, realize that these are other problems. Identify them, and bring them back to Ask Slashdot. We're Slashdot, we're not psychic.
If it's your outsourcing partner gouging prices, dump them for an outsourcing partner which doesn't gouge prices, or do it in-house.
If it's the inability to manage all those servers, get them to talk to each other, etc, that's a more interesting technical problem that Slashdot might be able to help solve.
There are a few exceptions -- you might be able to get away with something like Coda or AFS, though I don't know how well that scales to crappy bandwidth. But if so, that would imply that your only problem is managing strictly filesystem data -- it doesn't help at all if the problem is access to, say, an intranet webapp. So again, we need details, if we are to find the clever exceptions.
Otherwise, upgrade your bandwidth, and/or outsource your actual application servers to someone who can scale. If it's just web/email/docs, Google can do that. Otherwise, find someone who specializes in what you're doing (our SVN is run by cvsdude.com), or bite the bullet and buy some virtual servers.
Don't thank God, thank a doctor!
So how can you eat it?
Checkout Riverbed, Cisco, and many others. Basically they do caching, compress traffic, do TCP/IP traffic control the way it should be done (with the hindsight of 30+ years experience) and some application specific round-trip optimization (some even do voodoo optimization :).
Not cheap - but easy.
It's time for your company to seriously examine your outsourcing company's contract with you. The consolidation recommendation obviously did not fully examine the needs of the remote offices. They have to bear some of the brunt of this mistake ...or lose their contract with cause.
Server consolidation is great for centralized offices. Until we reach the bandwidth critical mass where the pipe is wider than the need, removing server capabilities from satellite offices is a ridiculous idea. Even if it's a store-n-forward device, you will need local access capabilities.
There is really no excuse for the consultancy making a flub this big. They should either be fired or forced to float the cost difference for their mistake. In the long run, you should look at replacing them anyway. You don't want the company's crown jewels in the hands of incompetents.
I have been looking at this product for a similar situation I am in: http://www.packeteer.com/products/ishaper/
Basically it is a WAFS box, with WAN traffic shaping, caching, etc, plus it acts as a Domain Controller, print server, authentication, dns/dhcp, etc.
If it works like they say it will it would be a good solution for you based on the problem description. Basically it is a server, plus WAFS, without being a server...
I wonder if anyone here has some hands on experience they could share?
Might well be a nice solution, assuming that your remote users are frequently throwing large files around.
ICA, RDP, and some X variants work well over slow connections. Do applications need to be executed locally, or can you run a farm of application servers with fast connections to the storage. Then put diskless, fanless thin clients (I typically use Wyse V50s), which DHCP configured to give them a config file to load on each startup. This gives you data security (no data is stored locally, or even at a branch office like your situation - someone steals a thin client, you are only out the hardware, application roll outs and updates are centrally managed, no rouge software can be installed (i.e. no weather bugs, cutsy screensavers, etc) by users, and many more advantages.
I publish applications via Citrix (Windows Apps.) and X (*nix apps)...They run on the same thin client desktops and the user knows no difference as to which server the application is actually running on - it appears local to them...I've also experimented with publishing OS X applications via vnc, but that requires the whole OS X desktop be served (not just the application)
We're a largeish company with one HQ (and associated data center), about 400 field offices, and four regional field service centers. Our approach was to centralize everything but printing, but that means EVERYTHING -- so people use Terminal Services to go into HQ. This means that once they've done the TS hop, everything is local, because they're accessing their files, running their apps, and accessing databases locally to where the terminal server is. Printing is, of course, still done in the office, via print servers in HQ.
The users don't seem to complain of speed issues -- then again, this whole thing is running on fairly old hardware (6-7 year old PCs) in the field, and they're not doing anything particularly high-performance (e.g. video).
Check out a company called Riverbed, http://www.riverbed.com/ they have a WAN optimization appliance called Steelhead that solves the exact problem you are describing. I won't turn this post into a sales pitch -- read their website, call them up and ask for a demo, then decide for yourself. I would insist on a proof of concept or pilot implementation before making an enterprise wide committment.
Given what the US markets are going to do tomorrow, it wouldn't surprise me if this is the way that your management chooses to "solve" the problem.
What I did in a previous job was implement terminal services across the board.
Stuck an AD server in each remote offices for workstation authentication, dns, dhcp, updates, etc.
Files were stored centrally.
Accessibility was increased (eveyone had access to their files which ever office they were in without them being dragged across the network.
Bandwidth has grown as the number of people in offices (and the amount they print) has grown.
... seems to be that your oursourcing partner has you on the Merry-Go-Round. They work it like this...
1. Propose a WAN-based solution.
2. When that slows to a crawl, propose a branch server solution.
3. When that proves to be too expensive to administer, propose a centralized solution.
4. When that proves to be difficult, unproductive, or slow, propose a branch office solution with accelerators, DFS, and all the goodies.
5. When that proves too expensive to administer, propose a thin client/remote app solution.
6. Repeat steps 2-5 as needed, substituting current technology for at least three iterations.
7. If you still have this client, you may now feel free to propose ANYTHING, including cans and string, or gerbils. They will buy it. Change your technical onsite staff every 6 months, rotating in fresh and untrained candidates. Rotate out those who show promise to be re-deployed at newer clients who are at step 4 or earlier in the process.
It's kinda sad. Consulting outfits can rarely make a living by doing right for a large client. Sooner or later, they either get replaced when the client starts 'analysing' the operation, or get replaced when some other outfit has a stronger line of bull to offer management.
Of course, there's incompetence, but my former boss isn't involved. He's busy screwing people in a different business, when he's not busy screwing his employees.
deleting the extra space after periods so i can stay relevant, yeah.
There would be two major paths I would investigate.
If you're in a Windows environment, look at getting Citrix (or something similar) set up. Centralized files, centralized management, and it works very well. The one major issue is printing, although we use a product called Uniprint at work that is fucking fabulous. We went from 60% of helpdesk calls being "reset print spooler" down to 0% when we rolled out Uniprint. Very impressive stuff. We use Citrix at work primarily for our DB-intensive apps (so we don't return millions of rows over the VPNs, just the end result via the user interface), but we do have it in use for Word, Outlook, Excel, etc, as well.
The other option is WAN acceleration. There are many vendors that have them now (Juniper, Cisco, Packeteer, yadda yadda). They're expensive and I'm not sure how well they work if each office only has a few users (only a couple people may not 'seed' the cache sufficiently to make a major impact), but I've heard they work well for larger offices.
http://www.riverbed.com/products/appliances/
or something similar; I mention Riverbed because it is what we use. Good luck.
But seriously, it sounds like your company followed some pretty bad advice. It may have allowed you to cut costs, but it also introduced a new set of problems for which there is no cheap and easy solution. Except perhaps what I've outlined above. Yes, strictly speaking thatt would mean adding "dedicated servers", but it would not be an expensive solution and it certainly sounds a lot less expensive to me than your current daily loss of productivity from 1,000 employees.
I've done a light evaluation of riverbed's steelhead appliances in the past (less from the efficiency stand point and more for manageability). To call it a dell server with centos is an understatement since there's a lot of software intelligence intercepting various protocols and caching the data that may be transmitted. Handling file locking, multiple email recipients of the same large attachment, and be transparent to the network, aren't easy problems to solve at the protocol level, so I'd say they deserve a few kudos. They weren't a simple WAFS, multiple protocols were included, it would simulate the reply from the remote server when possible, and all traffic to another data center or office with a steelhead would be compressed regardless of protocol (it's been a few years, so feel free to double check those facts). I believe they also included some physical bypass hardware so if the box completely died or needed to be rebooted, you wouldn't lose your network. All in all, I thought it was a nice solution. And no, I have no affiliation with the company.
Let's see...
1000+ employees/100 locations = 10 employees/location = 0.25-0.5 FTE in IT per location.
Step 1. Hire an inhouse IT staff to operate core systems.
Step 2. Deny outsourcing partner a role in testbed project.
Step 3. Choose 5 remote sites for testbed.
Step 4. Hire 2 IT support professionals for testbed remote sites.
Step 5. Implement inexpensive directory server for each testbed site.
Step 6. Configure a VPN over DSL for each testbed site.
Network printers these days don't need a print server. But if you feel you prefer one, use the directory server. It won't be breaking much of a sweat handling the authentication of 10+ employees and synching with corporate.
Client-server interaction that needs to happen between offices can happen over the 3Mbit DSL line. That should easily handle the traffic of 10+ employees.
Because you haven't provided any details of the nature of the inter-office data traffic, it's hard to design any further than that. However, it might be completely appropriate to make all the user machines in the testbed offices be thin clients, netbooting off the directory server.
It is a bit odd to me to hear a company with 1000+ remote employees (and some additional non-remote employees?) skimp on buying office servers. If you can afford to pay 1000+ employees (including 100 office managers) and pay the rent and utilities on 100 remote office locations... Can't you spring a little money for an office server?
Figuring costs for just one of the five testbed sites averaging 10 employees:
$300K Salaries (1)
$120K Benefits (2)
$ 20K Office rental (3)
Ignoring furniture, utilities, PCs, security, janitors, etc., etc., each remote site costs $450K per year to operate. Let's round it to an even $500K. And that is still on the very conservative side.
Over the three year life of a server, that is $1.5 million to operate the remote location.
But we can't spring for a $1500 server?
* The sample numbers above are extremely conservative and could easily be double those shown here. For instance, a site with 15 employees could easily cost $5 million over three years. And you still can't squeeze for a $1500 server?
(1) Figuring an average salary of $30K. This is obviously a very wild gueww, having no clue of industry, geography, etc. The numbers here are all very conservative.
(2) Figuring 40% labor burden.
(3) 10 employees * 100 s.f./employee. Again this is very conservative. With bathrooms, water coolers, and other common areas thrown in, this would be a very cramped sweatshop. Figure $20/s.f./year.
Here where I work, we replaced pretty much all the conventional applications (the ones which are required globally within the organization) for web-based ones. No, it didn't happen from a day to another.
.doc/.xls/etc documents and stuff like that. Such cases are processed locally and only the relevant files are sent (either through FTPS or e-mail), SMB shares are not transported through WAN at all.
We have pretty much everything centralized, except cases when you simply cannot escape from
It helps our structure reflects (most of the time) the physical segmentation of our organization.
Currently most of our (typical) traffic is HTTP (~80%) and e-mail (>10%).
We do have quite tight WAN links (1Mbps in most cases, slower in other places) so we apply a fairly elaborate QoS and, for HTTP besides the obvious local HTTP cache we also compress that with Ziproxy (what renders it less than half its size, in our case).
Hard drive space is very cheap. You could probably replicate all your company's databases in every office. That leaves you with the problem of syncing the databases but there are some solutions. Lotus Notes took that approach about fifteen years ago iirc. It worked well. WAN traffic was greatly reduced and performance was quite acceptable even with the slower WANs of the day.
I haven't done this kind of thing for a while so I googled on "replicate sync database" and got lots of relevant hits.
I used to work in the high tech industry with companies that made lots and lots of money. These companys had the fastest bandwidth, and the most creative people coming up with cool solutions to solve problems. But basicly the point was, everyone made lots of money, so if IT infastructure was a problem, they threw money at the problem, and it was solved...period. Since that time, I have seen general compression of the $$ side of things, the bright people go somewhere else, and the people outsource the smart clever IT folks that worked at the the tech company to some outsourcing firm...
and all the call centers are shipped off to India.
So... I think... where is all the money now, and clever people?
Google.
Just ask Google to host your IT applications, they already index the rest of the damn web anyway.
This would beat Googgle to their next big thing anyway... why not just host the world at Google?
Storing your sensitive financial information will be just a spec of content compared to the rest of the web. Then buy some good fiber connections from Verizon. (I'm spoiled with my FIOS service at home...better than the DSL at my companies remote office)... and viola, problem solved. Besides, then anyone can get to your data from anywhere.... the security issue is a myth... who has time to look up all this financial information anyway... most people are reading Dilbert cartoons about how your company outsourced the network.
Plus, you can tell all your clients to buy Google stock, prior to handing over all the data.
-- R
Ross Youngblood
Sounds like you need another IT partner, at the least.
And good luck having branch offices with no server. Only way i can think of doing that is 100% terminal services.
Oh whats the difference beteen a "branch office in a box" and a branch server? I bet nil.
---- Booth was a patriot ----
Where I have used them the costs of comms links was such that the Steelheads paid for themselves in around 18 months.
Of course your mileage may vary, but remember that cached data is bandwidth saved and that's either money in your pocket, or additional bandwidth for other uses.
In fact don't run a domain at all. Let the end users manage their own PCs / laptops / printers and run a real virtual organization. You'll save heaps of cash using Skype, Salesforce, GoToMeeting and other solutions designed for this. If you want to manage your end points, buy a solid endpoint management solution like Kaseya (Disclaimer: I work for Kaseya) rather than trying to customize something with GPOs.
I've worked with both trying to get a domain structure running over a wide area network with slow/cheap bandwidth links, and not running any kind of domain structure at all and the later is by far the best way to go. Forget trying to lock down local machines, manage user data and so on. It's like holding a leaky bucket.
Yes, you lose control of your data. The only way to avoid that is to centralize completely, go with a Citrix solution and do ridiculous things like prevent users printing or connecting any USB devices to their machines. There are solutions out there that completely lock and encrypt all data on the user endpoints, but you said that your company doesn't want to spend any money, so I'm assuming that they aren't going to fork out for any kind of real solution.
Thank God, I'm not the only one grappling with this problem.
Astronomical real estate prices in Vancouver have made it difficult to justify consolidating our two offices into one location. So management has come up with the great idea of running our two offices as a single LAN. It sounds like a great idea at first, but when you get down to the nitty gritty it becomes decidedly less practical. We deal with big files and need a speedy ODBC database connection, so our IPSec over WAN tunnel just isn't cutting it. Management was surprised to find that my estimates of several thousand dollars a month for leasing a dedicated fiber connection were, in fact, entirely accurate. I've suggested cloning our server equipment, but again, cost is balked at.
The future is not-quite-now, it seems.
OpenAFS for Windows 1.5.30 (1.5.3001.0) using MIT Kerberos for Windows 3.2.2 for authentication.
I attended a Cisco bash last year where they were expounding the virtues of their ACE (Cisco Application Control Engine) technologys.
Basically you use a couple of routers in between your server room and your remote office which know
about layer 4-7 of the protocol stacks. This allows the routers to short-cut a lot of the protocol
handshaking that causes the latency in things like HTTP, SMB, SQL etc.
These are meant to be quite effective for remote sites & greatly improve performance. Cisco claim that these engines have been optimized for a wide range of common office protocols.
Have a talk to your Cisco rep, they'd be more than happy to do a presentation & possibly lend you
some loan gear for testing.
read all of this: near the bottom it mentions other associated & relevant technologies such as "Application Velocity System"
http://www.cisco.com/en/US/prod/collateral/modules/ps2706/ps6906/prod_brochure0900aecd804595e1.html
'Branch office in a box' appliances look ideal, but they don't implement WAFS.
I'm pretty sure that the "Branch office in a box" servers from Packeteer (formerly Tacit) do implement WAFS, or something very similar.
Branch office in a box turnkey servers seem great on paper, but the reality is, you'll still need to manage them just like any other server. They're not quite as "fire and forget" as the manufacturers would like you to think.
In the end, network traffic compression is a better solution.
Look to Riverbed for these type of solutions. They reduce traffic significantly.
I'd go with Citrix (I don't think MS Terminal Services is there just yet) and deploy MS Office to those servers and then distribute all other software via Microsoft Softgrid (soon to be called Microsoft Application Virtualization): http://www.microsoft.com/systemcenter/softgrid/default.mspx
The combination of Citrix + Softgrid is a pretty powerful combination - there's no need to silo your Citrix farm any more, and apps deployed via softgrid don't leave any junk behind on the filesystem or registry (since both are virtualised). Use a Citrix access gateway (basically an SLL VPN device that integrates with Citrix) to publish a windows desktop and then your remote offices just need a decent connection to the internet (budget approx 50 kbit/sec per user with 30% concurrent usage). Users can then work from home or from a notebook with a 3G data card too. Or forget the access gateway and connect the offices to the data centre with dedicated leased lines / MPLS links etc.
In each office install network printers onto each local device and then use the Citrix Universal Printer driver to send compressed print jobs from the data centre to the printer via the citrix client. Or if you have the bandwidth, install the printer on a print server located in the data centre and send jobs directly from the print server, over the WAN, to the printer in the remote office (this is easier to manage).
Lock down the citrix servers and client desktops with AppSense http://www.appsense.com/ and you'll then have a secure, remotely accessible system which is managed centrally.
Unfortunately there are no silver bullets to solve this problem, no "remote office in a box" solutions that will solve 100% of your problems. I can pretty much guarantee that.
I work for a company that is committed to WAFS 100%, using Packeteer's iShare solution. They spent several months building their own homebrew iShare (software) on top of Win2K3 Server so they could have iShare and SMS on the same server. This setup was blessed by Packeteer after thorough testing. It is used in over 80 remote offices worldwide over a wide variety of WAN conditions. Some of these WAN conditions are quite bad.
This environment is carefully integrated into DFS so users connecting from remote offices get referrals to the proper regional file server for their WAFS-accelerated files. Obviously they want to avoid users in India getting files from the U.S. or referring through the U.S. if a file server cluster exists in India.
Presently none of the iShare boxes run in-line with the WAN connection, which basically means they're not taking full advantage of iShare's capabilities like TCP, Exchange, and Web acceleration. In a previous incarnation I used Riverbed's WAN accelerator boxes in-line and found that helped our remote sites quite a bit. I never got around to upgrading them to use the Riverbed's WAFS feature set before we were bought out, however, so can't speak to Riverbed's strength or weaknesses there.
All this said, iShare, while helpful, isn't magical. CAD applications in particular haven't been helped much and forget it if you want WAFS to help with any file that does internal locking (e.g., Access DBs). If you have lots of Access DBs across your organization, WAFS, iShare or otherwise I suspect, very likely will not help you. You need to go to enterprise-friendly databases. Access is a very hard habit to break, however, and if you're anything like my company you may have tens of thousands globally to deal with. CAD applications that may have thousands of small files will often bog down in the WAFS world. And CAD (or other) applications that require client-server version control like through PDMWorks or Teamcenter are not helped at all by WAFS. TCP acceleration could likely be helpful here, however.
The print queues remain on the local iShare server for each site since we rolled our own Win2K3 Server environment for iShare. I am not sure how feasible this would be if we used the actual iShare appliance--probably not, I'd wager.
Pure appliances are probably fine if all you need are WAFS and not much else. Beyond that a single box to do it all is more pie-in-the-sky marketing than reality.
Just have everyone telecommute to the central office. Problem solved!
Tsunami -- You can't bring a good wave down!
www.openafs.org
Its not only a nice idea, it works fine too.
Windows support?
http://www.openafs.org/windows.html
Cisco WAAS units are what you are looking for. They will do network packet optimizations as well as network caching. It keeps a hash database of the largest possible chunks of data, it sends the hashes first - if it gets a hit in the remote devices database it doesn't have to send all the data. Very effective when it works.
They can also serve as local print servers.
"Linux is not our destination, it is simply the open road to tommorow"
Having installed Citrix in close to 200 organizations, from 25 users to 400,000 users, I can say that it is a great solution. With the current version [4.5], you even have the option of streaming an app to a machine, so in the case of a laptop user who wants to get on a plane and be completely disconnected, you can still access your applications at all times. Your access speeds are great, since everything is on high speed. People complain about the cost, but they don't understand the ROI or where the dollars are going and where they are saved. Is the software more expensive than a distributed server only? Yes. But the savings of centralized manangement and increased efficiency are magnitudes higher. For example, a bottom line savings at a 4000 user installation I did showed that they saved a million a year in IT costs from decreased downtime and faster responses from the application in the first year, and 2 million a year after that [the cost of the implementation reduced the bottom line savings the first year].
The same is true at almost every level. The little 25 user network running on Citrix basically had no need for an in-house helpdesk person any longer since the system never appeared to be 'down' at any time and all the apps suddenly had no inconsistencies from user to to user.
Good luck to you and I hope you find an answer quickly.
Speaking of WAFS, brocade had a product suite based on an architecture called FAN's (file area networks). Originally it was several cobbled together disparate bits of software and an "appliance" running windows server 2003 - though i believe the components that make up tapestry now look more like they belong together rather then the way they used to look where it was very obvious the products were all from different vendors and had different design paradigms. Take a look though, http://www.brocade.com/products/tapestry.jsp (brocade arent the only ones that do this, so look around).
And if you look here: http://en.wikipedia.org/wiki/File_Area_Network - this is the generic term for most of the technology involved, file area networks.
Assuming your running windows everywhere (which wouldn't be a leap) then its not a bad solution - the on-site box is literally a "branch office in a box" solution that incorporates wafs/distributed locked/etc and runs a version of windows server, which i believe can be a AD server as well. But the point of it all is that the remote side has no real date unto itself (Everything goes back to head office) but can manage everything at a remote site (including such things as printers) as well as being easy to replace (in fact, its supposed to be constructed in such a way that if the branch office box fails, people shouldn't notice, everything just starts going back to "head office" in a seamless way). Supposedly its operates over very small amounts of bandwidth, but i can imagine the first time someone opens a large file being a painful excersize.
Still, ive not seen the product except in demo's, but i have heard good things about it.
The dude needs to re-read Elements of Style, for verbosity and consistency of tone. That post would have come off better as a parody.
That said unless your remote offices barely use the LAN, you already have a really fast WAN, or really high end equipment plus the in house resources to manage it, none of which appear to be true, all these options are going to be expensive.
Limited server consolidation can be a good thing, and large companies with really fat network pipes can actually centralize even file servers, and sometimes they even save money doing it(at least if they needed the network pipes anyway), but if you were with one of those companies you wouldn't be asking for a solution.
Your only real solution is to fire your outsourcing company, whichever meat head manager on your side thought it was a good idea, and anyone in a network or server role who didn't have the balls to say this was a terrible idea. If you're one of the above start by resigning. Then use the money you were going to spend on them to hire a few competent people and put servers back where they belong.
The first step is to sue the outsourcing partner for damages. Any settlement money from there could go to a better solution.
Step 2 - draft a contract, which spell out black and white financial liabilities and benchmarks for the new solution provider for the new implementation.
The rest is a matter of IT decisions.
Your other option is to turn the problem inside out. Run everything in your data center and use thin clients remotely.
Yes, running things like video, youtube, high definition images, etc will suffer, but this is an office, isn't it?
You should be able to run reasonable business desktop and productivity apps across your wan more efficiently than opening files, ginormous PST files, etc. etc.
This isn't free, and if you are a Microsoft shop, you still have to pay the piper with CALS. VMWare has their VDI, and there is the Sun solution, and the whole LTSP deal. Might not work for everyone, but might be a solution for a large number of them.
While there are a lot of unknowns about your question (e.g. number of locations, current size of WAN links, windows/linux enviroment, types of wan traffic, types of applications etc) really your only choices are:
... and to a certain extent, because its cheaper than a "real" file server:
1. Upgrade WAN links
2. Implement a citrix enviroment
3. Using comodity PC hardware, run file servers with DFS, and backup the DFS at the main office. If you are using Windows this could work really well, with file/folder authentication, and domain authentication can be done over the WAN to the main office. On the other hand, there are risks using comodity hardware...
The compression boxes that others have mentioned, which might work for you, work best depending on what *type* of WAN traffic you have. If its mostly MS Orifice, that will compress reasonably well. If you have other application/proprietory traffic, maybe not so good. For example, I worked for a company that ran their remote offices with citrix clients on 64k to 128k links. After adding more clients than the links cound handle, the company starts to scream, but does not want to spend the money. Our consulting partner tells us about these whizz bang boxes that can compress data into almost nothing. So we ran a trial, and we found that (just as I thought we would) that citrix ran slower than before. Why? Citrix compression was already enabled. I wonder what made the consultants think they could compress already compressed data.
I lost me sig.
Then they can't price gouge you on the local servers, which is the best idea.
Seriously though.
Actually put WAFS servers or in router devices in each office with decent size disks. They are linux devices and can be configured to do local auth as well as file and print.
Power Corrupts,Absolute Power Corrupts Absolutely, leaving one person(group)in charge is absolutely corrupt.
How Would You Make a Distributed Office System? Me? I am old fashioned, plain old traditional oak, well worked, using my leet router skillz.
"I work for a financial company which went through a server consolidation project approximately six years ago, thanks to a wonderful suggestion by our outsourcing partner. Although originally hailed as an excellent cost cutting measure, management has finally realized that martyring the network performance of 1000+ employees in 100 remote field offices wasn't such a great idea afterall. We're now looking at various solutions to help optimize WAN performance. Dedicated servers for each field office is out of the question, due to the price gouging of our outsourcing partner. Wide area file services (WAFS) look like a good solution, but they don't address other problems, such as authenticating over a WAN, print queues, etc. 'Branch office in a box' appliances look ideal, but they don't implement WAFS. So what have your companies done to move the data and network services closer to the users, while keeping costs down to a minimum?"
Basically, you got screwed. But that's not the worst of it. The bad news is that you're still in the midst of being screwed. Others have offered up good advice: drop the outsourcing company for a competent one.
Another good point others have made: your problem isn't well defined, define it and determine what your needs are. What problem are you really trying to solve.
But... back to the broad question you posed. The short answer is that you need branch office servers. Makes no sense to route a 8MB print job to the central office over the WAN, so that it can be routed over the same pipe _again_ to print at a printer that was originally 3' away from the desktop computer that issued the massive print job.
Here's how one might set it up on a low budget basis:
Central Office:
- Dynamic DNS server
- AD or LDAP to manage access control/etc.
- PRINT repo ( for all those print jobs and sox compliance )
- EMAIL repo
- File Server(s) (if you can afford it, get a good SAN storage for your central office with NAS/NFS heads to export storage. EMC/NETAPP/etc comes to mind)
- PRINT SERVER
- VPN server to bridge all offices securely
- VOIP Server/Asterix
Satellite Offices:
- GATEway server that can use Dynamic DNS to let the central office know where it is.
- VPN client
- Local AD/LDAP secondary(caching)
- Local DNS (caching)
- Local Print Server, which archives jobs back to central office for SOX
- QoS switch/routing so email/ssh/telnet/etc can operate while large file xfers are occuring
- VOIP node/Asterix
- LOCAL email repo, based on employees who are stationed at the location. Does periodic sync with central, but allows offline email access.
- LOCAL shared file server that periodically syncs up with central.
Implementation will be the hard part, but basically, that's the floorplan for getting a branch offices setup and not completely reliant upon the central office. Yes, you can use desktop hardware, which is less reliable. You can also use cheap NAS units for storage. It just depends on your performance requirements and what you are willing to pay for.
At the end of the day, you can't get something for nothing. And no matter how good the plan, an incompetent worker is still an incompetent worker. Fire the outsourced company that screwed you over.
Winged Power Photography
Ping someone who works @ Sun. Yeah I know they're yesterday's news, but they solved this problem with their thin-clients years ago! My biggest beef with their solution is that it takes a lot of work to get it to drive Windows apps. So the real question is around what kind of apps you have.
All of the other solutions I've seen for this problem are waaaay too convoluted. K.I.S.S. wherever possible.
Good luck!
Of a good a idea that worked well in one area but is not ready for full adoption. Wide Are Network has too much latency to simply turn local office systems global.
Your company is trying to cheat their development model. Rather than setup a distributed IT application they have simply tried to distribute a small office network worldwide. If you look back to the tried and true OSI model. 7 layers. The 7 layer model doesn't speak of Network File Sharing, it speaks of Hardware and Application. TCP/IP (which we have taken quite for granted) is around/below the application level. If you have an application that runs at the TCP/IP level you are good to go.
I have setup distributed systems for several ISPs in the late 90's. We didn't think about what we were doing or why it worked. It looked like we could long haul anything we wanted. A little lag in sending mail or a few extra milliseconds to authenticate LDAP is no big thang. The Internet is distributed by nature. Sometimes DNS was a little slow but that was acceptable for 56k modems and DSL customers. But we spent 2 years working on a central web based administration/billing/customer support application with 1 SQL base in the center. We didn't distribute the application and have it write to the SQL base directly or move files around.
But you can't distribute the file layer. SANs in a local building have had some of the same problems. Any lag affects all applications and you solve it by throwing a big fat fiber backbone in the local building, but it break downs when you try to long haul over WAN links.
If your company is thinking it can sneak around coming up with a decent workflow model, and then implementing that in an application by simply given MS Office and Exchange (or whatever they have employed) to everybody they are sadly mistaken.
But worry not. You are not alone. Many business execs scratch their heads as to why the simply can't share out MS Project and their Excel Spreadsheets to 25 plus people teams and it will work fine. You still need to do the leg work of figuring out the work flow and reducing that to a transaction based system centrally located. That's it. All we've done in the last 20 years is replaced printouts with emails and spreadsheets, and the night operator (a job I used ta do) with scripts (or procedures) that dynamically update or run every 10-15 minutes. You still need a central system and then distribute parts of it, or have slim down interface that everyone can use remotely. Look at how a bank does it, just good ole dumb terminals.
No magic bullets yet. We need faster broadband and much lower latency before you can share out at the file layer using a network stack meant for transaction based appilications.
Let yourself off the hook. No mortal IT person can turn this tide....
You need local servers to reduce the latency. You need some decent thought on the application, not the OS and Office Suite. Good luck!
"Don't fear death... fear not living..." -me
. There is no silver bullet to consolidate every operation into just a few data centers
. Look for specific options for each of your needs (authentication/authorization, storage, development, support...) there are many providers that offer different solutions, but each works for a specific problem.
. Whenever you think that reducing the number of data centers is the solution, then you have to think that you will spend more money in connectivity (sometimes a lot more money)
Its expensive, but it works over the crappiest links ever. It makes branch offices easy because they put new computers in, connect them to anything faster than 28.8 dialup, and they connect to the server and do their work remotely. Sure its a productivity kill if the link or server goes down, but it works for who I work for.
It also solves the file system issue. One server in the biggest office holds all the files. And you can print to local printers from it.
I use OpenVpn with Samba for all remote connections to the companies I do work for. The company with the most users has about 500 systems in play. at 5 geographically separate locations from Florida to Northern Pennsylvania. I set up 'mini' servers at the major locations (these are white boxes which cost about $3,000 each). The operational data load is around 1TB, and the data storage is around 5TB. Using Samba, rsync, OpenVPN, Unison, and a few custom scripts, all remote white boxes data, and programming are maintained from one central location. The shares for the white boxes are designed for quick access to data with immediate need, and remote sync for data that is not immediate need. At the main office is a set of three $11,000 servers which are running virtual servers for all the white boxes, and a central repository for the 'not so immediate needed' data. These boxes are kept in sync, and can replace and/or take over the load of one of the others should they fail, or need to go down for maintenance. I have on a few occasions had to fix, or repair remote 'mini' servers, but in the instance of failure for one of these, the end user's shares are redirected to main office. There are still three systems out there that I have not had to physically touch for 3 years, still going strong. It sounds like some companies need to hire a real consultant to help with the intrinsics of the company IT operations. My idea of a consultant is an expert in the field who can advise on how to setup, operate, maintain and project future requirements the system or systems while keeping the operations cost to a minimum.
F5's WANJet worked well for us. It is very similar to the product(s) from Riverbed and in fact, they are direct market competitors.
"I have learned, without a doubt, that in IT what one pays is usually quite unrelated to what one gets."
Apparently the same can be said about the sex industry. Maybe you two should compare notes?
Indeed when dealing with branch offices in rural locations you sometimes only get crappy DSL connections - unless you want to spend a bundle on the real stuff. I end up just buying a few of those DSL links from the cheapest providers - trying to get links from different providers as well if possible - and then aggregating those DSL links with some intelligence which allows me to use all those links as one fat pipe. Works pretty well and can get pretty cheap. The only problem left is that you have to consider some latency sensitive applications if applicable. Just to comment on this subject. It seems bandwidth is a nagging problem for a lot of folks and sometimes and I think it can be solved with low-cost products and fairly inexpensive high-tech solutions. Sorry if I digress.
It'd be nice if IT services were free, but obviously your company has found that they are not AND they cost more to keep in house.
So, the goal of the outsource was to reduce cost. This works simply by removing all the "fluff" that the IT department provided.
That said, you are on the pendulum swing that reduces IT expenditures by centralizing everything so the equipment can be cared for by fewer people. Your company wanted this! The surprise here being not that the equipment costs change, but the cost associated to the people change when doing this.
Now you personally want the service and support back that you originally had when the IT department was internal. Go Figure! Fewer people.
So, now its time for you to grow up. Get your head back into the real world where its not only your your company's job to reduce cost but, and here's the stinger, IT IS YOUR JOB TO REDUCE COST.
Get your head into the game. That's all it is. Start managing your outsource partner, not the other way around. Pay for the IT you NEED. Make suggestions, get budget and stop whining about how unfair the situation is. You are forgetting, this is not 7 years ago, its today. If you are the GUY, figure out what this stuff really costs and see if it is really a gouging. If it is, re-bid. Educate yourself.
By the way, WAFS will NOT fix your problem, it seems to be much deeper.
Hp Have a WAN Accelerator, basically it hashes up the data and transfers the information faster, the hash tables grows with use. Sort of like a cache of sorts. And its a WAN device not a server !
Your server consolidation project rolled out six years ago and your bosses aren't ready to spend serious money on a new model? Things must be going pretty well.
This is a topic near and dear to my heart.
1. First off, you dismissed WAFS-style accelerator solutions - I wouldn't. I think that's going to go a long toward your solution.
3. Get more bandwidth bang for your buck by consolidating all your connections through 1 carrier (realistically it probably isn't possible, but you might get close.) Something like Megapath. See if you can find someone to build you an MPLS network so you can guarantee layer 3 throughout. Build QoS policies on that. By going with 1 carrier you'll make your account management simpler and they might give you more bandwidth because they're carrying all your traffic. Be prepared to sign a 3 or 5 year contract for the best deal.
2. You were worried about authentication over the WAN - shouldn't matter because that's fairly low bandwidth. Print queues over the WAN? Egad - you don't want to do that so come up with a novel solution for dropping some kind of network printer on their LAN. Novel might just mean a sleeve on the side of the printer that holds the driver installation CD. Users can be walked through that. At the same time, make sure all your printers are the same so your help desk isn't bogged down in troubleshooting a million different ones.
4. Application selection: this is pretty critical. Don't purchase heavy apps. Purchase lightweight, web-based ones. For example, your accounting system might have a fat client for everyone in the home office and then a web version for everyone else. That'll work fine since they're probably just doing boring things like generating PO's. Microsoft Project has a web client within their MS Project Server, so that'll work for something like that. Just keep this in mind when purchasing any application.
----- obSig
Wait. Are you looking for print queues over the WAN? What happens after the document has printed. Does the head office FedEx it to you? Or is it quicker when they use the fax.
Do you or your partner snore? - Visit www.snoring.com.au
or, in other words, how do I put a P2P network to connect the branch offices. I you don't know how a P2P network works, let me know, it's been wondering me for a while.
Just go ahead and put a server in each branch office.
Are the magic words, but please do prepare your brain for a roller-coaster ride.
OpenAFS
and
Kerberos
We use SunRay thin client for this purpose and also to help with a number of other IT cost issues. It is quick, has good support for windows, is fast, easy to administer and the webtop version is great for people on the move or for helping those with older laptops or who need access to certain apps not installed on their laptop be productive.
All in all money very well spent.
Yeah... I love Open Source Software too.
My beliefs do not require that you agree with them.
For a filesystem, I would recommend openafs. For printing, I recommend setting up CUPS servers. AFS lets you have distributed servers that are centrally maintained. AFS is location agnostic. The filesystem is split into volumes, which can be located on any server and seamlessly moved between servers without needing to change the file path. Check it out at http://www.openafs.org/
It also has manual read-only replication so that you can have a local read-only copy of frequently accessed files.
See subject.
For example: As I said, our SVN repository is hosted by someone else. This implies that we must connect to them over the Internet, which means that, yes, if Internet in the office is down, we can't check in.
We also all have laptops. If necessary, we can just pick up our laptops, head down to the coffee shop, and continue working, with cups of coffee and chai being brought to us, until the Internet at work is fixed.
And we also have 100 mbit fiber shared among maybe five people, so if the Internet is up, no one person's BitTorrent is going to cause problems.
My point is not that being dependent on an Internet connection is always wrong or unworkable, but that if you want to not be dependent on the Internet, you're going to have to spend some money.
Don't thank God, thank a doctor!
We have several heavy use remote locations and were running into the same issues as the poster. We selected the iShaper and iShared devices from Packeteer after trialling them for the following reasons:
1) The iShaper is two devices in one - the regular Packeteer shaper to QoS WAN traffic and a Windows 2003 Storage server plane connected with an internal gigabit switch. The Windows side can be setup as Domain/DHCP/DNS/Print and app server.
2) The device can be placed inline.
3) With an iShared in your datacenter, the iShapers can pull content file share content with them via DFS and their specific protocol they have been working with MS on. They use a "hot/Cold" system for you to prepopulate the device with the user shares and other file shares, and then the protocol tracks the changes to make file shares uber fast over 128k and above connections. In our lab testing, it has been at least a x2 to x10 improvement in file load time.
We have a similer infrastructure and had to come up with a solution for many remote offices. Samba and cheap servers came to the rescue. The nice part about using Linux (Debian Distro) is we can keep adding more software as new challenges arrive, something you can't do with appliance. We wrote a couple of scripts for centralized backup to our data center and Landesk for management of workstations. This was rolled out in 2004 so I did not have as many options but I checked the many choices including Cytrix and other options. Remove the gui and you will be amazed how little power is needed for a linux box doing a ton of tasks including file and print serving.
with freenx of course.
Dedicated servers for each field office is out of the question, due to the price gouging of our outsourcing partner.
Gee that's an easy one, lose the price gouging outsourcing "partner". It sounds like he's the problem.
Seriously, local servers sound like the way to go for you, usually the only way to go, because North American bandwidth is still shite in the 21st century.
-Billco, Fnarg.com
Based on what I read in your post on the top of the page, you are asking the same questions that many of my customer ask me about. The Riverbed Steelhead and the Riverbed Mobile Server can solve all of your problems that you are looking at. It will allow you to reduce your bandwidth on your wan links to save money by accelerating your applications, backup jobs, print servers. Also you will have no problems with authentications over the wan, I have never seen a customers have any sort of that problem. You can also make this branch office in a box, it can be set up in a way that will make it easy to send to office and set up. If you have more questions please email me.
At least for Web-based applications, simply invest in the WebAccelerator from F5 Networks. It literally provides almost LAN-like performance for web applications over the WAN.
we use cisco WAAS boxes now in all our locations, but we still have servers in the locations due to ad authentication issues, i heard rumors that a new software release will run a virtual machine with dc functionality. Printing is supported by the box, handeld by CUPS. Performance is great, allthough it really depends on the application you run, and the amount of data your talking about. As compared, money wise, against a server, a decent server will win against a comparable waas box, so if its not political issues against a server in your location then go for a server in you location. Through the company that we buy our cisco equipment from, it was very easy to get a testsetup to do a pilot in one of our locations to get a feeling on performance gains using our applications. If there is a dc in your location (or when the waas box does the dc functiality itself) you'd still have access to your central fileservers when the wan goes down, any other centrally offered services arent cachable. just my 2 cents ..
"That post would have come off better as a parody."
I thought it was a parody.
I'm sorry, I'll pay more attention in the future.