Xerox "Routine Backup Test" Leave 17 States Without Food Stamps
An anonymous reader writes "People in Ohio, Michigan and 15 other states found themselves temporarily unable to use their food stamp debit-style cards on Saturday, after a routine test of backup systems by vendor Xerox Corp. resulted in a system failure. Xerox announced late in the evening that access has been restored for users in the 17 states affected by the outage, hours after the first problems were reported. 'Restarting the EBT system required time to ensure service was back at full functionality,' spokeswoman Jennifer Wasmer said in an email. An emergency voucher process was available in some of the areas while the problems were occurring, she said. U.S. Department of Agriculture spokeswoman Courtney Rowe underscored that the outage was not related to the government shutdown."
N/t
"[A politician] underscored that the outage was not related to the government shutdown."
These words should never have to be said.
Backups don't always work - that's why you test them. This time they did not work - much better that you experience problems when you anticipate them than when everything else is going wrong, too. It's unfortunate that the system was down, but it seems they got it back up in a reasonably quick time frame. Moreover, merchants are supposed to have manual means of recording EBT payments for just such a scenario.
Douglas Whitaker
Hahaha I was walking thru Kroger's yesterday and they kept announcing over the speakers "We cannot accept EBT today because our computers are having problems."
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Xerox makes extensive use of GNU/Linux.
People in Ohio, Michigan and 15 other states found themselves temporarily unable to use their food stamp debit-style cards on Saturday,
Why is it that a convenience -- our credit cards, are able to weather a failure like this by simply allowing all purchases, but our food stamp cards simply stop working? Credit card systems are, at every level, designed to cope with a failure by simply authorizing the purchase. Only a very small number of transactions would have been failed anyway for insufficient funds, etc., and these are reconciled when that part of the system is restored to service... meaning there's very little loss to the provider for this.
For that matter, if they've decided to design the system in this fashion, where were the redundancies? If a routine backup can result in failure on this scale, then it begs the question of where and how the backup of the actual systems, not just the data, got overlooked.
#fuckbeta #iamslashdot #dicemustdie
The local Walmart was lacking in any backup method. They had at least 50 buggies packed full of food sitting around the registers and a lot of pissed off customers. Glad they got it back up, I don't look forward to that riot.
Some idiot said, "Hey, let's test the backup system on a production server."
At least Xerox does backups, and tests them. Sure beats not testing and risking catastrophe.
This situation could have resulted in violence (or worse) if this wasn't rectified quickly.
Moreover, merchants are supposed to have manual means of recording EBT payments for just such a scenario.
Those lead to fraud loopholes, and not just EBT. Someone can claim, "oh, my card doesn't work because the system is down, just fill out the paperwork for me, please." Thats more problematic when nothing distinguishes a "DECLINE-card has no funds" from a "DECLINE-system is broken" to the cashier.
John
Backups don't always work - that's why you test them. This time they did not work - much better that you experience problems when you anticipate them than when everything else is going wrong, too. It's unfortunate that the system was down, but it seems they got it back up in a reasonably quick time frame. Moreover, merchants are supposed to have manual means of recording EBT payments for just such a scenario.
Exactly. Imagine a more catastrophic meltdown down the road and all of the Nancy Naysayers saying, "WhyoWhy didn't anyone test it?"
Let me remind you all of Senator Obama's words from 2006 regarding the raising of the debt ceiling. He voted against raising the debt ceiling at that time.
"The fact that we are here today to debate raising America's debt limit is a sign of leadership failure. It is a sign that the US Government can not pay its own bills. It is a sign that we now depend on ongoing financial assistance from foreign countries to finance our Government's reckless fiscal policies. Increasing America's debt weakens us domestically and internationally."
Source
How true are those words? I only wish President Obama still believed what he did as Senator.
Routine means that is done more often, not that is is done correct or even approved....
I'll just leave this here... https://www.youtube.com/watch?v=NzspsovNvII
"[A politician] underscored that the entry of the US into WW2 was not related to the attack on Pearl Harbor."
That statement is no longer operative.
Food stamps? Why does your country need food stamps?
Is this a war-ravaged african country we're talking about?
Some neoliberal-ravaged western country perhaps?
... much better that you experience problems when you anticipate them than when everything else is going wrong, too.
So what you're saying is: if you make a 1 million dollar mistake, your response should be "Phew! At least I didn't lose 5 million!".
An outside observer might suggest that losing the $1 million is bad on its face. Mitigating the outrage by making false comparisons is the sort of thing politicians do, as a dodge for responsibility.
Should we be sanguine about these sorts of problems because they're not the worst possible scenario? Is that an acceptable excuse?
In the late 1990s, the company I worked for was one of many processing EBT card transactions for grocery stores in New Mexico when they first switched to it from paper food stamps. The bank that was the approving authority for them (the next higher link up the chain from us) had a system problem and had been down for about 45 minutes.
I got a call from a very stressed sounding manager at a store in a bad neighborhood of Albuquerque and explained that the outage was statewide, and I'd already called the next highest level.
His response: "You don't understand! These people carry guns."
I really didn't have a good answer for that one, but certainly sympathized.
They later changed the rules so that when the statewide system was down, they could approve it at the store and then take out any overuse from later payments. That got abused, but it made some store managers a lot less nervous.
It's hardly fair to expect people to get a job just to eat. Everyone is entitled to food, shelter and reasonable transportation. It say's so in the US Constitution.
It does? Where? Since when?? The closest my US Constitution comes is "Life, Liberty, and the Pursuit of Happiness."
Do you know what happens when people get desperate? They stick knives and guns in your face.
Yeah, sure they may get caught, but in the meantime, you're dead. Got your own gun? Doesn't work when they stab or shoot you in the back.
Get it? These social pograms are to prevent folks from doing some very desperate things.
But that's just practicality. Let's talk about just being a human being.
We can act like a human being or we can act like animals where it's the survival of the "fittest"; which means in this culture, making money.
But when you are someone who grew up not learning the skills - material or social - to get a job, it is very difficult if not impossible. And I can tell you from experience, nobody clues you in about any defects one may have - you hear nothing or "you are not a good fit". No one ever points out what one's difficiences are. Many of these folks grew up in broken homes, were abused, live in the shittiest areas and have no ability to move.
And the job market being so tight as it is, it is completely unrealistic to "just go out and get a job". We have college graduates - folks with degrees in nursing, engineering and CS let alone "worthless" degrees having a horrible time.
You people take your lives and your opportunities for granted. You grew up where you had the opportunities or at least the knowledge how to get those opotunites. I grew up in a working class family that didn't go to college. I was the first to go to college because I was fortunate enough that my parents were able to afford to live in a middle class town and I could go to a school with college prep classes. I was able to have friends that had college educated parents and knew the "system" - like there are folks to help you get into college and better your life. Many of these poor folks don't even realize sugh things exist - really.
And when you live a life of no hope, you may even give up. he thought of "why should I even try when the rules are against me."
Yeah, yeah, yeah, cue the folks who grew up in a log cabin and pulled themselves up by their bootstraps. That's great that you knew how to do that. That - knowing how to pull yourself up by your bootstraps - is a skill in itself.
I myself am getting helpless too. I see the billionaires just rigging the game more and more. I don't beleive the American myth that one can work hard, take a chance, and make it big. In reality it is know someone in the right spot. I tried a couple of businesses and they failed. I'm tapped out at 50 and i'm scared about my future - it looks like SS and dog food in my retirement. Well dog food - SS is going to be confiscated by the student loan people. I stupidly went back to school for retraining. Yeah, no one hires 40+ year olds for entry level positions.
I mean really, when Mitt Romney was running, he said all he did was "examine reports" and made no decisions; like closing down factories or anything that cost people jobs. he made over $200 million by just examing reports - a $50K a year "analyst job". I could do that! But I don't have a well connected daddy to get me a cushy job that let's me make millions doing what a low level peon does.
Just count your blessings and stop judging other's lives and their characters because you are where you had opportunites that many others don't have.
Risk exists, and appropriate management of it is how the world moves forward. I'm not privy to the inner-workings of Xerox, and it is entirely possible that they were not following best practices and that a substantial portion of this is due to operator error. However, even routine tests have risk associated with them. My point is that immediately blaming Xerox is not a good reaction: making and testing backups is an effort to mitigate risks (with much worse outcomes). If minor problems arise from time to time in the course of protecting against a larger future risk, that should be accepted. How the short-term and long-term risks offset each other is a discussion for the 17 states, the USDA, and Xerox.
Douglas Whitaker
Comment removed based on user account deletion
With various reports referring to it as a power outage and others as a test of backup systems, I'd guess this was a generator load test where something went wrong with the transfer switch. We do those off-hours monthly at the data center where I work and, being the nervous sort, I'm grateful they usually coincide with one of my days off, although ours have gone smoothly.
Did you just pull random speculation out of your ass? If the system works, the card gets used. Most people on the registers are not going to know about any backup system. Want to use your card? swipe it first. Doesn't work? swipe it. I need to know what the problem is so I can ask my manager.
Manager comes over. What happens when you swipe it?
Now, if you're talking about friend of the cashier, that would raise lots of red flags to have piles of swipes work, followed by a single transaction by the friend. It would work once.
What happens after the data is reconciled for manual backups? If you exceed your limit or the card is not authorized for you, some accounting will find you.
Xerox asked retailers to revert to a manual system, meaning customers could spend up to $50 until the system was restored.
That sounds like a reasonable compromise, once everyone is aware the system actually is down.
That addresses this for most cases, so you don't get fraud.
Sounds like they didn't just let people buy whatever they wanted, because the plan b was down. Now, why did you waste our time posting horseshit?
The local Walmart was lacking in any backup method. They had at least 50 buggies packed full of food sitting around the registers and a lot of pissed off customers. Glad they got it back up, I don't look forward to that riot.
Ye gods, the crowd could get ugly...
Too late.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
I actually feel for the Xerox people working on this. I know what it is like to have a system with intense and broad functionality and you are frozen in fear every time you make the slightest change. Testing ahead of time is great but can you be 100% certain that you have 100% test coverage? If you are then you are a fool.
If I had to point any fingers it would be that they should have a multi-layered deployment system where they deploy to the test center, a small random group, a larger group, and then nationally. It sounds like they might be halfway there with this outage only affecting 17 states. Ideally it would only be affecting a few counties in a couple of states.
Plus you want to keep the domain of your test area down to the point where you can fix crap manually.
Errare humanum est
Seven puppies were harmed during the making of this post.
Should we be sanguine about these sorts of problems because they're not the worst possible scenario? Is that an acceptable excuse?
To some degree yes, mistakes happen, especially with large complex systems. We should count having avoided the worst cases scenario as a success and see what can be done to mitigate the failure mode that did occur in the future, and the answer to that question might very well be nothing or nothing less costly than the future number of anticipated similar failures.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
Government spending on the construction and maintenance of roads and other infrastructure creates positive externalities for businesses located near the infrastructure.
Where is the teaching people how to do needed work?
On the Internet, if Nicholas Negroponte's One Laptop Per Child project is to be believed.
Where is the work to be done as people have been taught?
In your own business that you started. Do you think Jesus's adoptive father Joseph was the equivalent of a W-2 employee, or was it more likely that he owned his own shop?
A point to consider: How many hours was perishable food left in all these abandoned carts before being put back on the shelf? I'd steer clear of buying anything perishable from a store accepting EBTs for a while, at least.
Xicor was referring to the Fourteenth Amendment to the Constitution, which does say "property".
Well, I have to say that this might be good.
I was in Target last week, at the checkout, behind a woman who paid approx $330 with a benefit card. At least 90%, if not more, was processed or manufactured food, apparently 'acceptable' to the system. It beggars belief that so much money was spent on sugar
The article is vague as to what Xerox provides. I gather they're not just the hardware vendor. Do they supply the software? Do they handle clearance, like a credit card company? What exactly is their role?
When i test failover, I announce the test in advance so people can have a plan ready in case the failover doesn't. It sounds like this was un-announced and nobody had any sort of plan to deal with a failure.
you increase the top earner's rates on income over a certain amount. In the 50s and 60s we had the highest growth in real wages and middle class incomes the country (maybe even the species) has ever seen with a 90% top tax bracket. How? Because that 90% wasn't a flat "Give us 90% of your income" it was "90 % over 1 Million" or about $9 million in todays money. So if you made over $9 million dollars in a SINGLE YEAR then you paid 90% of that to the gov't. This kept wealth inequality in check and forced top earners to really work for that money over $9 million. If you wanted to be filthy, stinking rich you really had to work at it (people still did). Meanwhile gov't programs redistributed the wealth. Maybe not evenly, but it's better than phoney job creators hording it and holding up human progress by sitting on their fat rears with all the money in the world...
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
I've known people on gov't assistance. It's a few hundred dollars a month and you have to be making about half the poverty line to get it. If you're sister is on gov't assistance for real then there's something wrong with her. I don't mean that as an insult. I mean there really is something wrong, and she needs the help. You don't get enough from the gov't to live, you get enough so that if your family is giving you a lot of help you can just barely eat.
Is this an astro turfer or something? I'd like to believe noone is this much of a jerk in real life...
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
... and it's also why, when you do test them, you make sure to test them on the data affecting 17 different states!
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
I bet those numbers in the stamps were juggling around like hungry children running after US Aid truck delivering American wheat after teachings about contraceptives had been dropped from the curriculum.
...this has to do with government is the fact that, once again, a PRIVATE COMPANY that the government has to use because of the fascist conservatives in this country have forced privatization every chance they get... has screwed some people over. That, and the obligatory rants by said conservatives about how bad the government is (even though, once again, this was the fault of A PRIVATE COMPANY).
Fascism: An authoritarian and nationalistic right-wing system of government and social organization. See also: NAZI's
Because of this news story I learned a new term....Free Shit Army (FSA).
the health care system made so people where better off not work then working even an part time job with no health care or an plan that did not cover anything while they end makeing to much pay so they got kicked off there government assistance plan.
Despite what your third grade communist fearin' teacher might have told you. After $9 million a _year_ you can't honestly say that the person in question is generating enough raw wealth to be 'worth' it. They're just able to obtain it through a combination manipulation of the political system, military power and indoctrination of the working class. The Koch brothers are a prime example. Their money comes from gov't contracts to run hospitals and gov't granted mineral rights. They add no value to either process. They're just good at manipulating the gov't and the populace to increase their wealth.
The super rich didn't 'earn' their wealth. They the gov't to obtain and maintain it. You can argue they shouldn't be allowed to do this, but you can't stop them from doing it. Their wealth makes them too powerful. To ignore that fact not only plays right into their hands, but it's pointless too. I've said it before, I'll say it again: Gov't is a tool. The rich are going to use it to their advantage. The only question is, are you?
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
"'Restarting the EBT system required time to ensure service was back at full functionality,'"
See also: "One does not simply RESTART a mainframe.
"Have you heard a Whoooosssshhhhh! lately?"
Yes, and it sure applied to you missing the point big time because you "assumed".
Don't waste your breath. The tea party fools cheering for the ultra rich would have been the ones cheering for the British aristocrats back then. It's quite ironic that they label themselves as the rebels of the time, but look at what they say and it's all about taking the rights away from the people and giving them back to the aristocrats.
Yes, the top rate was 90%. BUT NOBODY PAID 90%!!! There were all sorts of write offs, loop holes, etc and people paid close to what we paid today.
"A plan fiendishly clever in its intricacies"- Homer Simpson
Most of the people I've seen on food stamps could quit eating for 2 months and still be fat.
80% of the taxes collected for helping people goes to the social(ist) workers and anything related to the administration of those programs. Most of the rest of the money goes to private organisations in the community operated by "connected" families to administor. They use it for themselves to help the economicically disenfranchised to quit their sinful ways and direct them to a church for guidance and care; something anyone could do for themselves for free.
In other words, the money is really used to finance a staff of directory personnel, their friends, and their relatives; over paid parasites living off the taxpayer who think they can do a better job of helping people with my money than I could.
Maybe they can blame their buggy copiers. Didn't this used to be a quality company?
Liberty in your lifetime
tried to drag us into conflicts in Libya, Egypt, and a completely bone-headed Syrian intervention that would have had us in a face-off with Putin over something Putin is every bit as committed to (Syria and his naval base there) as Obama is to "Obamacare"...... and all while DOUBLING the nation's debt.
Oh, and by the way, there never was a Clinton-era budget surplus; Politicians in both parties (Democrats AND "establishment" Republicans) love to pretend they have been fiscally responsible by simply leaving things "off the books". The so-Called Clinton surplus numbers were only projections and only valid if you pretended some of the biggest items on the books (Social Security and Medicare) had no future obligation (and therefore no need to actually save/invest the money coming into the programs, freeing that money up for current spending). Corporate executives who do their accounting this way (not including wall st bankers with Washington lobbyists) go to jail.
Bush inherited a mess too... a recession, the popping of the first internet bubble ("pets.com" anyone???) and years of no American response to terrorism accompanied by legal blocks preventing intel agencies sharing data with eachother.
EVERY president inherits things and some inherit very bad messes; Reagan inherited a crippled military, double-digit inflation, double-digit interest rates, gas lines (people lined-up around the block to buy gas and only allowed to buy every-other day depending on your license plate#) double-digit unemployment...AND a Democrat congress that used the debt ceiling and the budgets to TRY to block his every action. Obama is LYING when he says the current situation has never been faced by a previous president - I remember it quite well. By the time Reagan ran for re-election he had every one of the economic indicators turned around and things were so good his campaign theme was "morning in America". Obama stepped into a much better situation than Reagan inherited by EVERY measure but the "mainstream"/Democrat press runs interference for him and he has become dependent upon their support; he's had FIVE YEARS and we actually have fewer full-time workers now than we did when he was sworn in. Oh, and Obama was no innocent bystander to the 2008 meltdown, he was a senator in the Democrat-run senate at the time which (working in concert with the Democrat-run House) wrote the laws and budgets in 2007 and 2008 after they pronounced Bush's proposals "dead on arrival"
It does not matter what you inherit..... what matters is what you make of it.
Bush's last deficit was $458.6 Billion. President Obama immediately drove this up to $1.4 Trillion (his 2009 "stimulus" ended up costing almost $1T in addition to the "regular" deficit) and then after 2009 the democrats stopped passing budgets (preferring to run the govt on "continuing resolutions" that locked-in the new high levels as the baseline while avoiding any embarrassing votes on budgets with massive deficits). Now with "sequestration" in effect the deficits are falling to about half of Obama's worst (but still FAR higher than Bush's worst) and Obama has the audacity to brag that he has halved the deficit! Blatant dishonest propaganda which he would be publicly challenged on if we had any journalists not connected to the Obama administration.
The situation is NOT improving as long as we are still deficit spending (we continue to go deeper into debt, just at a reduced rate) with each passing year we are heaping more and more debt onto our kids and grandkids and this will become a 1930's-style nightmare if interest rates go up to any degree (which they are bound to do given that they are currently at artificially-depressed rates AND the fed is printing money like a counterfeiter) Even with insanely low rates we are spending much more money every year in interest payments on the debt than we spend on NASA - just imagine all the cool things we could do with existing tax revenue if we were not already so deeply in debt and having to pay the lenders!
Xerox used to make printers, scanners copiers, etc (you know, all that old-fashioned real physical product stuff?) but now as a "service provider" they can mooch off the taxpayers by getting in on the "poor starving children" bandwagon (KA-ching! with each purchase on an EBT card) All of the incentives are there to grow the welfare state: more EBT card users = move voters supporting it, more crony capitalist corporate campaign contributions and lobbying to keep the ball rolling, more political power for the politicians running the gravy train, etc. The only person who loses here is the taxpayer... and he's outnumbered. Washington: meet ancient Rome... It boggles the mind that any young American who plans to still be here in 20 years is not absolutely frightened by the numbers; the US now owes approx $17 Trillion and has promised to pay-out another $100 Trillion in promised future benefits ..... more money than exists on Earth and SOMEBODY will have to pay it.
That which cannot be sustained, will not be sustained
cashier Eliza Shook said dozens of customers at Corner Grocery had to put back groceries when the cards failed Saturday because they couldn't afford to pay for the food.
Thats the store employees job. Customers don't have to put stuff back if they can't pay for it. Just leave it for the employees to put back on the shelf.
You say you think 90% top bracket is unacceptable to you. So what percentage would seem fair to you? 50%? 60%? How do you propose state and federal income is generated if you lower that percentage? It's okay to say you think something is unfair, but you have to come up with an alternative that does seem fair to you if you want to solve things.
I was promised a flying car. Where is my flying car?
The situation is NOT improving as long as we are still deficit spending
Say you wanted to reduce deficit spending to $0.00 by December 31. Most of the difference would have to come from entitlement cuts. How do you plan to fix Social Security and Medicare?
past $9 million a year then no. You're not 'working hard to earn more' any more then. You're riding on someone else's hard (that sounds dirty). Do you seriously think the Chinese billionaire that owns Foxconn 'earned' that? Or to go more extreme how about the southern cotton plantation owners in the 1800s? Yeah, I'm being inflammatory, but the point is still valid.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Backups don't always work - that's why you test them. This time they did not work - much better that you experience problems when you anticipate them than when everything else is going wrong, too.
That's also why you don't do a test restore into your production environment. Testing backups 101.
Check this out 2 Walmarts got pretty cleaned out when they let people use their ebts as if there was no limit on them. Watch the video it's amazing.
http://www.ksla.com/story/23679489/walmart-shelves-in-springhill-mansfield-cleared-in-ebt-glitch
Paul: Father... father, the sleeper has awakened! - Dune
Several years ago the majority of comments would have been about Xerox's inability to test software without causing mass hunger. Now every single comments devolve a rediculous policy debate written by a bunch of wannabe wonks who are obviously watching too much television.
the last one should recognize that he had a _lot_ of help along the way (which he did) and be willing to pay it forward.
In the real world kids from the projects don't make it big. In the real world they're crushed by daily life and their lack of education. Look up the unemployment rate and average income of project kids (especially the ones that speak Ebonics, which sadly makes them more or less unemployable outside of manual labor and fast food). It's not a fun read.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/