Slashdot Mirror
USPS Server Meltdown
m2pc writes "The US Postal Service is experiencing major server issues for its shipping API web services. After spending about an hour debugging my own eCommerce software for a client, I found the problem was with the USPS shipping servers being unavailable. Further research showed that message boards for OS Commerce and other e-Commerce packages are filling with posts from angry users who are experiencing crashing Web store applications and frustrated customers. Developers are scrambling to find interim solutions, from hard-coding fixed price shipping, to 'rolling their own' shipping calculation APIs based on the USPS Fixed Rate Zone Tables, to disabling the USPS option altogether. One user reported yesterday that a call to USPS yielded the response 'we expect it to be down all day.' As of 9:20 AM PST the service is still unavailable."
Sounds like the USPS is just angling to get some of that federal bailout money! Oh wait...
And that's what you get for writing e-commerce packages that rely on 3rd party sites for basic functionality...
Don't say I didn't tell you so...
My blog
"Neither snow nor rain nor heat nor gloom of night stays these couriers from the swift completion of their appointed rounds"
But the computers will!!!
Provide a web service, apparently.
Only a really terrible developer would hit a web services API and not code for it to fail. No one should expect a third party service to be up 100%. The apps should fail gracefully. Anyone finding their e-commerce software handling this situation poorly should find another package.
If a store offers only the USPS delivery method and the web service is down, the user could be directed to call the sales number to place their order. If the store offers other deliver methods the store front could instruct the user that USPS isn't currently available and they must choose another method.
Developers: We can use your help.
This is sure to piss off a lot of people being between that Thanksgiving and Christmas period. Lets hope nobody that ordered me something is having problems.
"I don't have to think. I only have to do it. The results are always perfect, but that's old news." - Meat Puppets
...but not through server crashes, apparently. :-)
-- http://ninthagenda.com/
This is a great example of why SAS (Software as a Service) in its current form will eventually fail. The very nature of the Internet is to be disconnected and stateless. If there is no guaranteed delivery at the 5 - 9's level (99.999%), then how can business expect to depend on the service? Mind you, I don't have a better model, but we had better come up with one if we intend to continue using the Internet for commerce!
It would probably snailmail patches to all the vendors connecting to their site ;-)
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Try it. Go to a USofA-based commercial site and check the shipping charges for your purchase.
Most of the sites are offering free shipping. I'm guessing that these two items are related.
Our website has been suffering from time outs and dropped connections to the UPS rate calculator web service as well since yesterday. Seems to be intermittent, refreshing the page seems to help and it will eventually connect. Luckily none of our customers see the problem as our sales tools are all internal. Happy Holidays!
TODO: Insert witty sig
I came in yesterday morning to find the USPS module non-functional. Worse, the only working option was DHL overnight - and in case you've missed the news, DHL is now about an order of magnitude worse than the post office for domestic delivery. Even for places they say they can do next-day delivery to, actual delivery can take more than a week.
Why? Because they hand it off to the post office rather than deliver it themselves. Why it takes the post office a week to deliver it when I can get it there in two days by sending it by priority mail myself is a mystery. In any case, DHL's out of the (US) domestic game entirely next month.
My site was up last time I checked, but if the USPS option goes down again, I think it's time for a 'free economy shipping' promotion. No messy rate tables to deal with!
Um... The post office is private, but protected. Even the best hardware and software fails on occasion. That was my point earlier - we need a better service model if we expect SAS to work. If the small Government crowd had their way the post office would still be the Pony Express. Let's see you get a letter acorss the country in a week for less than 50 cents. UPS and FedX can't - they get $5 for the smallest item. So much for off loading services to business!
An unexpected error occurred. An exception occured in module:
USPS
at address: 0x1234 Main St., Hometown USA, zc=0x10001
posting anonymously to cover my ass... We upgraded to MySQL 5.1 last week. We had some major table corruption and no backups. Sadly, no one will be fired or even reprimanded for any of this major cluster fuck.
This is why my company ships 2 of everything that is ordered. One via USPS and the other by UPS, just in case one gets lost. Always need redundancy.
UPS and FedEx can't, because it's illegal.
USPS has IT people? Oiy veih I can imagine that job.
A few years ago, I used to be one of them...
Oy Vey, indeed...
Goofy, Geeky Gifts and More!
The company I work for provides a hosted e-commerce shopping cart solution, SEO-Cart, which supports the USPS Web Tools. Of course the first call coming in for the day was from a client using USPS and having incorrect shipping prices being calculated for their store.
I went ahead and called USPS and the lady who answered was quite rude and explained to me that they had a Worldwide outage which affected other applications than just their Webtools API, and also that they hire a 3rd party company to handle their Webtools API software. She couldn't provide any other information at all and I told her a company of that size should have some sort of fail over plan in place to prevent them from being down as long as they have been. I was really disappointed in the fact she didn't even ask me for my name, phone number, or company by time the conversation was over, but she was probably being bombarded with phone calls all day.
After figuring out that USPS was completely down, I looked through our fail over code and found the following equations seem to come close to the USPS pricing:
National shipping: [cart-weight]*1.6+3.00
International shipping: [cart-weight]*1.6+15.00
These also include pricing for insurance.
After tweaking the fail over pricing code to this, it seemed that everyone using USPS were happy with the results. We also had to decrease the connection timeout set for the request to the USPS Webtools API which was also slowing things down.
The Webtools API seems to be both up and down today, with some orders having shipping prices directly from USPS and others having the fallback pricing. Either way, hopefully their IT department learns from this and also provide us information as to what exactly went wrong.
On that note, this is a prime example that I use when speaking to prospects about the advantages of using a hosted shopping cart solution rather than a licensed/free download solution. Besides the obvious IT benefits that you get with a good hosted shopping cart solution, hosted shopping cart software is typically a centralized application that can provide quick updates to problems like these. Of course this is assuming that the prospect is serious about their online store and doesn't want to handle technical support themselves.
To be fair, what sort of "backup" calculation would you have done here, short of reverse-engineering the USPS algorithm for calculating shipping rates?
I'm not usually a rabid free-market libertarian, but this here can be seen as a result of the fact that the USPS isn't really beholden to its customers. Can you imagine FedEx or UPS being afflicted by such an issue? And, if they were, would they blow off inquiries with a glib, "We expect the servers to be down for the rest of the day?" Of course not, because, for FedEx, UPS, DHL, et. al. such an outage directly affects the health of the organization. If people can't calculate shipping rates, they can't ship, and if they don't ship, the company doesn't make money. The close linkage between revenue and working services tends to put more impetus behind keeping things working and making sure that they get fixed quickly if they do happen to go down.
We all know what to do, but we don't know how to get re-elected once we have done it
The government can't even manage to keep a simple web service online, and people still believe that it would be wise to let them control health care.
Get real. On several occasions, I've had to manually intervene to fix idiotic billing f*ckups between my PRIVATE insurer and a PRIVATE hospital, who had entered into mutual contracts to be in the same "network". For some reason, they couldn't get their own computers to talk to each other and I had to fix their bugs by going deciphering cryptic paper printouts myself and wasting hours calling customer service. This kind of stupid private healthcare IT problem happens routinely to millions of people every year. Therefore, using your reasoning I conclude that due to a clear history of incompetence, it is unwise to let private parties handle health care, and such practice should be banned.
Because it's no longer reliable enough. The S&H charges change based on too many values for the few factors (carrier, size, weight, destination, type of delivery, timing of delivery, etc.), and things change. So it's not a necessarily easy calculation to perform.
Truth is like the sun. You can shut it out for a time, but it ain't goin' away. - Elvis Presley (source: imdb.com)
"Neither snow nor rain nor heat nor gloom of night stays these couriers from the swift completion of their appointed rounds"
But the computers will!!!
You do realize that this has never been the motto or creed of the USPS.
It is taken from the courier service of the Persian Empire. See Wikipedia.
--
This is not the sig you are looking for... Move along...
Apart from the trouble reported in this /. article which I found occuring on one of the existing sites I wrote yesterday (simply because there were no USPS prices being returned, no error, but took about 30 seconds to work out what happened), USPS simply sucks ass.
Here's why:
Some time ago, they had an API to get rates, it was called RateV2.
Then they "updated", and now have RateV3.
RateV3 is the only specification published.
To get access to the Rate API servers, you must first test your implementation against thier testing servers successfully, when they see that they let you on the production server....
Thier testing servers only work with a limited version of RateV2.
So, in order to use the USPS API, you must:
Write to the now unpublished RateV2.
Test that RateV2 on the test servers.
Ask USPS to allow you to use Production (and get the keys etc) because you have successfully tested.
Write completely new code against RateV3.
Test that RateV3 on the production servers.
And if you try and show the USPS staff the logical problem in this process, they will reply "I can not put you on production servers, until you have done three successful tests on the test servers".
NZ Electronics Enthusiasts: Check out my Trade Me Listings
It doesn't take a brain surgeon to look at a postal rate schedule - the postal service does provide them for all shipping services with clearly marked zones to figure out how much to pay for postage.
Did you get the security guy who came in and told you not to surf porn on company time, then ramble about how the USPS computer network was more important than the DOD? I got that, and they tried to do good cop/bad cop on us - it was comical.
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
Does your company offer mail order brides?
Can you imagine FedEx or UPS being afflicted by such an issue?
Having worked in IT for one of them this year... yes, without question, yes.
The same has been true with the USPS since 1970. Their entire budget is financed by people buying stamps and other services. They don't get a dime of taxpayer money.
If you reply, do so only to what I explicitly wrote. If I didn't write it, don't assume or infer it.
The zones are clearly marked but determining which zone you are shipping to is not. They are based on distance from the originating ZIP Code so the application must use Postal Explorer to generate the zone chart ahead of time. (For now, it happens to be working.) And in the case of non-flat-rate Priority Mail, there's a huge price difference between zone 4 (zone and weight only) and zone 5 (zone and weight *OR* zone and size).
You do realize that most of the routing of letters is done by computers, using OCR, right?
Nerd rage is the funniest rage.
"Americans are pathetic sometimes -- they expect their government services to do as well as private industry, yet they don't give them the ability to charge what private industry charges. Amtrak is a similar situation, Amtrak is expected to be cash flow positive, yet they are not allowed to own their own tracks, those are owned by the freight companies, whereas their main competitors run on highways that are paid for completely by the taxpayer and gas taxes, or operate out of airports also funded by taxpayers."
Welcome to municipal broadband. Oh wait!
Shai Schticks:"You don't make peace with friends, you make peace with enemies"
The same has been true with the USPS since 1970. Their entire budget is financed by people buying stamps and other services. They don't get a dime of taxpayer money.
How did you ever get modded +5 informative with that load of bull?
The total funding for the Postal Service in the administration's 2005 budget is just over $61.7 million.
Appropriation, fiscal year 2004 $65,135,000
Appropriations, 1999 $100,195,000
etc, etc
When information is power, privacy is freedom.
The shipping api is down, however the rate calculator is available. We provide the rate calculation engine to the folks that provide the shipping api. The shipping api offers numerous services and we are just one of them.
If you require domestic rates calculations, as received by ShippingAPI.dll, you can access our rate engine directly.
Accessing the rate calculator api is very similiar to how you access this functionality from the shipping api.
Getting started:
A simple demo site that will allow you to learn/test a subset the API:
http://postcalc.usps.gov/domSDK/SDKXMLtest.asp
XML over HTTP Application access:
http://postcalc.usps.gov/SdkXml.aspx
Documentation is available but I do not have a publically available site to post this on. If a site is available, just let me know and I will post the SDK documentation there.
Let me know if I can help.
Adrian Griffith
ManTech Information Systems & Technology
Project Manager, USPS Postal Explorer and Rate Calculators
adrian.griffith@mantech.com