USPS Server Meltdown

m2pc writes "The US Postal Service is experiencing major server issues for its shipping API web services. After spending about an hour debugging my own eCommerce software for a client, I found the problem was with the USPS shipping servers being unavailable. Further research showed that message boards for OS Commerce and other e-Commerce packages are filling with posts from angry users who are experiencing crashing Web store applications and frustrated customers. Developers are scrambling to find interim solutions, from hard-coding fixed price shipping, to 'rolling their own' shipping calculation APIs based on the USPS Fixed Rate Zone Tables, to disabling the USPS option altogether. One user reported yesterday that a call to USPS yielded the response 'we expect it to be down all day.' As of 9:20 AM PST the service is still unavailable."

Comments

Bailout?

[erik76]

Sounds like the USPS is just angling to get some of that federal bailout money! Oh wait...

That's what you get....

[morgan_greywolf]

And that's what you get for writing e-commerce packages that rely on 3rd party sites for basic functionality...

Don't say I didn't tell you so...

Re:That's what you get....

[qoncept]

While in principle I agree with that, what are they supposed to do? They are quoting you a price for a service they don't provide themselves.

Re:That's what you get....

[Anonymous Coward]

And that's what you get for writing e-commerce packages that rely on 3rd party sites for basic functionality...

Like a credit card processor? How many web stores have more than one?

Re:That's what you get....

[smallshot]

If it's important enough they should have a backup. I know it's not always within budget, but their website doesn't have to crash because the USPS service is unavailable, it could simply fall back to an acceptable solution, such as a fixed price, an estimation table, or an error message that says the USPS shipping estimator is down and shipping will be calculated at the time of shipment, or they can try again later. Yeah, they'll lose some business, but they can lose less business with a decent backup.

Re:That's what you get....

[lysergic.acid]

that's why i design all my e-commerce sites to accept cash only. (always makes sure the bills are facing up before you feed them into the floppy drive!)

Re:That's what you get....

[JoelKatz]

"While in principle I agree with that, what are they supposed to do? They are quoting you a price for a service they don't provide themselves."

Fail gracefully. For example, state that they are unable to price that shipping option at that time. Offer you to accept the option without knowing the price, select another type of shipping, or get an email when they know.

Designing software to gracefully handle external failures is a vital real-world programming skill.

Proof that computers corrupt all!!!!

[banbeans]

"Neither snow nor rain nor heat nor gloom of night stays these couriers from the swift completion of their appointed rounds"

But the computers will!!!

What can Brown do for you?

[HeavyD14]

Provide a web service, apparently.

Re:What can Brown do for you?

[HeavyD14]

No kidding. UPS *is* up and running, while USPS *is not*. Therfore, Brown can provide a web service.

Crashing Web store applications?

[truthsearch]

Only a really terrible developer would hit a web services API and not code for it to fail. No one should expect a third party service to be up 100%. The apps should fail gracefully. Anyone finding their e-commerce software handling this situation poorly should find another package.

If a store offers only the USPS delivery method and the web service is down, the user could be directed to call the sales number to place their order. If the store offers other deliver methods the store front could instruct the user that USPS isn't currently available and they must choose another method.

Re:Crashing Web store applications?

[Frosty Piss]

osCommerce is well know to be a bowl of spaghetti code and in general a P.O.S. Zen Cart forked a long time ago and did a complete code re-write. osCommerce continues to keep their heads up their asses, and only a fool doesn't look for alternatives at the earliest opertunity.

Re:Crashing Web store applications?

[seanadams.com]

Only a really terrible developer would hit a web services API and not code for it to fail. No one should expect a third party service to be up 100%. The apps should fail gracefully. Anyone finding their e-commerce software handling this situation poorly should find another package.

I have developed a fulfillment system using UPS so I have some experience with this. This was a few years ago, but at the time, UPS's system was designed on the assumption that a shopping cart system would contact UPS during the order placement process (or checkout). USPS appears to be very similar. This is incredibly moronic, because it means that if their server doesn't respond or any reason, you simply lose the sale. Problems AFTER the order has been accepted are far less serious - that just means fulfillment is interrupted - you don't lose orders. It would be even better if you could print the labels without having to reach their server, but they don't provide this (UPS's own software, however, can.)

The only reasonable way to do it (obviously?) is to have your own shopping cart software calculate the shipping rate. At the time I did this in 2001, UPS did not offer reasonable data to let you do this. Their rating system is extremely complicated, and the spreadsheets they gave me had clearly been maintained by hand and needed a lot of massage to get them into a reasonably parseable format. Not only that but it was very difficult to get these tables in the first place, requiring multiple escalations through tech reps and ultimately a "god damn it just give him the tables" from our sales rep.

I imagine things might have improved a little since then and I know there are shopping carts that can now do the rate calculations internally. But if any shipping company forces its customers to contact a server to calculate shipping costs, then they are just as retarded as the merchants who use them (for a web checkout).

Re:Crashing Web store applications?

[Rorschach1]

osC is insidious. You install it because it's open source and a zillion other sites are using it, so it can't be THAT bad, right? Then you have to install a few contributions to add the basic functions you need - and with each contribution, adding the next gets harder and harder.

But you get up and running. Then you discover all of the little annoying usability quirks - US state names don't use proper abbreviations, customers in Ireland can't place an order without a post code even if you set the minimum length to zero, the payment option selection looks like it has a default selected when it doesn't, users don't know that 'authorize.net' is actually the credit card payment option, and so on.

So you spend hours fixing those. Meanwhile, you've got the database populated with all of your products, and you're using it to track sales figures. Maybe you add some back-end order processing stuff for your own convenience.

Eventually this mass of haphazardly patched spaghetti code becomes an absolute nightmare to do ANYTHING with, but by this point you've spent so many hours working on it that it's easier to just keep slogging away at it than to just abandon the whole thing, install something new, and try to move your catalog and your sales records over to the new system.

And THAT is why there are so many osCommerce sites.

What would USPS do?

[140Mandak262Jamuna]

It would probably snailmail patches to all the vendors connecting to their site ;-)

Same here

[Rorschach1]

I came in yesterday morning to find the USPS module non-functional. Worse, the only working option was DHL overnight - and in case you've missed the news, DHL is now about an order of magnitude worse than the post office for domestic delivery. Even for places they say they can do next-day delivery to, actual delivery can take more than a week.

Why? Because they hand it off to the post office rather than deliver it themselves. Why it takes the post office a week to deliver it when I can get it there in two days by sending it by priority mail myself is a mystery. In any case, DHL's out of the (US) domestic game entirely next month.

My site was up last time I checked, but if the USPS option goes down again, I think it's time for a 'free economy shipping' promotion. No messy rate tables to deal with!

Re:Healthcare?

[djnewman]

Um... The post office is private, but protected. Even the best hardware and software fails on occasion. That was my point earlier - we need a better service model if we expect SAS to work. If the small Government crowd had their way the post office would still be the Pony Express. Let's see you get a letter acorss the country in a week for less than 50 cents. UPS and FedX can't - they get $5 for the smallest item. So much for off loading services to business!

Express Delivery

[Statecraftsman]

An unexpected error occurred. An exception occured in module:

USPS

at address: 0x1234 Main St., Hometown USA, zc=0x10001

Re:Very poor design

[sunking2]

This is why my company ships 2 of everything that is ordered. One via USPS and the other by UPS, just in case one gets lost. Always need redundancy.

Re:Healthcare?

[Capybara]

UPS and FedEx can't, because it's illegal.

Re:Sounds like fun.

[MadMorf]

USPS has IT people? Oiy veih I can imagine that job.

A few years ago, I used to be one of them...

Oy Vey, indeed...

hosted vs downloaded/licensed e-commerce

[jrozzi]

The company I work for provides a hosted e-commerce shopping cart solution, SEO-Cart, which supports the USPS Web Tools. Of course the first call coming in for the day was from a client using USPS and having incorrect shipping prices being calculated for their store.

I went ahead and called USPS and the lady who answered was quite rude and explained to me that they had a Worldwide outage which affected other applications than just their Webtools API, and also that they hire a 3rd party company to handle their Webtools API software. She couldn't provide any other information at all and I told her a company of that size should have some sort of fail over plan in place to prevent them from being down as long as they have been. I was really disappointed in the fact she didn't even ask me for my name, phone number, or company by time the conversation was over, but she was probably being bombarded with phone calls all day.

After figuring out that USPS was completely down, I looked through our fail over code and found the following equations seem to come close to the USPS pricing:

National shipping: [cart-weight]*1.6+3.00
International shipping: [cart-weight]*1.6+15.00

These also include pricing for insurance.

After tweaking the fail over pricing code to this, it seemed that everyone using USPS were happy with the results. We also had to decrease the connection timeout set for the request to the USPS Webtools API which was also slowing things down.

The Webtools API seems to be both up and down today, with some orders having shipping prices directly from USPS and others having the fallback pricing. Either way, hopefully their IT department learns from this and also provide us information as to what exactly went wrong.

On that note, this is a prime example that I use when speaking to prospects about the advantages of using a hosted shopping cart solution rather than a licensed/free download solution. Besides the obvious IT benefits that you get with a good hosted shopping cart solution, hosted shopping cart software is typically a centralized application that can provide quick updates to problems like these. Of course this is assuming that the prospect is serious about their online store and doesn't want to handle technical support themselves.

Re:That's what you get... for not using FedEx

[quanticle]

To be fair, what sort of "backup" calculation would you have done here, short of reverse-engineering the USPS algorithm for calculating shipping rates?

I'm not usually a rabid free-market libertarian, but this here can be seen as a result of the fact that the USPS isn't really beholden to its customers. Can you imagine FedEx or UPS being afflicted by such an issue? And, if they were, would they blow off inquiries with a glib, "We expect the servers to be down for the rest of the day?" Of course not, because, for FedEx, UPS, DHL, et. al. such an outage directly affects the health of the organization. If people can't calculate shipping rates, they can't ship, and if they don't ship, the company doesn't make money. The close linkage between revenue and working services tends to put more impetus behind keeping things working and making sure that they get fixed quickly if they do happen to go down.

Re:Healthcare?

[Waffle Iron]

The government can't even manage to keep a simple web service online, and people still believe that it would be wise to let them control health care.

Get real. On several occasions, I've had to manually intervene to fix idiotic billing f*ckups between my PRIVATE insurer and a PRIVATE hospital, who had entered into mutual contracts to be in the same "network". For some reason, they couldn't get their own computers to talk to each other and I had to fix their bugs by going deciphering cryptic paper printouts myself and wasting hours calling customer service. This kind of stupid private healthcare IT problem happens routinely to millions of people every year. Therefore, using your reasoning I conclude that due to a clear history of incompetence, it is unwise to let private parties handle health care, and such practice should be banned.

USPS rate API SUCKS

[Bitsy Boffin]

Apart from the trouble reported in this /. article which I found occuring on one of the existing sites I wrote yesterday (simply because there were no USPS prices being returned, no error, but took about 30 seconds to work out what happened), USPS simply sucks ass.

Here's why:

Some time ago, they had an API to get rates, it was called RateV2.
Then they "updated", and now have RateV3.
RateV3 is the only specification published.
To get access to the Rate API servers, you must first test your implementation against thier testing servers successfully, when they see that they let you on the production server....

Thier testing servers only work with a limited version of RateV2.

So, in order to use the USPS API, you must:

    Write to the now unpublished RateV2.
    Test that RateV2 on the test servers.
    Ask USPS to allow you to use Production (and get the keys etc) because you have successfully tested.
    Write completely new code against RateV3.
    Test that RateV3 on the production servers.

And if you try and show the USPS staff the logical problem in this process, they will reply "I can not put you on production servers, until you have done three successful tests on the test servers".

Re:USPS rate API SUCKS

[SnarfQuest]

If you want to get USPS's attention, FexEx them a package. For some reason, it really pisses them off.

Re:Very poor design

[WilyCoder]

Does your company offer mail order brides?

Re:Couldn't have come at a worse time

[maxume]

The crap I get, I hope that people buying me things are having problems.

Re:USPS

[powerlord]

See Wikipedia.

Sure, but it would have been easier if you had provided a link at least.

The motto will also continue to be mistaken for the motto of the USPS so long as the USPS does nothing to correct that.

The motto is prominently carved over the U.S. General Post Office in NYC

The building prominently bears the inscription: Neither snow nor rain nor heat nor gloom of night stays these couriers from the swift completion of their appointed rounds, which is frequently mistaken as an official motto of the United States Postal Service. It was actually supplied by William Mitchell Kendall of the firm of McKim, Mead & White, the architects who designed the Farley Building and the original Pennsylvania Station in the same Beaux-Arts style. The sentence is taken from Herodotus' Histories (Book 8, Ch. 98) and describes the faithful service of the Persian system of mounted postal messengers under Xerxes I of Persia. The USPS does not actually have an official motto or creed, but nonetheless the inscription on the building is often recognized as such. The inscription was carved by Ira Schnapp, who later designed the Action Comics logo and many other iconic logos for DC Comics.

and the USPS even used it in their own television spot (albeit in an altered form):

The commercial, which ran after the September 11, 2001, attacks and the anthrax mailings, featured no voiceover, only the following text interspersed on title cards:
We are mothers and fathers. And sons and daughters. Who every day go about our lives with duty, honor and pride. And neither snow, nor rain, nor heat, nor gloom of night, nor the winds of change, nor a nation challenged, will stay us from the swift completion of our appointed rounds. Ever.[2]

Note that the [2] link in the above quoted Wikipedia article takes you to this page on the USPS site quoting the exact text of the tvspot.

Yes, they have no Motto, but considering how much they use the meme, they might as well have one.