Slashdot Mirror
HP Keeps Installing Secret Backdoors In Enterprise Storage
Nerval's Lobster writes "For the second time in a month, Hewlett-Packard has been forced to admit it built secret backdoors into its enterprise storage products. The admission, in a security bulletin posted July 9, confirms reports from the blogger Technion, who flagged the security issue in HP's StoreOnce systems in June, before finding more backdoors in other HP storage and SAN products. The most recent statement from HP, following another warning from Technion, admitted that 'all HP StoreVirtual Storage systems are equipped with a mechanism that allows HP support to access the underlying operating system if permission and access is provided by the customer.' While HP describes the backdoors as being usable only with permission of the customer, that restriction is part of HP's own customer-service rules—not a limitation built in to limit use of backdoors. The entry points consist of a hidden administrator account with root access to StoreVirtual systems and software, and a separate copy of the LeftHand OS, the software that runs HP's StoreVirtual and HP P4000 products. Even with root access, the secret admin account does not give support techs or hackers access to data stored on the HP machines, according to the company. But it does provide enough access and control over the hardware in a storage cluster to reboot specific nodes, which would 'cripple the cluster,' according to information provided to The Register by an unnamed source. The account also provides access to a factory-reset control that would allow intruders to destroy much of the data and configurations of a network of HP storage products. And it's not hard to find: 'Open up your favourite SSH client, key in the IP of an HP D2D unit. Enter in yourself the username HPSupport, and the password which has a SHA1 of 78a7ecf065324604540ad3c41c3bb8fe1d084c50. Say hello to an administrative account you didn't know existed,' according to Technion, who claims to have attempted to notify HP for weeks with no result before deciding to go public."
Comment removed based on user account deletion
Besides Apple, Intel, and every social networking site and cloud service provider.
"SO we bide our time, waiting for a purer kick to bloom and the future is still bleak, uncertain and beautiful" -GSYBE
Years ago I worked on HP3000 servers and there was an hpsupport user on those systems as well. But on the 3000 series it was documented and every sysadmin was aware of it and could change the password if desired. Looks like HP still cares about customer service, but no longer cares about ethics. Sad. They were once a really great company.
Rainbow Tables: enabling ontopic first posts since 2013.
Without reading TFA, which I expect to be even more sensationalist crap:
I grok this to mean that a backdoor exists for customer service, which can be activated by a customer (by two factors: permission and network access), and that without action on the part of the customer, said backdoor is closed.
Did I miss something?
If so, please synopsize in non-sensationalist terms.
Indeed, whatever the case: Please post a not-purposefully-scary summary of the actual problem below, because right now it sounds a whole lot like the not-backdoor that Remote Assistance is under Windows.
Kid-proof tablet..
Pretty every much hardware/software stack combination that I ever encountered over 30+ years of programming had a "back door" admin account to allow the vendor to get into the systems to repair damage. This is nothing new.
Yes, it's a security hole.
But it's also standard practice and should come as no surprise to anyone.
I do not fail; I succeed at finding out what does not work.
The password you're looking for is badg3r5. So there. Go forth, my minions! In other news, Slashdot's corporate overlords apparently no longer believe in full disclosure, as it had in the past, and now omit critical information probably because their lawyers have more say in the editorial process than the submitter, editors, or anyone with a clue to spare. :(
#fuckbeta #iamslashdot #dicemustdie
I work for a large networking appliance company. We know these backdoors are a bad idea from a security standpoint. The problem is, customers demand them. They call up and want something fixed--or a customization or diagnosis or whatever--and many times the only way to resolve the issue is to access the box. Most times it's a configuration problem on their end, but often the quickest way to figure this out is to access the internal databases.
On our appliances our backdoors are completely optional--if you disable it, support is completely unable to access the box, period (I know because I helped to write it). But you wouldn't believe how irate customers become when you tell them that you can't help them, even though they're the ones who _chose_ to disable the support access, and clicked through all the warnings.
Could these backdoors be made more secure? Absolutely. But developing, say, a storage appliance and developing a secure remote access protocol (both in terms of software as well as access control) are worlds apart. SSH and SSL are just tiny elements in an overall solution.
I'm not one to argue that convenience and security are necessarily opposed. But it is incredibly hard to find the small set of solutions that provide both maximum convenience and maximum security. And even if you've found a solution in that set, it's incredibly hard to prevent it from degrading over time as developers come and go, introducing bugs as they add and fix features.
When you buy an 'Merican product you are buying Freedom!
The earlier article said they can reset user passwords, if they can do that, they can grant themselves access to the data.
http://www.theregister.co.uk/2013/07/09/hp_storage_more_possible_backdoors/
" lost admin passwords are resettable by HP. One, from November 2011, states: “You will need to call support and they can get into the backed and reset it for you. 1-800-633-3600 'Lefthand Solutions'”. The other, posted by a LeftHand product manager in 2009, states: “Call support. They can reset the password remotely.”
So they CAN get access to the data, because they can change the configuration to give themselves access.
I used HP3000s back in high school. They had plenty of other security holes, too.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
They're going bankrupt anyway so this issue will take care of itself.
NEXT!
On the system I worked on, there is a manufacturing mode that only someone with Admin privilege AND a manufacture mode password generator can enable. This means only HP support personnel can turn it on if the customer allows it.
Once it is turned on, root access can be gained using a private key.
My worst fears of how deep spying has gone keep getting confirmed. Pretty much every single major vendor is backdoored by the NSA one way or another.
Good-bye
I work for a large networking appliance company. We know these backdoors are a bad idea from a security standpoint. The problem is, customers demand them. They call up and want something fixed--or a customization or diagnosis or whatever--and many times the only way to resolve the issue is to access the box. Most times it's a configuration problem on their end, but often the quickest way to figure this out is to access the internal databases. On our appliances our backdoors are completely optional--if you disable it, support is completely unable to access the box, period (I know because I helped to write it). But you wouldn't believe how irate customers become when you tell them that you can't help them, even though they're the ones who _chose_ to disable the support access, and clicked through all the warnings.
This was my exact experience when working on telco infrastructure equipment years ago. We knew it was bad security but customers wanted it.
If working on such equipment today I would expect that we would incorporate a time-based one-time password that the customer would have to provide to our support person. Hardly perfect but a bit better than what seems to be common place today.
The right answer is a service account they can have activated, if needed. On the EqualLogic (Dell) we have that is how it is done. When they need to work on the system, they have you connect to a WebEx session. They then request control of the PC. They have you log in to the system using your admin account, and they can then set the password on an "fse" account, which they can use to access service functions you aren't supposed to get at. Once they are done, they encourage you to change the fse account to a different password.
That is how it is properly done: They get in using your system, with you monitoring what they do, and you lock out access after they are done.
Now maybe they are going to have access all the time for proactive monitoring. Fine, that is a service some like (we may take Dell up on it if they start offering it). Again the right method is an account set up by the customer, not one hardcoded in. Why? Well because of shit like this. If it is hardcoded in, and you can't change it, then if someone discovers the access, it is bad times.
For that matter I've never seen this on Cisco stuff either. The recovery for that is via serial, I've never seen a remote override from Cisco. Maybe it is there, but I've never seen them use it.
It's worse than that. The DOJ uses a HP system.
That's a pretty nifty idea.
Is it possible to engineer the appliance so that instead of using passwords sent remotely to access the appliance, access is only granted when a physical switch is flicked on by the consumer? i.e.
Operator: Okay, we are connected to your system, press the red button now.
Customer: *press*
Operator: Okay now were in. Gimme a few minutes while we check your system.
No one listens to the security group no matter how badly they get hammered. This is just dumb shit. If I ran the world everyone who was involved with implementing this would be fired immediately.
Remote access for customer support is a great thing...just build it right. It's really not that hard at all to build it right...probably even easier than building it this stupid ass way.
Would you rather deal with Rainbow Tables or Bobby Tables?
If the computer belongs to the corporation the CEO works for then chances are he already has authorization.
Actually, through to the early 80's. Hewlitt and Packard, the men, had a true sense of worth of their employees and treated them with respect. That was pretty much gone as the 80's rolled on. Packard was a changed man from his stint(s) in Washington. Then, of course, by the time Patricia Dunn was in charge, the company was a toilet. Pretexting, anyone? Yeah, sad.
The Feds probably paid for the backdoor.
What changed under Obama? Nothing Good
Why not both?'); UPDATE vulnerabilities SET failtype = 'Bobby' WHERE admin = 'fool'; --
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
It's probably buried in the TOS that you implicitly agreed to when you opened the box, so they're covered.
If HP had decided to store their passwords properly, by using Bcrypt or Scrypt with a decently high work factor, we would not be having this discussion... their password could be badg3r5, and it would take at least 5 or 6 hours to crack using a dicitonary search with l33t-speak substitution, so there probably wouldn't be 50+ people having discovered it within a couple days :)
Your SSH ports should never be exposed to the public internet directly. Generally you want a "jump" box that is a very tight and tied down system (selinux/freebsd) with RSA keys to get in. Just Saying
That looks suspiciously like the sort of simple password my ex-boss used to insist we use for things like Domain administrator accounts on Windows. He was an HP-UX admin at one point - does HP offer a free "find a crappy password" tool?
#DeleteChrome
Huawei ... rumors
Am I the only one that remembers the actual holes? https://www.computerworld.com/s/article/9229785/Hackers_reveal_critical_vulnerabilities_in_Huawei_routers_at_Defcon
(Sure it might not have been an intentional backdoor but still works as one. I don't see why we shouldn't treat security issues like this.)
Those laws are for people that does things against the government/corporations, not for corporations doing it for the government. Having backdoor will be the new normal, at least if people keeps buying from them.
And don't think the "consequences" will include removing them, the fix will only just put them more hidden, or reinstall them with the next update.
78a7ecf065324604540ad3c41c3bb8fe1d084c50 ? Really ? Crap... that's the combination to my luggage.
Just out of curiosity, since it's relevant but perhaps well known to most here but me, are rainbow tables capable of mixed letters and numbers and, say, 8 character pw length already widely available and searchable that fast with ordinary hardware? Are all my passwords (for those places still not accepting passphrases, which is most I deal with) that vulnerable once /etc/shadows is accessed?
The subject who is truly loyal to the Chief Magistrate will neither advise nor submit to arbitrary measures (Junius)
You people do realize that for *years* high end disk arrays shipped with *gasp* modems.
So if a problem occurred the array could 'phone home', open a case, upload logs and tell the vendor a problem took place. Then the vendor could dial in, diagnose the problem and dispatch a CE with the replacement part.
The techs accessing the arrays over the modems couldn't 'download' the customer data. Yes there were some companies that wouldn't allow the modem to be installed and would often have to sign very long legal documents basically saying that if a hardware failure happened and the vendor wasn't notified, the customer assumed responsibility.
Woo, go badgers! By the way, I'm fairly certain they have little to no presence in Wisconsin. I'm an IT manager in WI and the closest HP support and sales agent is in Illinois or something like that. The password was either randomly generated, related to the meme video, or some other strange source.
Rainbow Tables: enabling ontopic first posts since 2013.
if it's that then it's the same as the previous.. unless the badgers post was joke then and now.
world was created 5 seconds before this post as it is.
They need to get a snake grip on this before it mushrooms.
How is the Riemann zeta function like Trump rallies? Both have an endless number of trivial zeros.
They don't even have the common decency to at least choose a password that isn't already in every rainbow table on the planet.
If I were to make a back door system, I'd make sure customers knew about it. I'd make it so that a physical switch had to be activated on the device itself in order for the back door to be used. Activating the switch would be plainly obvious, with both physical indicators on the device and in management software, with auditing and warnings that the back door has been activated - and detailed logging of that account logging in. I'd use a 30-character randomly-generated password at least, if not some kind of public-key system, to authenticate the back door login.
If having to go to the physical device is a pain for you the customer, you can always just leave the switch always activated - you'd still be better off than those badg3r5 at HP.
"Screw Sun, cross-platform will never work. Let's move on and steal the Java language." - Visual J++ Product Manager
I'm an IT manager in WI and the closest HP user support and sales agent is in Illinois
They definitely have people they don't let you talk to, and I'm betting those guys wrote this account into the software.
Finally had enough. Come see us over at https://soylentnews.org/
Let's face it, it's far more likely to be "HP1234" than anything as complex as l33t-speak.
No sig today...
Oh, wait... the SHA1 of "badg3r5" is actually 78a7ecf065324604540ad3c41c3bb8fe1d084c50.
(mushroom, mushroom)
No sig today...
Actually for those that administer these systems this is old news...
I worked for a financial institution that had two four node clusters of storage products [one was SATA based and the other SAS based] which were developed by LeftHand Networks which ran on HP storage servers [DL320s] and Dell hardware as well. Shortly after we installed the clusters, HP bought LeftHand... and the LeftHand OS [then called SAN i/Q] became an HP only product [they dropped support for other hardware]. at that time (2008) this support back door already existed... I had occasion to allow a LeftHand support engineer access to a node which had taken itself off line... and the only way to bring it back was the command line backdoor -- It was part of the LeftHand OS / Cluster Administration software... LeftHand OS is a actually Linux with some custom cluster control / management software.
The real issue of this account is that it allows a third party access to an interface that the owner of the hardware cannot access-- yep, that is right LeftHand did not trust its clients with access to the command line on their storage server products... you were buying a very complex "Storage Appliance" which *required* a support contract... they were designed as a RAID 5 Cluster. Each Node was set up as a RAID 50 array and then the nodes were then clustered as Raid 5... you could lose a lot of drives and still have a cluster which at the time was something unique on the market
On the other had (the left one?) the Support Engineers at LeftHand were extremely knowledgeable of their products [It was then a start up and at least in part employee owned] and they were actually concerned and responsive to the needs of their customers... I was sad that the senior Support folks cashed out and moved on when HP bought them...
When HP took over that all went out the window... by 2009 the front end of the support operation went to Mexico and if you really did need a support engineer they would have the engineer call you [previously the Support number was a direct line to the support engineers]
Now my recollection was the reason that HP bought them was that LeftHand had a product that was better than HP's offerings at the time... so it should not be surprising that the LeftHand code base evolved / moved into other products...
The bottom line is that the only way to get access to the command line of a LeftHand node required either SSH access or a modem connection. As an administrator, giving network access to black hats by failing to block access SSH access to sensitive systems from unknown IP space just shows you are an idiot. While I understood the reason for the back door my only real fear of it was that some HP trainee engineer would wipe a cluster and take down the vSphere cluster that the storage cluster supported. The fact that my boss did not know the password made the system safe... since my boss knew nothing about server systems or networks...
When I played with MD5 rainbow tables, probably 10-15 years ago, it was an interesting experience.
I signed up to a website and was given a large block of passwords to crunch. I can't remember my block, but it was full of 7 character alpha-numeric passwords. There were some 6 character password blocks left to crunch, but 99% of them were complete.
My P3 450 crunched them all weekend and beyond. In return, I was given complete access to the MD5 rainbow tables, through some forms on a website.
It was a near-instant search.
Assume that your 8 character passwords are fully hashed. All alpha-numeric passwords 7 characters and under were complete back then.
Asking Google to search for hashes is also fun.
This post contains benzene, nitrosamines, formaldehyde and hydrogen cyanide.
The point is not that such access exists, the point is that it is NOT DOCUMENTED.
Oh wait... I thought you were joking!
The SHA1 of "badg3r5" really is "78a7ecf065324604540ad3c41c3bb8fe1d084c50".
http://www.sha1-lookup.com/index.php?q=78a7ecf065324604540ad3c41c3bb8fe1d084c50
HP used "badgers" in leet-speak for an NSA backdoor? Smells like they wanted people to know, to me. Maybe they didn't like what they were supposed to be doing, and stuck their tongue firmly in cheek at the implementation stage? "Screw the NSA - we'll give them a back door if they want it so much - and we'll make it so that researchers find it easily, so our business isn't damaged in the long term ("If we wanted you data so much, we'd have done a better job of hiding it - blame your government")
This tagline was transcoded to result in at least one smirk. If you experience failure to smirk, please consult your Gen
You've got it backwards. The computer abuse laws are for jailing the evil hackers who published the information.
Yep, that's why Eugene Volokh and his dad Vladimir made money selling that security package (yes, that Eugene Volokh, the law professor at UCLA. He was a kid genius programmer.)
Peter predicted that you would "deliberately forget" creation 2000 years ago...
Customers demand backdoors, period. And as long as there are backdoors, there are going to be exploits for them.
Backdoors for service accounts don't have to be trivial to exploit. If you ship a secret backdoor with the sort of simple password protection this one has, you deserve to go out of business over the resulting negligence lawsuits.
You didn't need to swear at the tech guy, manners goes a long way.
Fuck you.
You self-important, overblown dick.
Hypocrite much?
Peter predicted that you would "deliberately forget" creation 2000 years ago...
Would this fact not have been obvious the first time someone called support?
Troll is not a replacement for I disagree.
"Mister potato head . . . MISTER POTATO HEAD . . . back doors are not secrets!"
977533ed14dd55576b6bf27f869b040b68e39bd7
Troll or just stupid?
A succinct explanation for people that might be dumb enough to think like that: if all your security can be bypassed remotely with a click of a button then you have no security.
HP is not an ethical company. Another example is their printers. They now put small computer chips on every ink cartridge, which allows them to distinguish their cartridges from another brand. Those embedded chips also allow the printer to distinguish a new ink cartridge from one that has been refilled. The result? My $160 OfficeJet printer refuses to work with a non-HP ink cartridge, AND it refuses to work with a refilled cartridge. Luckily, I found a bit of a hack on the Internet - every time I print something, I have to physically go to the printer and open and close the access door for the ink cartridges. That at least lets me use HP cartridges that have been refilled at a local refilled-ink-cartridge store. But the hack doesn't work for refillable cartridges that can be purchased cheaply online.
Personally, I will never buy ANY HP product again.
Badgers! We don't need no stinkin' badgers! (But apparently we get them anyway.)
AJ Henderson
A 7 character password is stupidly short these days. This is just retarded, as a brute force attack against a password that short takes very little time now (few hours at most). Even a 'leetspeak' password at double that length would take much much longer, but still get knocked out in less than a month.
So make it something that can be enabled if needed, but is disabled by default. Maybe a special network port that should only be plugged in for recovery or a serial console that requires physical access (both still needing a proper login), or something like a jumper/switch/etc that are off by default by when turned on toggle a recovery console/account.
78a7ecf065324604540ad3c41c3bb8fe1d084c50 78a7ecf065324604540ad3c41c3bb8fe1d084c50 (repeat x6)
Mushroom mushroom!
78a7ecf065324604540ad3c41c3bb8fe1d084c50 78a7ecf065324604540ad3c41c3bb8fe1d084c50 (repeat x6)
A snake a snake! Snake a snake ohhh it's a snake...
The damn Slashdot Lameness Filter won't let me fully write it out without the (repeat x6). Grrrr.
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.
Reality check - What good would having administrative access ONLY to your storage array be to the NSA? The only thing they can do is to screw up your storage.... they won't be able to READ any of your precious data.
This just underlines the need for a separate administrative network that doesn't have access to your data network.
Hell have you ever had to call HP support? if it was more complex than "HP001" I'd be surprised,maybe "HP#1" if they wanted to get fancy ;-)
ACs don't waste your time replying, your posts are never seen by me.
You think that's all they have?
What is the back door for iLO, the HP remote admin for servers? You don't think they'd put one in the storage but leave out the servers do you?
I administer HP p4500g2 mulit-site SAN clusters where I work and they're wonderfull. I saw this "backdoor" in action once when hard drive firmware bricked an entire shelf. I am very thankful HP added this functionality.
I also believe the unnamed Register source has it wrong. Rebooting specific nodes should never "cripple the cluster". I could spend all day rebooting nodes with zero downtime. That's the reason HP has you install failover managers on local ESX(i) storage.
If you properly implement VLANs like you should why would this be a concern? What am I missing?
I'm not an HP advocate but I feel this story is misleading.
http://www.youtube.com/watch?v=EIyixC9NsLI
Custom electronics and digital signage for your business: www.evcircuits.com
Maybe they like http://www.youtube.com/watch?v=6joOVjEemh4
It was the first link on google when searching for the hash. Not even salted, well done HP.
A NSA backdoor would let you get at the data. This is a support backdoor.
Well he said HP3000 servers so about then, yes.
HAHAHAHA. I work at HP, doing enterprise customer support. I have five different companies I support, and have at least 12 different "bosses" I report to. I can totally understand why they never heard from HP. I submit issues with various Citrix portals, access to exchange boxes, etc, and never hear anything either. The bureaucracy there is so thick it makes Brazil look organized. I've been waiting on access to RSA admin accounts for over seven months now...just when some progress seems to be happening someone goes on vacation, someone moves to a new position, etc...and the whole process has to be restarted, only to stall again because the paperwork runs into itself where it was already sitting! I'm still using my lead's authentication...which we're all pretty sure is a violation of NERC's security policy but no one cares. We have so many outsourced business parts that it's become a game of bouncing tickets around to avoid SLA violations, different companies ignoring our tickets until they are almost an SLA violation, then assigning them back to us and not telling us, because we don't have access to whatever system that notifies people about ticket re-assignment. Every day I have tickets I submitted a month or so ago come back to me with notes like "not our department, please re-assign"...and the tickets have gone from me to their t2 pool, then their t2 pool sits on it and then sends it to a (seemingly mostly random) department who sits on it until the last second because it's "not their job" and they assign it to "misroute", then the computer sits on it for a few days and auto-assigns it back to the originating person (me). They should go back to t2, since all I mostly do is re-assign it to them...and the whole time the customer STILL ISN'T RESOLVED. Gods forbid actually giving me any access to fix these issues myself...even when I'm supposed to be able to, half the time I feel I'm breaking the law by using someone else's credentials to log into whatever and do it. I wouldn't be surprised if their request is still slowly winding it's way through a dozen different departments so that HP's response will be "compliant" with company PR releases.
Um.....
http://en.wikipedia.org/wiki/United_States_Deputy_Secretary_of_Defense
That is why experienced and highly intelligent bosses, like myself, use 1 2 3 4 5!
HP's doomed Itanium was _ahead_ in the 64 bit race. It was one of the first 64 bit processors that gained reasonable market share.
The true reason for its demise is the lack of backwards compatibility. They decided to fix everything in one go: 64 bit, increased execution parallelism without programmer effort etc.
Years later AMD came up with x64 that was compatible with x32 and Intel quickly hoped on board as it saw the marked liked backward compatibility.
I have seen the Itaniums, if the program was slightly optimized, it would beat the daylights out of other architectures in terms of performance. Too bad that didn't count.
Nevermind the fact that their laptops are the least-service-friendly machines I've ever laid a screwdriver on.
Are you confusing HP with Apple?
Obviously it was so they did not know what their RightHand was doing.
The irony is that Badger5 is a Wisconsin Lottery game. http://wilottery.com/lottogames/badger5.aspx