You joke about management, but I wonder if that's the root of the problem - too many chiefs, not enough Indians. 30,000 employees to operate 567 stores? That's more than 50 employees per store. I realize they have a corporate HQ and all, but the figure still seems excessive.
You know, you can pay online without making it automatic.
You raise an excellent point. However, they (typically) stop sending paper bills in favor of email notices once you start paying them online. With postfix and spamassassin, email occasionally gets misflagged, misfoldered, or otherwise misrouted. Forgetting that a certain bill is due, or not receiving the email notice for some reason, is IMO even worse than having an automatic payment set up. The physical paper bill is just as much a part of my fiscal responsibility process as is the physical paper check.
An odd aside. My utility company, Memphis Light Gas & Water, allows customers to register online to view their current statement. I did this once after I received an erroneous Cut-Off notice. Now, every month, their online billing system sends me two emails:
a) "Your MLGW Bill is Ready"... then, two weeks later, after they've already cashed my check...
b) "Your MLGW Bill is Overdue!"
Their system assumes that if you're signed up to receive your statement online, then you must also be paying online. The online system doesn't talk to the offline system. So if you're set up to receive e-statements, but you pay by check via mail, the online system never registers a payment and will email a late notice every freaking month.
It's clunky, and doesn't do much to inspire my confidence in online bill-pay.
Check out the contents of your/var/log/auth.log. I run FreeBSD and haven't had root on a linux machine in awhile, but also take a look and see if you have a/var/log/security.
Things like this make me nervous about switching to otherwise-tempting online bill payment, but checks are dangerous, too.
I'm one of those holdouts who still use paper checks, envelopes, and stamps to pay my bills. Once a month or so I'll bring the stack into the office and take care of it during downtime, and folks look at me like I'm transmitting morse code over a telegraph. I do bank online, but I don't do online bill pay.
One reason I still cling to checks is that they allow me to be the final arbiter and gatekeeper of my money, and I have better fiscal responsibility when I'm directly involved in disbursement. Each time I physically write out a check, there's a bit of mental bookkeeping that takes place. You can't sit down and write "One thousand one hundred ninety-eight and 32/100" without pausing for a moment to think, holy shit, that's X% of my paycheck. If you elect not to use online bill pay, you have to actually look at your credit card statements each month, instead of just setting up a $200 monthly ACH and ignoring the current total.
I'm afraid that if I set everything up to be paid automatically, I'd very quickly wake up to discover that my checking account is overdrawn because I wasn't paying enough attention. Writing checks and licking envelopes is my way of keeping tabs on what's going out the door each month. The potential security benefits don't hurt, as anyone screwing around with mailed bills faces the wrath of the United States Postal Inspection Service. Unlike most online fraud, fucking with the mails will actually get you in trouble, and USPIS doesn't blow you off if you haven't suffered hundreds of thousands of dollars in losses.
I do miss the one benefit that physical checks had up until a couple of years ago, the float. Check21 pretty much ruined that, but maybe it was for the better. Come to think of it, I haven't overdrafted since Check21.
Long live the check, just stay away from my routing numbers.
So we are still user password-based SSH authentication?
The problem is that in any sort of working environment, where you have a very heterogeneous user base, it's really really hard to enforce anything else.
Users - even the most basic of users - can be trained to enter a username and a password. They do it on Hotmail, they do it on Google, they do it on MySpace, they're used to the idea that when they want to login somewhere, they have to enter a username and a password. "That's how the internet works." So when their job functions require that they PuTTY into a box and make a couple choices from a shell-script menu, training them to enter a username and password is no big deal. Getting them to wrap their brains around a different authentication scheme is very difficult, even if your user base is fairly adept. Trying to set it all up for them is beyond the scope of most IT departments.
I've come to use passwordless key-based auth for ssh, but not so much for security as for convenience. I share a single DSA key across 6 or 8 machines because it's damn easy to generate a key on one box, append it to ~/.ssh/authorized_keys2 on all of them, and forget all about it from there on out. ssh just works. svn just works. rsync just works. You create your key and make it common among your systems, everything is...fluid. But try convincing someone who isn't a sysadmin, and doesn't have to deal with multiple machines, and doesn't use other applications that tunnel on top of ssh, that there's a benefit to setting up "weird encryption key stuff."
I have a 1u (personal, non-work-related) server in a colo facility. There are fewer than 10 users, all close friends, all tech savvy, all CS/IT types. Even with this very specialized audience, I couldn't convince all of them to switch to key-based auth; if I disabled PasswordAuthentication, I wouldn't hear the end of it. Temporarily moving sshd to a different port was hard enough. I can't even begin to imagine the hell that would ensue if they suddenly went key-based only at work.
It's all good. Everyone likes to ask, "Who watches the watchers?" But without fail, the IT department watches itself, because nobody else knows how to.:)
The only solution I can see to this would be to take an approach similar to the centralized spam-fighting solutions; a DNSBL [wikipedia.org] specialized for brute-force botnets.
Damn! I just posted about this as a reply to a previous thread, and it definitely belongs here instead.
Anyway, check out BruteForceBlocker, it's exactly what you describe, but it's implemented as a plaintext list instead of a DNSBL. Hosts using BruteForceBlocker can report attacks back to the central server. The list of recently reported attackers is public.
It's meant for BSD variants using the pf firewall, but I rolled my own implementation to parse FreeBSD ipfw logs (and report back) in a day or two. A daring volunteer could easily create a fork that works with Linux auth logs and iptables instead. Or set up a DNSBL that parses the list every hour or so and creates the appropriate zones.
While it's been pointed out that fail2ban isn't effective against this particular attack, I wanted to point out a similar utility called BruteForceBlocker.
It was written as a reactive firewall that parses pf logs on OpenBSD and FreeBSD (pf is "the iptables of BSD"). The coolest feature IMO is that it's a community effort, in that each participating host can elect to share its logs with a centralized server. That server then publishes a list of recently reported SSH attackers which you can script into your firewall rules, even if you aren't running the client. It's like a Vipul's Razor for SSH bruteforce reports.
Since I still use ipfw instead of pf on FreeBSD, I rolled my own implementation, but it still contributes back to the master database of recent attackers.
As an aside, for those who aren't familiar with DShield, it's a community effort where thousands of people submit their IDS logs to create aggregate statistics about intrusion attempts worldwide. And if you happen to run FreeBSD with ipfw as your firewall, check out FreeBSDShield, my DShield reporting client for FreeBSD.
Okay, how is this different than previous patterns of hacking activity, other than the fact that they're aquiring compromised machines via a bot net?
You're sort of missing the point, I think, in that what's different about this pattern of activity is precisely the fact that it's being done with a botnet.
For one thing, there's a new level sophistication, primarily in that this bruteforce campaign is not the least bit random. I'm being hit by thousands of distinct attackers, yet the progression of usernames being attempted is undeniably alphabetical. Occasionally a particular username is attempted more than once, but it's typically sequential. One attempt per username with the attacking hosts only making one attempt every few hours.
The level of coordination required for this sort of attack is unprecedented. Across thousands of bots, each one at any given moment is able to determine:
That I am among the pool of targets to be probed
That I am, at this precise second, the next target to be probed
That this particular bot hasn't probed me recently and is now eligible to probe me again
Which usernames have already been probed on my machine
The next username, in sequence, that should be attempted on my machine
In the past, brute force SSH attacks have always been obvious. Typical hit and runs. One host will spew hundreds or thousands of attempts at a target, typically in quick succession, typically focusing on system accounts, and typically trying a shitload of passwords against each account. Firewalls and IDS deployments far and wide will now easily detect (and often block) these attacks immediately because they're so easy to recognize.
This attack is very different. It's not targeting system accounts, it's hoping to get lucky against a vast list of potential userland lognames. It's only trying once or maybe twice per account. And it's distributing these attempts, round-robin style, across an impressive number of sources, with enough logic so that bot B will not attack host H unless all other bots in the network have sequentially exhausted their "token" attempt on host H.
What we're seeing is flying under the radar of a shit-ton of IDS/firewall implementations, and is harder to fight.
I would love to get my hands on the C&C database being used to coordinate all of this. Much as I hate to admit it, the architecture of this attack is unique and innovative, and I'd like to see what makes it tick.
Slashdot Mirror
Annals of Neurology
Hmm, parent may contain a Goatse link!
nless you explicitly block Skype, YIM, and others, cellphones are no longer relevant as anyone can use VoIP
RTFA, they're going to block VoIP.
Wait, when did the scientologists get involved?
50% of Americans won't know how long it takes the earth to go around the sun
Heresy! Everyone knows the sun revolves around the Earth, and it takes 6,000 years for it to pass around all four corners.
Because driving drunk is ok??
No one's making that argument; the objection is to how "drunk" is defined in many states now (mostly thanks to MADD).
what's so bad about Nickelback? I like a few songs from them...
You mean you like the song from them?
what hope is there for Sarah Connor and Dollhouse on a Friday night
Well I don't know about a dollhouse, but Summer Glau is welcome at my place any Friday night...
pdf is unsearchable
Rush, is that you?
Hi! we run a non-profit website that gets 100 million visitors a day
Oh cool, Moot's on Slashdot.
I too watched my stations switch-off their analog, but it was anticlimatic.
Well then, they should have signed-off with the weather team showing doppler ra
BTW, I've run my own DNS. Not dong that again.
I think you were running the wrong daemon...
Did someone use their phone to take a picture of a Congressman's daughter drunk at a college party?
Well, yeah, but it's probably happened again since then...
You joke about management, but I wonder if that's the root of the problem - too many chiefs, not enough Indians. 30,000 employees to operate 567 stores? That's more than 50 employees per store. I realize they have a corporate HQ and all, but the figure still seems excessive.
I find it unlikely that a significant number of US armed forces would turn their tanks, aircraft, and artillery against American citizens.
Robots (and drones), on the other hand...
Most airlines have something very similar that is created every time you make a reservation.
Sounds good to me. The question is, what the fuck is the government doing with it?
This is one of the most informative posts I've ever seen on Slashdot. Thanks for taking the time to write it!
You forgot:
You raise an excellent point. However, they (typically) stop sending paper bills in favor of email notices once you start paying them online. With postfix and spamassassin, email occasionally gets misflagged, misfoldered, or otherwise misrouted. Forgetting that a certain bill is due, or not receiving the email notice for some reason, is IMO even worse than having an automatic payment set up. The physical paper bill is just as much a part of my fiscal responsibility process as is the physical paper check.
An odd aside. My utility company, Memphis Light Gas & Water, allows customers to register online to view their current statement. I did this once after I received an erroneous Cut-Off notice. Now, every month, their online billing system sends me two emails:
a) "Your MLGW Bill is Ready" ... then, two weeks later, after they've already cashed my check...
b) "Your MLGW Bill is Overdue!"
Their system assumes that if you're signed up to receive your statement online, then you must also be paying online. The online system doesn't talk to the offline system. So if you're set up to receive e-statements, but you pay by check via mail, the online system never registers a payment and will email a late notice every freaking month.
It's clunky, and doesn't do much to inspire my confidence in online bill-pay.
Check out the contents of your /var/log/auth.log. I run FreeBSD and haven't had root on a linux machine in awhile, but also take a look and see if you have a /var/log/security.
Things like this make me nervous about switching to otherwise-tempting online bill payment, but checks are dangerous, too.
I'm one of those holdouts who still use paper checks, envelopes, and stamps to pay my bills. Once a month or so I'll bring the stack into the office and take care of it during downtime, and folks look at me like I'm transmitting morse code over a telegraph. I do bank online, but I don't do online bill pay.
One reason I still cling to checks is that they allow me to be the final arbiter and gatekeeper of my money, and I have better fiscal responsibility when I'm directly involved in disbursement. Each time I physically write out a check, there's a bit of mental bookkeeping that takes place. You can't sit down and write "One thousand one hundred ninety-eight and 32/100" without pausing for a moment to think, holy shit, that's X% of my paycheck. If you elect not to use online bill pay, you have to actually look at your credit card statements each month, instead of just setting up a $200 monthly ACH and ignoring the current total.
I'm afraid that if I set everything up to be paid automatically, I'd very quickly wake up to discover that my checking account is overdrawn because I wasn't paying enough attention. Writing checks and licking envelopes is my way of keeping tabs on what's going out the door each month. The potential security benefits don't hurt, as anyone screwing around with mailed bills faces the wrath of the United States Postal Inspection Service. Unlike most online fraud, fucking with the mails will actually get you in trouble, and USPIS doesn't blow you off if you haven't suffered hundreds of thousands of dollars in losses.
I do miss the one benefit that physical checks had up until a couple of years ago, the float. Check21 pretty much ruined that, but maybe it was for the better. Come to think of it, I haven't overdrafted since Check21.
Long live the check, just stay away from my routing numbers.
So we are still user password-based SSH authentication?
The problem is that in any sort of working environment, where you have a very heterogeneous user base, it's really really hard to enforce anything else.
Users - even the most basic of users - can be trained to enter a username and a password. They do it on Hotmail, they do it on Google, they do it on MySpace, they're used to the idea that when they want to login somewhere, they have to enter a username and a password. "That's how the internet works." So when their job functions require that they PuTTY into a box and make a couple choices from a shell-script menu, training them to enter a username and password is no big deal. Getting them to wrap their brains around a different authentication scheme is very difficult, even if your user base is fairly adept. Trying to set it all up for them is beyond the scope of most IT departments.
I've come to use passwordless key-based auth for ssh, but not so much for security as for convenience. I share a single DSA key across 6 or 8 machines because it's damn easy to generate a key on one box, append it to ~/.ssh/authorized_keys2 on all of them, and forget all about it from there on out. ssh just works. svn just works. rsync just works. You create your key and make it common among your systems, everything is...fluid. But try convincing someone who isn't a sysadmin, and doesn't have to deal with multiple machines, and doesn't use other applications that tunnel on top of ssh, that there's a benefit to setting up "weird encryption key stuff."
I have a 1u (personal, non-work-related) server in a colo facility. There are fewer than 10 users, all close friends, all tech savvy, all CS/IT types. Even with this very specialized audience, I couldn't convince all of them to switch to key-based auth; if I disabled PasswordAuthentication, I wouldn't hear the end of it. Temporarily moving sshd to a different port was hard enough. I can't even begin to imagine the hell that would ensue if they suddenly went key-based only at work.
Sh-h-h-h!
It's all good. Everyone likes to ask, "Who watches the watchers?" But without fail, the IT department watches itself, because nobody else knows how to. :)
The only solution I can see to this would be to take an approach similar to the centralized spam-fighting solutions; a DNSBL [wikipedia.org] specialized for brute-force botnets.
Damn! I just posted about this as a reply to a previous thread, and it definitely belongs here instead.
Anyway, check out BruteForceBlocker, it's exactly what you describe, but it's implemented as a plaintext list instead of a DNSBL. Hosts using BruteForceBlocker can report attacks back to the central server. The list of recently reported attackers is public.
It's meant for BSD variants using the pf firewall, but I rolled my own implementation to parse FreeBSD ipfw logs (and report back) in a day or two. A daring volunteer could easily create a fork that works with Linux auth logs and iptables instead. Or set up a DNSBL that parses the list every hour or so and creates the appropriate zones.
While it's been pointed out that fail2ban isn't effective against this particular attack, I wanted to point out a similar utility called BruteForceBlocker.
It was written as a reactive firewall that parses pf logs on OpenBSD and FreeBSD (pf is "the iptables of BSD"). The coolest feature IMO is that it's a community effort, in that each participating host can elect to share its logs with a centralized server. That server then publishes a list of recently reported SSH attackers which you can script into your firewall rules, even if you aren't running the client. It's like a Vipul's Razor for SSH bruteforce reports.
Since I still use ipfw instead of pf on FreeBSD, I rolled my own implementation, but it still contributes back to the master database of recent attackers.
As an aside, for those who aren't familiar with DShield, it's a community effort where thousands of people submit their IDS logs to create aggregate statistics about intrusion attempts worldwide. And if you happen to run FreeBSD with ipfw as your firewall, check out FreeBSDShield, my DShield reporting client for FreeBSD.
Okay, how is this different than previous patterns of hacking activity, other than the fact that they're aquiring compromised machines via a bot net?
You're sort of missing the point, I think, in that what's different about this pattern of activity is precisely the fact that it's being done with a botnet.
For one thing, there's a new level sophistication, primarily in that this bruteforce campaign is not the least bit random. I'm being hit by thousands of distinct attackers, yet the progression of usernames being attempted is undeniably alphabetical. Occasionally a particular username is attempted more than once, but it's typically sequential. One attempt per username with the attacking hosts only making one attempt every few hours.
The level of coordination required for this sort of attack is unprecedented. Across thousands of bots, each one at any given moment is able to determine:
In the past, brute force SSH attacks have always been obvious. Typical hit and runs. One host will spew hundreds or thousands of attempts at a target, typically in quick succession, typically focusing on system accounts, and typically trying a shitload of passwords against each account. Firewalls and IDS deployments far and wide will now easily detect (and often block) these attacks immediately because they're so easy to recognize.
This attack is very different. It's not targeting system accounts, it's hoping to get lucky against a vast list of potential userland lognames. It's only trying once or maybe twice per account. And it's distributing these attempts, round-robin style, across an impressive number of sources, with enough logic so that bot B will not attack host H unless all other bots in the network have sequentially exhausted their "token" attempt on host H.
What we're seeing is flying under the radar of a shit-ton of IDS/firewall implementations, and is harder to fight.
I would love to get my hands on the C&C database being used to coordinate all of this. Much as I hate to admit it, the architecture of this attack is unique and innovative, and I'd like to see what makes it tick.