Slashdot Mirror
Crazy Firewall Log Activity — What Does It Mean?
arkowitz writes "I happened to have access to five days worth of firewall logs from a US state government agency. I wrote a parser to grab unique IPs out, and sent several million of them to a company called Quova, who gave me back full location info on every 40th one. I then used Green Phosphor's Glasshouse visualization tool to have a look at the count of inbound packets, grouped by country of origin and hour. And it's freaking crazy looking. So I made the video of it and I'm asking the Slashdot community: What the heck is going on?"
Comment removed based on user account deletion
That's what I thought it was for. Srsly, they're your firewall logs. You should have some clue where inbound traffic is coming from and why. If you've got a webserver serving some sort of information that changes, this could be rss readers hitting your site. Or it could be pings of death being dropped by your firewall. It could be web surfers getting to work and hitting you up for information, or browsers grabbing some active information on your site. It could be googlebots. It could be slashdot hits for all I know. These are just theories, because this isn't my firewall or my traffic.
It's a little wrong to say a tomato is a vegetable. It's a lot wrong to say it's a suspension bridge.
Anyone else tempted to hum the theme tune to 2001 when they looked at that?
And also... "oh my god... it's full of stars"
Is this post an advertisement for Quova or Green Phosphor's Glasshouse?
I'm actually a lot more interested in the vertical stripes than the horizontal ones. It looks like at certain times, every country in the world sends a packet . .
Three days from now?? Thats tomorrow!! ~Peter Griffin
RTFV: this is one of the more interesting problems ive seen posted in years.... Especially as a China resident... Odd... Thought /. community?
"Does this mean anything?"
---- The real Slashdot is still here. You just have to browse at -1 to read the comments.
Somebody who doesn't forgets Poland.
(even if traffic from there wasn't unusual in any way)
One that hath name thou can not otter
Wait, is this just an advertisement for Glasshouse? The voice in the video on Green Phosphor's website is exactly the same.
What gives?
It's pretty interesting. You can see the countries with the largest botnets in the log... which also seems to suggest that a large majority of the packets are coming from the one botnet... since a good number of them kick in at the same time.
It also looks cool. Which is critical.
So there I was, scribbling down some notes off the PC screen by hand, when I reached for the keyboard and Ctrl-S'd.
The striping across all countries is a check whether your site is reachable from that part of the botnet, the purpose of the traffic is unclear; either to do a large data grab or it's a (very unsuccessful) bandwidth attack, or something. You should adjust it for number of internet connected users per country first then revisualize that.
Is this guy filtering out backscatter like DNS replication and time updates? If it's from a State agency it's entirely possible that are running a root DNS server on-site (I work st a State agency and we are). Also, what timezone is he in? Knowing that might help explain the spike at 21:00. Is that GMT? Need input!
I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
So you have access to these firewalls but you don't know how to go about diagnosing the problem aside from an Ask Slashdot? Am I the only one who's a little baffled by this?
Were you unaware that botnets spanned the globe, or that certain countries have a higher incidence of compromised systems? If you don't understand those things, maybe you should get someone else to manage your firewalls?
I would have to say that the countries of interest on the graph seem to be the countries of interest from a malware/hacking perspective. Perhaps it's bot net activity where there is a large amount of port scans that kickoff from all over the world and then some of the "increase" after the lines would be further recon activity. All very interesting.
It looks to me like the lines of major activity likely corresponded to major news events or other events that caused people to look at the relevant government agency. Without more data it is difficult to speculate. It might be possible to look at the approximate date (Early September of 2009) and find a specific event that would cause this. Indeed, it might then be possible to actually make a guess as to what government agency the firewall belonged.
Looking at the pop-up labels that show up when you mouse-over the data, there seems to be a huge temporal discontinuity in your data set: right at the first vertical stripe, the displayed date/time labels jump from 2009-09-17 to 2009-09-27. Maybe I'm just misreading the display, but a 10-day discontinuity would seem to account for the anomaly you describe.
It couldn't be that easy, could it?
Yes. Some context would be helpful, including what's behind the firewall, the kinds of traffic you think you're accepting, and public expectations of the services available.
Visualizing by port or protocol would be a great way to begin figure out what the traffic is.
Also, CERT and related may remember if any interesting 0-days were released just prior to the first band, etc.
There are 1.1... kinds of people.
Don't waste their time with this shameless advertisement.
Yes, he knows the firewall and the traffic. The question is - why is there suddenly traffic suddenly appearing from every country in the world at the same time? and again a number of hours later? And again 5 or 6 times? Suddenly there is inbound packets from every country in the world, for an hour or two, then it dies off. For some countries, the first 'stripe' is also the start of consistently higher traffic from that country. Does this mean anything?
I think it might be more useful to know the actual dates, and see if this corresponds with any spikes in spam or virus activity. What would be most useful would be know the dest port number of the inbound traffic, that could give us much better clues as to the reasons behind the patterns.
Ho! Haha! Guard! Turn! Parry! Dodge! Spin! Ha! Thrust!
it means that this is an ad for Quova and Green Phosphor's Glasshouse
Am I the only one who found the five minutes of this video to be about as interesting as listening to a stoned person describe the cracks on the ceiling?
You designed the visualization, buddy. If it's "freaking crazy looking," rather than yielding any useful insight, then obviously you did not visualize it in a meaningful way. You failed, in other words.
But as an earlier poster noted, this is just a Slashvertisement for the visualization tool in question. No doubt it will be quite effective on the kind of people who talk as slowly as the guy in the video.
Breakfast served all day!
First, we would need to know what kind of traffic we are seeing. TCP/UDP? Web? DNS?
On the other hand, I think you have only partial logs, that would explain many of the blanks on your data. Some blanks are too geometric to be correct, you are probably missing a shitload of data.
You have to take into account that, and timezones. Timezones are the key to this. This is probably some public service that gets hit at regular intervals (root DNS server, webserver holding news/stock/climate or similar information, etc). Timezones would explain the pattern. We would need to check times for each country against a timezone table to see if they correlate.
I'm also pretty sure that if someone took the time to look at the most active countries, and the less active countries, and some groups in between, we would be able to probably determine what kind of traffic this was.
Some people mentioned botnets, and it's a big chance that they have a huge influence on this graphs, again, matching timezones against this graph would help us understand.
I don't know what kind of information does the submitter have on the logs, or how he got them, but if he could post at least a small sample, that would help a lot. /methinks that submitter has a lot to do with the tool he's using, and this is just another slashvertisement.
WTF am I doing replying to an AC at 5 A.M on a Friday night?
Web robots. Just put a robots.txt file in your web directory and that pretty much shuts it down.
Also take into account that China, Russia, et al are +12 from us So that might explain some of it. In other words, they might be caching your site.
So why is he using State property for personal gain? My guess is his logs for his website were way too boring.
Shouldn't there be some agency in Florida who does not want their logs posted, even in cartoon format, in an internet video. I'm guessing this is probably either the Florida Dept. of Revenue or the Florida Dept. of Financial Services.
*DrugCheese rants*
(this is a guess, obviously. Full netflow data would tell me more, but only way to be really sure would be a full packet trace)
This just shows that you're being scanned with random source IP adresses (that's why the vertical stripe lights up). It is essentially a check to see if part of the botnet has more firewall access than other parts, or if a loadbalancer directs stuff to different firewalls, or if you have additional BGP uplinks, some of which might not be quite as secure.
Then the real scan starts, which uses the information gained in the first phase to make sure it tests out all the firewalls the target network has. Especially in the case of backup bgp links, where traffic comes in on physically and administratively different lines (say 1 verizon, 1 at&t, if you've got money to burn, and most govt. idiots feel the need to burn money). If the company in addition to the multiple uplinks outsources firewalls to those ISPs (or "security", not knowing what they're buying and getting nothing more than a smug false sense of security), again this is done by too many govt. agencies, you are bound to find holes this way. This uses actual bandwidth, and cannot be done on some networks. So what you're seeing is a disproportionate amount of scanning traffic coming from countries with fast networks and few watchful netadmins (or netadmins that just don't care, in Turkey's case), and many unsecured computers (and dear God, Turks and Russians really do not see any need for virusscanners, but generally you'd see a few other countries in there too. Heh the Russians are probably worried that running a virusscanner will interfere with their development of new viruses)
The regular repeats of vertical lines are probably to rescan reachability information, in case something changed. BGP can be twitchy, especially with incompetent local admins (on the botnet side of the network I mean)
From the (low) speed of the attack you can further deduce that it was an advanced attack, meant to stay below rate limiters, and presumably meant to stay below the radar. And from the resources required to pull this off you can deduce that this was not a lone hacker. Perhaps an organization (these days, tracing source ip's for security attacks almost invariably yields an IP address in far inland China, which is not because the russians have stopped attacking networks, but the Chinese are putting quantity above quality it seems these days).
And frankly, if someone has this kind of patience, generally they will find at least something, even in a well maintained network. Best hope it was only some files left out in the "public" folder or ~username folders. It's a good bet they probed the network security in other ways too (esp. googling), with IP's that will tell you much more about where the attack is coming from (using many hops is possible, but results in very slow page loads. And we're all human)
Btw : looking up a net's country can be done quickly via dns, no need for external company, no need for any tax dollars :
[kimmy@t61 ~]$ host -t TXT 104.79.125.74.cc.iploc.org
104.79.125.74.cc.iploc.org descriptive text "US"
(don't forget to reverse the IP address : looking up 1.2.3.4 is done by host -t TXT 4.3.2.1.cc.iploc.org)
My guess is that it's a bot net becoming active.
The countries with higher traffic during that period are countries that are widely known to have high bot net activity they are also more likely to have server bot net activity, which is why they don't stripe like the over countries.
The stripes are likely day/night where infected PCs are turned off when not in use.
...and that is all I have to say about that.
http://jessta.id.au
So 300p/hr = 12000p/hr?
How many more years will slashdot have an off-by-one error on your Score in your profile?
Not really. It makes green phosphor look like laggy shareware. Somthing with no effort spent on beautifying the interface and even less effort spent on cheating enough to make it visually smooth.
It made me think, "that's a really cool idea. If I had to do that kind of visualization (large dataset over two independent variables), I'd definitely be interested in something like that. But done well, instead."
Can you be Even More Awesome?!
You're trying imagine shapes in clouds, there is no context. Video conference call, maybe? Also, could be synchronization, or backups. Spooky garbage for the tin foil hat crowd, I hear theres a good business in it these days. It's an ad for a 3D graphing service.
The force that blew the Big Bang continues to accelerate.
Where are mod points when you need them? Parent desperately needs to be modded something OTHER than informative.
Over the past week I've had the following countries hitting my ssh:
108 location: RO
121 location: CZ
122 location: HU
133 location: AU
142 location: HK
143 location: MX
145 location: BR
151 location: TH
152 location: CO
158 location: IN
183 location: MU
184 location: NL
191 location: ES
205 location: ININ
234 location: JP
252 location: FR
270 location: CA
306 location: PL
313 location: GB
314 location: TW
355 location: CNCN
364 location: IT
379 location: RU
399 location: KR
632 location: DE
1361 location: CN
there's absolutely no context given at all here. and the fact that ips are coming from different countries could simply mean that proxies are being used in those countries. you say you work for the government?
~don't feel threatened by my pineal~
"I happened to have access to five days worth of firewall logs from a US state government agency..."
"While skimming through my grandmother's cookbook, I stumbled upon a recipe for processing yellowcake uranium..."
"In passing, a close personal friend mentioned to me that he would deploy ~30k troops to a Mideastern country, but he's worried that the local restaurantuers won't serve fresh babaganoush ..."
"While I was talking to a famous adult film star about my successful experiment with cold fusion..."
"I was fighting against an alien invasion of the Soviet Union the other day. Natalie Portman and I prepared a platoon of sharks with frickin' hotgrits cannons on their heads, but the unwelcome overlords kept jumping the sharks..."
If that's not the voice of Peter Gibbons from Office Space, then slap me silly!
"...Well, I generally come in at least fifteen minutes late, ah, I use the side door - that way Lumbergh can't see me - and, uh, after that I just sorta space out for about an hour and visualized activity by hour and country. I... took a bunch of the IP's from the logs, sent them to a company called Initech; Initech took every... (sent millions of them) Initech took every 40th one and sent them to Lumberg's house."
Exactly. This guy is advertising his own not-very-creative service.
Sure - he just happens to have access to the US State Deapartment logs, but isn't smart enough to look at the packets?
Astroturf.
GenerateCrazyFirewallActivity( struct in_addr dest[NUM_TARGETS], int hour, int minute ) { .... } ;
int i,SpoofPackets[NUM_COUNTRIES][HOURS_OF_THE_DAY]
= { { 10, 17} ,
for(j=0;j<NUM_TARGETS;j++) for(i=0;i<NUM_COUNTRIES) { count=SpoofPackets[j][hour] * random_fraction() + (confuse_the_hell_out_of_them ? 100 : 0); SpoofPacketsTo(dest[i],count) }
}
I'd guess you are seeing a bot-net attack. The bot-net army would have the greatest numbers in IT-heavy countries (US, India, China). The command structure would cause them all to attack at (roughly) the same time, regardless of time zone.
Or maybe you've been slashdotted.
Place nail here >+
Is no-one else bothered by the fact he has access to raw logs from a government system? Are there no privacy concerns from a private citizen being allowed to scan for users of government system? For instance, let's imagine it's the local IRS server - he now knows exactly what forms you were downloading, or perhaps visitors to a government site to help people find providers of mental health care. Really I don't care what the site was, it just seems like there's no valid reason for anyone to have raw data rather than aggregated data outside that department.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Where can one get a list of IP addresses for countries like China and India so that server admins like myself can block these countries entirely?
You want complaining? How about this: This visualization is terrible.
The video took five minutes to watch and most of it was him rolling over the bars in the 3-D chart so you can see what each of the lines means. If that's supposed to be a useful visual aid, I'll eat my hat. It's bad enough that you have to manually roll over every data element to figure out what it is; scrolling through the graph seemed dead slow. I hope that's not a limitation of the product itself.
Simple labels on the axes of the graph would have been nice. Far be it from anyone to try stick little flags next to the lines to represent different countries. Hell, just color-coding them in a totally arbitrary way would have made the graph easier to read.
BTW, a quick look at the Glasshouse site reveals all their output looks pretty much just like this demo. And there's no evidence that you can export one of their rudimentary 3-D graphs to "pretty it up" in a real 3-D app. Instead, their raison d'être appears to be allowing you to run around looking at these graphs... in Second Life.
I'm sorry, but if you're doing something like plotting fractals, for example, where visual similarity to patterns is the whole point, I can forgive you for coming to the conclusion that "it's crazy looking." If what you're doing is trying to provide a visual to aid in the interpretation of data, then the visual should -- y'know -- aid interpretation. A glance at this graph, on the other hand, reveals nothing; not even what it's supposed to represent.
In summary, Edward Tufte will be rolling in his grave when he dies from looking at this graphic.
Breakfast served all day!
This just doesn't seem like a big deal. The countries he points out are all in the same timezones so it's probably just their normal day starting. So this probably correlates to dns refresh or some other aspect (vertical) of general internet operations landing on the same hour.
He needs tcp port analysis and to compare days - the pattern is probably the same from day to day.
You are checking your backups, aren't you?
Video conference calls do not last for hours or days. And why would somebody in China or Romania be "backing up" data from a state government website?
Nothing really "interesting". What you notice is that around 9:00PM a bunch of East Asian countries start to show some spiked traffic. My guess is botnets on computers that are being turned on during the day generating a lot of traffic data. Or just computers coming on in general, for anything. There's no context as to what data they were requesting, it could have been simple search hits or image hits, or link hits in google or whatever else. But what it shows to me is nothing more than "hey look, the eastern half of the world wakes up when it's evening time in the US."
I normally I'd love this sort of thing. I pour over logs in my spare time - for kicks even, but this video just bored me. For nearly half the video this thing never goes beyond "look! people in different countries are active at different times!".
Even the few things that almost start to seem interesting leave you unable to gain any insight because there is just no information. There isn't any useful data to work with.
What this fails to provide us with is what kind of traffic this was in the first place. Any reasonably large site is going to get hit with all kinds of background noise, and so the fact that they found themselves with large amounts of "traffic" from 'nearly every country' doesn't surprise me.
This seems to be nothing more than an example of a very dull and uninformative way to display a large collection something very very common.
It's an ad for a 3D graphing service.
Indeed, the guy from the graphing service is the same guy who made this.
c++;
Maybe the fact that you put random chunks of data from days apart next to each other has something to do with it?
for a powerful client, but i need, you, random slashdork, to help me out here
no, i'm not a salesman
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
... you should hire someone who knows what they're doing, and/or quit acting like the kids from Jurassic Park. Pretty pictures alone don't tell the whole story in the real world.
Furries make the internet go.
I see no reason whatever that it would be necessary to use either Quova or Green Phosphor. Any competent programmer could have sampled the data, used whois to get location, and then used about 1000 different programs to visualize the data just as well. (Like Crystal Reports or Seagate.)
The fact that OP did neither, and is involved at a high level with one of the two companies, makes this whole post suspicious.
My best guess is that OP thought he had discovered a way to freely advertise via Slashdot, and victimized us as a result.
I get enough Spam. I don't need to see even more, on Slashdot. Can this user be blocked?
The vertical stripes, indicating worldwide activity at the same time, are probably the result of botnets being ordered to target an area that includes your IP pool. (or possibly, specifically your organization - depending on where you got the logs this may be more or less likely) The horizontal stripes are of course showing continuous activity from specific regions, which can indicate activity of a regional botnet doing general penetration scans looking for more machines to infect. For example, botnets that tend to post their driveby installer on russian web pages will be primarily comprised of participants from russia or other russian-speaking countries
You should also consider the sensitivity of the graph. Only having two axis is unhelpful. Could for example, one high bandwidth box at a single IP doing an intensive DDoS or password brute force on you be responsible for any of the horizontal lines? (in which case the graph is only showing number of connections, not number of UNIQUE IP connections) From that graph alone it's impossible to say if the attacks are distributed or simply high bandwidth solo, which can lead to different conclusions. A single compromised akami server could similar to a minor botnet on that graph.
You'd be advised to take a horizontal or vertical slice you are interested in and examine it alone, creating a new 2d graph with other information on the other axis. More patterns are bound to develop and you can further regraph with new information until clear patterns stop, and then you can consider the patterns you've identified as a group.
I work for the Department of Redundancy Department.
Better colors, please. Perhaps labels for the axes, as well.
Pretty sure I'd lose my job for posting this kind of stuff on slashdot....
||| I still can't believe Parkay's not butter.
p.s. The comments on you tube are funnier than here.
Uh...a bot net?
That would explain most of it.
Also is he plotting this based on potentially spoofed IP addresses? I'm thinking not just a botnet, but a botnet that doesn't care if it's getting packets back or not. It may not be every country in the world, just a bunch of random IPs coming from zombies which may (or may not) be in far-flung places.
Mal-2
How is the Riemann zeta function like Trump rallies? Both have an endless number of trivial zeros.
while you were processing the numbers? did you use Microsoft(TM) Windows(R) Moviemaker(C) to make the Youtube(TM) video?
--
Stay tuned for some shock and awe coming right up after this messages!
It does seem like a type of coordination of interest in the site possibly a bot-net but it could also be due to press releases or other media publications since it is a gov site. You would have to look over many days and not just hours to come up with something conclusive but it is none the less interesting that every country even those in different time zones accessed at the same time and it is odd that the Chinese are interested that much in a US gov site at the same time but I digress. Overall more information is needed and over a longer time frame to make any real conclusions.
A loop, by its nature, continues. If that didn't make sense, start reading this sentence again.
Looks like a botnet to me yo. What the fella doesn't explain is if that activity was inbound or outbound.
Yes, he knows the firewall and the traffic. The question is - why is there suddenly traffic suddenly appearing from every country in the world at the same time? and again a number of hours later? And again 5 or 6 times?
I get a lot of distributed dictionary attacks like that. Its pretty normal.
http://michaelsmith.id.au
"Whaddayou think *this* is?"
"It's something like a man's penis, only smaller."
(Spider Robinson, "Fivesight", _Time Travellers Strictly Cash_, no commercial association)
Don't you mean WTFV?
Infuriate left and right
Conference calls, backups, and synchronization from damn near every country on earth? For an agency within a single US state? No.
Also, too: The packet rates are far too low for those activities. If you watch TFV, you'll see that the largest users are only up to around a couple hundred packets per hour, which is such small number that even if you multiply it by 40 (due to the scaling done by the geo-IP service[1]), it's still far too small for those activities that you listed.
Any other theories?
[1]: It's not clear, to me, if we should be looking at these packet counts as they're shown in TFV, or multiplying them to account for the selection performed on the data.
Kid-proof tablet..
Maybe all the bots are part of the same botnet and were programmed to attack at the first spike.
The fact they are located in different countries doesn't mean anything, it's simply hiding whoever is really behind the attack.
I don't know the meaning of the word 'don't' - J
when something happens all ovr the world at the same time in DC, it is likely a zombie computer network hitting EVERYTHING. The countries with activity rising in general after the first blast probably indicates that the zombies in those countries are successful, and are increasing their attacks.
Shoes for Industry. Shoes for the Dead.
Nice visualization. Wonder if there is some way to do it in real time.
I've done networking and security for a university for the last 10 years. I can guess what this kind of activity would be if it was at my institution. Basically, there are several reasons why every country in the world will suddenly talk to us. They include P2P/Gnutella's, P2P/Swarmcasting, Bittorrent, Skype, P2P-poisoning, P2P-misdirection, and hacker/bot activity.
When we have pulses like you are observing, it is usually BitTorrent.
The Gnutella P2P variants don't usually have that many peers. And, they tend to last for several hours or days.
The various Swarmcasting P2P variants look very similiar to BitTorrent, but again, the users tend to leave them running for hours or days.
A popular Torrent makes connections to hundreds of locations at once, and usually the local user shuts down in minutes (or an hour) when they get their file.
Skype won't be narrow bands. It will be every country in the world talking to you all the time. We have had computers promote themselves up the Skype infrastructure until they are constantly talking to over 600K peers. Of course, it is more normal to see a Skype node talking to 10K to 20K peers, but still Skype won't be bands. Skype raises the floor for the entire graph.
P2P-poisoning would closely match your bands. For several years we observed pulses where every member of a large P2P cloud would attempt to talk to a non-existing IP at our institution. Eventually, we realized that somebody was attempting to render the P2P cloud non-functional by poisoning the P2P community with info on non-existing peers. Of course, since this is a Denial of Service (DoS) attack, this is technically illegal, but we saw it happening for years. But, it appeared to stop a couple years ago (about the time Obama replaced Bush) and we haven't seen any evidence of it lately.
P2P-misdirection is where a cloud will attempt to confuse traffic analysis by throwing out random connections/packets to random IPs. Typically, this misdirection happens all the time, and not in bursts/bands.
Bot attack activity doesn't match your patterns either. We observe several types. None would look like your bands:
- The spoofed attacks will look like every one of your IPs getting acks from a few remote IPs.
- The mapping activity will look like a representative sample of your IPs getting traffic from a few dozen IPs.
- An incoming DoS would have a few of your IPs get (spoofed) traffic from everywhere, but it would be sustained.
- Portscans will only involve a handful of remote IPs.
- The Tag-team SSH password guessing is close. During the last week, we observed about 3000 sources located all over. But, it happens all the time (in the aggregrate), not in bursts. And the sources this week are concentrated in Italy, Poland, Eastern Europe, Colombia, and Brazil. They aren't really all over the world.
So, I'm guessing it is BitTorrent. But, your situation may be way different from mine.
Miles
Vertical stripes may be from spoofed addresses -- nothing from real sources, even botnets, can be that uniform across the whole address space. It would make sense to check how much of traffic comes from unallocated address space, as packets from there are guaranteed to be spoofed. Why would anyone do such a thing? As a direct portscan it would be useless (he can't see the responses), however it might be used as a smokescreen to hide a real portscan or attack from some of those addresses. It may even be an attack that floods the DNS servers with fake responses in the attempt to poison DNS cache, thus redirecting some of the traffic to the attackers' addresses.
Then, after whatever kind of discovery was completed, you have seen some targeted host scans, [D]DoS attempts or actual exploits causing large amount of traffic (horizontal stripes).
Another possibility is that those packets are responses caused by something on your network being coerced into sending packets uniformly to the whole address space. It may be something as stupid as a web page with random redirects, however more likely it is a worm on some of your computers looking for other members of his botnet. After such discovery some hosts joined the botnet[s], producing horizontal stripes composed of traffic from other botnet members.
Contrary to the popular belief, there indeed is no God.
It means that some IP spaces are used more heavily, and that if you don't care about getting a response (hello, UDP) then you can make your traffic come from anywhere.
The real question is - if you don't know what this means, why in gods name did the US Gov hand over the logs to you??
I want to delete my account but Slashdot doesn't allow it.
Just a guess. Stripes are caused by either a redundant firewall arrangement, or redundant feeds to the internet, where the load balancer (or telco) is moving traffic one way or the other.. Simple enough to reverse out your own IP..no need to outsource that part. Lastly, you are working with a small sample size, (no more than a few hundreds of packets per line) so any small change appears larger.
and sent
several million
of them to a company called Quova, who gave me back full location info on every 40th one.
Well, there you have it. Unless you can prove, that that filtering that Quova does, does not influence your results, you can’t really draw any information from it. Could just be selectivity, applied by Quova. Or a otherwise bad filter.
Only if you are safe in that regard, would you first have to look at the actual outgoing traffic, in case there are correlations. (Which, considering the data, seems very likely.)
Any sufficiently advanced intelligence is indistinguishable from stupidity.
"Video conference calls do not last for hours or days."
Maybe not in your world, but then again it's likely you've never been in a Camfrog room. Also, on Skype, my UK and AUS partners and I just leave the conversation going. If any of us are near the computer and hear the others, we'll speak up and start a conversation. It's much simpler. Our machines are all located in our in-home offices.
I usually leave Camfrog and Skype open and connected 24/7. It's just much simpler that way.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
It appears that the big countries, like china, and india shows up with more hits than the small countires like angola and cuba.
I wonder what that can mean? Is it similar to the statistical fact that most truck accidents happen in US made trucks?
In the latter, until you factor in that 95% of US trucks are made in the US, you have only meaningless statistics.
It seems that current incarnation of this analysis tool suffers the same flaw.
don't cut it off www.mgmbill.org
Computers are used by people. People who wake up, work, play, sleep, have weekends, business holidays, religious holidays, events and a pantheon of other reasons why they might act in seeming semi-concert.
You're suggesting that for the five day period in question, the majority of people work up at the same time GMT? Not 7am local time, but 9pm GMT everywhere in the world? Or did you just not actually look at the video (which shows spikes of data from every country in the world at the same time)? "Timezone effects" should eliminate these sorts of lines, not cause them, by spreading that kind of activity out over 24 hours.
"Convictions are more dangerous enemies of truth than lies."
I think it's safe to say the traffic spikes, especially from asia and the south pacific, is due to the tsunami that hit. http://en.wikipedia.org/wiki/2009_Samoa_earthquake
It's elementary my dear Watson. P2P. Someone's firing up Bittorrent (hence, every country in the world with long streams to those actually grabbing data).
This is the guy whose product we're talking about. He wants to explain himself. If you think he tried to use Slashdot to advertise his product, you don't have to mod him up, but if you mod him down to -1 then he'll drop below a lot of people's thresholds and they won't even see that he tried to participate. That's not being fair.
Breakfast served all day!
No, I'm suggesting that people who work across timezones are aware of other people's schedules and organise their own to coordinate.
I work regularly with a group in Arizona and another in Shanghai. I know what times they get to work, what times they leave and I plan my activities to work with that. They do the same for my GMT +8 timezone.
It is an increasingly common mode of work.
"I've got more toys than Teruhisa Kitahara."
No, I'm suggesting that people who work across timezones are aware of other people's schedules and organise their own to coordinate.
Actually, that'd be the opposite of what you were previously suggesting. You're replacing "a seeming semi-concert" caused by people tending to do the same thing at the same time of day with actual, intentional coordination.
"Convictions are more dangerous enemies of truth than lies."
Everyone wants to think botnet but I have a second theory that's kinda out there. Maybe it's Google's international search engine robot servers all hitting it on a schedule to update their indexes. Why they'd be that synchronized I don't know for sure though. Maybe the google master controls decide it's time to update that website's index and it doesn't want the UK's google to show something 2 days later than the US version or people might find out and bitch about second rate service for their country so they signal all the robot servers everywhere to hit it at once.
Google's Super Secret Search Algorithm: SELECT @search_results FROM internet WHERE @search_results = 'good'
And this would explain hits from every country in the world at the same time on the same day?
IMO looks like botnet activity.
Literalism isn't a form of humor, it's you being irritating.
Some people also work at night and browse at odd hours you know. Just because it doesn't coincide with your personal ratrace schedule means nothing.
I have lived in 6 countries and it's readily noticeable to anyone that people keep different schedules in different places. Don't try to apply your limited little world on everyone else.
Um, that was my point. Try some reading comprehension. The facts you've just stated should spread the traffic out, and thus do not explain worldwide time-coordinated spikes.
"Convictions are more dangerous enemies of truth than lies."
People getting to work in US Mountain Time Zone IS "a seeming semi-concert". Do you think they all get together and plan their commutes?
"I've got more toys than Teruhisa Kitahara."
He finds it surprising that people actually look at government websites?
I find it surprising too! :)
Does this mean anything?
It means the logs are incomplete
Bingo. My thoughts exactly.
Unless his gives up some more data, hard to tell for sure.
But, I agree, it sounds like someone is using their employer's (government)bandwidth to torrent. Could be a machine that someone shuts off the monitor on but P2P downloads overnight with a scheduled P2P app.
The peaks/valleys might be explained by reset packets introduced by the ISP temporarily killing the outbound requests and it takes the inbound requests awhile to trickle off.
You can see this same type of log traffic by simply starting a torrent, waiting a little bit, then stopping the P2P client, waiting awhile again, then restarting it. Rinse, repeat and you will see something that looks awfully close to what you have.
Reset packets essentially create the same traffic pattern, but for a different reason (ISP- introduced traffic "shaping").
While this looks interesting, there isn't enough information to determine the cause. What were the packets?
It could be DNS requests based on the expiration of a domain on the stripes. The content on the website may be expiring at particular times. Someone may be posting on blogs, or tweeting with a link to the page.
Simply put, without knowing what the content is, and filtering out "explainable" traffic, then looking at the result, any pattern is just an interesting curiosity, nothing more.
I'm more thinking that there is one or more machines behind his firewall that were in the process of sending spam or were infected by a bot of some type during the times where there was a cross country stripe, and that made his IP address visible to the world, and that in turn started probes back, or even that it was bot control attempts that then did show up as country specific stripes.
So I would suggest a check of all machines behind the firewall for virus infections as a first measure.
What can be seen in the log is just the symptom, not the cause.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
I would wager that if he was to look at outbound traffic at the same time as the inbound "stripes" he would indeed find a correlation. For example, if you ping some IP address it should send you back a packet of data. Perhaps those strips aren't so representative of everyone else all of a sudden looking at the site but the site looking at everyone else and getting some kind of answer back?
I'm no sys-admin, but it's a logical hypothesis.
Tamran
Ahah. So you are why my costs for bandwidth are so high.
Yeah he does. All the plotted traffic is inbound. And yeah, botnet seems the most likely explanation.
Strange how they don't appear in the first half of the graph though. I didn't know that botnets took the weekends off.
Ho! Haha! Guard! Turn! Parry! Dodge! Spin! Ha! Thrust!
Not sure what it means, but I'm tempted to plug-in Guitar Hero and jam along to your firewall logs.
Just let me finish my Klax game first.
Piss-poor ad though. How many people saw the video and thought "I must get me some of this graphing tool!"? My first thought was "interesting way of presenting information, but his graphing tool is crap".
Ho! Haha! Guard! Turn! Parry! Dodge! Spin! Ha! Thrust!
I'm thinking that the strips are a news item relevant to the agency running on a world-wide news channel (BBC/CNN/Al Jazeera) and then when a local media picks up the story.
I would say that they have to be. He didn't take 7.5 packets from his 1/40th filtered data and multiply by 40 to get 300, did he?
If so, it's pretty stupid of him to not scale the numbers for presentation or at least mention that "Oh yeah, that 300 really means probably about 12,000..."
And not doing the geolocation on all of the data himself - WTF?
Yes they do. They plan to all get in my way. It's a vast government conspiracy to have everyone in Denver go to work at 8am and leave at 5pm. Well, they're in my way on my way from work, as I'm one of those opposite-hours people.
It's a little wrong to say a tomato is a vegetable. It's a lot wrong to say it's a suspension bridge.
Why is the ad "piss-poor"? If it's on a technical basis, yeah it didn't convey much in the way of information, but it didn't sound like he was really trying to convey much. Which is a different reason to be displeased with him. And if you're criticizing it on an artistic basis...well, I don't see why everything in front of our eyes must be smoothly rendered with swooping camera angles that cause some kind of visual orgasm when you see it. It was perfectly sufficient for me in that regard.
Unity? Screw that: XFCE. Slashdot Beta? Screw that: SoylentNews. Australis? Screw that: Pale Moon. UX developers DIAF
Oh crap. Ad---I just got that. If he's trying to sell the renderer thing as a product, yes, it sucks. My apologies.
Unity? Screw that: XFCE. Slashdot Beta? Screw that: SoylentNews. Australis? Screw that: Pale Moon. UX developers DIAF
The graph is kind of misleading, its not actually to scale and its not showing the 5 days he claims in the youtube description. Go to around the 3:05 mark and watch the time stamp when he mouses over Romania. On the far right you can see an early date of 2009-09-15, as he scrolls to the right we can see a date of 2009-09-28 at the second stripe which is roughly in the middle of the graph, continuing on the far right hand side portion of the graph is dated 2009-09-30. The left hand side of the graph shows results over the span of 13 days and the right hand side taking up the same visual space only shows 2-3 days. Basically I just wasted 15 minutes looking over worthless data on a random youtube video that doesn't actually say anything.
And when did the first stripe occur, say, 0600 Monday local time?
My first thought. I bet the first day of this 5-day-period is saturday.
In hind sight, this MAY be very dangerous.
It all depends on what government agency you are talking about. If sensitive information is at stake, you could have serious problems.
A scheduled P2P application uploading the contents of a hard drive as a torrent would be a worst case scenario. Judging by the logs, it would seem that if this IS a P2P app, a LOT of people are interested in that torrent.
It wouldn't be that hard to script a drive imaging application to create an .iso of the drive then another script to periodically upload the newest image as a torrent.
Fun stuff.
Without a whole bunch of more information we are guessing in the dark:
Until we know more -- it is not worth spending time on this.
It's the times when the latest 'Hero', 'Chuck', 'Castle' etc version gets posted and the planet's consumers grab it simultaneously.
During weekends not much is posted.
"Hi! I've created this awesome freaky-looking visualisation tool! It's so fucking useless that even I, the author, can't actually determine any useful information from the output it shows me, so I have to go ask some random commenters on a website if they've got a single sodding clue what's going on.
Wanna buy it?"
Wul, way to go picky pants, now it's modded off topic and down. You were the classroom snitch back in school, weren't ya?
*Repent!Quit Your Job!Slack Off!The World Ends Tomorrow and You May Die!
DDOS bots got a new target. That would explain the simultaneous bombardment from different countries.
Patents Drive Free Software as Hurricanes Drive Construction Industry
My first impression was botnet too, but just IP info alone isn't enough to come to a conclusion. Give me port info and packet size too. His graph is enough to go "Huh. That's interesting" and then look into it further. That's it.
I'm not sure, I can't really parse the data mentally on a 2-D plane. Perhaps he should get one of the computers these guys use and cook up some 3D cityscape models.
Random Thoughts From A Diseased Mind (Not For Dummies)
A botnet attack? But then the activity shouldn't be concentrated by country, but spread around the world about evenly.
Or it could be that someone's seeding a torrent from behind the firewall. That would explain the suddenly starting continuous activity. It might also explain the concentration by country (language or timezone). It would help if the graph could be organized by such factors.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
No botnet attack, just bittorrent traffic, with the stripes being times when the client used inside the "US government agency" is uploading traffic data at the request of the tracker, and the high activity countries being those with good broadband connectivity and serving lots of packets :P
Yeah he does. All the plotted traffic is inbound. And yeah, botnet seems the most likely explanation.
Strange how they don't appear in the first half of the graph though. I didn't know that botnets took the weekends off.
botnets don't take the weekends off, but owners of infected machines are likely to turn off their computers over the weekend.
Only his tendency toward a dazed stupor prevented him from screaming aloud.
Look at the chart. LOOK. it's obvious what this means.
BSD is dying!
* Carthago Delenda Est *
My first thought was "why does everybody have to make everything a video?"
The graph is kind of misleading, its not actually to scale
I think the point is to show that time prior to a point of interest shows one behavior, and the time after that shows another. If he had only shown 2-3 days prior, it would have looked basically the same.
Yes, he is misleading in the video, but having extra data is forgivable.
Basically I just wasted 15 minutes looking over worthless data on a random youtube video that doesn't actually say anything.
The validity of the data has nothing to do with the rest of your post. Is 15 days worth of data suddenly less worthwhile than 5?
All my liberal friends think I'm a conservative, all my conservative friends think I'm a liberal.
Do these stripes have to do anything with your business hours....
Starting work say a 9:00 everyone logs in checks and sends their email out getting server responces will create a stripe. If other countries work with the organization they probably have their schedules match to the US time zones. (AKA working nights) So you get you start of the day stripe. After cofee break stripe, lunch stripe, etc...
The stripes are not really a big deal. I would pay more attention to the active countries
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
My intuition says the lines are caused by some type of botnet activity.
Only his tendency toward a dazed stupor prevented him from screaming aloud.
Some context would be helpful, including what's behind the firewall, the kinds of traffic you think you're accepting, and public expectations of the services available.
Exactly. Without context, we can merely speculate that the server is pwned or under attack. This conjecture could be assessed better if there were also logs of outbound traffic as well as inbound. Also, it's not clear if the packets were being rejected (attack resisted), or being passed through (active attack or already pwned).
The stripes in the inbound packets look rather like botnet c&c traffic, which is presumably distributed worldwide. There are not many other activities which would be synchronous worldwide. Traffic from specific countries rises and stays high for extended times after some of the stripes. This could be payload updates, or other nefarious activity. Was there outbound traffic to the same sites also?
What kind of services does this site provide? One would not expect the same traffic profile for a LOC or NASA web server as for a stratum 1 or 2 NTP server, for instance. Maybe the traffic pattern is all innocent, maybe not...
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
It's not the government, at least not my part of it.
Btw, did you enjoy The Truman Show?
"I've got more toys than Teruhisa Kitahara."
What I find a bit odd is that nobody has even thought to question what business the submitter has with 5 days' worth of server logs from a US state government agency.
E.T. Phone Home?
...what new sites or services started the day of the first stripe? Given that you did a every 40th sample this could potentially be a sampling error, or a moire pattern caused by said sampling error.
I'd go with a site or service coming online somewhere within the organization where the data came from. If it's a higher-education institution it could have been anything or anyone setting up a website, or it could even have been a trojan or virus that is now using a machine to tunnel through the firewall and share music, video or warez.
Happy Hunting!
>I would wager that if he was to look at outbound traffic at the same time as the inbound "stripes" he would indeed find a correlation.
For firewall logs related to procotols that the firewall treats as stateful, the log entries usually have a source ip and destination ip. A single log entry covers both the packets generated by the local system as well as the responses received from the remote system. For a single stateful session, there are not separate log entries for the "outbound" requests and the "inbound" replies. In your example with ping (ICMP), some firewalls treat a ping as a stateful connection and will log just one entry that covers the outbound echo request and the inbound echo reply.
Are you linked to the video professor
First, as many people have noted, these stripes could easily be due to events that have world-wide interest, and the spikes due to regional events. Without knowing the site involved there's not much point in speculating.
Second, if I was the admin at the unnamed site I'd be pissed that he'd disclosed firewall trace information to a data-mining company.
Third, not disclosing his relationship to the graphing company is pretty dodgy.
No, not really. It should be spread across vulnerable computers about evenly, which means it should be concentrated in large countries with significant technological infrastructure. China and India are both huge, and while a decent chunk of them are impoverished, I suspect they're still quite up there in terms of internet connected systems.
Some kind of distributed attack, possibly, or something completely benign. Look at the packets and see what they are. Looking at source alone isn't that revealing.
It would be clearer if we knew the destination port distribution of these "stripes".
If its all to port 80, it could be there's a web page with a refresh that updates every few hours, and people happen to have it up on their screens...
- "History shows again and again how nature points out the folly of men" -- Blue Oyster Cult, 'Godzilla'
By keeping this article up you are rewarding spammers and inviting more of it. When I receive spam, I mark and delete it. Don't you, Slashdot?
Second Life? Is that where you go after you die? Not everyone believes in reincarnation.
- James
Now that you have isolated a few categories of interesting packets, you should study samples of those interesting packets in more detail. Many have speculated about what the "stripes" mean, but you can find out more by investigating one stripe.
if it's Romania in there, it's BitTorrent.
Dugg for Romania!
oh wait...
Somewhere, Edward Tufte is rolling over in his bed.
How sure are we that the resulting data from this service is accurate? Is there a pattern between the times and resulting countries because they're mistakenly parsing the date/time of the log instead of the actual IP address? Or if they're only parsing every 40th entry maybe they're injecting bursts of "wrong" data as part of a trial?
I see no reason to jump to any conclusion as long as there may be doubt about the validity of the data you/we are looking at.
This comment does not necessarily represent the views and opinions of the author.
Glib.
Surely, you can type, dear one - defend your rating.
http://slashdot.org/my/journal
~hylas
advertising in the guise of an ask slashdot.. annoying.
first and only theory i'm buying here so far..
Calling out bogus battery capacity claims.
What's sad is I have access to about 50/50 (wireless bridging of my personal internet connection with my fiance's university account via the wireless signal that reaches our apartment) and I saturate it mostly with sending live research data/footage to those partners as well so they can watch how their money gets spent and used.
Imagine if I had a 100/100 connection. I'd be putting your bills thru the skyscraper antenna!
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
Right but this is only 300 packets or so from each country. How much data is 300 packets, potentially? 20KB? Bit torrent usually pushes an order of magnitude more data than that.
moox. for a new generation.
I would assume it's from global botnets running scripted attacks in unison at particular times.
A botnet attack? But then the activity shouldn't be concentrated by country, but spread around the world about evenly.
Or it could be that someone's seeding a torrent from behind the firewall. That would explain the suddenly starting continuous activity. It might also explain the concentration by country (language or timezone). It would help if the graph could be organized by such factors.
You people have lost your damn minds. The highest "peak" was ~350 packets PER HOUR. Not per second, but PER HOUR the narrator said. That's so low I'm guessing this probably is router-router traffic, BGP updates or similar. 350 packets isn't even a meg. It's slightly over 512KB assuming ethernet frame size of 1514 bytes. There is so little traffic described here it's not worth analyzing. Seriously. The guy who came up with this is a loser.
googlebot, msnbot, slurp, insert your favorite web crawler here.
Who knows, we don't have enough information. We need to analyze individual streams in both directions with port numbers and ideally even packet payload. We also need to know the firewall action (drop, allow, etc).
Not even close to enough information. Anyone who claims to have any idea what this means is grasping at straws.
Try using some other sample intervals and see if your patterns stay consistent. You might be aliasing.
I'd try 101 and 17 right off the bat, since prime numbers work best for detecting aliasing in my experience (I'm not a mathematician so my methods are empirical, I stole those numbers from bamboo and locusts respectively).
Those plaids may be an artifact of your sampling interval. The real patterns might even be more interesting!
The day someone invents a working greasemonkey script that lets me remove meta-whining from conversations I will throw a party.
Personally, I am interested enough in these data visualizations that I don't care if you are "advertising" your company and/or products.
I don't know if you read my earlier comment about aliasing, but the data filtering you used here (that removed the "plaid" effect) could easily be acting as a poor-man's anti-aliasing system.
Try using prime numbers in your sampling intervals. You might be surprised what happens. Most networks have broadcast traffic that hits at regular 60-second intervals (due to unimaginative default settings in commercial software and hardware) that introduces regular "pulses" into the data flows. In very large switched networks this can create amazing patterns as the switches dynamically fiddle with the broadcast traffic to optimize per-port throughput. In my experience no pattern is real unless it shows up using multiple sample intervals on the same traffic. Check out this video where Burton MacKenzie abuses the Nyquist limit.