Slashdot Mirror


What Kind Of Logs Should ISPs Keep?

Effugas asks: "An engineer at a rather large ISP recently asked me a rather simple question that I didn't have a particularly good answer for: What logs should they be storing? He wasn't asking about the simple question of whether their own servers should be watched closely--that's obvious. He was asking about his routing infrastructure. I told him they of course musn't record the actual data being routed through their network; however, endpoint to endpoint route logs(since the establishment of those routes is the ISP's raison d'etre) did seem viable. But now, I'm not so sure--if there's one thing we learned from Kenneth Starr's subpeona of Lewinsky's book purchase records, it's that Barnes and Noble stored such records in the first place! But on the flip side, I've certainly had friends be harassed and threatened online, and turning a blind eye to everything but attacks directly against the network doesn't seem right either. So I ask, without passing judgement in either direction: What options does a network administrator have for retaining forensic evidence in case of abuse, which ones are ethically justified, and what are the actual router configurations which implement such ethical systems?"

52 of 176 comments (clear)

  1. Everything by Ed+Avis · · Score: 2

    The ISP should feel free to log anything. Anyone who wants their data to be secure will be using https or ssh anyway.

    Okay, that might be a bit extreme, but it seems the only workable and enforceable policy. If you choose any other criterion, there will always be some unscrupulous ISPs which ignore it, and it gives people a false sense of security.

    --
    -- Ed Avis ed@membled.com
    1. Re:Everything by Spasemunki · · Score: 2

      But I think for the most part, one would not have the expectation that those would be kept secret. The phone company has the same information about the calls that you make. What is important then is to make sure that there are policies in place that determines who gets that information, and for what purpose. If you are going to keep any sort of logs or records of activity, who/where/when is probably one of the most basic ones, and I would expect most people know it will be logged. Just important to 1)make sure people know that sort of information is subject to logging 2)Clearly define the circumstances when the logs will be made available, and make sure your customers know that too.

      "Sweet creeping zombie Jesus!"

    2. Re:Everything by journeyman101 · · Score: 2

      I run a small ISP and I agree with most of the people here that trying to make ISPs log routing is a bit much. Do restaurant owners have to keep a log of what everyone of their customers eats everytime they visit? Does Walmart keep track of every product you buy when you are there? I don't think so. The logs I keep are to ensure everything is running correctly and to HELP my customers with problems...nothing more. My 2 cents worth

  2. You asked for it... by Trinition · · Score: 4
    Anyone, including ISPs, should always keep dry logs. The type of wood varies with personal preference. For example, Pine has a strong, well... piney odor -- especially when combusted.

    To help keep their logs dry, they should purchase a log rack, or simply arrange the logs on top of a makeshift support system so that the logs do not directly contact the ground.

    Moist logs tend to attract bugs and decompose much faster.

    1. Re:You asked for it... by Signal+11 · · Score: 2
      Pine leaves alot of sap residue in your fireplace. Honestly, I would recommend using dry oak as it burns dry and hot. You can also turn the dampener down quite abit and it will happily burn for hours and hours. The only wood I wouldn't recommend is poplar. Up here in Minnesota, that wood is very common, however it burns very crappy. Just avoid it, trust me.

    2. Re:You asked for it... by Surak · · Score: 2

      A couple of four-by-four posts work pretty well if you stack the logs right .... :)

  3. logging should be kept down by Anonymous Coward · · Score: 4

    Logging in this country has gone way too far and is an abuse that cannot be permitted to progress any further. Not only does abuse of this cost many of us what they view as their birthright, but it also scares the hell out of those who haven't lost anything due to it yet. Sure, there are definitely some political and corporate interests who benefit by letting this sort of thing run rampant, but can we really afford it? And who's this world for, anyway -- the corporations or the people?

    And when this does spiral out of control, efforts to redress the wrongs that have been committed, no matter how good-intentioned or extensive, will never fully wipe out the harm that has been caused within the lifetimes of those who have really been hurt most. Once you go too far, you can never truly come back.

    So I would definitely urge keeping logging to an absolute minimum if you can't eliminate it entirely. If you can't really appreciate the wisdom of not logging, I strongly urge you to take a hike.

    And then, after you come back from your tromp through tree-lined trails, to reconsider. :)

  4. Time Frame by Dungeon+Dweller · · Score: 3

    It's really not a question of what logs to keep, but how long to keep them. You should keep logs of requests, attacks, e-mails, routing information, anything that you might actually need, but only keep them for the appropriate period of time. You really don't want to have to dedicate a tape changer to this anyway, do you?

    E-mail, routing information, and the like, should have a relatively short lifespan, if a person is being harassed, they should report it quickly. You should allow them a week or 2 for turnaround in such cases, and burn the necessary information to a CD or other storage media for any followup needed, when there is a report. You shouldn't, however, keep a long papertrail on your users, this only invades their privacy. If there is a legitimate need for such logs, it will arise relatively quickly.

    Attack logs should be kept longer. All attack logs should be analyzed and damage should be evaluated. Appropriate individuals should be informed of the attack based on what has been compromised. Even these, however, should be trashed after a period of time. Do you really care about an unsuccessful attack 2 years ago? Probably not, you might, however, care about someone who root-kitted your server a year ago, since they probably still have the passwords of at least a few of your users.

    --
    Eh...
  5. Crypto by Signal+11 · · Score: 5
    That depends. You could get in trouble for taking this advice, depending on what form of tyranny^H^H^H^Hgovernment you happen to live under...

    Personally, I would encrypt them all using public-private key crypto. The "public" key is what is used to feed the data into syslog, and the private key can be used to decrypt it if you need it. If your systems are physically or otherwise compromised, the attacker still cannot derive the private key as long as you maintain due diligence in maintaining the security of the logging host(s). This means you can log everything to your hearts content and not worry about privacy concerns, as much. Just make sure to put the standard disclaimers in your AUP.

    I suspect, however, that wasn't quite the answer you were looking for. Honestly, in order to compromise most people's privacy requires an ungodly large harddrive to store all that information. Simply monitoring a T1 with a packetsniffer doing decent filtering can easily trash a fast 30GB HDD. The security industry is replete with stories of how crackers were caught because their packet sniffers went amok trying to log everything, and crashed the system trying.

    I'd recommend logging the source and destination of mail, and when it was retrieved. If you are using RADIUS servers, log the times they signed on and off, and keep the system clock religiously on-time. Have the facilities to monitor each user (ie, be familiar with how to use a packetsniffer, and have a box on standby if you need to use it). A quick cheat would be to configure the RADIUS server to tell $SUSPECT connection to only use $MONITORED_IP and then tell the packetsniffer to dump everything from $MONITORED_IP to disk. It's simple, but it works.

    As far as advice on law enforcement.. it depends on your situation. If you have been compromised, it still may do you more harm than good to report it due to the administrative overhead involved in prosecuting them. Generally, however, they are quite helpful on getting you the information you need to prosecute. Don't expect them to get too involved though unless your SMTP logs say that a message was sent from l335h4x0r@yourisp.com to president@whitehouse.gov with a subject line mentioning what he's going to do with a box of cigars and a can of surgical lubricant. In that case, you probably won't have any choice but to cooperate. :)

    Hope this helps,

    1. Re:Crypto by WNight · · Score: 4

      You don't use the public key to encrypt the logs, you randomly generate a session key, encrypt the logs with that, then use the public key to encrypt the session key. Rotate keys every few minutes and don't leave them sitting around. Ditto with the logs. Have a seperate machine which only accepts one incoming connection, that which dumps logs onto it. Then the log holding machine has *no* idea of the way the log was encrypted, nor, if the logs are removed (via console) to another machine, preferably laptop, for examination, would it know how to decrypt the logs.

      Because the public key only encrypts a 128 bit (or whatever) session key every ten minutes or so, it's fairly quick, and two-way crypto is very quick, easily enough to dump logs through.

      If you ever implement a log system and don't want them modified, keep an ID # in each packet of logs, along with a MD5 hash of the previous packet of logs (including the previous-packet hash of the log file before it.) This way if a log is modified, the attacked has to change all logs after that point.

      Ideally you'd also have the log catcher dumping logs to a write-only media, like CDs. Preferably in a session-based way, so it didn't have to wait too long between getting logs and writing them.

    2. Re:Crypto by WNight · · Score: 2

      The idea behind the crypto is that the logs, once public key encrypted, couldn't be decrypted, to gain information that could be used in further cracking attempts, or to lessen anyone's privacy.

      Simple logs, based on multiple users, such as bandwidth usage, number of connections to various services, etc, should all be plaintext to make them easier to use. But logs of individual connections, when someone picked up email, what sites people went to, MD5 hashes of outgoing mail, etc, shouldn't be plaintext.

      And the specifics of many of these logs would be unimportant, you rarely need to prove a user did or didn't mail something, so if it sits encrypted on a CD for a year, and then takes ten minutes to decrypt and view, no big deal. Much better that it can't be easily accessed by someone unauthorized.

      Of course, if your random number generation was flawed, all of your session keys could be compromised, but ideally you'd use fairly strong methods. And yeah, there are swap issues and all to deal with, but I'd left the details to the user's discretion.

      And as for the trashing, that's why you'd offload them to CD or hardcopy frequently. Perhaps you'd dump the MD5 hash to a lineprinter every time a log bundle came in and dump it to a CD every time you got a few MBs... (Depending if you can write multisession CDs.)

      But, to summarize. Not only are logs for catching bad guys, but they're also private info, which if you collect sensitive stuff, needs to be guarded properly.

  6. Elm by Dungeon+Dweller · · Score: 2

    I think that ELM has a much longer burn time, and a stronger, prouder smell, that reminds me of my heritage.

    --
    Eh...
  7. Logging by Syberghost · · Score: 4

    Assume they log everything, for purposes of guaranteeing your own privacy.

    Assume they log nothing, for purposes of maintaining your own documentation.

    Because the fact is, they probably don't log what you need them to log, and log all sorts of crap you wouldn't want them to.

    What they should log, IMHO, is everything they can, but only keep it for a couple of weeks.

    Having made use of everything from error logs to snooped IRC traffic to bust intruders on my systems, I recognize both the value of such logs, and the potential for abuse.

    --

  8. it's subjective by sethgecko · · Score: 2
    just like anything else in life, it's a subjective question. different ISP will have different needs.

    What it really comes down to, IMHO, is that information itself is rarely bad. Having information is neither good nor bad in itself.

    Consider a widespread DDOS attack--in this case tracking down the origin is difficult enough, and having profuse logs would be a real plus not just for the ISP, but for the net at large.

    On the other hand, logging routing traffic which shows that users X,Y, and Z downloaded metallica songs which they did not own, thereby making it possible to prosecute and put them in jail for a long time would come under the heading of a Very Bad Thing.

    Notice that in each of these cases, having the data in itself is not bad--it depends entirely on what is done with it. The real question which should determine what logs should be kept is, how likely is it that this information will be abused?

    disclaimer: I don't think that people will really go to jail for downloading metallica MP3's-that was just an example to illustrate a point-that if the existence of logs in a given situation, in this case a police state situation, were this likely to be abused, it would be a consciencious netizens duty to come up with a convincing reason why logging was impossible. Something about the data bandwidth of (n-1)^10000 exceeding possible logging potential of network based systems under primary load conditions. Impossible to argue with that, now, isn't it?

    --
    Be ot or bot ne ot, taht is the nestquoi.
  9. Ethics and Pragmatics by werdna · · Score: 2

    In view of the competing interests and liabilities of the ISP, it is probably pragmatically necessary for the ISP to maintain as comprehensive a set of logs as possible.

    Whatever policy is adopted, a breach of ethics would not arise from the maintenance of logs, but rather from the failure to inform customers that such logs are being maintained. By informing the customers, each customer is on notice to take steps to assure the security of any information sent in the clear or over the wire.

  10. Keeping logs.... but what logs and why. by Vapula · · Score: 2
    Keeping logs of who connected when may be interresting (in order to help finding those who used their access to hack someone else)... but are useless after the phone company purged their log.

    At least one belgian ISP got his password file very often cracked. So, if you can't track the connexion via the phone company, the information is useless. Even more, accesses are often pirated using tools like Back Orifice and such. So the information of what user connected is useless by itself...

    Connexions made should definitively not be logged, for privacy and practical reasons. The people who do craking/pirating visit many web/ftp sites, connect to many machines each time they use internet. Those who only make 2 or 3 connexions are those who log on the net, connect to IRC and check their mail. Without forgetting about those web sites with so many ad-banners/counters/... that to visit one page, about 10 different IP are accessed !

    Bad formed packet could be logged in order to spot people trying DoS, spoofing and such. again, how long is the question. If you can't track the real people connecting, it's useless.

    Mail server use should also be tracked. but no mail content. (remind me of the FIDOnet time when many unscrupulous Sysops spend their time reading the mail going through their machine)

    For the rest, AFAIK, log files can be modified at will. So I can't see how they could be used as legal evidence. IMHO, they could only be used as a tool to spot problems. But nothing more. So I think that all what is not needed for such purpose should not be logged.

  11. Logging at our ISP... by tzanger · · Score: 3

    ... consists of tcplog and our RADIUS log.

    Essentially that gives us a list of which IPs are in communication with our own, and a list of who was on what IP at the time the communication occurred. We keep our RADIUS log for 6 months for billing purposes and dispute settlements over billing, and our tcplogs are kept for one month.

  12. Re:When the bacon comes a knockin by PigleT · · Score: 2

    Quite so. I wonder what interest the police have in seeing your webmail logs and IP#s and stuff, anyway. OTOH, I would maintain a log of TCP connections when/where and list of who was logged in when, that you could use to restrict it all down a bit. Letting the pragmatic work in favour of the "let it all be private" ethics...

    . o O ( I wonder which the most popular cookie in my ONE GIG of httpd ERROR logs was? )
    ~Tim
    --
    .|` Clouds cross the black moonlight,

    --
    ~Tim
    --
    .|` Clouds cross the black moonlight,
    Rushing on down to the circle of the turn
  13. Re:Logs.... by -brazil- · · Score: 3
    They shouldn't unless they're committing a crime.

    Yeah, suuure. Get yourself a copy of "1984", read it, and learn why total surveillance is a bad thing. The very existence of such data is a danger in itself, because it can be used to commit crimes, and you can never be sure who eventually gets hold of it.

    --

    The illegal we do immediately. The unconstitutional takes a little longer.
    --Henry Kissinger

  14. Re:Logs.... by Constantine · · Score: 2

    They should all be archived indefinitely.

    I think you will find that most states have document retention laws, which specify for how long you are able to keep certain kinds of documentation. Lawsuits have been lost because companies did not comply with these laws, i.e. kept logs/documents for too long.

    You might want to recheck the laws in your state before you start keeping stuff "indefinitely".

  15. Avioding faked emails. by KahunaBurger · · Score: 4
    But on the flip side, I've certainly had friends be harassed and threatened online, and turning a blind eye to everything but attacks directly against the network doesn't seem right either.

    What options does a network administrator have for retaining forensic evidence in case of abuse

    This also ties into the carnivore question about faked emails. I've gotten some harrassing emails and considered forwarding them back to the sys admin of the jerk in question. However, realisticly, I could send anything I wanted with FWD in the title, and without digital signatures, they wouldn't know if I was forwarding a real email or not. But what kind of logs could they keep that they could confirm the authenticity of a message without invading the privacy of the user?

    Now, bearing in mind that I don't do this for a living, wouldn't it be possible to set up a logging program that ran a metric on each message that came through, based on date, to and from and message content, that could not be reversed to actually produce that data, but would have an astronomically improbable chance of being reproduced by a fake message?

    That way, the logs kept, just looking at them (even by the ISP) would tell them nothing but how many messages had gone to and fro from the whole ISP. But if someone came to them with an "incriminating" or "harrassing" email they could (at their discretion or under warrent) confirm the authenticity of that message actually having been sent by their service. If each ISP used their own metrics and kept them private, it would be very difficult for anyone to fake email evidence. This would be useful for both law enforcement/people being harrassed and the innocent but framed.

    So, is this kind of log possible, and would it satisfy privacy advocates, since you couldn't even tell "how aften and when used" for any given user?

    -Kahuna Burger

    --
    ...will work for Chick tracts...
  16. UK RIP (snooping) Bill by martin · · Score: 2

    Says log everything and forward it to the police on request (well, over simplification of the truth but thats about what it amounts to).

    Alot of ISPs are _threatening_ to pull of the UK because of this.

  17. Perhaps a more interesting question.... by NRLax27 · · Score: 2
    Instead of commenting on what you should log, or as is mentioned in many of the other posts her, how long you should keep your logs, I'd like to pose another question. Who should have access to these logs. Obviously this is going to have something to do with the law in the area that the ISP is located. However, should the RIAA have access to logs in order to prosecute people who have downloaded MP3's? Should individual artists have the right to examine ISP logs to determine who downloaded their songs? Should these parties, in fact, be able to demand that an ISP keep logs of this information?

    While most Slashdotters will answer a resounding no to those questions, what happens when child pornography comes into play? Should a police officer, or the FBI even, be able to demand an ISP hand over their logs, and examine them for people who have downloaded child porn? (Not exactly the easiest search, but I suppose doable none the less).

    I think that determining who has access to the logs is perhaps even more important than determining what to log in the first place.

    ./configure
    make comment
    make post

  18. Nothing by dingbat_hp · · Score: 2

    The ISP should log nothing, out of their own self-interest. Anything they need to log, for their own purposes, should be destroyed after use.

    Although it may be useful to Starr to find Lewinsky's book buying history, it's not good press for Barnes & Noble to have the existence of this log disclosed. Similarly it's never going to be in the ISPs interests to be at risk of having logs subpoenaed. The only legally-secure defence against this is to not have the logs in the first place (and this may require a traceable and provabel process to show that any that did exists have been destroyed).

    1. Re:Nothing by Panda+Star · · Score: 2

      As someone who works at an ISP let me clarifiy somewhat about what logs we deem keepable...
      We keep logs of connections to our boxes, the ip number given to that connection, the login name of the person connecting, what they connected at and for how long. We do not log where they went after they connected just that they connected and got a good solid IP number for the machines dns resolution.

      This not only helps us with router and dailup box diagnoses incase of trouble, but it also allows us to better help a customer who is having connection problems because it shows, when, how, and how fast they connected, and how they disconnected, timeout or dropped carrier.

      IMHO this is really all the log needed kept as far as what our customers are doing internal system logs on the other hand we keep very extensible because of the high number of hack attempts that go on against larger ISP's.

      --
      WinLinux the Operating system for LLamas
  19. In the UK... by DrWiggy · · Score: 4

    The new Regulation of Investigatory Powers bill is due to be passed in the UK soon, which means that on request (i.e. a warrant issued by the Secretary of State is produced), an ISP must be able to intercept all traffic that a particular customer sends or receives. If you haven't got such a warrant when you intercept traffic coming from or destined to a UK citizen, then you are in breach of the Interception of Communications Act, and so you shouldn't be doing any logging at all.

    To be honest, I don't think the harm is in the logging - it's what is done with the logs. Disclosure to third parties is definitely illegal and unethical, but the use of this sort of data within an organisation can also be dubious. How much would your marketing department like to know about the 'real' (read 'secret') interests of all of your customers?

    I say you guys have got it pretty easy in the US, but at least we're now getting clear legislation (even if it is b0rked) saying what we can and can't do over here in the UK. To easily answer this question in the UK though, does require a few hours with a copy of the Data Protection Act, the Interception of Communications Act and the Regulation of Investigatory Powers bill. Even then, you're probably wrong.

    As far as we what we do is concerned (as an ISP) - we log enough for billing, and we have some machines running an IDS in promisc. mode to pick up scans, viruses, etc. going across the network. Apart from that, it's all pretty standard syslog-out-of-the-box.

    --

  20. Spammers/child porn/death threats/etc by muldrake · · Score: 3
    I think ISPs should keep point-to-point logs as well as log, say, an rc hash of certain content such as email and news.

    With the hash, the data can not be retrieved as such, but it is possible to verify objectionable content as genuine and not forged. This would be in the "kiddie porn/death threat/Metallica song" category.

    These logs should be expired in a reasonable period of time. Any sufficiently serious death threat could not fail to be investigated within 30 days. Any behavior which is not repeated within that period of time can be considered at an end. Tough for the slowpoke.

    Otherwise, no content logging, and no intrusive logging such as unauthorized snooping on what software is being used and how.

  21. Transparent Web proxies: are you being logged? by peterw · · Score: 5
    Last year, my ISP, without any announcement, began using a transparent Web proxy. Most of my outbound traffic to TCP port 80 gets re-routed through machines running some Inktomi transparent HTTP proxy software.

    Naturally, my ISP keeps logs for that traffic (Inktomi boasts that its Traffic Server can write many different log formats), in part to deal with abuse.

    As you might also expect, the privacy policy does not directly cover these logs. It makes promises about some very specific types of information, but does not make any general statements that obviously pertain to types of information not covered in the enumerated, specific types. Result: I think most lawyers would say my ISP could sell access to DoubleClick, the FBI, or anyone else.

    Checking your system

    So are you using a proxy, but don't know it? You can check pretty quickly (though I should warn you, while a positive/proxy result is conclusive, a negative/no-proxy result may be a result of the proxy configuration, as the systems can be set up to bypass the proxy for certain sites, or to only use the proxy for certain sites, etc.).

    Step 1: what's your address?
    Check your current address for whatever network adapter (ethernet card, PPP/dialup device, etc.). In Unix or Linux, something like '/sbin/ifconfig eth0' will do; in Windows 9x, run 'winipcfg'; in Windows NT, 'ipconfig'.

    Step 2: what address do web sites see?
    Go to a URL that will show you the environment variables passed to CGI scripts, like http://www.cgihost.com/cgi-bin/env.cgi or http://www.ualberta.ca/htbin/dumpenv.pl . Look at REMOTE_ADDR. Reload several times. Does it change? You might see some other proxy-specific variables like HTTP_CLIENT_IP and HTTP_VIA, depending on the proxy server's configuration.

    Step 3: interpreting the results
    If you ever see a REMOTE_ADDR value in Step 2 that doesn't match the local address from Step 1, yet you don't have a Manual or Automatic proxy configured in your browser, then congratulations, you're behind a transparent proxy, and should assume that all your Web traffic is being logged.

    http:// vs https:// For regular HTTP, there's a lot they can conceivably record. The URL. Your cookies. Where you came from. Etc. For https:// it's a bit better. All they can do is record where you connected to, and when. Even this information might be deemed valuable, e.g., someone frequently connecting to many banking sites probably isn't eligible for low income tax credits. https:// is somewhat like encrypting your email: they can't tell what you're doing, but they can tell who you're contacting.

    I've complained via email a few times, and received a couple polite emails from the technical staff. But nothing has changed in the official policy, so my ISP is still free to share my complete Web usage history with whomever they wish. Highest bidder? Most pushy government agency? I can't say.

    -Peter

    1. Re:Transparent Web proxies: are you being logged? by Surak · · Score: 3

      Those of you who have used IP Masquerading (ipchains) on Linux will know what I'm talking about ... routers can log *EVERYTHING* that passes through them...no proxy is required.

  22. Re:Logs, none, incorrect by thesparkle · · Score: 5

    Maybe some of you have not worked at an ISP, but ISP's keeping logs is very important, if only to combat SPAM and other forms of abuse.

    These logs should include:

    * Radius logs - username, port, and time, (Caller ID or npanxx info if you can get it), and IP assignment.

    * SMTP logs - SMTP ID. Actual copies of emails would require too much space than available to any ISP.

    * NNTP logs - again ID information only (NNTP post ID, date, time, etc).

    * Accounting logs as relevant to specific devices - for instance, shell and web servers which allow for telnet/ssh access, ftp servers, etc. This is not spying, this is good system administration.

    * DNS - knowing about those lame delegations is a big help. Especially when your customers routinely register domain names with your name servers as authoritative but fail to alert you!

    * Most important, accounting logs for root level commands as executed by the system's administrators. This can be a sore spot with some admins, but logging into a machine as root or su'ing immediately to root after login does not present accurate data as to what the admins are doing on a box. Using sudo or one of the other packages and maintaining an adherence policy to its' use should be expected. (Yes, yes there are ways around it..).

    Most of these things are standard practices for any of you who have worked for an ISP. I could care less what people were doing online unless they were violating our TOS/AUP and generated complaints. At that point, we needed to know who was doing what in order to fufill our contractual obligations to all of our customers.

  23. Re:Freeze! by thesparkle · · Score: 2

    Yep, but Mr. ISP can check the headers of the suspected email, compare it with his SMTP logs and radius logs of Alledged Perp and go from there.

    What course of action the ISP takes is subjective. However, most abusers go quietly or make up some wildly outrageous story that nobody believes.

  24. Re:Logs.... by Wheely · · Score: 3

    The argument that nobody should mind unless they are committing a crime always pops up when some invasion of privacy is being advocated. What constitutes a crime may well look very black and white when you mention child pornography but what happens when some future or in some case current laws start to gnaw away at basic freedoms.

    Regards

  25. This sounds remarkably like... by Spiff28 · · Score: 2
    ...what corporations are going through. The issue of logging is definetly a double edged sword for them. It's all a question of what you want the logs to be used for, and then conversely what they could be used for. Basically it sounds like what you're asking is what's the best trade-off.

    What do you log? As has been said, packet sniffing content would take ungodly amounts of storage, and if you're an ISP, you really shouldn't be doing it. It's Just Wrong (tm). Once again, it depends how tyrannical you want to be, but I think that just monitoring what IP's are hitting your boxes when is sufficient for most security concerns. At the most I'd say take note of traffic patterns, just incase a customer's box has been broken into and is doing things it didn't normally do.

    Should logs be permanent? We all should be able to come up with one real simple example of a corporation that was burned by e-mail leaking out that honestly shouldn't have. Corporations are now beginning to take a policy of purging e-mail stores often, so it doesn't come back to bite them in the ass. Is this ethical? Probably not. Which is why you have every right to be dumping your logs too. If corporation XYZ comes to you looking to see if the maintainer of corporationxyzsucks.com is one of your customers... sorry, you dumped the log. Don't get me wrong here, I'm not saying that ISP's shouldn't help big evil corporations if someone from them DoS'd them. I'm just saying that ISP's have a right to 'lose' information just like corporations do. Things are much less of a hassle that way.

    Legal issues. If I were a customer of an ISP that suddenly decided to start logging everything, they damn well better tell me that their terms of service are changing. Anonymity is something I value, and is a key factor in my ISP choice. What with all the DoubleClick-ish privacy things going on right now, I would not get yourself into that mess. Let your customers know exactly what you're logging, they have every right to know.

    Perhaps this is all remarkably obvious, and the opinions have been karma whored up by now, but I just thought I'd offer my two cents.

  26. I still maintain... by mindstrm · · Score: 2

    It's a public network. Use encryption. ISP's should log as much as they want.

    Of course, what they do with this information is the important part.

  27. All data logged by blakestah · · Score: 2

    ISPs should and probably do log ALL data coming through their routers. After all, that is very valuable data to some people.

    And don't kid yourself that many ISPs are not. And unless you are administering the ISP yourself, don't kid yourself that YOU are not having all your network traffic recorded.

    It is like Microsoft or Real sending pings of your net traffic back to home base across the net. There is little motivation for an ISP to abstain from such activity. It is very tough to get caught. And some people will pay for your data, especially if you preprocess it properly.

  28. Whatever you keep, destroy them soon by John+Jorsett · · Score: 2

    If I were running an ISP, whatever server logs I did decide to keep, I wouldn't keep them long; I'd be too concerned about potential abuse by overzealous law enforcement or litigants to want to retain them. If you consult a tort lawyer today, you'll be told to get rid of your company's old email fairly rapidly so that it can't be used against you in court. I think that this would be a smart strategy for server logs as well.

  29. Forged Packets by Greyfox · · Score: 2

    Drop forged packets (Essentially anything not inside your address range originating from within your address range) and log those too.

    --

    I'm trying to teach myself to set people on fire with my mind... Is it hot in here?

  30. Re:Yea by dattaway · · Score: 2

    For the wood-burning emergency generator. Duh!

    No. Nice try, though. Its the emergency backup heavy duty LART stick.

  31. Someone else.. by mindstrm · · Score: 3

    pointed out (so I'm stealing their post to a degree) that there is a difference between security and privacy.

    Should you use encryption, to keep your data secure? *YES* absolutely.
    Should your ISP be forced to keep your surfing habits private? *ABSOLUTELY*
    Should they be allowed to log as much data as they want for their own analysis later? *ABSOLUTELY*. Why? Because they *can*. It's *THEIR* network. If we say 'they can't' they can just put it in the contract; you want to use @home, you agree that we may log as much information about packet flow as we want. Period.

  32. Re:Freeze! by mindstrm · · Score: 2

    I'm not sure that ISP's are really *common carrier*. I realize that they are generally assumed to be, but I'm not sure they have the legal designation.

    After all, my ISP has a transparent proxy on my web traffic. That's not 'common carrier'. That's interfering.

    Common carrier means that they simply move data, and have 0% responsibility as to what kind of data, or where it goes.
    @home telling me *don't run a server* and scannign me for servers is *NOT* the behavior of a common carrier, as they are dictating what types of traffic I may produce.

  33. my 0.02 by davebooth · · Score: 2

    First, I do not currently work at an ISP, but I have done. I also have administered arrangements for remote access at educational estabishments, thereby effectively being an ISP for the students and staff. This was a VERY thorny question for us in all those cases. We recorded who connected when, with what IP, and who accessed the services we provided, again recording the source IP. Those logs were kept for a few months. Logs of suspected probes were kept for a few weeks, overt attacks for longer. That was it. With this info we were able to pin down the account associated with any abuse reports and spot a few compromised user accounts (usually because somebody used the same password for everything and it got cracked somewhere else) by seeing the same user pop up twice from different locations at the same time.

    The logs we kept on OURSELVES though were much more thorough. Anything one of our machines did was watched somewhere and whilst most of those logs were short-term and verbose enough to require scripted assistance to scan in any meaningful manner we made damn sure that we looked into everything that poked up above the background noise level there.

    Privacy was important too - in all cases it was clearly understood that discussing logged info with anyone outside the admin team apart from the customer who owned a suspect account was cause for getting fired immediately. To even discuss it with the customer required written authorisation. If anyone else wanted the info it had to go through the head of the admin team. Marketing folks, the billing dept, top level management (by their own request) or support staff did not have access to that raw data and it would only be turned over to anyone outside the company with a court order.

    Other guys at the company sometimes accused us (the admin team) of being anal about it and I guess we were, but the complaints sure dried up when the policy saved us from getting our ass sued.

    # human firmware exploit
    # Word will insert into your optic buffer
    # without bounds checking

    --
    I had a .sig once. It got boring.
  34. Lions and tigers and bears, oh my! by Randy+Rathbun · · Score: 2

    I have to respond to this for one reason - namely the line "I've certainly had friends be harassed and threatened online, and turning a blind eye to everything but attacks directly against the network doesn't seem right either."

    I don't think a lot of people really think about this stuff when they do it. Yes we all want to be safe. We want our friends to be safe. Sometimes we even want those we despise to be safe as well.

    But where do we draw the line? This type of thinking is as dangerous as blanket "log everything no matter what!" As the story suggests, Barnes and Noble learned a very important lesson when they kept track of everything.

    Remember, the moment you give up just one of your rights to privacy is the moment you have given them all up. Also remember that "protecting the little children", as the religious right likes to say all the time, does not mean that MY rights as an adult should be erroded because of whatever draconian law they want passed.

  35. None whatsoever. by panda · · Score: 2

    If it's your policy not to keep logs of any sort, then they can't be subpoena'd in court.

    --
    Just be sure to wear the gold uniform when you beam down -- you know what happens when you wear the red one.
  36. They should log what someone could be sued over by Builder · · Score: 2

    A friend of mine recently lost a database on his site by someone 'social-engineering' a username and password to the web interface from one of his staff. The site logged the ip that the person doing the deletions came from, but it was a proxy's IP. The ISP in question didn't store their cache logs for more than a day or two, and so could not tell him the account he was attacked from. They had backups of the data and were back up in minutes, and the pleb who leaked the password was beaten soundly, but it would have been nice to know who-dunnit :-)
    /* Wayne Pascoe

  37. Sniffer log overflow? by Christopher+Thomas · · Score: 3

    Simply monitoring a T1 with a packetsniffer doing decent filtering can easily trash a fast 30GB HDD.

    A T1 is 1.5 megabits/sec. To fill up 30 gigabytes recording _all_ data sent across the T1 (no filtering) would take about 44 hours. If a cracker leaves a sniffer unattended for that long, I have little sympathy for them.

    Overflowing a user account I can believe, but I would be amazed if drive overflow was a significant problem for the vast majority of packet-sniffing crackers. Heck, cut out HTTP and take only the first few packets of an FTP or POP session's data and you've reduced your data load by a factor of 100 or more, while keeping the information you're interested in (passwords).

    In summary, I don't think that drive space is a problem for a half-way competent sniffer.

  38. Re:Full lyrics :) by Surak · · Score: 2

    Yeah, the only thing less stable then Netscape Communicator is Mozilla ... (which has an excuse, since it is still alpha... :)

  39. yes you need logging by josepha48 · · Score: 2
    It depends on what you are doing as an ISP> IF you are a web portal then you should log certain data so you can analyze who is comint to your traffic and possible why. If you have asearch engin then you need to log the search data so you can analyze it and find out what is being searched on most frequently so you can work at building a better search engine that returns better results. If you also do web hosting you may need to log IP addresses and more so that you can provide your customers with who is visitng their web site and let then have the information that they need.

    Yes you must respect privacy, but you shoudl also state clearly and in laymans terms what your privacy policy is, and stick with it even when times are tough.

    What you should log depends on what your needs are and also what services you provide. Remeber though that you may be held responsibe for someone abusing your network, so it may be wise to keep track of who is on it and from where.

    send flames > /dev/null

    --

    Only 'flamers' flame!

  40. Re:Full lyrics :) by Fishstick · · Score: 2

    OT: Nick is going to start airing the original R&S episodes again soon. Keep the kids away from the TV! :-)

    --

    There is much cruelty in the universe, John.
    Yeah, we seem to have the tour map.

  41. It's in the Echelon Users Manual by Tom7 · · Score: 2

    What, he didn't see that section in the Echelon Users Manual that the FBI/CIA/NSA sent him when he started his ISP?

    Good ol' USA!

  42. Rejected packets by Animats · · Score: 2
    From a purely technical perspective, I used to log all "rejected packets". Packets dropped by TCP, IP, or UDP (we had our own TCP/IP stack) for any reason were logged, along with the reason. This included TCP packets that didn't advance anything, i.e. duplicates. Packets dropped due to congestion were logged as much as possible. (Only packet headers were ever logged; content belongs to the user.)

    Today, if you did that, you'd be overwhelmed. But it's useful to have the capability and to log such stuff during a peak period now and then, just to get an indication of what junk is out there. Most denial-of-service attacks will show up in such logs, of course.

  43. Not quite analogous... by jfrisby · · Score: 2

    <>

    That's a different issue. Lewinsky's purchase history was based on *financial* records. Financial records MUST MUST MUST be kept to help eliminate errors, and correct them when they do arise. Not to mention for tax purposes...

    -JF

    --
    MrJoy.com -- Because coding is FUN!
  44. Re:Logs, none, incorrect by sapphire42 · · Score: 2

    I agree with all of the above. We recently were slapped with a subpeona in a child pornography case. We are a small ISP in a small town, if we had nothing to give the police, it would have been very bad for business in this small town. The only time ANY info about any customer is given out is by subpoena, and it's happened twice since I started at this ISP. Some may cry that it's a violation of their freedom, but people pay to use a service that WE provide, on our equipment, and ISP's need to be able to protect themselves against things you might do 'in their name'. Our owner takes it quite personally when a user does illegal things on our service toting our e-mail address, etc. Granted, the logs we keep do not delve too much into someone's actual activity. Unless a user has static IP, it's nearly impossible to even see what webpages are being looked at in the logs on the cache machine. Sure, if I went to see what dynamic IP was assigned to a person at a specific time, then went to the cache logs to see what matches of the IP within the specific time frame there were, I could probably see, but it's too much trouble to even attempt without a court order involved. Radius logfiles in particular are very useful in terms of technical support. We can see why people get disconnected, etc, and we do have caller ID on all of our modems, and have caught people who were using other peoples' accounts, etc. with it. When someone calls and says that they can't get online, and we see that they are already logged in, but the phone number matches little Timmy's best friend's house, and that's where little Timmy is right now, parents get a tad upset ;-) The same people that would complain that these things are invasions will most likely be *helped* by these records at some point in the future.