Slashdot Mirror

← Back to Stories (view on slashdot.org)

Obtaining Multi-Tier Application Logs for Reseach?

Posted by Cliff on from the tricky-requests dept.

arohann asks: "I'm a research assistant in a well-known university in the US. As part of the research work my group is doing, we need access to the logs from a production system of an n-tier web-application. I've been looking around for a while with no result. Most places reply with a flat 'No!'. I was wondering if there anyone who could help/advise with this. Please read about our requirement below and do let me know if you can help?" "We want to examine the request arrival behaviour of a real-world web-application and will also need to examine how long each request takes to be processed at each tier. We would collect this data over a few days and then use it to build a real-world model of the request behaviour of an internet application. This model would be used in our analysis and profiling of clustered, multi-tier, internet applications.

Of course, we realize it maybe that some of this data cannot be shared due to client privacy concerns. However, let me assure you that we are not interested in any client details and we're not particularly concerned with what kind of an application it is as long as its at least 3-tier, is a production system (we need a real-world model), and is used daily. We are also willing to sign a confidentiality agreement if necessary and follow any company protocol required to ensure that security and confidentiality are preserved.

Of course, if this results in any research paper publications, we would give credit to the supplier of the data.

Hoping to hear back from everyone soon ;)"

40 comments

  1. own them by Lehk228 · · Score: 0

    i doubt you will get to see logs like that without taking them by force

    --
    Snowden and Manning are heroes.
    1. Re:own them by SillySnake · · Score: 2, Insightful

      Oddly enough, you might own them. Surely a 'well known' university in the US has a website that gets plenty of hits a day. I'd start by looking there, and if they refuse to give them straight to you, a professor with some pull should be able to get any information you'd need from the IT department.

    2. Re:own them by bob_marlin · · Score: 1

      http://fancyasian.gotdns.com/thumb.asp?menu=Porn&p age=1

  2. Don't think so by MarkRose · · Score: 0, Troll

    No!

    --
    Be relentless!
  3. Have you tried... by woolio · · Score: 1

    a candy bar?

    Before you laugh, it goes amazingly far where I work....

    1. Re:Have you tried... by Anonymous Coward · · Score: 0

      Ethiopia?

    2. Re:Have you tried... by Glonoinha · · Score: 1

      You would be surprised what you can get for $8 and a candy bar.

      That said, the OP may find that if he gave us the log analysis tools and algorithms he wants to apply to the log files, a bunch of us would run the analysis on our own logs and send him the results. That way he would get the benefit of a slew of different data sets, instead of just one or two.

      Companies are not too keen on sending out their internal datasets (Sarbanes / Oxley might have something to do with that, or the thought of being caught in some sort of phishing scam) but may be a bit more comfortable in simply sending him the analysis results on their data.

      --
      Glonoinha the MebiByte Slayer
  4. My Suggestions, by colemanguy · · Score: 3, Insightful

    My guess is your gonna need to try to contact them using something other then email, probably some sort of ceritifed letter.

  5. no by chris_mahan · · Score: 1, Insightful

    I will echo that: No!

    System logs are for the machine's administrators and for software developers, not researchers.

    If you guys want research material, build your own systems and sink in the tens of miillions of dollars to do that. If your app is decent you'll have more log data that you could possibly wish for.

    --

    "Piter, too, is dead."

    1. Re:no by Nik13 · · Score: 1

      Well, I doubt they'd manage to get logs from my workplace either (not for free at least)... But it should not cost tens of millions to build a somewhat simple 3 tier web app (especially with all the free tools that can generate a good part of the code for you, or using O/R mappers like [N]Hibernate). Then collecting data is a non issue (you could even offer someone to make them an app to solve a real world problem in exchange for some logs perhaps - sounds like a good deal).

      Although I see limited use for a single set of logs, from a single app. I'm thinking the type of results you'd get from analyzing the logs will differ, depending on too many factors (how well it's made, how much time was spent in profiling/optimization, how much caching is involved, platform/database/programming language used, network delays, how heavy is the business logic and more). To have relevant results you'd most likely need logs from more than one app.

      Besides, most logs might not be so helpful (many apps mostly log errors and such, which are of limited use for something like this).

      --
      ///<sig />
    2. Re:no by Nik13 · · Score: 1

      (I hate replying to myself but anyways...)

      and will also need to examine how long each request takes to be processed at each tier.

      That can vary greatly in N-Tier apps. In N-Tier apps, many people put business logic in sprocs in the database (and sometimes some in the clients... poor design usually although can be used to "double check" things), while others will have exactly 100.0% of the BL in the Business Logic Layer and none anywhere else (none in sprocs whatsoever). Things like that will affect results greatly.

      Beides, the way they want these logs, most apps I've seen would need some work done (add some custom logging code). Go thru that, redo unit tests and all that stuff (perhaps not everyone would bother), update your production servers (that can be somewhat of a pain), fil your HDs with logs (slowing things down somewhat), to hand over your logs for free and have to undo those changes later?

      --
      ///<sig />
    3. Re:no by Anonymous Coward · · Score: 0

      "Beides, the way they want these logs, most apps I've seen would need some work done (add some custom logging code). Go thru that, redo unit tests and all that stuff (perhaps not everyone would bother), update your production servers (that can be somewhat of a pain), fil your HDs with logs (slowing things down somewhat), to hand over your logs for free and have to undo those changes later?"

      Also, remember that in many IT shops, downtime and problems get you fired. All of this is risky. It isn't like you can just ask the guy admining the box for the data. This would need approval from multiple levels of management, schedule coordination, etc.

      Take a raw estimate of how long it would take to code up a logging module and triple that, then add about 6 months and you might have your data around that time. Now, if the company is really research friendly and thinks that you might be able to come up with something that will save them money in this area, cut 3 months off of the time estimate. Either way, this isn't quick, easy, or cheap.

    4. Re:no by Anonymous Coward · · Score: 0, Flamebait

      What are you? An idiot perhaps. Your systems wouldn't exist without the research that paved the way for those technologies. When researchers want to examine something with the mind of improving things, you should oblige. Of course there are all sorts of agreements that can be made regarding confidentiality, etc.
      You can use the results to improve your own system! It's a win-win situation if you'd stop being so closed mineded.

      Our group does this with large cellular providers. They actually PAY us to do this. Stop being silly.

    5. Re:no by Anonymous Coward · · Score: 0

      "What are you? An idiot perhaps. Your systems wouldn't exist without the research that paved the way for those technologies. When researchers want to examine something with the mind of improving things, you should oblige. Of course there are all sorts of agreements that can be made regarding confidentiality, etc."

      Re-read the original statement. The research assistant is asking for this data in return for a mention of support...yeah like that is going to improve our systems. Now, if they give specific recommendations, that *might* be another matter.

      The primary focus of business is to make more money. If you aren't helping them make more money then you won't get any support from them. As business people, our job isn't to improve the community as a whole, it is to extract as much money as we can from the community and do it with as little cost as possible.

    6. Re:no by Anonymous Coward · · Score: 0

      Making specific recommendations tend to be the end result of research, or else it's just a waste of time. The cellular companies certainly do their part to tell us what they want to know, and we do our part and oblige, and we get to use the data for our own ends as well (of course within the bounds of the agreement). It is up to the business, when approached, to recognize these opportunities (although we academics should know how to forsee what sorts of things would be interesting and present those -- this guy seems like he's new to all this and has some learning to do).

      The primary focus of business is to make money. However, ethical business also gives back to the community from which they derive profit. If this isn't done, then things start to break down. In the short-term balance sheet, giving back may not give you those big numbers, but over the long term it does. Some businesses realize this, others don't. Those who do tend to do better.

    7. Re:no by chris_mahan · · Score: 1

      Oh, lovely AC. I work for a fortune 200, in healthcare.

      We've sunk $20+M in one project for 1 app. Granted the app is very complex, with all sorts of crap in it. I am sure we've passed the 100,000 man hours mark too in the last 8 months. It's got some .NET, some Classic ASP, some jsp on weblogic, some MSSQL stored procs, some informatica scripts, some Oracle databases, several MSSQL databases, tibco, SOAP, Perl, python scripts, and FTP drop and reads to outside vendors. Requires 9 production servers just for the front-end data entry and reporting, and an untold number of servers on the back end.

      Also, we are required by law not to disclose the data (HIPAA), so no, agreements can't be made for confidentiality.

      And, ah, imagine that, our company's book value increased by 67% in the past year. You should have bought some stock, no? Instead of calling us ID10Ts.

      For the rest of you fellow slashdotters, sorry to have humored such a trollish remark. I'll stop now.

      --

      "Piter, too, is dead."

    8. Re:no by Anonymous Coward · · Score: 0

      So, you've wasted a tonne of money on something overly complex. Typical short-sighted business types. Who cares. You don't have to reveal confidential information in order to facilitate research! Imagine that... you can anonymize data, though I sincerely doubt the poster is after your precious data, more likely just wants usage data. Believe me, there are processes allowing data to be disclosed even under HIPAA regulations. It's not my fault you're ignorant of that fact.

      And what's this? We made money so we must be right! Well I can pimp out your mother and your sister so I guess doing that would be right too!
      In fact, considering your field is healthcare -- you're doing almost the same thing -- profiting from people's misery. And I'm the troll.
      Pathetic.

    9. Re:no by chris_mahan · · Score: 1

      "We made money so we must be right!"

      Welcome to America.

      --

      "Piter, too, is dead."

    10. Re:no by Anonymous Coward · · Score: 0

      Yep, and that's why 50 million people in this country have no health care coverage. Why the majority of the country is run-down and not much better off than a third-world country, and why this country is going straight to hell.

      That's not the America I know and love, that's the perversion that is destroying it.

    11. Re:no by chris_mahan · · Score: 1

      There's two of us and 290 million of them. Who's going to change?

      Besides, I lived in France. It's no better.

      The key is that if you work hard, at least you CAN.

      This country is not going straight to hell, since, well, countries don't go to hell. Or heaven, for that matter.

      --

      "Piter, too, is dead."

  6. Probably not. by Seumas · · Score: 1

    I'm not familiar with this "reseach" that you speak of, but it doesn't sound like something I'd want to donate my logs to. It sounds like some sort of weird internet version of felching.. Ew. No thank you.

  7. Talk to research friendly companies by jhoger · · Score: 4, Informative

    I'd start with companies that are already offering internships through your university. Find professors and graduate students that already have a working relationship with private sector folks and get introduced through them.

    Just cold calling or sending in letters or email is about as effective as you've found it to be.

    Also you should try looking through published artcles in trade journals and find out which companies are sponsoring research in your field by association with existing published research.

    The fact is that you'll certainly have to sign an NDA and likely they will have to scrub the data anyway. One way or another it's going to cost the donors $$$ that you aren't going to reimburse. Your project will have to fit in with their research goals or they'll be returning a favor from someone else.

    -- John.

  8. Use your professors by deranged+unix+nut · · Score: 3, Insightful

    Your best bet would be to have your professors call in a favor from former students or their contacts in the industry.

    Most companies will consider this to be a security risk. They don't even want you to know the rough design of their backends let alone collect data from it.

    Some companies wouldn't know how to gather what you want and wouldn't risk letting you touch their systems.

    Most of these systems are probably messy, kludged together by former employees and hacked by current employees just enough to keep them running.

    If you have some time, get an internship and do your research on the side. :)

  9. What's it worth to the supplier? by stienman · · Score: 2, Informative
    I'm a research assistant in a well-known university in the US. As part of the research work my group is doing, we need access to the logs from a production system of an n-tier web-application.

    Welcome to capitalism, we hope you enjoy your stay. While here, please note that TANSTAAFL.

    Asking for data from a business requires a lot of work on your part. You must somehow convince them that all the effort they are going to spend collecting, sanitizing, and providing you with the information is going to pay off for them in a reasonable way. Since this request involves several months of data, and more employee involvement than a 5 minute survey you'll have to build a strong relationship with a company who has this data.

    Opportunites include:
    • The research will help you identify areas where improvement will save $$$ in [bandwidth|speed|latency|etc]
    • We can supply one or more interns to do all the internal work as well as work on a few other projects of your choosing
    • You (manager, CEO, IT lackey) got your degree here and still have fuzzy feelings for the school
    • Oh benevolent ones! May we sip at the firehose? Verily, this research will help this university provide graduates of the caliber which will dazzle the eyes! Yea, they will be cheap, too.
    The key here, as in everything to do with business, is to network, network, network. Don't email - you cannot possibly explain your research in a way that will make them go, "Gee, I think I'd like to devote company resources to these kids tha the university of whatever!" in an email. At best send an email such as, "Dear sir, blah blah blah, we are researching n-tier applications and would like to spend a few moments talking with you about your architecture. When would be a good time to call?" Give it two days - Call them in any case except if they patently refuse to talk to you. Don't engage in email conversations - in order to get good buy-in, you need to talk to them (if only briefly) so they can associate a voice with the email. Then email all you want.

    You may have better luck calling at the outset, intriducing yourself and your research, then asking who at the company would be suited to help you out with your research. Then engage that person. Don't get too low on the totem pole or you may end up with someone who is inneffective within the company at getting you what you want. Certian companies (Google, forinstance) are resource rich and may be easier to work with, especially if you can get one or two workers involved and spending their 20% time helping you. If your research isn't exciting on a general level, you're in for a rough ride.

    Once you've started a conversation (with several people at different companies - you're still trying to get something they will be reluctant to give) then you can start edging into what you need to complete your research. This whole process will take 2-6 months just to set everything up. I hope you've started early.

    Good luck.

    -Adam
    1. Re:What's it worth to the supplier? by Anonymous Coward · · Score: 0

      This is good advice. Too many geeks think that simple bussiness skills are beneath them claiming that they strive for higher goals then money. But most great achievements require help. Which, in turn, require some simple skills to organize and manage.

      You are someone who should go far in life. May you achieve your goals, whatever those goals may be.

  10. be more helpful, talk to open-source website by sonamchauhan · · Score: 2, Interesting

    ...
    > if this results in any research paper publications,
    > we would give credit to the supplier of the data.

    If that's all you offer in return, which company will allocate the resources to verify:?
      (a) this breaches no privacy laws (b) business advantage isn't sacrificed? ...And rightly so for companies whose constitution is 'maximise profit'.

    Some suggestions:

    1. Offer a quid-pro-quo to companies you contact: in return for access, you will deliver (say) a multi-page detailed architectural review and specific recommendations on potential improvements, reviewed, say, by your professor.

    2. Talk to people who run websites for non-profits, or open-source/ creative-commons websites like wikipedia.org, sourceforge.net, even slashdot. The attitude there may be more sympathetic to your efforts and the admins more willing to knock up a few Perl scripts to strip logs of sensitive information.

    3. Offer to be a website maintainer for a large indepedent open-source / community effort and obtain agreement on your access to logs.

  11. I can probably help by abradsn · · Score: 2, Interesting

    I've written a couple of these (including one for an extremely large software company that I'm sure you've heard of), and I'm currently working on one right now on the side (for my own personal gain).

    If you reply to this comment with your email address, then maybe we can work something out.

    I need some help with testing my current project, and you need some data. It 's actually more work for me to have someone besides myself test the software but the quality should be higher and it could help you out.

    It would probably take at least a few weeks of work on your end, but you would definately have your data at the end of it. Thanks, Brad

    1. Re:I can probably help by arohann · · Score: 1

      Hi, Yes I'm interested in the logs you have to offer for our research. Can you tell me a bit more about the application itself ? My email is arn129@psu.edu Thanks and regards Arjun

      --
      ....In a world without walls, who needs Windows ?
  12. Are you for real? by Saeed+al-Sahaf · · Score: 0

    If you are from a "large university", how come you can find any big app log files right on campas? Most "large universities" have plenty of "n-tier web-applications". Me thinks your request smells bad.

    --
    "Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck
    1. Re:Are you for real? by thegrassyknowl · · Score: 1

      If you are from a "large university", how come you can find any big app log files right on campas? Most "large universities" have plenty of "n-tier web-applications". Me thinks your request smells bad.

      At my university the response would be a big fat NO!. I have asked the admins for some software that I desparately needed to do my research work efficiently (OSS) and they just said "no that's a security risk". I fail to see how some basic image procesing software is a security risk, but they have it in their head it is.

      University admins tend to run much bigger systems that do a lot more. I can see why they would say a big-fat-no to your request and why a smaller local company would be a better place to ask.

      --
      I drink to make other people interesting!
    2. Re:Are you for real? by UnrefinedLayman · · Score: 0
      If you're a school bus driver filling up the tank on Old Yeller and one of your students comes up to you and asks, "Hey bus driver! Will you add this bottle of Opus's Secret Solution to the oil? It'll make the ride for seat #12 all smooth for me!" are you going to do it?

      Of course not, and here are a few reasons:

      • It's not your bus; you just drive it
      • You don't know what's in that bottle of OSS
      • It may make seat 12 great but cause the engine to seize up
      • You're talking to a kid whose bus-maintenance experience could be summed up as having read The Magical School Bus for all you know
      • Your responsibility isn't to make one kid's ride pleasant, it's to get all the kids to school every day
      • Busses are expensive and there may be no replacement engines at the depot if the bottle of elixir breaks everything
      • The bus is no longer under its 10,000 mile warranty so anything that stops working will have to be replaced out of pocket and serviced by yourself
      • There's no company to contact to ask for a service contract for Opus's Secret Solution so that in the event of a problem while using it Opus can lend assistance until it's resolved
      • You don't want to have to watch the news every night to find out if there's been a reported problem with OSS that causes the engine to shut off when a Lexus pulls up next to you and that a new bottle of OSS is required to correct that issue
      • Kid 12 is only going to be at school for another year, the bus needs to run for four years; after kid 12 leaves, do you try to safely remove the elixir or leave it in?


      I would say the point I'm trying to make is that system administrators have in their heads the knowledge to administer systems safely and correctly. My experience is that people who try to second guess them lack fundamental knowledge and experience (or have just forgotten it) to understand why they do what they do.

      Sys admins aren't the giant assholes everyone thinks: part of the job is to look out for the safety of everyone, and similar attitudes and responses are prevalent in similar roles of responsibility in other fields.
  13. Underlying reason by x_terminat_or_3 · · Score: 1

    You say you want it to make a business model etc, but you don't say what your final goal is.

    Since you're already on ./ tell us what you really want in the end, and we might find a workaround. One rule with asking questions is never assume you are right. You ask for help, so let us draw the conclusions.

    --
    Only those who risk going too far can possibly find out how far they can go. T. S. Eliot
  14. ever thougth ... by schmu_20mol · · Score: 1

    ... of building yourself a net of vmwares for this purpose?

    --
    "Nae Kin! Nae Quin! Nae laird! Nae master! We willna be fooled again!"
  15. Smells fishy by slashflood · · Score: 1

    I can't help, but it smells fishy to me. Why don't you tell us the name of the "well-known university"? Where is your email address? Why don't you answer the post that asks for your email address? Why are the questioners of an Ask Slashdot question never replying to the answers?

  16. Send a data shrouder with your requst by davecb · · Score: 1
    Send a perl programs that changes all the fields that might be sensitive, so that the person can test and convince themselves thye're not going to leak information to you.

    That will make them realize that you understand some of the constraints they are under, and that you'e a nice person (:-))

    In particular, transform
    http://www.sin.com/porno into
    193'd-seen-HTTP-address

    --dave

    --
    davecb@spamcop.net
  17. Sourceforge? by maxume · · Score: 1

    Does sourceforge fit your definition?

    --
    Nerd rage is the funniest rage.
  18. use your own? by Gothmolly · · Score: 1

    Um, if you're at a "well-known university in the US", then I'm sure your campus IT and Admin departments have some interesting setups. Try getting log data from the on-line registration system, during the register/add/drop period at the beginning of a semester - plenty of traffic and data moving around.

    +0, Obvious?

    Christ, what do they teach in schools these days?

    --
    I want to delete my account but Slashdot doesn't allow it.
  19. we can give you access... by Anonymous Coward · · Score: 0

    we can give you access to one of our servers logfiles provided that we remain anonymous and do not tell you what the server does. we log each function and request with granualities of +/- 1ms. the server also uses a reflecting proxy so all requests logged are from localhost but that shouldnt matter to your environment. the loads and relevant requests are all real world.
    post your email so we can send it to you.

  20. Sweeten the pot by dubl-u · · Score: 1

    The way to make this happen is to offer something more than credit in return. If you are going to collect a lot of data on an application, you'll have the opportunity to give them some cool reports about where the bottlenecks are. Offer the reports in exchange for the data. Sure, they *could* make the reports themselves, but most places never do.

    If you can't get in on your own, convince some important vendor (e.g., IBM's Websphere group) that you're legit and can help them if they'll help you.

    And if nobody bites on the "we'll give you reports for free" thing, then charge them $30k for your reports or $20k if they let you reuse the anonymized data in your research. You'll need a business listing and a sales guy with a nice suit to pull this off, but I promise that it will work, especially if you let the sales guy keep all the money.