Slashdot Mirror


London Stock Exchange Tackles System Problem

DMandPenfold writes "The London Stock Exchange has taken steps to resolve a system problem that occurred at 4.30pm Tuesday, which saw a delay to the start of the closing auction and knocked out automatic trades during a 42 second period. The problem occurred a day after the high profile launch of its new matching engine on the main equities market, based on the SUSE Linux system from Novell."

127 of 237 comments (clear)

  1. First posters are lame by VMSBIGOT · · Score: 1

    So, they replaced a C#/.NET infrastructure with C++/Linux and apparently didn’t test it? Now, I’m not the systems engineer involved with this, but I have a hard time believing that something like this didn’t come up in testing.

    At least now they have the ultimate answer, now they just need to work on the ultimate question.

    1. Re:First posters are lame by mwvdlee · · Score: 4, Insightful

      No test can ever be a 100% accurate representation or real use.
      It's probably somewhere in the 0.1% mismatch where this problem occurred.

      --
      Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
    2. Re:First posters are lame by michelcolman · · Score: 2, Insightful

      If I were to introduce a new trading system like that, I would probably run it in parallel with the old system for a couple of months, feeding it the same data and checking the results. I can't imagine them just doing some offline tests and then throwing the switch and hope for the best? Surely not?

    3. Re:First posters are lame by pieterh · · Score: 4, Insightful

      This works for a component with one set of inputs and one set of outputs.

      A trading system is essentially chaotic in the way it processes data because it gets so many inputs and their relative arrival times determine the system's behavior. You'd have to be replacing the old system with an identical new one, and then add heavy and slow synchronization to all the inputs going to both systems (so e.g. a trade A hitting the old system one microsecond before trade B also hits the new system the same way).

      So yes, it comes down to running a whole lot of offline tests using real data and then bringing it online.

    4. Re:First posters are lame by iserlohn · · Score: 4, Informative

      The system has been online since the middle of last year. Nice for you to drop by and dish out out your golden nuggets of wisdom tough.

    5. Re:First posters are lame by michelcolman · · Score: 1

      I did add a "surely not" at the end, didn't I?

    6. Re:First posters are lame by Anonymous Coward · · Score: 4, Informative

      The summary is even lamer. It didn't knock out any trading. What happened is that the closing auction announcement was delayed by 42 seconds. Normally any trades submitted prior to the announcement get ignored. However, in this case I'll quote from the article:

      LSE traders are required to wait for the message before trading. Normally auction trade instructions sent before the message would not complete, however because of the surprise delay, on this occasion order book trades were allowed to complete later.

      So, no outage, no downtime, no lost trades.

    7. Re:First posters are lame by hattig · · Score: 1

      So, no outage, no downtime, no lost trades.

      So everything worked fine even when an unforeseen issue occurred and this should be seen as a glowing endorsement of the system that is in place now?

    8. Re:First posters are lame by jrumney · · Score: 1

      They have been running it in parallel since at least October. Search for "LSE Turquoise bug" for all the past outages they've experienced. There's lots of talk about "human error", "suspicious circumstances" and "network problems", but noone seems to be pointing the finger at the software itself, which was thrown together in a few months by a previously unknown software development house in Sri Lanka which LSE bought outright for less than they pay Microsoft and Accenture every year for maintenance of the previous system.

    9. Re:First posters are lame by ThePhilips · · Score: 1

      Platform changes don't actually address the underlying issues of incompetent programmers to begin with. Platform changes just wash the bad taste of a now fired programmer out of your mouth when you replace them with someone else who works in a different environment.

      I would argue that in that particular case platform matters.

      Most modern platforms (C#/.Net, Java) encourage sloppy programming. (*) That works fine on desktops and even in corporate data centers, where minor problems can be brushed off, where excessive resource consumption can be mitigated by throwing in more hardware. But that isn't acceptable for customers like LSE, where even adding piece of dumb hardware might require exhausting test: to make sure that software works with more hardware and the extra hardware itself works without problems too.

      (*) Do not get me wrong. As developer, in general, I'm all for the sloppy coding practices, anything what makes life of developer easier.

      --
      All hope abandon ye who enter here.
    10. Re:First posters are lame by LingNoi · · Score: 1

      Perhaps you'd like to compare it to the old system which shutdown trading for a whole day if not longer? I don't remember the specifics.

      A 42 second delay at the end of trading is nothing. I really don't understand why people hold this Linux platform to a much higher standard and overlook the poor performance of the previous system which was terrible.

    11. Re:First posters are lame by dave87656 · · Score: 1

      Had this been a Windows based system and it stopped for only 42 seconds it would have been hailed as high reliability. But you've got the Windows fanbois who can't wait to find an issue, any issue, please, with a Linux system.

      Remember the article a couple of weeks ago where there was an outage on the LSE while still running on the C# system? The article's title was something like LSE Outage during Linux test. Of course, the Linux test had nothing to do with it, but they had to find something.

    12. Re:First posters are lame by dave87656 · · Score: 1

      So, no outage, no downtime, no lost trades.

      It amazes me how quickly the press tries to make a big deal about a Linux outage that isn't even an outage. The Windows monopoly must be nervous.

  2. Newsworthy? by Anonymous Coward · · Score: 1

    Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

    1. Re:Newsworthy? by c0lo · · Score: 1

      Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

      I'm afraid I can't remember what you said about the name of company you work for, can you please repeat it?

      --
      Questions raise, answers kill. Raise questions to stay alive.
    2. Re:Newsworthy? by Anonymous Coward · · Score: 1

      Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

      I'm afraid I can't remember what you said about the name of company you work for, can you please repeat it?

      One of the major exchanges in Chicago, as well as one one of the bigger global banks. Not a small firm.

      Besides, if Reuters or Currenex had this same issue for two hours, we STILL wouldn't hear about this.

    3. Re:Newsworthy? by c0lo · · Score: 2

      Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

      One of the major exchanges in Chicago, as well as one one of the bigger global banks. Not a small firm.

      Ah, I see... well, maybe it would be news-worthy, but... too pity there's too much noise coming from the open outcry trading in the pit... nobody hears about an outage and even during an outage the business goes as usual in that pit... everybody furiously shouting and gesticulating.
      (just kidding)

      Maybe it will help to compare the LSE trade-volume per day with the same for your workplace?

      --
      Questions raise, answers kill. Raise questions to stay alive.
    4. Re:Newsworthy? by DeBaas · · Score: 5, Funny

      Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

      Simple, when a Linux system has a 42 seconds outage that's news. When a windows systems has several hours outage, there is nothing new about.
      Odd that you didn't get that..

      --
      ---
    5. Re:Newsworthy? by h4rm0ny · · Score: 1

      Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

      I'm afraid I can't remember what you said about the name of company you work for, can you please repeat it?

      One of the major exchanges in Chicago, as well as one one of the bigger global banks. Not a small firm.

      Besides, if Reuters or Currenex had this same issue for two hours, we STILL wouldn't hear about this.

      I can't answer why it is "news", but I can answer why it is on Slashdot. Over the past couple of years, Slashdot has increasingly posted stories that have flamebait as their primary feature and where it isn't intrinsic to the story, it is often editorialised to add a little prompting to get us all arguing. This has an ostensible Linux vs. MS angle hence qualifies to be featured prominently for the sake of the post-count boosting flamewars it could induce. I say "ostensibly" because in reality, any change over between two such large, complex and hyper-used systems will experience problems.

      --

      Aide-toi, le Ciel t'aidera - Jeanne D'Arc.
    6. Re:Newsworthy? by Gruturo · · Score: 1

      Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

      Cause it's about Linux, it's News for nerds, stuff that matters (to us, at least).

      --

      Vacuum cleaners suck. Kings rule.
    7. Re:Newsworthy? by cbope · · Score: 1

      Actually, cue someone claiming that X millions of dollars or pounds were lost during that 42 seconds of downtime...

      It's all about trading speed these days, with milliseconds being critical. Not that I agree, because usually this means someone is gaming the system through automated computer-based trading systems. I would like to see a mandatory delay added to all trades to slow things a bit and let the market become more manageable, by you know, humans. After all, it's we, as humans, that are ultimately doing the trades. The problem is automated trading is too widely used and distorts the markets. I would even go out on a limb and say this was a major contributor to the markets crashing.

    8. Re:Newsworthy? by Americano · · Score: 1

      God, why won't this stupid meme die? "Inserting a mandatory delay" does NOTHING to solve the problem you're asserting it will. All it does is mean that it'll take $delay seconds after market open for high-speed / automated trading to begin hitting the exchange, and it will continue ALL DAY LONG until market close.

      This assertion that a delay would somehow make the market "manageable" for individual humans is like saying that making everybody wait in their driveway for 2 minutes before they begin their commute to work will somehow alleviate traffic jams.

    9. Re:Newsworthy? by jrumney · · Score: 1

      Because those several hour long outages happened during the testing phase. From Monday the system went live, so outages are causing real problems now.

    10. Re:Newsworthy? by DrVxD · · Score: 1

      Cause it's about Linux

      It's also about 42

      --
      Not everything that can be measured matters; Not everything that matters can be measured.
    11. Re:Newsworthy? by An+ominous+Cow+art · · Score: 1

      I can't answer why it is "news", but I can answer why it is on Slashdot. Over the past couple of years, Slashdot has increasingly posted stories that have flamebait as their primary feature and where it isn't intrinsic to the story, it is often editorialised to add a little prompting to get us all arguing.

      It's all starting to make sense, now. The entity calling itself "CmdrTaco" is one of those energy beings from the Start Trek "Day of the Dove" episode, feeding on the strife it creates.

    12. Re:Newsworthy? by Gruturo · · Score: 1

      $delay won't solve squat as you point out, but rounding everything down to the nearest second might alleviate some issues. "All orders between 08:13:35.000 and 08:13:35.999 are considered simultaneous and will all be recorded as 08:13:35".

      With 1 second "tick" even a human player might have a chance vs computers! Cue Watson jokes.

      --

      Vacuum cleaners suck. Kings rule.
    13. Re:Newsworthy? by Americano · · Score: 1

      I guess I don't understand what problem you think this will solve, at all. "Treating them as simultaneous" doesn't change the fact that they're received in an order, and processed in a particular order. All you're going to do is invent a new mouse trap for the mice to engineer around - either they'll stuff the queue with more orders to try and increase the chances that some of their trades will be randomly selected to go first, or they'll figure out the delay and algorithm you're using to process orders, and game that.

      A "regular human player" who's trying to time trades in terms of seconds is not going to win. Computers are faster, and the amount of money required to build the trading systems and eke a profit out of arbitrage using these systems is not "stealing money" from anybody. They are making money in fractions of a penny per share per trade, and have to trade thousands of shares just to make a couple bucks. It's a game for institutional investors with deep pockets and a lot of patience. Normal humans are far better served buying and holding long-term investments based on the fundamentals of the stocks they're buying. The computers are trying to eke a profit out of normal intraday and cross-exchange fluctuations in prices. At the same time, they're providing additional liquidity to the market, and acting as a damping mechanism on that price volatility.

      Automated trading did not cause the meltdown in 2008. Automated trading did not cause the meltdown in 2001. What problem, exactly, are you trying to solve by introducing latency (and thus pricing & trade inefficiency) into the financial market, and what mechanism do you specifically expect that your solution will use to solve that problem?

    14. Re:Newsworthy? by h4rm0ny · · Score: 1

      It's all starting to make sense, now. The entity calling itself "CmdrTaco" is one of those energy beings from the Start Trek "Day of the Dove" episode, feeding on the strife it creates.

      Or he gets ad revenues based on page hits, but yours is good too. ;)

      --

      Aide-toi, le Ciel t'aidera - Jeanne D'Arc.
    15. Re:Newsworthy? by Vaphell · · Score: 1

      At the same time, they're providing additional liquidity to the market, and acting as a damping mechanism on that price volatility.

      that liquidity is fake and artificial. HFT algorithms trade the handful of shares back and forth to set trends, confuse competing algorithms and discover optimal prices triggering trades. Flash crash showed how great they are at providing liquidity and damping the price volatility.
      Introducing ticks would mean that there is less profit in cutting into a trade that would happen anyway. There would be no time to probe buyers and sellers with bogus transactions to discover their prices and use that knowledge against them to pocket the difference. Mere fractions of pennies per share certainly haven't prevented GS from getting over $100M of revenue on record DAYS.

    16. Re:Newsworthy? by Americano · · Score: 1

      Well, since it's fake AND artificial, I'm sure your point is completely valid.

      If all HFT algos were doing was passing the same set of shares back and forth, nobody would be making any money at it. But I'm sure you already knew that. They're buying shares that investors want to sell, and selling shares that investors want to buy. They make money off it... and if they weren't there offering to buy & sell those shares, the price would probably be different, or at least certainly more volatile, when the investors wanted to buy or sell a position.

      The "Flash Crash" wasn't caused by HFTs, though I know that's a popular meme which allows us to keep being angry at all those "fat cats" in their "business suits" on "Wall Street". It was triggered by a large mutual fund company deciding to sell an abnormally large number of E-mini contracts, which basically exhausted the pool of buyers for that instrument. This caused the prices to decline, and while it was *exacerbated* by algo trading shedding futures and real shares into a market where most buyers were already exhausted, it was not "caused" by the HFTs. The conditions would have existed for that sharp drop regardless of whether or not a computer was doing the trading.

      So, once again: How does adding latency to every trade do ANYTHING to solve this problem? We've had large, very long-lasting point swings long before computers and HFT algorithms were ever in existence. Given that everybody loves to point to the Flash Crash as "proof of the evil of HFTs," I'd suggest that if that singular example is all you can cite, then you're really not looking to "make the market more fair," you're simply looking to "keep somebody else from having something." If you had money invested before that event, and you didn't participate in the stupid short-term, short-sighted panic and sell everything at the bottom of the flash crash, you lost nothing, and your investments have grown in value since then.

      The "$100M of revenue on record DAYS" that you're complaining would have been simply wasted, lost to friction in the market, if you prevent anybody from trying to capture it. Should FedEx introduce random latency and delays into their systems in order to make bike couriers more competitive? The bike courier can't compete with FedEx's scale & reach. Maybe in the interests of fairness, we should force FedEx to use only bikes as well? After all, if one company can take advantage of its greater size & scale to make money in a given industry, surely that source of profit is illegitimate and should be completely outlawed!

  3. Well, well, well... by LordNacho · · Score: 1, Interesting

    This just shows that it's hard to build these highly available, low latency, massive usergroup systems. Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)

    The problem is more likely to be internal organisation than specific platforms.

    1. Re:Well, well, well... by _Sprocket_ · · Score: 4, Insightful

      This just shows that it's hard to build these highly available, low latency, massive usergroup systems. Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)

      Yes. And let us not forget that a lot of that chatter came from Microsoft's PR department.

    2. Re:Well, well, well... by the_womble · · Score: 3, Interesting

      Given that this was a showcase client for MS, it does not make them looks good.

      Given that the MS was involved in developing the TradeElect system (the Windows based one), so even is the fault lies in TradeElect rather than the MS platform, then its still at least partly MS's fault.

      Microsoft and Accenture developed a system that turned out not to be as good as the one developed by a small Sri Lankan company no-one had ever heard of.

  4. "Oh well I guess Linux sucks then by atomicbutterfly · · Score: 1

    ... should have stuck with Windows eh?"

    I just know some Microsoft fanboys (they do exist, check out Neowin.net for examples) are gonna latch onto any and all problems that occur after the switchover, regardless of merit.

    1. Re:"Oh well I guess Linux sucks then by atomicbutterfly · · Score: 1

      you mean exactly the same way that Linux fanboys latched onto any and all problems that occurred before the switchover, regardless of merit?

      Actually, yes, exactly like those Linux fanboys. I dislike fanboys of any camp.

      What? You didn't think I was totally brainwashed by Slashdot did you? :)

    2. Re:"Oh well I guess Linux sucks then by slimjim8094 · · Score: 4, Insightful

      To be fair, if this was Windows you *know* the Linux fans (myself included) would be berating them for choosing such a crappy platform. And it was Windows, and we did berate them for it...

      Of course, the truth is somewhere in the middle - isn't it always? The most important part of a trading computer system set up is, well, the trading computer system software. The system's biggest problems aren't going to be due to Coolwebsearch or a bluescreen, they'll be with the trading software itself. The OS doesn't really matter. Buggy software is buggy software.

      That being said, Linux is just a better platform to build something like this on. Sure, you can do it with Windows and make it work, but it's just more and unnecessarily difficult.

      --
      I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
    3. Re:"Oh well I guess Linux sucks then by michelcolman · · Score: 4, Funny

      They should have just used a Mac. iStockExchange is a free download for MobileMe members.

    4. Re:"Oh well I guess Linux sucks then by Anonymous Coward · · Score: 1, Interesting

      That being said, Linux is just a better platform to build something like this on. Sure, you can do it with Windows and make it work, but it's just more and unnecessarily difficult.

      I keep hearing this, but never see any technical details. Why is this so? From hearsay, the use of Linux in finance/trading shops is widespread (for servers... clients are typically Windows), it could be true... or it could just be to reduce licensing costs rather than for purely technical reasons.

    5. Re:"Oh well I guess Linux sucks then by tgetzoya · · Score: 1

      I can image them doing that, even after Apple stopped selling their server systems and told everyone to use their mac minis in the meantime.....

    6. Re:"Oh well I guess Linux sucks then by ADRA · · Score: 1

      That's a sad but true commentary. If I heard of a high profile Linux->Windows migration that failed in some way, I'm sure I would've jumped on it. The truth is that any new systems roll-out is hard, and the pressures on anything involving large amounts of money is even worse. That's why it takes forever to develop and deploy these systems. I hope their issue was something perceivable and easily addressable.

      --
      Bye!
    7. Re:"Oh well I guess Linux sucks then by Anonymous Coward · · Score: 3, Funny

      But wont Apple keep 30% of the money that flows through it?

    8. Re:"Oh well I guess Linux sucks then by ErroneousBee · · Score: 1

      I am not camp!

      Also, I just think Linux is a better operating system, with reduced time spent managing the system, fixing errors and waiting for it to open applications.

      --
      **TODO** Steal someone elses sig.
    9. Re:"Oh well I guess Linux sucks then by the_womble · · Score: 3, Interesting

      Licensing costs are trivial in the context of these sorts of systems. TradeElect cost £40m, the new system was £50m - but that was to buy the company, not just the software.

    10. Re:"Oh well I guess Linux sucks then by the_womble · · Score: 1

      The move to Windows happened three and a half years ago. That is hardly a new system: its mature and its still crap.

    11. Re:"Oh well I guess Linux sucks then by the_womble · · Score: 1

      Sorry, I misunderstood the GP.

    12. Re:"Oh well I guess Linux sucks then by aliquis · · Score: 1

      iStockExchange is a free download for MobileMe members.

      But Apple take 30% on every trade ...

    13. Re:"Oh well I guess Linux sucks then by Anonymous Coward · · Score: 1

      ...changed ...from C# to C++ which definitely is a huge downgrade.

      bozo

      They are trading a higher-level language with (builtin) garbage collection, huge standard library, runtime protection, linq and runtime protections for one which hasn't ...

      Yeah, because the non-deterministic garbage collection, the bloated standard library, 'runtime protections' and linq don't contibute to the solution at all, and if anything add possible failure points.

    14. Re:"Oh well I guess Linux sucks then by Anonymous Coward · · Score: 1

      Of course, the difference is that Microsoft were hailing the LSE system as a great example of why you should use Microsoft's software stack, with the usual marketing dreck about how Microsoft software makes everything possible, blah blah blah.

      It had problems because of the crap they built on top of it. We all knew that the underlying platform didn't really matter with these kinds of systems, but Microsoft wanted to publically take all the praise and positive publicity. So instead, they took the criticism and the blame.

      It kind of was Microsoft's fault. They had a big hand in developing the software that ran the exchange, and they deliberately picked technologies based on what Microsoft could use as a showcase, rather than what actually made sense for the project.

      After that, the LSE basically decided to build their own. It happens to use Linux. This doesn't really matter, because the underlying platform is irrelevant with these kinds of systems. It's hardly even worth mentioning, except for the history.

      This time, it's their own system. It has a bug. They fix it. No blame, no complaining, and no scrapping the entire system and starting over.

    15. Re:"Oh well I guess Linux sucks then by anonymous+cupboard · · Score: 2

      I worked on a trading system back in the early days. We hit lots of "edge" performance cases. To take full advantage of what a system offers us and to code around problems we usually have source code to look at. We didn't change it, but we had to have the access. MS would gladly give their source code to major customers, but frankly there is more expertise around Linux kernels than Windows.

    16. Re:"Oh well I guess Linux sucks then by shutdown+-p+now · · Score: 1

      I keep hearing this, but never see any technical details. Why is this so?

      I wonder if that is because there are some hard real-time requirements for those systems (or at least parts of them)? Stock Linux isn't that, either, but there are hard RT forks which still count as "Linux". There's no such option with Windows.

    17. Re:"Oh well I guess Linux sucks then by DMandPenfold · · Score: 1
  5. 42 seconds, that's eternity! by michelcolman · · Score: 4, Insightful

    Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.

    1. Re:42 seconds, that's eternity! by haploc · · Score: 2

      Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.

      At least it would be The Answer to.. !

    2. Re:42 seconds, that's eternity! by Anonymous Coward · · Score: 1

      Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.

      Now Imagine finding out that there really was not an outage of any length, all the trades completed, and nobody lost anything, but you had already made such claims in a very public forum.

      Amazing how reading the actual article will prevent you from looking like a complete idiot.

    3. Re:42 seconds, that's eternity! by Americano · · Score: 1

      Man, I want you to come talk to my boss and tell him that 42 seconds isn't an "outage of any length."

      When you work with market data and trading systems, a "significant outage" is anything with time > 0 that your functionality is unavailable. Business critical applications are, susprisingly, considered critical to the operation of the business.

      This is why you have hot-spare High Availability / Disaster Recovery configurations, robust monitoring and load balancing, and a team of operations staff standing by to respond at the first sign of a problem. And that's also why it costs tens of millions of dollars to design, build, test and deploy a system like this, despite the script jockeys' assertions that "LOL 6 guys and 6 months could build that."

    4. Re:42 seconds, that's eternity! by ignavus · · Score: 1

      Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.

      "Hi, 1970s trader! I am just going to skip over the momentous fact that I have accomplished time travel - the ultimate cool technology - to let you know that one day people will be even more obsessive about the stock market than they are in the 1970s.

      Thought you'd like to know. Oh, and in case you are interested, I can travel through time. Now going back to talk to HG Wells about a great book idea I have."

      --
      I am anarch of all I survey.
  6. 42 seconds by tal_mud · · Score: 2

    Liffe[sic] the universe and everything.

    1. Re:42 seconds by DamonHD · · Score: 2

      LIFFE != LSE

      --
      http://m.earth.org.uk/
    2. Re:42 seconds by Luyseyal · · Score: 1

      Restate my assumptions:

      One, Mathematics is the language of nature.

      Two, Everything around us can be represented and understood through numbers.

      Three: If you graph the numbers of any system, patterns emerge. Therefore, there are patterns everywhere in nature.

      Evidence: The cycling of disease epidemics;the wax and wane of caribou populations; sun spot cycles; the rise and fall of the Nile. So, what about the stock market? The universe of numbers that represents the global economy. Millions of hands at work, billions of minds. A vast network, screaming with life. An organism. A natural organism. My hypothesis: Within the stock market, there is a pattern as well... Right in front of me... hiding behind the numbers. Always has been.

      — Maximillian Cohen, Pi

      --
      Help cure AIDS, cancer, and more. Donate your unused computer time to worldcommunitygrid.org. Join Team Slashdot!
  7. Someone skimped on testing by PolygamousRanchKid+ · · Score: 4, Insightful

    I've seen this too often in my 25+ professional years in IT. The system test manager produces an excellent plan, that fully simulates the anticipated workload. But it requires X testers, Y test case developers and Z machines. The program manager rejects the plan, "because he is under pressure to reduce costs." The program manager says, "The testing that the developers do should be enough." He then moves on, before the system goes into production.

    The result? It always ends in tears.

    --
    Schroedinger's Brexit: The UK is both in and out of the EU at the same time!
    1. Re:Someone skimped on testing by daid303 · · Score: 4, Interesting

      Testing is just hard. Even with a perfect plan, and a perfect execution you're still going to miss things.

      I just spend 2 weeks finding a 6 year old bug, which only showed itself when the temperature of our product is somewhere at -5C, and then only on 75% of the total devices. Devices where tested at low as -40C, but those where lucky and kept working correctly, so nobody noticed before. Even after I found it I spend a week looking at the code to find what's really wrong.

    2. Re:Someone skimped on testing by pak9rabid · · Score: 1

      Do you really think with a project this large and crucial, that the people in charge would willingly skimp on testing? I mean shit, this isn't BP we're talking about here. I'd imagine this is a pretty huge system. I'm willing to bet that whatever caused this outage was just simply overlooked when developing the test plan.

    3. Re:Someone skimped on testing by mvdwege · · Score: 1

      Do you really think with a project this large and crucial, that the people in charge would willingly skimp on testing?

      Yes.

      --
      "I know I will be modded down for this": where's the option '-1, Asking for it'?
  8. Teething problems by mangu · · Score: 4, Insightful

    Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)

    It's one thing to have a 42 seconds glitch in the first day a totally new system is powered up. That's perfectly normal, and had been predicted:

    "Observers watching today's Linux-based launch will likely note that such a large change could bring about some teething problems, as with any technology overhaul."

    It's a totally different thing to have it stop for a whole day after having been in operation for three months.

    So, in conclusion, yes, it's about the platform. .NET, MSSQL 2003, etc aren't robust enough for this kind of job.

    1. Re:Teething problems by UnknowingFool · · Score: 1

      To be fair we don't know the exact reasons TradElec went down. It was mentioned that an upgrade went wrong. It could have been the platform.

      There was talk that the platform was slow and wasn't performing as expected. The outage was just the last straw. Like a poor performing employee, you'd like to replace him/her but until they screw up royally you don't have enough justification for HR.

      Or it could have been working fine but a public embarasment like that needed a scape goat. Someone had to be fired; something had to be changed. Publicly.

      Based on the fact that this was the first foray of MS using .NET into this type of application: high availability, low latency, high performance, it wasn't yet mature enough. Contrasted that to Linux which has been used for years in big iron my opinion is that it was a combination of platform and inexperience.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    2. Re:Teething problems by man_of_mr_e · · Score: 1

      You do realize that it's not .NET, MSSQL and Windows 2003 alone that made up the trading system, right? It was also a boat load of custom developed software.

      There is no evidence to suggest the problems that TradeElect had were directly the problem with the technology it was based on, but more likely due to problems in the custom software developed for it. I'd venture a guess that Microsoft's own web sites, like Hotmail, MSN, etc.. generate more load and have more complexity than TradeElect, and they handle things just fine.

      Likewise, the problem today was unlkely to be a problem with Linux, C++, or any other frameworks or databases they're using. And to claim it was, will be ridiculous.

    3. Re:Teething problems by UnknowingFool · · Score: 1

      True Hotmail may serve more clients per day but the type of load is vastly different. If it takes 3 seconds instead of 2 to load your mail, very few people care. Exchanges like this are all about transactional processing and execution and performance is measured in milliseconds. Volumes are chaotic in nature. On hotmail you might see a small spike during parts of the day but not an order of magnitude jump because of some news announcement. Bottom line MS has little experience with exchanges before this. Their platform simply may not have been ready.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    4. Re:Teething problems by man_of_mr_e · · Score: 1

      That may be, but there is no evidence to suggest it was the basic stack that was causing the problem.

      It may simply be that the TradeElect software itself was not written to withstand the loads that were placed upon it.

      Without a thorough analysis, there is no way for us to know. Therefore, claiming it's the stack is just as valid or invalid as claiming it's the custom software.

      In any event, this is not an either/or situation. Microsoft did not write the TradeElect software, a third party did... Accenture. Although Microsoft was definitely involved, it was ultimately Accenture's baby.

      I thnk what this boils down to is that MilleniumIT is a Trading system company. That's what they do. Accenture is a consulting company, that's what THEY do.

      By that, I mean that MilleniumIT is focused on building the worlds best trading system, and everything they do revolves around that. Accenture is just trying to write software for their specific clients needs, and thus have their hands in many pies.

    5. Re:Teething problems by UnknowingFool · · Score: 1

      Yes we can't really know the exact nature. However the main problem with comparison is the sample size here. I don't know of any other exchange that uses Windows as trading platform. I would say that no one else has even built one on Windows or that Windows is typically not used in cases of high performance is more of an indicator. It's kinda like this: Black & Decker is big name in tools; however, professional contractors don't use that brand as much as they use others like DeWalt. It's not Black & Decker is bad; it's just more designed for consumers. Windows may be fine for email servers or websites; it's not really good for this application.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    6. Re:Teething problems by mangu · · Score: 1

      It's interesting that you use power tools as an analogy. I once made a presentation to managers in my company on "Why Linux?" where I had the following analogy: Windows XP on personal computers is like Black & Decker drills, cheap and easy to use, found everywhere. Windows server is like Makita drills, sturdier, more professional, and more expensive, but very much like an amateur tool on steroids.

      Linux is like an air drill. Most people aren't even aware that such thing exists. But it's the only solution where an industrial strength tool is needed.

    7. Re:Teething problems by objectdisoriented · · Score: 1

      There is not enough public information to make specific conclusions about the contributing factors for this outage.

      We can make specific broad comments about systems that have these type of requirements, performance and otherwise.

      Just as there are platforms that have a security model that make them more (or less) secure than other platforms, there are platforms that are inherently better (or worse) at performance.

      There are message passing schemes that well suited for this type of system.

      There are programming languages that make it easier to develop robust bug-free applications.

      There are systems with built-in high-availability fail-over capabilities (as apposed to a typical multiple vendor, multi tiered "solution").

      I'm sorry, if they were patching the system after 3 months running in parallel, they probably have much more fundamental problems than the application not yet being production ready.

      I think the biggest problem of all is the extreme hubris of vendors and consulting firm who sell the idea that they can apply their products, methodologies, and "industry best practices" (what a load of excrement!) to ANY project, even though they have never attacked a problem in the same class before! We'll have our Super Certified Windbags meet with the other vendors Account Superheros and your Subject Matter Expuds, and we'll have a full project plan and budget on your desk by this Tuesday.

      Bzzt! Wrong!

      The best case is that they simply fail miserably. Worst case is the get it almost-right and go through the outage/patch cycle for the next decade.

      Oh, and for any system that must have near-perfect availability, you want to avoid patching as much as possible. Annually is a nice goal. Every Tuesday, not nice at all. That's begging, pleading, screaming for trouble.

      --
      Performance must be inherent in every aspect of the system. It is not an afterthought, but always thought. - me
    8. Re:Teething problems by DMandPenfold · · Score: 1
    9. Re:Teething problems by smellotron · · Score: 1

      I don't know of any other exchange that uses Windows as trading platform.

      I do.

    10. Re:Teething problems by UnknowingFool · · Score: 1

      If you actually read the case study by MS, it mixes both present and future tense. "has selected" and "will use". Has this thing been built or will it be built?

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    11. Re:Teething problems by smellotron · · Score: 1
      FTFA:

      "Both the Direct Edge and ISE platforms had been built on Windows Server technology," says Bonanno. "It's where our expertise lies. Based on our experience with those systems over five years, we felt that Microsoft had what it took to be a mission-critical, high-performance platform. It had always been a requirement of our earlier systems to handle massive numbers of transactions, and to do so quickly. For example, we built a real-time processing engine on Microsoft technology that published several hundred thousand transaction messages per second. If Microsoft hadn't been able to meet our needs, we'd have looked elsewhere—but there was no need to."

      So both DirectEdge and ISE used MS as a platform prior to the case study, and DirectEdge sounds like happy customers. Amusingly, the lead-in there seems to imply that they chose the platform because of their own knowledge base, not because it was actually better:

      This is the point in the story at which most companies would consider their technology options—but not Direct Edge.

      "Both the Direct Edge and ISE platforms had been built on Windows Server technology," says Bonanno. "It's where our expertise lies. [et cetera]"

    12. Re:Teething problems by UnknowingFool · · Score: 1

      My bad I didn't read case studies from any company themselves but was looking at google articles. Sometimes case studies by companies are somewhat skewed. For example, previously MS presented a case study of how NASDAQ uses their technology in "trading". If you read the press release from MS, you'd think that Windows was the base for their trading platform. However, you read other articles, it was reported that NASDAQ uses Windows for messaging for their trading platform (sending out stock prices like in tickers) and not transactional processing for their trading platform. They actually use a Linux backend for that. If you do a google search on the topic, every link is full of present and future tense. Most of them originated in July 2010 or December 2010. It isn't clear is if it is 100% Windows. Is it? Many companies rely on a variety of technologies and platforms like NASDAQ above.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    13. Re:Teething problems by smellotron · · Score: 1

      It isn't clear is if it is 100% Windows. Is it? Many companies rely on a variety of technologies and platforms like NASDAQ above.

      I would imagine that there are a variety of technologies in use by any exchange. OTOH, DirectEdge is smaller (and newer) than NASDAQ so it's possible that they're a single-platform shop. Their web server identifies itself as Microsoft-IIS/7.0.

    14. Re:Teething problems by man_of_mr_e · · Score: 1

      Again, I see no evidence to suggest that Windows or .NET was where the problem was. You're making generalizations based on speculation.

      Again, I suspect the real issue here is one of market specialization. MilleniumIT made a better product because they specialize in that product. Accenture merely build a product based on the specifications of their client (which may or may not have been what the client actually needed).

      This is one reason I typically recommend against hiring consulting firms to build applications for my clients. Consulting firms are only interested in generating more billable hours, while product based companies are interested in making more sales. Thus, the latter is has more self-interest in creating a better product.

  9. What I was wondering by aliquis · · Score: 2

    Was the consequences of the loss of the automatic trade?

    Everybody won? :D

  10. Proposed patch by Norsefire · · Score: 1

    -- sleep(life_the_universe_and_everything);

  11. The Unix Philosophy by mangu · · Score: 1

    That being said, Linux is just a better platform to build something like this on. Sure, you can do it with Windows and make it work, but it's just more and unnecessarily difficult.

    I keep hearing this, but never see any technical details. Why is this so?

    I had extensive Windows programming experience before I switched to Linux about ten years ago and, basically, it boils down to the Unix philosophy.

    Linux is a system made by programmers for programmers. Windows is a system made following directions of the marketing department.

  12. Simple by snookerhog · · Score: 1

    42

    1. Re:Simple by atisss · · Score: 1

      Six by nine

  13. What hour ? by psergiu · · Score: 1

    4:30. What were the sys admins doing 10 minutes before this ? ;-)

    --
    1% APY, No fees, Online Bank https://captl1.co/2uIErYq Don't let your $$$ sit in a no-interest acct.
  14. Re:TradElect & InfoLect lasted longer... apk by Americano · · Score: 1

    Remember folks, you heard it here first:

    APK says Windows is better than Linux.

  15. Cherry picking much, Mr. Ballmer? by mangu · · Score: 1

    That's NOT what London Stock Exchange reps said:
    "LSE denied that the collapse was TradElect's fault" - http://blogs.computerworld.com/london_stock_exchange_to_abandon_failed_windows_platform by Stephen J. Vaughn-Nichols

    Let's quote from your own link. First of all, the line you put between quotes was entirely out of context, because you omitted the initial "while" and the following lines:

    "they also refused to explain what the problem really was. Sources at the LSE tell me to this day that the problem was with TradElect.

    Since then, the CEO that brought TradElect to the LSE, Clara Furse, has left without saying why she was leaving. Sources in the City-London's equivalent of New York City's Wall Street--tell me that TradElect's failure was the final straw for her tenure. The new CEO, Xavier Rolet, is reported to have immediately decided to put an end to TradElect."

    You also failed to mention that the article's title was "London Stock Exchange to abandon failed Windows platform" and that the author concluded:

    "So, might I suggest to the LSE that they consider Linux as the foundation for their next stock software infrastructure? After all, besides working well for Chi-X, Linux seems to be doing quite nicely for the CME (Chicago Mercantile Exchange), the NYSE (New York Stock Exchange), etc., etc."

    1. Re:Cherry picking much, Mr. Ballmer? by UnknowingFool · · Score: 1

      Interesting that you are actually trying to argue with apk. Having some experience here, let me tell you how this will go. You might say something completely innocuous like:

      "I like vanilla."

      Pretty soon you will get a 3000 word ramble from an AC about how chocolate is good for you and how Thalian chocolate is the best since they age the beans 400 years. Then you'll get a 4000 word diatribe from an AC about how chocolate is bad for you because he's an expert in Thalian chocolate. If you're lucky, he'll accidentally sign his name to both of them otherwise you'd wonder why any two different ACs would bother posting rambling filibusters similarly written but completely contradictory and irrelevant to the topic and whether it was the same person.

      Should you feel to respond and actually point out important facts like Thalian chocolate only exists in Star Trek and it technically isn't real, he'll search for every post and website you may have ever visited until he finds that post you made three years ago where you said:

      "You know chocolate isn't that bad a color for the Zune.

      You might point out your handle is "mangu" and not "manguts" and that wasn't you at all. Nevertheless, he'll take your post as proof that you were wrong about MS and the Zune. Fortunately for you, slashdot closes these threads after a while or he would go on and on in this thread not for weeks, not for months but for years about how your post on vanilla completely supports his assertion that 32-bit will win out against 64-bit, how the HOSTS file is the solution to all things security, and how ramdisks will solve any performance problem. Unfortunately for you, he'll can follow you on slashdot as AC and respond to every post from now until eternity how wrong you were about Thalian chocolate.

      Now you may wonder later how this all got started. It's rather simple. He disagrees with you; therefore he must destroy you.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
  16. Linux MilleniumIT system @ LSE down, 2x now? LOL! by Anonymous Coward · · Score: 1

    "Sources at the LSE tell me to this day that the problem was with TradElect." - by mangu (126918) on Thursday February 17, @10:03AM (#35232010)

    What 'sources' are those? The "horses' mouth" said otherwise:

    "LSE denied that the collapse was TradElect's fault" - http://blogs.computerworld.com/london_stock_exchange_to_abandon_failed_windows_platform [computerworld.com] by Stephen J. Vaughn-Nichols

    (OR, can't you read?)

    ---

    "You also failed to mention that the article's title was "London Stock Exchange to abandon failed Windows platform" and that the author concluded: "So, might I suggest to the LSE that they consider Linux as the foundation for their next stock software infrastructure? After all, besides working well for Chi-X, Linux seems to be doing quite nicely for the CME (Chicago Mercantile Exchange), the NYSE (New York Stock Exchange), etc., etc." - by mangu (126918) on Thursday February 17, @10:03AM (#35232010)

    LMAO - they abandoned a platform that ran SOLID for them for 2 yrs. time... and for what? A Linux one that has crashed 2x now in less than 1 yr.!

    (Facts are FACTS, and today's article substantiates my statement above in reply to yours, easily!)

    I have noted that Linux does the job elsewhere, just fine, myself -> http://linux.slashdot.org/comments.pl?sid=1995760&cid=35212722 , so, you can't pull the "you're not being fair/complete" b.s. on me...

    (I just tell it how it is, and I use backing documentation... like the above quote, for example, FROM THE HORSES' MOUTH LSE (not some "sources" who remained unnamed - that's NOT "good enough" for me!))

    APK

    P.S.=> Still, the fact remains - LINUX WENT "DOWN FOR THE COUNT" IN LESS THAN 1/2 a YEAR ON THE LSE! and, it's NOT THE FIRST TIME EITHER boys, so... "read 'em & weep" + "argue with the facts" I put out, not myself... apk

  17. What's a closing auction ? by doperative · · Score: 1

    "Due to a technical problem, that system sent out the daily automatic message to clients - stating that the closing auction was beginning seconds late"

  18. Re:Linux @ LSE, down 2x in less than 1 yr. - FACT! by Americano · · Score: 1

    Cool story, Bro.

  19. 42? by molecular · · Score: 1

    well there we have an answer to a popular question

  20. why you don't need to worry about HST by Thud457 · · Score: 1

    High speed trading is just the froth on top of the market -- a million little bubbles.
    Granted, it increases costs overall for everybody in the market, but those costs should be spread evenly. This shouldn't greatly adversely impact long term investors.
    But it does still amount to an privileged class siphoning value out of the system. That's bad.

    --

    the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff

  21. Re:Americano goes off topic? LMAO! by Americano · · Score: 1

    Y U MAD BRAH?

  22. Re:Americano goes off topic? LMAO! by Americano · · Score: 1

    Don't worry, I'm on the case. Keep your hosts files updated, and your Windows recovery disk warm.

  23. Re:It's not ME, it's the "dancing penguins", lmao! by Americano · · Score: 1

    One trick pony only has one trick. :(

  24. Re:Your use of all CAPS indicates you are mad by Americano · · Score: 1

    Oh APK, you are amazing.

  25. Re:1 "trick" (truth) is all one needs vs. trolls by Americano · · Score: 1

    tl;dr. Please try formulating a coherent point before posting.

  26. Re:versace sunglasses by atisss · · Score: 1

    thanks, now i'll be hitting in face anybody who wears those frackin sunglasses

  27. Re:American STILL "off-topic"/adhominem trolling? by Americano · · Score: 1

    Damn. You have uncovered my secret: I'm part of Linus Torvalds' secret Linux Astroturf Brigade.

    You win twelve internets for your insight.

  28. Re:Try being on topic, you off topic TROLL, lol! by Americano · · Score: 1

    It's not an ad hominem attack, you fool. An ad hominem attack would take this form:

    "APK is an unstable psychotic. We may therefore conclude that all of his points are invalid."

    I am asking you to formulate a coherent point, rather than your rambling mess of capitals, emphatic text, and "PS" addenda, which is both physically painful to read, and logically incoherent.

  29. Re:It surely looks like an adhominem attack from U by Americano · · Score: 1

    An ad hominem attack is the logical fallacy of using negative statements about your opponent to discredit their arguments.

    As I said, "APK is a psychotic. Therefore his points are wrong." Is an ad hominem attack. And it is a fallacy, because whether or not you are a psychotic isn't relevant to whether or not your argument holds water.

    Calling you a 1-trick pony is an observation of your behavior: You flog the same snippets of text repeatedly with a rambling mish-mash of capital letters, bold typefaces, and other assorted walls of text. In other words - you make arguments that are incomprehensible to the reader, and only serve to undermine the strength of any point you try to make.

  30. Re:tl;dr as repeatedly off topic + "DEFINITION TIM by Americano · · Score: 1

    From your own link:

    "An ad hominem (Latin: "to the man"), also known as argumentum ad hominem, is an attempt to link the validity of a premise to a characteristic or belief of the person advocating the premise."

    In other words: If I were trying to dismiss your concerns by calling you a one trick pony, that would be an ad hominem argument.

    Since I have simply observed that all you do is flog the same points with poor grammar and rambling and incoherent walls of text, that is not an ad hominem argument. It is simply observing the fact that that is all you do, and it doesn't matter whether your points are relevant or not, because you do such a horrible job of FORMULATING them that nobody can even tell what your point is. You're just cluttering up Slashdot with pointless self-promotion.

    Name calling is, technically speaking, name calling: It is not a logical fallacy to call you a self-aggrandizing ass, it is simply an observation of your behavior, followed by me stating my opinion of it.

  31. Re:You're still off topic? tl:dr by Americano · · Score: 1

    APK, why do you speak of yourself in the third person in these thinly veiled attempts at sock puppetry?

    Noting that you're a self-aggrandizing bore is simply an observation. It is not an ad hominem attack, because frankly, I'm not arguing with any of your points, I'm simply pointing out that you love to repeat yourself!

    Y U MAD, BRAH?

  32. Re:Not attacking my points & me instead IS Ad by Americano · · Score: 1

    APK, I'll explain this to you one last time in small words with few syllables: for me to be conducting an "ad hominem" attack, I'd have to be saying something negative about you, *and then* suggesting that that negative trait is why your points are wrong and invalid. Name-calling and getting you all lathered with frustration like this is what we call "just a bit of fun," and has no bearing on whether or not your points are true, false, or completely disconnected from all reality.

    You're the one who went off topic with your diatribe about SQL Server versions and "windows being better."

    Fail harder though, seriously.

  33. Re:AmericanoTobacco: 4 trolls who get "smoked"! by Americano · · Score: 1

    Yawn.

    You're boring me, APK.

    1) Learn what constitutes an ad hominem argument. The definition is very clear, and you haven't been subjected to one.
    2) Attacking you? Hardly. I've simply called you some names and poked fun. If I were attacking you, you'd know it.

  34. Re:Are you threatening me? Answer that... apk by Americano · · Score: 1

    I see you're able to quote the definition I gave you - good for you! Next, perhaps you can work on understanding what all those big words mean.

    If I were attacking you, you'd know it. And not even your precious hosts file would help you!

  35. Re:Testing? No, No, No, No, No by objectdisoriented · · Score: 1

    Testing can only prove the presence of bugs, not the absence.

    It doesn't matter if you have 10,000 test cases or 10,000 times as many. Best case is you catch the bugs you were able to anticipate. And if you think you will be able to simulate real-life loads and traffic patterns of your first year in production, well, good luck with that.

    If you are relying on testing to deliver "bug free" systems, you've already lost half the battle.

    Do yourself a favor and dump most of your "test case" staff and hire yourself some real engineers. You'll find you didn't need those folks and have higher quality systems.

    --
    Performance must be inherent in every aspect of the system. It is not an afterthought, but always thought. - me
  36. UPDATE: Anger mounts over LSE price data errors by DMandPenfold · · Score: 1
  37. Re:"YOU gave me"? What a line of bullshit! by Americano · · Score: 1

    You provided a link to the definition of ad hominem, which surprisingly, you appear incapable of comprehending.

    I quoted the relevant definition of exactly what it was (not just a link). I win, you lose.

    And why do you insist on asking if I'm making threats? I told you I wasn't, because if I was, you'd know it. Since you claim to not know whether or not I am, the answer, by process of deduction, is that I have not made any threats.

    Really, APK. Elementary logic. You should read a book sometime.

  38. Re:Have you even taken LOGIC, Americano? by Americano · · Score: 1

    APK, stop with the sock puppets. It does nothing to bolster your arguments, just makes you look like a foolish child, desperate for attention.

    (And as far as hiding... who, exactly, is hiding behind an "Anonymous Coward" label here? Yeah... I thought so.)

  39. Re:A douchebag says what? by Americano · · Score: 1

    APK, you seem intent on viewing every response of mine as a threat. Is it because you're a paranoid schizophrenic? Seek help.

    We all know that you've got a hosts file and you're not afraid to use it!

    But I'll do you a favor - my ip is 127.0.0.1 - please block me with your hosts file. Also, you probably want to block out a system named "localhost," that's sort of my thing, I use that identifier.

  40. Re:Threatening to attack my system? What's my IP?? by Americano · · Score: 1

    Again with the paranoid delusions.

    You must live in such a scary world.

  41. Re:Show us your PHD in Psychiatry? How about CSC?? by Americano · · Score: 1

    Don't need a PhD in psychiatry to diagnose you. Just good old-fashioned common sense, girlfriend.

    APK == OFF TOPIC TROLL!

  42. Re:Show us a PHD in Psychiatry & proof you too by Americano · · Score: 1

    Prognosis is a term to describe a prediction of the "likely outcome of an illness." Diagnosis is a determination of what your illness is.

    What I offered was a speculative diagnosis, based on my observations of your behavior here. My prognosis is that you will die alone, unloved, probably of liver failure because you'll turn to alcohol to fill that deep void in your life that Slashdot and your Hosts file can't fill.

    See the difference?

    APK == OFF TOPIC TROLL!

  43. Re:Prove you've taken LOGIC Americano (lol, U can' by Americano · · Score: 1

    If a college degree is what gives you your mangled interpretations of "ad hominem," "prognosis," and other standard terms, I'm glad I'm not a graduate of whatever backwater troll factory you call your alma mater.

    I do, in fact, have a college degree. You're wrong on all counts, but you are providing a remarkable demonstration of ad hominem attacks on me, and also, an effective case study in pissing into the wind. I'll let you know when I write you up in JAMA, the specifics of your case are fascinating.

    APK == OFF TOPIC TROLL!

  44. Re:OK, WHAT IS MY IP ADDRESS BIG TALKER? by Americano · · Score: 1

    APK == OFF TOPIC TROLL.

  45. Re:Americano = another degreeless douche wannabe by Americano · · Score: 1

    I have no need to prove myself to you, you troll.

  46. Re:Prove you have a degree then, douche by Americano · · Score: 1

    "We"? What, you got a mouse in your pocket now?

    You don't know what an ad hominem argument is, you don't know what the difference between "diagnosis" and "prognosis" are, and you clearly have issues.

    Cry more troll. Cry more.

  47. Re:Americano running from proving he took logic? by Americano · · Score: 1

    Now you're just flapping your gums to hear yourself talk. Hope you enjoy it, you off-topic troll!

  48. Re:Americano FAILS at producing PROOFS? LMAO! by Americano · · Score: 1

    Wow, 2 posts that say the same thing, but which obviously aren't copies of one another. Are you having a psychotic break?

  49. Re:What's my IP Address, big talking Mac Fag? by Americano · · Score: 1

    BOOOOOOORING.

  50. Re:Americano prove to us you have a degree by Americano · · Score: 1

    Or it could be that I have nothing to prove to a raving lunatic whose sole source of excitement appears to be talking about how he edited a hosts file once.

    Fail more, APK.

  51. Re:Prove you have a college degree Mac Fag by Americano · · Score: 1

    YAWN.

    You finished?

  52. Re:You fail at proving your own words by Americano · · Score: 1

    I have nothing to prove to you. And why are you so insistent on me being a "fag" - hoping to score? Sorry to disappoint you, APK, but, psychotic and male isn't exactly my type.

  53. Re:Americano LIED about having a college degree? by Americano · · Score: 1

    No, I will not date you.

  54. Re:Americano LIED about having a college degree? L by Americano · · Score: 1

    you just want my name so you can look me up and try to seduce me. No thanks, brah.

  55. Re:LMAO yes you do (prove U took logic, & your by Americano · · Score: 1

    Sorry to disappoint, bro. Got a date, you'll have to amuse yourself for the rest of the night!