Update on SuperK Detector Failure

This note came in from Director Totsuka to the press and other scientists. Hemos and I felt it deserved more than just a regular SlashBack reference, as we feel that this is an important project. (I belive this comes form a translation from japanese, so forgive the errors) this is an update to the original post on the Super-K malfunction.

As a director of the Kamioka Observatory, which owns and is responsible to operate and maintain the Super-Kamiokande detector, it is really sad that I have to announce the severe accident that occurred on November 12 and damaged the significant part of the detector. The cause and how to deal with the lo ss in future will be discussed by newly found committees. However, even before discussing with my colleagues of the Super-K and K2K collaborations, I have decided to express my intension on behalf of the staff of the Kamioka Observatory.

We will rebuild the detector. There is no question. The strategy may be the following two steps, which will be proposed and discussed by my colleagues.

• 1. Quick restart of the K2K experiment.
  • (1) We will clear the safety measures which may be suggested by the committees.
  • (2) reduce the number density of the photomultiplier tubes by about a half.
  • (3) use the existing resources.
  • (4) resume the K2K experiment as soon as possible; the goal may be within one year.
  2. Preparation for the JHF-Kamioka experiment.
  • (1) Restore the full Super-Kamiokande detector armed with the state-of-the-art techniques.
  • (2) The detector will be ready by the time of the commissioning of the JHF machine.

To achieve our objective is formidable but we are determined to do so. But we certainly need your encouragement, advice and help. I should appreciate it very much if you could support our effort as you have kindly done so before.

Best regards,
Yoji Totsuka
director, Kamioka Observatory
On behalf of the Kamioka Observatory staff

Comments

Boy, that clears that up.

[Bowie J. Poag]

Still no formal explanation..This is beginning to sound an awful lot like, "Dad, I totalled the car..A telephone pole jumped infront of my car, and I couldn't swerve around it in time! Honest, Dad!!"

Something tells me these guys made a titanically stupid mistake, and they're afraid of letting the cat out of the bag before they have a chance to circle the wagons and defend their multi-million dollar "oops".. See, its kinda hard to rebuild the detector when your funds have been cut due to findings of gross negligence.

Again, I move we refer to it as the "Special K" detector from now on. :)

Cheers,

Re:Boy, that clears that up.

[N9VLS]

It seems more likely that the PMTs weren't manufactured to spec, and the enclosures failed as a result of pressure during the refill.

Gross negligence? Doubtful.
You don't get to spend that kind of money without at least pretending to account for possible problems. Thing is, no one expects the fscking detectors to implode like this....

Re:Boy, that clears that up.

[Performer Guy]

How about they refilled too quickly, or never equalized the pressure in the sensors. It seems clear that SOMETHING went wrong here, that detector used to be filled with heavy water, second time around the enclosures implode. It beggs the question, what did they do wrong the second time that they did right before? It seems foolish to suggest that the detector enclosures were not up to spec. Besides, if they weren't they should have known about it. It would be simple to pressurize the enclosures, no need to withstand the many atmospheres of pressure at the bottom of that tank.

Re:Boy, that clears that up.

[markmoss]

It would be simple to pressurize the enclosures A photomultiplier is a vacuum tube, 20 inches long in this case. It can't be pressurized, and one end has to be transparent. So whether it's the tube itself or a glass enclosure around it, you've got to have a big glass vessel that can support the pressure. Glass is actually pretty strong in compression, so if the shape is rounded for arch-like load transfer (a sphere or a cylinder with rounded ends), the tubes would hold up quite well to static pressure. Shock is a whole different matter.

But when you have 21,000 pieces of glass, it shouldn't come as a surprise when one gets broken. Why didn't they have baffles or shock-resistant enclosures around the tubes to prevent chain-reactions?

Re:Boy, that clears that up.

[Performer Guy]

There are large enclosures which house these tubes, they aren't just floating in the drink. The load over a small tube is much less that that over a large enclosure.

And we're not talking about one breaking here, there are clearly large numbers which broke, by the sounds of it around half.

Re:Boy, that clears that up.

[Eric Sharkey]

There is *no* enclosure around the tubes. They *are* just floating in the drink. Well, not really floating. There are metal straps which wrap around them and hold them down. Since they're mostly vaccuum inside, they're much less dense then water and would float to the surface very quickly if you let them go. The theory is that the shockwave hit them at such an angle that they twisted and it was these metal straps which held them too tightly and caused enough pressure at fairly narrow places to cause them to implode.

Re:Boy, that clears that up.

[darkonc]

It seems more likely that the PMTs weren't manufactured to spec, and the enclosures failed as a result of pressure during the refill.

I'm guessing that it was only one unit which either wasn't manufactured to spec, or got hit by a worker during refill -- then the shock wave from the implosion caused a couple of it's neighbours to implode

"Then they told two friends, and they told two friends, and so on and so on....",

The problem appears to center around the fact that all of the tubes were interconnected. Thi mad it posssible to drain and refill the thousands of tubes without taking a really long time. Unfortunately, there wasn't anything (or, at least, enough) set up to dampen the shockwave in the case of a failure. It's the kind of thing that's obvious in hindsight, but only after you think of it, or it happens...

It's kinda like using box cutters to hijack a plane and use it as a suicide bomb.... We never really considered the possibility of 20 suicidal hijackers getting together to create mayhem with office implements.

Repair estimates top $30M

[Camel Pilot]

It has been estimated that the US will spend $100 Billion in a year pursuing justice in the hinterlands of Afghanistan. That comes out to approximately $300 million per day! Or we will spend what is required to fix the SuperK in 12 hours. Kind of puts things in perspective.

Re:Repair estimates top $30M

[PD]

Can't do much science when people can't safely work in office buildings or post offices.

War is immoral, and spending outrageous amounts of money on war is immoral, but it is even more immoral to ignore evil.

Hope THAT puts things into perspective.

Re:Repair estimates top $30M

[Zeinfeld]

It has been estimated that the US will spend $100 Billion in a year pursuing justice in the hinterlands of Afghanistan. That comes out to approximately $300 million per day! Or we will spend what is required to fix the SuperK in 12 hours. Kind of puts things in perspective.

The estimates reported by the BBC are a billion a year. The '100 billion' figure is the amount of corporate welfare the Republican party wants to ram through Congress under the pretense it is a stimulus package.

Before anyone gives these folk any more money they should be able to explain why the previous detectors went pop. Otherwise there is every chance the replacements will fail in exactly the same way.

$30 million is a pretty large chunk of change to lose. For the same money you could fund an awful lot of interesting Comp Sci research.

Re:Repair estimates top $30M

[Camel Pilot]

It all depends who you believe and what they count. The $100B was mention on CNN a while back. But I did find this link which gives a good cost comparision of this and other wars. A quote from the reference link:

"Because of its global scale and long-term nature, the war on terrorism probably will cost more than the Persian Gulf War, which totaled about $80 billion in constant fiscal-year 2002 dollars"

BTW, we are little quick to call someone stupid are we not. You should show a little restraint. I am sorry i accidently fat fingered the 1 and 2.

Re:Repair estimates top $30M

[astrophysics]

It's estimated to take 2 years to replace the PMTs. That's a matter of logistics not funding.

Re:Repair estimates top $30M

[brunes69]

BBC huh? you sure thats not A British billion? In America, a Billion generally means a thousand million. (1x10^9) In England, a billion means a million million (1x10^12). So, if the BBC is reporting it as 1 billion dollars, that is really 1 trillion dollars in American terms. Now, Im not saying that it is either of these, maybe somehwer in between?

Re:Repair estimates top $30M

[j-beda]

Do I keep one on my person at almost all times? Yes... and I think you are a coward if you don't.

This makes one a coward how?

Now in your mind it is safer to carry a gun than to not do so, but generally doing the more dangerous thing is not what defines a coward. In fact, it is usually the opposite.

A "classic" coward is unable to uphold their beliefs due to fear. They either act (or fail to act) in a way consistent with their beliefs because they are afraid of the consequences of that action (or not performing that action).

It seems to me that not carrying a gun would only be cowardice if the person thought that they *should* carry a gun from some philosophical reason, and yet did not due to fear. Generally this is not the case. I would say that the majority of people involved in this type of debate who do not carry guns have a philosophical reason NOT to carry a gun. Thus pointing out the danger of that decision demonstrates bravery rather than cowardice. Similarly, pointing out the dangers of carrying a gun to someone who thinks that gun possession is philosophically an important issue, might highlight the bravery of the individual in that case.

Granted, one or the other (or both for that matter) of these philosophical positions could be based on ignorance and stupidity, but nobody ever said that stupid people could not be brave.

But to get back on topic - I think it is a shame that we are so willing to invest in military strength and in environmentally unfriendly governments, businesses and practices and all those sorts of "bad" types of things and yet we spend such a relatively small amount on basic research.

Re:Repair estimates top $30M

[Richy_T]

FWIW, billion=1000 million has pretty much become standard usage in the UK. I think the milliard is on its way out.

Rich

Re:Repair estimates top $30M

[PD]

I'm a liberal. I own no weapons. I oppose evil instead of being evil.

Re:Repair estimates top $30M

[blair1q]

And I spent $7.77 on a cheesesteak, fries, and a coke for lunch today.

--Blair
"And the student was enlightened."

Japanese Engrish

[Robber Baron]

I know, I know, I'll probably get modded to hell for this but I've got the points to burn and simply can't resist giving in to the Dark Side and Posting a link to this site.

Re:Japanese Engrish

[DNS-and-BIND]

Wow, racist humor. I sure thought that stuff was dead and buried, but thanks for sharing!

Re:So...

[spaceyhackerlady]

Just what is the SuperK?

A detector for neutrinos. Have a look at their web page.

I attended a talk last night by one of the scientists from the Sudbury neutrino detector. One of their Big Issues at the moment is figuring out why all the best neutrino detectors only pick up a fraction of the neutrinos predicted by all the best theories on the innards of stars.

...laura

Wait how is this gonna work?

[jgaynor]

(2) reduce the number density of the photomultiplier tubes by about a half.

If they can up and cut the number of sensors in half will they still detect the "blue streak" of the Nuetrino if one happens to pass through? If so why were their that many photosensors in it in the first place?

Additionally - the tank will again be flooded with the same amount of water, and correspondingly, water pressure. With only half the amount of sensors - wont these sensors each have more pressure placed on them? Wasnt a collapse because of water pressure what caused the initial sensor implosion chain reaction?

This seems like a real cut-throat solution, I wish there was more of an explanation than just a few lines . . . Good to hear they're rebuilding though.

Re:Wait how is this gonna work?

[brainboyz]

Using double the desity of Photomultiplier Tubes allows them to get a better resolution picture of the energy released when the nutrino passes through. They won't get half the pictures, but they'll see them half as well.

It's a good solution for the time being because at least they can take pictures. If they waited until longer to get all the PMTs replaced, then they'd have less pictures overall instead of less resolution for a short period of time.

Re:Wait how is this gonna work?

[DerekLyons]

the tank will again be flooded with the same amount of water, and correspondingly, water pressure. With only half the amount of sensors - wont these sensors each have more pressure placed on them?

No, the pressure on a sensor tube is a function of the depth of water at the sensor, not of the number of tubes in the array.

Wasnt a collapse because of water pressure what caused the initial sensor implosion chain reaction?

That's the purpose of reducing the amount of sensors in the array. Increasing the spacing will reduce the chances of another chain reaction. (The strength of the shock wave falls off according to the square power law. (IIRC)) Array sensitivity will suffer a hit, but loss of half the dectectors does not always mean loss of half the capability. I suspect that angular resolution will suffer more than the absolute detection threshold.

Re:Wait how is this gonna work?

[dragons_flight]

Yes they will detect it. What you lose by reducing sensors is resolution as to direction and energy.

It's actually rather unlike that they'll miss nuetrino events because of such a change. I've had the oppurtunity to look at individual event plots and raw data, and the Cerenkov light from a single event actually registers in a considerable fraction of the tank. IIRC, typically 5-30% of detectors see each event.

They use the timing of when each detector becomes active to reconstruct the path and speed of the particle generating the light. So fewer PMT tubes means less accuracy in determining the direction and energy of the nuetrino that produced the event. I would guess that it's not the case that half as many tubes means half the accuracy. If I were to make an estimate I'd say you're probably increasing the error on individual measurements by around 30-60% (as opposed to 100%, if it were doubled). This is most important on electron nuetrino events which were somewhat hard to accurately determine to begin with, compared to their muonic cousins.

With only half the amount of sensors - wont these sensors each have more pressure placed on them?

No. Hydrodynamics doesn't work that way.

Wasnt a collapse because of water pressure what caused the initial sensor implosion chain reaction?

Well the machine worked successfully for several years at the same amount of pressure, so this shouldn't be the initial cause of the accident. However it is entirely likely that the pressure facillitated the disasterous chain reaction once some faulty equipment or human error got it started.

This is an exotic size of tube and most of the replacements will have to be manufactured (which takes time), so this is probably the best solution we can expect in the near term.

Re:Wait how is this gonna work?

[astrophysics]

Since the PMTs have a vacumb inside, when the tubes are tightly packed, there's no (little) pressure tangential to the radius vector. I'd guess they'll put in "dummy detectors" or something to fill the space and keep the water from getting anywhere but on one face of the detectors.

Re:Wait how is this gonna work?

[Brett Viren]

No. Hydrodynamics doesn't work that way.

Well, at least hydrostatics doesn't, which is the useful discipline in expressing water pressure as a function of depth.

You are both half wrong/right. The pressure at a certain depth is hydrostatic, but the implosion cascade is a hydrodynamic effect caused by the pressure wave from the first imploded PMT producing a pressure differential (and thus a net force) across the neighbor PMTs. If we assume a pressure differential of 1 atmoshphere, this translates more than 2 tons of net force (not balanced hydrostaic force) on the PMT. Those PMTs are extreamly strong, but drop a car on one and they will break.

Re:Wait how is this gonna work?

[Eric Sharkey]

It's actually rather unlike that they'll miss nuetrino events because of such a change. I've had the oppurtunity to look at individual event plots and raw data, and the Cerenkov light from a single event actually registers in a considerable fraction of the tank. IIRC, typically 5-30% of detectors see each event.

This isn't entirely true. It depends a lot on the type of event. The pictures you probably saw were either of an atmospheric neutrino event, a cosmic ray background event, or one of the K2K events. In all of these cases, the particle in the detector will have somewhere between 100 and 5000 MeV of energy (and in some cases more).

As a particle travels through matter, it loses energy as it goes. The more energy it has to start with, the longer it will go before it stops. A general rule of thumb is that a muon travelling through water loses 2MeV for every centimeter travelled. So a 100MeV muon produced by a neutrino interaction would travel for 50cm. The higher the energy, the longer the track, the longer the track, the more light produced, the more light produced, the easier it is to see.

Very high energy cosmic ray muons will produce so much light that every tube in the detector will register a hit. A typical high energy event picture is here. Would you see the pattern with half as many pixels? Of course.

The problem is that "solar neutrinos", neutrinos which come from the sun, typically have much lower energies. (Only 1-10 MeV) So low, that even before the accident, SK would miss most of them (anything below 5MeV) because they just didn't produce enough light to be distinguishable from random noise in the dector or the decay of stray radon particles. If you look at pictures of these events, normally you can't see anything by eye. There's just a few photons (5-10) which are recorded which can only be identified as a real event by their timing because you can triangulate back to a single point based on their arrival time at the PMTs. Solar neutrino physicists rarely post event display photos because there's so little to see in them. Even then, it's hard to distinguish solar neutrino events from noise. In fact, it's not possible to identify an individual event as a solar neutrino event and not a radon event that looks like a solar neutrino event. It can only be done statistically. (Radon events don't point in any particular direction, while solar events all come from the sun, so you can compare the number of events coming from the sun and the number of events apparently coming from other directions and do a background subtraction.)

It's these types of events which will be hurt most by the loss of the extra tubes.

Somewhere in the Japan....

[zulux]

Mr. Tanaka: You have failed the SuperK - Dr. E.! Our German contacts are not pleased with the latest ramifications of the 'device.'
Dr. E.: Wah! But the Gaia force was in alignment, this can not be!
Mr. Tanaka: Your latest failure is being undue attention to our cause.
Dr. E.: Wah! But Pretty-Girl likes to SCUBA in the detector. Makes fresh-wind in water and boom - becomes divine-wind chain reaction.
Mr X.: Doctor, your failure is now at hand!
Dr. E: Wah! I give my body to the Emperor! Pretty-Girl, be saying Goodbye! (Slice) (Slice)

No need to ask forgiveness

[The Pim]

I belive [sic] this comes form [sic] a translation from japanese, so forgive the errors

Don't be ashamed, Chris! We're quite used--indeed endeared--to the editors' barely intelligible brand of English. For Taco, that would be a good post.

Oh, you meant the quoted part ...

The world revolves around the sun??

[pjbass]

I just got through reading through the description of the super-k and what it is supposed to do, and found myself hitting dictionary.com quite a bit. I was very impressed and excited about this project, but came across something that I read a few times and it still doesn't make any sense to me:

If the problem of solar neutrinos would be caused by the oscillation of neutrinos, it is predicted that the number of solar neutrinos is
different in the day and at night ; however, there is not much difference in intensity of
solar neutrinos between the day and night.

So the assertion (or hypothosis) is that the amount of neutrinos emitted from the sun's core is different during night than day?? If I'm missing something, please someone let me know. I find this difficult to understand, since the sun really doesn't give a damn what earth is doing, especially when you're talking about night in Japan vs. night in America. I honestly welcome clarification on this if anyone has any. Thanks!!

Re:The world revolves around the sun??

[merlin_jim]

Actually, according to current theory, neutrinos pass through normal matter without interacting with it (much)... that's why you need such a big and delicate detector to find them at all; they are practically inert as far as normal matter is concerned.

So, I don't know why this statement was made, except to say that the neutrino's rate of interaction with matter is one of the more hotly debated questions in science right now. Its rate of interaction and its mass are two variables that play an important role in all of cosmology (for details, lookup Mach's theory, Ober's Sky, Missing Mass, and the Omega constant, sometimes referred to as the Hubble constant)

Re:The world revolves around the sun??

[Brand X]

IANAP but i believe the oscillation of neutrinos refers to the theory that they switch forms as they travel through space.

I used to be a student of one of the American Physicists more heavilly involved in Super-K. Spent a lot of time learning this theory. You have it mostly right. They don't so much switch forms (transforming from one to another) as oscillate between mixed states. The particle in question ends up being, say, 90% electron neutrino (I'm pulling the numbers out of the air here), 8% mu neutrino, and 2% tau neutrino, in one of the theories. (Another one has a neutral charge neutrino in the mix too...) and the mixed state travels as a single packet of quantum potential. If it interacts at a particular time, the chances of the interaction being muon is X, electron being Y, etc. Except that if one of the neutrino types has mass, the packet is moving at a finite, though high, fraction of the speed of light, and the probability of interaction is affected by the phase of the different particles in the potential with respect to one another (which would be uniform if they were all massless and moving at the speed of light), and the phase is further affected by medium of transmission (matter versus vaccuum), which is why there is a statistical variation due to time of day and day of year... different distances traveled in different seasons, and different amount of matter traveled through. Course, although IUTBAP, IANCAP. This is from distant memory echoing through fog and cobwebs. YMMV.

Night and Day

[MarkusQ]

So the assertion (or hypothosis) is that the amount of neutrinos emitted from the sun's core is different during night than day??

No, the same number are emitted, but if they have to travel through the bulk of the earth before reaching the detector, it will effect how many you detect. That's true of photons too (you see a lot more of them durring the day, even though the sun emits at a ~constant rate), but here it is even more interesting; the neutrinos aren't being absorbed by the earth, they are being converted between two forms, one of which is easier for a particular detector to detect. So you can wind up detecting more at night!

--MarkusQ

Re:Night and Day

[j-beda]

Generally, I think that this analysis is correct, the size of the earth compared to the radius of the orbit is pretty small. In fact I think that the eleptical deviations of the earth's orbit are larger than the size of the earth. Additionally, the volume of the sun from which the neutrinos are created is probably larger than the volume of the earth, so we already have a pretty large uncertainty in knowing where the neutrinos each were created in the first place.

However this does not necessarily mean that the earth has no effect on the situation. The detector itself might not be symmetric - maybe it is more or less accurate when pointing down compared to pointing up? Theories involving neutrino oscillations might have measurably different rates when the neutrinos pass through the earth than when they pass through empty space.

Re:Night and Day

[krlynch]

As you pointed out, the size and shape of the earth is miniscule compared to variations in the orbit. However, if neutrinos oscillate between different mass eigenstates during their travels, then you would expect the oscillation rate inside matter (i.e. the earth) to be different from the rate of oscillation outside matter (i.e. the vacuum of space), and from knowledge of the oscillation rate in one, you can predict the oscillation rate in the other (well, given some other information that doesn't concern us here...). The day/night variation would arise because the neutrinos from the sun that are detected during the day travel through less matter than do the neutrinos at night, which have to travel all the way through the earth first, and hence would have reached a different point in their oscillation. You would also expect to see a seasonal variation as the earth moves radially toward or away from the sun during the year, because the distance over which the neutrinos oscillate would change. Finally, from knowledge of the matter effects, you would also predict a difference in the number of neutrino events passing through the detector from teh top and from the bottom. It was actually the last effect which was the evidence for oscillations, if I'm not mistaken.

Re:Night and Day

[MarkusQ]

Since an observatory is only looking at a small percentage of neutrinos on a relatively thin path between here and sun's core, I don't think you could establish any day/night difference, even with years of observations.

To quote a teacher I once had, I would agree with you if not for the fact that you're wrong. The day/night neutino detection cycle isn't a theoretic effect they are trying to observe, it's an experimental effect they are trying to explain. The present best explanation is that they switch between two families (muon & tau) with different masses.

-- MarkusQ

Re:If I understand this correctly...

[Tsar]

Super-K is basically a huge underground cylindrical tank, about 40 meters wide and 40 meters deep, containing 50,000 tons of very nearly pure water. The sides, top, and bottom of the tank are covered with PMT's, the photomultiplier tubes which serve as detectors in the telescope. They are all pointed inward toward the mass of water, ready to detect the slightest Cherenkov light. (And slight it is—the Cherenkov light generated by the shockwave of a single muon is about as bright to the detector as a single candle seen from the Moon.)

Fortunately, each PMT is sensitive enough to detect a single photon of Cherenkov light. How does it do this? The same way you eat an elephant—one bite at a time. First, the photon hits a photo-cathode on the inner surface of the PMT's glass bulb, and the photo-cathode, in turn, releases an electron. The electron is attracted to a dynode, which carries a high-voltage positive charge, and accelerates toward it. When it hits, its great kinetic energy causes the dynode to emit several electrons, which are attracted to a second dynode with an even higher positive charge. The process repeats once for every dynode in the detector, until the final dynode is deluged with electrons, and sends a signal indicating that it has detected a photon. Neat, eh?

As you can imagine, PMT's are expensive ($3000 each, in this case), delicate, precision instruments, and you don't move them around like lightbulbs on a Christmas tree. Especially if you've recently gone from having 11,242 of them to having only 4,000 or so in one horrific oops.

Re:Why TUBES ??

[dragons_flight]

Fine. You show them a better way to be able to detect SINGLE photon events, roughly measure their wavelength and do it with sub-nanosecond resolution and I'm sure they will be very interested.

probably no single stupid mistake

[mj6798]

Given that a lot of smart people were working on this for a long time, I doubt it was "titanically stupid mistake".

Here is a random guess at a scenario: someone dropped something or the detector was filled to quickly. The implosion of the first detector caused a chain reaction and caused nearby detectors to implode as well. You have to expect that these kinds of accidents happen.

I suspect that the people who built the device simply didn't expect a chain reaction of implosions. Maybe one can argue in hindsight that they "should have" thought of it, but it's not like people regularly build things that have thousands of vacuum tubes deep under water.

What would be stupid is if they anticipated the possibility of a chain reaction of implosions and decided "oh, we just aren't going to drop anything accidentally". We'll have to see whether anything like that eventually comes out. Until then, I'd hold my judgement.

Re:probably no single stupid mistake

[EasyTarget]

Given that a lot of smart people were working on this for a long time, I doubt it was "titanically stupid mistake".

Unlike, say, sending a probe all the way to Mars then having it burn up because two teams used different measurement units and forgot to convert them?

History is full of examples of very gifted and smart people making very simple but catastrophic mistakes, or totally failing to anticipate the consequences of their actions, this looks like another of them. At least nobody died in this one!

No matter how hard we (humanity) tries, things will go wrong, given the complexity of todays world it is probably unavoidable. But it is important that we at least learn.. And that is the good thing about this article, they are going to find the 'what' and 'why', and (if I read it correctly) make sure it does not happen again.

Re:probably no single stupid mistake

[JabberWokky]

Unlike, say, sending a probe all the way to Mars then having it burn up because two teams used different measurement units and forgot to convert them?

When, pray tell, did *that* ever happen? I recall the rather recent example of the Mars Lander, which crashed due to cumulative drift of significant digita due to repeated conversion of units back and forth, but no Mars probe that burned up "because two teams used different measurement units and forgot to convert them?"

--
Evan

Re:probably no single stupid mistake

[JabberWokky]

CNN (and dozens of news agencies including Reuters and possibly the Washington Post) go on and on about this anthrax virus that is going around. Damn shame there's no such thing. But there *is* a bacteria named anthrax that's being mailed.

If you believe that news is accurate...

BUT - in this case it was... I just read the Official Mishap Investigation Board Phase I Report... turns out that the problem was a small program called SM_FORCES that was to read a table of pound-second figures, while the table provided for the flight was in newton-second figures. Read the whole thing here.

--
Evan "Not too proud to admit when he's wrong"

Re:probably no single stupid mistake

[EasyTarget]

It's cool, I was going from memory of the reports at the time, so I would not have been at all surprised if a different explanation had arrived later (and cumulative errors actually sounds more likely to me, I've seen similar effects in feedback modelling systems I once worked with).

It just seemed like a fairly relevent example of the great and good cocking up.. There are others, but of course it's the failures that we remember, and we forget the vast majority of times when everything goes perfectly (because the same people got it right). Perhaps the real lesson here is that success needs more recognition..

PS. 100% with you on the quality of news reporting (look here), Always amazes me that the the news media are so quick to critisize errors in others, when they are the least accurate of all..

Re:probably no single stupid mistake

[mj6798]

The problem there wasn't the fact that people used wrong units in that one case. The problem was that (1) the US hasn't standardized on the metric system along with the rest of the world, and (2) that the NASA engineers didn't have software engineering procedures in place that required the presence of units on all data. Those are on-going, deep-rooted problems, and they are going to cause crashes and other problems over and over again until they get addressed.

Re:probably no single stupid mistake

[mj6798]

Unlike, say, sending a probe all the way to Mars then having it burn up because two teams used different measurement units and forgot to convert them?

You can't rely on people remembering to convert units--they will make these mistakes. The problem is that the US hasn't converted to the metric system, and that NASA software apparently does not use type systems and data files containing units throughout. Those aren't stupid mistakes, they are on-going, deep-rooted problems, and they will cause more crashes, guaranteed.

Why use PMTs over solid-state light detectors?

[sigwinch]

When you want to sense the raw quantity of light arriving (i.e., you don't care about direction, image, and color), PMTs are ludicrously good. They are absurdly linear over the range of one photon/year to millions of photons/second. (Solid-state detectors are notoriously nonlinear.) PMTs have a tremendous dynamic range. PMTs can measure the time of arrival of individual photons to the nearest nanosecond. (Solid-state devices tend to be much slower.) I don't know for sure, but I strongly suspect that large PMTs are vastly more reliable than equivalent solid-state detectors.

The real kicker is cost. Solid-state devices cost on the order of $1,000,000 per square meter of active area! PMTs are on the order of $100,000 per square meter. If you want hundreds of square meters of active area -- like in a neutrino observatory -- PMTs are the only way to go.

Re:yeah but...

[Shimbo]

there is still no fucking formal explanation, Mr. Man.

Well no. It will be looked into by 'committees' - reading between the lines, some sort of board of inquiry will be set up. When you decide to do a formal investigation, you don't preannounce the results even if you have a damn good idea what happened. You make sure your investigation process is transparent and fair to anyone whose career might suffer. These things take time to do properly.

Re:So...

[SEWilco]

It is suspected that neutrinos can oscillate between several types, and the detectors are sensitive to certain types. The MINOS project will test this starting in 2003, by firing neutrinos from Fermi Lab near Chicago, under Wisconsin, to the detector in a mine near Soudan, Minnesota.

Re:The Replacements

[SEWilco]

They replaced about 100 which were not working.

Gee, they don't have a PayPal link so we can sponsor a PMT?

Re:Why TUBES ??

[markmoss]

If you want to count each individual photon, photomultiplier tubes are the only choice.

In a PMT, a photon hitting the first plate releases an electron. The first plate (cathode) is negatively charged, so the electron flies off towards the less-negative 2nd plate, picking up enough energy to knock several electrons loose. These hit the third plate, knocking out more electrons, and so on. After many plates, the pulse of electrons is large enough to be easily measured, so they are collected and output on a wire at the back of the tube (anode). You can either measure the average current to determine photons/seconds, or detect each pulse to determine when each photon arrived. The super-K uses the latter method, since it has to compare photon arrival times to find the position of the event which created a burst of photons.

The PMT has very high gain and a remarkably good signal to noise ratio. "Gain" is the number of electrons out for one freed electron in, and you just add plates (and increase the overall voltage) until you get what you need. "Noise" would be an electron spontaneously flying off from the cathode, and this is pretty rare.

Solid-state detectors also start with a photon energizing one electron to jump somewhere it wouldn't normally go. Then you need an amplifier. It's possible to build solid-state circuits that will amplify a single electron to a measurable pulse, but to make it that sensitive you must also make it possible for electrons to just tunnel through the first amplifier stage on their own, and this is indistinguishable from detected photons. So it's hard to sort out the signal from the noise.

Re:anyone know...

[Brett Viren]

The main page, www-sk.icrr.u-tokyo.ac.jp has some
"press" sized pics, last I checked. Yes, here:
http://www-sk.icrr.u-tokyo.ac.jp/doc/sk/photo/hi gh .html

K2K

[ErfC]

Incidentally, K2K is sort of the other half of Super-K's job. It's an experiment where the KEK accellerator creates a neutrino beam and fires it through Japan (through the ground, through towns, farmers' fields, through the Japanese people...) at Super-K. The nice thing about neutrino beams is that you know what you're starting with and you can control the rate.

(I imagine it's probably also kind of hard to aim, since neutrinos are so hard to see in the first place... They have a "front detector" at KEK which gives them an idea of how many neutrinos they're starting with, and I think where they're shooting them. KEK and Super-K are 250 km apart, so even a slight miss can have a big impact on whether they hit Super-K or not, I think.)

Re:K2K

[krlynch]

I imagine it's probably also kind of hard to aim, since neutrinos are so hard to see in the first place...

Actually, it is (relatively) easy to aim, as long as you are done aiming before the neutrinos are produced... neutrino beams are made by accelerating protons into targets, which produces beams of charged pions, which are collimated and sent down a beam pipe pointing directly at Super-K. Some fraction of the pions (checking... checking... 99.9877% ) of the pions decay to a muon and a muon neutrino, going in the same direction as the pion (i.e., pointed directly at Super-K). All the muons and undecayed pions are stopped at the end of the beam pipe, and don't contribute to the beam. The hardest part of aiming is digging the tunnel in the right direction, but GPS makes that relatively easy these days.

Re:K2K

[Eric Sharkey]

I imagine it's probably also kind of hard to aim, since neutrinos are so hard to see in the first place.

It's not that hard. Don't think of it as aiming a gun. Think of it as aiming a flashlight. As the neutrino beam travels, it spreads out into a cone. The further away you get the "dimmer" the beam gets because the neutrino density goes down, but as long as you can generally aim the beam with some coarse precision, it doesn't matter how far away your target is, you can hit it.

Thursday update

[Tsar]

PhysicsWeb has this story providing additional detail about the accident, and their recovery plans.

Re:No, it isn't possible

[angst_ridden_hipster]

Ah, bit the pessimists assert that it was half empty at the time of 'detonation'.

scoop on what happened -insiders info

[paraquat]

I have heard from people close the the community several things.
1. The detectors that imploded/ did not implode
where seperated by the water line. I.E. Those under water imploded and the above the water line did not implode.
2. There is about 3 meters of broken glass in the bottom of the tank.
3. Nobody is sure of what will happen to funding or the experiment. (I realize this contradicts the the main thread explanation)

4. there was about 20 million in damage just in destroyed tubes. This is not counting water which was very expensive, near the theroetical purity of water. Or the cleanup and redisign cost.

_____________
Now for some speculation/opinion
1. It is the opinion of some people in the field that this could have possibly been prevented, by baffles, and partitions.
They either did not fully consider the affects of what could happen or they dismissed it.

2. This was a pretty prestigious experiment.
Liken this to Fermilab exploding in the US or
CERN in europe. This was one of the biggest if not the biggest experiment of its kind in the world. Also the most sensitive. (vs say Homestake)

3. Because of the prestige for the Japanese scientific community there is a very good chance funding to bring it back will come through.

4. Unless the tank itself is leaking. These tanks were not designed to survive a catastrophic event like this. If it leaks it probably will not be repairable. and the experiment is over

5. The tank will need to be drained and the glass removed, About 3 meters deep worth, and they will need to design a baffle system to keep this from happening again before they star again.

6. this could have been prevented acording to people in the community and was a known danger.

9. This suddenly makes funding for other competing projects in other place more available. Which is good for the other places. They may be secretly glad.

10. This is great for Hamamatsu, because they make the tubes and may get and order to replece them.

Mystery solved

[leonbrooks]

http://www.electric-universe.de/ (warning: contains flash)

http://www.kronia.com/

...there are more, but those should be enough.

.

Re: You try doing some of this stuff

[Kris_J]

History is full of examples of very gifted and smart people making very simple but catastrophic mistakes, or totally failing to anticipate the consequences of their actions, this looks like another of them.

Just to clear things up, the reason why history isn't quite so full of ordinary people making stupid mistakes that that it's far more common and usually doesn't involve innovative projects -- thus rarely of historical note. Just thought I'd put that in to make the scientists feel a little less like everyone's favourite chew toy.

New Sign

[Wanker]

I bet when it's all fixed they have a new sign in 1000-point lettering which says:

NO FLASH PHOTOGRAPHY

Whoops. ;-)