Slashdot Mirror


What's Inside the Mars Rovers

Captain Zion writes "Space.com has a story about the hardware and software of Mars Rovers Spirit and Opportunity. Basically, they're radiation-shielded, 20MHz PowerPC machines wirh 128Mb RAM and 256Mb of flash memory, running VxWorks. I wonder if I could make a nice firewall with one of these for my home network..."

35 of 458 comments (clear)

  1. Radiation hardness by swordboy · · Score: 5, Interesting

    Does anyone know what the deal was with the flash memory that caused the outage? I heard something about a "solar event" that caused a problem with the flash memory that led to the outage. It was subsequently resolved by disabling the flash. If so, BAE Aerospace has a possible solution with their upcoming line of rad-hard memory.

    --

    Life is the leading cause of death in America.
    1. Re:Radiation hardness by TwistedGreen · · Score: 5, Interesting
      It appears to be a software error, and not hardware-related. It actually looks like it ran out of swap space and the OS crashed. This article explains what they think happened, and this article has more information on their recovery plans.

      A quote:
      It is now believed that the rover's flash memory had become so full of files that the craft couldn't manage all of the information stored aboard. Spirit bogged down because it didn't have enough random access memory, or RAM, to handle the current amount of files in the flash -- including data recorded during its cruise from Earth to Mars and the 18 days of operations on the red planet's surface.
      Raises some interesting questions about software reliability, I think. Did nobody think about running out of disk space?
    2. Re:Radiation hardness by AaronW · · Score: 4, Interesting

      If anyone saw my earlier posts on VxWorks they would see I am not at all surprised about the problems NASA is having.

      As someone with first-hand experience with VxWorks let me say that VxWorks' memory handling code sucks. Their malloc implementation has got to be the worst one ever designed. It fragments horribly and when fragmented has unusable performance. A malloc call can take many milliseconds when memory gets fragmented. Our box used to crash due to the fragmentation. I replaced Wind River's code with DLMalloc and that fixed the memory issue. We went from many tens of thousands of fragments to only a few dozen. Our reliability significantly increased as well after dumping Wind Rivers brain-dead malloc code. BTW, glibc uses a variant of Doug Lea's malloc code, so it's been widely tested.

      Furthermore, in VxWorks there is no way to identify what process malloced a block of memory. There is no memory protection either (think DOS). If a task has a memory leak, VxWorks does not provide any method of tracking down the culprit. I had to add that support to the DL Malloc code I ported so we could find memory leaks and general memory usage by task.

      Since malloc is critical to the working of VxWorks, if you run out of memory you are basically completely dead. Often the only way to recover is to wait for the hardware watchdog timer to kick in and reboot the system.

      If a task dies in VxWorks with a bad pointer there is no way to recover other than reboot. The OS will not clean up after a task (i.e. free memory, close files, release semaphores).

      As far as flash goes, VxWorks supports the FAT16 file system. As you know, FAT16 sucks. It only supports a limited number of files in the root directory. It's relatively easy to corrupt, and when corrupt it tends to corrupt itself even worse. There is no wear leveling support. If the FAT table or root directory gets corrupted, you're screwed. VxWorks is even worse since there arn't tools to fix a corrupt file system.

      VxWorks is not a scalable OS. The OS gets slower as the number of tasks increases. Realtime support sucks. Although it has support for things like priority inheritance to prevent priority inversion, the best guaranteed realtime latency is half the system tick rate (the tick rate is usually 10ms).

      Also remember that unlike open source operating systems, the source code to VxWorks is not available unless you pay some major $$$. Without the source you're basically working blind.

      VxWorks is an old RTOS, and its age is definitely showing. It is not a robust OS.

      As far as turning VxWorks into a firewall, you'll need to write all your own code. The VxWorks TCP/IP stack is an archaic vulnerable version of the BSD stack. TCP sequence number guessing is trivial. There is no built-in support for firewall support, NAT, or anything else. I have heard many many complaints about the VxWorks networking code. Although the box I'm working on is a router and broadband remote access server, we don't use the VxWorks TCP/Ip stack much. I am sure the VxWorks stack is vulnerable to many of the current DOS attacks as well.

      Some of you might say why not use VxWorks AE, which adds memory protection. Think hacked-in memory protection far worse than Windows 95. It's slow, very buggy, and poorly supported. We tried AE and after many months we dumped it and went back to the non-AE version. Very few companies actually went with AE.

      Today there are many other choices for an RTOS that are better than VxWorks. For our next major project we're looking at both QNX and TimeSys Linux. QNX is a true microkernel design with the core being around 70K. Every driver is protected and can be restarted if it crashes. I think you can even buy a medical grade version of QNX. TimeSys Linux is also pretty cool, with excellent real-time support and all the advantages of Linux. For something like the Mars rover, QNX would be better due to the limited amount of memory and greater robustness.

      Wi

      --
      This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
    3. Re:Radiation hardness by AaronW · · Score: 4, Informative
      While I would agree with you that avoiding malloc and preallocating memory is the way to go, but it is not always possible. In my case, we are using various 3rd party libraries, and changing them to use static memory allocation would be prohibitave. In at least one case, the third party library source code was not available. Also, in many cases the dynamic nature of some algorithms requires dynamic memory management. You cannot statically allocate everything, especially in a limited memory environment.

      I know we're not the only ones to have been burned by Wind River's malloc. I know several major companies that also had to replace Wind River's code.

      As far as being able to dynamically replace code, VxWorks isn't alone in that. Numerous other RTOSes out there can do the same thing, including QNX. QNX even supports the concept of a hot standby process to take over if the main process dies.

      To give you an idea about how Wind River's malloc works, they keep a sorted linked list of fragments from the smallest to the largest. When you try and allocate a block, it walks the linked list until it finds a block large enough. Likewise, when you free a block it checks if it can coalesc the block with a neighboring block. It then goes through the linked list looking for a slot to insert the free block.

      Yes, VxWorks may have been around since the 80's, but that's part of the problem too and it is showing its age. In the 80s embedded processors typically did not have MMUs. Now MMUs are quite common in the more powerful embedded processors.

      You say you can't have low latency and memory protection? QNX proves that you can. It is low latency and *very* robust. If your driver dies, no problem, restart it. Timesys Linux also has a very low latency, although not as low as QNX. Timesys also has an interesting feature where you can guarantee CPU and networking resources. I can schedule a task to be guaranteed 5.8ms of execution every 8.3ms and it will guarantee that that task will get the CPU time allotted to it with the desired resolution. This is without increasing the system tick rate (usually 10ms). Timesys can also schedule a task to be higher priority than an interrupt. I'm not as familiar with QNXs scheduler, but it's also quite flexible from what I've heard.

      As far as FAT, it is not a robust filesystem. It never has been. If the FAT gets corrupted or a directory entry gets corrupted it's difficult to recover. Other than possibly having 2 copies of the FAT cluster table, any corruption can be difficult to repair. If the FAT table gets corrupted, which table is corrupt and which is not? If a directory entry gets corrupted, it can be impossible to fix. For flash memory, unless you are using a device with special wear-leveling, FAT is about the worst choice since any file write that changes the size of a file requires a write to the directory entry and possibly the FAT table. If the table gets corrupted and you don't run a repair operation (which often ends up leaving orphaned files as lost clusters), the file system can happily corrupt itself to death. Why do you think every time DOS/Windows9x/ME crashed it had to repair the disk with scandisk? FAT is a poorly designed file system that was originally designed for 160K floppies and scales poorly. FAT32 is an improvement, but it's still not very robust. For flash, something like Linux's journalling flash file system 2 (JFFS2). More information on VxWorks file system support can be found here.

      Basic VxWorks information can be found http://www.slac.stanford.edu/exp/glast/flight/docs /VxWorks_2.2/vxworks/guide/.

      --
      This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
  2. What's inside the rover? by Chilltowner · · Score: 5, Funny

    Darn. Interesting articles, but I was hoping that inside it was filled with creamy nougat center. Oh, wait. I'm thinking Mars bar. Nevermind.

  3. Wait a second... by deitel99 · · Score: 5, Insightful

    The machines aren't as slow as the top post says... they don't run at 20MHz, they are "capable of carrying out about 20 million instructions per second". Depending on the complexity of the instructions, the processor actually runs several times faster than 20MHz.

    1. Re:Wait a second... by Zog+The+Undeniable · · Score: 4, Insightful
      Actually, they're probably slower :-P

      Modern superscalar (pipelined) processors have a lot more MIPS than megahertz.

      --
      When I am king, you will be first against the wall.
    2. Re:Wait a second... by Rootbear · · Score: 5, Informative

      Actually they are running at 20MHz. I've seen several write ups which clearly state that. The RAD6000 can apparently run at up to 33MHz, with a claimed 35MIPS. The rovers are "underclocked", probably due to power budget concerns.

      Go to
      http://www.iews.na.baesystems.com/space/rad600 0/ra d6000.html
      and click on the rover picture to get a PDF brochure, which gives the 33MHz/35MIPS figure.

      Rootbear

  4. Self-warming by Faust7 · · Score: 5, Funny

    To survive the frigid Martian night, MER computers are housed in warm electronics boxed heated by a combination of electric heaters, eight radioisotope heater units as well as the natural warmth from the electronics themselves.

    Just leave off the heatsinks and fans, and everything should be fine.

    1. Re:Self-warming by Cyclopedian · · Score: 5, Interesting
      To survive the frigid Martian night, MER computers are housed in warm electronics boxed heated by a combination of electric heaters, eight radioisotope heater units as well as the natural warmth from the electronics themselves.[Emphasis added by me]

      If obsessed environmentalists don't like NASA sending up probes with any radioactive material ('it might blow up, ohh..'), then how did this little tidbit get by them? Do they consider it non-radioactive? If they're only concerned by radioactive propulsion systems, then I think they're a bunch of hypocrites. Radioactivitiy is radioactivity whether it's propulsion or heating.

      If they don't mind it, then let's send up those dune buggies with RTG and 18-inch wheels and cover a lot more of Mars.

      -Cyc

    2. Re:Self-warming by Zog+The+Undeniable · · Score: 5, Funny

      If only they'd used an Athlon, the planet would have been habitable in Bermuda shorts by the end of the week.

      --
      When I am king, you will be first against the wall.
    3. Re:Self-warming by JDevers · · Score: 4, Informative

      If I'm not mistaken, virtually all probes have some sort of radioisotope heater...

      Radioactivity is NOT radioactivity when you are considering things like this. Saying the people who don't want nuclear powered rockets should hate this as well or else they are hypocrites is tantamount to saying that the people who don't like oil spills should bitch about how some motor oil ALWAYS stays in the plastic container it is shipped in. Not quite the same problem. Afterall, these things aren't much more radioactive than a Coleman lantern wick or a smoke detector element...

    4. Re:Self-warming by The+Fun+Guy · · Score: 4, Informative

      Radioisotope thermoelectric power units need to be hot enough to allow for electricity to be generated by thermocouples placed between the unit and the heat sink (space). A quick Google search gives 200-500 watts of power generated from multiple interleaved stacks of plutonium-238 or strontium-90, average radioactive source strength of around 50,000 curies, depending on design.

      Radioisotope heaters use much less material, as they only need enough heat to keep the warm electronics box above -40F or so. From the Environmental Impact Statement in the Federal Register ([wais.access.gpo.gov][DOCID:fr10de02-54]):

      "Each rover would employ two [calibration] instruments that use small quantities of cobalt-57 (not exceeding 350 millicuries) and curium-244 (not exceeding 50 millicuries) as instrument sources. Each rover would have up to 11 RHUs that use plutonium dioxide to provide heat to the electronics and batteries on board the rover. The radioisotope inventory of 11 RHUs would total approximately 365 curies of plutonium."

      Nothing you'd like to swallow, but still, much smaller than a radioisotope power unit.

      --
      The man who does not read good books has no advantage over the man who cannot read them. - Mark Twain
  5. If you've got money to burn, sure by Powercntrl · · Score: 4, Funny

    But I'd take a Linksys over a hacked Mars Rover anyday... Billions cheaper, ya know.

    --

    ---
    DRM is like antifreeze, to the MPAA/RIAA it's sweet, to the consumers it's poison.
  6. Radiation Shielding by kyknos.org · · Score: 4, Interesting

    How is it done? Some external armor, or even insides of the chip are different?

    ---

    --

    SHE does throw dice.
    1. Re:Radiation Shielding by pi+eater · · Score: 5, Funny

      A modded alienware case perhaps?

      geeky stuff.. offensive stuff!

    2. Re:Radiation Shielding by the+real+darkskye · · Score: 5, Informative

      The CPU is fabricated to withstand the radiation, a brief summary can be found here or by googling

      --
      Music is everybody's possession.
      It's only publishers who think that people own it.
      Fuck Beta
      ~John Lenno
  7. Re:What's the bus speed on that thing? by gerf · · Score: 5, Insightful

    Does a 20mhz processor really need 128mb of ram? I mean, with a bus speed that low, you can probably put the data to flash ROM just as fast. What are the chances of you using all 128mb of ram?

    I imagine they can use all the storage they can, since there's no hard drive. So, the RAM acts as a cache for everything that is transmitted (which is a lot, actually). The Flash is used for more permanant software, like OS, commands, other files, ect. I'm amazed they can do it all with as little as they have.

  8. Re:What's the bus speed on that thing? by CaptainAlbert · · Score: 4, Informative

    > Does a 20mhz processor really need 128mb of ram?

    A processor of any speed doesn't need RAM of any size.

    The application you want to run needs both processing power and memory. How much of each? Depends on the application.

    --
    These sigs are more interesting tha
  9. Re:What's the bus speed on that thing? by GoofyBoy · · Score: 5, Funny

    >What are the chances of you using all 128mb of ram?

    Ask Bill Gates.

    --
    The surprise isn't how often we make bad choices; the surprise is how seldom they defeat us.
  10. A nice Firewall? by blorg · · Score: 5, Funny

    "I wonder if I could make a nice firewall with one of these for my home network..."

    You could, but the latency would be a bitch.

  11. Flying VxWorks to Mars by Hiroto.+S · · Score: 5, Informative
    I googled across following presentation with a little more details.

    Flying VxWorks to Mars

  12. Re:I hope the flash memory was not commodity hardw by Rootbear · · Score: 5, Interesting

    There is very little on the Rovers that is "commodity" in any sense. The CCD image sensors, the computers, everything, is all custom made. Everything has to be made to withstand the rigors of flight and the harsh environments of space and Mars. The CPU does not have a backup, which is a bit unusual for NASA (I'm a contractor at NASA/Goddard, but not involved in any flight missions). However, the particular computer used on the rovers (the RAD6000) has a very good record. There are something like 150 in use on various spacecraft and they've all worked very well.

    And the flash memory has probably not failed. It seems to have been a software problem, not hardware.

    Rootbear

  13. Internet conneciton by vpscolo · · Score: 4, Funny

    And it can still send back at 128 kbits/sec which is faster than my connection can managed. Just waiting for it to start getting spam advertising pr0n and viagra.

    Spirit Rover: Staying up longer and harder

    Rus

  14. Save HST! Sell Sojourner knock-offs by bhima · · Score: 4, Interesting
    What NASA should do is to hire a Taiwanese firm to build inexpensive knock-offs of Sojourner. They already have the design, I'm sure a few bright minds could cut the chassis price down significantly; after all we don't need all the exotic materials. I'm sure IBM still makes a PPC variant that would make a new cheap board layout easy. As far as the OS: of course we don't need VxWorks (Nor could the project afford it) we have NetBSD!

    The profits from Slashdot alone could extend the life of HST or launch the James Web Space Telescope early.

    I thought about the current rovers, but I think they are a bit large to be successful!

    --
    Nothing in the world is more dangerous than sincere ignorance and conscientious stupidity.
  15. Re:Redundency Check? by vofka · · Score: 5, Informative

    If I recall correctly, the Shuttle has 5 GPC's (General Purpose Computers), three of which are "online" at any one time.

    The online GPC's each carry out the same set of calculations (potentially each uses code designed to do the same thing, but written by different programmers), and they compare each others results. If any single GPC is considered to be too far wrong, the offline GPC's submit their answers. The three GPC's that are in closest agreement then become the new online GPC's, and the remaining two go offline. The GPC's can reboot themselves if they are too far out of whack, if they fail in one of the "results elections", and of course when they are told to do so by the crew.

    Also, whenever a GPC is sent offline by one of the others, a specific caution indicator (and potentially the master caution indicator and klaxon) is activated, and the relevant error codes are shown on one of the forward CRT's. The error codes, along with other information such as the currently running program and the current mission phase, determine the crew's actions. Actions can be as simple as disabling the master caution klaxon for the current alert, all the way to hand-checking certain results and manual GPC restarts.

    This is all from memory (from about 5 years back), so some of this may have changed recently, particularly on Atlantis with the "glass cockpit" upgrade that happened 18 months or so ago, but the general gist should be about right (and I'm sure I'll soon know if it isn't!!)

    --
    Disclaimer: I meant what I thought, not what I wrote! What? You can't read my Mind? Oh dear!
  16. space shuttle uses 1969-vintage ibm 360 computers by peter303 · · Score: 5, Interesting

    The space shuttles run on five AP-101 computers, originally designed in 1969. The started with 32 kilowords of magnetic core memory for radiation protection, since upgraded to semiconductor memory. These computers were chosen due to their success in the Apollo, Skylab, and B52. For science and personal work the astronaut specialists usually bring personal laptops which are thousnds of times more performant.

  17. Processor is *not* a PowerPC by GileadGreene · · Score: 5, Informative
    radiation-shielded, 20MHz PowerPC machines

    No, they're not.

    The processors in MER are RAD6000's, which are radiation-hardened versions of the RS/6000, the predecessor to the PowerPC (see this for details). The RAD6000's younger brother, the RAD750, is indeed a rad-hardened PowerPC.

    As an aside, there is a big difference between a radiation-shielded processor and a radiation-hardened processor. Shielding implies just sticking some kind of rad-absorbent material between the processor and the environment. A rad-hardened processor is actually manufactured in a different way - different gate layout, different design rules, often different materials (Silicon-on-Insulator is popular). These things are done to minimize or prevent the effects of single-event upsets (when a bit is flipped by high-energy particles) and single-event latchups (which basically turn a couple of gates into a glorified short-to-ground). The materials changes may also improve the overall total dose tolerance of the processor. The work required for redesign is one of the reasons that space-qualified rad-hard processors lag the commercial market. The NASA Office of Logic Design has some good papers on space processors available online if you're interested in learning more.

    1. Re:Processor is *not* a PowerPC by addaon · · Score: 4, Interesting

      The Rad750, btw, is a deeply cool chip. Once it's mature enough to start using for scientific-level stuff, it will be a real revolution in what we can do. One of the limitations with Hubble was that it had so little processing, a full data dump needed to be done for even checking orientation; there was no ability to offload processing to the sat. If the 750, or something similar (not that I know of anything too similar) is up there for our next big telescope, it will make a real difference in the efficiency of how it is used.

      --

      I've had this sig for three days.
  18. They used to use sapphire... by emil · · Score: 4, Informative

    ...as the substrate of the chip, rather than a silicon wafer, so the chip was a "sapphire" chip rather than a silicon chip (although doped silicon could then be used to form transistors, as could Gallium Arsenide or Germanium, through the regular lithographic process).

    This is the classic "Silicon On Insulator." IBM has a process of embedding a layer of glass beneath the surface of a standard silicon wafer, allowing SOI using silicon substrates. This and their work with copper set them apart from the other large silicon transisitor foundries (TSMC, Intel, etc.).

    The processors on the rovers are probably SOI, but I don't know which process is used.

  19. Re:What's the bus speed on that thing? by 3waygeek · · Score: 5, Funny

    In other words, the Spirit is willing, but the flash is weak.

  20. FPGA's by retostamm · · Score: 4, Interesting
    There are also Xilinx FPGA's in the Rover. Cool thing because they can be reconfigured if you find a bug while the thing is in transit.

    Xilinx radiation-tolerant Virtex(TM) FPGAs are being used in the "main brain" of the rover vehicle, controlling the motors for the wheels, steering, arms, cameras and various instrumentation, enabling the vehicle to travel about the planet.

    They also controlled the Pyrotechnical stuff during landing.

    [Disclaimer: I work for this great company.]

  21. RAD6K by Anonymous Coward · · Score: 5, Informative
    I am an engineer that works with the RAD6K processor boards. A couple of observations here.

    1. The RAD6K really does run at 20 Mhz. They're creakingly slow. They're spec'd to run up to 33 Mhz, but the customer can get them to clock at lower speeds (I've seen them run at 12.5 Mhz). The only drawback is the PCI bus is also clocked as the same speed as the CPU. This is a mixed bag - but a slower PCI bus helps improves signal integrity and decreases power consumption.
    2. The board is PCI, but NOT compact PCI. There is a proprietary PCI connector and a proprietary PCI backplane. You cannot plug commercial PCI products unless you have an adapter to interface to the proprietary PCI connectors.
    3. For those who are not aware, there are three types of memory being used on the rovers. There is the SRAM (the RAD6K boards use SRAMs, not DRAMs), the EEPROM, and apparently, FLASH RAM. The EEPROM and the SRAM are on the processor board itself - there is probably more EEPROM memory in the system on another board. The EEPROM usually holds the flight code, and there are usually two copies. An original version that was launched with the spacecraft, and one patched version made possible via uplinks.
    4. I am amazed at the presence of FLASH RAM's. I am not aware of any rad-hardened FLASH RAM devices for spaceflight use. In addition to radiation hardness, the device must be made reliable with an approved QML-Q or V manufacturing flow. Radiation hardness is just icing on the cake, but the key is that the device must be reliable to withstand temperature extremes, shock and vibration. So, I have yet to see a FLASH RAM device that can be used. I am aware of the Chalcogenide based RAM's which are essientially uses the same substrates on CD-ROMs as memory cells. These products are hard to come by right now and are a high risk because we don't have sufficient data and flight heritage. A catch-22 in flight design is if it hasn't flown before, we don't want to fly it. But at some point, someone has to fly the first generation (someone who is willing to take a huge risk). Anyway, the FLASH RAM's on the rovers are in all likelihood upscreened commercial products. In other words, a mass buy of an entire lot of dies of commercial FLASH RAM's may have been bought, packaged in-house or through a vendor, and then screened for reliablity at extended specifications. This is not the same as the manufacturer who can guarantee the specs by design by designing it from the outset with increased reliability in mind.
    5. Radiation shielding? Minimal at best! The RAD6K shields its SRAMs by placing it on the underside of the processor board and orienting it such that the particles hit the processor side of the board instead of the RAM side of the board. There is some degree of radiation shielding for particles of sufficiently low energy. The truly high speed particles are going to get through and the only thing that will truly stop them is shielding whose thickness is measured in feet. That amount of shielding is too heavy for launch. The best we can do is mitigate the effects of radiation by guaranteeing devices can withstand a certain amount of radation dosage (measured in kRads) and design for latchup protection (latchup is a parasitic condition in which an ionizing particle impacts a transistor structure in a way that causes a SCR to be formed and a runaway current condition is initated leading to the device being burned out by high currents). Radiation effects in the form of SEE's (single event effects) such as bit flips can be mitigated by redundancy and voting circuits, memory scrubbing, and error checking using checksums/CRC's.
    1. Re:RAD6K by demachina · · Score: 4, Informative

      "A catch-22 in flight design is if it hasn't flown before, we don't want to fly it. But at some point, someone has to fly the first generation (someone who is willing to take a huge risk)."

      Or you fly it as a non mission critical experimental payload which is what we did back in the day I worked on avionics. You fly it as an experimental package so it gets the stress but if it breaks either its not in the mission critical loop, or if it is in the loop you can switch back to proven hardware. I kind of assumed this would be a standard part of qualifying electronics for space flight as well though its obviously a lot more expensive. Its not feasible to test it on Mars due the expense but you could test it in geostationary orbit where it will get lots of radiation and temperature extremes, as well as launch vibration and G's.

      --
      @de_machina
  22. The CPU is POWER, not PPC by rshadowen · · Score: 4, Informative
    The RAD6000 is based on a POWER CPU called RSC (RIOS single chip) - it's not a PPC chip. This was a design that consolidated the 5 - 6 chip RIOS processor complex onto a single lower performance die for low end workstations. I worked on the development team at IBM.

    The RSC design played a key role in bringing Apple and Motorola together with IBM to create the PowerPC line of CPUs. The 601 was the first PPC and was basically a redesign of RSC. It supported both POWER and PPC architectures, although there were deviances from PPC since the architecture was actually being defined at the time we were working on the chip.

    The RAD6000 version of the design happened because IBM wanted to pursue some government contracts, so had the RSC specially qualified. Another group then took the design and performed the radiation hardening.

    After Pathfinder we had some cool IBM/Mars posters hanging around the building, but oddly enough they vanished very quickly...

    --
    "I want my job to be the guy who kicks George Bush in the face all day, only stopping to make out with him."