Slashdot Mirror

← Back to Stories (view on slashdot.org)

Google Glitch Took Thousands of Chromebooks Offline (geekwire.com)

Posted by EditorDavid on from the no-internet-for-you dept.

Slashdot reader Bismillah was the first to notice stories about Chromebooks going offline. GeekWire reports: Tens of thousands, perhaps millions, of Google Chromebooks, widely prized by schools due to their low cost and ease of configuration, were reported to be offline for several hours on Tuesday. The apparent cause? A seemingly botched WiFi policy update pushed out by Google that caused many Chromebooks to forget their approved network connection, leaving students disconnected.
Google eventually issued a new network policy without the glitch -- but not everyone was satisfied. The Director of Technology at one school district complains Google waited three and a half hours before publicly acknowledging the problem -- adding that "manually joining a WiFi network on 10,000+ Chromebooks is a nightmare."

48 of 77 comments (clear)

  1. manually joining a WiFi network on 10,000+ by pecosdave · · Score: 2

    manually joining a WiFi network on 10,000+ Chromebooks is a nightmare

    Yep.

    Bet you're happy they've quit putting Ethernet ports on everything huh? You know, with an Ethernet port, even if they almost never used them, you could dust it off, plug in a cable later and get the update that would fix your WiFi issue. If only you had the port.

    I stay mad at Macbooks over the lack of Ethernet jacks these days, Chromebooks are a mixed lot, but one thing I do know, Samsung put out some really thin systems with a fold-down Ethernet port so there's no excuse not to have and and having one is much more reasonable than expecting everyone to keep up with a dongle.

    --
    The preceding post was not a Slashvertisement.
    1. Re:manually joining a WiFi network on 10,000+ by Anonymous Coward · · Score: 1

      No you don't get it, the dongle costs like $40. They can put out a fancy "ultradongle" later for $150 or something insane, for "power users" who "want ethernet" and such.

      This has got to be a deliberate strategy, like Ajit Pai allowing Comcast to put up toll roads while crowing about fantasy bullshit free-market rhetoric. It's about $, duh.

    2. Re:manually joining a WiFi network on 10,000+ by pecosdave · · Score: 1

      I've paid around $7 for a 10/100/1000 USB 3 dongle before. You just have to be willing to put your trust into cheap stuff from China if you do that. I have to say, it works rather well. For around $20 I bought a 10/100/1000 dongle with a three port USB 3 hub in it. That one is super handy.

      --
      The preceding post was not a Slashvertisement.
    3. Re:manually joining a WiFi network on 10,000+ by ShanghaiBill · · Score: 1

      Bet you're happy they've quit putting Ethernet ports on everything huh?

      I bet they are. At $5 a pop, that would have added $50k to the cost of a 10k unit order. For what? An external USB-to-Ethernet dongle ($10 on Amazon) is far more economical, since you don't need one for every Chromebook (unless your router/server has 10,000 ethernet ports).

      Adding hardwired ethernet to every computer "just in case" makes about as much sense as including an RS-232 and a parallel printer port. Those can also be done with a dongle.

    4. Re:manually joining a WiFi network on 10,000+ by b0s0z0ku · · Score: 1

      Does a Chromebook have a driver/support for such a dongle?

    5. Re:manually joining a WiFi network on 10,000+ by pecosdave · · Score: 1

      Depends.

      Just like nearly everything it depends on which chips are on-board. I know in many cases even stuff that's well supported in Linux is sometimes stripped out of a Chromebook kernel for performance and lock-in reasons. My guess - maybe. I've had good luck mostly with modern USB devices, but I did run into one USB C item that required a driver install when I put it on a Mac.

      --
      The preceding post was not a Slashvertisement.
    6. Re:manually joining a WiFi network on 10,000+ by pecosdave · · Score: 2

      An Ethernet port is nowhere NEAR RS-232, PS/2, Parallel port territory. I'm personally considering turning off the WiFi in my own home and just going straight wire on everything, the phones are the only reason I don't. Ethernet is still in heavy use throughout the world and I still encourage my users to plug their damned cable in when they're at their desk, especially if they want me to remote in and fix something.

      --
      The preceding post was not a Slashvertisement.
    7. Re:manually joining a WiFi network on 10,000+ by Solandri · · Score: 1

      It's actually closer to 10 cents a pop. So it adds $1000 to a 10k unit order.

      If the poor IT guy has to wait 10 seconds for the USB-to-ethernet dongle to be recognized (we'll assume "it just works" and no time is needed for driver installation), for 10k units that's 27.8 hours of extra billable time. If he makes $50k/yr, that's $25/hour. So the extra cost to use a USB-to-ethernet dongle per 10k units is $695 per incident. Just two incidents and the built-in ethernet port is cheaper.

      RS-232 and parallel ports aren't added because they're rarely used. Ethernet is. Every year in my computer consulting business, I probably spend a cumulative 3-5 hours at client's offices (billed at $100/hr) twiddling my thumbs because they asked me to set up a device without an ethernet port, and to do that I need the WiFi password. They don't know the password, and have to run around digging through papers or making phone calls to figure out the password. That right there is several hundred dollars my clients could save if they'd bought devices with built-in 10 cent ethernet ports.

    8. Re:manually joining a WiFi network on 10,000+ by apoc.famine · · Score: 1

      Lol, really?

      You think that a school district somehow has the switches and cables to manually plug in 10,000 normally wifi enabled devices?

      You think they're going to put network devices where students can access them to even be able to plug them in?

      You think it's reasonable to take a class of 20-30 kids and string cables across the room?

      I get the hate for Macs losing their ports, and that's one reason I moved away from them personally. But chromebooks are very strongly marketed towards education, and specifically, non-computer-lab education. They are general purpose devices used in classrooms, which don't have an established power or network infrastructure. WAPs are put in the ceilings, and the chromebooks are charged on the storage carts they wheel them into each classroom on.

      This is one use-case where a wired infrastructure wouldn't work, and would be far worse as a backup than not having it at all.

      --
      Velociraptor = Distiraptor / Timeraptor
    9. Re:manually joining a WiFi network on 10,000+ by pecosdave · · Score: 1

      You think that a school district somehow has the switches and cables to manually plug in 10,000 normally wifi enabled devices?

      You say that like every one of them has to be plugged in at the same time so patches will install.

      You think they're going to put network devices where students can access them to even be able to plug them in?
      I would argue that WiFi access points are network devices where students can access

      You think it's reasonable to take a class of 20-30 kids and string cables across the room?
      Do you even I.T.?

      I'm going to argue that most schools are old enough there's probably at least one network jack in each classroom for the the teachers computer from 1990's early 2000's.

      I'm also going to argue that most schools probably have the infrastructure to have at least one computer lab from the same era that is similarly wired.

      I'm going to then argue that selecting one particular event throughout the day - say homeroom or study period, or whatever kind of staggered less responsible time they have scheduled for students throughout the day could be utilized to have groups of kids go report to the old computer lab, set their computer on the table, plug in the cable then hit "run updates" or whatever the computer guy tells them to do would be enough to knock out the entire school in a few days.

      --
      The preceding post was not a Slashvertisement.
    10. Re:manually joining a WiFi network on 10,000+ by apoc.famine · · Score: 1

      Spoken like someone who hasn't seen the inside of a school since the chromebook revolution.

      Chromebooks live on carts, usually in the library. Those carts are charging stations. Teachers reserve the carts, then someone hauls them down to their classroom, the kids pull the chromebooks off the carts, pop them open, use them, throw them back into the cart, and it goes back to the library. In general, there are enough for a sizable percent of the kids in a school to use at one time. Many schools are one-to-one, so every single student has one.

      WAPs are in the drop ceiling in the classrooms and halls. While there are some computer labs, they don't really get used like they used to even 5 years ago. Computing is now a mainstream classroom thing, and it happens in most of the classrooms if not all of them at the same time.

      All of the policies are centrally managed, and IT doesn't generally touch any of the carts filled with chromebooks. Broken ones might get looked at, but more than likely get shipped back to the vendor for a replacement, because they're under warranty. Or just tossed because the staff time in trying to fix a multi-year-old $200 device just isn't worth it.

      When something like this breaks, the one guy on the IT payroll for a school suddenly has hundreds of non-functional appliances on his hands. There isn't an established network in each classroom that's accessible, and now that computing is a daily classroom activity, there just isn't lab capacity. Curriculum is designed around using chromebooks in about every classroom, meaning you can't just wait until homeroom to fix this shit.

      Is this a pretty poor situation that schools have put themselves into? Yes. But that's reality.

      --
      Velociraptor = Distiraptor / Timeraptor
    11. Re:manually joining a WiFi network on 10,000+ by pecosdave · · Score: 1

      Some districts. My buddy who lives in the same town as me, but on the other side of the county line so is in a different district than my kids has a different setup. His district wants every kid to have a Chromebook owned by the school (they wanted a $200 deposit - yes I see the same thing you do with that) that the parents are fully responsible for financially, but is under the schools control. Every kid is expected to be walking around the school with a Chromebook at all times.

      There are ways around this - you can provide whatever you want to your kid as long as it can run Chrome and actually run what the school wants it to run.

      During this year so far his daughter has broken an older Mac Book Pro, a Surface Pro I think Gen 2, and a slim Dell.

      I think it would cheaper for him to just buy his own Chromebooks or HP Streams. My kid would be working off the costs of each of them if she kept breaking them.

      --
      The preceding post was not a Slashvertisement.
  2. Re:There's no cloud... by r1348 · · Score: 2

    Quite true, considering that Chromebooks are aimed at users using other people's computers (students using school's computers, employees using company's computers, etc.), I'd say they pretty much cover the role.

  3. You get what you paid for by DogDude · · Score: 3, Insightful

    You get a cheap gadget created a a giant corporation used for collecting and re-selling information about the "users" and expect to use that as a PC. What'd you expect?

    --
    I don't respond to AC's.
    1. Re:You get what you paid for by Bing+Tsher+E · · Score: 2

      In Education, they got a cheap gadget that pupils can use for their schoolwork. It should be possible for School Districts to host all the 'stuff' that a Chromebook needs on a local wifi based network. The Chromebook apps should all be pushed from the local network and it should be possible for the school district to unplug from the Internet as a whole, or strictly firewall it, to maintain an enclosed learning environment.

    2. Re:You get what you paid for by ShanghaiBill · · Score: 4, Insightful

      ... and expect to use that as a PC.

      Schools do NOT expect to use Chromebooks as PCs. They use them as thin clients, running everything in a browser. The kids can't mess them up, because, well, there is nothing on them.

      The schools know what they want, and Chromebooks deliver it. Nobody is being deceived.

    3. Re:You get what you paid for by 93+Escort+Wagon · · Score: 1

      The schools know what they want, and Chromebooks deliver it.

      Without a network connection, these Chromebooks weren’t delivering anything.

      --
      #DeleteChrome
    4. Re:You get what you paid for by Osgeld · · Score: 1

      there's plenty of other cheap gadgets that could fit that bill

    5. Re:You get what you paid for by Osgeld · · Score: 1

      The kids can't mess them up

      only google, and apparently on a massive scale

    6. Re:You get what you paid for by thegarbz · · Score: 2

      Interesting comment.

      a) They aren't cheap.
      b) Google doesn't resell user data from people who's accounts are tied to Google Edu for legal reasons
      c) "users" are actually users, there's no need for quotes.
      d) These aren't PCs, they are specific devices designed to run specific software for specific purposes replacing locked down iPads
      e) You don't expect your cloud vendor to knock you offline, especially someone with the size, and presumed professionalism of Google. What you normally expect is better, more stable and more reliable service than what can be provided by a lowest bidder IT service.

    7. Re: You get what you paid for by Anonymous Coward · · Score: 1

      Found the Google employee.

    8. Re:You get what you paid for by spire3661 · · Score: 1

      And thus we are back where we started before the PC. Mainframes that control everything and the user is dictated to. Good job google, you set us back to the mainframe era..

      --
      Good-bye
    9. Re:You get what you paid for by ShanghaiBill · · Score: 1

      For a school with thousands of students, what do you suggest as an alternative?

      If you give them "real" PCs, they will screw them up, and you will need dozens of IT staff to delete viruses and reformat drives. At any given time, you will have hundreds of students without working computers.

      The old solution was to reformat the drive everyday to a standard image, which means that the PC is just an expensive thin client. With a Chromebook, you get the thin client for less money, without the hassle of reformatting, and without a fragile magnetic drive.

    10. Re:You get what you paid for by spire3661 · · Score: 1

      You get a thin client that you control...that is the difference. Updates should go through a LOCAL QA process before being deployed on your network. That is a professional job that you cant just hand-wave away to pretend to save money. You arent supposed to outsource everything. You dont outsource your local network, that is just insanely stupid. If you are deploying for 10,000 kids, you NEED an in-house IT staff that controls the machines end-to-end.

      --
      Good-bye
    11. Re:You get what you paid for by Bing+Tsher+E · · Score: 1

      Why google should be allowed to not allow that is a better question.

      The idea that school districts should be tossed to the mercy of Microsoft 'or else' is rather obscene.

      An enterprising group of hackers just needs to crack the google chromebook app server connection and figure out how to mirror a server on a local network.

  4. The dark side of the cloud you can't control by zerofoo · · Score: 3, Insightful

    We were impacted by this glitch. We are a small school, so manually connecting 40 or so chromebooks was not a huge deal.

    However, this is not the first time we have been impacted by a Google screwup. We've had outages where Google's authentication service failed and no one could login to their chromebooks.

    We've since decided that we are walking back Chromebooks for staff members and putting them back on Windows laptops. Between the functional limitations of a chromebook, and the centralized parts we can't control, we've decided that an entirely cloud strategy for students and teachers is too risky.

    Admin and teachers will be provisioned on on-prem systems. Students will be cloud provisioned. At least this way when Google's infrastructure shits the bed, the business side of the school can keep going.

    1. Re:The dark side of the cloud you can't control by Gravis+Zero · · Score: 2

      We've since decided that we are walking back Chromebooks for staff members and putting them back on Windows laptops. Between the functional limitations of a chromebook, and the centralized parts we can't control, we've decided that an entirely cloud strategy for students and teachers is too risky.

      So, out of the frying pan and into the fire?

      --
      Anons need not reply. Questions end with a question mark.
    2. Re:The dark side of the cloud you can't control by Fly+Swatter · · Score: 1

      Students will be cloud provisioned. At least this way when Google's infrastructure shits the bed, the business side of the school can keep going.

      At least the business side can keep spending tax payer money eh?

    3. Re:The dark side of the cloud you can't control by thegarbz · · Score: 1

      With 40 units it doesn't sound like an unmanageable situation that would benefit from outsourcing in the first place.

    4. Re:The dark side of the cloud you can't control by zerofoo · · Score: 1

      Ha! - yeah a bit.

      Think of it this way - out of a frying pan over which we have no control into a fire that we can at least somewhat control.

  5. Re:God forbid they learn how to use a damn compute by Calydor · · Score: 1

    Maybe, just maybe, they don't want the students to actually KNOW the WiFi password, with which they can connect unauthorized devices to the network. Thus, someone with the password has to go around to type it in on all the devices around the school when the network policy shits the bed.

    --
    -=This sig has nothing to do with my comment. Move along now=-
  6. Re:God forbid they learn how to use a damn compute by b0s0z0ku · · Score: 1

    So implement MAC control or something similar -- only devices with approved MACs get on. All other devices are either banned or throttled. Yeah, yeah, students can still spoof a MAC, but they can steal a WiFi password too.

  7. Re:God forbid they learn how to use a damn compute by Jetstream · · Score: 1

    I can't recall ever having had to read one bit of a man page to set up wi-fi. In both Linux Mint & windows, all I've had to do was click the icon in the tray, select the network, and enter the password. Surely, Chromebooks would be just as simple, since they're designed to be used on wi-fi?

  8. Lie with dogs, get fleas... by b0s0z0ku · · Score: 1

    Chromebook -- a cloud-centric device that even the owner doesn't control. Basically, time-share computing updated for the 2000s -- should anyone be surprised that the people (Google) calling the shots occasionally screw up and screw their users? (I won't say "customers", since Google's real "customers" are advertisers and firms that buy customer data.)

  9. No School Today! by Fly+Swatter · · Score: 4, Insightful

    ..because the vendor pushed out an untested update again. Now we are teaching children to accept being dependent on a single point of tech failure. Good old pen and paper is still better, your brain retains the info better too.

    -Time to seed my lawn, get off.

    1. Re:No School Today! by DogDude · · Score: 1

      Now we are teaching children to accept being dependent on a single point of tech failure

      That's exactly right. "Google's broken today, so we can't work! Oh, and there's literally nothing we can do about it. We can't even call Google. We just have to wait until Google deems we're ready to begin doing things again. That's the way we live our lives these days."

      --
      I don't respond to AC's.
    2. Re:No School Today! by thegarbz · · Score: 2

      Now we are teaching children to accept being dependent on a single point of tech failure.

      Haven't we always? Given the way people use their own personal devices I will happily recommend cloud based solutions to a lot of people. That "someone else" who runs "someone else's computer" is much better at it than the general population.

      Good old pen and paper is still better, your brain retains the info better too.

      At what? The latter part of your sentence seems to imply that they are reading from them, that's about one of the few things that Chrome books don't get used for in my wife's school. Now are pen and paper better at taking automated online tests? Sending emails? Editing documents? Visualizing complex math formulas? Displaying 3D models of the body?

      Sounds quite like you're making a lot of very critical assumptions to bolster your argument.

  10. Still better than Microsoft's KB4051033... by greenwow · · Score: 1

    which hammered our Windows servers for more than a week since it was a 1.1 GB update, and they for some reason decided to save the update in 11 KB files with another about 10 KB index file for every one of those files.

    1. Re:Still better than Microsoft's KB4051033... by Hal_Porter · · Score: 1

      You're supposed to run this -

      https://en.wikipedia.org/wiki/...

      The first version of WSUS was known as Software Update Services (SUS). At first, it only delivered hotfixes and patches for Microsoft operating systems. SUS ran on a Windows Server operating system and downloaded updates for the specified versions of Windows from the remote Windows Update site which was operated by Microsoft. Clients could then download updates from this internal server, rather than connecting directly to Windows Update. Support for SUS by Microsoft was originally planned to end on December 6, 2006, but based on user feedback, the date was extended to July 10, 2007.

      WSUS builds on SUS by expanding the range of software it can update. The WSUS infrastructure allows automatic downloads of updates, hotfixes, service packs, device drivers and feature packs to clients in an organization from a central server(s).

      --
      echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
  11. What? by 140Mandak262Jamuna · · Score: 1

    Manually joining 10000 chromebooks? There are 10,000 students and their care givers. Teach them to join wi-fi. It is a school! Teach them!.

    --
    sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
  12. Re:God forbid they learn how to use a damn compute by cdsparrow · · Score: 1

    Can't be that hard to change the wifi password to password1234 for the afternoon...

  13. What they wanted... by Dan+East · · Score: 1

    Isn't this what they wanted? To put all their eggs in one basket? Even though they knew Google was the one holding the basket.

    --
    Better known as 318230.
  14. "Tens of thousands, perhaps millions" by mlookaba · · Score: 1

    Hyperbole much?

  15. No, not quite that "simple" a problem by Pollux · · Score: 2

    No school today...because the vendor pushed out an untested update again.

    Let me help you, as it appears you didn't do 30 seconds of Googling to help yourself. Chrome OS is heavily beta tested, and is built upon Chromium OS, which it itself is heavily beta tested. As a Google admin for a public K-12 school (~1200 Chromebooks), I have the option of assigning all my Chrome devices into one of three categories of development. Google "recommends" I activate a policy that will randomly assign 5% of all devices to a beta channel* to assist them with testing and development, though our district chooses to use stable-only software.

    Occasionally, a serious bug actually does make it through to a stable, but if it is found, Google has been incredibly quick to prioritize its fix and release an update. It's only when there's a doozie like this where suddenly everyone starts the finger-shaming.

    * The first time I turned this on, the very next day, we had about five Chromebooks all come into our office. Every one of them had Chrome crashing randomly, usually within about 30 seconds of it opening up. All had the exact same version of Chrome on it, v.51 I think, when every other one of our Chromebooks had v.50 or below. The only way we were able to fix them was to use a CrOS repair drive utility to reinstall CrOS with a previous version. When I saw that other Chromebooks that had v.50 couldn't be upgraded to v.51, I reasoned that these were the beta tested Chromebooks. I turned that feature back off, but I still saw a few more Chromebooks trickle into my office over the next few days that also "got lucky". After that, never again.

  16. Many have survived their educational experience... by AmazingRuss · · Score: 1

    ... without a chromebook. The Children will be fine for 3 hours without one.

  17. Taxpayers by zerofoo · · Score: 1

    We have two schools, both with special-ed populations. One is taxpayer funded, the other is not.

    Administration has things to do besides spend money - take attendance, communicate with child study groups and parents, complete IEP documentation, deal with medical issues and records...etc. All of these tasks require technology - practically nothing is done with paper anymore.

    How do schools run in your area? With magic and free volunteers?

  18. Google Glitch by CustomBuild · · Score: 1

    Google glitch sounds like a new product. How did it take machines offline again?

  19. Re:Only 40 units affected by thegarbz · · Score: 1

    That makes a bit more sense.