Around 2:30 p.m. on Tuesday, Adam Henderson began receiving a flood of reports from across his district’s 17 schools saying their Chromebook devices were no longer working.
Henderson, who is the Director of Technology Systems at Nassau County Schools in Florida, was among the many educators who experienced the issue, which caused Chromebook devices to logout of existing internet accounts and fail to reconnect.
In Nassau County, 5,000 devices experienced the outage, but it’s expected that tens of thousands, and perhaps millions of devices around the country, could have experienced the same glitch.
Shortly after devices went down, users took to both Twitter and Reddit looking for answers.
Google did not respond to questions from EdSurge about the issue or how it was resolved, but it’s believed to originate from a “botched WiFi policy update pushed out by Google that caused many Chromebooks to forget their approved network connection,” Frank Catalano wrote in GeekWire.
ITnews also reported that “Chromebook systems administrators say it is likely related to Google Apps for Education devices having their WiFi passwords and base station identifiers deleted.”
Google commented on the issue in a tweet stating that “The issue is resolved.” The company’s suggestion to get Chromebooks back online was to “reboot & manually join a WiFi network or connect via ethernet to receive a policy update.”
Since the issue involves forgetting a device’s existing SSID, schools must either create a new network for students to log into, or log the devices into existing alternative networks. Henderson says his district is taking the latter approach, and will re-connect devices to a “guest” network and update the policies. From there, Google claims the devices will automatically switch back over to the preferred network.
Still, logging into a guest network can be risky, as Catalano points out, because it puts students on an unsecure network. And logging each device into the guest account will likely take district IT workers several hours come Wednesday morning.
“Tomorrow we will be there early,” says Henderson. “We will really feel it in the morning.”
Google has since noted the error on its support page, which now reads: “Managed Chrome devices may have received invalid network policies and lost connectivity to passphrase-protected networks configured through admin policies. We have addressed the root cause of this issue, but due to the nature of issue, the fix does not take effect until affected devices connect to networks and refresh policies.”
The company also published a help article for users to tell if they were affected and how to manually reconnect devices.
Some users were still unsatisfied with Google’s late support and tweet, however, which came nearly three hours after reports of the issue began circulating on the internet and educators began trying to contact Google.
Henderson says it took him more than an hour to connect with Google to get a temporary solution. During that time, the district’s middle schools, which are 1:1, were in session and students and teachers using them had to quit their digital activities.
An educator in Chicago claimed on Reddit to have a similar experience, writing that teachers were in the middle of testing when the outage reached their classrooms.
For Henderson, the event is unprecedented. “We have had these in our district for four years and this is the biggest issue that we faced,” he says. “That’s the downside to a 100 percent cloud-based solution. If something happens in the cloud, it’s out of local IT control.”
He added that “diversifying how our devices authenticate, locally or on the cloud, is something that we will really take a look at going forward.”