Fcc lied about website hack taking down net neutrality comments

[bsaambl]

A senior US official has admitted to being the source behind a claim that the FCC was “hacked” in 2014 during the net neutrality debate. Internally, however, the agency’s security team had assessed there was no evidence of a malicious intrusion.

Dr. David Bray, who was the FCC’s chief information officer until last month, spoke privately with a reporter at Motherboard roughly a week after the FCC’s public comment website—known as the Electronic Comment Filing System (ECFS)—locked up after comedian John Oliver, host of HBO’s Last Week Tonight, directed his audience to flood the FCC with comments supporting net neutrality. Bray told the reporter that the agency had been the target of a “malicious attack.”

Bray was also the first US official to announce that the FCC had been attacked this year, too, after Oliver asked his audience once again to submit pro-net neutrality comments using the ECFS. Afterwards, the system became inaccessible on and off for roughly eight hours beginning the night of May 7, 2017. The FCC’s decision to withhold detailed analysis of the attack has prompted skepticism from reporters and the public at large.

Multiple FCC sources—including one with direct knowledge of the agency’s security operations—tell Gizmodo that, in June 2014, no evidence was ever produced that a cyberattack occurred. In the wake of Oliver’s net neutrality segment, the agency’s Network Security Operations Center (NSOC) pored over data collected by various logs. But it was unable to locate any proof to support Bray’s claim that a malicious attacker was responsible for the comment system’s failure.


Drawing from the statements of a senior FCC official (Bray), Motherboard described on June 10, 2014 a “malicious” attack carried out against the ECFS, a legacy system that had received few upgrades since its Clinton-era rollout. The ECFS was initially designed for lawyers and other knowledgeable sources to provide feedback on pending FCC regulations; but in a new era of digital civic engagement, the system became the principal tool for aggregating comments from the public about proposed rules to gut net neutrality.

Motherboard described a “malicious” attack carried out against the FCC, attributing the tip to a high-level agency source: The agency had been “hacked” by “unknown digital assailants” using what was described as “database Denial of Service tactics.” It was an “onslaught,” the site said.

Motherboard’s source was so well placed, in fact, the author wrote confidently that the FCC itself had “confirmed” the news. (The claim was supported by a second source as well, who had used words like “exploited” and “assaulted” to describe the incident.)

“It was never the official position of the FCC that it was a DDoS attack.”
But the tip was apparently based on the assumptions of the senior US official whose opinion did not comport with the findings of his agency’s security professionals.

“We couldn’t find any evidence of the attack,” said a former security contractor, who spoke on condition of anonymity to discuss their work at the agency. “We never took any remediation or mitigation steps with regard to security. There was no attack.”

The FCC’s press office was quick to refute reports that “scripts or automated bots” were responsible for the comment system’s troubles. “If anything, a high volume of traffic caused the collapse,” a reporter for Engadget wrote after speaking with the agency’s spokesperson.

“We stand by our story,” Motherboard’s editor in chief tweeted in response, saying that a “high-level FCC source” had described a “malicious attack.” (Motherboard confirmed last week that its source—whom Gizmodo has confirmed was Bray—used that term explicitly.)

“It was never the official position of the FCC that it was a DDoS attack,” Gigi Sohn, former counselor to then-Chairman Tom Wheeler, told Gizmodo. Yet, Bray “did not deny and there was never any doubt that he talked to Motherboard,” she said.

“My goal was to communicate on background that the commenting system had experienced abnormal ‘dead record locks’ and [had] not crashed from high comment volume,” Bray told Gizmodo on Saturday. “Multiple events were happening and the abnormal activity observed raised concerns that this was a form of malicious attack to tie up the system.”

“When pressed on the term ‘hack,’ I emphasized the system was not compromised,” he said, despite having given Motherboard a green light to use of word “hacked,” which appeared in its headline. 

Bray was interviewed later that year by TechRepublic and the Washington Post about ongoing efforts to revamp the FCC’s aging IT infrastructure. He never refers to a cyberattack crippling the ECFS.

Dead-locked records

In its official statement, the agency said in that a byproduct of receiving such a high volume of comments is what’s known as a “dead record lock,” whereby the ECFS’s database was overwhelmed in June 2014 and eventually froze. “This created difficulty for people trying to submit and search for filed comments,” it said. But the agency made no mention of any malicious activity.

Moreover, a “dead record lock” is not itself indicative of an attack. When overwhelmed, database systems are designed to initiate a “record lock” to preserve its integrity—i.e., prevent the database from being corrupted. While in this state, the ECFS would be unable to accept new comments, which is what happened on June 2, 2014, following the Last Week Tonight net neutrality segment.

Following the segment, the security operations center reviewed data collected in the FCC’s system logs, in its intrusion detection system, and from the multiple web and appliance-based firewalls from which logs were aggregated into a security information event manager, or SIEM. provided by McAfee. The security team came up empty handed.

The former security contractor told Gizmodo that the presence of any automated bots or scripted activity would have been detected through the use of meta-data analysis. The millisecond latency of requests coming from the same IPsource or session ID would have been a dead give away. Request activity faster than 10 milliseconds, for example, is almost certainly botnet activity. No abnormalities were detected, however.

The source described how an attack on the ECFS could have taken advantage of the record-lock procedure to force the system to freeze. A bot could have been engineered to flood the ECFS with comments attributed to hundreds or thousands of fictitious or stolen identities. Immediately after the comments were filed, the bot would’ve then sent a request to view the comment before the system had sufficient time to actually create the record. A flood of these requests would’ve inevitably overwhelmed the system.

“I checked for evidence of the theoretical attack above at the FCC in 2014 and could not find evidence of this,” the source said. Instead, the logs showed a high volume of commenters requesting access to the FCC web page that by default shows a list of newly submitted comments, what the source described as “normal intended use of the website which is in no way malicious.”

Weakness in the FCC codebase

After the record lock, the security team and the agency’s contracted developers discovered a weakness in the ECFS’s Sybase software, which was outdated by more than a decade. (A “weakness” is viewed as being less threatening than a “vulnerability” exploitable by hackers.) The software was, essentially, not configured to update new database rows properly, which created an inefficient procedure for adding new comments. This caused the system to lock up just after Oliver directed his viewers to to swarm the FCC’s site.

The development team documented the discovery in an application called Jenkins—the management system used to test and track updates to the FCC’s entire codebase.

“The security team was in agreement that this event was not an attack,” the former contractor said. “The security team produced no report suggesting it was an attack. The security team could not identify any records or evidence to indicate this type of attack occurred as described by Bray. The security team did not provide Bray with access to any security systems or logs that he might have performed his own independent analysis to come to this conclusion.”

His position as chief information officer notwithstanding, Bray’s access to security logs were restricted, the source said, under the principle of least authority—you only give people access to systems necessary to perform their job. If there was a security threat, Bray would have had to have relied on the security team to provide proof.

When Bray reached out on June 3, 2014 to the FCC leadership to report that someone was intentionally trying cripple the ECFS, the security team had detected nothing to suggest the kind of “malicious attack” Bray later described to Motherboard.

In a statement on Wednesday, Bray noted that fewer comments were received after Oliver’s show (roughly 25,000 per day) than in September 2014, when the ECFS handled more than 200,000 comments a day without the same issue. (The ECFS did experience periods of downtime that month.)

“The security team could not identify any records or evidence to indicate this type of attack occurred...”
However, a June 18, 2014 email acquired by Gizmodo, sent by Mary Ellen Seale, the Homeland Security official who served as FCC’s resiliency officer, states explicitly that the agency’s IT team had “installed a fix to ECFS to repair the record lock out problem.” Seale added: “They believe it is fixed for now.”

Bray initially told Gizmodo: “These allegations are false, and it is worth noting that these sources are suddenly raising allegations three years after the fact.” He continued: “To the extent that anyone at the FCC felt that the information publicly provided to the press in 2014 by everyone at FCC including myself was somehow inaccurate, they should have spoken up at the time, and the fact that they didn’t should cast suspicion on these unsubstantiated allegations that only have arisen now.”

On Friday, Bray clarified that when he said the “allegations are false,” he did not mean to suggest that he hadn’t spoken to Motherboard, only that the FCC’s IT team—which is separate from the security and development teams—had produced evidence that the ECFS “experienced an abnormal number of dead record locks.”

There are varying accounts of why, if the FCC actually believed an attack had occurred, it did not notify the US Computer Emergency Readiness Team (US-CERT)—the Department of Homeland Security organization that guidelines dictate cyber threats be reported to pursuant to the Federal Information Security Management Act (FISMA). The first, of course, is that there was no cyberattack.

The second, by those who believe an attack may have occurred—despite the lack of proof—is that ECSF is not considered mission critical. The FCC didn’t, in other words, want to waste US-CERT or the Federal Bureau of Investigation’s time. (The agency did contact the FBI this year, however, after the ECSF crashed following Oliver’s second net neutrality segment, though, the bureau declined to investigate.)

Several of Bray’s former colleagues declined to speak to Gizmodo on the record, though some who worked alongside him for years suggested his motivations were entirely pure. A former FCC advisor who worked closely with Bray touted the “tremendous job” he had done saving taxpayer money while updating the agency’s legacy systems.

“Even with transparency, we tech professionals all know there is no common definition of IT terms, and understanding of words can easily become interpreted differently by all humans involved,” the source said. “As someone from the Valley, I can attest that David and fellow change agents do the best for the public with limited IT resources that they can.”

Renewed controversy

Despite migrating from dilapidated servers to a cloud-based system, the ECFS continues to be a source of controversy for the FCC. Former Commissioner Ajit Pai, appointed FCC chairman by President Donald Trump, began a campaign this year to rollback Obama-era net neutrality regulations that made it illegal for internet providers to discriminate against content by blocking or slowing traffic to websites whenever they choose.

While the ECFS has remained online in 2017 an estimated 99 percent of the time, Bray’s announcement of a “cyberattack” against the system in May has been viewed with intense skepticism by the public and the press. The issue was compounded by the FCC’s reluctance to produce any definitive evidence of the attack. In response to a Freedom of Information Act request by Gizmodo in May, the agency withheld any records of substance that might exist indicating a malicious attack as the cause.

The ECFS has been inundated this year by fake comments allegedly from both sides of the net neutrality debate. A conservative group determined this year that hundreds of thousands of pro-net neutrality comments originated from addresses in foreign countries, including the United Kingdom and Russia. Likewise, thousands of anti-net neutrality comments appear to have been generated using stolen identities.

Twenty-seven Americans signed a letter addressed to Chairman Pai in late May asking the FCC to remove fraudulent comments attributed to them from the ECFS. “Whoever is behind this stole our names and addresses, exposed our private information in a public docket without our permission, and used our identities to file a political statement we did not sign onto,” the letter reads. It further warns that “hundreds of thousands of other Americans may have been victimized too.”

Pai has demonstrated a reluctance to exclude comments from the system—minus those falsely attributed to himself and the likes of “Joseph Stalin.” Gizmodo has learned that the agency’s officials fear being perceived as limiting the comment system in any way. The result, they believe, would be accusations that the FCC is trying to hinder the public’s participation.

Last month, the FCC was named defendant in lawsuits in Washington D.C. and the State of New York for improperly withholding documents in response to public records requests. One of the requests, to which the agency did not respond, demanded the release of documentation about the ongoing astroturf campaigns targeting the public comment system.