How Two Amateur Sleuths Looked for FinSpy Software

Egyptian protestors found a Gamma Group proposal to sell FinSpy to Egypt’s security forces.Egyptian protestors found a Gamma Group proposal to sell FinSpy to Egypt’s security forces.

On Thursday, one day after security researchers disclosed that FinSpy, an elusive, off-the-shelf surveillance software, was spying on computers and mobile phones from servers in as many as 15 countries, the servers began to go dark.

One by one, servers in Singapore, Brunei, Indonesia and Mongolia disappeared. Another in Bahrain went down then appeared to quickly reincarnate elsewhere.

The article on Friday in The New York Times outlines the use of the software around the world. The tale of how the two sleuths discovered it is an interesting one. Here are more of the details.

On Friday, as Americans head out the long weekend, Bill Marczak and Morgan Marquis-Boire will be glued to their computers screens, analyzing malware samples and watching as servers in Turkmenistan, Ethiopia and the United Arab Emirates continue to track unsuspecting targets.

Both have day jobs. Mr. Marquis-Boire, 32, works as a security engineer at Google. Mr. Marczak, 24, is earning his Ph.D. in computer science at the University of California-Berkeley. And until a few months ago, they had never met. In May, a journalist asked the two of them to take a look at some suspicious e-mails sent to three Bahraini activists, one based in London, one in Washington and a third in Manama, Bahrain’s capital.

What followed was four months of painstaking forensics work that — with every new clue — pointed to the widespread use of a sophisticated, off-the-shelf cyberespionage tool by governments like Qatar and Brunei that hgave questionable track records on human rights.

Theirs is yeoman’s work. The spyware is so targeted in scope that the two are treading where antivirus giants — like Symantec and McAfee — will not go. But the spyware is invasive enough, and sly enough, that Mr. Marquis-Boire and Mr. Marczak have given up their nights and weekends to find out who is being tracked on the other end.

The two discovered that the e-mails sent to the Bahrani activists all reported back to the same command-and-control server in Bahrain. They found that the spyware could grab images of computer screens, record Skype chats, turn on cameras and microphones and log keystrokes. But what made it especially sophisticated was how well it flew under the radar. Its creators had specifically engineered it to elude antivirus makers like Kaspersky Lab, Symantec, F-Secure and others.

But their most intriguing finding was a small but telling word embedded in the spyware’s code: “FinSpy.”

FinSpy is one product in a larger FinFisher surveillance product line sold by Gamma Group, a British company that says it sells monitoring software to governments to track criminals.

FinSpy first gained attention in March 2011 after Egyptian protesters raided the state’s security offices and discovered a Gamma Group proposal to sell FinSpy to Egypt’s security forces for 287,000 euros, or $353,000.

Nine months later, Wikileaks published leaked Gamma Group brochures that summarized what FinSpy could do: record e-mail, instant messages and Skype chats, spy on Web cameras and microphones, log keystrokes and circumvent 40 different antivirus systems — precisely the same functions of the spyware Mr. Marczak and Mr. Morgan-Boire discovered in Bahrain. The company says it is used by governments to track criminals.

But the apparent use of the spyware to spy on Bahraini activists — none of whom had any criminal history — suggested it had been used to focus on dissidents.

That was the conclusion Mr. Marczak and Mr. Marquis-Boire came to when they published their initial findings with the Citizen Lab of the Munk School of Global Affairs at the University of Toronto last July.

Since then, the two have started collecting malware samples from other security researchers as well as activist groups who suspect that they may have been tracked with FinSpy, too.

The two have spent the last month studying those samples from their respective apartments and sharing their findings through encrypted chat and e-mail services. Occasionally, they meet over coffee to share their findings in person.

They were helped by a computer researcher at Rapid7 who identifies himself as Claudio Guarnieri, who took a close look at the command-and-control server in Bahrain and found that when he sent it an unexpected message, it responded: “Hallo Steffi.” It turned out to be an important clue.

Rapid7, based in Boston, scoured the Internet to see if other servers returned the same message and found 11 I.P. addresses in 10 more countries: Indonesia, Australia, Qatar, Ethiopia, the Czech Republic, Estonia, Mongolia, Latvia, the United Arab Emirates and the United States, where an Amazon-hosted server appeared to be running FinSpy. Amazon did not return requests seeking more information about that I.P. address. But Mr. Marczak said that the server appeared to be a proxy that conceals traffic for another server.

The Gamma Group denied that the servers Rapid7 discovered were running FinSpy. “FinFisher servers would not respond in such a way and would not be able to be fingerprinted with such a technique,” said Martin J. Muench, a Gamma Group managing director. “None of our server components send out strings like ‘Hallo Steffi.’”

Using a different fingerprinting method, Mr. Marczak and Mr. Marquis-Boire also scanned the Internet and stumbled on many of the same servers as Rapid7. In one case, Mr. Marquis-Boire discovered a sample that was speaking to the server Rapid7 had identified in the Czech Republic. “I was holding the other end of what they had found,” Mr. Marquis-Boire said.

On Wednesday, the researchers announced one of their biggest discoveries yet. They discovered new mobile versions of the spyware that had been customized for Apple’s iOS, Google’s Android, Windows Mobile, Nokia’s Symbian and BlackBerry’s mobile operating systems.

Each could monitor calls, text messages, e-mails and, in the case of BlackBerry, read BBM messages. They could also steal a user’s address book, transmit a target’s location and spy on third-party applications like WhatsApp, a popular, free texting app.

One of the more intriguing tidbits to come out of the research by Mr. Marquis-Boire and Mr. Marczak was a familiar name: Johnny Debs.

They said that the version of the spyware that focused on Nokia’s Symbian system infected phones through a fake software update. But software updates — even fake ones — require certification. The certificate had been signed by an employee at a company called Cyan Engineering. A Web site for Cyan Engineering was registered anonymously and clicks led to several “Under Construction” messages.

But the researchers noted that Cyan Engineering Services also appeared as the registrant for a second Web site, www.it-intrusion.com. That Web site was registered by somebody by the name of Johnny Geds, a name that had popped up once before.

That name — Johnny Debs — was listed as Gamma Group’s sales contact on the FinSpy proposal uncovered in the raid on Egypt’s security headquarters.

Asked about the possible connection, Mr. Meunch confirmed Johnny Debs is a sales employee at Gamma Group but did not elaborate.