Fast Company/Steven Melendez: Facebook downplays “ambient audio” tech that can eavesdrop on you

Fast Company   

    06.29.186:00 am

Facebook downplays “ambient audio” tech that can eavesdrop on you
At least two Facebook patents describe listening to users’ TVs. Facebook says it will “never” deploy the technique, but other firms are using similar methods.
[Animation: joshborup/Pixabay; Josy_Dom_Alexis/Pixabay]

By Steven Melendez 5 minute Read

Amid persistent rumors that Facebook is spying on you through your phone’s microphone–rumors that the company has denied since at least 2016–reports emerged this week that the company has applied for a patent on technology that listens to “ambient audio” to detect what you’re watching on TV.
advertisement

Essentially, your phone or another device would detect, via Bluetooth or other signals, that it’s near your turned-on TV and would record short snippets of sound. TV shows and ads could include high-pitched signals identifying precisely what you’re watching, and that data could be beamed back to a central server to “update user profiles of the individuals” in your household in order to better target users with personalized content, according to the patent application filed in late 2016.

A diagram from a 2016 Facebook patent application for analyzing “ambient audio” [Photo: courtesy of United States Patent and Trademark Office]
This isn’t the company’s only technology for capturing and identifying audio signals in a user’s environment. In another patent application filed in 2015, Facebook describes a system that can capture data about what a user is watching by using “acoustic fingerprinting” to analyze ambient audio, or even by analyzing electrical signals emitted from an HDMI cable or a television. Neither patents mention listening in on users’ voices or conversations.

When reached by Fast Company, Facebook said it will “never” actually put the patent-pending technology to use. According to an emailed statement from Allen Lo, VP, deputy general counsel and head of intellectual property at Facebook:

    It is common practice to file patents to prevent aggression from other companies. Because of this, patents tend to focus on future-looking technology that is often speculative in nature and could be commercialized by other companies. The technology in this patent has not been included in any of our products, and never will be. As we’ve said before, we often seek patents for technology we never implement, and patent applications should not be taken as an indication of future product plans.

And when asked during his April U.S. Senate testimony whether Facebook used “audio obtained from mobile devices to enrich personal information about its users,” Zuckerberg answered, “No.”

Of course, the company stores audio and video deliberately uploaded and shared by its users, and its terms of service give it fairly wide latitude to analyze and process that data. But the company says it has no plans to use ambient sounds to shape ads or users’ feeds.

“Facebook has never used your phone’s microphone to inform ads or to change what you see in News Feed, and we have no plans to do so in the future,” said Facebook’s vice president of ads Rob Goldman in a statement shared with Fast Company.
advertisement

Meanwhile, rumors suggest that Facebook will soon launch a smart speaker product similar to Amazon Echo and Google Home, which would likely revive theories of spying, even unintentionally. Amazon devices have been known to misinterpret conversations as commands to record and transmit audio, and the devices generally work by constantly listening for cues in people’s conversations, even if they don’t retain that incidental audio or send it to the cloud.
It’s already happening

Capturing audio to figure out what you’re watching might sound outlandish, but it’s actually being done. A company called Alphonso does just that, using modules in apps such as games to listen to TV audio and identify shows, the New York Times reported last year. The company says it doesn’t pick up human voices, and users of apps with the code can opt out at any time.

In 2016, the Federal Trade Commission (FTC) wrote to developers warning them that if they used another TV-observing app module by data firm Silverpush and didn’t properly notify users, they could be violating the law. At the time, Silverpush said its technology wasn’t in use in the United States, according to the FTC, but vowed that it would kill the software anyway due to a “business decision.”

A year later, however, researchers found over 200 Android apps still using Silverpush’s inaudible sound “beacon” technology to spy on what TV commercials users were watching. The apps, including some used overseas and connected with large companies like McDonald’s and Krispy Kreme, had collectively been downloaded millions of times from the official Google Play market, the researchers wrote in a May 2017 paper.

Many smart TVs already do phone home with information about what you’re watching, provided you give consent in the often-dense menus you see when you first get the device home and plug it in. The TVs generally don’t need to listen to external audio to learn what you’re watching, since they already have that data. By gathering your IP address, broadcasters and advertisers can also use that information to match your viewing habits with other information known about your online behavior.

For instance, 4C, an analytics company and early Facebook and Twitter marketing partner, uses various techniques to help determine users’ offline activity and pair digital ads with live TV moments and commercials. Before it collapsed, disgraced elections firm Cambridge Analytica was touting similar techniques.
advertisement

Even if marketers aren’t literally listening in, hackers and stalkers have been known to take over webcams and microphones to spy on innocent people. (Even Facebook CEO Mark Zuckerberg reportedly covers the camera and mic on his laptop.) Last year, a leak of CIA cyberweapons revealed a tool for capturing audio recorded by Samsung’s Smart TVs. And some companies offer spying apps that you can install on smartphones (often targeted at employers and parents) to monitor use, including turning on the microphone to eavesdrop as you see fit.

Related: Hackers Could Use A Pop Song To “Watch” You Through Your Smart Speaker

To prevent clandestine audio monitoring by apps, users can use their smartphone privacy settings to disable an app’s access to the microphone. This, however, can also prevent a number of useful features, like recording video with sound.

The bottom line is that advertisers have found they like the wealth of data they can get about your online habits, and there’s likely to be a continual push to find ways to get similar data about what you’re watching, listening to, and even discussing in the physical world. Even if Facebook isn’t listening, that doesn’t mean it technically can’t or that it won’t some time in the future.
About the author

Steven Melendez is an independent journalist living in New Orleans.

More
You Might Also Like:

    7 warning signs that you shouldn’t accept a job offer
    Remember, Alexandria Ocasio-Cortez also wrote her own winning ad
    This RV Full Of Lawyers Is Touring Rural America To Save Young Immigrants From Deportation
    Video: This company wants you to drink your beer--and eat it, too

Technology Newsletter
Receive special Fast Company offers.
See All Newsletters
Sponsored Financial Content
Powered by Dianomi
Where is the clever money going?
MarketViews
Where is the clever money going?
Latin America’s Renewable Energy Revolution
LatAm Investors
Latin America’s Renewable Energy Revolution
Content Marketing in Financial Services - 2019 Benchmarking Survey
Editions Financial
Content Marketing in Financial Services - 2019 Benchmarking Survey
Video
How AI could be impacting childhood development
Ideas
Ideas
Trump’s tax law will likely hurt small charities the most
Ideas
Activism built America’s infrastructure once, and can do it again
Ideas
Nonprofits remain victims of their own ambition
Entertainment
Entertainment
Top 5 Ads Of the Week: Netflix diversity, Airbnb travels forward
Entertainment
Why AT&T Should Pay Attention to Go90’s Demise
Entertainment
CAA database of TV writers of color addresses dearth of diversity
Co.Design
Cities & Spaces
Design is like a drug
Products
Apple and Samsung’s truce is the end of an era for design
Cities & Spaces
How the Alexandria Ocasio-Cortez campaign got its powerful design
Fast Company
Leadership
I left my dream job at Facebook: Here’s the best way to quit a job
Leadership
10 digital nomads share how they stay focused working abroad
Entertainment
Top 5 Ads Of the Week: Netflix diversity, Airbnb travels forward

    AdvertisePrivacy PolicyTermsContactAbout UsSite MapFast Company & Inc © 2018 Mansueto Ventures, LLCDigital Advertising Alliance (DAA) Self-Regulatory Program