MIT Technology Review/Arm: From cloud to the edge: On-device artificial intelligence boosts performance

MIT Technology Review

Artificial Intelligence

From cloud to the edge: On-device artificial intelligence boosts performance
AI can boost performance, security, and cost savings—but building any AI-enabled product requires careful use of optimized computing.
May 16, 2019
Sponsored Content

Produced in association with Arm

If artificial intelligence (AI) goes according to plan, we’ll barely notice it taking hold. As a result, and despite the hyperbole, AI may be the quietest major computing revolution the world has ever known. What’s happening at one of the world’s leading children’s hospitals is a great example.

Great Ormond Street Hospital (GOSH) clinicians see more than 300,000 children every year, many of them with critical care needs. To ensure its patients receive the best possible care in a safe and secure environment, GOSH began testing an AI-based person recognition system where medical staff, patients, and authorized visitors receive access to certain secure areas of the hospital while any unauthorized entrants are either stopped or flagged by the system. The solution uses a network of AI-enabled smart cameras to examine each person’s face, body structure, and gait. The system then automatically cross-checks facial features against a database of registered people. The system has increased hospital security and has clinical benefits, too. For example, if a child requires immediate care, an emergency room doctor can quickly be located and notified, ensuring the team is ready to spring into action when needed.
ARM Image
everythingpossible/123RF

In the past, coping with such a sophisticated system would have required a sprawling data center and its associated costs. But the AI revolution has sparked a movement to perform AI computing differently. Instead of a cloud link, data generated by GOSH’s innovative cameras is processed locally on the cameras themselves using a tiny chip. Not only does this “AI at the edge” system process data faster and more cost efficiently, it never leaves the confines of the hospital.
The Next Era of Computing – Machine Learning for Every Device

    Learn MoreThe way we interact with machines is changing. Arm’s Project Trillium will transform our lives through a new class of advanced and ultra-efficient machine learning processors purpose-built to redefine device capabilities.

Machine learning in action

A branch of AI, machine learning (ML) uses sophisticated algorithms in models that can learn from data and identify important patterns. By uncovering connections, ML helps businesses make better decisions without the need for human input.

Today, ML is powering all kinds of applications, many of which are mobile, as smartphone users climb to an anticipated 3.8 billion by 2021. Examples range from fingerprint recognition and photo-sorting to more innovative use cases, including:

Smart inhalers: AI-powered inhalers run real-time ML algorithms that calculate a patient’s lung capacity and breathing patterns. This data is then interpreted on the device itself and sent to a smartphone app, enabling healthcare professionals to personalize regimens for asthma sufferers based on detailed sensor data.

A branch of AI, machine learning (ML) uses sophisticated algorithms in models that can learn from data and identify important patterns

Robot companions: An AI-driven social robot for senior citizens uses ML to understand the preferences, behavior, and personality of its owner. Based on these interactions, the robot can automatically connect older adults to stimulating digital content, such as music or audiobooks, as well as recommend activities, remind the user about upcoming appointments, or connect to family and friends through social media. And unlike most AI systems, which require voice activation, the robot proactively communicates with its user. For example, if a senior citizen has been sitting for an extended period of time, the robot can automatically recommend calling a friend or taking a walk.

Reindeer cam: A smart camera system detects herds of reindeer through ML algorithms as they approach train tracks in remote parts of Norway where the animals are often needlessly killed. By processing information on the device itself, the system can warn train operators in real-time to reduce speeds when the animals are present, thereby preventing accidents and train delays.
The edge advantage

Hardware vendors are taking note and increasingly equipping devices with ML-capable chips. As a result, these devices are capturing and processing data in real time, providing instantaneous situational analysis, identifying patterns, and supporting quick AI-enabled decision making.

Edge AI devices are mainly running ML inference workloads—where real-world data is compared to a trained model. The models they use are mostly built in the cloud due to the heavy compute requirement of building an AI model. However, even with AI training, we are starting to see edge devices used as trainers as they learn in real-world environments.

The timing couldn’t be better. We’ve reached a critical mass of compute resources in the cloud. Around 29 billion connected devices are predicted by 2022, of which nearly 18 billion will be related to the Internet of Things. At the same time, the average consumer will own 13 connected devices by 2021 as autonomous vehicles populate our roads and sensors spread from factory floors to rural farms, each vying for precious compute power.

“With today’s exponential explosion of intelligent devices, there simply aren’t enough data centers in the world to send all of the data to the cloud,” says Ian Bratt, an engineering fellow at Arm Limited, where he leads the machine learning technology group.

Moving ML workloads to the edge can provide a number of key advantages. These include:

Heightened speed and performance: Whether searching for a weather update or driving directions to a restaurant, today’s mobile user demand fast access to critical information. But sending data back and forth to the cloud can result in latency, which can negatively impact time-critical applications. On-device learning, however, delivers increased responsiveness for immediate insights.

Enhanced privacy and security: The global average cost of a data breach is $3.86 million, according to a study by IBM and the Ponemon Institute. Unfortunately, shipping data to the cloud for processing creates an opportunity for that information to get hijacked by a cybercriminal. One sure-fire way to prevent an attack is by making sure sensitive data never leaves a device. On-device ML also provides decentralization, making it more difficult for hackers to launch an attack compared to a single, centralized server.

Cost savings:  A 2018 Frost & Sullivan survey reveals that 49 percent of IT decision-makers struggle to manage costs associated with running cloud workloads. And a TECHnalysis Research report on AI reveals that 50 percent of non-AI users cite cost as a chief concern. ML at the edge can alleviate the financial burden by reducing reliance on costly cloud services and infrastructure.

Together, these advantages, combined with an exponential growth in AI compute, are spurring an on-device ML revolution.
Sussing Silicon: How to select the microprocessor that’s right for your task

Yet with the amount of compute power used in AI-training models doubling every hundred days, many organizations are questioning where ML workloads should be performed—on a central processor (CPU), a graphics processor (GPU), or a neural processor (NPU)?

The answer hinges on a number of factors, including how quickly tasks need to be executed, the compute performance required, and whether it’s worth the extra cost of adding an NPU or GPU to a system design.

Some training and inference algorithms are so complex, and the data sets so large, that faster computation solutions, such as a GPU and NPU, are worth considering as co-processors to the CPU every system will already have. But it’s also critical that organizations consider the benefit of including custom silicon when CPUs are becoming increasingly AI-capable.

“A lot of AI is still being done on CPUs,” says Bob O’Donnell, founder and chief analyst of TECHnalysis Research. “Yes, it’s great that there are other kinds of chips that can be used for certain AI tasks and workloads. However, because every single device out there has CPU, it’s an excellent baseline.”

Dean Wampler agrees. A vice president at Lightbend, which provides an open-source platform for developing cloud-native applications, Wampler says companies are rethinking how they use GPU resources with an eye toward trying “to minimize the compute overhead required” for a task.

Instead, he says, more and more clients are realizing they “can be clever and exploit the compute power that they already have” in a CPU. The result: minimal overhead and less strain on resources for maximum throughput.
When CPU isn’t enough

That’s not to suggest, however, that there’s not enormous value in GPU and NPU. “In some of today’s more advanced, forward-thinking applications, GPUs are incredibly important,” says O’Donnell. Although best known for graphics, video, and photo processing, GPUs are gaining favor in the finance and scientific research sectors for accelerating computational workloads.

NPUs, on the other hand, are several times faster than GPUs, making them best-suited for compute-intensive tasks and heavy workloads, while the general availability and programmability of CPUs make them an excellent default option for mobile inference.

The key is to take advantage of all three forms of computational power depending on the ML-related task at hand. Certainly, CPU may be the first choice for ML processing, but in cases where responsiveness or power efficiency are imperative, it helps to complement a CPU with a dedicated NPU which offers greater efficiency and higher performance. In many situations, a layer of software that lives on the CPU can help play real-time traffic cop, determining the right processor for the right task.
A look forward

These are early days for intelligent devices. “We’re in the first phase of the AI and machine learning revolution,” says O’Donnell. But use cases are fast evolving from voice recognition and photo filters to life-saving devices, driving the demand for unprecedented compute power. Moving ML workloads to the edge can help improve performance and efficiency. But carefully considering which programming approach is best, and on which platform, is what will ultimately ensure staying in the game.
This story was commissioned by an advertiser and was produced by a dedicated custom content team, not our journalists. Please see our advertising guidelines to learn more.
Share
Popular

    01.
    Ford thinks robots and self-driving cars could team up to deliver packages
    02.
    Female voice assistants fuel damaging gender stereotypes, says a UN study
    03.
    Google’s AI can now translate your speech while keeping your voice

Blockchain
The cryptocurrency startups trying to save Argentina from itself
By holding their money in “stable” cryptocurrency, Argentines might avoid the notorious volatility of their peso. First, though, they have to be persuaded.
Photograph of Buenos Aires
Artificial Intelligence2h
Ford thinks robots and self-driving cars could team up to deliver packages

Autonomous vehicles and robots could share sensor data to help them better navigate the world around them....

The news: Ford is going to test a legged robot that unfolds from the back of an autonomous car to bring parcels to people’s doors. It’s teamed up with Agility Robotics, using its “Digit” robot to try out the idea. One day, a driverless taxi trip could double as a delivery service, dropping packages off between rides, Ford’s CTO suggested in a blog post.

Digit: Ford says the robot can carry packages up to 40 pounds, walk up and down stairs, work around obstacles, and regain its balance if it’s bumped. Bipedal robots have some advantages over wheeled ones: they can deal with obstacles and stairs more easily. However, they’re slower and less stable. Could Digit get up again if it was pushed over, for example?

A crowded market: The boom in home deliveries means companies are scrambling to find quicker, cheaper ways to get goods to your front door. Amazon and FedEx are working on their own pilots, and several smaller players have launched deliveries on college campuses.

A compelling combination: The combination of driverless car and robot is compelling, especially because the two could share camera and LiDAR sensor data to help each better understand their surroundings. The robot could also charge in the car, helping to reduce the need for lots of bulky batteries.

However, we’re still many, many years away from this concept becoming a reality. Ford’s own CEO recently admitted that driverless cars are still years away, and there are still plenty of technical barriers to overcome before we ever see the robot/car duo launched in the wild.

Sign up here to our daily newsletter The Download to get your dose of the latest must-read news from the world of emerging tech.
Share
Author

Charlotte JeeI write The Download, the only newsletter in tech you need to read every day. Before joining MIT Technology Review I was editor of Techworld. Prior to that I was a reporter covering the intersection of politics, the public sector and technology. In my spare time I run a venture called Jeneo aimed at making tech events more inclusive. I regularly do public speaking and crop up on the BBC from time to time. Sign up for The Download here.

Expand
Artificial IntelligenceMay 22
America and its economic allies have announced a “democratic” vision for AI

The Trump administration might be building walls between America and some countries, but it is eager to forge alliances when it comes to shaping the course of artificial intelligence....

The Organisation for Economic Co-operation and Development (OECD), a coalition of countries dedicated to promoting democracy and economic development, has announced a set of principles for artificial intelligence. The announcement came at a meeting of the OECD Forum in Paris.

The OECD does not include China, and the principles outlined by the group seem to contrast with the way AI is being deployed there, especially for face recognition and surveillance of ethnic groups associated with political dissent.

Speaking at the event, America’s recently appointed CTO, Michael Kratsios, said, “We are so pleased that the OECD AI recommendations address so many of the issues which are being tackled by the American AI Initiative.”

The OECD Principles on AI read as follows:

— AI should benefit people and the planet by driving inclusive growth, sustainable development and well-being.

— AI systems should be designed in a way that respects the rule of law, human rights, democratic values and diversity, and they should include appropriate safeguards—for example, enabling human intervention where necessary—to ensure a fair and just society.

— There should be transparency and responsible disclosure around AI systems to ensure that people understand AI-based outcomes and can challenge them.

— AI systems must function in a robust, secure and safe way throughout their life cycles and potential risks should be continually assessed and managed.

— Organisations and individuals developing, deploying or operating AI systems should be held accountable for their proper functioning in line with the above principles.


Share
Author

Will KnightWill Knight is MIT Technology Review’s Senior Editor for Artificial Intelligence. He covers the latest advances in AI and related fields, including machine learning, automated driving, and robotics. Will joined MIT Technology Review in 2008 from the UK science weekly New Scientist magazine.
ImageOECD

Expand
ComputingMay 22
US political parties are still making basic cybersecurity blunders

Many of the biggest political organizations in the US still have awful cyber hygiene ahead of next year’s election....

The news: Researchers at cybersecurity firm SecurityScorecard spent the first quarter of 2019 analyzing the anti-hacking defenses of the parties, including both the US Republican National Committee (RNC) and the Democratic National Committee (DNC). They found that both have some serious holes to address.

The dirty truth: The flaws include exposed personal data about employees that could be used to create fake identities; older versions of software that could let hackers steal usernames and passwords fairly easily; and malicious software, or malware, that could be used to spy on party activities and compromise user accounts.

Why this matters: Ahead of the 2016 US presidential election, hackers penetrated the DNC’s systems and stole e-mails and other data to cause chaos. With European Union parliamentary elections looming and the US about to enter another presidential election year, more attacks on political organizations are inevitable.

Bigger is (somewhat) better: The researchers acknowledge that the RNC and DNC have put significant effort into bolstering their cyber defenses since 2016 but say they still found some (undisclosed) weaknesses. Another, smaller party was using a tool that leaked voter names, dates of birth, and addresses. This flaw was fixed after the party was told what SecurityScorecard had found.
Share
Author

Martin GilesI am the San Francisco bureau chief of MIT Technology Review, where I cover the future of computing and the companies in Silicon Valley that are shaping it. Before joining the publication, I led research and publishing at a venture capital firm focused on business technology. Prior to that, I worked for The Economist for many years as a reporter and editor, most recently as the paper’s West Coast-based tech writer.
ImageAP

Expand
Biohacking
This celebrity biohacker is under investigation for practicing medicine without a license
He taunted the health authorities. Now he stands accused of pretending to be one.
01.
A biotech CEO explains why he injected himself with a DIY herpes treatment on Facebook Live
February 2018
02.
Biohackers disregard FDA warning on DIY gene therapy
December 2017
03.
Why Kickstarter’s glowing plant left backers in the dark
July 2016
04.
Can hobbyists and hackers transform biotechnology?
April 2011
Climate ChangeMay 22
How scientists unraveled the mystery behind the return of a banned greenhouse gas
Artificial IntelligenceMay 22
This robot watches you flex to learn to be a better teammate
An MIT robot collaborates with a person by tracking his or her muscles.
Apollo 50th anniversaryMay 22
The NASA engineer who helped Apollo astronauts catch their ride home
To get back from the lunar surface, the astronauts had to rendezvous in lunar orbit. Sheila Thibeault helped make that possible.
Read more
Silicon ValleyMay 22
Female voice assistants fuel damaging gender stereotypes, says a UN study

Products like Amazon Echo and Apple’s Siri are set to sound female by default, and people usually refer to the software as “her.”...

Embedding bias: Most AI voice assistants are gendered as young women, and are mostly used to answer questions or carry out tasks like checking the weather, playing music, or setting reminders. This sends a signal that women are docile, eager-to-please helpers without any agency, always on hand to help their masters, the United Nations report says, helping to reinforce harmful stereotypes. The report calls for companies to stop making digital assistants female by default and explore ways to make them sound “genderless.”

Who’s blushing: The report is titled “I’d blush if I could,” after a response Siri gives when someone says, “Hey Siri, you’re a bi***.” It features an entire section on the responses to abusive and gendered language. If you say “You’re pretty” to an Amazon Echo, its Alexa software replies, “That’s really nice, thanks!” Google Assistant responds to the same remark with“Thank you, this plastic looks great, doesn’t it?” The assistants almost never give negative responses or label a user’s speech as inappropriate, regardless of its cruelty, the study found.

The report: Its aim is to expose the gender biases being hard-coded into the technology products that are playing an increasingly big role in our everyday lives. The report also suggests ways to close a gender skills gap that is wide, and growing, in most parts of the world. It found women are 25% less likely to have basic digital skills than men, and are only a fourth as likely to know how to program computers. “These gaps should make policy-makers, educators and everyday citizens ‘blush’ in alarm,” the report says.

Sign up here to our daily newsletter The Download to get your dose of the latest must-read news from the world of emerging tech.
Share
Author

Charlotte JeeI write The Download, the only newsletter in tech you need to read every day. Before joining MIT Technology Review I was editor of Techworld. Prior to that I was a reporter covering the intersection of politics, the public sector and technology. In my spare time I run a venture called Jeneo aimed at making tech events more inclusive. I regularly do public speaking and crop up on the BBC from time to time. Sign up for The Download here.
ImageAssociated Press

Expand
People are calling for Zuckerberg’s resignation. Here are just five of the reasons why.
How the data mining of failure could teach us the secrets of success
Sign up for The Download — your daily dose of what's up in emerging technology
Also stay updated on MIT Technology Review initiatives and events?
YesNo