Edge computing is a multi-choice puzzle for IT architects and embedded developers. Ultimately, it can create cutting-edge AI, enabling faster and richer decision making.
AI-based machine learning techniques go beyond the cloud-based data center, as the processing of vital IoT sensor data approaches where the data first came to be.
The move will be made possible by new chips equipped with artificial intelligence (AI). They include built-in microcontrollers with narrower memory and power consumption requirements than GPUs (graphics processing units), FPGAs (field-programmable fields), and other specialized IC types used for the first time to answer data scientists ’questions in cloud data centers Amazon Web Services, Microsoft and Google.
Machine learning and a related neural network exploded in those clouds. But the rise of the IoT has created assault data that also required edge-based machine learning.
Now cloud vendors, Internet of Things (IoT) platform manufacturers and others see the benefit of edge data processing before handing it over to the cloud for analytics.
Edge-based AI decision-making reduces latency and makes real-time response to sensor data more convenient and possible. Yet what people call “marginal intelligence” takes many forms. And how to power it with next-generation IoT presents challenges in terms of presenting quality data that can be exploited.
Edge computing workloads are rising
Edge-based machine learning could spur significant growth in artificial intelligence in the IoT market, which Mordor Intelligence estimates will grow to 27.3% CAGR by 2026.
This is supported by the 2020 Eclipse Foundation IoT Group survey, which found AI at 30% as the most frequently cited workload among IoT developers.
For many applications, replicating endless server shelves that enable parallel machine learning in the cloud is not an option. There are many cases of IoT edges that benefit from local processing, and they are highlighted by various cases of monitoring operations. Processors, for example, monitor events caused by changes in the manometer on an oil rig, by detecting an anomaly on a remote transmission line, or by recording video surveillance of problems at the factory.
The latter case is one of the most researched. The application of AI that parses the image data at the edge proved to be a fertile area. But there are many complex processing needs to process events using data collected on IoT devices.
Edge Compute value
Still, cloud-based IoT analytics will hold up, said Steve Conway, a senior advisor to Hyperion Research. But the distance data must overcome the processing delay. Moving data in and out of the cloud naturally creates lag; the return trip takes time.
“There’s something called the speed of light,” Conway said. “And you can’t exceed it.” As a result, a processing hierarchy has developed at the edges.
In addition to board-level devices and implementations, this hierarchy includes IoT gateways and production data centers that extend the architectural options available for next-generation IoT system development.
In the long run, the edge AI architecture is another generational shift in the focus of data processing – but crucial, according to Saurabh Mishra, senior product marketing manager in SAS’s IoT and Edge departments.
“There’s progress here,” he said. “At one time the idea was to centralize your data. “You can do that for certain industries and certain uses – those where the data has already been created in context, for example in a data center,” he said.
In fact, it is not possible to efficiently – and cost-effectively – move it into the analysis cloud, ”said Mishra, noting that SAS has created proven edge IoT reference architectures on which customers can build AI and analytics applications. Achieving a balance between cloud and edge AI will be a fundamental requirement, he said.
Finding a balance begins by considering the amount of data needed to run a machine learning model, according to Frédéric Desbiens, program manager, IoT and Edge Computing of the Eclipse Foundation. Here come the new intelligent processors.
“AI accelerators on the edge can perform local processing before sending data elsewhere. But this requires you to consider the functional requirements, including the software package and the storage required, ”Desbiens said.
Abundance of AI chip edges
The rise of cloud-based machine learning has been driven by the rise of high-bandwidth GPUs, often in the form of NVIDIA semiconductors. That success has caught the attention of other chipmakers.
Own intelligence-specific processors are tracked by hyper-scales of the cloud, Google, AWS, and Microsoft.
That battle with the AI chip was joined by leading lights such as AMD, Intel, Qualcomm and ARM Technology (which in turn was acquired by NVIDIA last year).
In turn, embedded microprocessors and on-chip systems, such as Maxim Integrated, NXP Semiconductors, Silicon Labs, STM Microelectronics, and others, began to focus on adding intelligence capabilities.
Today, the need for IoT and edge processing has attracted the launch of AI chips that include EdgeQ, Graphcore, Hailo, Mythic and others. Edge processing is limited. Obstacles include available memory, energy consumption and costs, points out Steve Conway of Hyperion.
“Built-in processors are very important because energy use is very important,” Conway said. “GPUs and CPUs are not tiny matrices, and GPUs in particular use a ton of power,” he said, referring to the relatively large silicon form factors that GPUs and CPUs can take.
Data movement is a factor in energy consumption at the edge, advises Kris Ardis, CEO of Maxim Integrated for microcontrollers and software algorithms. The company recently released the MAX78000, which connects a low-power controller with a neural network processor that can run on battery-powered IoT devices.
“If you can do the budget at the very edge, you’ll save bandwidth and communication power. The challenge is to take the neural network and adapt it to the part, ”Ardis said.
Individual chip-based IoT devices can power IoT gateways, which also play a useful role, combining data sets from devices and further filtering data that can go into the cloud to analyze overall operations, he said.
Other semiconductor device manufacturers are also adapting to the trend that shows that computing is approaching where the data is. They are part of an effort to expand the capabilities of developers, even as their hardware choices grow.
Bill Pearson, vice president of Intel’s IoT group, admits that there was a time when “the CPU was the answer to all problems.” Trends like the edge of AI believe in it now.
The term “XPU” is used to represent different types of chips that support different purposes. But, he adds, the variety should be supported by a single application programming interface (API).
To help software developers, Intel recently released Version 2021.2 of the OpenVINO Edge Systems Inference Toolkit. It provides a common development environment among Intel components, including CPUs, GPUs, and Movidius visual processing units. Also, Intel is offering DevCloud for Edge software to predict the performance of neural network conclusions on a variety of Intel hardware, according to Pearson.
The drive for simplification is also marked on the GPU of NVIDIA.
“The industry needs to make it easier for people who aren’t intelligence experts,” said Justin Boitano, vice president and general manager for Enterprise and Edge Computing, NVIDIA.
It can take the form of NVIDIA Jetson, which includes a low-power ARM processor. Named the head of a 60s sci-fi cartoon series, Jetson is designed to provide GPU-accelerated parallel processing in mobile embedded systems.
Recently, to facilitate the development of the vision system, NVIDIA released the Jetson JetPack 4.5, which includes the first production version of its Vision Programming Interface (VPI).
Over time, AI development tasks will focus more on IT stores and less on AI researchers with deep knowledge of machine learning, Boitano said.
A tiny ML roaring
The skills required to migrate machine learning methods from a huge cloud to a limited edge device are not easy to acquire. But new software techniques are being applied that enable compact AI while making the task of the programmer easier.
In fact, the industry has seen an increase in access to “Tiny ML”. They are satisfied with less power and use limited memory, while at the same time achieving capable ratings in inference per second.
Various machine learning tools have emerged to reduce edge processing requirements, including Apache MXNet, Edge Impulse EON, Facebook Glow, Foghorn Lightning Edge ML, Google TensorFlow Lite, Microsoft ELL, OctoML’s Octomizer and others.
The main goal is to process the shrinking neural network, and the technique is several. Among them are quantization, binization, and cropping, according to Sastry Malladi, chief technical officer at Foghorn, a software platform maker that supports a variety of edge and local applications.
The quantization of neural network processing focuses on the use of low bit width mathematics. Binaryization, in turn, is used to reduce budget complexity. Circumcision is also used to reduce the number of neural nodes that need to be processed.
Malladi admits that this is a frightening range for most developers – especially through the entire hardware. The efforts behind Foghorn’s Lightning platform, he said, aim to abstract the complexity of machine learning on the edge.
The goal, for example, is to allow line operators and reliability engineers to work with drag-and-drop interfaces, rather than program programming interfaces and development software packages, which are less intuitive and require more coding knowledge.
Software that simplifies development and runs multiple types of state-of-the-art AI hardware is also the focus of Edge Impulse, the manufacturer of the embedded machine learning development platform.
In the end, the maturation of machine learning means a certain miniaturization of the model, according to Zach Shelby, CEO of Edge Impulse.
“Once the direction of research was geared towards ever-increasing models of increasingly complex,” Shelby said. “But as machine learning hit its peak, people began to worry about efficiency again.” This led to Tiny ML.
Software is needed that can run on existing IoT infrastructure while supporting the path to new types of hardware, he said. Edge Impulse tools allow you to model algorithms and events on available cloud hardware, Shelby continued, so users can try out different options before making a selection.
Keep an eye on
Eventually, computer vision has become a prominent use case of AI, especially in the form of deep learning, which uses multiple layers of neural networks and unsupervised techniques to achieve results in image pattern recognition.
The architecture of the Vision system is undergoing changes today as edge cameras add processing capabilities through embedded deep learning hardware, according to Kjell Carlsson, chief analyst at Forrester Research. But finding the best goals for the app can be a challenge.
“The problem with AI on the edge is that as often as possible you end up seeing usage cases that are ‘net new,'” he said.
The development of these greenfield solutions has an inherent risk, Carlsson said, so it is a useful tactic to focus on use cases that offer a high benefit-cost ratio, even though pattern recognition accuracy can follow that of full-fledged existing systems.
Overall, Carlsson said the edge of AI could help deliver on the original IoT promise, which sometimes fell behind when implementers sorted out countless potential uses.
“The IoT itself had certain limitations. “Now with AI, machine learning and deep learning make IoT more applicable – and valuable,” he said.