New AI Sensor ‘Sniffs’ Out Spectral Targets

Meet the sniffer dog of spectroscopy tools: an AI-enhanced sensor that can “sniff and seek” target objects in real-time.

Spectral imaging tools — cameras that capture colors beyond the RGB spectrum visible to our eyes — are vital for gleaning information about an object’s material and structural properties. Marrying them with machine learning has provided a powerful pipeline for identifying features in real-world applications including semiconductor fabrication, pollutant tracking, and crop monitoring. By folding AI algorithms into the camera’s sensor itself, researchers at the Department of Energy’s Lawrence Berkeley National Laboratory (Berkeley Lab) have now eliminated a data-processing bottleneck that has long plagued the performance of spectral imaging technology. The result is an intelligent sensor capable of identifying chemicals and characterizing materials quickly and efficiently.

“We focused on enhancing the speed, resolution, and power efficiency of existing spectral machine vision technologies by more than two orders of magnitude,” said Ali Javey, the scientist who led the Science study reporting the device. Javey is a senior faculty scientist at Berkeley Lab and a professor of materials science and engineering at UC Berkeley. The work was performed in close collaboration with Aydogan Ozcan at UCLA.

“We focused on enhancing the speed, resolution, and power efficiency of existing spectral machine vision technologies by more than two orders of magnitude.”
– Ali Javey

The sensor design illustrates how novel functionality can be built into semiconductor devices themselves to improve their efficiency and utility, and enable a new class of AI vision hardware.

Building algorithms with light

Today’s spectral imaging technologies have separate sensor and computational modules. The sensor first captures a stack of images, each of which corresponds to a certain color. Then the dense image stack gets sent to a digital processor for further computation, which produces the object-identification results. That’s where the problems arise.

“The sensors must collect and send much more data to the digital processor than normal cameras, roughly ten- to hundred-times larger in volume,” said Dehui Zhang, a postdoc in Berkeley Lab’s Materials Sciences Division and the lead author on the study. Consequently, the sensor and computer hardware are often overwhelmed, making object-recognition tasks extremely slow and power-hungry.

Instead, the Berkeley Lab team developed sensors that perform AI computation and spectral analysis during the image capturing — or photodetection — process itself.

“Photodetection can be perceived as an automatic physical computational process,” explained Zhang. When light hits the sensor, its intensity automatically gets mapped to the strength of an electrical current. Because the sensor’s responsivity to light can easily be adjusted, the researchers have a tuning knob for selecting which spectral signatures get highlighted and which get suppressed. The current that departs the sensor to be read by a circuit, therefore, serves as an inference about the image’s spectral content.

“We proved that the computational process mathematically resembles an algorithm typically used for digital machine learning,” said Zhang. This analogy made it possible to use the sensor as a machine learning computer and perform the machine learning computations on the incoming light itself.

Training the machine

Any AI or machine vision model first needs to learn what it’s supposed to identify. That means “showing” it enough examples of the spectral signatures of interest — say, the infrared patterns that come from a real leaf versus an artificial one; or the pixels in an image that belong to a bird’s plumage versus a tree’s similarly colored bark — that it can find these signatures in an untrialed test case.

In the training step, the researchers showed the sensor dozens of images of colorful birds in wooded settings. Rather than examining every pixel of each image, the sensor “sniffed” a random selection of pixels, each of which was labeled as belonging either to the bird or to the unwanted background. An external computer sent an electrical signal to the sensor commanding it to “identify bird” or “identify background,” and recorded the sensor’s output for each command. Software then determined the best command combination for teaching the sensor to highlight the bird region while suppressing everything else.

In the test step, they showed the sensor a new image and told it to find a bird, using the command combination developed during training. The sensor gave positive output signals only for pixels that belonged to the bird. This result meant the sensor had learned from the examples to identify target objects, even when they belonged to an image it had never seen before.

“For me, the most exciting part is the concept of giving intelligence to sensors,” said Javey. Normal sensors simply collect raw environmental information, leaving the intelligent recognition tasks to digital processors.

By co-designing the semiconductor materials, devices and the algorithms, the team enabled the sensors to learn and compute without the need for digital post-processing of data.

But applications for the technology go way beyond identifying birds. Using photodiodes of black phosphorus (capable of detecting mid-infrared light with tunable responsivity), the researchers experimentally demonstrated several other intriguing possibilities. They successfully identified oxide layer thicknesses in semiconductor samples — which manufacturing giants need to be perfectly uniform — as well as hydration states in different plant leaves, object segmentation in optical images, and transparent chemicals in a petri dish.

“I’m also optimistic about the future of such devices for broader applications,” Javey said. In the future, the smart sensors could find a home not only in spectral machine vision but in “other advanced optical sensing and beyond.”

The work was funded by the US Department of Energy’s Office of Basic Energy Sciences. It received support from the DOE’s Microelectronics Energy Efficiency Research Center for Advanced Technologies, one of the DOE’s three Microelectronics Science Research Centers.

For information about licensing this technology, contact UC Berkeley.

###

Lawrence Berkeley National Laboratory (Berkeley Lab) is committed to groundbreaking research focused on discovery science and solutions for abundant and reliable energy supplies. The lab’s expertise spans materials, chemistry, physics, biology, earth and environmental science, mathematics, and computing. Researchers from around the world rely on the lab’s world-class scientific facilities for their own pioneering research. Founded in 1931 on the belief that the biggest problems are best addressed by teams, Berkeley Lab and its scientists have been recognized with 17 Nobel Prizes. Berkeley Lab is a multiprogram national laboratory managed by the University of California for the U.S. Department of Energy’s Office of Science.

DOE’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit energy.gov/science.