By Rachel Robey
Professor Mohit Gupta’s work on single photon imaging gains unprecedented traction as industry interest rises.
It’s a tale as old as start-ups: The world’s greatest inventions always begin in a garage.
Such was the case for Computer Sciences (CS) Professor Mohit Gupta, whose research in single-photon cameras (SPCs) and single photon avalanche diodes (SPADs) had beginnings in his garage along with his lab. Now a career-defining offshoot of his research in computer vision, Gupta’s exploration of these devices started purely as curiosity.
It’s easy to see why: Single photon imaging devices are capable of detecting light down to individual photons. “As academics, we’re often concerned with what happens when we push something to its limit,” says Gupta. “For imaging, that’s detecting individual photons. There’s nothing beyond that.”
Initial tests were rudimentary. A single-photon camera imaged Gupta slowly cycling around his unlit garage to test the camera’s capabilities in low light and high movement conditions. During a particularly memorable late-night experiment, concerned neighbors who were unfamiliar with Gupta’s work almost called the police on him as he drove through the neighborhood with the research camera strapped to the dashboard of his car; another time, a police car actually flashed lights and stopped Gupta and researchers for questioning while performing late night experiments. The results of these early experiments — which amounted to just a few pixels — were nevertheless promising.
At WISION Lab, single-photon cameras “see the world in a new light”
In the years since, Gupta has continued research on single-photon imaging in the WISION Lab, the computational imaging and computer vision group he directs within the Department of Computer Sciences at the University of Wisconsin–Madison. His work — which has garnered awards from the National Science Foundation (NSF), the Institute of Electrical and Electronics Engineers (IEEE), and the European Conference on Computer Vision (ECCV), among others — reimagines a camera all the way down to its core algorithms and the kinds of pictures it captures.
Unlike typical consumer cameras that perform best in bright conditions, ultra sensitive single-photon cameras are able to detect light down to its most fundamental unit: the photon. Not only does this make SPCs more flexible and capable across a spectrum of light conditions, but it allows researchers like Gupta to pursue answers right up against the foundational laws of physics.
“These cameras will allow us to capture the visual world around us to the maximum fidelity that physics allows,” Gupta continues. “If you’re able to consider every photon, then that’s virtually all the information there is.” In a sense, the images SPCs create are the truest possible representations of reality.
Swiss Federal Institute of Technology Lausanne (EPFL).
Naturally, the possibilities for this kind of technology are endless. With their impressive dynamic range and suitability for suboptimal visual conditions, SPCs are a plausible solution for situations ranging from extreme robotics (for example, a robot exploring an underground cave) to fluorescence lifetime microscopy (an imaging technique used in cancer research) or even in virtual reality (VR) headset sensors.
At Ubicept, a Boston-based start-up exploring SPCs where Gupta serves as a founding advisor, development of a “new kind of camera that sees in the dark (and around corners)” is especially promising for self-driving vehicles. Over at the Computational Optics Group within the Department of Biostatistics and Medical Informatics, Professor Andreas Velten, a close collaborator of Gupta’s and likewise a founding advisor for Ubicept, researches the biomedical imaging and remote sensing applications of this technology. Meanwhile, Canon’s recent release — a $25,000 SPAD device — is intended for high security and surveillance applications.
But before any of these possibilities are realized, more research and development are needed. With industry backing, it’s happening faster than ever.
With industry support, innovation at the speed of light
Years ago when Sony, an industry leader in photosensors, unveiled their Faculty Innovation Award supporting “pioneering research,” Gupta decided to submit his work. “I sent a proposal, and it landed in the right place with the right people,” says Gupta, who won the award in 2020 and received additional funding in 2021. “It started off as a one year thing, but now [the collaboration with Sony] is in its fourth year.”
Around the same time that Gupta first submitted his proposal, big names including Sony, Canon, and Apple all began to invest heavily in the nascent single photon imaging technology. A veritable space race ensued, causing research in the area to explode. Seven years ago, Gupta’s lab had access to just one pixel that cost more than $10,000. Today, SPAD technology is already a consumer device that’s been tucked inside the latest generations of iPhone.
“That level and speed of progress is almost unheard of with a technology like this,” says Gupta, who uses research cameras with nearly a quarter megapixels.
Given the technical difficulties presented by SPCs, government investment and industry support will be necessary to aid the research required to fully develop the technology. “One major practical hurdle for these kinds of sensors is they consider every individual photon that comes in,” says Gupta. “In a second you may see trillions of photons, so for these things to ever be practical, we need ways to deal with this data deluge.” For Gupta, this means going back to first principles to rethink algorithms related to computer vision, machine learning, and image processing.
Daunting as it may be, Gupta feels grateful to be able to work on this. Especially among busy junior faculty, the tenure track doesn’t typically incentivize researchers to pursue intellectual property rights or commercialize their inventions. Yet here at UW–Madison this academia-wide trend is shifting, with divisions like the Wisconsin Alumni Research Foundation (WARF), the Technology Entrepreneurship Office (TEO), the Creative Destruction Lab (CDL), Discovery to Product (D2P), and the School of Computer, Data, and Information Sciences (CDIS) pushing faculty to think more about entrepreneurship.
“Here at UW, we’re fortunate to have partners like WARF. They really push people,” says Gupta, who has worked with WARF on securing nearly 25 patents since joining the CS faculty in 2016. “For me, the most encouraging thing is that a large fraction of [our patents] aren’t just gathering dust – they’ve actually been licensed for practical use by companies.”
Yet while the licensing, patent granting, and industry investing are welcome encouragement, Gupta’s core motivation remains the same as ever: to always follow curiosity and explore novel ways of “seeing the world in a new light.”