Jaskolski dug in, poring over wind models and social media sightings to feed his software, called RAIC (rapid automatic image categorization), new swathes of satellite data from the company Planet Labs. The tool is designed to make it possible to search large image collections for objects of interest using a single example image. “We drew a big arc across time and space and started searching that,” Jaskolski says. Having found the balloon once, Synthetiatic’s software could be trained with a real image of the balloon to further guide its search. Over the next several days, Jaskolski put RAIC to work. The company has since compiled six sightings of the balloon (five confirmed, one still being investigated) on its satellite imagery and has used wind data to estimate how it moved between those points. “We can draw a 1-kilometer-wide track across the whole of the United States and just follow the balloon,” he says. “We have a track from where it entered from Canada, all the way to South Carolina, where it got popped, with six points along that arc.” Jaskolski’s stratospheric scavenger hunt may have been made possible by smart software, but it also required human expert knowledge. His initial drawing of the craft looked more like a technicolor snowman—stacked red, green, and blue circles. The aim was to mimic the way satellites generally capture different wavelengths of light using separate sensors that aren’t always synced in time, creating multiple disjointed views of objects. And it throws up false positives. But the ability to map a surveillance balloon’s path with such clarity could be a game changer for national security, says Arthur Holland Michel, senior fellow at the Carnegie Council and author of a book on drones and surveillance. “The combination of AI with satellite imagery is undoubtedly a very powerful technology for surveillance and espionage and counterespionage,” he says. Holland Michel also points out that satellite imagery and AI have their limitations. The method by which Synthetatic first found the balloon—using a drawing—could result in false positives if the object of interest was something more complex or less publicly documented, such as a tank. “Things often look a bit weird and unfamiliar from above,” he says. “There’s undoubted potential there,” Holland Michel says, “but it’s easy to think this combination of satellites and AI is an all-seeing capability that will lay everything bare.” It’s useful in certain cases, like the balloon, he says, but likely not all scenarios. That’s something Jaskolski acknowledges—but he also considers the project an example of how human expertise and grunt work can be elevated by AI. “This human-machine collaboration is my idea of how AI works today,” he says. “And it’s definitely how we build our product.” The tool is currently used for humanitarian purposes, including by the UN World Food Program to find flood victims. The pursuit of the balloon isn’t over just because Jaskolski has managed to track it across the United States. He says the process is “resource-intensive” because the software isn’t perfect and turns up many potential sightings that have to be whittled down by people. “But we’d like to still continue to track it,” he says. “Whether we go all the way back to China or not, we feel like we solved a technical problem at least. We’d be crazy not to try.”
