Twitter’s Cortex AI System Recognizes What’s Happening in Live Video Clips


Right now, someone somewhere is live-streaming something interesting. Thanks to technology being developed by a team of artificial intelligence researchers at Twitter, you may soon be able to find it.

Live-streaming is becoming ever-more popular through smartphone apps such as Periscope from Twitter, Meerkat, and, most recently, Facebook Live. But live video content usually isn’t tagged or categorized well, often because people don’t know what they’ll record until the camera begins rolling.

Twitter’s AI team, known as Cortex, has developed an algorithm that can instantly recognize what’s happening in a live feed. The algorithm can tell, for instance, if the star of a clip is playing guitar, demoing a power tool, or is actually a cat hamming it up for viewers.

“Content is always changing on Periscope, and more generally on live videos,” says Clement Farabet, who is the technology lead for Cortex. Farabet demonstrated the video-recognition technology to MIT Technology Review, showing a screen of about two dozen Periscope feeds, all being tagged in real-time.

Identifying the content of live video is a pretty impressive trick. Researchers have made impressive progress in recent years with algorithms that can identify objects in photographs, but it is much more difficult to do with a live video of varying quality. To do it instantly also requires considerable computing power. Twitter effectively built a custom supercomputer made entirely of graphics processing units (GPUs) to perform the video classification and serve up the results. These chips are especially efficient for the mathematical calculations required for deep learning, but normally they are just one part of a larger computer system.

Read the source article at MIT Technology Review