Videogenic: Video Highlight Generation via Photogenic Moments
Abstract
This paper investigates the challenge of extracting highlight moments from videos. To perform this task, a system needs to understand what constitutes a highlight for arbitrary video domains while at the same time being able to scale across different domains. Our key insight is that photographs taken by photographers tend to capture the most remarkable or photogenic moments of an activity. Drawing on this insight, we present Videogenic, a system capable of creating domain-specific highlight videos for a diverse range of domains. In a human evaluation study (N=50), we show that a high-quality photograph collection combined with CLIP-based retrieval (which uses a neural network with semantic knowledge of images) can serve as an excellent prior for finding video highlights. In a within-subjects expert study (N=12), we demonstrate the usefulness of Videogenic in helping video editors create highlight videos with lighter workload, shorter task completion time, and better usability.
Example Results
Videogenic identifies the most highlight-worthy moments of an activity or event. Examples include the officiant address of the wedding, the cars drifting, the skateboard kickflip, the graduation hat toss, the breakdance power move, the bird carrying its prey, and the weightlifter completing the clean and jerk.
Video Sources
- https://www.youtube.com/watch?v=kOSbdmOlKUk
- https://www.youtube.com/watch?v=_hTesCkABtM
- https://www.youtube.com/watch?v=ynlZXzQrCig
- https://www.youtube.com/watch?v=9cqd6OMuicg
- https://www.youtube.com/watch?v=PznYLQ1wf1Q
- https://www.youtube.com/watch?v=JOfk4_o6Au4
- https://www.youtube.com/watch?v=EiqEFUFM-KI
- https://www.youtube.com/watch?v=gpZxiJKHB9
- https://www.youtube.com/watch?v=TWgyMCVenrE
- https://www.youtube.com/watch?v=Ets4wohr0Z4
- https://www.youtube.com/watch?v=HM5w7Aq8buw
- https://www.youtube.com/watch?v=jCpqNoumNIc
- https://www.youtube.com/watch?v=zxrxzvA2sQU
- https://www.youtube.com/watch?v=TTwT1-TpFhE
- https://www.youtube.com/watch?v=IA-ZEDJAbfE
- https://www.youtube.com/watch?v=tw-xgdupCOY
Highlight Graph
The highlight graph visualizes the distribution of predicted highlight scores across the video (a). The user may scrub through the graph to inspect a corresponding video frame and its highlight score (b).
The user may brush through the highlight graph to select an interval of the video to use for the highlight video (a). The interface displays a dashed line and a text label to indicate the average highlight value of the selected interval (b).
Example video frames and their corresponding highlight scores within a long skydiving video, using the keyword skydiving. The top-left corner displays the photograph collection used by Videogenic.
More Example Highlight Graphs
wedding
fireworks
breakdance
rafting
