Motion-based video retargeting with optimized crop-and-warp

Yu Shuen Wang, Hui Chih Lin, Olga Sorkine, Tong Yee Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

We introduce a video retargeting method that achieves high-quality resizing to arbitrary aspect ratios for complex videos containing diverse camera and dynamic motions. Previous content-aware retargeting methods mostly concentrated on spatial considerations, attempting to preserve the shape of salient objects in each frame by removing or distorting homogeneous background content. However, sacrificeable space is fundamentally limited in video, since object motion makes foreground and background regions correlated, causing waving and squeezing artifacts. We solve the retargeting problem by explicitly employing motion information and by distributing distortion in both spatial and temporal dimensions. We combine novel cropping and warping operators, where the cropping removes temporally-recurring contents and the warping utilizes available homogeneous regions to mask deformations while preserving motion. Variational optimization allows to find the best balance between the two operations, enabling retargeting of challenging videos with complex motions, numerous prominent objects and arbitrary depth variability. Our method compares favorably with state-of-the-art retargeting systems, as demonstrated in the examples and widely supported by the conducted user study.

Original languageEnglish
Title of host publicationACM SIGGRAPH 2010 Papers, SIGGRAPH 2010
EditorsHugues Hoppe
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450302104
DOIs
Publication statusPublished - 2010 Jul 26
Event37th International Conference and Exhibition on Computer Graphics and Interactive Techniques, SIGGRAPH 2010 - Los Angeles, United States
Duration: 2010 Jul 262010 Jul 30

Publication series

NameACM SIGGRAPH 2010 Papers, SIGGRAPH 2010

Other

Other37th International Conference and Exhibition on Computer Graphics and Interactive Techniques, SIGGRAPH 2010
CountryUnited States
CityLos Angeles
Period10-07-2610-07-30

Fingerprint

Crops
Aspect ratio
Masks
Cameras

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design
  • Computer Vision and Pattern Recognition
  • Software

Cite this

Wang, Y. S., Lin, H. C., Sorkine, O., & Lee, T. Y. (2010). Motion-based video retargeting with optimized crop-and-warp. In H. Hoppe (Ed.), ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010 [90] (ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010). Association for Computing Machinery, Inc. https://doi.org/10.1145/1778765.1778827
Wang, Yu Shuen ; Lin, Hui Chih ; Sorkine, Olga ; Lee, Tong Yee. / Motion-based video retargeting with optimized crop-and-warp. ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010. editor / Hugues Hoppe. Association for Computing Machinery, Inc, 2010. (ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010).
@inproceedings{f88550d6c57d45498922f1cf3e44dcb7,
title = "Motion-based video retargeting with optimized crop-and-warp",
abstract = "We introduce a video retargeting method that achieves high-quality resizing to arbitrary aspect ratios for complex videos containing diverse camera and dynamic motions. Previous content-aware retargeting methods mostly concentrated on spatial considerations, attempting to preserve the shape of salient objects in each frame by removing or distorting homogeneous background content. However, sacrificeable space is fundamentally limited in video, since object motion makes foreground and background regions correlated, causing waving and squeezing artifacts. We solve the retargeting problem by explicitly employing motion information and by distributing distortion in both spatial and temporal dimensions. We combine novel cropping and warping operators, where the cropping removes temporally-recurring contents and the warping utilizes available homogeneous regions to mask deformations while preserving motion. Variational optimization allows to find the best balance between the two operations, enabling retargeting of challenging videos with complex motions, numerous prominent objects and arbitrary depth variability. Our method compares favorably with state-of-the-art retargeting systems, as demonstrated in the examples and widely supported by the conducted user study.",
author = "Wang, {Yu Shuen} and Lin, {Hui Chih} and Olga Sorkine and Lee, {Tong Yee}",
year = "2010",
month = "7",
day = "26",
doi = "10.1145/1778765.1778827",
language = "English",
series = "ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010",
publisher = "Association for Computing Machinery, Inc",
editor = "Hugues Hoppe",
booktitle = "ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010",

}

Wang, YS, Lin, HC, Sorkine, O & Lee, TY 2010, Motion-based video retargeting with optimized crop-and-warp. in H Hoppe (ed.), ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010., 90, ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010, Association for Computing Machinery, Inc, 37th International Conference and Exhibition on Computer Graphics and Interactive Techniques, SIGGRAPH 2010, Los Angeles, United States, 10-07-26. https://doi.org/10.1145/1778765.1778827

Motion-based video retargeting with optimized crop-and-warp. / Wang, Yu Shuen; Lin, Hui Chih; Sorkine, Olga; Lee, Tong Yee.

ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010. ed. / Hugues Hoppe. Association for Computing Machinery, Inc, 2010. 90 (ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Motion-based video retargeting with optimized crop-and-warp

AU - Wang, Yu Shuen

AU - Lin, Hui Chih

AU - Sorkine, Olga

AU - Lee, Tong Yee

PY - 2010/7/26

Y1 - 2010/7/26

N2 - We introduce a video retargeting method that achieves high-quality resizing to arbitrary aspect ratios for complex videos containing diverse camera and dynamic motions. Previous content-aware retargeting methods mostly concentrated on spatial considerations, attempting to preserve the shape of salient objects in each frame by removing or distorting homogeneous background content. However, sacrificeable space is fundamentally limited in video, since object motion makes foreground and background regions correlated, causing waving and squeezing artifacts. We solve the retargeting problem by explicitly employing motion information and by distributing distortion in both spatial and temporal dimensions. We combine novel cropping and warping operators, where the cropping removes temporally-recurring contents and the warping utilizes available homogeneous regions to mask deformations while preserving motion. Variational optimization allows to find the best balance between the two operations, enabling retargeting of challenging videos with complex motions, numerous prominent objects and arbitrary depth variability. Our method compares favorably with state-of-the-art retargeting systems, as demonstrated in the examples and widely supported by the conducted user study.

AB - We introduce a video retargeting method that achieves high-quality resizing to arbitrary aspect ratios for complex videos containing diverse camera and dynamic motions. Previous content-aware retargeting methods mostly concentrated on spatial considerations, attempting to preserve the shape of salient objects in each frame by removing or distorting homogeneous background content. However, sacrificeable space is fundamentally limited in video, since object motion makes foreground and background regions correlated, causing waving and squeezing artifacts. We solve the retargeting problem by explicitly employing motion information and by distributing distortion in both spatial and temporal dimensions. We combine novel cropping and warping operators, where the cropping removes temporally-recurring contents and the warping utilizes available homogeneous regions to mask deformations while preserving motion. Variational optimization allows to find the best balance between the two operations, enabling retargeting of challenging videos with complex motions, numerous prominent objects and arbitrary depth variability. Our method compares favorably with state-of-the-art retargeting systems, as demonstrated in the examples and widely supported by the conducted user study.

UR - http://www.scopus.com/inward/record.url?scp=85026926010&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85026926010&partnerID=8YFLogxK

U2 - 10.1145/1778765.1778827

DO - 10.1145/1778765.1778827

M3 - Conference contribution

AN - SCOPUS:85026926010

T3 - ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010

BT - ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010

A2 - Hoppe, Hugues

PB - Association for Computing Machinery, Inc

ER -

Wang YS, Lin HC, Sorkine O, Lee TY. Motion-based video retargeting with optimized crop-and-warp. In Hoppe H, editor, ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010. Association for Computing Machinery, Inc. 2010. 90. (ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010). https://doi.org/10.1145/1778765.1778827