Scalable and coherent video resizing with per-frame optimization

Yu Shuen Wang, Jen Hung Hsiao, Olga Sorkine, Tong Yee Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

40 Citations (Scopus)

Abstract

The key to high-quality video resizing is preserving the shape and motion of visually salient objects while remaining temporallycoherent. These spatial and temporal requirements are difficult to reconcile, typically leading existing video retargeting methods to sacrifice one of them and causing distortion or waving artifacts. Recent work enforces temporal coherence of content-aware video warping by solving a global optimization problem over the entire video cube. This significantly improves the results but does not scale well with the resolution and length of the input video and quickly becomes intractable. We propose a new method that solves the scalability problem without compromising the resizing quality. Our method factors the problem into spatial and time/motion components: we first resize each frame independently to preserve the shape of salient regions, and then we optimize their motion using a reduced model for each pathline of the optical flow. This factorization decomposes the optimization of the video cube into sets of subproblems whose size is proportional to a single frame's resolution and which can be solved in parallel. We also show how to incorporate cropping into our optimization, which is useful for scenes with numerous salient objects where warping alone would degenerate to linear scaling. Our results match the quality of state-of-the-art retargeting methods while dramatically reducing the computation time and memory consumption, making content-aware video resizing scalable and practical.

Original languageEnglish
Title of host publicationProceedings of ACM SIGGRAPH 2011, SIGGRAPH 2011
Volume30
Edition4
DOIs
Publication statusPublished - 2011 Jul 1
EventACM SIGGRAPH 2011, SIGGRAPH 2011 - Vancouver, BC, Canada
Duration: 2011 Aug 72011 Aug 11

Other

OtherACM SIGGRAPH 2011, SIGGRAPH 2011
CountryCanada
CityVancouver, BC
Period11-08-0711-08-11

Fingerprint

Optical flows
Global optimization
Factorization
Scalability
Data storage equipment

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design

Cite this

Wang, Y. S., Hsiao, J. H., Sorkine, O., & Lee, T. Y. (2011). Scalable and coherent video resizing with per-frame optimization. In Proceedings of ACM SIGGRAPH 2011, SIGGRAPH 2011 (4 ed., Vol. 30). [88] https://doi.org/10.1145/1964921.1964983
Wang, Yu Shuen ; Hsiao, Jen Hung ; Sorkine, Olga ; Lee, Tong Yee. / Scalable and coherent video resizing with per-frame optimization. Proceedings of ACM SIGGRAPH 2011, SIGGRAPH 2011. Vol. 30 4. ed. 2011.
@inproceedings{ef04bd1ae12c4ed7bd39f964bb98dcdf,
title = "Scalable and coherent video resizing with per-frame optimization",
abstract = "The key to high-quality video resizing is preserving the shape and motion of visually salient objects while remaining temporallycoherent. These spatial and temporal requirements are difficult to reconcile, typically leading existing video retargeting methods to sacrifice one of them and causing distortion or waving artifacts. Recent work enforces temporal coherence of content-aware video warping by solving a global optimization problem over the entire video cube. This significantly improves the results but does not scale well with the resolution and length of the input video and quickly becomes intractable. We propose a new method that solves the scalability problem without compromising the resizing quality. Our method factors the problem into spatial and time/motion components: we first resize each frame independently to preserve the shape of salient regions, and then we optimize their motion using a reduced model for each pathline of the optical flow. This factorization decomposes the optimization of the video cube into sets of subproblems whose size is proportional to a single frame's resolution and which can be solved in parallel. We also show how to incorporate cropping into our optimization, which is useful for scenes with numerous salient objects where warping alone would degenerate to linear scaling. Our results match the quality of state-of-the-art retargeting methods while dramatically reducing the computation time and memory consumption, making content-aware video resizing scalable and practical.",
author = "Wang, {Yu Shuen} and Hsiao, {Jen Hung} and Olga Sorkine and Lee, {Tong Yee}",
year = "2011",
month = "7",
day = "1",
doi = "10.1145/1964921.1964983",
language = "English",
isbn = "9781450309431",
volume = "30",
booktitle = "Proceedings of ACM SIGGRAPH 2011, SIGGRAPH 2011",
edition = "4",

}

Wang, YS, Hsiao, JH, Sorkine, O & Lee, TY 2011, Scalable and coherent video resizing with per-frame optimization. in Proceedings of ACM SIGGRAPH 2011, SIGGRAPH 2011. 4 edn, vol. 30, 88, ACM SIGGRAPH 2011, SIGGRAPH 2011, Vancouver, BC, Canada, 11-08-07. https://doi.org/10.1145/1964921.1964983

Scalable and coherent video resizing with per-frame optimization. / Wang, Yu Shuen; Hsiao, Jen Hung; Sorkine, Olga; Lee, Tong Yee.

Proceedings of ACM SIGGRAPH 2011, SIGGRAPH 2011. Vol. 30 4. ed. 2011. 88.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Scalable and coherent video resizing with per-frame optimization

AU - Wang, Yu Shuen

AU - Hsiao, Jen Hung

AU - Sorkine, Olga

AU - Lee, Tong Yee

PY - 2011/7/1

Y1 - 2011/7/1

N2 - The key to high-quality video resizing is preserving the shape and motion of visually salient objects while remaining temporallycoherent. These spatial and temporal requirements are difficult to reconcile, typically leading existing video retargeting methods to sacrifice one of them and causing distortion or waving artifacts. Recent work enforces temporal coherence of content-aware video warping by solving a global optimization problem over the entire video cube. This significantly improves the results but does not scale well with the resolution and length of the input video and quickly becomes intractable. We propose a new method that solves the scalability problem without compromising the resizing quality. Our method factors the problem into spatial and time/motion components: we first resize each frame independently to preserve the shape of salient regions, and then we optimize their motion using a reduced model for each pathline of the optical flow. This factorization decomposes the optimization of the video cube into sets of subproblems whose size is proportional to a single frame's resolution and which can be solved in parallel. We also show how to incorporate cropping into our optimization, which is useful for scenes with numerous salient objects where warping alone would degenerate to linear scaling. Our results match the quality of state-of-the-art retargeting methods while dramatically reducing the computation time and memory consumption, making content-aware video resizing scalable and practical.

AB - The key to high-quality video resizing is preserving the shape and motion of visually salient objects while remaining temporallycoherent. These spatial and temporal requirements are difficult to reconcile, typically leading existing video retargeting methods to sacrifice one of them and causing distortion or waving artifacts. Recent work enforces temporal coherence of content-aware video warping by solving a global optimization problem over the entire video cube. This significantly improves the results but does not scale well with the resolution and length of the input video and quickly becomes intractable. We propose a new method that solves the scalability problem without compromising the resizing quality. Our method factors the problem into spatial and time/motion components: we first resize each frame independently to preserve the shape of salient regions, and then we optimize their motion using a reduced model for each pathline of the optical flow. This factorization decomposes the optimization of the video cube into sets of subproblems whose size is proportional to a single frame's resolution and which can be solved in parallel. We also show how to incorporate cropping into our optimization, which is useful for scenes with numerous salient objects where warping alone would degenerate to linear scaling. Our results match the quality of state-of-the-art retargeting methods while dramatically reducing the computation time and memory consumption, making content-aware video resizing scalable and practical.

UR - http://www.scopus.com/inward/record.url?scp=80051908713&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80051908713&partnerID=8YFLogxK

U2 - 10.1145/1964921.1964983

DO - 10.1145/1964921.1964983

M3 - Conference contribution

AN - SCOPUS:80051908713

SN - 9781450309431

VL - 30

BT - Proceedings of ACM SIGGRAPH 2011, SIGGRAPH 2011

ER -

Wang YS, Hsiao JH, Sorkine O, Lee TY. Scalable and coherent video resizing with per-frame optimization. In Proceedings of ACM SIGGRAPH 2011, SIGGRAPH 2011. 4 ed. Vol. 30. 2011. 88 https://doi.org/10.1145/1964921.1964983