Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge: Report

Andrey Ignatov; Cheng Ming Chiang; Hsien Kai Kuo; Anastasia Sycheva; Radu Timofte; Min Hung Chen; Man Yu Lee; Yu Syuan Xu; Yu Tseng; Shusong Xu; Jin Guo; Chao Hung Chen; Ming Chun Hsyu; Wen Chia Tsai; Chao Wei Chen; Grigory Malivenko; Minsu Kwon; Myungje Lee; Jaeyoon Yoo; Changbeom Kang; Shinjo Wang; Zheng Shaolong; Hao Dejun; Xie Fen; Feng Zhuang; Yipeng Ma; Jingyang Peng; Tao Wang; Fenglong Song; Chih Chung Hsu; Kwan Lin Chen; Mei Hsuang Wu; Vishal Chudasama; Kalpesh Prajapati; Heena Patel; Anjali Sarvaiya; Kishor Upla; Kiran Raja; Raghavendra Ramachandra; Christoph Busch; Etienne De Stoutz

doi:10.1109/CVPRW53098.2021.00284

Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge: Report

Andrey Ignatov, Cheng Ming Chiang, Hsien Kai Kuo, Anastasia Sycheva, Radu Timofte, Min Hung Chen, Man Yu Lee, Yu Syuan Xu, Yu Tseng, Shusong Xu, Jin Guo, Chao Hung Chen, Ming Chun Hsyu, Wen Chia Tsai, Chao Wei Chen, Grigory Malivenko, Minsu Kwon, Myungje Lee, Jaeyoon Yoo, Changbeom KangShinjo Wang, Zheng Shaolong, Hao Dejun, Xie Fen, Feng Zhuang, Yipeng Ma, Jingyang Peng, Tao Wang, Fenglong Song, Chih Chung Hsu, Kwan Lin Chen, Mei Hsuang Wu, Vishal Chudasama, Kalpesh Prajapati, Heena Patel, Anjali Sarvaiya, Kishor Upla, Kiran Raja, Raghavendra Ramachandra, Christoph Busch, Etienne De Stoutz

Institute of Data Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

12 Citations (Scopus)

Abstract

As the quality of mobile cameras starts to play a crucial role in modern smartphones, more and more attention is now being paid to ISP algorithms used to improve various perceptual aspects of mobile photos. In this Mobile AI challenge, the target was to develop an end-to-end deep learning-based image signal processing (ISP) pipeline that can replace classical hand-crafted ISPs and achieve nearly real-time performance on smartphone NPUs. For this, the participants were provided with a novel learned ISP dataset consisting of RAW-RGB image pairs captured with the Sony IMX586 Quad Bayer mobile sensor and a professional 102-megapixel medium format camera. The runtime of all models was evaluated on the MediaTek Dimensity 1000+ platform with a dedicated AI processing unit capable of accelerating both floating-point and quantized neural networks. The proposed solutions are fully compatible with the above NPU and are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

Original language	English
Title of host publication	Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021
Publisher	IEEE Computer Society
Pages	2503-2514
Number of pages	12
ISBN (Electronic)	9781665448994
DOIs	https://doi.org/10.1109/CVPRW53098.2021.00284
Publication status	Published - 2021 Jun
Event	2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 - Virtual, Online, United States Duration: 2021 Jun 19 → 2021 Jun 25

Publication series

Name	IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
ISSN (Print)	2160-7508
ISSN (Electronic)	2160-7516

Conference

Conference	2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021
Country/Territory	United States
City	Virtual, Online
Period	21-06-19 → 21-06-25

All Science Journal Classification (ASJC) codes

Computer Vision and Pattern Recognition
Electrical and Electronic Engineering

Access to Document

10.1109/CVPRW53098.2021.00284

Cite this

Ignatov, A., Chiang, C. M., Kuo, H. K., Sycheva, A., Timofte, R., Chen, M. H., Lee, M. Y., Xu, Y. S., Tseng, Y., Xu, S., Guo, J., Chen, C. H., Hsyu, M. C., Tsai, W. C., Chen, C. W., Malivenko, G., Kwon, M., Lee, M., Yoo, J., ... De Stoutz, E. (2021). Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge: Report. In Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 (pp. 2503-2514). (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops). IEEE Computer Society. https://doi.org/10.1109/CVPRW53098.2021.00284

Ignatov, Andrey ; Chiang, Cheng Ming ; Kuo, Hsien Kai et al. / Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge : Report. Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society, 2021. pp. 2503-2514 (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops).

@inproceedings{3935fd6536c74b6eb3ab1e396198ce73,

title = "Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge: Report",

abstract = "As the quality of mobile cameras starts to play a crucial role in modern smartphones, more and more attention is now being paid to ISP algorithms used to improve various perceptual aspects of mobile photos. In this Mobile AI challenge, the target was to develop an end-to-end deep learning-based image signal processing (ISP) pipeline that can replace classical hand-crafted ISPs and achieve nearly real-time performance on smartphone NPUs. For this, the participants were provided with a novel learned ISP dataset consisting of RAW-RGB image pairs captured with the Sony IMX586 Quad Bayer mobile sensor and a professional 102-megapixel medium format camera. The runtime of all models was evaluated on the MediaTek Dimensity 1000+ platform with a dedicated AI processing unit capable of accelerating both floating-point and quantized neural networks. The proposed solutions are fully compatible with the above NPU and are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.",

author = "Andrey Ignatov and Chiang, {Cheng Ming} and Kuo, {Hsien Kai} and Anastasia Sycheva and Radu Timofte and Chen, {Min Hung} and Lee, {Man Yu} and Xu, {Yu Syuan} and Yu Tseng and Shusong Xu and Jin Guo and Chen, {Chao Hung} and Hsyu, {Ming Chun} and Tsai, {Wen Chia} and Chen, {Chao Wei} and Grigory Malivenko and Minsu Kwon and Myungje Lee and Jaeyoon Yoo and Changbeom Kang and Shinjo Wang and Zheng Shaolong and Hao Dejun and Xie Fen and Feng Zhuang and Yipeng Ma and Jingyang Peng and Tao Wang and Fenglong Song and Hsu, {Chih Chung} and Chen, {Kwan Lin} and Wu, {Mei Hsuang} and Vishal Chudasama and Kalpesh Prajapati and Heena Patel and Anjali Sarvaiya and Kishor Upla and Kiran Raja and Raghavendra Ramachandra and Christoph Busch and {De Stoutz}, Etienne",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 ; Conference date: 19-06-2021 Through 25-06-2021",

year = "2021",

month = jun,

doi = "10.1109/CVPRW53098.2021.00284",

language = "English",

series = "IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops",

publisher = "IEEE Computer Society",

pages = "2503--2514",

booktitle = "Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021",

address = "United States",

}

Ignatov, A, Chiang, CM, Kuo, HK, Sycheva, A, Timofte, R, Chen, MH, Lee, MY, Xu, YS, Tseng, Y, Xu, S, Guo, J, Chen, CH, Hsyu, MC, Tsai, WC, Chen, CW, Malivenko, G, Kwon, M, Lee, M, Yoo, J, Kang, C, Wang, S, Shaolong, Z, Dejun, H, Fen, X, Zhuang, F, Ma, Y, Peng, J, Wang, T, Song, F, Hsu, CC, Chen, KL, Wu, MH, Chudasama, V, Prajapati, K, Patel, H, Sarvaiya, A, Upla, K, Raja, K, Ramachandra, R, Busch, C & De Stoutz, E 2021, Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge: Report. in Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, IEEE Computer Society, pp. 2503-2514, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021, Virtual, Online, United States, 21-06-19. https://doi.org/10.1109/CVPRW53098.2021.00284

Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge : Report. / Ignatov, Andrey; Chiang, Cheng Ming; Kuo, Hsien Kai et al.

Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society, 2021. p. 2503-2514 (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge

T2 - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021

AU - Ignatov, Andrey

AU - Chiang, Cheng Ming

AU - Kuo, Hsien Kai

AU - Sycheva, Anastasia

AU - Timofte, Radu

AU - Chen, Min Hung

AU - Lee, Man Yu

AU - Xu, Yu Syuan

AU - Tseng, Yu

AU - Xu, Shusong

AU - Guo, Jin

AU - Chen, Chao Hung

AU - Hsyu, Ming Chun

AU - Tsai, Wen Chia

AU - Chen, Chao Wei

AU - Malivenko, Grigory

AU - Kwon, Minsu

AU - Lee, Myungje

AU - Yoo, Jaeyoon

AU - Kang, Changbeom

AU - Wang, Shinjo

AU - Shaolong, Zheng

AU - Dejun, Hao

AU - Fen, Xie

AU - Zhuang, Feng

AU - Ma, Yipeng

AU - Peng, Jingyang

AU - Wang, Tao

AU - Song, Fenglong

AU - Hsu, Chih Chung

AU - Chen, Kwan Lin

AU - Wu, Mei Hsuang

AU - Chudasama, Vishal

AU - Prajapati, Kalpesh

AU - Patel, Heena

AU - Sarvaiya, Anjali

AU - Upla, Kishor

AU - Raja, Kiran

AU - Ramachandra, Raghavendra

AU - Busch, Christoph

AU - De Stoutz, Etienne

PY - 2021/6

Y1 - 2021/6

N2 - As the quality of mobile cameras starts to play a crucial role in modern smartphones, more and more attention is now being paid to ISP algorithms used to improve various perceptual aspects of mobile photos. In this Mobile AI challenge, the target was to develop an end-to-end deep learning-based image signal processing (ISP) pipeline that can replace classical hand-crafted ISPs and achieve nearly real-time performance on smartphone NPUs. For this, the participants were provided with a novel learned ISP dataset consisting of RAW-RGB image pairs captured with the Sony IMX586 Quad Bayer mobile sensor and a professional 102-megapixel medium format camera. The runtime of all models was evaluated on the MediaTek Dimensity 1000+ platform with a dedicated AI processing unit capable of accelerating both floating-point and quantized neural networks. The proposed solutions are fully compatible with the above NPU and are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

AB - As the quality of mobile cameras starts to play a crucial role in modern smartphones, more and more attention is now being paid to ISP algorithms used to improve various perceptual aspects of mobile photos. In this Mobile AI challenge, the target was to develop an end-to-end deep learning-based image signal processing (ISP) pipeline that can replace classical hand-crafted ISPs and achieve nearly real-time performance on smartphone NPUs. For this, the participants were provided with a novel learned ISP dataset consisting of RAW-RGB image pairs captured with the Sony IMX586 Quad Bayer mobile sensor and a professional 102-megapixel medium format camera. The runtime of all models was evaluated on the MediaTek Dimensity 1000+ platform with a dedicated AI processing unit capable of accelerating both floating-point and quantized neural networks. The proposed solutions are fully compatible with the above NPU and are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

UR - http://www.scopus.com/inward/record.url?scp=85115992041&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85115992041&partnerID=8YFLogxK

U2 - 10.1109/CVPRW53098.2021.00284

DO - 10.1109/CVPRW53098.2021.00284

M3 - Conference contribution

AN - SCOPUS:85115992041

T3 - IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops

SP - 2503

EP - 2514

BT - Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021

PB - IEEE Computer Society

Y2 - 19 June 2021 through 25 June 2021

ER -

Ignatov A, Chiang CM, Kuo HK, Sycheva A, Timofte R, Chen MH et al. Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge: Report. In Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021. IEEE Computer Society. 2021. p. 2503-2514. (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops). doi: 10.1109/CVPRW53098.2021.00284

Learned smartphone ISP on mobile NPUs with deep learning, mobile AI 2021 challenge: Report

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this