Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform

Yi Wei Huang, Sheng-Tzong Cheng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recent advances in sensor technology has led to varieties of sensors. Problems facing now is how to exploit and compute these data efficiently. Map Reduce is a programming model to solve these problem. However, it may cause I/O bottleneck. In-memory computing (IMC) comes up to solve the problems. However, network bandwidth remains a bottleneck. It restricts the speed of receiving the information from the source and dispersing information to each node. To our observation, some data from sensor devices might be similar due to temporal or spatial dependency. Therefore, compression technology could be an effective solution. It replaces data with smaller sizes of streams for improving data utilization. This study presents an effective reduce transmission scheme on a distributed real-time IMC platform-Spark Streaming. We design and implement a system to provide a high compression ratio in a small batch data from the source. It is expected to reduce data transmission with little delay time in the soft real-time fashion.

Original languageEnglish
Title of host publicationICUFN 2018 - 10th International Conference on Ubiquitous and Future Networks
PublisherIEEE Computer Society
Pages244-249
Number of pages6
ISBN (Print)9781538646465
DOIs
Publication statusPublished - 2018 Aug 14
Event10th International Conference on Ubiquitous and Future Networks, ICUFN 2018 - Prague, Czech Republic
Duration: 2018 Jul 32018 Jul 6

Publication series

NameInternational Conference on Ubiquitous and Future Networks, ICUFN
Volume2018-July
ISSN (Print)2165-8528
ISSN (Electronic)2165-8536

Other

Other10th International Conference on Ubiquitous and Future Networks, ICUFN 2018
CountryCzech Republic
CityPrague
Period18-07-0318-07-06

Fingerprint

Data communication systems
Sensors
Data storage equipment
Electric sparks
Time delay
Compaction
Bandwidth
Big data

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture

Cite this

Huang, Y. W., & Cheng, S-T. (2018). Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform. In ICUFN 2018 - 10th International Conference on Ubiquitous and Future Networks (pp. 244-249). [8436821] (International Conference on Ubiquitous and Future Networks, ICUFN; Vol. 2018-July). IEEE Computer Society. https://doi.org/10.1109/ICUFN.2018.8436821
Huang, Yi Wei ; Cheng, Sheng-Tzong. / Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform. ICUFN 2018 - 10th International Conference on Ubiquitous and Future Networks. IEEE Computer Society, 2018. pp. 244-249 (International Conference on Ubiquitous and Future Networks, ICUFN).
@inproceedings{fa370b3177f4410aa81ad84a80870e2f,
title = "Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform",
abstract = "Recent advances in sensor technology has led to varieties of sensors. Problems facing now is how to exploit and compute these data efficiently. Map Reduce is a programming model to solve these problem. However, it may cause I/O bottleneck. In-memory computing (IMC) comes up to solve the problems. However, network bandwidth remains a bottleneck. It restricts the speed of receiving the information from the source and dispersing information to each node. To our observation, some data from sensor devices might be similar due to temporal or spatial dependency. Therefore, compression technology could be an effective solution. It replaces data with smaller sizes of streams for improving data utilization. This study presents an effective reduce transmission scheme on a distributed real-time IMC platform-Spark Streaming. We design and implement a system to provide a high compression ratio in a small batch data from the source. It is expected to reduce data transmission with little delay time in the soft real-time fashion.",
author = "Huang, {Yi Wei} and Sheng-Tzong Cheng",
year = "2018",
month = "8",
day = "14",
doi = "10.1109/ICUFN.2018.8436821",
language = "English",
isbn = "9781538646465",
series = "International Conference on Ubiquitous and Future Networks, ICUFN",
publisher = "IEEE Computer Society",
pages = "244--249",
booktitle = "ICUFN 2018 - 10th International Conference on Ubiquitous and Future Networks",
address = "United States",

}

Huang, YW & Cheng, S-T 2018, Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform. in ICUFN 2018 - 10th International Conference on Ubiquitous and Future Networks., 8436821, International Conference on Ubiquitous and Future Networks, ICUFN, vol. 2018-July, IEEE Computer Society, pp. 244-249, 10th International Conference on Ubiquitous and Future Networks, ICUFN 2018, Prague, Czech Republic, 18-07-03. https://doi.org/10.1109/ICUFN.2018.8436821

Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform. / Huang, Yi Wei; Cheng, Sheng-Tzong.

ICUFN 2018 - 10th International Conference on Ubiquitous and Future Networks. IEEE Computer Society, 2018. p. 244-249 8436821 (International Conference on Ubiquitous and Future Networks, ICUFN; Vol. 2018-July).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform

AU - Huang, Yi Wei

AU - Cheng, Sheng-Tzong

PY - 2018/8/14

Y1 - 2018/8/14

N2 - Recent advances in sensor technology has led to varieties of sensors. Problems facing now is how to exploit and compute these data efficiently. Map Reduce is a programming model to solve these problem. However, it may cause I/O bottleneck. In-memory computing (IMC) comes up to solve the problems. However, network bandwidth remains a bottleneck. It restricts the speed of receiving the information from the source and dispersing information to each node. To our observation, some data from sensor devices might be similar due to temporal or spatial dependency. Therefore, compression technology could be an effective solution. It replaces data with smaller sizes of streams for improving data utilization. This study presents an effective reduce transmission scheme on a distributed real-time IMC platform-Spark Streaming. We design and implement a system to provide a high compression ratio in a small batch data from the source. It is expected to reduce data transmission with little delay time in the soft real-time fashion.

AB - Recent advances in sensor technology has led to varieties of sensors. Problems facing now is how to exploit and compute these data efficiently. Map Reduce is a programming model to solve these problem. However, it may cause I/O bottleneck. In-memory computing (IMC) comes up to solve the problems. However, network bandwidth remains a bottleneck. It restricts the speed of receiving the information from the source and dispersing information to each node. To our observation, some data from sensor devices might be similar due to temporal or spatial dependency. Therefore, compression technology could be an effective solution. It replaces data with smaller sizes of streams for improving data utilization. This study presents an effective reduce transmission scheme on a distributed real-time IMC platform-Spark Streaming. We design and implement a system to provide a high compression ratio in a small batch data from the source. It is expected to reduce data transmission with little delay time in the soft real-time fashion.

UR - http://www.scopus.com/inward/record.url?scp=85052498363&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85052498363&partnerID=8YFLogxK

U2 - 10.1109/ICUFN.2018.8436821

DO - 10.1109/ICUFN.2018.8436821

M3 - Conference contribution

AN - SCOPUS:85052498363

SN - 9781538646465

T3 - International Conference on Ubiquitous and Future Networks, ICUFN

SP - 244

EP - 249

BT - ICUFN 2018 - 10th International Conference on Ubiquitous and Future Networks

PB - IEEE Computer Society

ER -

Huang YW, Cheng S-T. Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform. In ICUFN 2018 - 10th International Conference on Ubiquitous and Future Networks. IEEE Computer Society. 2018. p. 244-249. 8436821. (International Conference on Ubiquitous and Future Networks, ICUFN). https://doi.org/10.1109/ICUFN.2018.8436821