Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform

  • 黃 奕崴

Student thesis: Master's Thesis


Recent advances in sensor technology have led to the availability of a multitude of the sensor e g sound luminosity and humidity Huge raw data is a difficult problem to exploit and compute these data efficiently Hadoop MapReduce has been used to solve this issue but the operations which need iteration is not an efficient to handle these data Hence “In-memory Computing concept (IMC)” is come up to resolve the problem of Hadoop I/O bottleneck In in-memory computing the data is computed parallel in random access memory (RAM) instead of slow disk drives We can train patterns and analyze large data frequently by IMC technique However IMC platform does not provide an effective reduce transmission scheme in the real-time system It may limit some applications like wireless sensor network It may be impractical for transmitting entire data from each sensor node due to weak resource such as CPU Memory Power etc Compress data before sending is an effective way to make good use of sensor nodes limited power supply and make better the life of sensors According to our observation most of the sensor data has a similar pattern due to time dependence and spatial dependence Therefore we can improve compression efficiency by these characteristics This study presents an effective reduce transmission scheme on a distributed real-time IMC platform “Spark Streaming” which is used to collect data in real-time We describe the whole system design and implement that provides a high compression ratio in a small batch data from the source It is expected to reduce data transmission with a little delay time in the soft real-time system
Date of Award2017 Aug 23
Original languageEnglish
SupervisorSheng-Tzong Cheng (Supervisor)

Cite this