Spigot: Fragment-level file sharing and consistency on grids

Conventional data grid systems require the application to access remote data through explicit APIs, consequently sacrificing user transparency. Thus, some systems enable the application to transparently access remote data by copying the entire file to a user's local storage before executing the application, even when only a tiny file fragment is required. Such an approach consumes unnecessary transmission time and storage space, with the additional problem of maintaining replica synchronization. Traditionally, replica consistency treats shared files as read-only, consequently sacrificing guaranteed replica consistency. This paper presents a DSM-based fragment-level file sharing framework called "Spigot". Spigot allows the users to design their applications with the POSIX I/O interface. Moreover, Spigot transfers only the necessary file fragments on user demand, thereby reducing transmission time, wasted network bandwidth and required storage space. Data waiting time is further reduced by overlapping data transmission and data analysis. The DSM concept maintains replica synchronization. Real experiments show reduced turnaround times in data-intensive applications.