Load Balancing For Cloud-Based Dynamic Data Processing

Abstract

The Map/Reduce paradigm has dominated cloud computing since its beginnings. However, there are some scenarios in which Map/Reduce is not the best model. Once such situation is a system that collects data dynamically, with intermittent arrival times. In this study, we study a modified form of Map/Reduce that uses a load balancer to distribute work, rather than simply assigning a Map node in an ad-hoc fashion. We show that this approach performs significantly better than standard Map/Reduce. In particular, it reduces the amount of time data is waiting in a queue to be processed.