Tame the Big Data beast!
GPI-Space delivers faster results and brings you a competitive advantage while others still wait for theirs.
Big Data: MapReduce
The typical problem that is solve with MapReduce is extracting information out of the data contained in a wast amount of records.
A MapReduce program is composed of a map part that performs filtering and sorting (such as sorting all the words in a long text into queues, one queue for each word) and a reduce part that performs a summary operation on the results of the mapping (such as counting the number of words in each queue, yielding the frequency of occurance). The graph on the left shows execution times for such a simple word count problem.
With GPI-Space, those kind of MapReduce problems can be solved in-memory and thus much faster than on any other framework using traditional disk I/O. Especially for time-critical applications, such as MonteCarlo simulations in the financial sector, this method can deliver a competetive advantage. And with the GPI-Space framework, all a user has to implement are the map, reduce and partition methods and the framework takes care of everything else.
Streaming Data: Lambda Architecture
GPI-Space is unifying batch and speed layer of the Lambda Architecture, allowing new approaches to existing big data challenges by adding a layer for streaming data into the GPI-Space core. With this new functionality, real time data analysis projects can be realized without leaving the familiar development environment of GPI-Space.
The goal of a software project by Fraunhofer ITWM and some of its industry partners is, to build a real-time monitoring system for smart meters in large building complexes, such as hospitals, hotels, office buildings etc. Live data from smart meters, attached to key distribution boxes in the building, is fed into GPI-Space where advanced algorithms evaluate the data for specific patterns. With non-intrusive appliance load monitoring, individual consumers are identified without the need for direct metering. Live data analysis as well as analysis of historic data in storage are used to optimize a building’s energy consumption. Some of the information, that can be extracted via data mining from the smart meter data is:
Prediction of future energy consumption, based on previous years, months, weeks and days.
Monitoring of individual devices with high energy consumption to implement auto-off or fault-detection.
Use energy consumption forcast to optimize the use of renewable energy sources in the building, such as solar panels.