Objective

2

As data collection methods mature and diversify

data sources such as personal smart devices

floating car GPS

internet of things

and social media are becoming increasingly abundant

and the amount of data have been accumulating in an explosive manner. Big data hold spatio-temporal information and high-dimensional features. Spatial and temporal features refer to attribute fields with spatial position and time tags. High dimensional features mean that the target data often contain other valuable attributes. Visual analysis is a highly important method for big data research as it can quickly and intuitively help researchers analyze and understand intrinsic values. However

because of its massive volume

spatio-temporal correlation

and high dimensions

big data visualization poses many challenges to current implementations

including large memory consumption

high rendering delay

and poor visual effects.

Method

2

In this study

we propose a generic multi-dimension aggregation pyramid (MAP) model on the basis of the well-known 2D tile pyramid model. This MAP model can support the hierarchical aggregation of time

space

and attributes simultaneously and transform the aggregated results into discrete key-value pairs for scalable storage and efficient retrieval. Then

we use the high-performance Spark cluster as a parallel preprocessing platform and the distributed HBase as final storage to store the generated MAP data. Finally

with the generated MAP datasets

we design and implement an open-source distributed visualization framework (MAP-Vis).

Result

2

The experiments use the open New York taxi data

which cover 30 months from January 2014 to June 2016. A single record contains trip-related information

including the location and time of the taxi origin/destination

trip duration

and distance. The visualization interface is implemented on the MAP-Vis framework

which uses HTML

CSS

and JavaScript. Leaflet and OpenStreetMap are used for road network display; the timeline and attribute histogram sections use the d3 library to support user interaction. Three efficiency metrics are collected to evaluate the performance of the MAP model and MAP-Vis system in terms of model validation

storage scalability

and system scalability. In the experiment of model validation

as the size of the raw data increases

the response time curve remains flat and does not show a significant linear increase; the values slightly fluctuate between 0.7 s and 1 s. This result indicates that the MAP model can scale well with the size of spatio-temporal data sets

guarantee a sub-second response

and achieve a smooth interactive visualization experience. In the experiment of storage scalability

as the number of clusters increases

the overall response time decreases dramatically from 3.2 s to 0.9 s

and the parallel efficiency is improved by approximately 2.4 times. This finding can be attributed to distributed storage. More storage nodes are used and the possibility of access to only one region and the access queue time are reduced. Therefore

by increasing the number of HBase storage regions

the proposed framework enhances query efficiency

fully exploits the parallelism of distributed clusters

and significantly improves the visual interactive experience. In the experiment of system scalability

the number of worker nodes in the Spark cluster is changed to measure how the pre-processing time changes (excluding the time of importing the HBase database). An increase in the number of nodes leads to the reduction of pre-processing time from 360 min to 160 min

and the efficiency is improved by approximately 1.3 times. Therefore

with computation nodes

the Spark cluster uses worker nodes and executor processes to share pre-processing tasks

thereby significantly improving the pre-processing efficiency.

Conclusion

2

Given its large size

space-time properties

high dimension

and other characteristics

spatial-temporal big data face various challenges such as large memory consumption

high rendering delay

and poor visual effect. To solve this problem

we first propose a spatio-temporal big data organization model

namely

the MAP

which integrates the tile pyramid model and the key-value matching method. The MAP model can consider the time and space dimensions

attribute information

and the three aggregate aggregations step by step

thereby adapting to the rapid and high visualization of time and space big data. On the basis of the MAP model

an open-source visualization framework

MAP-Vis

is implemented on a Linux cluster. The MAP-Vis system uses Spark as a pre-processing tool and HBase as a distributed storage platform. Experiments validate the efficiency of the proposed MAP model

and the undrerlying distributed platforms provide high scalability for visualization and processing. With the cluster

the MAP-Vis realizes sub-second data query and achieves good interactive visualization. Future work can be conducted in the following aspects. 1) This framework has strong support for point type data

but visual elements

including line type elements

polygon type elements

images

etc. should be considered compatible with other data types as much as possible. 2) A simple visual display cannot fully explore the law and value of big data. Hence

joining data analysis modules could be taken into consideration to make the MAP-Vis framework function complete.