浏览全部资源
扫码关注微信
Received：11 March 2019，
Revised：2019-4-16，
Accepted：23 April 2019，
Published：16 October 2019
移动端阅览
目的
2
近年来，随着我国遥感技术的快速发展，遥感数据呈现出大数据的特点，遥感数据的时效性增强，针对新环境下遥感算法编程语言众多，程序运行和部署环境需求多样，程序的集成和部署困难的问题，提出了一种遥感算法程序快速封装与Docker容器化系统集成架构。
方法
2
该系统架构主要包括：1）遥感算法程序的镜像自动化封装制作；2）镜像的分发管理，达到算法程序镜像的共享；3）遥感信息产品生产流程的容器化编排服务，将相关联的算法程序镜像串联，以满足特定遥感信息产品的生产；4）容器的调度运行，调用镜像，实现特定遥感产品的容器化运行。本文在上述容器化系统集成架构下，以Landsat5数据的NDVI、NDWI信息产品的生产作为容器化生产实例，并同物理机、KVM（kernel-based virtual machine）虚拟机在运行时间、内存占用量、部署效率等性能进行了对比。
结果
2
Docker容器虚拟化环境下的产品生产和物理机环境下在运行时间和内存占用量上几无差别，优于KVM虚拟机。Docker容器虚拟化环境和KVM虚拟机环境下在部署上能够节省大量时间，相比于物理机环境能够提高部署效率。
结论
2
容器化的系统集成方式能够有效解决遥感算法程序集成和部署困难的问题，有利于遥感算法程序的复用和流程的共享，提高系统集成效率，具备较强的遥感数据实时快速处理能力。
Objective
2
Remote sensing data have obvious big data characteristics owing to the continuous improvement in the spatial
temporal
spectral
and radiometric resolution for remote sensing data. The seamless integration and deployment of remote sensing image processing algorithms are becoming a big challenge in the era of remote sensing big data and cloud computing due to diverse and complex remote sensing image processing algorithms. Virtualization technology provides a feasible solution to the above problems. Docker is a new open source virtualization container technology. Compared with traditional virtualization technologies
such as KVM (kernel-based virtual machine)
Docker container is a virtualizing operating system and has the advantages of lightweight and resource efficiency. In this paper
we propose a system framework for rapid integration of remote sensing algorithms on the basis of Docker container.
Method
2
The framework consists of an automated image encapsulation mechanism for remote sensing algorithms
a unified image distribution management
a containerized orchestration service for production of remote sensing information product
and a container scheduling scheme about daemons. We use the Dockerfile files to package automatically the base image
program dependencies
and remote sensing algorithm programs to build a new image layer by layer. A container is made up of a number of readable layers and a readable and writable layer. The Docker image is a read-only Docker container template that contains the file system structure
and its contents needed to start the Docker container. An image is the basis for starting a container. The Docker image is a static view of the Docker container
which is the running state of the Docker image. We upload the image to the repository such as "DockerHub" via the "Docker push" command. Users on other machines download the corresponding image to the local via the "Docker pull" command. The image arrangement refers to the serial connection of the associated containers in a logical sequence according to the production of the remote sensing information product. The image is represented by a compose file
which consists of three parts:version
services
and networks. The compose file is used to reproduce and share about the running process of flow. We design a container operation scheme based on the JAVA platform. First
the user sends an order task to the back end through the front-end interactive interface and backend parsing commands. The backend creates a specific compose.yaml file based on the template file. Finally
the running image of the remote sensing algorithm program is containerized. Production of NDVI and NDWI information products based on Landsat5 data is an example of prototype system and containerized production. In this experiment
remote sensing data are distributed storage using the distributed file system GlusterFS. The system consists of three server hosts and one client host. The server host is used to store data in a distributed manner. The client host reads the data of the server host by mounting. The client host is also the environment where the Docker container is integrated and running. In this experiment
Landsat5 DN value data are used as input data. The final output data are the binarized product data. The algorithm programs used include radiation correction programs written in C++
typical feature inversion index programs written in Python
and binarization programs written in MATLAB. We perform computational and deployment performance experiments
such as runtime
memory usage
and deployment efficiency in Docker virtualization environment
KVM virtual machine environment
and physical machine environment. Deployment efficiency refers to the configuration complexity required to deploy the same multiple applications in different environments. Run time is the time that takes to cycle through the same application in three environments. Memory usage refers to the number of memory usage of the same application running in parallel in the three environments.
Result
2
Almost no differences in performance are observed in system load metrics
such as operational efficiency and memory footprint
running applications in Docker container virtualization environments
and physical machine environments. The three environments are better than the KVM virtual machine environment. However
when installing and deploying a production environment on a new machine
the Docker container environment and the KVM virtual machine environment can reduce the amount of configuration and facilitate program migration and reuse compared with the physical machine environment.
Conclusion
2
The containerized system integration method can solve effectively the problem of difficulty in the integration and deployment of remote sensing algorithm programs. The Docker container runs different remote sensing algorithm programs
which enable resource isolation and eliminate the problem of dependency conflicts among different programs. The deployment of applications using the Docker container enables one-click deployment and operation
which facilitates the reusability of remote sensing algorithm programs and facilitates software-level sharing. Our containerized framework is promising for improving the efficiency and seamlessness of system integration compared with physical and virtual machine environments.
Li D R, Zhang L P, Xia G S. Automatic analysis and mining of remote sensing big data[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12):1211-1216.
李德仁, 张良培, 夏桂松.遥感大数据自动分析与数据挖掘[J].测绘学报, 2014, 43(12):1211-1216. [DOI:10.13485/j.cnki.11-2089.2014.0187]
Antunes R R, Blaschke T, Tiede D, et al. Proof of concept of a novel cloud computing approach for object-based remote sensing data analysis and classification[J]. GIScience&Remote Sensing, 2019, 56(4):536-553.[DOI:10.1080/15481603.2018.1538621]
Ren F H, Wang J N. Turning remote sensing to cloud services:technical research and experiment[J]. Journal of Remote Sensing, 2012, 16(6):1331-1346.
任伏虎, 王晋年.遥感云服务平台技术研究与实验[J].遥感学报, 2012, 16(6):1331-1346.]
Zhu J Z, Shi Q, Chen F E, et al. Research status and development trends of remote sensing big data[J]. Journal of Image and Graphics, 2016, 21(11):1425-1439.
朱建章, 石强, 陈凤娥, 等.遥感大数据研究现状与发展趋势[J].中国图象图形学报, 2016, 21(11):1425-1439. [DOI:10.11834/jig.20161102]
Zhang B, Huang W J, Zhang H, et al. The current and future situation of dynamic monitoring technology for the earth resource and environment[J]. Journal of Remote Sensing, 201620(6):1470-1478.
张兵, 黄文江, 张浩, 等.地球资源环境动态监测技术的现状与未来[J].遥感学报, 2016, 20(6):1470-1478. [DOI:10.11834/JRS.20166369]
Wang X Z, Zhang H M, Zhao J H, et al. An interactive web-based analysis framework for remote sensing cloud computing[C]//ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2015, Ⅱ-4/W2.[ DOI: 10.5194/isprsannals-Ⅱ-4-W2-43-2015 http://dx.doi.org/10.5194/isprsannals-Ⅱ-4-W2-43-2015 ]
Yan J N, Ma Y, Wang L Z, et al. A cloud-based remote sensing data production system[J]. Future Generation Computer Systems, 2017, 86:1154-1166.[DOI:10.1016/j.future.2017.02.044]
Kivity A, Kamay Y, Laor D, et al. KVM: the linux virtual machine monitor[C]//Proceedings of Linux Symposium, 2007, 1: 225-230.
Cui Z Y, Zhao H Q. Research and application of virtualization based on KVM[J]. Computer Technology and Development, 2011, 21(6):108-111, 115.
崔泽永, 赵会群.基于KVM的虚拟化研究及应用[J].计算机技术与发展, 2011, 21(6):108-111, 115. [DOI:10.3969/j.issn.1673-629X.2011.06.029]
Li G Q, Huang Z C. Data infrastructure for remote sensing big data:integration, management and on-demand service[I]. Journal of Computer Research and Development, 2017, 54(2):267-283.
李国庆, 黄震春.遥感大数据的基础设施:集成、管理与按需服务[J].计算机研究与发展, 2017, 54(2):267-283. [DOI:10.7544/issn1000-1239.2017.20160837]
Kozhirbayev Z, Sinnott R O. A performance comparison of container-based technologies for the cloud[J]. Future Generation Computer Systems, 2017, 68:175-182.[DOI:10.1016/j.future.2016.08.025]
Yang P, Ma Z C, Peng B, et al. Performance research of OpenStack cloud platform integrated with Docker container[J]. Computer Engineering, 2017, 43(8):26-31.
杨鹏, 马志程, 彭博, 等.集成Docker容器的OpenStack云平台性能研究[J].计算机工程, 2017, 43(8):26-31. [DOI:10.3969/j.issn.1000-3428.2017.08.005]
Wu Y. Research on virtualization technology based on Docker[J]. Information Technology, 2016, (1):121-123, 128.
伍阳.基于Docker的虚拟化技术研究[J].信息技术, 2016, (1):121-123, 128. [DOI:10.13274/j.cnki.hdzj.2016.01.031]
Combe T, Martin A, Di Pietro R. To Docker or not toDocker:a security perspective[J]. IEEE Cloud Computing, 2016, 3(5):54-62.[DOI:10.1109/MCC.2016.100]
Salza P, Ferrucci F. Speed up genetic algorithms in the cloud using software containers[J]. Future Generation Computer Systems, 2019, 92:276-289.[DOI:10.1016/j.future.2018.09.066]
Wan X L, Guan X J, Wang T J, et al. Application deployment using microservice and Docker containers:framework and optimization[J]. Journal of Network and Computer Applications, 2018, 119:97-109.[DOI:10.1016/j.jnca.2018.07.003]
De Benedictis M, Lioy A. Integrity verifcation of Docker containers for a lightweight cloud environment[J]. Future Generation Computer Systems, 2019, 97:236-246.[DOI:10.1016/j.future.2019.02.026]
Watts T, Benton R, Glisson W, et al. Insight from a Docker container introspection[C]//Proceedings of the 52nd Hawaii International Conference on System Sciences. Hawaii, USA: HICSS, 2019: 6875-6884.[ DOI: 10.24251/HICSS.2019.863 http://dx.doi.org/10.24251/HICSS.2019.863 ]
Felter W, Ferreira A, Rajamony R, et al. An updated performance comparison of virtual machines and Linux containers[C]//Proceedings of 2015 IEEE International Symposium on Performance Analysis of Systems and Software. Philadelphia, America: IEEE. 2007: 171-172.[ DOI: 10.1109/ISPASS.2015.7095802 http://dx.doi.org/10.1109/ISPASS.2015.7095802 ]
Joy A M. Performance comparison between Linux containers and virtual machines[C]//Proceedings of 2015 International Conference on Advances in Computer Engineering and Applications. Ghaziabad, India: IEEE, 2015: 342-346.[ DOI: 10.1109/ICACEA.2015.7164727 http://dx.doi.org/10.1109/ICACEA.2015.7164727 ]
Xavier B, Ferreto T, and Jersak L. Time provisioning evaluation of KVM, Docker and unikernels in a cloud platform[C]//Proceedings of the 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing. Cartagena, Colombia: IEEE, 2016: 277-280.[ DOI: 10.1109/CCGrid.2016.86 http://dx.doi.org/10.1109/CCGrid.2016.86 ]
Kim Y M, Poline J B, Dumas G. Experimenting with reproducibility:a case study of robustness in bioinformatics[J]. Gigascience, 2018, 7(7):giy077.[DOI:10.1093/gigascience/giy077]
Knoth C, Nüst D. Reproducibility and practical adoption of geobia with open-source software in Docker containers[J]. Remote Sensing, 2017, 9(3):290.[DOI:10.3390/rs9030290]
Boettiger C. An introduction to Docker for reproducible research[J]. ACM SIGOPS Operating Systems Review-Special Issue on Repeatability and Sharing of Experimental Artifacts, 2015, 49(1):71-79.[DOI:10.1145/2723872.2723882]
Piccolo S R, Frampton M B. Tools and techniques for computational reproducibility[J]. GigaScience, 2016, 5(1):#30.[DOI:10.1186/s13742-016-0135-4]
Morris D, Voutsinas S, Hambly N C, et al. Use of Docker for deployment and testing of astronomy software[J]. Astronomy and Computing, 2017, 20:105-119.[DOI:10.1016/j.ascom.2017.07.004]
Zhang P, Zhang H M, Li J H. A large-scale remote sensing data cloud service platform based on Linux container[J]. E-science Technology&Application, 2015, 6(2):47-55.
张品, 张海明, 黎建辉.一种基于Linux容器技术的大规模遥感数据云服务平台[J].科研信息化技术与应用, 2015, 6(2):47-55. [DOI:10.11871/j.issn.1674-9480.2015.02.006]
Li S P, Ji X, Hu Q X, et al. Research on containerized management technology of power grid GIS platform based on cloud computing[J]. Electric Power Information and Communication Technology, 2017, 15(7):7-12.
刘淑平, 纪鑫, 胡强新, 等.云架构下的电网GIS平台容器化管理技术研究[J].电力信息与通信技术, 2017, 15(7):7-12. [DOI:10.16543/j.2095-641x.electric.power.ict.2017.07.002]
He Z P, Zhang X D, Cao P. The research on continuous integration for big data platform of intelligent campus based on container technology[J]. Intelligent Computer and Applications, 2017, 7(5):38-40.
贺宗平, 张晓东, 曹萍.基于容器技术的智慧校园大数据平台持续集成研究[J].智能计算机与应用, 2017, 7(5):38-40. [DOI:10.3969/j.issn.2095-2163.2017.05.011]
Tong F, Wang X L, Li J Y, et al. Application of Docker technology in bioinformatics[J]. Military Medical Sciences, 2016, 40(7):614-617.
佟凡, 王小磊, 李江域, 等. Docker技术在生物信息学中的应用[J].军事医学, 2016, 40(7):614-617. [DOI:10.7644/j.issn.1674-9960.2016.07.019]
Software Engineering Lab of Zhejiang University. Docker Container and Container Cloud[M]. The second edition. Beijing:Posts&Telecom Press, 2016.
浙江大学. Docker容器与容器云[M]. 2版.北京:人民邮电出版社, 2016.
Mouat A. Using Docker: Developing and Deploying Software with Containers[M]. Beijing: Posts&Telecom Press, 2017.Mouat A. Docker
开发指南[M].黄彦邦, 译.北京: 人民邮电出版社, 2017.
Xiong W, Yu Z B, Xu C Z. A characterization and analysis of distributed file systems[J]. Journal of Integration Technology, 2012, 1(4):58-63.
熊文, 喻之斌, 须成忠.几个常见分布式文件系统特征分析和性能对比[J].集成技术, 2012, 1(4):58-63.
Modak A, Chaudhary S D, Paygude P S, et al. Techniques to secure data on cloud: Docker Swarm or Kubernetes?[C]//Proceedings of the 2nd International Conference on Inventive Communication and Computational Technologies. Coimbatore, India: IEEE, 2018: 7-12.[ DOI: 10.1109/ICICCT.2018.8473104 http://dx.doi.org/10.1109/ICICCT.2018.8473104 ]
Gupta R R, Mishra G, Katara S, et al. Data storage security in cloud computing using container clustering[C]//Proceedings of the 7th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference. New York, NY, USA: IEEE, 2016: 1-7.[ DOI: 10.1109/UEMCON.2016.7777913 http://dx.doi.org/10.1109/UEMCON.2016.7777913 ]
相关文章
相关作者
相关机构
京公网安备11010802024621
