A data pipeline for Optical and Near-infrared Solar Eruption Tracer

分类： 天文学 >> 天文学 提交时间： 2021-03-12 合作期刊: 《天文研究与技术》

Xinhua Wang Dong Chen Tao Deng Hongbing Dai Yongyuan Xiang

摘要： With the advent of large astronomical equipments, the traditional development model for data reduction faces problems such as redundancy of programs and conflicting environmental dependencies; Besides as a cluster is a highly coupled computing resource, serious environmental conflicts can lead to the unavailability of the entire cluster. To address this problem, we have developed a new pipeline framework using the concept of microservices. This paper presents the ONSET data pipeline developed through this framework. To achieve near real-time data processing, we optimize the core program using MPI and GPU technologies and evaluate the final performance. The results show that this development model can be built in a short time to meet the requirements of the pipeline, and we believe that this development model has implications for future multi-band and multi-terminal astronomical data processing.

 点击量 7224  下载量 925  评论