Scaling Notebooks as Re-configurable Cloud Workflows

Subjects: Computer Science >> Integration Theory of Computer Science submitted time 2022-11-28 Cooperative journals: 《数据智能（英文）》

Yuandou, Wang Spiros, Koulouzis Riccardo, Bianchi Na, Li Yifang, Shi Joris, Timmermans W. Daniel, Kissling Zhiming, Zhao

Abstract： Literate computing environments, such as the Jupyter (i.e., Jupyter Notebooks, JupyterLab, and JupyterHub), have been widely used in scientific studies; they allow users to interactively develop scientific code, test algorithms, and describe the scientific narratives of the experiments in an integrated document. To scale up scientific analyses, many implemented Jupyter environment architectures encapsulate the whole Jupyter notebooks as reproducible units and autoscale them on dedicated remote infrastructures (e.g., high#2; performance computing and cloud computing environments). The existing solutions are still limited in many ways, e.g., 1) the workflow (or pipeline) is implicit in a notebook, and some steps can be generically used by different code and executed in parallel, but because of the tight cell structure, all steps in the Jupyter notebook have to be executed sequentially and lack of the flexibility of reusing the core code fragments, and 2) there are performance bottlenecks that need to improve the parallelism and scalability when handling extensive input data and complex computation. In this work, we focus on how to manage the workflow in a notebook seamlessly. We 1) encapsulate the reusable cells as RESTful services and containerize them as portal components, 2) provide a composition tool for describing workflow logic of those reusable components, and 3) automate the execution on remote cloud infrastructure. Empirically, we validate the solution’s usability via a use case from the Ecology and  Earth Science domain, illustrating the processing of massive Light Detection and Ranging (LiDAR) data. The demonstration and analysis show that our method is feasible, but that it needs further improvement, especially on integrating distributed workflow scheduling, automatic deployment, and execution to develop as a mature approach.

 Hits 1531  Downloads 227  Comment

Improvement and transcriptome analysis of root architecture by overexpression of Fraxinus pennsylvanica DREB2A transcription factor in Robinia pseudoacacia L. ‘Idaho’

Subjects: Biology >> Botany >> Plant biochemistry, plant Biophysics submitted time 2016-05-04

Yu Xiu Arshad Iqbal Chen Zhu Guodong Wu Yanping Chang Na Li Yu Cao Wenbiao Zhang Huiming Zeng Shouyi Chen Huafang Wang

Abstract：

Transcription factors play a key role to enable plants to cope with abiotic stresses. DREB2 regulates the expression of several stress-inducible genes and constitutes major hubs in the water stress signalling webs. We cloned and characterized a novel gene encoding the FpDREB2A transcription factor from Fraxinus pennsylvanica, and a yeast activity assay confirmed its DRE binding and transcription activation. Overexpression of FpDREB2A in R. pseudoacacia showed enhanced resistance to drought stress. The transgenic plant survival rate was significantly higher than that of WT in soil drying and re-watering treatments. Transgenic lines showed a dramatic change in root architecture, and horizontal and vertical roots were found in transgenic plants compared to WT. The vertical roots penetrated in the field soil to more than 60 cm deep, while horizontal roots expanded within the top 20–30 cm of the soil. A physiological test demonstrated that chlorophyll contents were more gradually reduced and that soluble sugars and proline levels elevated more sharply but malondialdehyde level stayed the same (P < 0.05). Plant hormone levels of abscisic acid and IAA were higher than that of WT, while gibberellins and zeatin riboside were found to be lower. The root transcriptomes were sequenced and annotated into 2011 differential expression genes (DEGs). The DEGs were categorized in 149 pathways and were found to be involved in plant hormone signalling, transcription factors, stimulus responses, phenylalanine, carbohydrate and other metabolic pathways. The modified pathways in plant hormone signalling are thought to be the main cause of greater horizontal and vertical root development, in particular.