The SDC was initially intended as a single-site data centre for solar observation data, mainly from the German solar telescopes in Tenerife, Spain, located at KIS in Freiburg; this early version was built on MongoDB/GridFS, and extensive automation of data processing and incorporating third-party data was not a priority. Although this version of the SDC was already successful, fast and robust beyond our initial expectations, it became clear relatively soon that it would be beneficial from a scientific point of view to open up to data from other sources and replace the technical foundation with standard products used in similar national and European efforts for data management.
After two years of development, testing and integration, the transition of the SDC's technical foundation from MongoDB/GridFS to Rucio/dCache is now complete. All metadata and non-scientifically usable data will continue to be hosted in an additional MongoDB; one of the following steps will be to make this separation fully transparent to the user by extending Rucio's application programming interface (API) for metadata queries into our MongoDB.
Rucio is a data management solution used in several modern experiments and large-scale data projects, e.g. in Punch4NFDI on a national and ESCAPE on a European level. Thus, the completed transition lays the foundation to seamlessly federate and interoperate with these data lakes.
The web interface interface for locating and downloading data in the SDC has also been greatly improved. Previously based on jQuery, SemanticUI and Java EE, it is now a modern, significantly accelerated single-page application based on Next.js, Bootstrap and Spring Boot, where search parameters can be subsequently adjusted directly on the results page.
The milestones achieved in data management, online data search and the underlying data abstraction layer represent a significant step towards our goal of correlating solar data across different instruments and sources by merging, processing and offering data from third-party sources in a homogeneous, standardised way.
The SDC Tech Team will next focus on automatic data transport between OT and SDC, the implementation of site and data level independent access embargoes (of which only a proof of concept exists to date) and the injection of new data from sources other than OT.
