NOMAD facilitates and integrates high performance computing (HPC) usage in Horizon 2020.
Key partners providing HPC services are European centers with petascale compute systems and storage: Barcelona Supercomputing Center in Spain, CSC - IT Center for Science in Finland, the Leibniz Supercomputing Centre and the Max Planck Computing and Data Facility in Germany. Together, partners defined and developed the architecture of the infrastructure - the technology platform.
The NOMAD platform consists of a distributed multi-layered storage system connected to scalable computing capacity. Where feasible, already existing solutions at the supercomputing centers have been and will continue to be employed to minimize the need for new hardware and will enable the use of dedicated servers or e.g. cloud capacity.
The NOMAD platform is mainly built using Docker containers orchestrated by Kubernetes, as shown below. Virtualization makes it easier to set up the services on new sites. The NOMAD services use centralized user management. A Shibboleth-based Identity Provider (IdP) provides a single sign-on access to all NOMAD services. For example, saving Data Analytics Notebooks from the NOMAD Analytics Toolkit for later use requires setting up a NOMAD user account as a self service, but the NOMAD tools can also be tested anonymously without disclosing any personal information.
The NOMAD Laboratory CoE Infrastructure Platform is based on virtual machines. The Repository runs directly on virtual machines, whereas the Encyclopedia services run on containers on separate virtual machines, and the parsing infrastructure, the remote visualization and the Analytics Toolkit all run on Docker containers, managed by a shared Kubernetes cluster.
The distributed NOMAD infrastructure will be operated at a European scale. New sites, beyond the participating supercomputing centres, will also be supported to set up local installations of the NOMAD services, as needed. A local installation would enable providing extended capacity for eligible users. Services and individual processing tasks have been configured to minimize data redistributions in the global workflow.
NOMAD has signed a Memorandum of Understanding with the HPC-Europa3 (HPCE3) programme. HPCE3 funds international research visits and can be, for example, used to start collaboration projects with NOMAD developers to speed up adopting and develop together NOMAD analytics tools. Detailed description and eligibility for HPCE3 funded visits can be found at the HPCE3 website.
The paradigm shift towards data-driven science has also inspired other European initiatives to create data repositories for easy access and reuse of research data, for example the SeaDataCloud. The NOMAD team has accumulated important practical experience on these in close collaboration with HPC centers, researchers and industry.
Contact concerning general aspects of the CoE: Jessica Pietsch
Contact concerning HPC Services and Infrastructure: Atte Sillanpää