WG3 - Data processing and visualisation

Driver	@Adrian Priceputu @Bruno Chareyre
Approver	@Daniel Barreto
Contributors	Bruno Chareyre,@John P. Morrissey and waiting for others to join
Informed
Objective	Produce a common standard output file format which may be used across different DEM open-source codes Generate a protocol in which data from less commonly used contact models can be generated using the Open DEM format Provide a set of visualization tools that can work as plug-ins of open-source tools that are widely used
Due date
Key outcomes	D2.1 Inventorying open-source high performance visualisation tools capable to receive specific developments through plugin mechanism (September 2024) D2.2 Selecting the data to be saved in a common DEM file covering a range of applications (Technical Report - December 2024) D2.3 Defining a list of the most common DEM post-processors and data analyses (June 2025) D1.2 Writing the file format specifications (December 2025) D1.3 Implementing the Open DEM file format (Technical Report - April 2027) D2.2 Development of post-processing plugins for the software defined at D2.1 (July 2027) D2.4 Enriching visualisation tools (July 2027) - needs a bit of clarification
Status	in progress

Problem Statement

This Working Group proposes to produce a common standard output file format which may be used across different DEM open-source codes. The aim is to provide this in the form of an optimised file format (Open DEM format) that will enable faster post-processing and analysis of data across multiple common applications. This will at least include the most widely used contact models.

Scope

Must have:	a set of the most common post-processing tools and algorithms list of data needed to be saved for exchanging between DEM codes list of data needed to be saved for post-processing a framework to add this data by anyone, independent on the code a protocol to expand the list of data, allowing less commonly used codes to be integrated plugins to be added to DEM software for visualizing results

Must have:	a set of the most common post-processing tools and algorithms list of data needed to be saved for exchanging between DEM codes list of data needed to be saved for post-processing a framework to add this data by anyone, independent on the code a protocol to expand the list of data, allowing less commonly used codes to be integrated plugins to be added to DEM software for visualizing results
Nice to have:	a list of most commonly used contact models (from WG2)
Not in scope:	…

Timeline

Common DEM file format

Visualisation and post-processing

Communication

enumerating data to save

writing the file format specifications

implementing the Open DEM file format

inventorying the available tools

defining a list of common DEM post-processing tools

developing plugins to read the Open DEM file format

implementing post-processing plugins

communicating about these new features

Milestones and deadlines

Milestone	Owner	Team	Deadline	Status

Milestone

Owner

Team

Deadline

Status

enumerating the list of data to save

Bruno Chareyre

@John P. Morrissey

@Adrian Priceputu

Ulrich Odefey

Sep 4, 2024

this is a partial deadline

in progress

inventorying open-source visualisation tools capable of receiving plugins

@Adrian Priceputu

Sep 4, 2024

Reference materials

HDF5 based VELaSSCo format - Developing a standardised file format for large scale DEM visualisation and Analytics
VTK HDF file format for visualizing large data sets: https://www.kitware.com/vtk-hdf-reader/

VELaSCCo Project (Visualization For Extremely Large-Scale Scientific Computing): http://www.cimne.com/projects/velassco/index.html
The CFD community has General Notation System (CGNS) which “provides a general, portable, and extensible standard for the storage and retrieval of computational fluid dynamics (CFD) analysis data.”. While they may be slightly ahead of the DEM community, it is a very good example of how to define a common standard. Lots of detail. Read the variable name identifiers as an example: https://cgns.org/standard/SIDS/appendix_a.html. Their HDF5 file format is described here: https://cgns.org/standard/hdf5.html
There is a HDF5 based file format extension available for LAMMPS: https://arxiv.org/abs/1308.6382#. Also https://github.com/pdebuyl/pyh5md/tree/master. This only stores data and requires additional steps for visualisation.