Dear Ady,
sorry for the late reply.
In general, I recommend that you upgrade to the latest NEST version (3.3 as of writing) as to exclude any bugs that might have been fixed in the (quite long) meantime.
Regarding MPI4Py, it can well be that this interacts badly with structural plasticity. I don't know of anyone who has tried this before.
If the data you want to write out is coming directly from a recording device in your simulation (i.e. is not created by your own Python code), you could used the recording backend for SIONlib, which internally uses MPIIO to write the data in a distributed fashion to a bunch of binary files. The documentation has more: https://nest-simulator.readthedocs.io/en/v3.3/models/recording_backend_sionl...
You coud then use https://github.com/nest/nest-sionlib-reader to read the data. The reader might still be a bit rough around the edges, though.
Cheers, Jochen!
On 16.07.22 12:52, Aadhar Sharma wrote:
Hi there!
I am using the structural plasticity module to simulate large-scale networks on HPC. The way I run these simulations with MPI creates a large number of small data files (.npy) which I organize into a HDF5 file in a post hoc manner. This has hitherto not been a problem, however, I recently increased the size of my simulations and the number of files that I create exceeds my HPCinode quota. So, I have to come up with a way of handling all these files or the way I export them.
Ideally, I would like to concurrently write to a file (like zarr) with multiple processes. However, parallel zarr writes are only optimal when the chunk size are uniform. For me, they are not uniform---so, I abandoned the idea of using zarr.
Currently, I am trying to use mpi4py to gather the data I want and save it through a mother MPI process. I ran some test simulations and all was fine (see attached script). However, when I use structural plasticity with mpi4py I get an error that:
"nest.lib.hl_api_exceptions.DimensionMismatch: ('DimensionMismatch in SetStatus_id: Expected dimension size: 1\nProvided dimension size: 20', 'DimensionMismatch', <SLILiteral: SetStatus_id>, ': Expected dimension size: 1\nProvided dimension size: 20".
I believe that mpi4py somehow interferes with nest's internal mpi mechanism. Perhaps it signals nest kernel to use 1 process rather than the mentioned 20. If I remove the mpi4py import call, all works fine. So, I cannot run the simulation using this mpi4py hack.
Do you notice something wrong with my approach? I'm just beginning to use mpi4py, so I may be missing something here.
Alternatively, could you recommend a way to reduce the number of exported files. For example, through zarr or other file formats which support parallel writes.
I'd appreciate any suggestions. Thanks!
pyNEST version: 2.20.2 python version: Python 3.7.10 mpiexec/mpirun version: MPI Library for Linux* OS, Version 2021.3
Best, Ady
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
-- Dr. Jochen Martin Eppler Phone: +49(2461)61-96653 ---------------------------------- Simulation Laboratory Neuroscience Jülich Supercomputing Centre Institute for Advanced Simulation
------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------