Hi, Has anyone explored the external library support? I am on Fugaku and the jobs can create casually tens of thousands of files in a single directory; this aborts jobs.
Itaru.
Dear Itaru,
we have created the new I/O infrastructure (and the SIONlib backend in particular) to address exactly the problem you describe. It is available in NEST master since quite some time and the user-level documentation can be found here:
https://nest-simulator-jougs.readthedocs.io/en/latest/from_cpp/recording_bac...
In order to enable it, you first install SIONlib (see this scripthttps://github.com/nest/nest-simulator/blob/master/extras/install_sionlib.sh for an example on how to do that) and then pass the path to NEST’s CMake using the --Dwith-sionlib=<path>.
Once everything is installed, you can just set the record_to property of any recording device to sionlib to let it write to SIONlib binary files. A minimal reader for the files can be found here: https://github.com/nest/nest-sionlib-reader
Please let me know of any problems you run into. As this is quite new and has not been tested extensively in the wild, there might still be some glitches.
Cheers, Jochen!
On 09.10.20 08:55, Itaru Kitayama wrote:
Hi, Has anyone explored the external library support? I am on Fugaku and the jobs can create casually tens of thousands of files in a single directory; this aborts jobs.
Itaru. _______________________________________________ NEST Users mailing list -- users@nest-simulator.orgmailto:users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.orgmailto:users-leave@nest-simulator.org
-- Dr. Jochen Martin Eppler Phone: +49(2461)61-96653 ---------------------------------- Simulation Laboratory Neuroscience Jülich Supercomputing Centre Institute for Advanced Simulation
------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
Jochen, Is it usable from the SLI interface? I've been running NEST on Fugaku using SLI.
On Mon, Oct 12, 2020 at 8:27 PM Jochen Martin Eppler j.eppler@fz-juelich.de wrote:
Dear Itaru,
we have created the new I/O infrastructure (and the SIONlib backend in particular) to address exactly the problem you describe. It is available in NEST master since quite some time and the user-level documentation can be found here:
https://nest-simulator-jougs.readthedocs.io/en/latest/from_cpp/recording_bac...
In order to enable it, you first install SIONlib (see this script for an example on how to do that) and then pass the path to NEST’s CMake using the --Dwith-sionlib=<path>.
Once everything is installed, you can just set the record_to property of any recording device to sionlib to let it write to SIONlib binary files. A minimal reader for the files can be found here: https://github.com/nest/nest-sionlib-reader
Please let me know of any problems you run into. As this is quite new and has not been tested extensively in the wild, there might still be some glitches.
Cheers, Jochen!
On 09.10.20 08:55, Itaru Kitayama wrote:
Hi, Has anyone explored the external library support? I am on Fugaku and the jobs can create casually tens of thousands of files in a single directory; this aborts jobs.
Itaru. _______________________________________________ NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
-- Dr. Jochen Martin Eppler Phone: +49(2461)61-96653
Simulation Laboratory Neuroscience Jülich Supercomputing Centre Institute for Advanced Simulation
Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
Dear Itaru,
the backend infrastructure is in the kernel and thus independent of the user interface you are using. In SLI you can query the list of available recording backends using
``` GetKernelStatus /recording_backends get info ```
and set the backend for a given recorder `recorder` using
``` recorder << /record_to /<backend_name> >> SetStatus ```
I just realized that I posted the link to my personal build of the documentation, which has not been updated in a while. The *official* and current version is actually here:
https://nest-simulator.readthedocs.io/en/latest/guides/recording_from_simula...
Cheers, Jochen!
On 13.10.20 02:59, Itaru Kitayama wrote:
Jochen, Is it usable from the SLI interface? I've been running NEST on Fugaku using SLI.
On Mon, Oct 12, 2020 at 8:27 PM Jochen Martin Eppler j.eppler@fz-juelich.de wrote:
Dear Itaru,
we have created the new I/O infrastructure (and the SIONlib backend in particular) to address exactly the problem you describe. It is available in NEST master since quite some time and the user-level documentation can be found here:
https://nest-simulator-jougs.readthedocs.io/en/latest/from_cpp/recording_bac...
In order to enable it, you first install SIONlib (see this script for an example on how to do that) and then pass the path to NEST’s CMake using the --Dwith-sionlib=<path>.
Once everything is installed, you can just set the record_to property of any recording device to sionlib to let it write to SIONlib binary files. A minimal reader for the files can be found here: https://github.com/nest/nest-sionlib-reader
Please let me know of any problems you run into. As this is quite new and has not been tested extensively in the wild, there might still be some glitches.
Cheers, Jochen!
On 09.10.20 08:55, Itaru Kitayama wrote:
Hi, Has anyone explored the external library support? I am on Fugaku and the jobs can create casually tens of thousands of files in a single directory; this aborts jobs.
Itaru. _______________________________________________ NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
-- Dr. Jochen Martin Eppler Phone: +49(2461)61-96653
Simulation Laboratory Neuroscience Jülich Supercomputing Centre Institute for Advanced Simulation
Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
[Re-sending with rendered Markdown]
Dear Itaru,
the backend infrastructure is in the kernel and thus independent of the user interface you are using. In SLI you can query the list of available recording backends using
|GetKernelStatus /recording_backends get info |
and set the backend for a given recorder |recorder| using
|recorder << /record_to /<backend_name> >> SetStatus |
I just realized that I posted the link to my personal build of the documentation, which has not been updated in a while. The /official/ and current version is actually here:
https://nest-simulator.readthedocs.io/en/latest/guides/recording_from_simula...
Cheers, Jochen!
On 13.10.20 02:59, Itaru Kitayama wrote:
Jochen, Is it usable from the SLI interface? I've been running NEST on Fugaku using SLI.
On Mon, Oct 12, 2020 at 8:27 PM Jochen Martin Eppler j.eppler@fz-juelich.de wrote:
Dear Itaru,
we have created the new I/O infrastructure (and the SIONlib backend in particular) to address exactly the problem you describe. It is available in NEST master since quite some time and the user-level documentation can be found here:
https://nest-simulator-jougs.readthedocs.io/en/latest/from_cpp/recording_bac...
In order to enable it, you first install SIONlib (see this script for an example on how to do that) and then pass the path to NEST’s CMake using the --Dwith-sionlib=<path>.
Once everything is installed, you can just set the record_to property of any recording device to sionlib to let it write to SIONlib binary files. A minimal reader for the files can be found here: https://github.com/nest/nest-sionlib-reader
Please let me know of any problems you run into. As this is quite new and has not been tested extensively in the wild, there might still be some glitches.
Cheers, Jochen!
On 09.10.20 08:55, Itaru Kitayama wrote:
Hi, Has anyone explored the external library support? I am on Fugaku and the jobs can create casually tens of thousands of files in a single directory; this aborts jobs.
Itaru. _______________________________________________ NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
-- Dr. Jochen Martin Eppler Phone: +49(2461)61-96653
Simulation Laboratory Neuroscience Jülich Supercomputing Centre Institute for Advanced Simulation
Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
Hello Jochen, all,
Thanks. I'll experiment with the IO library when I receive another round of machine time allocation. Success stories on JURECA or JUSUF encourage us a lot.
Itaru.
On Wed, Oct 14, 2020 at 5:31 PM Jochen Martin Eppler j.eppler@fz-juelich.de wrote:
[Re-sending with rendered Markdown]
Dear Itaru,
the backend infrastructure is in the kernel and thus independent of the user interface you are using. In SLI you can query the list of available recording backends using
GetKernelStatus /recording_backends get info
and set the backend for a given recorder recorder using
recorder << /record_to /<backend_name> >> SetStatus
I just realized that I posted the link to my personal build of the documentation, which has not been updated in a while. The official and current version is actually here:
https://nest-simulator.readthedocs.io/en/latest/guides/recording_from_simula...
Cheers, Jochen!
On 13.10.20 02:59, Itaru Kitayama wrote:
Jochen, Is it usable from the SLI interface? I've been running NEST on Fugaku using SLI.
On Mon, Oct 12, 2020 at 8:27 PM Jochen Martin Eppler j.eppler@fz-juelich.de wrote:
Dear Itaru,
we have created the new I/O infrastructure (and the SIONlib backend in particular) to address exactly the problem you describe. It is available in NEST master since quite some time and the user-level documentation can be found here:
https://nest-simulator-jougs.readthedocs.io/en/latest/from_cpp/recording_bac...
In order to enable it, you first install SIONlib (see this script for an example on how to do that) and then pass the path to NEST’s CMake using the --Dwith-sionlib=<path>.
Once everything is installed, you can just set the record_to property of any recording device to sionlib to let it write to SIONlib binary files. A minimal reader for the files can be found here: https://github.com/nest/nest-sionlib-reader
Please let me know of any problems you run into. As this is quite new and has not been tested extensively in the wild, there might still be some glitches.
Cheers, Jochen!
On 09.10.20 08:55, Itaru Kitayama wrote:
Hi, Has anyone explored the external library support? I am on Fugaku and the jobs can create casually tens of thousands of files in a single directory; this aborts jobs.
Itaru. _______________________________________________ NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
-- Dr. Jochen Martin Eppler Phone: +49(2461)61-96653
Simulation Laboratory Neuroscience Jülich Supercomputing Centre Institute for Advanced Simulation
Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
-- Dr. Jochen Martin Eppler Phone: +49(2461)61-96653
Simulation Laboratory Neuroscience Jülich Supercomputing Centre Institute for Advanced Simulation
NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org
-
Itaru Kitayama -
Jochen Martin Eppler