Hi Allen,

It would also be interesting if you could take a look at how memory consumption grows while the simulation is running (using top or similar).

Are you recording spike or membrane potential data to memory during the simulation? With you very long-running simulations, that could flood memory over time. A quick back-of-the envelope calculation:

648 neurons x 1 membrane potential per ms x 300000 ms x 24B / values (V_m, sender, time) = 4.3 GB

If you do record data to memory during the entire simulation, try running without recording. If problem still occur, data may be collecting elsewhere and we would need to check further. Have you tried a comparable simulation without MUSIC?

Best, Hans Ekkehard

--

Prof. Dr. Hans Ekkehard Plesser Head, Department of Data Science

Faculty of Science and Technology Norwegian University of Life Sciences PO Box 5003, 1432 Aas, Norway

Phone +47 6723 1560 Email hans.ekkehard.plesser@nmbu.nomailto:hans.ekkehard.plesser@nmbu.no Home http://arken.nmbu.no/~plesser

On 24/11/2020, 10:10, "Trensch, Guido" <g.trensch@fz-juelich.demailto:g.trensch@fz-juelich.de> wrote:

Dear Allen,

a “bad_alloc” exception does not indicate a disk space problem, but rather refers to a memory shortage. Please try to increase the base memory of your VM.

Best regards, Guido

From: Allen Rabayda [mailto:arabayda@masonlive.gmu.edu] Sent: Monday, 23 November 2020 17:24 To: NEST User Mailing List users@nest-simulator.org Subject: [NEST Users] error std::bad_alloc: virtualbox

Dear Nest Community,

Has anyone encountered a "bad_alloc" error like the one below and if so, any recommendations? It appears to be a VM memory issue but only using 21% of harddrive space (ref: below).

My simulation successfully completes for 200,000 ms but errors out at 98% complete for 230,000 ms, 75% for 300,000 ms and 56% for 400,000 ms.

I'm running on NEST 2.18.0 VirtualBox lubuntu 18.04 (ref: image of settings below).

Thank you for any suggestions.

Best Regards, --Allen ********************************************** **** Error Message *****

...

# SIMULATION nest.Simulate(300000)

Nov 21 17:10:28 NodeManager::prepare_nodes [Info]: Preparing 684 nodes for simulation.

Nov 21 17:10:28 MUSICManager::enter_runtime [Info]: Entering MUSIC runtime with tick = 1 ms

Nov 21 17:10:28 SimulationManager::start_updating_ [Info]: Number of local nodes: 684 Simulation time (ms): 300000 Number of OpenMP threads: 2 Number of MPI processes: 1

75 %: network time: 223698.0 ms, realtime factor: 0.6277Traceback (most recent call last): File "<stdin>", line 3, in <module> File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/ll_api.py", line 246, in stack_checker_func return f(*args, **kwargs) File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/lib/hl_api_simulation.py", line 66, in Simulate sr('ms Simulate') File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/ll_api.py", line 132, in catching_sli_run raise exceptionCls(commandname, message) nest.ll_api.std::bad_alloc: ('std::bad_alloc in Simulate_d: C++ exception: std::bad_alloc', 'std::bad_alloc', <SLILiteral: Simulate_d>, ': C++ exception: std::bad_alloc') ********************************************

**** Folder Space on VirtualBox after Error **** nest@nestvm:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 5.2G 0 5.2G 0% /dev tmpfs 1.1G 1.1M 1.1G 1% /run /dev/sda1 99G 20G 76G 21% / tmpfs 5.2G 0 5.2G 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 5.2G 0 5.2G 0% /sys/fs/cgroup SharedNest2 917G 447G 470G 49% /media/sf_SharedNest2 tmpfs 1.1G 16K 1.1G 1% /run/user/1000 /dev/sr0 74M 74M 0 100% /media/nest/VBox_GAs_6.0.10

***********************************

**** VirtualBox Settings *******

------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

Dear Guido,

I forgot to email you to let you know your suggestion worked perfectly. It appears I had 4 processing threads so nest was recording 4 potential files where each one > 42 GB. This storage was exceeding my available Ubuntu disk drive space and host available storage as it was recording.

Thank you.

Best Regards, --Allen

On Wed, Jan 20, 2021 at 2:05 PM Trensch, Guido g.trensch@fz-juelich.de wrote:

Dear Allen,

please can you set the parameter ‘to_memory’ for the recording devices, especially for the multimeter, explicitly to false?

I was able to reproduce the increase in memory consumption when I ran the simulate250.profile.py script you provided. Setting the parameter solved the problem for me.

Best regards,

Guido

*From:* Hans Ekkehard Plesser [mailto:hans.ekkehard.plesser@nmbu.no] *Sent:* Tuesday, 24 November 2020 10:20 *To:* NEST User Mailing List users@nest-simulator.org *Subject:* [NEST Users] Re: error std::bad_alloc: virtualbox

Hi Allen,

It would also be interesting if you could take a look at how memory consumption grows while the simulation is running (using top or similar).

Are you recording spike or membrane potential data to memory during the simulation? With you very long-running simulations, that could flood memory over time. A quick back-of-the envelope calculation:

648 neurons x 1 membrane potential per ms x 300000 ms x 24B / values (V_m, sender, time) = 4.3 GB

If you do record data to memory during the entire simulation, try running without recording. If problem still occur, data may be collecting elsewhere and we would need to check further. Have you tried a comparable simulation without MUSIC?

Best,

Hans Ekkehard

--

Prof. Dr. Hans Ekkehard Plesser

Head, Department of Data Science

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

On 24/11/2020, 10:10, "Trensch, Guido" g.trensch@fz-juelich.de wrote:

Dear Allen,

a “bad_alloc” exception does not indicate a disk space problem, but rather refers to a memory shortage. Please try to increase the base memory of your VM.

Best regards,

Guido

*From:* Allen Rabayda [mailto:arabayda@masonlive.gmu.edu arabayda@masonlive.gmu.edu] *Sent:* Monday, 23 November 2020 17:24 *To:* NEST User Mailing List users@nest-simulator.org *Subject:* [NEST Users] error std::bad_alloc: virtualbox

Dear Nest Community,

Has anyone encountered a "bad_alloc" error like the one below and if so, any recommendations? It appears to be a VM memory issue but only using 21% of harddrive space (ref: below).

My simulation successfully completes for 200,000 ms but errors out at 98% complete for 230,000 ms, 75% for 300,000 ms and 56% for 400,000 ms.

I'm running on NEST 2.18.0 VirtualBox lubuntu 18.04 (ref: image of settings below).

Thank you for any suggestions.

Best Regards,

--Allen

---

**** Error Message *****

...

...

# SIMULATION nest.Simulate(300000)

Nov 21 17:10:28 NodeManager::prepare_nodes [Info]: Preparing 684 nodes for simulation.

Nov 21 17:10:28 MUSICManager::enter_runtime [Info]: Entering MUSIC runtime with tick = 1 ms

Nov 21 17:10:28 SimulationManager::start_updating_ [Info]: Number of local nodes: 684 Simulation time (ms): 300000 Number of OpenMP threads: 2 Number of MPI processes: 1

75 %: network time: 223698.0 ms, realtime factor: 0.6277Traceback (most recent call last): File "<stdin>", line 3, in <module> File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/ll_api.py", line 246, in stack_checker_func return f(*args, **kwargs) File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/lib/hl_api_simulation.py", line 66, in Simulate sr('ms Simulate') File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/ll_api.py", line 132, in catching_sli_run raise exceptionCls(commandname, message) nest.ll_api.std::bad_alloc: ('std::bad_alloc in Simulate_d: C++ exception: std::bad_alloc', 'std::bad_alloc', <SLILiteral: Simulate_d>, ': C++ exception: std::bad_alloc')

---

**** Folder Space on VirtualBox after Error ****

nest@nestvm:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 5.2G 0 5.2G 0% /dev tmpfs 1.1G 1.1M 1.1G 1% /run /dev/sda1 99G 20G 76G 21% / tmpfs 5.2G 0 5.2G 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 5.2G 0 5.2G 0% /sys/fs/cgroup SharedNest2 917G 447G 470G 49% /media/sf_SharedNest2 tmpfs 1.1G 16K 1.1G 1% /run/user/1000 /dev/sr0 74M 74M 0 100% /media/nest/VBox_GAs_6.0.10

---

**** VirtualBox Settings *******

---

---

Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt

---

---

---

NEST Users mailing list -- users@nest-simulator.org To unsubscribe send an email to users-leave@nest-simulator.org