6 May
2021
6 May
'21
7:19 p.m.
Hi Hans,
Thanks for the tip! I ended up doing something slightly different, since in my case the
simulation can go from regular speed to infinitely slow pretty fast (so that the last 10ms
Run call before freezing never finishes).
In case it's useful to someone:
* I run in 10ms steps
* If the simulation starts slowing such that one of the steps takes longer than 2s, I
run the next 10ms in "ministeps" (equal to the kernel resolution)
* if 3 consecutive "ministeps" last longer than 1s I raise an error
Below is the corresponding code,
Cheers!
```
class HangingSimulationError(Exception):
pass
def nest_run(sim_time):
import nest, time
t = time.time()
nest.Run(sim_time)
return time.time() - t
def simulate_in_steps(simulation_time):
"""Simulate in steps and catch hanging simulations."""
import nest
step = 10 # (virtual time)
ministep = nest.GetKernelStatus('resolution')
N_break = 3
split_t = 2 # (real time) (s) Run next chunk in mini steps if previous took longer
break_t = 1 # (real time) (s) break if `N_break` consecutive mini-steps takes longer
total_sim_time = 0
kernel_time = nest.GetKernelStatus('time')
with nest.RunManager():
while total_sim_time < simulation_time:
next_step_t = min(step, simulation_time - total_sim_time)
step_real_t = nest_run(next_step_t)
total_sim_time += next_step_t
if step_real_t > split_t:
# Run next {step}ms in ministeps
# Break if 3 consecutive ministeps last too long
count = 0
print(f"Running next {step}ms in ministeps")
for _ in range(int(step/ministep)):
if simulation_time == total_sim_time:
break
next_ministep_t = min(ministep, simulation_time - total_sim_time)
t = nest_run(next_ministep_t)
total_sim_time += next_ministep_t
if t > break_t:
count += 1
else:
count = 0
if count >= N_break:
raise HangingSimulationError(
f"Simulation froze at
{nest.GetKernelStatus('time')}ms"
)
assert nest.GetKernelStatus('time') - kernel_time == simulation_time
```
________________________________
From: Hans Ekkehard Plesser <hans.ekkehard.plesser(a)nmbu.no>
Sent: Thursday, May 6, 2021 9:49 AM
To: NEST User Mailing List <users(a)nest-simulator.org>
Subject: [NEST Users] Re: Stop hanging simulations
Hi Tom,
As a DIY workaround, you can use the RunManager context to simulate in small steps and
break if it gets too slow. I haven't tested the code, just sketching from memory.
Instead of calling nest.Simulate(1000), use
with nest.RunManager():
for _ in range(100):
t = time.time()
nest.Run(10)
if time.time() - t > 5:
break
The logic is as follows: You split the 1000 ms into 100 times 10 ms. This is fast with
Run() within the RunManager(). You then use Python's time to see how long it takes to
simulate 10 ms and break if it takes too long, here a 5 s limit. You can then use
GetKernelStatus to get the current time in the simulation.
It would be interesting to add this as a kernel feature. Let me know if it works!
Best,
Hans Ekkehard
--
Prof. Dr. Hans Ekkehard Plesser
Head, Department of Data Science
Faculty of Science and Technology
Norwegian University of Life Sciences
PO Box 5003, 1432 Aas, Norway
Phone +47 6723 1560
Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no>
Home http://arken.nmbu.no/~plesser
On 06/05/2021, 16:08, "TOM BUGNON"
<bugnon@wisc.edu<mailto:bugnon@wisc.edu>> wrote:
Hi all,
Under some circumstances simulations can slow down up to the point where the
nest.Simulate() does not advance anymore and stays stuck at a given virtual time, with a
"realtime factor" of 0. I suppose this can happen for instance when a network
falls into a regime of runaway excitation in which a massive number of spikes are being
exchanged.
I'm looking for a way to stop the simulation in such a case, (say when the realtime
factor goes below a set threshold, or when the output files are not updated for a certain
duration), ideally in such a way that the program can continue running rather than
crashing. If anyone has a suggestion about how to go around this issue I'd be happy to
hear it.
Thanks in advance! Best, Tom