Hi Tom,

I now created a NEST issue to put in place a proper stopper in case things become too slow:

https://github.com/nest/nest-simulator/issues/2033 .

BTW, I assume that you are using the ht_neuron? That might slow done quite a lot because the of the adaptive stepsize solvers for the highly non-linear membrane currents.

Best,

Hans Ekkehard

--

Prof. Dr. Hans Ekkehard Plesser

Head, Department of Data Science

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

On 06/05/2021, 19:19, "TOM BUGNON" <bugnon@wisc.edu> wrote:

Hi Hans,

Thanks for the tip! I ended up doing something slightly different, since in my case the simulation can go from regular speed to infinitely slow pretty fast (so that the last 10ms Run call before freezing never finishes).

In case it's useful to someone:

· I run in 10ms steps

· If the simulation starts slowing such that one of the steps takes longer than 2s, I run the next 10ms in "ministeps" (equal to the kernel resolution)

· if 3 consecutive "ministeps" last longer than 1s I raise an error

Below is the corresponding code,
Cheers!

```

class HangingSimulationError(Exception):

    pass

def nest_run(sim_time):

    import nest, time

    t = time.time()

    nest.Run(sim_time)

    return time.time() - t

def simulate_in_steps(simulation_time):

    """Simulate in steps and catch hanging simulations."""

    import nest

    step = 10 # (virtual time)

    ministep = nest.GetKernelStatus('resolution')

    N_break = 3

    split_t = 2 # (real time) (s) Run next chunk in mini steps if previous took longer

    break_t = 1 # (real time) (s) break if `N_break` consecutive mini-steps takes longer

    total_sim_time = 0

    kernel_time = nest.GetKernelStatus('time')

    with nest.RunManager():

        while total_sim_time < simulation_time:

            next_step_t = min(step, simulation_time - total_sim_time)

            step_real_t = nest_run(next_step_t)

            total_sim_time += next_step_t

            if step_real_t > split_t:

                # Run next {step}ms in ministeps

                # Break if 3 consecutive ministeps last too long

                count = 0

                print(f"Running next {step}ms in ministeps")

                for _ in range(int(step/ministep)):

                    if simulation_time == total_sim_time:

                        break

                    next_ministep_t = min(ministep, simulation_time - total_sim_time)

                    t = nest_run(next_ministep_t)

                    total_sim_time += next_ministep_t

                    if t > break_t:

                        count += 1

                    else:

                        count = 0

                    if count >= N_break:

                        raise HangingSimulationError(

                            f"Simulation froze at {nest.GetKernelStatus('time')}ms"

                        )

    assert nest.GetKernelStatus('time') - kernel_time == simulation_time

```

---

From: Hans Ekkehard Plesser <hans.ekkehard.plesser@nmbu.no>
Sent: Thursday, May 6, 2021 9:49 AM
To: NEST User Mailing List <users@nest-simulator.org>
Subject: [NEST Users] Re: Stop hanging simulations

Hi Tom,

As a DIY workaround, you can use the RunManager context to simulate in small steps and break if it gets too slow. I haven't tested the code, just sketching from memory. Instead of calling nest.Simulate(1000), use

with nest.RunManager():

    for _ in range(100):

        t = time.time()

        nest.Run(10)

        if time.time() - t > 5:

            break

The logic is as follows: You split the 1000 ms into 100 times 10 ms. This is fast with Run() within the RunManager(). You then use Python's time to see how long it takes to simulate 10 ms and break if it takes too long, here a 5 s limit. You can then use GetKernelStatus to get the current time in the simulation.

It would be interesting to add this as a kernel feature. Let me know if it works!

Best,

Hans Ekkehard

--

Prof. Dr. Hans Ekkehard Plesser

Head, Department of Data Science

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

On 06/05/2021, 16:08, "TOM BUGNON" <bugnon@wisc.edu> wrote:

Hi all,

Under some circumstances simulations can slow down up to the point where the nest.Simulate() does not advance anymore and stays stuck at a given virtual time, with a "realtime factor" of 0. I suppose this can happen for instance when a network falls into a regime of runaway excitation in which a massive number of spikes are being exchanged.

I'm looking for a way to stop the simulation in such a case, (say when the realtime factor goes below a set threshold, or when the output files are not updated for a certain duration), ideally in such a way that the program can continue running rather than crashing. If anyone has a suggestion about how to go around this issue I'd be happy to hear it.
Thanks in advance! Best, Tom