TLDR: add a 5.5v zenner diode between the digital pin on the spark and ground, and a resistor in serial from the pin to the switch.. so switch goes 5v, switch, then splits through a resistor to ground and a resistor to digital input, which then connects over to ground through the zenner diode (placed facing the opposite way to a normal diode - the painted on circle should face towards the digital input)
I bet this is your problem:
when you start or stop pressing the button, it connects or disconnects the little circuit through the resistor, and those relatively long wires to the switch have a small inductive capacity. When the switch is connected and power runs through it, the wires charge up, and when the switch disconnects, it's metal plates very slowly (in electron timescales) move apart.. picometers, then nanometers, eventually as much as a milimeter or two.. and along the way, it creates a spark gap. The inductive property of the wire tries to continue moving the same amount of current, and the only way to do that is to swap amps for volts, so the voltage on the wire rises up.. 5v, 6v, 10, 20, 100... maybe hundreds of volts.. enough to spark across the gap, which requires higher and higher voltages as it widens, sending jolts through the entire system easily enough to fry the digispark. Usually this isn't a problem - usually the linear regulator on the spark seems to do a decent job of making sure the spark isn't powered over 5v, so excess voltage bleeds out through the clamping diodes in the attiny85 and get thrown away as heat in the linear regulator. if the voltages involved were high enough through, I bet this wouldn't work. Zenner Diode should solve it through.
I've fried heaps of stuff from a similar situation - using unreliable connectors to connect things over 1m wires.. the connectors (audio jacks in this case) would make tiny sparkgaps as they moved slightly, and those tiny sparks would fry the attiny chip.