Ah, but is that just with the internal pullups? As if that is the case, you are enabling the internal pullup resistors on the digispark, so that's where your resistors are. In other words, if you did a pinMode of INPUT_PULLUP, or setting it as an input and then doing a digitalWrite HIGH (I think I got the order of the second case right... has been a while since I did that particular one).
The answer is yes, basically setting a pin to OUTPUT and HIGH, and then connecting it to GND will damange that pin. Momentary contacts like those of switches or rotary encoder might be fine, but no longer. Doing something like a INPUT_PULLUP actually connects the pin internally to VCC through an internal resistors, hence why that is safe.
I'm of course assuming you did something like the
Trinket USB Volume control, but with a digispark instead

Doe that answer your question?