According to the
Differences (from the Arduino) and limitations section of the
Digispark Basics page (
https://digistump.com/wiki/digispark/tutorials/basics)
Pin 3 and Pin 4 (P3 and P4) are used for USB communication and programming, while you can use them in your circuit if you are not using USB communication, you may have to unplug your circuit during programming if the circuit would impede the pin states or dramatically affect the voltage levels on these pins.
Try moving your input to another PWM pin, P0 or P1 (since you are not using any other pins in your circuit.) See if that does the job.