That is pretty weird.
Did you wire up this circuit on a breadboard (following your KiCad schematic exactly,) yet?
If not, try that, and see if it works. Although, I think that it will. From your description, it sounds like there must be some kind of cross-talk, or some other interference, causing this trouble. If you can get it working on your breadboard, you will be able to verify that your schematic is correct. Then you can concentrate on looking for the exact trouble in your PCB layout.
Also, did you let KiCad route the traces for you? If you did, try manually routing the traces, and leave out the ground plane (while a good idea, you really don't need the ground plane for this simple of a circuit.) The ground plane should protect your circuit from this kind of stuff, but if it touches the wrong point, it can act like an antenna, and bring in all sorts of noise.