As I understand it the VirtualWire library is quite timing sensitive. This is one of the core reasons for the creation of the Manchester library (which I've ported to the digispark with some success). If you'd like to improve the timing accuracy of your digispark you can install the unsupported bootloader upgrade by going in to the Tools menu, and choosing Burn Bootloader, then plug your digispark in, wait for programming to finish, then wait a further five seconds. After this, your bootloader will have been upgraded to 1.06, which fixes the clock speed to 16.5mhz regardless of if it was plugged in to USB when it started or powered in a different way. The 16.0, 8, and 1mhz board options know about this feature and automatically detect it, so 16.0mhz mode will adjust down from 16.5mhz at startup regardless if it was started from USB or not, and the resulting clock speed will therefore be more stable, as with current versions there can be two different 16.0mhz clock speeds depending if the chip started up naturally or with USB, one adjusting down from the precisely calibrated 16.5mhz speed and the other just using the default atmel configuration which is only specified as accurate to within 10% of 16.5mhz (rather than the <1% deviation of the USB calibrated speed).
Digistump wont replace your digisparks if the bootloader upgrade doesn't work correctly, so be careful not to bump it during the upload and unplug shields if you can.