are we talking about using the digispark for a (virtual) serial comport or a (virtual) keyboard?
As far as I know, using the digispark as a virtual USB comport, the baudrate is adapted automatically by the driver on the PC. When you start a terminal on the PC and choose some baudrate, the PC driver will adapt to it.
Talking about the (virtual) digispark keyboard: there is no baudrate.
The software USB emulation keeps the digispark very busy. He is doing as fast as he can. If it is too slow for you, you must choose a microprocessor with a hardware USB unit, for example the Arduino Micro. They are much faster and more reliable.