OK I tried it at 2.4ms latency, and results were a lot better. Still early (ish) but not the 50ms delay there was before.
But thats 1 instrument with nothing else playing. What happens when I want to record while playing along to the track ? At that buffer size, the CPU will overload.