When I record with a vsti with my midi pickup I have to set the buffer at 64, no noticable latency and no audio problems.
Same when playing/recording live guitar with Amplitdue as an insert on that inputchannel, no problems at 64 buffer.
a high buffersetting is only needed for mixing, playing lots of tracks with plugins.