Correct.
When you hit a pad on the PD, a MIDI message is sent to the virtual instrument. This is close to instantaneous.
The virtual instrument then has to generate (or play) the sound.
That sound enters the audio system’s buffer, where it will stay until the buffer is sent to the interface.
If that buffer is set to a high value, then the sound will be more delayed.
All audio makes use of this buffer setting.
The only way to lessen the delay (it can never be totally eliminated) is to reduce the buffer setting.
This setting can be found in your interface’s control panel.
However, setting this too low will introduce distortion, cracks, and/or pops into your sound.
You’d need to experiment to find a value that works for you.