What buffer size did you have originally, and what did you “reduce” it to as mentioned above?
Might also try to do the same recording technique with the buffer size at MAXIMUM. Do the MIDI notes move earlier than the 8 msec they are now (1/64th note at 120 BPM)?