Maybe try to switch to another filter that doesn’t have such initialization delay, like poly-sinc-ext2 or poly-sinc-xtr (closest ones)?
I’m mostly using poly-sinc-ext2 myself at the moment, after a long while of poly-sinc-short-mp. Actually surprising change, given how different the two are, while delivering similar properties but in a vastly different way. I didn’t expect this change at all.