OK, first let me indicate that I have been experiencing the problems stated here (panning & zooming delays) with my Visio 2010 program (as most/all others here are expressing this issue with Visio 2013)
What I've found is that if I have shapes that I've made - that I insert into my drawing and they are drawn with solid lines --- pan & zoom are fine.
If I take that same shape and change it to a dashed line - pan & zoom slow to a CRAWL.
Here is a shape of mine that I inserted into a blank drawing.
Shape - (Solid Lines)
So in that same drawing, if I change the shape to either of these type Dashed lines

Shape - (Dashed Lines)

.....then pan & zoom slow to a CRAWL.
If I go back and change the shape back to solid lines --- Pan & Zoom work fine again.
I don't know if this is the reason for the delays that everyone is experiencing, but this is what I have noticed in my experiences with Visio 2010. Hopefully, something as simple as this could also apply for the problems everyone is experiencing in Visio
2013.