Glad another person had the same idea too!
I tried running SDT and SFC as recommended by Marrion_Y but it didn't find any issues with my SB2, nor had any different outcome to my Zoom sound issue.
I was thinking along the same line as LarryB99, that there must be something on the signal processing side causing it - because it usually happened to me when there are large number of participants in the call.
When I tried to reproduce the issue with just two participants, it didn't happen even after an hour. This experiment was immediately following a rather embarrassing Zoom meeting where I gave all attendees a very unpleasant shock to their ears - so I thought I could reproduce it but no.
So maybe it is CPU trying to do some smart signal processing when there are large number of sound input coming through?
Anyway the best candidate for possible 'excessive signal processing' I could find was Noise Suppression in
- Settings -> Sound -> Input | Device Properties -> Device Properties -> Additional Device Properties -> Enhancements -> Noise Suppression.
Every since I unticked the checkbox for Noise Suppression, I haven't had this issue reoccur (YET).
Not sure if it is a permanent solution or I am just fortunate enough not to experience the problem again.
Would love to hear others' experience after trying out this setting?