I have been facing this issue for over one year now and I eventually noticed it might not actually be related to the Game Input. Yes, Game Input reconfigure is always there after a crash/restart but it is also there if you manually restart your computer. In addition it appears that it reconfigures Game Input <after> the reboot, therefore I don't think it is the one causing the crash.
I've tried everything, nothing works. It really looks like we have a few pieces of hardware that are just not compatible with Xbox application?
I never had this issue until I installed Xbox app and Age of Empires IV to play with Gamepass.