So I had this issue too,
The fix of taking the server worked for connecting offline but then my GF couldn't connect from her place. So I had to set it back to "true" and try another fix.
I restarted the Xbox (as opposed to turning it off and on as I use the instant on setting so powering off doesn't actually work as a way of resetting it as it never really turns off)
I'm not sure if that was actually what fixed it or not as the following also changed
I opened Xbox companion on the PC running the server (not sure how that would have changed anything though)
I tinkered with my Xbox privacy settings (not sure how that would have made a difference either)
I drank a can of coke (I am 100% sure that wasn't what fixed it but include it so as to be thorough) :)
If it happens again il come back and add further info to the post if not, consider it as having Perma fixed the issue for me.