Please be informed that this is an intended behavior seen by all O365 admins. This information is for Admin password reset as well as for service communications.
You can move away from the page by clicking "cancel", click on the "admin" tile or by navigating back to the admin page via the link here
portal.office.com/.../default.aspx.
Do take note on the potential risk if you are a single admin or you may missed out from any service communication if the information is not updated.
@Mad Dad, please have a look at the information below that could address your concern.
Disable self service password reset in your organization. Your admin will still get the prompt but this address your concern on admin self service password reset for admin who left the org.
Also, if clicking "cancel" keeps you in a loop, let's try the other 2 options i have shared earlier.
msdn.microsoft.com/.../dn194127.aspx
Set-MsolCompanySettings -SelfServePasswordResetEnabled $false