Well, for what I have been able to figure out is the number value is for grahpics/video driver use, port assignment. No way to change this that I know of.
You can make any monitor/TV primary display in monitor resolution support on your system.
I have (3) video ports, HDMI, DVI, and VGA.
My TV is attached to HDMI, which is assigned number 1.
My main monitor (my primary monitor which I set when I set up my system) is attached to DVI, which is assigned number 2.
My second monitor is attached to VGA which is assigned to number 3.
If I remove the HDMI cable, my DVI becomes number 1, and my VGA becomes number 2, If I also remove the DVI cable, my VGA becomes number 1.
It does not matter which ports I have attach too, HDMI on my PC will always show number 1 when attached. DVI will only show number 1 when there is no HDMI attached, VGA will only show number 1 when there is no HDMI or DVI attached.
Port numbering does not hinder monitor or TV use.
And yes there is things that are annoying, if you have to do a fresh OS install you will need to reassign primary monitor, (a wireless mouse is handy to have).