I replied to this thread on June 26, 2024, to say I was having the same issue.
I just wanted to post an updated reply to see if maybe I could provide some insight into the issue with other things I've experienced, and to maybe offer some help for others in the same boat.
To start, I have been taking screenshots of my category list any time I make a change. This will be important in a moment.
Eventually I had given up with respect to this issue and decided to just work around it. In taking screenshots, I figured, I would just remake the categories and work from there. In my world, I use categories to sort current tasks, and once completed, the category is no longer important to me. So if completed items lost their category, it wasn't a big deal.
I did notice, however, that for items that had "lost" a category (and by this I mean the categories I had made that had disappeared), the item still retained something of a "ghost" of a category. It was basically just a blank/white category with the name of the old one, but it no longer existed in my category list. Here is what that looked like:

This is so common for my team that we just call them "ghost" categories colloquially... LOL
Here is what I noticed, based on the information above...
Because I was taking screenshots of my categories, and re-adding them in Outlook, I began to notice that the "ghost" categories began to regain their colour. And what's more, they were "shifted" over to the new categories that I re-made in Outlook! It's like the system picked up on the same colour and title and just... switched it back.
This isn't a fix to this problem, as I am in a position where losing my categories is just annoying rather than consequential to my work flow, as is the case for others.
I just wanted to provide my insight for those that may find it beneficial.
Cheers,
Christine