Using Classic Outlook on Windows for personal email, calendar, and contact management
This functionality seems to have been added now in the new Outlook - when you hit send on the modified invite a pop-up window asks if you want to send to all attendees or just to the newly added.
I also endorse this works. I was so scared of pressing the send button in case it would not work that I thought I may as well share it.
To modify the event I opened (new) outlook -> opened the calendar tab on the left -> searched for my event and right-clicked on it -> (hit "all the event in the series") -> added the participant -> clicked on send at the top left corner. A pop-up appeared to only notify the new or removed attendees.
The attendee now appears in the meeting chat and the participants list, unlike when forwarding the meeting like others have suggested. This way we could also delete participants and not notify everyone.