Managing sound and video settings during Teams meetings and calls for optimal communication
This exactly what I've been doing. You cannot start a meeting or join from the calendar.
If someone starts a meeting, and you get the "meeting has been started" notification, you can join from the notification and click cancel on the needs permission warning.