Using New Outlook on Windows for professional communication and productivity
I thought so at first too, but I was not able to resolve the problem by upgrading licenses - and in addition, I am not having the problem on my own user accounts where one of them are Exchange Online licenses. So it's not specific to licensing.
In the screenshot you can see 3 accounts (domains removed) are currently connected to the new preview outlook, but one of them is Exchange Online 1.