Hi All,
I found the root cause of this issue and fixed the RDS collections for 3 clients of my company.
The setup is specific, so I doubt it will help all of you, but maybe it will make life easier for some of you.
Unfortunately, I can’t add screenshots to it (got an error when uploading it☹ ), all described instead.
As the history of the issue is long I will chunk it into sections:
1 Initial issue
2 Investigation with details
3 First Client with the issue and applied solutions
4 Second Client with the issue and applied solutions
5 Third Client, under investigation 6 Summary of the investigation
7 Solution 8 ConclusionIf you want just the solution please scroll to pt 7 and 8.
1. The issue :
On RDS collection, when users try to open Outlook (in the bigger picture all O365 apps) they get:
That’s from one of the first affected users:
>>>>>>>>>>
I got Error “Something Went Wrong [1001]
After clicking close on the above error, the dialogue box appears :
Cannot start Microsoft Outlook. Cannot open the Outlook window …
After clicking OK the screen returns to just the desktop.
>>>>>>>>
In the beginning, users were able to close errors and open Outlook, but after some time Outlook did not start after Error.
The issue spread to all O365 applications.
Environment setup that is affected by this issue :
RDSs host on WS 2019 and O365
2. The whole my investigation story from the beginning:
Last Year I got the task to upgrade OSs on all servers, as a part of upgrading all clients to the newest / supported versions of the OS. I Started September/October, as it is the upgrade of the Live system, it depended on available time, usually OOH.
I started with "First Client".
I upgraded most of the servers, I left RDS farm for the end as it caused many issues with users.
All upgrade was in place upgrade from 2012 > 2016 >2019
I upgraded RDS first (2,3 and 4, RDS01 left as old working backup)
That was the time when the issue started happening, the first report from users 15/11/2023.
In the beginning, users got the error, but they were able to close it and eventually, Outlook opened.
After a few days more people reported the same, but with worse symptoms, like Outlook couldn't start at all. Lost connection to Exchange and couldn't reconnect. Basically, it paralyses users' work.
I was lucky to still have as a backup RDS01 that wasn't upgraded, and it uses user profiles v2 and upgraded RDS use profiles V6.
That allows to separation of affected users.
The only solution to allow users to work again is to force the system to recreate the user profile (roaming profile stored on a separate server ) and clean the registry (related to the affected user) on all RDSs in the collection.
I did the investigation, I searched everything I could and I found an official MS Support article :
https://support.microsoft.com/en-us/office/error-something-went-wrong-1001-signing-in-to-microsoft-365-desktop-applications-6f63238d-d83c-437c-a929-de72fe819793
where the proposed solution, not working or not applicable to our environment set-up.
I found MS forum where admins around the world complaining about this issue since 7 July 2023
https://answers.microsoft.com/en-us/outlook_com/forum/all/outlook-365-app-error-1001-on-rds-environment/e6ae8598-81a0-4859-846a-d9509b246bf4?page=23
I tried all the proposed and applicable changes I found there over the last 3 months. Non is stable.
Summary of proposed changes from the forum :
>>>>>>>>>>>>>>>>>>>>>
Hi everyone!
I've been following this thread for a while and so far, our client's environment has been stable since end of November. Here are the things to note so far:
Deploy these registry keys:
[HKLM\SOFTWARE\Policies\Microsoft\Windows\WorkplaceJoin]
"BlockAADWorkplaceJoin"=dword:00000001
[HKLM\SOFTWARE\Policies\Microsoft\Windows\WorkplaceJoin]
"autoWorkplaceJoin"=dword:00000000"
Delete these registry keys, if possible, Microsoft doesn't recommend using these anymore:
Computer\HKEY_CURRENT_USER\SOFTWARE\Microsoft\Office\16.0\Common\Identity
EnableADAL
DisableAADWAM
DisableADALatopWAMOverride
If you're using FSLogix, make sure to enable this policy through main GPO:
RoamIdentity
Lastly, if users did register the RDS to Azure AD, might have to follow these steps:
>>>>>>>>>>>>>>>>>>
3. First Client Setup looks like now:
Desktop -> RDS01 on WS2012 > OLD stable RDS
TestCollection -> RDS02,03 and 04 > last applied solution :
I set up GPO with this change :
[HKLM\SOFTWARE\Policies\Microsoft\Windows\WorkplaceJoin]
"BlockAADWorkplaceJoin"=dword:00000001
[HKLM\SOFTWARE\Policies\Microsoft\Windows\WorkplaceJoin]
"autoWorkplaceJoin"=dword:00000000"
Additionally, I reverted upgrades of O365 to 2023 March 20 Version 2302 (Build 16130.20332) as I found that this may cause it as well.
I checked in Entra registered devices and removed any RDS servers.
This was stable for 5 days and it starts again for random users.
I uninstalled o365 (32bit ) and installed a fresh o365 64-bit version.
Revert upgrades to March last Year.
Applied above GPO, 3 users are testing this last 7 days, but yesterday I got first user affected on this setup as well
My last resolution is to build RDS from scratch, however, this involves 3rd party to set up the application, so for now is a last resort, but still, there is no guarantee it will work.
In the end, there was no other way than to rebuild the RDS host from scratch and build a new collection based on it
After 6 months, the solution seems to be stable, but as a precaution, I saved the original new RDS built as a template in case the issue comes back to replace the whole farm quickly.
4. Second client affected :
The client was built on WS 2019 a few years ago.
The same issue started 3 months ago with one user, after a few days there were a lot of affected users.
I applied this GPO WorkplaceJoin to the RDS collection and I believe that stopped or at least slowed down the issue.
Setup :
Desktop > RDS01,2,3 and 04, users using it on BAU > Applied GPO
That settled down the situation a bit
Desktop - x64 Test > RDS05 > Applied GPO
Test server with RDS with O365 64 bit, we doubled resources temporarily as we agreed with the Client to route any affected users to this server. As it should be stable most of the time.
I asked the Client to check Entra for registered devices and remove any servers from there.
I applied the newest updates to the OS and O365, but the issue persisted.
Eventually, I had to build a new RDS from scratch the same way I did with the First affected Client.
5 Third Client,
Client has 2 RDS farms:
Old > access internally on WS 2012 R2 with O365 32 bit , this farm was scheduled for upgrade to WS 2019, the project started and the first RDS was tested.
New > accessed externally, built on WS 2019 with O365 32-bit a few years ago. This setup was ok until 27/09 when the first user reported an error when trying to log in to Excel.
Error “Something Went Wrong [1001]
I applied to GPO, and the registry has been cleaned but it doesn’t work.
I cloned one of the RDSs for testing, cleaned the registry and reinstalled O365 with the 64-bit version.
This didn't work either.
I get different errors but the outcome is the same None of the o365 apps is working.
Error “Something Went Wrong [5fcl8]
I tried everything I could to make the existing RDS work again.
The client is extremely unhappy as the pick season is just about to start and they are already busy.
Building and testing new RDS is not something that the client can afford over pick season.
The issue dramatically complicates the company workflow.
6 Summary of the investigation
The issue is related to the O365 itself and WS 2019.
The issue happened randomly on random users, and there is the minimum pattern I noticed
What I noticed, is when the user is logged first time to the collection after the profile reset, it works ok sometimes.
As soon as over the next login RDG routes the user to a different RDS server than the first one, the issue kicks in, but it's not the rule.
This affects 3 Clients but as you can see it can affect any client with a similar setup.
7 Solution
I have logged the ticket with MS Support.
They advised me with a few fixes, but none worked.
The ticket has been passed to the Escalation Engineer.
We had a remote session where I explained the whole situation.
The engineer explained that the error I got is generic and the root may differ, depending on the setup.
He advised as well, that the WAP plugin is used by O365 to authenticate. And it is highly possible it is not working properly. He asked me to follow the guide to fix WAM.
https://learn.microsoft.com/en-us/microsoft-365/troubleshoot/authentication/cannot-sign-in-microsoft-365-desktop-apps
We went through and got an error when I tried to run PS script to reinstall WAM.
Add-AppxPackage -Register "$env:windir\SystemApps\Microsoft.AAD.BrokerPlugin_cw5n1h2txyewy\Appxmanifest.xml" -DisableDevelopmentMode -ForceApplicationShutdown
Add-AppxPackage -Register "$env:windir\SystemApps\Microsoft.Windows.CloudExperienceHost_cw5n1h2txyewy\Appxmanifest.xml" -DisableDevelopmentMode -ForceApplicationShutdown
But I got error that this application is blocked by appLocker
As the error said the app is blocked by AppLocker. The client RDS servers are heavily locked by GPO.
I spotted this guide earlier, but when I hit the error I thought it was a dead end.
I asked for another meeting the next day, in the meantime, I cloned another RDS server to isolate it from Domain and GPO added to the MGMT network.
I tested the PS script again, same results.
I checked the local GPO, and there was nothing set up, as intended (Only Domain GPO should be applied to the server).
I searched a bit and found out that local GPO app locker policies are stored in files in C:\Windows\System32\AppLocker\
I checked there and found app locker files.
AppCache.dat 8KB
Apps.AppLocker 1KB
Dll.AppLocker 1KB
Exe.AppLocker 64KB
Msi.AppLocker 1KB
Script.AppLocker 1KB
I removed all files. I double-checked later, and the policy that blocked PS script was Exe.AppLocker.
I checked what was inside and it looked like it is a partially readable mix of actual AppLocker Policies from GPO and some trash.
Anyway, I removed the files, and I ran PS script again :
Add-AppxPackage -Register "$env:windir\SystemApps\Microsoft.AAD.BrokerPlugin_cw5n1h2txyewy\Appxmanifest.xml" -DisableDevelopmentMode -ForceApplicationShutdown
Add-AppxPackage -Register "$env:windir\SystemApps\Microsoft.Windows.CloudExperienceHost_cw5n1h2txyewy\Appxmanifest.xml" -DisableDevelopmentMode -ForceApplicationShutdown
It worked straight away.
I checked O365, all apps allow login with Client email, I tested as well with company email.
All start to work again.
8 Conclusion
I'm not 100% sure but I think something in the past, like Windows Updates had a bug that messed up local AppLocker files and the system for some reason is reading and applying it. Even more, it is blocking MS apps/plugins.
Files were removed from all RDS hosts, additionally, I applied GPO :
[HKLM\SOFTWARE\Policies\Microsoft\Windows\WorkplaceJoin]
"BlockAADWorkplaceJoin"=dword:00000001
[HKLM\SOFTWARE\Policies\Microsoft\Windows\WorkplaceJoin]
"autoWorkplaceJoin"=dword:00000000"
That is now working ok, no issues have been reported.
Thanks
Greg.