FsLogix - Unclean logoff causing locked files until server reboot

Hi All, we battled variations of this and took us a while to pin ours down because we believed it to be centered on FSLogix. We experienced many of the symptoms laid out here and it was very intermittent and unpredictable.

Environment:

• Citrix Virtual Apps (Published Desktops / Shared Session Hosts) Version 1912.
• Windows Server 2019
• Tried with two FSLogix versions: 2.9.7117.27413 and 2.9.7979.62170

Our Experience:

• Users VHDX would get locked.
• The Citrix Session would be logged off most of the time but not always.
• Regardless of ^^ the session would show active from VHDX storage using "Net Session" sourcing from the VDA they were logged in from.
• They would get temporary profiles due to profile lock.
• Sometimes while in the middle of working, they would lock up.
• Rebooting the non-persistent VDA seemed to be the only way to truly clear it along with manually closing the open profile on the VHDX Profile share.
• This was random across 20+ VDA servers.

Our Troubleshooting:

• We would watch the session timers on the shared storage using Net Session and keep an eye on ones that would have high counters.
• We would review FSLogix logs on users who were getting temp profiles.
• We would review event logs.
• Lots of reading which brought me here.

Summary:

The real pointer came when we were able to remote to a user who was actively having a session lock while still working. Since the VDA's were being rebooted daily to help get through the issue, we would miss some log entries.

Ultimately, that user was creating a ton of Event ID 107 Citrix.Authentication.IdentityAssertion in the app log.

That led us to article: https://support.citrix.com/article/CTX255423

That was our issue, the Kerberos Renew Time was set to 10 hours. Many of our users due to various reasons have been working longer hours and we are requried to have very high session limit timers for idle but active then disconnected. If that user had never disconnected and reconnected during the day or even had their 20 minute idle screen come up, their ticket would expire and not renew causing all access to network resources to come to a halt. Primary indicator was loss of access to the VHDX. The session wouldn't tear down correctly due to this. We discovered this in our GPO's and modified that at the domain level and we have been error free since.

Another helper was a log aggregator that allowed us to see TGT requests and if the user only had the initial request and no subsequent requests from that machine, you could begin to predict the failure.

Check your Kerberos settings in your Default Domain Policy or whatever policy is applied at the domain level that manages Kerberos ticket times.

I am sure this won't solve it for all but based on what I read here, it might be contributing.

It was very frustrating to figure out. Looking back, since this has been this way since before I started working here, it explained some other strange access issues that got written off after reboots.

Hope this helps!

As a temporary solution, I use this script when a user logs out using group policies.

$SID = ([System.Security.Principal.WindowsIdentity]::GetCurrent()).User.Value

Set-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileService\References\$SID" -Name RefCount -Value ([byte[]](0x00,0x00,0x00,0x00))

I have two different Citrix environments. Both are pretty much identical. Same FSLogix version, same GPOs, same patch level. Almost same everything except for the language and some apps.

Both run on Citrix PVS, Windows Server 2019 with XenApp 1912 LTSR CU3, FSLogix 2105. Both vdisks are monthly rebuilt from scratch, fully scripted.

On one env I have the issue:

[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)
[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)
[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)
[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)
[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)

On the other one I don't:

[15:17:25.851][tid:000017d4.000068e0][INFO] Successfully removed C:\Users\local_x

Both run Windows Defender with the EXACT same settings (they even share the same GPO).

I checked BrandonClark's command fltmc filters and they give the same output on both envs.

So why does it not work on env A and why does it work on env B?

Luckily, I don't have the issue where the users can't re-logon and the vdisk is locked. The vdisk actually is not locked any longer after logoff and people can just relogon. If they happen to hit the same server it will usually create a different folder like local_username_1 or something like that.

Hey all, just so you're aware. I troubleshot this error along with Microsoft Support for 48 hours straight. In that time I got to learn FsLogix pretty well. In OUR case, it was Symantec DLP, the Data Loss Protection software. The issue was our DLP software was OUT OF DATE. DLP (from Symantec/Broadcom) uses a FILTER DRIVER. This out of date filter driver was causing ONEDRIVE to hang, which in turn caused the Office Container to hang because of a stuck handle. Once we ugpraded our Symnatec DLP from 15.5 to 15.7, the issue went away. This was confirm with an open ticket to Broadcom as well.

Has anyone tried to write a script for a temporary solution ? There will be a fix in November. https://learn.microsoft.com/en-us/answers/questions/229309/fslogix-unclean-logoff-causing-locked-files-until.html?childToView=568300#answer-568300