PC crashing - BSOD Watchdog violation 1x133 (possibly nVidia issue)

Looks like flashing my BIOS didn't do the trick, nor did updating the motherboard firmware.  Still BSOD after 1 min when using both GPUs.  Here is my mini dump if anyone can take a look.

https://drive.google.com/file/d/1u3i1nG71cqZ7lm\_3IocfvKP7oKM72YBP/view?usp=sharing

For now I'm going to try disabling one card and see if it remains stable.  Only other thing I can think to do now is update Windows to 1803.

Recently I have built a 3D rendering station for a client of mine. The machine runs, but crashes about once or twice a day with a BSOD (watchdog violation 1x133. Blue Screen View mentions either nvlddmkm.sys or ntoskrnl.exe as culprit.

Things I have tried so far:

• Clean install Windows 10
• Clean install Windows 7
• DDU removal of graphics drivers
• latest nVidia drivers
• older nVidia drivers
• extensive RAM testing

Specs:

• Asus X99-WS 10G E
• Intel 6850K
• 7x Zotac 1080ti Blower Edition (transformed to water cooling)
• 64GB RAM
• Samsung 960 Pro 512GB (system drive)
• 3x WD Red in RAID 0 (data drive)
• 2x Cooler Master V1200 PSU
• Custom EK watercooling loop
• Obsidian 900D

Speccy Snapshot: http://speccy.piriform.com/results/qchWoHD0m7mueWTEVccxFlD

Any help or insight on this issue would be greatly appreciated, as this issue now causes great delays in my client's workflow. 

I narrowed down the issue to the NVIDIA drivers for Windows 10. It affects specifically the high-end Workstation Motherboards that feature PLX PEX 8747 PCI-E switches from Broadcom, with three or more GTX Pascal GPU cards.

Whether this is by accident or intentional by NVIDIA I don't know, but I'm very frustrated they've known it and haven't acted for several months. The drivers are broken since last July/August, for 9 months already.

An old but stable driver is version 382.53 (supports up to CUDA 8.0), from June / July 2017. You can download it directly from NVIDIA. The issue was first introduced in the newer drivers that followed.

Following lots of pressure, NVIDIA's Tech Support have confirmed they're looking into this and they recently added the issue in the Known Issues section, in the Release Notes of the recently released drivers (May).

So, I hope this information is helpful to everyone with the aforementioned computer configuration. Use driver 382.53 as a workaround and report your problem to NVIDIA's Tech Support, mentioning the model of your WS motherboard and why you need NVIDIA to address this issue in its latest drivers.

this is not what I was hoping to hear.  I have 6 machines built off of the x99 e ws and multiple GPUs.

The problem does persist with less than 3 pascal cards, however.  My system with 2 1080 cards still exhibits the same behavior.  

For now I will have to limit the system to one card and hope that it stays stable.  As I need CUDA 9.1 for Octane render.

Thanks for the follow up, and I'll keep reporting to NVIDIA until we get this fixed.  If you or anyone does find a fix that includes not going back to CUDA 8 please let us know.

After chatting with support the only suggestion they have is to update to 1803 and try these settings in the NV control panel:

Open NVIDIA control panel (Right click on desktop and click NVIDIA control panel) > Go to Manage 3D settings under 3D settings > Select Program settings > Select the app/game from the dropdown menu > if it is not listed in the dropdown menu then click Add then select the respective .exe to the list > And now change these settings from Option 2.

• Power management mode - Prefer maximum performance

• Triple buffering - Off

• Threaded optimisation - ON

• Vertical Sync - off

Apply these settings, refresh the Desktop several times, start the game play and check the performance. "

I'm unable to try this at the moment, but will report back as soon as I get a chance to try this.

this is not what I was hoping to hear.  I have 6 machines built off of the x99 e ws and multiple GPUs.

The problem does persist with less than 3 pascal cards, however.  My system with 2 1080 cards still exhibits the same behavior.  

For now I will have to limit the system to one card and hope that it stays stable.  As I need CUDA 9.1 for Octane render.

Thanks for the follow up, and I'll keep reporting to NVIDIA until we get this fixed.  If you or anyone does find a fix that includes not going back to CUDA 8 please let us know.  

Check if your 2 GPUs are on PCI-E slot 1 and 5. In this configuration, each GPU will be attached to a different PLX PEX8747 PCI-E switch.

If you've alread instelled the GPUs as above and it doesn't work, try Slots 1 and 3, i.e. on the same PCI-E PLX switch.

Lower cost motherboards, without the high-performance PLX switches aren't affected.

The good news is that NVIDIA has acknowledged the issue in its latest driver Release Notes, page 14:

"> Random DPC watchdog violation errors occur when using multiple GPUs on motherboards with PLX chips. [2079538]"

Source: http://us.download.nvidia.com/Windows/397.93/397.93-win10-win8-win7-desktop-release-notes.pdf

Furthermore, NVIDIA Tech Support informed me that they're working on a solution but, unfortunately, they couldn't promise when that will become available.