Method 1: is not available in Windows 7 Home Premium , only in the Pro version. And I could not find any registry value that would match the gpedit.msc corresponding option.
Method 2: works on my system. Without "ExtendedUIHoverTime" registry entry:
With "ExtendedUIHoverTime" registry entry:
So it prevents (for the number of milliseconds specified) the pop up of the taskbar thumbnail, but does not remove the additional (!) pop up (text box black on white) I guess known as "taskbar icon name".
After hours of investigation on the Internet, the only efficient solution I could find is indeed this light program that also brings lots of additional functionalities. It works also on Windows 8.1, does not change anything to the regular registry (quoting FAQ -> "The only registry keys the tweaker modifies are it’s own settings. There are no registry keys for the options it provides. The tweaker does that by injecting a DLL to explorer, hooking/subclassing/some other methods of the dark side.")
http://rammichael.com/7-taskbar-tweaker
I now use it, no more registry manual manipulations ("ExtendedUIHoverTime" registry entry removed), I love it.