For all these wasted man-hours, microsoft must be awarded the "World's largest open source Employer".
Here is the fix for a HP Pavilion dm4 1041tx with a AMD Radeon HD 5000 series graphics card.
It is the graphics card driver issues which was stopping the upgrade process midway.
Everything including compatibility test passed and went through till restart and than screen blacked out.
- Uninstall the graphics card from the device manager and don't restart the system till the upgrade process restarts it, else windows will again re-install it.
- Start the upgrade and start praying.
- After the installation if you see the Microsoft driver for the graphics card you can update it from the device manager, will work fine and will automatically update to the latest AMD driver. No issues after the installation.
Mission accomplished after 6 installation and 3 downloads, got the start button which points to the windows 8 home page.