If you are trying to boot to the recovery usb flash drive using the 2 button power + volume down power up that may be your problem. That procedure does not work on my sp4 i7 and may not work on any sp4s.
Method to boot to Recovery usb
Make sure your sp4 is shutdown, insert your usb flash drive, then hold down the Volume UP button and tap the power button momentarily, as soon as the SURFACE logo appears release the Volume UP button. This will bring you to the UEFI configuration screens. Go to the Boot configuration screen and make sure the "Enable boot from usb devices" is checked and that USB Storage is selected(the box will be blackened if it is selected). To boot the usb swipe your finger from the right to left on the "USB Storage" line. Answer the following message to boot from that device and your sp4 should boot to recovery drive. Your usb stick should be formatted as fat32 and the recovery image should be unzipped in its entirety to the usb stick. Rufus is not necessary if you formatted your flash drive on the sp4.
Note that I have used both 32gb and 16gb flash drives successfully. You do not have to disable Secure Boot.