Andrian Nur Prabawa,
I understand that you would like to remove the Previous devices that you have linked with the Phone Link application on Windows 11.
I have tested this from my end and followed the instructions from the Microsoft article below to successfully remove a device.
Unlink a device in the Phone Link-
https://support.microsoft.com/help/9382f770-b203-7e0e-c81e-18d4053cf4e5
In image 1, you might see two devices that I have linked.

In image 2, you might see the device (Realme c1) removed from the lists.

To remove the device successfully, I have followed the steps below (on both Android and Windows):
- Opened (link to Windows) app downloaded, and selected settings (Gear icon on top right corner**)** on my Android device.
- Select your Account (e-mail address) and click 'sign out'.
- Once completed, go to this link https://account.microsoft.com/devices/android-ios to remove the device from your Microsoft account as well.
- Once done, Reset the Phone Link app from the App settings and restart it again. The device should be removed completely from the lists.
Note: For Samsung devices, the application "Link to Windows" might be pre-installed, you will have to follow the instructions for "If you have the Link to Windows (pre-installed) app on your Android device:" from the above article to successfully remove the device.
But if you no longer have the older device to select Sign out, I think there are currently no available options or instructions to remove the device or if it's no longer active. I suggest submitting your feedback to Microsoft through the feedback hub (press the Windows key + F) to submit your feedback.
Best regards,
Gunasekar N