If you get the following error:
To make it so you can delete the Recovery Services Vault. Open the Storage Accounts under “Backup Infrastructure” by navigating to, Recovery Vault >- Backup Infrastructure -> Storage Accounts
Then hit the three dots & select delete. Now you will be able to delete the vault that you were trying to delete.