Explanation
You need to set the Sharing settings to 'Existing Guests'. This setting allows sharing only with guests who are already in your directory. These guests may exist in your directory because they previously accepted sharing invitations or because they were manually added.
Reference:
https://docs.microsoft.com/en-us/sharepoint/turn-external-sharing-on-or-off