When Contacts to Multiple Accounts is enabled, a contact must always be associated with a primary account, even if the account field is not required on the page layout. The primary account determines how the contact is shared and how roll-up summary fields calculate values from the contact records. To remove the primary account from a contact, the contact must be deleted or merged with another contact. References: https://help.
salesforce.com/s/articleView?id=sf.relating_contacts_to_multiple_accounts.htm&type=5