IdentitiesOnly
Specifies that ssh(1) should only use the authentication identity
files configured in the ssh_config files, even if ssh-agent(1) or
a PKCS11Provider offers more identities. The argument to this
keyword must be “yes” or “no”. This option is intended for situ‐
ations where ssh-agent offers many different identities. The
default is “no”.
You need IdentitiesOnly=yes; from ssh_config(1):
IdentitiesOnly
Specifies that ssh(1) should only use the authentication identity
files configured in the ssh_config files, even if ssh-agent(1) or
a PKCS11Provider offers more identities. The argument to this
keyword must be “yes” or “no”. This option is intended for situ‐
ations where ssh-agent offers many different identities. The
default is “no”.