gpg-agent / ssh / ed25519 / agent refused

Back to overview Older post: recap 2023 - updates van OSSO

gpg-agent / ssh / ed25519 / agent refused

  • Written by
    Walter Doekes
  • Published on

After putting in all the work to get ED25519 OpenPGP keys in my Yubikey smart card, I was slightly frustrated that the SSH support "sometimes" didn't work.

I thought I had tested that it worked, but today it didn't.

$ ssh my_server
sign_and_send_pubkey: signing failed for ED25519 "cardno:000612345678" from agent: agent refused operation
walter@my_server: Permission denied (publickey).

That's odd.

Adding some debug to the gpg-agent — debug 1024 in gpg-agent.conf — got me this:

DBG: chan_10 -> SERIALNO
DBG: chan_10 <- S SERIALNO D2760001240103040006123456780000
DBG: chan_10 <- OK
DBG: chan_10 -> GETINFO card_list
DBG: chan_10 <- S SERIALNO D2760001240103040006123456780000
DBG: chan_10 <- OK
DBG: chan_10 -> SERIALNO --demand=D2760001240103040006123456780000
DBG: chan_10 <- S SERIALNO D2760001240103040006123456780000
DBG: chan_10 <- OK
DBG: chan_10 -> GETATTR $AUTHKEYID
DBG: chan_10 <- S $AUTHKEYID OPENPGP.3
DBG: chan_10 <- OK
DBG: chan_10 -> GETATTR SERIALNO
DBG: chan_10 <- S SERIALNO D2760001240103040006123456780000
DBG: chan_10 <- OK
DBG: chan_10 -> READKEY OPENPGP.3
DBG: chan_10 <- [ 44 20 28 31 ...(83 byte(s) skipped) ]
DBG: chan_10 <- OK
DBG: chan_10 -> GETATTR $DISPSERIALNO
DBG: chan_10 <- S $DISPSERIALNO 000612345678
DBG: chan_10 <- OK
ssh request handler for request_identities (11) ready
ssh request handler for sign_request (13) started
DBG: chan_10 -> SERIALNO --demand=D2760001240103040006123456780000
DBG: chan_10 <- S SERIALNO D2760001240103040006123456780000
DBG: chan_10 <- OK
detected card with S/N D2760001240103040006123456780000
DBG: chan_10 -> SETDATA 3021...>
DBG: chan_10 <- OK
DBG: chan_10 -> PKAUTH D459...
scdaemon[898750]: app_auth failed: Invalid value
DBG: chan_10 <- ERR 100663351 Invalid value <SCD>
smartcard signing failed: Invalid value
ssh sign request failed: Invalid value <SCD>

When I tried a different server, things did work:

...
DBG: chan_10 -> SERIALNO --demand=D2760001240103040006114652250000
DBG: chan_10 <- S SERIALNO D2760001240103040006114652250000
DBG: chan_10 <- OK
detected card with S/N D2760001240103040006114652250000
DBG: chan_10 -> SETDATA 3021...>
DBG: chan_10 <- OK
DBG: chan_10 -> PKAUTH D459...
DBG: chan_10 <- [ 44 20 1d 30 ...(52 byte(s) skipped) ]
DBG: chan_10 <- OK

No explanations to be found here. But the different server did provide the clue that the OpenSSH version was involved.

I tried a few different OpenSSH versions:

trusty - 1:6.6p1-2ubuntu2.13 - Accepted publickey ... ED25519 de:14:...
bionic - 1:7.6p1-4ubuntu0.7  - Accepted publickey ... ED25519 SHA256:lKE...
focal  - 1:8.2p1-4ubuntu0.9  - Accepted publickey ... ED25519 SHA256:lKE...
jammy  - 1:8.9p1-3ubuntu0.6  - <failure>

And in fact (tested with a bit of SSH key forwarding) using an older client against the newer server also worked.

OpenSSH client side debugging started turning up interesting differences after the third -v:

-- OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022 --
-- debug1: Reading configuration data /home/walter/.ssh/config --
-- debug3: kex names ok: [diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1] --
...
1. debug1: Connecting to old_server port 22.
 2. debug1: Connecting to my_server port 22.
  3. debug1: Connecting to rsa_server port 22.
   4. debug1: Connecting to hostkey_server port 22.
-- debug3: set_sock_tos: set socket 3 IP_TOS 0x10 --
-- debug1: Connection established. --
...
-- debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 --
1. debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.9
 2. debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
  3. debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
   4. debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
1. debug1: compat_banner: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.9 pat OpenSSH* compat 0x04000000
 2. debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 pat OpenSSH* compat 0x04000000
  3. debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 pat OpenSSH* compat 0x04000000
   4. debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 pat OpenSSH* compat 0x04000000

Server 1 was an older server, server 2, 3 and 4 are the new server, with: (2) failure, (3) an RSA key, and (4) a forced ssh-ed25519 host key algorithm.

The log continues..

-- debug2: fd 3 setting O_NONBLOCK --
1. debug1: Authenticating to old_server:22 as 'walter'
 2. debug1: Authenticating to my_server:22 as 'walter'
  3. debug1: Authenticating to rsa_server:22 as 'walter'
   4. debug1: Authenticating to hostkey_server:22 as 'walter'
-- debug3: send packet: type 20 --
-- debug1: SSH2_MSG_KEXINIT sent --
-- debug3: receive packet: type 20 --
-- debug1: SSH2_MSG_KEXINIT received --
-- debug2: local client KEXINIT proposal --
1. debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
 2. debug2: KEX algorithms: curve25519-sha256,ext-info-c,kex-strict-c-v00@openssh.com
  3. debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
   4. debug2: KEX algorithms: curve25519-sha256,ext-info-c,kex-strict-c-v00@openssh.com
1. debug2: host key algorithms: ecdsa-sha2-nistp256
 2. debug2: host key algorithms: ecdsa-sha2-nistp256
  3. debug2: host key algorithms: ecdsa-sha2-nistp256
   4. debug2: host key algorithms: ssh-ed25519

The difference here is me explicitly setting the HostKeyAlgorithms=ssh-ed25519 for server 4.

Some more uninteresting logs..

-- debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com --
-- debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com --
-- debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 --
-- debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 --
-- debug2: compression ctos: none,zlib@openssh.com,zlib --
-- debug2: compression stoc: none,zlib@openssh.com,zlib --
-- debug2: languages ctos: --
-- debug2: languages stoc: --
-- debug2: first_kex_follows 0 --
-- debug2: reserved 0 --
-- debug2: peer server KEXINIT proposal --
1. debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
 2. debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com
  3. debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com
   4. debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com
1. debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
 2. debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
  3. debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
   4. debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
-- debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com --
-- debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com --
-- debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 --
-- debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 --
-- debug2: compression ctos: none,zlib@openssh.com --
-- debug2: compression stoc: none,zlib@openssh.com --
-- debug2: languages ctos: --
-- debug2: languages stoc: --
-- debug2: first_kex_follows 0 --
-- debug2: reserved 0 --
1.
 2. debug3: kex_choose_conf: will use strict KEX ordering
  3. debug3: kex_choose_conf: will use strict KEX ordering
   4. debug3: kex_choose_conf: will use strict KEX ordering

This could be something. will use strict KEX ordering is only seen on the newer servers.

-- debug1: kex: algorithm: curve25519-sha256 --
1. debug1: kex: host key algorithm: ecdsa-sha2-nistp256
 2. debug1: kex: host key algorithm: ecdsa-sha2-nistp256
  3. debug1: kex: host key algorithm: ecdsa-sha2-nistp256
   4. debug1: kex: host key algorithm: ssh-ed25519
-- debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none --
-- debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none --
-- debug3: send packet: type 30 --
-- debug1: expecting SSH2_MSG_KEX_ECDH_REPLY --
-- debug3: receive packet: type 31 --
-- debug1: SSH2_MSG_KEX_ECDH_REPLY received --
1. debug1: Server host key: ecdsa-sha2-nistp256 SHA256:abc...
 2. debug1: Server host key: ecdsa-sha2-nistp256 SHA256:abc...
  3. debug1: Server host key: ecdsa-sha2-nistp256 SHA256:abc...
   4. debug1: Server host key: ssh-ed25519 SHA256:def...
1. debug3: record_hostkey: found key type ECDSA in file /home/walter/.ssh/known_hosts:1234
 2. debug3: record_hostkey: found key type ECDSA in file /home/walter/.ssh/known_hosts:1234
  3. debug3: record_hostkey: found key type ECDSA in file /home/walter/.ssh/known_hosts:1234
   4. debug3: record_hostkey: found key type ED25519 in file /home/walter/.ssh/known_hosts:1234
1. debug3: load_hostkeys_file: loaded 1 keys from old_server
 2. debug3: load_hostkeys_file: loaded 1 keys from my_server
  3. debug3: load_hostkeys_file: loaded 1 keys from rsa_server
   4. debug3: load_hostkeys_file: loaded 1 keys from hostkey_server
1. debug1: Host 'old_server' is known and matches the ECDSA host key.
 2. debug1: Host 'my_server' is known and matches the ECDSA host key.
  3. debug1: Host 'rsa_server' is known and matches the ECDSA host key.
   4. debug1: Host 'hostkey_server' is known and matches the ED25519 host key.
-- debug1: Found key in /home/walter/.ssh/known_hosts:1235
-- debug3: send packet: type 21 --
1.
 2. debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
  3. debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
   4. debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
-- debug2: ssh_set_newkeys: mode 1 --
-- debug1: rekey out after 134217728 blocks --
-- debug1: SSH2_MSG_NEWKEYS sent --
-- debug1: expecting SSH2_MSG_NEWKEYS --
-- debug3: receive packet: type 21 --
1.
 2. debug1: ssh_packet_read_poll2: resetting read seqnr 3
  3. debug1: ssh_packet_read_poll2: resetting read seqnr 3
   4. debug1: ssh_packet_read_poll2: resetting read seqnr 3

The ssh_packet_send2_wrapped and ssh_packet_read_poll2 is also new.

-- debug1: SSH2_MSG_NEWKEYS received --
-- debug2: ssh_set_newkeys: mode 0 --
-- debug1: rekey in after 134217728 blocks --
-- debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused operation --
-- debug1: get_agent_identities: agent returned 3 keys --
1. debug1: Will attempt key: cardno:000612345678 ED25519 SHA256:lKE... agent
 2. debug1: Will attempt key: cardno:000612345678 ED25519 SHA256:lKE... agent
  3. debug1: Will attempt key: cardno:000611111111 RSA SHA256:vYJ... agent
   4. debug1: Will attempt key: cardno:000612345678 ED25519 SHA256:lKE... agent

The RSA key is obviously different...

-- debug2: pubkey_prepare: done --
-- debug3: send packet: type 5 --
-- debug3: receive packet: type 7 --
-- debug1: SSH2_MSG_EXT_INFO received --
1. debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
 2. debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
  3. debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
   4. debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>

Extra server signature algorithms shouldn't matter.

1.
 2. debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
  3. debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
   4. debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>

kex_input_ext_info: publickey-hostbound@openssh.com is also new...

-- debug3: receive packet: type 6 --
-- debug2: service_accept: ssh-userauth --
-- debug1: SSH2_MSG_SERVICE_ACCEPT received --
-- debug3: send packet: type 50 --
-- debug3: receive packet: type 51 --
-- debug1: Authentications that can continue: publickey --
-- debug3: start over, passed a different list publickey --
-- debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password --
-- debug3: authmethod_lookup publickey --
-- debug3: remaining preferred: keyboard-interactive,password --
-- debug3: authmethod_is_enabled publickey --
-- debug1: Next authentication method: publickey --
1. debug1: Offering public key: cardno:000612345678 ED25519 SHA256:lKE... agent
 2. debug1: Offering public key: cardno:000612345678 ED25519 SHA256:lKE... agent
  3. debug1: Offering public key: cardno:000604261342 RSA SHA256:vYJ... agent
   4. debug1: Offering public key: cardno:000612345678 ED25519 SHA256:lKE... agent
-- debug3: send packet: type 50 --
-- debug2: we sent a publickey packet, wait for reply --
-- debug3: receive packet: type 60 --
1. debug1: Server accepts key: cardno:000612345678 ED25519 SHA256:lKE... agent
 2. debug1: Server accepts key: cardno:000612345678 ED25519 SHA256:lKE... agent
  3. debug1: Server accepts key: cardno:000604261342 RSA SHA256:vYJ... agent
   4. debug1: Server accepts key: cardno:000612345678 ED25519 SHA256:lKE... agent
1. debug3: sign_and_send_pubkey: using publickey with ED25519 SHA256:lKE...
 2. debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com with ED25519 SHA256:lKE...
  3. debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com with RSA SHA256:vYJ...
   4. debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com with ED25519 SHA256:lKE...
1. debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:lKE...
 2. debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:lKE...
  3. debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:vYJ...
   4. debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:lKE...
1.
 2. sign_and_send_pubkey: signing failed for ED25519 "cardno:000612345678" from agent: agent refused operation
  3.
   4.
...
1. Authenticated to old_server:22 using "publickey".
 2. debug1: No more authentication methods to try.
  3. Authenticated to walter-dev.osso.nl ([217.21.199.35]:22) using "publickey".
   4. Authenticated to hostkey_server:22 using "publickey".

Here we saw that:

  • Connecting to the old server succeeded with the ED25519 key.
  • Connecting to the new server failed with the ED25519 key.
  • Connecting to the new server succeeded with an RSA key.
  • Connecting to the new server succeeded with HostKeyAlgorithms set to ED25519.

The culprit looks like publickey-hostbound-v00@openssh.com. According to SSH agent restriction documentation, this option passes the destination to the client, where it should be validated and approved.

If either the client or the server doesn't support this extension, things work just fine. But if they both do, then the gpg-agent fails to properly sign the public key negotiation data.

Destination restrictions in ssh-agent strongly depend on the agent being able to parse the data being signed, and the contents having all the information needed to compare against the restrictions listed for a given key. SSH user authentication requests have a format that meets these requirements, but other uses of the agent protocol are not likely to.

For now, my guess is that indeed gpg-agent does not support the extra hostname from the extension. The workaround is either:

  • Set -oHostKeyAlgorithms=ssh-ed25519 or
  • set -oPubkeyAuthentication=unbound on the client.

I don't know why the first option works, but it does. The second one should disable the publickey-hostbound-v00@openssh.com extension, making negotiation fall back to the original behaviour. So, if you're affected by this, it may make sense to add that to your ~/.ssh/config.

The relevant ticket for GnuPG is OpenSSH 8.9, 9.0, and 9.1 can't authenticate with gpg-agent and usb token.

Back to overview Older post: recap 2023 - updates van OSSO

Location

Helperpark 282A

9723ZA Groningen

Phone: +31 (0)50 210 3520

E-mail: info@osso.nl

Links

OSSO B.V.

  • IBAN NL45RABO0105881341EUR
  • KvK 02087026
  • BTW NL820103500B01
Copyright © 2024 OSSO B.V.