Troubleshoot access issues

This page provides troubleshooting tips for Bare Metal Solution access issues.

Check if your question or problem has already been addressed on the Known issues and limitations page.

SSH client unable to connect

If your SSH client fails to connect to a server, you might see one of the following errors:

connection timeout or connection refused: The SSH client can't connect.
Permission denied (publickey): The SSH client fails to authenticate.

To diagnose failed SSH connections, follow these steps:

Test connectivity.

Ensure that the host is reachable and that the SSH port (22) is open using ping, traceroute, and nc commands.
```
ping SERVER_NAME
```
```
traceroute SERVER_NAME
```
```
echo "" | nc SERVER_NAME 22
```
If this does not work, then the issue might be in the networking layer and not with the SSH.

Examine the client side debug output.

Turn on the verbosity of the SSH protocol.

ssh -v SERVER_NAME -i ~/.ssh/id_ecdsa

The command prints the debug output that shows the key events of the client side SSH protocol.

The following example output shows that the client sent its key, but the server rejected it. The server asked the authentication to continue with another public key, but the client has no additional keys to offer.

.. .. ..
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:V9cRYdqcAJv+RPfN+oofNTVdUxs6VlocP4uMWOxeGKI
debug1: Host 'bms-server' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering ECDSA public key: /root/.ssh/id_ecdsa
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

If the SSH verbose output does not clearly indicate the cause of the error message, run the strace command:
```
strace ssh SERVER_NAME -i ~/.ssh/id_ecdsa > strace-ssh.txt 2>&1
```
Examine the strace output for errors related to the core problem.

In some cases, the client side debug output is not enough to identify the problem. You might also need to run server side tracing to find the error. Until you can't SSH into your server, use the interactive serial console to perform the next steps.

Examine the server side debug output.
1. Find the current SSH settings.
```
grep -v "^#" /etc/ssh/sshd_config | grep -v "^$"
```
2. To obtain detailed information about SSH, set the following parameters in /etc/ssh/sshd_config file to obtain detailed information.
```
SyslogFacility AUTH
LogLevel DEBUG
```
3. To apply the changes, restart the service.
```
service sshd restart
```
4. At the client side, run the ssh command and extract the sshd messages from the logs:
```
grep sshd /var/log/messages
```
  Or, you can export all relevant messages for the time period using the following command:
```
journalctl -u sshd -S "START_TIME" -U "END_TIME" --utc
```
  Replace the following:
  - START_TIME: The start time of the time period in format yyyy-mm-dd hh:mm:ss.
  - END_TIME: The end time of the time period in format yyyy-mm-dd hh:mm:ss.
  Example:
```
journalctl -u sshd -S "2023-04-25 18:38:00" -U "2023-04-25 18:40:00" --utc
```

To fix these issues, consider the following steps:

Verify if the client key file is set with read-only permissions (flag 400). Otherwise, the SSH client does not accept it.

To set the read-only flag for the private key that's in use, run the following command:
```
chmod 400 ~/.ssh/id_ed25519
```
At the server side, check if the client public key is specified in the corresponding user's local configuration file (~/.ssh/authorized_keys).
In some cases, the problem might be related to the SSH protocol version or the SSH algorithm. The server side and the client side debug outputs might indicate such a problem. Typically, the man pages for ssh and sshd_config provide the details for maintaining the necessary configuration. For example, to find the key exchange algorithms or ciphers that are supported, use the following commands:
```
# Find key exchange algorithms
ssh -Q kex
# Find the symmetric encryption ciphers
ssh -Q cipher
```