I was having some issues with ssh connections taking longer than what I think they should to connect. They were fine once you connected but that first step of conneting was taking too long.
It was possible it was a slow connection in general but taking these steps did help so my suspicion was correct.
Update /etc/sshd/sshd_config on the server, these will already be there so you should be able to uncomment them:
UseDNS no
UsePam no
GSSAPIAuthentication no
On the client, also add GSSAPIAuthentication no to ~/.ssh/config.
Restarting logind after everything may also help.
sudo systemctl restart systemd-logind
Another way to speed up connections is by sharing connections. We can have the ssh connection persist for awhile before we close it fully.
This can be added to the client side: ~/.ssh/config:
Host *
GSSAPIAuthentication no
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
ControlPersist 600
This will persist the connection for 600 seconds so that future connections can use it without having it redo everything.