Nov 14

Google Cloud SSH Permission Denied Public Key

Debian LogoI recently ran into an issue with a Debian VM instance that had been upgraded from Stretch to Buster. Prior to the upgrade, the ssh via gcloud os login worked just fine. After the upgrade, however, every attempt resulted in a permission denied error.

Now, there are a lot of posts and threads about this particular error. In this case, however, none of them provided the necessary answers to solve this particular scenario. Worse, yet, the backup user account was (suddenly?) no longer or otherwise not in the sudoers group – which added a minor complexity to the troubleshooting.

Post upgrade problem appears :

gcloud beta compute ssh –zone “<zone name>” “<vm name>” –tunnel-through-iap –project “<project name>”

<username>@compute.: Permission denied (publickey).
ERROR: (gcloud.beta.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Troubleshooting – unique elements only

  1. Login to Cloud Platform
  2. Create a Snapshot of the VM
  3. edit the VM Instance
  4. Enable connecting to serial ports and save
  5. connect to the serial port (I found it useful to do this from a terminal window on a separate screen)
  6. gcloud compute –project=<projectname> connect-to-serial-port <vmname> –zone=<zonename>
  7. Reset the vm
  8. Look for the following from the console as the VM reboots : localhost systemd[1]: Reloaded OpenBSD Secure Shell server. [ 12.027817] google_guest_agent[378]: ERROR oslogin.go:147 Error updating NSS cache: exec: “google_oslogin_nss_cache“: executable file not found in $PATH. localhost google_guest_agent[378]: ERROR oslogin.go:147 Error updating NSS cache: exec: “google_oslogin_nss_cache”: executable file not found in $PATH.
    (note – if you still have sudo access via the serial console, you can obviously skip this next step since it’s just adding an existing user to the sudoers group)
  9. Edit the VM Metadata
  10. key : startup-script
    value : #!/bin/bash usermod -aG sudo
  11. Reset the VM
  12. Login using the indicated username and verify sudo
  13. remove the startup-script from metadata
  14. sudo systemctl list-unit-files | grep google | grep enabled
  15. Verify the following : google-disk-expand.service enabled
    google-guest-agent.service enabled
    google-osconfig-agent.service enabled
    google-shutdown-scripts.service enabled
    google-startup-scripts.service enabled
    google-oslogin-cache.timer enabled
    Note, especially, if google-oslogin-cache.timer is missing.
  16. sudo apt-get update
  17. sudo apt-get install
  18. curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add –
  19. DIST=$(cat /etc/os-release | grep “VERSION=” | sed “s/\”\|(\|)\|VERSION=//g” | awk ‘{print tolower($NF)}’) sudo tee /etc/apt/sources.list.d/google-cloud.list << EOM deb http://packages.cloud.google.com/apt google-compute-engine-${DIST}-stable main deb http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-${DIST} main EOM
  20. sudo apt update
  21. sudo apt install -y google-cloud-packages-archive-keyring sudo apt install -y google-compute-engine google-osconfig-agent
  22. sudo reboot
  23. check from your local machine to see if the problem is solved.
  24. gcloud beta compute ssh –zone “<zone name>” “<vm name>” –tunnel-through-iap –project “<project name>”
  25. if not, then : sudo apt update sudo apt install google-compute-engine google-compute-engine-oslogin google-guest-agent google-osconfig-agent
  26. Check again from your local machine to see if the gcloud compute ssh connection works.
  27. If it’s all working, remember to clean up any unneeded snapshots
  28. Edit the VM instance, disable “connecting to serial ports” and save

Wrap up and Links

Hopefully that helped solve your problem. If not, here are a few links that may guide you towards a successful resolution :

 

Dan Yedinak

Dan Yedinak is founder and owner of ECLAT Tech.
These days Dan finds himself focusing primarily in three areas. Small Business Consulting is his area of professional expertise. Whether it is managing a technology project, like site to site VPNs, router and firewall configuration and management, site wide wireless, servers, or websites and internet marketing, or installing and managing the systems of a small business, you can count on Dan to both get the job done, and work with you to document the work so that YOU have control over your Technology, whenever you need it. Dan also devotes a whole lot of time to Autism and to Gender and Socioeconomic Equity.