This guide documents the step-by-step process of recovering a compromised Google Cloud VM, troubleshooting complex SSH and system issues, and re-securing the server environment.<\/p>\n\n
If you’re locked out of your VM due to a compromised password, you can regain access using a startup script.<\/p>\n
Add a startup script to your VM’s metadata in the Google Cloud Console. This script creates a new user with sudo privileges.<\/p>\n
startup-script<\/code><\/li>\n - Value (The Script):<\/strong>\n
#!\/bin\/bash\nuseradd -m -s \/bin\/bash -G google-sudoers tempadmin\necho \"tempadmin:newpassword\" | chpasswd<\/code><\/pre>\n <\/li>\n <\/ul>\n - Save<\/strong> and then Stop\/Start<\/strong> the VM. You can now log in via the Serial Console<\/strong> with the username
tempadmin<\/code> and password newpassword<\/code>.<\/li>\n<\/ol>\n\n
\n\nStep 2: Troubleshooting the Serial Console<\/h2>\nProblem: Scrolling System Messages<\/h3>\n
The serial console can be flooded with kernel messages, making it difficult to type.<\/p>\n
Action: Silence the Console<\/h4>\n
Run the following command to tell the kernel to only show critical panic messages:<\/p>\n
echo \"1 4 1 7\" | sudo tee \/proc\/sys\/kernel\/printk<\/code><\/pre>\n\n
\n\nStep 3: Securing User Accounts<\/h2>\nProblem: Resetting Passwords and Checking the Root Account<\/h3>\n
After regaining access, immediately secure your original user and verify the root account’s status.<\/p>\n
Action: Reset Passwords and Lock Root<\/h4>\n\n - Find your original username:
ls \/home<\/code><\/li>\n - Reset its password:
sudo passwd your_original_username<\/code><\/li>\n - Check the root account’s status:
sudo passwd --status root<\/code>. The output should be root L<\/code> (Locked).<\/li>\n - If it’s not locked, lock it for security:
sudo passwd --lock root<\/code><\/li>\n<\/ol>\n\n
\n\nStep 4: Fixing Website and CDN Issues<\/h2>\nProblem: Website Down (Error 520) After VM Restart<\/h3>\n
Restarting a VM with an ephemeral IP address will change its IP, breaking DNS connections from your CDN.<\/p>\n
Action: Update DNS and Set a Static IP<\/h4>\n\n - Find the VM’s new External IP<\/strong> in the Google Cloud Console.<\/li>\n
- Update the ‘A’ record<\/strong> in your CDN’s (e.g., QUIC.cloud or Cloudflare) DNS settings to point to the new IP.<\/li>\n
- Prevent this in the future:<\/strong> In the Google Cloud Console, go to VPC network > IP addresses<\/strong> and reserve the VM’s current IP address to make it static<\/strong>.<\/li>\n<\/ol>\n\n
Problem: Using Two CDNs (Cloudflare and QUIC.cloud)<\/h3>\n
Chaining CDNs slows down your site.<\/p>\n
Action: Choose One CDN<\/h4>\n\n - If using LiteSpeed Server:<\/strong> Use QUIC.cloud as the primary CDN. In Cloudflare, set your domain\n\n\n\n
Recovering and Securing a Hacked Google Cloud VM: A Complete Guide<\/h1>\n\n
This guide documents the step-by-step process of recovering a compromised Google Cloud VM, troubleshooting complex SSH and system issues, and re-securing the server environment.<\/p>\n\n
\n\nStep 1: Regaining Initial Access When Locked Out<\/h2>\n
If you’re locked out of your VM due to a compromised password, you can regain access using a startup script.<\/p>\n
Action: Create a Temporary Admin User<\/h3>\n
Add a startup script to your VM’s metadata in the Google Cloud Console. This script creates a new user with sudo privileges.<\/p>\n
\n - Navigate to Compute Engine > VM instances<\/strong> and click EDIT<\/strong> on your VM.<\/li>\n
- Scroll to the Metadata<\/strong> section and add an item with:<\/li>\n
\n - Key:<\/strong>
startup-script<\/code><\/li>\n - Value (The Script):<\/strong>\n
#!\/bin\/bash\nuseradd -m -s \/bin\/bash -G google-sudoers tempadmin\necho \"tempadmin:newpassword\" | chpasswd<\/code><\/pre>\n <\/li>\n <\/ul>\n - Save<\/strong> and then Stop\/Start<\/strong> the VM. You can now log in via the Serial Console<\/strong> with the username
tempadmin<\/code> and password newpassword<\/code>.<\/li>\n<\/ol>\n\n
\n\nStep 2: Troubleshooting the Serial Console<\/h2>\nProblem: Scrolling System Messages<\/h3>\n
The serial console can be flooded with kernel messages, making it difficult to type.<\/p>\n
Action: Silence the Console<\/h4>\n
Run the following command to tell the kernel to only show critical panic messages:<\/p>\n
echo \"1 4 1 7\" | sudo tee \/proc\/sys\/kernel\/printk<\/code><\/pre>\n\n
\n\nStep 3: Securing User Accounts<\/h2>\nProblem: Resetting Passwords and Checking the Root Account<\/h3>\n
After regaining access, immediately secure your original user and verify the root account’s status.<\/p>\n
Action: Reset Passwords and Lock Root<\/h4>\n\n - Find your original username:
ls \/home<\/code><\/li>\n - Reset its password:
sudo passwd your_original_username<\/code><\/li>\n - Check the root account’s status:
sudo passwd --status root<\/code>. The output should be root L<\/code> (Locked).<\/li>\n - If it’s not locked, lock it for security:
sudo passwd --lock root<\/code><\/li>\n<\/ol>\n\n
\n\nStep 4: Fixing Website and CDN Issues<\/h2>\nProblem: Website Down (Error 520) After VM Restart<\/h3>\n
Restarting a VM with an ephemeral IP address will change its IP, breaking DNS connections from your CDN.<\/p>\n
Action: Update DNS and Set a Static IP<\/h4>\n\n - Find the VM’s new External IP<\/strong> in the Google Cloud Console.<\/li>\n
- Update the ‘A’ record<\/strong> in your CDN’s (e.g., QUIC.cloud or Cloudflare) DNS settings to point to the new IP.<\/li>\n
- Prevent this in the future:<\/strong> In the Google Cloud Console, go to VPC network > IP addresses<\/strong> and reserve the VM’s current IP address to make it static<\/strong>.<\/li>\n<\/ol>\n\n
Problem: Using Two CDNs (Cloudflare and QUIC.cloud)<\/h3>\n
Chaining CDNs slows down your site.<\/p>\n
Action: Choose One CDN<\/h4>\n\n - If using LiteSpeed Server:<\/strong> Use QUIC.cloud as the primary CDN. In Cloudflare, set your domain’s proxy status to DNS Only (Grey Cloud)<\/strong>.<\/li>\n
- If using NGINX\/Apache:<\/strong> Use Cloudflare as the primary CDN. Set the proxy status to Proxied (Orange Cloud)<\/strong> and point it directly to your VM’s static IP.<\/li>\n<\/ul>\n\n
\n\nStep 5: Solving Persistent SSH Failures (“Server refused our key”)<\/h2>\n
This was the most complex issue, caused by deep system changes from the hacker.<\/p>\n
Action 1: Fix Firewall for “SSH-in-browser”<\/h3>\n
The browser-based SSH client connects via Google’s IAP service. You need a firewall rule to allow this.<\/p>\n
\n - Go to VPC network > Firewall<\/strong> and create a rule.<\/li>\n
- Source IPv4 ranges:<\/strong>
35.235.240.0\/20<\/code><\/li>\n - Protocols and ports:<\/strong> TCP, port
22<\/code><\/li>\n<\/ul>\n\nAction 2: Fix File Permissions on the Server<\/h3>\n
The SSH service requires strict permissions. This is the most common cause of SSH key rejection.<\/p>\n
# Secure your home directory\nchmod g-w,o-w \/home\/your_username\n\n# Secure the .ssh directory\nchmod 700 \/home\/your_username\/.ssh\n\n# Secure the authorized_keys file\nchmod 600 \/home\/your_username\/.ssh\/authorized_keys\n\n# Ensure you own the files\nsudo chown -R your_username:your_username \/home\/your_username\/.ssh<\/code><\/pre>\n\nAction 3: Verify the SSH Server Configuration<\/h3>\n
If permissions are correct, ensure the SSH server itself is configured to allow key authentication.<\/p>\n
\n - Open the configuration file:
sudo nano \/etc\/ssh\/sshd_config<\/code><\/li>\n - Ensure the following two lines are present and are not commented out (they should not start with #):\n
PubkeyAuthentication yes\nAuthorizedKeysFile .ssh\/authorized_keys<\/code><\/pre>\n <\/li>\n - If you make changes, save the file and restart the SSH service:
sudo systemctl restart sshd<\/code><\/li>\n<\/ol>\n\nAction 4: The Final Fix – Restoring Sudo and Reinstalling SSH<\/h3>\n
The ultimate root cause was that the hacker had removed the user from the “sudoers” group, and the SSH server package was corrupted.<\/p>\n
\n - Restore Sudo Rights:<\/strong> Add a startup script to the VM’s metadata to add your user back to the admin group and restart the VM.\n
#!\/bin\/bash\nusermod -a -G google-sudoers your_username<\/code><\/pre>\n <\/li>\n - Fix Broken SSH Installation:<\/strong> After regaining sudo access, completely purge and reinstall the SSH server to fix any corruption.\n
# Force fix any broken packages\nsudo dpkg --configure -a\n\n# Purge the old, broken server\nsudo apt-get purge openssh-server -y\n\n# Install a fresh, clean copy\nsudo apt-get install openssh-server -y<\/code><\/pre>\n <\/li>\n - Re-add your Public Key:<\/strong> After reinstalling, the authorized_keys file will be empty. You must edit it (`nano ~\/.ssh\/authorized_keys`), paste your single-line public SSH key, and set the permissions one last time.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
Recovering and Securing a Hacked Google Cloud VM: A Complete Guide This guide documents the step-by-step process of recovering a compromised Google Cloud VM, troubleshooting complex SSH and system issues, and re-securing the server environment. Step 1: Regaining Initial Access When Locked Out If you’re locked out of your VM due to a compromised password, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-321","post","type-post","status-publish","format-standard","hentry","category-itai"],"_links":{"self":[{"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=\/wp\/v2\/posts\/321","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=321"}],"version-history":[{"count":4,"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=\/wp\/v2\/posts\/321\/revisions"}],"predecessor-version":[{"id":325,"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=\/wp\/v2\/posts\/321\/revisions\/325"}],"wp:attachment":[{"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/yinyang.taichi.us.kg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}