TAICHI

Ubuntu

yinyang

in

Recovering and Securing a Hacked Google Cloud VM: A Complete Guide

This guide documents the step-by-step process of recovering a compromised Google Cloud VM, troubleshooting complex SSH and system issues, and re-securing the server environment.

Step 1: Regaining Initial Access When Locked Out

If you’re locked out of your VM due to a compromised password, you can regain access using a startup script.

Action: Create a Temporary Admin User

Add a startup script to your VM’s metadata in the Google Cloud Console. This script creates a new user with sudo privileges.

  1. Navigate to Compute Engine > VM instances and click EDIT on your VM.
  2. Scroll to the Metadata section and add an item with:
    • Key: startup-script
    • Value (The Script):
      #!/bin/bash
useradd -m -s /bin/bash -G google-sudoers tempadmin
echo "tempadmin:newpassword" | chpasswd
  3. Save and then Stop/Start the VM. You can now log in via the Serial Console with the username tempadmin and password newpassword.

Step 2: Troubleshooting the Serial Console

Problem: Scrolling System Messages

The serial console can be flooded with kernel messages, making it difficult to type.

Action: Silence the Console

Run the following command to tell the kernel to only show critical panic messages:

echo "1 4 1 7" | sudo tee /proc/sys/kernel/printk

Step 3: Securing User Accounts

Problem: Resetting Passwords and Checking the Root Account

After regaining access, immediately secure your original user and verify the root account’s status.

Action: Reset Passwords and Lock Root

  1. Find your original username: ls /home
  2. Reset its password: sudo passwd your_original_username
  3. Check the root account’s status: sudo passwd --status root. The output should be root L (Locked).
  4. If it’s not locked, lock it for security: sudo passwd --lock root

Step 4: Fixing Website and CDN Issues

Problem: Website Down (Error 520) After VM Restart

Restarting a VM with an ephemeral IP address will change its IP, breaking DNS connections from your CDN.

Action: Update DNS and Set a Static IP

  1. Find the VM’s new External IP in the Google Cloud Console.
  2. Update the ‘A’ record in your CDN’s (e.g., QUIC.cloud or Cloudflare) DNS settings to point to the new IP.
  3. Prevent this in the future: In the Google Cloud Console, go to VPC network > IP addresses and reserve the VM’s current IP address to make it static.

Problem: Using Two CDNs (Cloudflare and QUIC.cloud)

Chaining CDNs slows down your site.

Action: Choose One CDN

  • If using LiteSpeed Server: Use QUIC.cloud as the primary CDN. In Cloudflare, set your domain

    Recovering and Securing a Hacked Google Cloud VM: A Complete Guide

    This guide documents the step-by-step process of recovering a compromised Google Cloud VM, troubleshooting complex SSH and system issues, and re-securing the server environment.

    Step 1: Regaining Initial Access When Locked Out

    If you’re locked out of your VM due to a compromised password, you can regain access using a startup script.

    Action: Create a Temporary Admin User

    Add a startup script to your VM’s metadata in the Google Cloud Console. This script creates a new user with sudo privileges.

    1. Navigate to Compute Engine > VM instances and click EDIT on your VM.
    2. Scroll to the Metadata section and add an item with:
      • Key: startup-script
      • Value (The Script):
        #!/bin/bash
useradd -m -s /bin/bash -G google-sudoers tempadmin
echo "tempadmin:newpassword" | chpasswd
    3. Save and then Stop/Start the VM. You can now log in via the Serial Console with the username tempadmin and password newpassword.

    Step 2: Troubleshooting the Serial Console

    Problem: Scrolling System Messages

    The serial console can be flooded with kernel messages, making it difficult to type.

    Action: Silence the Console

    Run the following command to tell the kernel to only show critical panic messages:

    echo "1 4 1 7" | sudo tee /proc/sys/kernel/printk

    Step 3: Securing User Accounts

    Problem: Resetting Passwords and Checking the Root Account

    After regaining access, immediately secure your original user and verify the root account’s status.

    Action: Reset Passwords and Lock Root

    1. Find your original username: ls /home
    2. Reset its password: sudo passwd your_original_username
    3. Check the root account’s status: sudo passwd --status root. The output should be root L (Locked).
    4. If it’s not locked, lock it for security: sudo passwd --lock root

    Step 4: Fixing Website and CDN Issues

    Problem: Website Down (Error 520) After VM Restart

    Restarting a VM with an ephemeral IP address will change its IP, breaking DNS connections from your CDN.

    Action: Update DNS and Set a Static IP

    1. Find the VM’s new External IP in the Google Cloud Console.
    2. Update the ‘A’ record in your CDN’s (e.g., QUIC.cloud or Cloudflare) DNS settings to point to the new IP.
    3. Prevent this in the future: In the Google Cloud Console, go to VPC network > IP addresses and reserve the VM’s current IP address to make it static.

    Problem: Using Two CDNs (Cloudflare and QUIC.cloud)

    Chaining CDNs slows down your site.

    Action: Choose One CDN

    • If using LiteSpeed Server: Use QUIC.cloud as the primary CDN. In Cloudflare, set your domain’s proxy status to DNS Only (Grey Cloud).
    • If using NGINX/Apache: Use Cloudflare as the primary CDN. Set the proxy status to Proxied (Orange Cloud) and point it directly to your VM’s static IP.

    Step 5: Solving Persistent SSH Failures (“Server refused our key”)

    This was the most complex issue, caused by deep system changes from the hacker.

    Action 1: Fix Firewall for “SSH-in-browser”

    The browser-based SSH client connects via Google’s IAP service. You need a firewall rule to allow this.

    • Go to VPC network > Firewall and create a rule.
    • Source IPv4 ranges: 35.235.240.0/20
    • Protocols and ports: TCP, port 22

    Action 2: Fix File Permissions on the Server

    The SSH service requires strict permissions. This is the most common cause of SSH key rejection.

    # Secure your home directory
chmod g-w,o-w /home/your_username

# Secure the .ssh directory
chmod 700 /home/your_username/.ssh

# Secure the authorized_keys file
chmod 600 /home/your_username/.ssh/authorized_keys

# Ensure you own the files
sudo chown -R your_username:your_username /home/your_username/.ssh

    Action 3: Verify the SSH Server Configuration

    If permissions are correct, ensure the SSH server itself is configured to allow key authentication.

    1. Open the configuration file: sudo nano /etc/ssh/sshd_config
    2. Ensure the following two lines are present and are not commented out (they should not start with #):
      PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
    3. If you make changes, save the file and restart the SSH service: sudo systemctl restart sshd

    Action 4: The Final Fix – Restoring Sudo and Reinstalling SSH

    The ultimate root cause was that the hacker had removed the user from the “sudoers” group, and the SSH server package was corrupted.

    1. Restore Sudo Rights: Add a startup script to the VM’s metadata to add your user back to the admin group and restart the VM.
      #!/bin/bash
usermod -a -G google-sudoers your_username
    2. Fix Broken SSH Installation: After regaining sudo access, completely purge and reinstall the SSH server to fix any corruption.
      # Force fix any broken packages
sudo dpkg --configure -a

# Purge the old, broken server
sudo apt-get purge openssh-server -y

# Install a fresh, clean copy
sudo apt-get install openssh-server -y
    3. Re-add your Public Key: After reinstalling, the authorized_keys file will be empty. You must edit it (`nano ~/.ssh/authorized_keys`), paste your single-line public SSH key, and set the permissions one last time.

Leave a Reply

Your email address will not be published. Required fields are marked *