The beginning
As the night gently whispered me to sleep, with everyone around me fading into a deep slumber.
I was in front of my laptop figuring out weak points for my remote server located in Mumbai. Recently a friend of mine; nicknamed Marty; asked if he could host something of production, which might gain some value soon as more and more people may start using it to hangout and have a gorgeous time in a virtual block game. Henceforth, I decided to bring forward the finest system-administration skills a 17 year old could pull off.
Having enabled fail2ban on the nginx side and the SSH side, along with 2fa on the pterodactyl panel; the one we used to manage the cluster of minecraft server instances for his minecraft server; I decided to put 2fa on the SSH side of things as well… Mainly because Marty also had SSH access to the server, and while I trusted him to my death, his technical skills; despite being beyond acceptable; were not the best at securing his own personal devices. Or so I assumed!
Checked the documentation, and it recommended libpam-google-authenticator for the TOTP generation, but being the skeptical one, I didn't trust google at all. Henceforth, I checked the git source thoroughly for any signs of backdoors or telemetry but was relieved to see none of any major concerns. Installed it on the server and configured SSH for it. Set it up for my user account and noted down both the backup codes and the TOTP seed.
Created a new tab in my terminal emulator and tried to SSH into the server to test it, whilst keeping the initial SSH connection open on the first tab as a precaution in case it didn't work out as planned, and I may need to reconfigure the 2fa.
At last, when I tried to SSH into it, I successfully got into the TOTP input screen. Blissfully, I put in the TOTP generated by the 2fa app and slammed enter with my fingers crossed.
It didn't let me in, wonder something was misconfigured and could be easily fixed by my initial SSH connection, right?
The downfall
I tried it another 2 times, put in the TOTP before confirming that was broken, so I thought “no worries” dawg, just use the backup code. Tried that, 2 eFF-ING TIMES and got notified by my terminal emulator that my initial SSH connection timed out.
I was confused as hell and somewhat scared, took me mere moments to realized that I tried to SSH 5 times in a short spam of time, failing all of them thus triggering my own security system against me making fail2ban blacklist my IP for 30 mins.
Wasn't worried at all because Marty also had SSH access to the server and I could ask him to SSH and just de-configure the 2fa from the SSHd config file… Called Marty and asked him to share his screen and SSH into the server.
He opened his terminal emulator and tried to SSH in, failing it. I asked why he was trying to authenticate with password and not with his SSH key. Trying to reaffirm me,he said that he didn't download the key I sent him yet to which I said I never sent you any key. He checked all our private messages across 3 social media platforms before confirming that he was wrong and I did in-fact never sent him one.
My heart sank, I had asked Marty before I disabled password authentication in SSH (a very common and basic procedure for any sysadmin) and he said I can go ahead. Apparently he missed that line.
The first fail
Sharing my screen, I logged into the Oracle cloud control panel, whilst reading the Oracle cloud's documentation simultaneously… I mentioned I could connect directly as the root user using the connection console.
Swiftly, I turned on the connection console on the instance and patiently waited for it to connect. At first, it timed out, making me think it must be a network error. Tried again and after a few minutes it notified me that there was an error in the connection.
At this point my soul was sinking deeper than Megumi Fushiguro's soul when Ryomen Sukuna took over his body (Jujutsu Kaisen manga reference haha)...
I started turning through the documentation pages of the cloud console, hoping to find another way to connect to the instance.
Alas! I didn't find a way to connect to it but found a way to run scripts in the server.
I opened that bit and started carefully crafting a script which would de-configure 2fa and restart the SSHd service.
Running the script, It didn't give any reply or output and the server was still refusing my attempts to SSH in. I tried to modify the script for any possible errors and reran, still nothing…
I made dozens of different scripts to test around and figure out how this system works. None of the scripts replied or gave an output, and the server was still locked away.
At last, the defeated unicorn I felt like accepted that it was probably my mistake in the instance… Wasn't sure if I disabled or removed some package important for the Oracle Cloud control panel's functions. Or did I?
Time: ~100 mins since getting locked
The last arrow in the quiver
Reading through similar cases where a server got locked out from every user… I found the last method, it should work regardless of the control panel as long as it had some maintenance or recovery mode.
Welp, Oracle Cloud control panel didn't D:
But I could simulate a recovery mode by detaching the boot volume of the broken instance and attaching it to a working one, and then modifying it to fix the issue.
Wasn't sure if that would work because nothing in the control panel was working either way, but it was the last strand.
Ryomen Sukuna's fire arrow to the rescue
Drops of sweat were oozing down my face as I was making the lowest spec possible instance compatible to my current one.
Skipped over a dedicated IPv4 address as I was tensed as to how much would this maneuver would add to my bill. Told myself, since this was a fresh instance, I could connect to it via the control panel.
Oh holy mahoraga was I wrong…
The connection console didn't work for that one either. :/
Had to add a dedicated IPv4 address for it. Had stopped caring about the bill, this was more of a challenge to my skill set than anything else at that point.
SSH-ed into the new instance, installed everything I could think of.
My heart could be felt as it raced when I was detaching the boot volume of the initial instance… Wasn't even sure how to attach it to the recovery instance I made. After 30+ mins of googling, I figured out how to attach it.
Voilà, I booted the recovery instance, but the documentation said it needed to be registered before it could be mounted; which indeed was the case as I couldn't locate the attached broken-volume via any means.
The docs mentioned how it needed a bunch of UUIDs and OCIDs. I couldn't locate how to find them anywhere. ChatGPT finally came into play, I asked it how I could find those… It said that it wasn't available in the control panel, but I needed to install the CLI controller on my local machine to get it. I went ahead and tried that… Oracle being the son-of-a-bitch didn't think it was necessary to give a compiled version of the CLI package for UNIX style-OSes. Finally, I managed to get it compiled after another 40+ minutes of pure agony and loaded it with my API key.
It didn't have any option to find the UUID I needed and when I enquired about the same to ChatGPT it apologized for its 'mistake' and told me it wasn't possible to find it… I googled for a while and found a random blog article about attaching storage volumes to instance, I wasn't sure if it was even relevant as it looked outdated, but it told me there was a button in the 3 dots menu to copy the register command. A grin appeared on my dull face as I found the same on the control panel as well.
It already had the details filled in, so I copy-pasted it in my recovery instance, and it successfully registered!
I could finally locate the new volume in the recovery instance and swiftly mounted it. At last, I could see some light at the end of the tunnel.
Mounted it to /mnt and then fixed the SSHd configuration file.
As I shut down the recovery instance and detached the broken-boot volume, followed by attaching it to the initial instance… I could feel my heart thumping in my throat!
Throughout heaven and the earth, I alone am the honoured one!
It booted and I was able to SSH in!
Felt like a lifetime worth of effort, but I am there.