3.7 KiB
3.7 KiB
How I designed this challenge
- Start with the main trick of the challenge
- The point is that this challenge is both going to suck and also be fun because it uses a VM-based protection scheme, which is the trick or point of the challenge.
- Figure out how challenge will work at a VERY high-level
- User types in flag as string, program reads flag and outputs if flag is correct or not
- Crackme-style challenge kinda, need to reverse the algorithm that takes a flag as input and checks if it is correct or not (in this case, the algorithm needs to be reversible)
- Design algorithm that obfuscates flag input
- Should take a plaintext flag, "scramble" it, then compare the scambled version with the stored scrambled flag
- Implement above algorithm in a "real" assembly language
- I used a limited subset of x86 so that I could more directly translate it to my imaginary architecture
- Realize this is actually a major pain in the ass, and purposefully limiting myself in a language that is already more powerful was no fun.
- Decide to heck with step 4, let's just get to writing the emulator!
- Design the imaginary architecture
- Stack- or register-machine? How many registers? Instruction encoding? Memory?
- Realize this wasn't going to happen either; decided to design ISA as I needed it
- However, as I started writing the ISA, I realized that stack machine instructions were "less work" in some ways to implement than a register machine (bc needing to handle many different possible registers instead of just one stack; future fun idea: implement a stack machine that supports TWO stacks and use different instructions to manipulate/switch between them).
- Write the base of the VM for the imaginary architecture
- Needs to be able to fetch, decode, and execute instructions, it would be helpful if the fetch/decode/execute/repeat cycle were easy/simple, so I decided on a fixed-width encoding using 8 bits for the instruction itself just to make things even easier. However, it had to at least be a little weird, so I used 3-byte instructions (okay, so not weird if you're from the 1960s, but wierd for 2021).
- Don't actually need to implement full ISA just yet; just literally 2-3 test instructions (ex. PUSH, POP, HALT)
- Write tooling for the imaginary architecture
- Tooling consists of a base for an assembler, disassembler, and debugger (base is made easier to get started since it only has to support 2-3 instructions at first; should be trivial or semi-easy to add new instructions)
- Should be able to display the assembly language representation given a raw machine code file (raw binary)
- Should be able to take a basic assembly language and translate it to a raw binary
- Should be able to step through instructions and print out some VM statistics
- Optional: debugger should support breakpoints (I ended up implementing this; BEST DECISION I EVER MADE FOR DEVELOPMENT!!!)
- Start coding and implementing/designing ISA
- My strategy: I won't know what I need until I need it, so I'll add it later. I stuck to using the instructions I had, and if I needed a new instruction either because I was missing a necessary capability or I wanted to increase architecture usability, I would just simply implement the new instruction in the switch-case with all the others and add it to my tooling
- I must be a masochist deep down because I could have actually written far more easy-to-implement instructions that would have made bare virtual assembly FAR less painful, but I didn't. Lesson learned: it costs almost nothing to add a new instruction if you're coding on modern computers, so when in doubt, add the instruction (it's just a CTF challenge anyway).