3ByteBadVM/doc/challenge-design-process.md

40 lines
3.7 KiB
Markdown

# How I designed this challenge
----------------------------------------------------------------
1. Start with the main _trick_ of the challenge
* The point is that this challenge is both going to suck and also be fun because it uses a VM-based protection scheme, which is the _trick_ or point of the challenge.
2. Figure out how challenge will work at a VERY high-level
* User types in flag as string, program reads flag and outputs if flag is correct or not
* Crackme-style challenge kinda, need to reverse the algorithm that takes a flag as input and checks if it is correct or not (in this case, the algorithm needs to be reversible)
3. Design algorithm that obfuscates flag input
* Should take a plaintext flag, "scramble" it, then compare the scambled version with the stored scrambled flag
4. Implement above algorithm in a "real" assembly language
* I used a limited subset of x86 so that I could more directly translate it to my imaginary architecture
* Realize this is actually a major pain in the ass, and purposefully limiting myself in a language that is already more powerful was _no fun_.
* Decide to heck with step 4, let's just get to writing the emulator!
5. Design the imaginary architecture
* Stack- or register-machine? How many registers? Instruction encoding? Memory?
* Realize this wasn't going to happen either; decided to design ISA as I needed it
* However, as I started writing the ISA, I realized that stack machine instructions were "less work" in some ways to implement than a register machine (bc needing to handle many different possible registers instead of just one stack; future fun idea: implement a stack machine that supports TWO stacks and use different instructions to manipulate/switch between them).
6. Write the base of the VM for the imaginary architecture
* Needs to be able to fetch, decode, and execute instructions, it would be helpful if the fetch/decode/execute/repeat cycle were easy/simple, so I decided on a fixed-width encoding using 8 bits for the instruction itself just to make things even easier. However, it had to at least be a little weird, so I used 3-byte instructions (okay, so not weird if you're from the 1960s, but wierd for 2021).
* Don't actually need to implement full ISA just yet; just literally 2-3 test instructions (ex. PUSH, POP, HALT)
7. Write tooling for the imaginary architecture
* Tooling consists of a base for an assembler, disassembler, and debugger (base is made easier to get started since it only has to support 2-3 instructions at first; should be trivial or semi-easy to add new instructions)
* Should be able to display the assembly language representation given a raw machine code file (raw binary)
* Should be able to take a basic assembly language and translate it to a raw binary
* Should be able to step through instructions and print out some VM statistics
* Optional: debugger should support breakpoints (I ended up implementing this; BEST DECISION I EVER MADE FOR DEVELOPMENT!!!)
8. Start coding and implementing/designing ISA
* My strategy: I won't know what I need until I need it, so I'll add it later. I stuck to using the instructions I had, and if I needed a new instruction either because I was missing a necessary capability or I wanted to increase architecture usability, I would just simply implement the new instruction in the switch-case with all the others and add it to my tooling
* I must be a masochist deep down because I could have actually written far more easy-to-implement instructions that would have made bare virtual assembly FAR less painful, but I didn't. Lesson learned: it costs almost nothing to add a new instruction if you're coding on modern computers, so when in doubt, add the instruction (it's just a CTF challenge anyway).