3ByteBadVM/doc/flag-obfuscation.md

1.7 KiB

Flag obfuscation algorithm


Flag: RS{B4bys_1st_VMPr0tect} # 23-bytes long Stored part of flag: B4bys_1st_VMPr0tect # 19-bytes long

reverse the string

...

Note: program will print 'RITSEC{' and '}' for you; you have to reverse the contents

Type in flag: 'b4bys_1st...' Flag passed to virtual machine verbatim

Step 0

Read the string into the virtualized memory space of the VM

  • String written to VM memory at constant address (at the time of the competition, this address was 0x0080)
  • Note: my entire VM assembly program assumes a 20-byte string. If the user enters fewer, the remaining characters will simply be NULLs (I should have changed this to padd with random chars to make reversing more difficult). The actual flag is 19-characters, so there was a guaranteed NULL terminator in this case.

Step 1

Reverse the string: tcet0rPMV_ts1_syb4B

Step 2

Originally, I was going to convert the string to a custom character encoding

  • ### 0-9 is 0-9, 'A-Z' is 10-35, 'a-z' is 36-61

Instead, due to time limitations, I just subtracted each character byte by 27

  • Reason: writing the bare virtual assembly was starting to get so painful that I switched to the simple subtraction despite it being slightly lamer

  • Note: the memory assigned to the string is of such a size that if the string is not divisible by 4, it will automatically be NULL-padded (the magic of memset)

  • For each 4-byte unit:

    • byte 0: xor byte by value 69
    • byte 1: switch with 2
    • byte 2: switch with 1
    • byte 3: add byte 0 to this (after xor by 69)

THAT is the stored value!

  • Note: this value is stored split up into 4-byte words in different places in the program's memory