stragglers

README.md

@@ -0,0 +1,59 @@
+# TypeTapper
+
+Audrey Dutcher
+CSE 578 Fall 2022
+
+TypeTapper is a tool to visualize structure usage patterns in binaries.
+
+## Installation
+
+To install, run the installation script: `./install.sh`.
+It is highly recommended you do this in a virtualenv!
+This will install dependencies, and then link TypeTapper into the plugin loading path for angr-management.
+
+## Booting
+
+Start angr-management: `python -m angrmanagement`.
+Click in the toolbar: Plugins -> Manage Plugins... and check the box next to TypeTapper, then press Ok.
+Load a binary to analyze from the File menu. Several good samples are provided in the `samples` directory.
+As an example, let's use the corewars-vm binary.
+Click through the loading and analysis options screens, using the default settings.
+Wait for autoanalysis to finish.
+Click on the code that appears and press the Tab key to decompile it.
+Finally, right-click on a variable usage you're interested in and select "Start TypeTapper".
+In corewars-vm, a good variable to use is any of the variables assigned from a call to `calloc`.
+
+![](pics/startup.png)
+
+## Usage
+
+The main view of TypeTapper is an interactive node-link diagram, initially populated by one node.
+
+![](pics/initial.png)
+
+### Interactions
+
+- To populate it with more nodes, click on the blue plus icon and then click on one of the nodes which appears to commit it to the graph.
+- Click and drag a node to reposition it.
+- Hold down the Z key to run the graph layout algorithm on the current layout.
+- Double-click on a node to enter a new view with that node's children in it. If you double-click on a node which has no children, you will be navigated to the corresponding point in the disassembly.
+- Double-click on the background to undo this operation and navigate to the parent view. To undo navigating to the disassembly, click on the tab in the tab bar labeled "TypeTapper".
+- Click and drag on the background to pan the view.
+- Hold Shift, then click and drag on the background to select multiple nodes.
+- Right-click on a selected node or nodes to view context actions, including creation and restructuring of node groupings.
+- Drag a node into another node to move the dragged node into the target node's group.
+- Hold down the X key to enter rapid-fire expand mode, where hovering nodes with the mouse will immediately commit and expand them.
+
+![](pics/complex.png)
+
+## Code structure
+
+- `data.py` is the record structures used by the analysis and the visualization, as well as some very primitive algorithms for manipulating them.
+- `engine.py` is the emulator logic for the static analysis. It describes how to interpret Valgrind's VEX IR as directives for producing the data domain we're working with.
+- `analysis.py` is the coordinator logic for the static analysis. It runs in two phases, first applying the engine to each basic block of the program, and then stitching the results together with the help of a control flow graph.
+- `knowledge.py` is the angr knowledge base plugin to store the static analysis result.
+- `procedures.py` is a collection of function summaries to provide meaningful results when analyzing imported library functions without having to look at their code.
+- `relative_graph.py` is the first-level data structure used during the visualization. It generates and stores a view of the graph from the static analysis results such that each node in the derived view contains properties relative to a starting node.
+- `hierarchy_graph.py` is the second-level data structure used during the visualization. It organizes nodes from the relative graph into a hierarchy, storing data in such a way that it is efficient to query relationships between nodes at different levels of hierarchy.
+- `plugin.py` is the angr-management plugin definition. It handles the UI elements outside the main TypeTapper view as well as coordinating the initial analysis.
+- `hierarchy_graph_view.py` is the visualization UI code. It defines the main TypeTapper widget, its rendering and interactions, the subcomponent graphical items, and their behaviors.

install.sh

@@ -0,0 +1,13 @@
+#!/bin/sh -ex
+
+pip install cffi unicorn pygraphviz
+pip install --no-build-isolation git+https://github.com/angr/archinfo@8742e791f6557e3f6d472d935102ea01d1881e52#egg=archinfo
+pip install --no-build-isolation git+https://github.com/angr/pyvex@b9fa0bcc91bce1b4169b096e19f528901250541d#egg=pyvex
+pip install --no-build-isolation git+https://github.com/angr/claripy@bccb40657174e2b02da96712b053ca81f143ba12#egg=claripy
+pip install --no-build-isolation git+https://github.com/angr/cle@3378ac6eba145ad04a84a9f3876b6395b4bd450c#egg=cle
+pip install --no-build-isolation git+https://github.com/angr/ailment@eb15befceb1cba4475afcdef450270da5b300c97#egg=ailment
+pip install --no-build-isolation git+https://github.com/angr/angr@b14a09b1e5bd662f9e0c83403e5f432f511600fb#egg=angr
+pip install --no-build-isolation git+https://github.com/angr/angr-management@5d3ae680978f8984f849d49362f89de6a03d756d#egg=angr_management
+
+mkdir -p ~/.local/share/angr-management/plugins
+ln -s $(dirname $(realpath $0)) ~/.local/share/angr-management/plugins

notes

@@ -0,0 +1,66 @@
+DESIGN:
+- output of analysis is a full-program dataflow graph
+  - nodes are atoms annotated with properties (usage notes)
+  - edges are directed, annotated with
+    - op sequences indicating how the data is transformed or related at the two locations (can be concatenated, simplified, inverted)
+    - control flow indicating how to treat this edge with respect to context sensitivity
+- graph is traversed through a "relative atom graph"
+  - initialized with respect to a single atom
+  - each node of the RAG corresponds to a single node of the atom graph plus context sensitivity info
+    - nodes are annotated with the relationship between the seed atom and the target atom (the sum of the ops along the path, simplified)
+    - in case of multiple paths to a single node, store all the op-paths.
+    - these paths can traverse reverse edges, the ops are inverted in this case
+    - control flow info prevents meaningless paths from being analyzed
+    - op sequences can be used as functions to transform the node properties of the target atom to be "about" the seed atom
+  - this graph is technically infinite, so it starts off with a single node and a "frontier" of edges that have yet to be explored
+    - methods for expanding the graph from each edge on the frontier
+- the complexity of this graph is managed through a hierarchical graph, a view on the RAG
+  - arbitrary collapsing of groups of atoms into single "group" nodes, recursively
+  - efficient algorithms
+
+passes:
++ per-block live data flow and manipulations
++ inter- and intra-block passive register data flow, data flowing across callsites
+- passive memory flow - given a store, track where live references to it propagate and are loaded
+- constant propagation - need to re-evaluate ops which are marked as variable-dependant?
+
+ideas for reducing chaos:
+- only allow paths which follow only forward or only backward edges (add edges from referents to references)
+- discard-with-connection nodes which have no interesting properties
+- group atoms which refer to the same variable
+- use sfdp on large graphs
+- set initial positions based on codeloc position in disassembly graph or something
+
+next in the queue:
+- memcpy procedure
+- recursive mark layout algorithm
+- search for node text in current view
+- start from disassembly atoms
+- start from function argument atoms
+- navigate to decompilation
+
+TODO
+- Attempt to expand graph automatically by refusing to recurse
+- globals...
+
+Research directions
+- Type confusion as weird structures
+- Pick more applications
+
+ReMind
+Why Johnny Can't reveres malware  https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7546501
+Dan Voytika
+Hackers vs testers
+something ghidra - Michell Mazerik x3
+
+https://dl.acm.org/doi/abs/10.1145/2896499
+
+applications:
+Type inference
+decompilation
+taint tracking
+
+targets:
+rust
+golang
+c++ virtual inheritance

typetapper/analysis.py

@@ -31,7 +31,9 @@ class TypeTapperAnalysis(angr.Analysis):
 
     def _analyze_active_flow(self):
         node: CFGNode
-        for node in self._cfg.graph.nodes():
+        for i, node in enumerate(self._cfg.graph.nodes()):
+            progress = i / len(self._cfg.graph) * 100
+            self._update_progress(progress, "Analyzing " + str(node))
             block: Block = node.block
             if block is not None:
                 self._engine.handle_vex_block(block.vex)
@@ -42,6 +44,7 @@ class TypeTapperAnalysis(angr.Analysis):
                     proc.analyze(node.addr)
 
     def _analyze_passive_flow(self):
+        self._update_progress(99.99, "Analyzing passive flow")
         queue = OrderedDict()
         for block_addr in self.manager.block_info.keys():
             queue[block_addr] = None

typetapper/data.py

@@ -7,18 +7,18 @@ import copy
 import networkx
 
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class CodeLoc:
     bbl_addr: int
     stmt_idx: int
     ins_addr: int
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class Atom:
     loc: CodeLoc
     size: int
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class RegisterAtom(Atom):
     name: str
     slot_name: str
@@ -26,21 +26,21 @@ class RegisterAtom(Atom):
     def __repr__(self):
         return f'{self.name} @ {self.loc.ins_addr:#x}'
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class MemoryAtom(Atom):
     endness: str
 
     def __repr__(self):
         return f'MEM @ {self.loc.ins_addr:#x}'
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class TmpAtom(Atom):
     tmp: int
 
     def __repr__(self):
         return f'TMP#{self.tmp} @ {self.loc.ins_addr:#x}'
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class ConstAtom(Atom):
     value: int
 
@@ -48,12 +48,12 @@ class ConstAtom(Atom):
         return f'CONST#{self.value:#x} @ {self.loc.ins_addr:#x}'
 
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class Op:
     def invert(self) -> 'Op':
         raise NotImplementedError
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class ConstOffsetOp(Op):
     const: int
 
@@ -68,12 +68,19 @@ class ConstOffsetOp(Op):
             const -= 2**64
         object.__setattr__(self, 'const', const)
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
+class StrideOffsetOp(Op):
+    stride: int
+
+    def invert(self):
+        return self
+
+@dataclass(frozen=True, slots=True)
 class NegOp(Op):
     def invert(self):
         return self
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class VarOffsetOp(Op):
     var: Any
 
@@ -81,26 +88,26 @@ class VarOffsetOp(Op):
         # TODO ????
         return self
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class DerefOp(Op):
     size: int
 
     def invert(self):
         return RefOp(self.size)
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class RefOp(Op):
     size: int
 
     def invert(self):
         return DerefOp(self.size)
 
-#@dataclass(frozen=True)
+#@dataclass(frozen=True, slots=True)
 #class OtherOp(Op):
 #    def invert(self) -> 'Op':
 #        return self
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class OpSequence:
     ops: Tuple[Op, ...] = ()
 
@@ -127,6 +134,20 @@ class OpSequence:
     def invert(self) -> 'OpSequence':
         return OpSequence(tuple(x.invert() for x in reversed(self.ops)))
 
+    def compute_unifications(self) -> List[Tuple[int, int]]:
+        base_offset = 0
+        strides = []
+        for op in self.ops:
+            if isinstance(op, ConstOffsetOp):
+                base_offset += op.const
+            elif isinstance(op, StrideOffsetOp):
+                strides.append(op.stride)
+            else:
+                base_offset = 0
+                strides = []
+
+        return [(base_offset, base_offset + stride) for stride in strides]
+
 def simplify_op_sequence(seq: List[Op]):
     i = 0
     while i < len(seq):
@@ -161,6 +182,11 @@ def simplify_op_sequence(seq: List[Op]):
             if i > 0:
                 i -= 1
             continue
+        if isinstance(cur, StrideOffsetOp) and isinstance(nex, StrideOffsetOp) and cur.stride == nex.stride:
+            seq.pop(i)
+            if i > 0:
+                i -= 1
+            continue
 
         i += 1
 
@@ -172,33 +198,27 @@ class DataKind(IntEnum):
     Float = auto()
     Pointer = auto()
 
-@dataclass
+@dataclass(slots=True)
 class Prop:
     self_data: Counter[DataKind] = field(default_factory=Counter)
-    struct_data: defaultdict[int, defaultdict[int, Counter[DataKind]]] = field(default_factory=lambda: defaultdict(lambda: defaultdict(Counter)))
+    struct_data: Counter[Tuple[int, int, DataKind]] = field(default_factory=Counter)
     unifications: Counter[Tuple[int, int]] = field(default_factory=Counter)
 
     def update(self, other: 'Prop'):
         self.self_data.update(other.self_data)
-        for offset, v1 in other.struct_data.items():
-            for size, v2 in v1.items():
-                self.struct_data[offset][size].update(v2)
+        self.struct_data.update(other.struct_data)
         self.unifications.update(other.unifications)
 
     def subtract(self, other: 'Prop'):
         self.self_data.subtract(other.self_data)
-        for offset, v1 in other.struct_data.items():
-            for size, v2 in v1.items():
-                self.struct_data[offset][size].subtract(v2)
+        self.struct_data.subtract(other.struct_data)
         self.unifications.subtract(other.unifications)
 
     def maximize(self, other: 'Prop'):
         for key, val in other.self_data.items():
             self.self_data[key] = max(self.self_data[key], val)
-        for offset, v1 in other.struct_data.items():
-            for size, v2 in v1.items():
-                for kind, val in v2.items():
-                    self.struct_data[offset][size][kind] = max(self.struct_data[offset][size][kind], val)
+        for key, val in other.struct_data.items():
+                self.struct_data[key] = max(self.struct_data[key], val)
         for key, val in other.unifications.items():
             self.unifications[key] = max(self.unifications[key], val)
 
@@ -212,24 +232,25 @@ class Prop:
         result = copy.deepcopy(self)
         for op in ops.ops:
             if isinstance(op, RefOp):
-                result.struct_data.clear()
-                result.struct_data[0][op.size] = result.self_data
-                result.self_data = Counter()
-                self.unifications.clear()
+                result.struct_data = Counter({(0, op.size, k): v for k, v in result.self_data.items()})
+                result.self_data.clear()
+                result.unifications.clear()
             elif isinstance(op, DerefOp):
-                result.self_data = result.struct_data[0][op.size]
+                result.self_data = Counter({k[2]: v for k, v in result.struct_data.items() if k[0] == 0 and k[1] == op.size})
                 result.struct_data.clear()
-                self.unifications.clear()
+                result.unifications.clear()
             elif isinstance(op, ConstOffsetOp):
                 items = list(result.struct_data.items())
                 result.struct_data.clear()
-                for k, v in items:
-                    result.struct_data[k - op.const] = v  # there is some JANK shit going on with this sign
+                for (offset, size, kind), v in items:
+                    result.struct_data[(offset - op.const, size, kind)] = v  # there is some JANK shit going on with this sign
                 saved = result.self_data.get(DataKind.Pointer, None)
                 result.self_data.clear()
                 if saved:
                     result.self_data[DataKind.Pointer] = saved
                 result.unifications = Counter((x - op.const, y - op.const) for x, y in result.unifications)
+            elif isinstance(op, StrideOffsetOp):
+                result.self_data.clear()
             elif isinstance(op, VarOffsetOp):
                 saved = result.self_data.get(DataKind.Pointer, None)
                 result = Prop()
@@ -239,53 +260,76 @@ class Prop:
                 result = Prop()
         return result
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class LiveData:
     """
     The in-flight data representation for the analysis. All sizes are in bytes
     """
-    sources: List[Tuple[Atom, OpSequence]]
-    const: Optional[int]
+    loc: CodeLoc
+    sources: Tuple[Tuple[Atom, OpSequence], ...]
     size: int
+    # if this is non-empty it means the data is characterized SOLELY by the sum of a0*x + a1*y + a2*z + ...
+    strides: Tuple[Tuple[Optional['LiveData'], int], ...]
+
+    @property
+    def const(self):
+        if len(self.strides) == 1 and self.strides[0][0] is None:
+            return self.strides[0][1]
+        return None
 
     @classmethod
-    def new_null(cls, size: int):
-        return cls([], None, size)
+    def new_null(cls, loc: CodeLoc, size: int, strides: Tuple[Tuple[Optional['LiveData'], int], ...]=()):
+        return cls(loc, (), size, strides)
 
     @classmethod
-    def new_atom(cls, atom: Atom) -> 'LiveData':
-        return cls([(atom, OpSequence())], None, atom.size)
+    def new_atom(cls, loc: CodeLoc, atom: Atom) -> 'LiveData':
+        return cls(loc, ((atom, OpSequence()),), atom.size, ())
 
     @classmethod
-    def new_const(cls, value: int, size: int, codeloc: CodeLoc) -> 'LiveData':
-        return cls([(ConstAtom(codeloc, size, value), OpSequence())], value, size)
+    def new_const(cls, loc: CodeLoc, value: int, size: int) -> 'LiveData':
+        return cls(loc, ((ConstAtom(loc, size, value), OpSequence()),), size, ((None, value),))
 
-    def appended(self, op: Op, size: int, const: Optional[int]=None) -> 'LiveData':
+    def appended(self, loc: CodeLoc, op: Op, size: int, strides: Optional[Tuple[Tuple[Optional['LiveData'], int], ...]]=None) -> 'LiveData':
         return LiveData(
-            [(atom, seq.appended(op)) for atom, seq in self.sources],
-            self.const if const is None else const,
-            size
+            loc,
+            tuple((atom, seq.appended(op)) for atom, seq in self.sources),
+            size,
+            self.strides if strides is None else strides,
         )
 
-    def unioned(self, other: 'LiveData', size: int, const: Optional[int]=None) -> 'LiveData':
-        return LiveData(self.sources + other.sources, const, size)
+    def unioned(
+            self,
+            loc: CodeLoc,
+            other: 'LiveData',
+            size: int,
+            strides: Tuple[Tuple[Optional['LiveData'], int], ...]=(),
+    ) -> 'LiveData':
+        return LiveData(loc, self.sources + other.sources, size, strides)
 
     def commit(self, target: Atom, graph: networkx.DiGraph):
+        prop = Prop()
         for src, seq in self.sources:
+            for start, end in seq.compute_unifications():
+                prop.unifications[(start, end)] += 1
             graph.add_edge(src, target, ops=seq, cf=[])
+        self.atom_prop(target, prop, graph)
 
     def prop(self, prop: Prop, graph: networkx.DiGraph):
         for atom, ops in self.sources:
             tprop = prop.transform(ops.invert())
-            try:
-                eprop: Prop = graph.nodes[atom].get('prop')
-            except KeyError:
-                graph.add_node(atom, prop=tprop)
+            self.atom_prop(atom, tprop, graph)
+
+    @staticmethod
+    def atom_prop(atom, tprop: Prop, graph: networkx.DiGraph):
+        try:
+            eprop: Prop = graph.nodes[atom].get('prop')
+        except KeyError:
+            graph.add_node(atom, prop=tprop)
+        else:
+            if eprop:
+                eprop.update(tprop)
             else:
-                if eprop:
-                    eprop.update(tprop)
-                else:
-                    graph.nodes[atom]['prop'] = tprop
+                graph.nodes[atom]['prop'] = tprop
 
     def prop_self(self, kind: DataKind, graph: networkx.DiGraph):
         prop = Prop()
@@ -297,7 +341,7 @@ class LiveData:
         prop.unifications[(offset1, offset2)] += 1
         self.prop(prop, graph)
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class RegisterInputInfo:
     atom: RegisterAtom
     callsites: Tuple[int, ...]
@@ -328,20 +372,20 @@ class RegisterInputInfo:
         graph.add_edge(source, self.atom, ops=OpSequence(), cf=actions)
 
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class ControlFlowAction:
     pass
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class ControlFlowActionPush(ControlFlowAction):
     callsite: int
 
-@dataclass(frozen=True)
+@dataclass(frozen=True, slots=True)
 class ControlFlowActionPop(ControlFlowAction):
     callsite: int
 
 
-@dataclass
+@dataclass(slots=True)
 class BlockInfo:
     outputs: Dict[str, RegisterAtom] = field(default_factory=dict)  # slot names
     inputs: Dict[str, RegisterAtom] = field(default_factory=dict)  # alias names

typetapper/engine.py

@@ -1,5 +1,6 @@
 from typing import Union
 import logging
+from collections import Counter
 
 import angr
 import pyvex
@@ -42,6 +43,15 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
             return self.kp.block_info[self._force_addr]
         return self.kp.block_info[self.irsb.addr]
 
+    def new_null(self, size: int, strides: Tuple[Tuple[Optional['LiveData'], int], ...]=()):
+        return LiveData.new_null(self.codeloc, size, strides)
+
+    def new_atom(self, atom: Atom) -> 'LiveData':
+        return LiveData.new_atom(self.codeloc, atom)
+
+    def new_const(self, value: int, size: int) -> 'LiveData':
+        return LiveData.new_const(self.codeloc, value, size)
+
     def handle_vex_block(self, irsb):
         self._force_addr = None
         super().handle_vex_block(irsb)
@@ -50,20 +60,20 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
         return self.const(const.value, get_type_size_bytes(const.type))
 
     def const(self, val, size) -> LiveData:
-        atom = LiveData.new_const(val, size, self.codeloc)
+        atom = self.new_const(val, size)
         self.blockinfo.atoms.append(atom.sources[0][0])
         return atom
 
     def _perform_vex_expr_RdTmp(self, tmp):
         if self.tmp_atoms:
-            return LiveData.new_atom(self.tmps[tmp])
+            return self.new_atom(self.tmps[tmp])
         else:
             return self.tmps[tmp]
 
     def _perform_vex_expr_Get(self, offset: LiveData, ty, **kwargs):
         size = get_type_size_bytes(ty)
         if type(offset.const) is not int:
-            return LiveData.new_null(size)
+            return self.new_null(size)
         name = self.project.arch.register_size_names[(offset.const, size)]  # unsafe
         return self.get(name, offset.const, size)
 
@@ -73,7 +83,7 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
         slot_info = self.project.arch.get_base_register(offset, size)
         if slot_info is None:
             l.error("??????? (%s, %s)", offset, size)
-            return LiveData.new_null(size)
+            return self.new_null(size)
         slot_name = self.project.arch.register_size_names[slot_info]
         reg_atom = RegisterAtom(self.codeloc, size, name, slot_name)
         self.blockinfo.atoms.append(reg_atom)
@@ -85,12 +95,12 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
             else:
                 pass  # alias mismatch
         elif name in self.blockinfo.inputs:
-            return LiveData.new_atom(self.blockinfo.inputs[name])
+            return self.new_atom(self.blockinfo.inputs[name])
         else:
             self.blockinfo.inputs[name] = reg_atom
             self.blockinfo.ready_inputs.add(name)
 
-        return LiveData.new_atom(reg_atom)
+        return self.new_atom(reg_atom)
 
     def _perform_vex_expr_Load(self, addr: LiveData, ty, endness, **kwargs):
         size = get_type_size_bytes(ty)
@@ -99,20 +109,20 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
     def load(self, addr, size, endness):
         prop = Prop()
         prop.self_data[DataKind.Pointer] += 1
-        prop.struct_data[0][size][DataKind.GenericData] += 1
+        prop.struct_data[(0, size, DataKind.GenericData)] += 1
         addr.prop(prop, self.graph)
 
         mem_atom = MemoryAtom(self.codeloc, size, endness)
         self.blockinfo.atoms.append(mem_atom)
-        addr.appended(DerefOp(size), size).commit(mem_atom, self.graph)
-        return LiveData.new_atom(mem_atom)
+        addr.appended(self.codeloc, DerefOp(size), size).commit(mem_atom, self.graph)
+        return self.new_atom(mem_atom)
 
     def _perform_vex_expr_CCall(self, func_name, ty, args, func=None):
-        return LiveData.new_null(get_type_size_bytes(ty))
+        return self.new_null(get_type_size_bytes(ty))
 
     def _perform_vex_expr_ITE(self, cond, ifTrue: LiveData, ifFalse: LiveData):
         assert ifTrue.size == ifFalse.size
-        return ifTrue.unioned(ifFalse, ifTrue.size)
+        return ifTrue.unioned(self.codeloc, ifFalse, ifTrue.size)
 
     def _perform_vex_expr_Op(self, op, args: List[LiveData]):
         return self.op(op, args)
@@ -126,39 +136,95 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
                 arg.prop_self(DataKind.Int, self.graph)
 
         size = get_type_size_bytes(ret_ty)
+        sign = None
+        mul0, mul1 = None, None
+        const0, const1 = None, None
         if op in ('Iop_Add8', 'Iop_Add16', 'Iop_Add32', 'Iop_Add64'):
             sign = 1
         elif op in ('Iop_Sub8', 'Iop_Sub16', 'Iop_Sub32', 'Iop_Sub64'):
             sign = -1
-        else:
-            sign = None
+        elif op in ('Iop_Mul8', 'Iop_Mul16', 'Iop_Mul32', 'Iop_Mul64',
+                    'Iop_MullS8', 'Iop_MullS16', 'Iop_MullS32', 'Iop_MullS64',
+                    'Iop_MullU8', 'Iop_MullU16', 'Iop_MullU32', 'Iop_MullU64'):
+            mul0 = args[0].strides
+            mul1 = args[1].strides
+            const0 = args[0].const
+            const1 = args[1].const
+        elif op in ('Iop_Shl8', 'Iop_Shl16', 'Iop_Shl32', 'Iop_Shl64'):
+            if args[1].const is not None and args[1].const >= 0:
+                const0 = args[0].const
+                const1 = 2**args[1].const
+                mul0 = args[0].strides
+                mul1 = ((None, const1))
+
         if sign is not None:
             assert size == args[0].size == args[1].size
-            addend0 = args[0].const
-            addend1 = args[1].const
-            if addend0 is not None and addend1 is not None:
-                const = addend0 + addend1 * sign
+            stride0 = args[0].strides
+            stride1 = args[1].strides
+            strideC = Counter()
+            if stride0:
+                for key, n in stride0:
+                    strideC[key] += n
             else:
-                const = None
+                strideC[args[0]] += 1
+            if stride1:
+                for key, n in stride1:
+                    strideC[key] += n * sign
+            else:
+                strideC[args[1]] += sign
+
             neg1 = args[1]
             if sign == -1:
-                neg1 = neg1.appended(NegOp(), neg1.size, -addend1 if addend1 is not None else None)
-            input0 = args[0].appended(ConstOffsetOp(addend1 * sign) if addend1 is not None else VarOffsetOp(neg1), size)
-            input1 = neg1.appended(ConstOffsetOp(addend0) if addend0 is not None else VarOffsetOp(args[0]), size)
-            result = input0.unioned(input1, size, const)
+                neg1 = neg1.appended(self.codeloc, NegOp(), neg1.size, tuple((k, -n) for k, n in neg1.strides))
+
+            input0 = args[0]
+            if stride1:
+                for (key, stride) in stride1:
+                    if key is None:
+                        input0 = input0.appended(self.codeloc, ConstOffsetOp(stride), size)
+                    else:
+                        input0 = input0.appended(self.codeloc, StrideOffsetOp(abs(stride)), size)
+            else:
+                input0 = input0.appended(self.codeloc, VarOffsetOp(neg1), size)
+
+            input1 = args[1]
+            if stride0:
+                for (key, stride) in stride0:
+                    if key is None:
+                        input1 = input1.appended(self.codeloc, ConstOffsetOp(stride), size)
+                    else:
+                        input1 = input1.appended(self.codeloc, StrideOffsetOp(abs(stride)), size)
+            else:
+                input1 = input1.appended(self.codeloc, VarOffsetOp(args[0]), size)
+
+            result = input0.unioned(self.codeloc, input1, size, tuple((key, n) for key, n in strideC.items() if n != 0))
+
+        elif mul0 is not None and mul1 is not None:
+            if const0 is not None and const1 is not None:
+                result = self.new_null(size, strides=((None, const0 * const1),))
+            elif const1 is not None and len(mul0) != 0:
+                result = self.new_null(size, strides=tuple((key, v * const1) for key, v in mul0))
+            elif const0 is not None and len(mul1) != 0:
+                result = self.new_null(size, strides=tuple((key, v * const0) for key, v in mul1))
+            elif const0 is not None:
+                result = self.new_null(size, strides=((args[1], const0),))
+            elif const1 is not None:
+                result = self.new_null(size, strides=((args[0], const1),))
+            else:
+                result = self.new_null(size)
         else:
-            result = LiveData.new_null(size)
+            result = self.new_null(size)
 
         return result
 
     def _handle_vex_expr_GSPTR(self, expr: pyvex.expr.GSPTR):
-        return LiveData.new_null(get_type_size_bytes(expr.result_type(self.irsb.tyenv)))
+        return self.new_null(get_type_size_bytes(expr.result_type(self.irsb.tyenv)))
 
     def _handle_vex_expr_VECRET(self, expr: pyvex.expr.VECRET):
-        return LiveData.new_null(get_type_size_bytes(expr.result_type(self.irsb.tyenv)))
+        return self.new_null(get_type_size_bytes(expr.result_type(self.irsb.tyenv)))
 
     def _handle_vex_expr_Binder(self, expr: pyvex.expr.Binder):
-        return LiveData.new_null(get_type_size_bytes(expr.result_type(self.irsb.tyenv)))
+        return self.new_null(get_type_size_bytes(expr.result_type(self.irsb.tyenv)))
 
 
     def _handle_vex_stmt_IMark(self, stmt: pyvex.stmt.IMark):
@@ -166,7 +232,7 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
 
     def _perform_vex_stmt_Put(self, offset: LiveData, data: LiveData, **kwargs):
         if type(offset.const) is not int:
-            return LiveData.new_null(data.size)
+            return self.new_null(data.size)
         name = self.project.arch.register_size_names[(offset.const, data.size)]  # unsafe
         return self.put(name, data, offset.const)
 
@@ -177,7 +243,7 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
         slot_info = self.project.arch.get_base_register(offset, data.size)
         if slot_info is None:
             l.error("??????? (%s, %s)", offset, data.size)
-            return LiveData.new_null(data.size)
+            return self.new_null(data.size)
         slot_name = self.project.arch.register_size_names[slot_info]
         reg_atom = RegisterAtom(self.codeloc, data.size, name, slot_name)
         self.blockinfo.atoms.append(reg_atom)
@@ -199,15 +265,15 @@ class TypeTapperEngine(angr.engines.vex.VEXMixin):
     def store(self, addr, data, endness):
         prop = Prop()
         prop.self_data[DataKind.Pointer] += 1
-        prop.struct_data[0][data.size][DataKind.GenericData] += 1
+        prop.struct_data[(0, data.size, DataKind.GenericData)] += 1
         addr.prop(prop, self.graph)
 
         mem_atom = MemoryAtom(self.codeloc, data.size, endness)
         self.blockinfo.atoms.append(mem_atom)
-        addr.appended(DerefOp(data.size), data.size).commit(mem_atom, self.graph)
+        addr.appended(self.codeloc, DerefOp(data.size), data.size).commit(mem_atom, self.graph)
         data.commit(mem_atom, self.graph)
 
     def _perform_vex_stmt_Dirty_call(self, func_name, ty, args, func=None):
         if ty is None:
             return None
-        return LiveData.new_null(get_type_size_bytes(ty))
+        return self.new_null(get_type_size_bytes(ty))

typetapper/hierarchy_graph.py

@@ -1,4 +1,5 @@
 from typing import Set, Union, TYPE_CHECKING, List, Optional, Dict, Iterable, Tuple, Callable, Any
+import copy
 from itertools import pairwise, chain
 from collections import defaultdict
 
@@ -41,6 +42,10 @@ class HierarchicalGraph(RelativeAtomGraph):
             else:
                 assert isinstance(node, RelativeAtomGroup)
                 assert node is self._root_group or node.parent is self._root_group or node.parent in self.__graph.nodes
+                prop = Prop()
+                for child in node.children:
+                    prop.update(self.prop(child))
+                assert prop == node.prop
 
         pool = set(self.__graph.edges)
         while pool:
@@ -129,8 +134,8 @@ class HierarchicalGraph(RelativeAtomGraph):
         else:
             yield node
 
-    def _prop_propagate(self, node: RelativeAtomOrGroup, add: bool):
-        prop = self.prop(node)
+    def _prop_propagate(self, node: RelativeAtomOrGroup, add: bool, prop: Optional[Prop]=None):
+        prop = self.prop(node) if prop is None else prop
         for parent in self._ancestry(node):
             if add:
                 parent.prop.update(prop)
@@ -138,11 +143,17 @@ class HierarchicalGraph(RelativeAtomGraph):
                 parent.prop.subtract(prop)
 
     def _add_node(self, relatom: RelativeAtom, path: OpSequence, has_gone_down: bool) -> bool:
-        res = super()._add_node(relatom, path, has_gone_down)
-        if res:
+        if relatom in self.__graph.nodes:
+            old_prop = copy.deepcopy(self.prop(relatom))
+        else:
+            old_prop = None
             self._atom_parents[relatom] = self._current_group
             self._current_group.children.add(relatom)
             self.__graph.add_node(relatom)
+        res = super()._add_node(relatom, path, has_gone_down)
+        if res:
+            if old_prop is not None:
+                self._prop_propagate(relatom, False, old_prop)
             self._prop_propagate(relatom, True)
         self.check_invariants()
         return res
@@ -188,6 +199,7 @@ class HierarchicalGraph(RelativeAtomGraph):
         # if item is a group, all edges will have two sides
         # if item is an atom, all edges will have one side
         if isinstance(item, RelativeAtom):
+            # if there is a self-loop, it will appear once from each side
             yield from ((None, (item, succ, key)) for succ, keys in self.__graph.succ[item].items() for key in keys)
             yield from (((pred, item, key), None) for pred, keys in self.__graph.pred[item].items() for key in keys)
         else:
@@ -260,14 +272,13 @@ class HierarchicalGraph(RelativeAtomGraph):
                 self.__graph.edges[new_edge][prev_next[outward]] = even_further_edge
                 if even_further_edge: self.__graph.edges[even_further_edge][prev_next[not outward]] = new_edge
 
-        self._prop_propagate(item, False)
         new_parent.children.add(item)
         parent.children.remove(item)
         if isinstance(item, RelativeAtomGroup):
             item.parent = new_parent
         else:
             self._atom_parents[item] = new_parent
-        self._prop_propagate(item, True)
+        parent.prop.subtract(self.prop(item))
         self.check_invariants()
 
     def move_node_in(self, item: RelativeAtomOrGroup, new_parent: RelativeAtomGroup):
@@ -302,6 +313,10 @@ class HierarchicalGraph(RelativeAtomGraph):
                 outward = False
             further_edge = self.__graph.edges[outer_edge][prev_next[outward]]
 
+            if outer == item:
+                # self loop!
+                continue
+
             # we only need to break the outer edge
             self.__graph.remove_edge(*outer_edge)
             # is this a collapse or an expand operation?
@@ -337,14 +352,13 @@ class HierarchicalGraph(RelativeAtomGraph):
                 self.__graph.edges[new_edge][prev_next[outward]] = even_further_edge
                 if even_further_edge: self.__graph.edges[even_further_edge][prev_next[not outward]] = new_edge
 
-        self._prop_propagate(item, False)
         new_parent.children.add(item)
         parent.children.remove(item)
         if isinstance(item, RelativeAtomGroup):
             item.parent = new_parent
         else:
             self._atom_parents[item] = new_parent
-        self._prop_propagate(item, True)
+        new_parent.prop.update(self.prop(item))
         self.check_invariants()
 
     def move_node(self, node: RelativeAtomOrGroup, new_parent: RelativeAtomGroup):

typetapper/hierarchy_graph_view.py

@@ -1,4 +1,4 @@
-from itertools import chain
+from itertools import chain, pairwise
 from typing import Tuple, Optional, Union, Dict, List, Set
 from collections import defaultdict
 from math import sqrt, sin, cos, pi
@@ -8,9 +8,10 @@ from threading import Lock
 import logging
 
 from PySide6.QtCore import QRectF, QPointF, QTimeLine, QEasingCurve, QMarginsF, QTimer, Qt, QEvent
-from PySide6.QtGui import QColor, QFont, QBrush, QPainterPath, QPen, QAction, QShortcut
+from PySide6.QtGui import QColor, QFont, QBrush, QPainterPath, QPen, QAction, QShortcut, QMouseEvent
 from PySide6.QtWidgets import QGraphicsItem, QGraphicsRectItem, QGraphicsSimpleTextItem, QHBoxLayout, QGraphicsScene, \
-    QGraphicsEllipseItem, QGraphicsSceneMouseEvent, QMenu, QDialog, QVBoxLayout, QLineEdit, QInputDialog, QMessageBox
+    QGraphicsEllipseItem, QGraphicsSceneMouseEvent, QMenu, QDialog, QVBoxLayout, QLineEdit, QInputDialog, QMessageBox, \
+    QGraphicsLineItem, QGraphicsTextItem
 
 import networkx
 import pygraphviz
@@ -40,6 +41,7 @@ PLUS_ICON_COLOR = QColor(0xff, 0xff, 0xff)
 PLUS_HOVERED_COLOR = QColor(0x40, 0xc0, 0xc0)
 PLUS_BORDER_COLOR = QColor(0x30, 0xc0, 0x50)
 PLUS_ICON_FONT = QFont("default", 12)
+ARC_COLOR = QColor(0xff, 0xff, 0xff)
 
 SCENE_MARGIN = 200
 DPI = 72
@@ -104,6 +106,7 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
         self._layout_animation = QTimer()
         self._layout_animation_controller = QTimer()
         self._mouse_is_down = False
+        self._rapid_expand = False
         self._selection_event_occurring = False
         self._layout_event_occurring = False
         self.faux_frontier = faux_frontier or set()
@@ -211,15 +214,17 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
     def begin_expand(self, model: RelativeAtomOrGroup):
         self.expansion_models = [adj for adj in set(chain(self.ng.succ[model], self.ng.pred[model])) if adj in self.faux_frontier]
         if not self.expansion_models:
-            l.error("How'd you do that")
             return
+        for other_model in self.expansion_models:
+            self._do_expand(other_model)
+        self._layout(0)
 
         qnode = self.qnodes[self.ug_reverse[model]]
         self.scene().clearSelection()
         qnode.setSelected(True)
         for other_qnode in self.qnodes:
             other_qnode.setAcceptHoverEvents(False)
-            other_qnode.setAcceptedMouseButtons(Qt.MouseButton.NoButton | 0)
+            other_qnode.setAcceptedMouseButtons(Qt.MouseButtons.NoButton)
             if qnode is other_qnode:
                 other_qnode.setOpacity(0.8)
             else:
@@ -239,7 +244,7 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
             qnode,
             other_qnode,
             other_model in self.ng.succ[model],
-            other_model in self.ng.pred[model]
+            other_model in self.ng.pred[model],
         ) for other_qnode, other_model in zip(self.expansion_qnodes, self.expansion_models)]
 
         tmp_ag = pygraphviz.AGraph()
@@ -257,7 +262,7 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
             self.scene().addItem(other_qnode)
             other_qnode.center = parse_pos(tmp_ag.get_node(idx).attr['pos']) - origin + qnode.center
             other_qnode.enter()
-            other_qnode.set_expandable(False)
+            other_qnode.set_expandable(self.can_expand(other_qnode.model))
 
         for qedge in self.expansion_qedges:
             self.scene().addItem(qedge)
@@ -273,7 +278,7 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
     def end_expand(self, accepted_qnode: Optional['PropChartHG']=None):
         for qnode in self.qnodes:
             qnode.setAcceptHoverEvents(True)
-            qnode.setAcceptedMouseButtons(Qt.MouseButton.AllButtons | 0)
+            qnode.setAcceptedMouseButtons(Qt.MouseButtons.AllButtons)
             qnode.setOpacity(1)
         for edge in self.qedges:
             edge.setOpacity(1)
@@ -298,22 +303,8 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
         self.expansion_qnodes.clear()
         self.expansion_models.clear()
 
-        if accepted_qnode:
+        if accepted_qnode is not None and accepted_qnode.model in self.faux_frontier:
             self.faux_frontier.remove(accepted_qnode.model)
-            existing_groups = [(g, self.label(g)) for g in self.current_group.children if isinstance(g, RelativeAtomGroup)]
-            existing_groups = {
-                self.instance.kb.functions[lbl].addr: g
-                for (g, lbl) in existing_groups
-                if lbl in self.instance.kb.functions
-            }
-            newgroups = self.hg.expand_by_key(accepted_qnode.model, self.current_group.am_obj, self.hg.kp.atom_to_function, dict(existing_groups))
-            for k, g in newgroups.items():
-                if k in existing_groups:
-                    continue
-                if k in self.hg.kp.kb.functions and self.hg.kp.kb.functions[k].alignment:
-                    self.ignore.add(g)
-                else:
-                    self.faux_frontier.add(g)
             self._layout(0)
 
     def set_drop_target(self, item: Optional['PropChartHG']):
@@ -439,6 +430,22 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
 
     # private interfaces
 
+    def _do_expand(self, model: RelativeAtomOrGroup):
+        existing_groups = [(g, self.label(g)) for g in self.current_group.children if isinstance(g, RelativeAtomGroup)]
+        existing_groups = {
+            self.instance.kb.functions[lbl].addr: g
+            for (g, lbl) in existing_groups
+            if lbl in self.instance.kb.functions
+        }
+        newgroups = self.hg.expand_by_key(model, self.current_group.am_obj, self.hg.kp.atom_to_function, dict(existing_groups))
+        for k, g in newgroups.items():
+            if k in existing_groups:
+                continue
+            if k in self.hg.kp.kb.functions and self.hg.kp.kb.functions[k].alignment:
+                self.ignore.add(g)
+            else:
+                self.faux_frontier.add(g)
+
     def _rekey_externs(self, old_qnode: 'PropChartHG', new_qnode: 'PropChartHG'):
         old_extern_model = ExternNode(old_qnode.model)
         new_extern_model = ExternNode(new_qnode.model)
@@ -646,12 +653,27 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
     def viewportEvent(self, event):
         if event.type() == QEvent.Type.ContextMenu:
             return False
+        elif isinstance(event, QMouseEvent) and event.type() == QMouseEvent.Type.MouseButtonPress:
+            if event.button() == Qt.MouseButton.LeftButton:
+                self._mouse_is_down = True
+
+            if (event.modifiers() & Qt.ShiftModifier) == Qt.NoModifier:
+                self.setDragMode(QZoomableDraggableGraphicsView.ScrollHandDrag)
+            else:
+                self.setDragMode(QZoomableDraggableGraphicsView.RubberBandDrag)
+                self.viewport().setCursor(Qt.CursorShape.CrossCursor)
         return super().viewportEvent(event)
 
     def keyPressEvent(self, event):
         if event.key() == Qt.Key_Z and not self.is_expanding:
             self._layout_animation.start()
             event.accept()
+        elif event.key() == Qt.Key_X:
+            if event.modifiers() & Qt.KeyboardModifier.ControlModifier:
+                self._rapid_expand ^= True
+            else:
+                self._rapid_expand = True
+            event.accept()
         else:
             super().keyPressEvent(event)
 
@@ -659,6 +681,12 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
         if event.key() == Qt.Key_Z:
             self._layout_animation.stop()
             event.accept()
+        elif event.key() == Qt.Key_X:
+            if event.modifiers() & Qt.KeyboardModifier.ControlModifier:
+                self._rapid_expand ^= True
+            else:
+                self._rapid_expand = False
+            event.accept()
         super().keyReleaseEvent(event)
 
     def mouseDoubleClickEvent(self, event):
@@ -670,16 +698,12 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
                 self.current_group.am_event()
 
     def mousePressEvent(self, event):
-        self._mouse_is_down = True
-        if event.modifiers() & Qt.ShiftModifier == 0:
-            self.setDragMode(QZoomableDraggableGraphicsView.ScrollHandDrag)
-        else:
-            self.setDragMode(QZoomableDraggableGraphicsView.RubberBandDrag)
-            self.viewport().setCursor(Qt.CursorShape.CrossCursor)
         super(QZoomableDraggableGraphicsView, self).mousePressEvent(event)  # lol. lmao even
 
     def mouseReleaseEvent(self, event):
-        self._mouse_is_down = False
+        if event.button() == Qt.MouseButton.LeftButton:
+            self._mouse_is_down = False
+
         if self.drop_target is not None:
             self.confirm_drop()
         super().mouseReleaseEvent(event)
@@ -721,6 +745,10 @@ class HierarchicalGraphWidget(QZoomableDraggableGraphicsView):
             self._selection_event_occurring = False
 
     def _on_selection_changed_internal(self):
+        if not self.items():
+            # don't do anything if we are closing the pane (there will be a desync between items() and qnodes/qedges)
+            return
+
         if self.is_expanding:
             items = self.scene().selectedItems()
             if not items:
@@ -781,7 +809,7 @@ class HGNode(AnimatableItem):
 
     def exit(self, duration=250):
         self.exiting = True
-        for edge in self.edges.values():
+        for edge in list(self.edges.values()):
             edge.orient(self, away=True)
             edge.exit(duration)
             edge.start.edges.pop(self.model, None)
@@ -849,7 +877,7 @@ class HGNode(AnimatableItem):
 
 
 class PropChart(QGraphicsItem):
-    def __init__(self, parent, max_width=200., default_unit=10., byte_height=10., margin_x=15., margin_y=5., padding_left=5.):
+    def __init__(self, parent, max_width=200., default_unit=10., byte_height=10., margin_x=15., margin_y=5., padding_left=5., gap_size=15., tick_granularity=4):
         self.max_width = max_width
         self.default_unit = default_unit
         self.unit = default_unit
@@ -857,6 +885,8 @@ class PropChart(QGraphicsItem):
         self.margin_x = margin_x
         self.margin_y = margin_y
         self.padding_left = padding_left
+        self.gap_size = gap_size
+        self.tick_granularity = tick_granularity
         self.width = 0.
         self.height = 0.
         self.objects = []
@@ -876,67 +906,114 @@ class PropChart(QGraphicsItem):
     def paint(self, painter, option, widget=...):
         pass
 
+    def layout(self):
+        raise NotImplementedError
+
     def _layout_marks(self):
         self.objects.clear()
         for item in list(self.childItems()):
             self.scene().removeItem(item)
 
-        data = [(offset, kind, size, count) for offset, v1 in self.prop.struct_data.items() for size, v2 in v1.items() for kind, count in v2.items() if count]
+        # layout vertical position mapping
+        offsets_set = {offset + suboffset for (offset, size, kind), count in self.prop.struct_data.items() if count for suboffset in range(size)}
+        offsets_set.update(o for offsets in self.prop.unifications for o in offsets)
+        first_offset = min(offsets_set, default=-1)
+        ypos_mapping = {first_offset: self.margin_y}
+        ypos_max = self.margin_y
+        ticks = []
+        gaps = []
+        run_start = first_offset
+        if first_offset % self.tick_granularity == 0:
+            ticks.append(first_offset)
+
+        for last_offset, offset in pairwise(sorted(offsets_set)):
+            if offset - last_offset > self.tick_granularity:
+                if len(ticks) == 0 or ticks[-1] < run_start:
+                    ticks.append(run_start)
+
+                run_start = offset
+                gaps.append(offset)
+                ypos_mapping[offset] = ypos_mapping[last_offset] + self.byte_height + self.gap_size
+
+                if offset % self.tick_granularity == 0:
+                    ticks.append(offset)
+            else:
+                ypos_mapping[offset] = ypos_mapping[last_offset] + (offset - last_offset) * self.byte_height
+
+                for suboffset in range(offset, last_offset, -1):
+                    if suboffset % self.tick_granularity == 0:
+                        ticks.append(suboffset)
+                        if suboffset != offset:
+                            ypos_mapping[suboffset] = ypos_mapping[offset] - (offset - suboffset) * self.byte_height
+
+            ypos_max = ypos_mapping[offset] + self.byte_height
+
+        if offsets_set and (len(ticks) == 0 or ticks[-1] < run_start):
+            ticks.append(run_start)
+
+        # layout boxes
+        data = [(offset, kind, size, count) for (offset, size, kind), count in self.prop.struct_data.items() if count]
         data.sort(key=lambda x: (x[0], x[1], -x[2], x[3]))
         cols_allocated = defaultdict(int)
-        row_allocated = 0
-        offset_allocated = None
-        marks = []
-        ticks = []
+        boxes = []
         for offset, kind, size, count in data:
             xpos = max(cols_allocated[suboffset] for suboffset in range(offset, offset + size))
             for suboffset in range(offset, offset + size):
                 cols_allocated[suboffset] = xpos + count
 
-            if offset_allocated is None or offset >= offset_allocated:
-                ypos = row_allocated
-                row_allocated += size
-                offset_allocated = offset + size
-                ticks.append((offset, ypos))
-            else:
-                ypos = row_allocated - (offset_allocated - offset)
-                stretch = max(0, offset + size - offset_allocated)
-                row_allocated += stretch
-                offset_allocated += stretch
-
-            marks.append((offset, kind, size, count, xpos, ypos))
+            boxes.append((offset, kind, size, count, xpos))
 
         total_width = max(cols_allocated.values()) if cols_allocated else 0
 
         if total_width * self.unit > self.max_width:
             self.unit = self.max_width / total_width
 
+        # create base object
         self.width = self.max_width + 2 * self.margin_x + self.padding_left
-        self.height = row_allocated * self.byte_height + 2 * self.margin_y
+        self.height = ypos_max + self.margin_y
         box = QGraphicsRectItem(QRectF(0, 0, self.width, self.height), self)
         box.setBrush(BOX_COLOR)
         box.setPen(QPen(BOX_BORDER_COLOR, BOX_BORDER_WIDTH))
         self.objects.append(box)
         self.object_bg = box
 
-        for offset, kind, size, count, xpos, ypos in marks:
-            rect = QGraphicsRectItem(QRectF(self.margin_x + self.padding_left + xpos * self.unit, self.margin_y + ypos * self.byte_height, count * self.unit, size * self.byte_height), self)
+        # create box objects
+        for offset, kind, size, count, xpos in boxes:
+            rect = QGraphicsRectItem(QRectF(self.margin_x + self.padding_left + xpos * self.unit, ypos_mapping[offset], count * self.unit, size * self.byte_height), self)
             rect.setBrush(kind_to_color(kind))
             rect.setPen(QColor(0x10, 0x10, 0x10))
             self.objects.append(rect)
 
+        # create tick objects
         tick_width = 0
-        for offset, ypos in ticks:
-            tick = QGraphicsSimpleTextItem(str(offset), self)
+        max_tick_ypos = 0
+        for offset in ticks:
+            tick = QGraphicsSimpleTextItem(format(offset, 'x'), self)
             tick.setBrush(TEXT_COLOR)
             tick.setFont(TEXT_FONT)
-            width = tick.boundingRect().width()
+            rect = tick.boundingRect()
+            width = rect.width()
             tick_width = max(width, tick_width)
-            tick.setPos(QPointF(self.margin_x - width, self.margin_y + ypos * self.byte_height))
+            tick.setPos(QPointF(self.margin_x - width, ypos_mapping[offset] - rect.height() / 2 + self.byte_height / 2))
+            max_tick_ypos = max(rect.height() + tick.y(), max_tick_ypos)
             self.objects.append(tick)
 
-        if tick_width > self.margin_x:
-            shift = tick_width - self.margin_x
+            ticktick = QGraphicsLineItem(self.margin_x + self.padding_left / 2, ypos_mapping[offset], self.margin_x + self.padding_left, ypos_mapping[offset], self)
+            self.objects.append(ticktick)
+
+        # create squiggle objects
+        for gap in gaps:
+            squiggle = SquiggleMark(self.margin_x + self.padding_left - 1.5, ypos_mapping[gap] - 12.5, self)
+            self.objects.append(squiggle)
+
+        # create arc objects
+        for offset1, offset2 in self.prop.unifications:
+            arc = UnificationArc(self.margin_x + self.padding_left, ypos_mapping[offset1], ypos_mapping[offset2], self)
+            self.objects.append(arc)
+
+        # if during rendering ticks we ran out of space horizontally, expand (and shift everything to the right of it)
+        if tick_width > self.margin_x - self.padding_left:
+            shift = tick_width - (self.margin_x - self.padding_left)
             for obj in self.objects:
                 if obj is self.object_bg:
                     rect = obj.rect()
@@ -946,12 +1023,23 @@ class PropChart(QGraphicsItem):
                     obj.setX(obj.x() + shift)
             self.width += shift
 
+        # if during rendering ticks we ran out of space vertically, expand
+        if max_tick_ypos + self.margin_y > self.height:
+            shift = max_tick_ypos + self.margin_y - self.height
+            rect = self.object_bg.rect()
+            rect.setHeight(rect.height() + shift)
+            self.object_bg.setRect(rect)
+            self.height += shift
+
 class PropChartHG(HGNode, PropChart):
     def __init__(self, parent, **kwargs):
         self.object_label = None
         self.object_btn: Optional[PlusButton] = None
         super().__init__(parent, **kwargs)
 
+        self.setAcceptHoverEvents(True)
+        self.setAcceptedMouseButtons(Qt.MouseButtons.AllButtons)
+
     @property
     def label(self):
         return self.hgw.label(self.model)
@@ -973,8 +1061,17 @@ class PropChartHG(HGNode, PropChart):
         self.setY(v.y() - self.height / 2)
 
     def layout(self):
-        lbl_height = 20
         self._layout_marks()
+        self.object_label = QGraphicsTextItem(self.label, self)
+        self.object_label.setTextWidth(self.width - self.margin_x * 2)
+        self.object_label.setPos(QPointF(self.margin_x, self.margin_y))
+        self.object_label.setFont(TEXT_FONT)
+        self.object_label.setDefaultTextColor(TEXT_COLOR)
+        self.objects.append(self.object_label)
+        self.object_btn = PlusButton(self, self.begin_expand, 16)
+        self.object_btn.setPos(self.width - 18, 2)
+        lbl_height = self.object_label.boundingRect().height()
+
         self.height += lbl_height
         for obj in self.objects:
             if obj is self.object_bg:
@@ -982,16 +1079,13 @@ class PropChartHG(HGNode, PropChart):
                 rect = obj.rect()
                 rect.setHeight(obj.rect().height() + lbl_height)
                 obj.setRect(rect)
+            elif obj is self.object_label:
+                pass
+            elif obj is self.object_btn:
+                pass
             else:
                 obj.setY(obj.y() + lbl_height)
 
-        self.object_label = QGraphicsSimpleTextItem(self.label, self)
-        self.object_label.setPos(QPointF(self.margin_x, self.margin_y))
-        self.object_label.setFont(TEXT_FONT)
-        self.objects.append(self.object_label)
-        self.object_btn = PlusButton(self, self.begin_expand, 16)
-        self.object_btn.setPos(self.width - 18, 2)
-
         self.setTransformOriginPoint(self.center - self.pos())
 
     def set_expandable(self, expandable: bool):
@@ -999,6 +1093,7 @@ class PropChartHG(HGNode, PropChart):
 
     def begin_expand(self):
         if self.timeline.state() == QTimeLine.NotRunning:
+            self.hgw.end_expand(self)
             self.hgw.begin_expand(self.model)
             return True
         else:
@@ -1052,8 +1147,13 @@ class PropChartHG(HGNode, PropChart):
                     break
             else:
                 self.hgw.set_drop_target(None)
+
         super().mouseMoveEvent(event)
 
+    def hoverEnterEvent(self, event):
+        if self.hgw._rapid_expand:
+            self.begin_expand()
+
     def emphasize(self, duration=100):
         self._start_animation(self._animate_emphasize, duration)
 
@@ -1274,6 +1374,58 @@ class PlusButton(QGraphicsItem):
         self.object_bg.setBrush(PLUS_COLOR)
         super().hoverLeaveEvent(event)
 
+class UnificationArc(QGraphicsItem):
+    def __init__(self, xpos, ypos1, ypos2, parent, bent=10., thickness=3.):
+        super().__init__(parent)
+        self.setPos(xpos, ypos1)
+        self.height = ypos2 - ypos1
+        self.bent = bent
+        self.thickness = thickness
+        self.setOpacity(0.5)
+        self.setAcceptHoverEvents(True)
+
+    def shape(self):
+        result = QPainterPath()
+        result.moveTo(0, 0)
+        result.lineTo(-self.thickness, 0)
+        result.cubicTo(-self.bent, self.height / 2, -self.bent, self.height / 2, -self.thickness, self.height)
+        result.lineTo(0, self.height)
+        result.cubicTo(-self.bent + self.thickness, self.height / 2, -self.bent + self.thickness, self.height / 2, 0, 0)
+        result.closeSubpath()
+        return result
+
+    def paint(self, painter, option, widget=...):
+        painter.fillPath(self.shape(), ARC_COLOR)
+
+    def boundingRect(self):
+        return QRectF(-self.bent, 0, self.bent, self.height)
+
+    def hoverEnterEvent(self, event):
+        self.setOpacity(1)
+
+    def hoverLeaveEvent(self, event):
+        self.setOpacity(0.5)
+
+class SquiggleMark(QGraphicsItem):
+    def __init__(self, xpos, ypos, parent, nlines=5, dx=3., dy=2., thickness=1.):
+        super().__init__(parent)
+
+        self.setPos(xpos, ypos)
+        self.nlines = nlines
+        self.dx = dx
+        self.dy = dy
+        self.thickness = thickness
+
+    def paint(self, painter, option, widget=...):
+        line = QPainterPath()
+        line.moveTo(0, 0)
+        for i in range(self.nlines):
+            line.lineTo(self.dx if i % 2 == 0 else 0, self.dy * (i + 1))
+        painter.strokePath(line, QPen(TEXT_COLOR, self.thickness))
+
+    def boundingRect(self):
+        return QRectF(-self.thickness, -self.thickness, self.dx + self.thickness*2, self.dy * self.nlines + self.thickness*2)
+
 def arrowhead_vectors(start: QPointF, end: QPointF, stem_width: float, flare_width: float) -> Tuple[QPointF, QPointF, QPointF, QPointF, QPointF, QPointF]:
     direction = end - start
     norm = normalized(direction)

typetapper/plugin.py

@@ -1,6 +1,8 @@
+import threading
 from typing import Optional
 
 from angr.analyses.decompiler.structured_codegen.c import CVariable
+from angrmanagement.data.jobs.job import Job
 from angrmanagement.plugins import BasePlugin
 from angrmanagement.ui.views import CodeView
 
@@ -15,14 +17,12 @@ class TypeTapper(BasePlugin):
         super().__init__(workspace)
         self.kp: Optional[TypeTapperManager] = None
 
-    MENU_BUTTONS = ["TypeTapper initial analysis"]
+    def handle_project_initialization(self):
+        threading.Thread(target=self._startup).start()
 
-    def handle_click_menu(self, idx):
-        assert idx == 0
-        cfg = self.workspace.main_instance.cfg
-        tt = self.workspace.main_instance.project.analyses[TypeTapperAnalysis](cfg)
-        self.kp = tt.manager
-        self._start(self.kp.lookup_reg(0x4144a5, "rdi"))
+    def _startup(self):
+        self.workspace.main_instance.join_all_jobs(0.200)
+        self.workspace.main_instance.add_job(TypeTapperStartupJob(self))
 
     def build_context_menu_node(self, node):
         # this is bad lol
@@ -56,3 +56,13 @@ class TypeTapper(BasePlugin):
         )
         self.workspace.add_view(view)
         self.workspace.raise_view(view)
+
+class TypeTapperStartupJob(Job):
+    def __init__(self, plugin: TypeTapper, **kwargs):
+        super().__init__("TypeTapper", **kwargs)
+        self.plugin = plugin
+
+    def _run(self, inst):
+        cfg = inst.cfg
+        tt = inst.project.analyses[TypeTapperAnalysis](cfg, progress_callback=self._progress_callback)
+        self.plugin.kp = tt.manager

typetapper/procedures.py

@@ -3,9 +3,9 @@ from typing import List, Optional
 import networkx
 
 from angr.calling_conventions import SimCC, SimFunctionArgument, SimRegArg, SimStackArg
-from angr.sim_type import SimType, parse_signature, SimTypeFunction, SimTypeBottom
+from angr.sim_type import parse_signature, SimTypeBottom
 
-from .data import BlockInfo, LiveData, CodeLoc, DataKind
+from .data import LiveData, DataKind
 from .engine import TypeTapperEngine
 
 class TypeTapperProcedure:
@@ -78,7 +78,8 @@ class TypeTapperProcedure:
         elif isinstance(loc, SimStackArg):
             return self.load(self._translate_stack_offset(loc), loc.size, self._engine.project.arch.memory_endness)
         else:
-            return LiveData.new_null(loc.size)
+            self._engine.stmt_idx += 1
+            return LiveData.new_null(self._engine.codeloc, loc.size)
 
     def _write_argloc(self, loc: SimFunctionArgument, value: LiveData):
         if isinstance(loc, SimRegArg):
@@ -92,7 +93,8 @@ class TypeTapperProcedure:
         args = [self._read_argloc(loc) for loc in arg_locs]
         result = self.run(args)
         if result is None and not isinstance(prototype.returnty, SimTypeBottom):
-            result = LiveData.new_null(prototype.returnty.size // 8)
+            self._engine.stmt_idx += 1
+            result = LiveData.new_null(self._engine.codeloc, prototype.returnty.size // 8)
         if result is not None:
             self._write_argloc(self.cc.return_val(prototype.returnty), result)
         if self._engine.project.arch.call_sp_fix:

typetapper/relative_graph.py

@@ -1,3 +1,4 @@
+import copy
 from typing import TYPE_CHECKING, List, Optional, Tuple, Set
 from dataclasses import dataclass
 import logging
@@ -25,7 +26,11 @@ class RelativeAtomAttrs:
     has_gone_down: bool = False
 
     def merge(self, other: 'RelativeAtomAttrs'):
+        changed = False
+        checkme = copy.deepcopy(self.prop)
         self.prop.maximize(other.prop)
+        if checkme != self.prop:
+            changed = True
         # TODO has_gone_down
         if self.path != other.path:
             # TODO unifications
@@ -34,8 +39,8 @@ class RelativeAtomAttrs:
             score_1 = sum(0 if isinstance(op, (ConstOffsetOp, RefOp, DerefOp)) else 1 for op in other.path.ops)
             if score_1 < score_0:
                 self.path = other.path
-                return True
-        return False
+                changed = True
+        return changed
 
 class RelativeAtomGraph:
     def __init__(self, kp: 'TypeTapperManager', baseline: List[Atom]):
@@ -57,7 +62,6 @@ class RelativeAtomGraph:
         If relatom is not present in the graph, add it.
         If it is present in the graph, merge the new information into its attrs
         """
-        from .data import RegisterAtom
         newattrs = RelativeAtomAttrs(
             prop=self.kp.graph.nodes[relatom.atom].get('prop', Prop()).transform(path.invert()),
             path=path,
@@ -116,9 +120,6 @@ class RelativeAtomGraph:
             edge_ops: OpSequence,
             is_pred: bool,
     ) -> Optional[RelativeAtom]:
-        #if str(relatom.atom) == 'rsp @ 0x4144a8' and str(succ) == 'MEM @ 0x4144d9':
-        #    import ipdb; ipdb.set_trace()
-
         weh = self._update_callstack(relatom.callstack, relatom.rcallstack, edge_cf, is_pred)
         if weh is None:
             return None