SimilarityDB: what makes it different

A summary of SimilarityDB's distinctive design decisions and features, collected from the repository (README, docs/, sql/, and the Rust crates featurizer/, engine/, frontend/, watermark/). Each section points to the source for details.

1. At a glance

sample ─► start.py ─► format extractors (PE/ELF/Mach-O, hashes, YARA, CAPA, DIE, malcontent, IOCs)
              │                                   └─► similaritydb_* tables
              └─ --decompile ─► IDA 9.5 + Hex-Rays (isolated child process)
                                   └─► IR (archived, the permanent record)
                                         └─► similaritydb-featurize (Rust) ─► code_ida_* (content / references)
                                                 └─► similaritydb-engine (Rust) ─► similarity_* tables
                                                         └─► similaritydb-web (Rust, no JS)

2. Technology choices and why

3. Storage model: content vs references (dedup + counts)

This is the central storage idea (docs/TOOLING.md, sql/code_ida_schema.sql).

Every code table family is split in two:

A library routine found in 10,000 samples is stored once in each content table and 10,000 times as a small reference row.

Families: decompiled (pseudocode), disassembled, low-level microcode, function features (v2), CFG, plus per-sample code_ida_sample_stats, code_ida_call_edges and code_ida_strings_by_binary.

Rule: a content key must hash every column of its row, so different functions can never overwrite each other. Anything that depends on addresses or on the sample (names, flags, callee addresses, string offsets) belongs on the references side.

Reference counts and prevalence

Because content and references are separate, "how common is this code?" is cheap to answer:

• Per-function reference count. The function page queries count() AS refs, uniqExact(sha256) AS samples, uniqExact(address) AS addrs, uniqExact(name) AS names for a disassembled_function_hash. It then lists every sample and address where the same function body appears, with the current sample first (frontend/src/functions.rs).
• Per-line prevalence. code_ida_pseudocode_line_stats is an AggregatingMergeTree fed through a Null table and a materialized view. For each normalized pseudocode line it keeps uniqState of the functions and samples it appears in, so re-ingestion is idempotent. The UI shades lines by heat ("seen in 24 of 24 samples (100 functions)") and has a common lines page (frontend/src/explore.rs).
• Storage / references admin page. Shows distinct content rows against reference rows per family, which is the dedup ratio (frontend/src/refs.rs).
• Document frequency in the engine. The same sharing counts drive IDF weighting, so common code counts less in lineage scoring (see §8).

Per-sample statistics and commit marker

code_ida_sample_stats records functions_total, functions_user, functions_library, functions_thunk, functions_external, functions_small, functions_decompiled, functions_failed, and name sources (names_symbol, names_flirt, names_goresym, names_dummy). It also records code coverage (exec_bytes, user_function_bytes, orphan_code_bytes, coverage_ratio), arch and platform, the packed flag, and every tool version (IDA, Hex-Rays, IR, featurizer).

It is written last and acts as the commit marker: a sample counts as analysed only when its newest row has status ok, so a half-finished export is retried. Re-ingesting a sample first deletes its old per-sample rows. Shared content rows are kept.

4. "Keep and flag", never filter

Every IDA function is exported with flags: type (USER / LIBRARY / THUNK / EXTERNAL / UNKNOWN), small, thunk, library (FLIRT, lib_verified), and name source. Nothing is dropped at extraction. Filtering decisions belong to the consumer. For example, the engine down-weights library code in lineage mode, but still reports library_matches.

5. Feature families (featurizer v1 → v2 → v3 + pseudocode analyses)

• Hashes at several abstraction levels: exact, normalized, PIC hash (IDA auto-names such as sub_/loc_ → ADDR), skeleton hashes, and pseudocode template hashes. Disassembly normalization has three levels: fully normalized, API-normalized and category-normalized (docs/CODE_ANALYSIS_APPROACH.md).
• MinHash / LSH: typed MinHash over both microcode levels, plus b-bit MinHash over dataflow strands and ctree subtrees.
• TLSH over disassembly and microcode. It is stored but deliberately not used as a matcher, because it hurt results in ablation.
• CFG / structure: BLAKE3 topology hash, MD-index (top-down / bottom-up), prime products, cyclomatic complexity E − N + 2P, loops, calls, Weisfeiler-Lehman features.
• Canonically renamed block hashes (MMAT_LVARS). These are architecture-neutral.
• API, constants and strings: normalized API sets, API-category MinHash, informative constants, strings with entropy, and IOC extraction.
• Call edges with thunk chains resolved.
• Eight opt-in pseudocode analyses (docs/pseudocode-analyses.md): Deckard-style AST vectors, statement winnowing, api_args (call argument shapes), types/structs, an idiom library (memcpy loops, XOR loops, CRC/base64 tables…), control dependence, dataflow (BSim/Esh-style strands), and value-set analysis. dataflow + api_args are on by default because they were chosen on the benchmark.
• Architecture-neutrality classification: every feature is labelled neutral, measured-not-neutral, or arch-specific (docs/featurizer-v2.md §8). Arch-specific features are only compared within one (processor, bits).

6. TLSH: computed and stored, deliberately not used for matching

TLSH (Trend Micro Locality Sensitive Hash) is a fuzzy digest built from bucketed counts of byte triplets in a sliding window. It comes with a distance score.

Implementation. featurizer/src/tlsh.rs is a pure-Rust port of the Apache-2.0 reference implementation, matching the py-tlsh 4.7.2 tlsh.hash() configuration: 128 buckets, 1-byte checksum, "T1" prefix, at least 50 bytes of input, non-conservative. Its output is byte-identical to the Python pipeline. The license note is in featurizer/NOTICE.

Where it is stored. Each digest is a Nullable(FixedString(72)) column:

The web UI shows all of them under "fuzzy hashes" on the function page (frontend/src/functions.rs).

Why it is not a matcher. - In the clustering prototype's ablation (docs/similarity-clustering-draft.md), a TLSH tier (tlsh_llil distance ≤ 30) carried 64% of inter-family matched weight but only 5% of intra-family weight. It raised the noise floor between unrelated packages from 0.005 to 0.043 (max 0.103), so it was switched off. - Function-level token strings are short and low-entropy, which is the worst case for this kind of digest. This agrees with Pagani et al. (2018) (docs/literature-review.md §3.4). - TLSH has no index: at a scale of 10⁹ samples it could only check candidate pairs that another tier already proposed, never generate them. The draft rejects adding more TLSH variants for this reason. - On the benchmark it is weak across architectures, for example tlsh_llil P@1 is 0.05 on the arch axis (bench/RESULTS.md). TLSH of disassembly is classed as arch-specific (docs/featurizer-v2.md §8). - The feature critique notes the redundancy: TLSH appears in three tables, and tlsh_llil_new duplicates tlsh_llil.

The literature review leaves one question open: whether TLSH could break ties for large functions only, where its statistics mean something.

7. Abstraction levels: from raw bytes to semantics

The idea that recurs across the whole design is to describe the same function at several levels of abstraction. Each level has its own hashes, and the engine prefers the most concrete level that still matches. A lower level is more precise; a higher level survives more change (compiler, flags, architecture, platform).

7.1 The IR ladder (what IDA exports)

7.2 Normalization inside each level

• Disassembly at three granularities (docs/CODE_ANALYSIS_APPROACH.md):
• Level 0, fully normalized: GPR, MEM, CONST, DATA_REF.
• Level 1, API normalized: GPR_DATA, MEM_STACK, CONST_LARGE, API_CreateFileW.
• Level 2, category normalized: REG_32, MEM_BASE_OFFSET, CONST_HEX, API_FILE_OP.
• Control-flow instructions are tagged <TARGET>.
• Exact → normalized → PIC hash: the PIC hash replaces IDA auto-names (sub_/loc_/…) with ADDR, drops short and collapses blanks. It is computed in SQL by the engine.
• Skeleton vs typed variants: skeleton keeps only the opcode sequence; typed adds operand kinds. Both exist for MinHash, TLSH and block hashes.
• Canonically renamed block hashes: variables and registers are renamed canonically, and an operand size equal to the pointer size becomes the token P, so 32- and 64-bit builds hash alike.
• Pseudocode templates: every local variable is renamed SDBL<i>_, which gives the template and template_skeleton_hash. These are also the lookup key for watermark detection. A normalized pseudocode hash drives the dnorm tier.
• Structural abstractions: CFG topology hash, MD-index, prime products, and ctree_control_hash (only the if / loop / switch nesting).
• Semantic abstractions (pseudocode analyses): AST characteristic vectors, dataflow strands, control-dependence paths, value sets, idiom tags, and API argument shapes.
• Platform-independent names and strings: API names lose __imp_, j_, leading underscores and stdcall decoration. String skeletons reduce letters to a and digits to 0 and keep format specifiers.

7.3 How the engine uses the ladder

Tier order puts concrete before abstract: exact → norm → dnorm → pic → bset → struct → MinHash tiers → blocks → consts → api → fused → callgraph. The greedy 1:1 assignment takes the best tier first. Each match records its tier and quality, and a separate neutral quality that counts only the arch-neutral levels. That neutral quality is what the cross-architecture scores (xarch_*) are built from.

8. Similarity engine: tiered, explainable matching

engine/README.md documents the engine.

• Tiered function matching through inverted indexes. It never compares all pairs of functions. Tiers, best first: exact, norm, dnorm (pseudocode), pic, bset (block-hash set), struct, mlil MinHash, minhash (llil), blocks, consts, api, fused, callgraph. Every match is tagged with the tier that produced it.
• Fused tier (v4). Sums five signals (ctree MinHash, dataflow b-bit MinHash, mlil MinHash, block Jaccard, api_args Jaccard). This raised cross-arch recall from 0.64 to 0.84 while precision stayed near 0.99.
• Call-graph propagation (BinDiff / VxClass style, deterministic). Unlike BinDiff, it requires evidence even for single candidates; dropping that requirement lowered precision from 0.89 to 0.76.
• Two scoring modes:
• lineage: IDF-weighted, library code excluded. Answers "same family?"
• reuse: all shared code at full weight. Answers "shared code?"
• Symmetric similarity and directional containment (contain_a_in_b, contain_b_in_a).
• Cross-architecture scores (xarch_*) estimate how much of the similarity would survive a recompile for another ISA.
• Arch gating: arch-specific tiers only propose pairs within the same arch.
• Flood-resistant document frequency: --df-unit sample | source | cluster. This stops an attacker or an unbalanced feed from hiding a family by uploading many near-copies.
• Clustering: threshold graph + union-find, plus an average-linkage (UPGMA) dendrogram built with the nearest-neighbour-chain algorithm. Cluster id = smallest sha256.
• Atomic runs: similarity_runs is written last, so a run is visible only when complete. A partial (--samples) run never replaces full-corpus results. similarity_latest_* views always point to the newest finished run.
• Every function match is stored (similarity_function_matches), with per-tier counts and weights for each sample pair.

9. Web UI: similaritydb-web

• Rust, server-rendered, no JavaScript. Navigation uses links and plain GET forms only, under a strict CSP.
• Pages: samples, sample detail (functions, similar samples, clusters, nearest neighbours, strings), function detail (highlighted pseudocode and disassembly, CFG as inline SVG, metrics, similar functions, reference list), search (hash → sample or function, names, ranking a sample's functions against all others), clusters with heat map and dendrogram, line heat shading, common lines, and a /help glossary.
• Security hardening: parameterized read-only queries through a dedicated read-only ClickHouse user with quota (sql/web_user.sql), whitelisted parameters, and generic error pages. The server is bounded: 8 workers, a queue of 64, 16 KiB request head, and per-IP limits of 10 req/s. The admin area uses PBKDF2-SHA256 (600k rounds) Basic auth with IP lockout, and refuses plaintext on non-loopback addresses.
• Analysis jobs are derived from the tables. There is no job-queue table.

10. Output watermarking (provenance)

watermark/ and docs/watermarking.md cover this.

• Pseudocode and disassembly are watermarked only when they leave the system, keyed by (deployment key, content hash). The same function always renders the same way for one deployment.
• Invariant: never watermark what is hashed. Stored content and all features keep the canonical IDA / Hex-Rays text.
• Leak detection: a leaked function is looked up by skeleton hash, the watermark bits are checked, and evidence across functions is combined into a p-value with Fisher's method.
• Self-contained crypto (SHA-256 / HMAC tested against NIST and RFC 4231 vectors). The only dependency is tinyjson.

11. Robustness and engineering discipline

• IDA isolation: IDA runs in a credential-free child process with a deadline enforced by the parent.
• Safe archive extraction: protects against zip slip and links, with member count and size limits.
• Packed samples are still decompiled, and flagged as packed.
• Fuzzing: cargo-fuzz targets for the watermark crate (9) and engine (7), run under ASan. Findings are fixed with regression tests (docs/FUZZING.md).
• Benchmark-driven tuning: 162 builds of Lua, zlib and cJSON across 9 targets, 3 opt levels, stripped and unstripped. Every tier is measured on opt, arch, platform and strip axes (bench/RESULTS.md), and engine thresholds were chosen from measured precision.
• Reference implementations: the Python featurizer is the specification for the Rust one, and scripts/cluster_prototype.py validates the engine.
• Self-critique and literature: docs/feature-design-critique.md (critique items are referenced in commit messages) and docs/literature-review.md.
• Scale projection (docs/performance.md): 10¹² functions ≈ 394,000 core-hours, about 16 days on 1,000 cores. The engine handles 4,000 synthetic samples in about 30 s on 12 threads.

12. Classic feature extraction (non-code)

Alongside code similarity, per-format extractors fill the similaritydb_* tables:

• PE: headers, imports, resources, overlay, sections, Authenticode signatures, .NET, Rich header, inconsistency and spoofing checks.
• ELF: segments, sections, dependencies, symbols, imports and exports, relocations, notes.
• Mach-O: segments, imports and exports, dylibs, code signature, similarity hashes.
• General: hashes, Detect-It-Easy, CAPA capabilities, YARA (YARA-Forge, precompiled .yarac), malcontent, IOC extraction, and Go metadata (GoReSym).

SimilarityDB: next moves (an engine-style projection)

This roadmap is planned the way a chess engine picks a move. It first evaluates the current position. Then it generates candidate moves, scores each one, and looks several moves ahead for the line that leaves the strongest position. Lines that lose material are pruned.

Every move has to be legal under the project's own rules (docs/literature-review.md §1.2):

• deterministic and explainable;
• no ML, no LLMs, no deobfuscation;
• architecture- and platform-neutral by default;
• the IR is the permanent record, because binaries are deleted;
• per-function cost O(n log n), at most about 64 bytes of storage per function per analysis;
• candidate generation never compares all pairs.

Tags used below: - [repo]: the repository already names this as a gap or open question. - [new]: a proposal that does not appear in the repository.

1. Evaluating the position

Strengths (material we have): - content / references storage, which gives deduplication and cheap prevalence counts; - the IR archive, so any feature can be recomputed without IDA; - 13 explainable matching tiers, with lineage and reuse modes, containment and cross-architecture scores; - a benchmark that covers optimization level, architecture, platform and stripping; - a hardened UI, watermarking and fuzzing.

Weaknesses (where the opponent attacks):

Evaluation: the position is strong. The project has excellent foundations and measurement, but three structural problems block the 10⁹ target: the batch engine, the missing band table, and per-sample IR. Precision has two cheap leaks: name leakage and CRT code. Fix the cheap leaks first. Then rebuild the foundation for scale, and only after that add features.

2. Candidate moves, scored

Columns: - Gain: expected value, 1 to 5. - Cost: effort, 1 (days) to 5 (months). - Risk: chance it doesn't pay off, low / mid / high. - Unlocks: the moves that become possible afterwards. In chess terms this is tempo: a move that opens many others is worth more than its own gain.

A. Forcing moves: cheap, immediate precision

B. Development: the foundation for scale

C. Better evidence: recall where we are weakest

D. Measurement: knowing what we've won

E. Attack: new capabilities for analysts

3. The principal variation: the line I would play

The search goes about 8 moves deep. Order matters: each move opens the next.

 1. A1  name-free normalizers           ── forcing: stops a precision leak
 2. A3  toolchain library flagging       ── forcing: unblocks Windows lineage
    (A2, A4, A5 in the same sprint: one-day moves)
 3. D2  gcc + MSVC benchmark             ── measure before the big work
 4. B1  LSH band table                   ── the foundation every later move needs
 5. B3+B4 content-addressed IR + cache   ── makes featurizer upgrades cheap
 6. B2  incremental engine               ── from batch to online; the key move
 7. E1  diff view  +  E7 explanations    ── visible payoff from existing data
 8. D1  calibrated significance          ── scores with a meaning
 9. E2  cluster history → E5 watchlists  ── operational product
10. E3  arch-neutral family signatures   ── from clustering to detection

Why this order: - Moves 1–2 are forcing: cheap, low risk, and every later measurement gets better. Tuning anything else while name leakage and CRT noise are in the data would tune against artifacts. - Move 3 comes before C1 and C6, so the opt-axis and compiler work have a target to hit. - Moves 4–6 are development. Without them every new tier makes the batch run slower, so the position gets worse with each feature added. B1 is cheap and is a prerequisite for B2, E3 and E10. - Move 7 cashes in data that already exists: per-pair tier counts, every function match, and statement hash lists. Users see value while the deep work continues. - Moves 8–10 turn SimilarityDB from "a similarity database" into "a lineage and detection system". Signatures (E3) need the stable cores that cluster history (E2) provides.

Best evaluation after the line: the engine is online at 10⁹ scale; Windows lineage is clean; recall across compilers has been measured, and its weaknesses are known; similarity has a calibrated meaning; analysts get diffs, explanations, cluster timelines, alerts and architecture-neutral signatures.

Side line (run in parallel, research): C1 (-O0 normalization), then C3 / C4 (context and rarity tiers), then D4 (adversarial axis). Each is measured on the benchmark from move 3 before it is promoted to a default tier.

4. Lines considered and pruned

5. Threats to watch (the opponent's best replies)

• Flooding: an attacker uploads near-copies to dilute df. Mitigated by --df-unit cluster. With B2 the engine must keep df per unit incrementally, or the defence lapses.
• Name-poisoning: fake symbols in an unstripped binary. A1 removes names from hashes; the UI should show names as untrusted.
• Function splitting / outlining: C2 (logical functions) and containment scoring.
• IDA upgrade drift: without B5, one IDA update can silently split every cluster.
• Signature evasion: once E3 exists, adversaries target the stable core. Leave-one-out tests and the adversarial axis (D4) must gate each release.

6. Feature projection beyond the line (moves 11+)

These are less certain and depend on how moves 1–10 turn out:

• Vulnerability propagation: mark a function as a known-vulnerable version (for example a CVE in a zlib or OpenSSL function) and list every sample that contains it, ranked by match tier. This is a content-hash lookup on the references side, made precise by D1.
• Time-travel queries: "what did cluster C look like on date D?" This needs E2 and run retention (B6).
• Federated corpora: two deployments exchange only content hashes and band values, never code, to find shared families. Watermarking already covers what leaks when code is shown.
• Containment-first search: "which samples contain ≥ 80% of this loader?" as a first-class query, using containment-MinHash [repo, open question].
• Corpus health dashboard: the dedup ratio over time, df distribution, library coverage, and coverage ratio per platform. All of these already exist as counts.