Hang Gao

# APK Code Analysis — Technical Annex

**Companion to:** APK_CODE_ANALYSIS_PDD.md
**Audience:** Engineers implementing APK code analysis
**Date:** 2026-03-07

---

## Table of Contents

1. [File Structure](#1-file-structure)
2. [System Dependencies Setup](#2-system-dependencies-setup)
3. [Phase 1 — Core Infrastructure](#3-phase-1--core-infrastructure)
4. [Phase 2 — Extractor and Data Export](#4-phase-2--extractor-and-data-export)
5. [Phase 3 — Similarity and Obfuscation](#5-phase-3--similarity-and-obfuscation)
6. [Phase 4 — Integration Testing](#6-phase-4--integration-testing)
7. [ClickHouse Schema](#7-clickhouse-schema)
8. [Androguard API Quick Reference](#8-androguard-api-quick-reference)
9. [Smali Format Reference](#9-smali-format-reference)

---

## SDLC Requirements

**Each phase ends with a quality gate:**

1. All new code has unit tests (mocked externals — no JADX/apktool/Java required)
2. All existing tests pass (`pytest tests/unit/` — must remain at 579+ tests passing)
3. New tests are added to the appropriate test file and documented
4. `TEST_INDEX.md` is updated with new test counts and file descriptions
5. Code passes `flake8`, `pylint`, `black`, `isort` (existing dev tools in `requirements.txt`)
6. Commit only after all tests pass. Move to next phase only after commit.

---

## 1. File Structure

### New files to create:

```
redb/extractors/decompiler/
├── DecompileAPK.py                    # Main extractor (like DecompileBinja.py)
├── apk/
│   ├── __init__.py
│   ├── analyzer.py                    # APKCodeAnalyzer — orchestrates analysis
│   ├── jadx_wrapper.py               # JADXDecompiler — subprocess + output parsing
│   ├── apktool_wrapper.py            # ApktoolDisassembler — subprocess + smali parsing
│   ├── method_extractor.py           # Per-method content extraction and hashing
│   ├── library_filter.py             # Package-based library filtering
│   └── smali_parser.py               # Smali file parsing — method boundary detection

tests/unit/
├── test_apk_code_analyzer.py          # Analyzer, library filtering, method enumeration
├── test_apk_jadx_wrapper.py           # JADX subprocess, output parsing, timeouts
├── test_apk_apktool_wrapper.py        # Apktool subprocess, smali parsing, timeouts
├── test_apk_method_extractor.py       # Hashing, content extraction, similarity
├── test_apk_decompile_extractor.py    # DecompileAPK extractor, export, schema

docs/
└── apk-code-schema.md                 # ClickHouse table definitions (like new-code-binja-schema.md)
```

### Files to modify:

```
redb/extractors/enum.py               # Add APK_DECOMPILED tag
redb/models/dataclasses.py            # Add APK code analysis dataclasses
redb/workers.py                        # Wire up DecompileAPK in APK dispatch
tests/TEST_INDEX.md                    # Update test inventory
```

---

## 2. System Dependencies Setup

### JADX Installation

```bash
# Option 1: Download from GitHub releases
wget https://github.com/skylot/jadx/releases/download/v1.5.5/jadx-1.5.5.zip
unzip jadx-1.5.5.zip -d /opt/jadx
ln -s /opt/jadx/bin/jadx /usr/local/bin/jadx

# Option 2: Package manager (if available)
# apt install jadx  OR  brew install jadx

# Verify
jadx --version
```

### apktool Installation

```bash
# Download wrapper script + jar
wget https://raw.githubusercontent.com/iBotPeaches/Apktool/master/scripts/linux/apktool
wget https://github.com/iBotPeaches/Apktool/releases/download/v2.12.0/apktool_2.12.0.jar
mv apktool_2.12.0.jar /usr/local/bin/apktool.jar
mv apktool /usr/local/bin/apktool
chmod +x /usr/local/bin/apktool

# Verify
apktool --version
```

### Java Runtime

```bash
apt install openjdk-17-jre-headless
java -version
```

### Docker considerations

If running in Docker, add to Dockerfile:

```dockerfile
RUN apt-get update && apt-get install -y openjdk-17-jre-headless
# Then install jadx and apktool as above
```

---

## 3. Phase 1 — Core Infrastructure

### Task 1.1: Library Filter (`library_filter.py`)

**What to build:** A class that determines whether a method belongs to a known library/framework package and should be filtered out of content tables.

**Behavior:**
- Accept a list of package prefixes (default list in PDD §2.3)
- Configurable via `APK_LIBRARY_PREFIXES` environment variable (comma-separated)
- Method `is_library(class_name: str) -> bool` — returns True if class_name starts with any filter prefix
- Method `get_filter_stats() -> dict` — returns counts of filtered vs. retained classes (for logging)

**Reference pattern:** Similar to `is_lib_or_thunk()` in `redb/extractors/decompiler/bninja/decompiler.py` (module-level function that checks `symbol.type` against `SymbolType.LibraryFunctionSymbol`, etc.)

**Tests:** Unit tests with various class names, custom prefix lists, env var override.

---

### Task 1.2: Smali Parser (`smali_parser.py`)

**What to build:** Parser that reads apktool's smali output files and extracts individual method bodies.

**apktool output structure:**
```
<tmpdir>/smali/com/example/MyClass.smali
```

Each `.smali` file contains one class with all its methods. Method boundaries are delimited by:
```smali
.method public onCreate(Landroid/os/Bundle;)V
    .registers 4
    .param p1, "savedInstanceState"

    invoke-super {p0, p1}, Landroid/app/Activity;->onCreate(Landroid/os/Bundle;)V
    const/high16 v0, 0x7f090000
    invoke-virtual {p0, v0}, Lcom/example/MyClass;->setContentView(I)V
    return-void
.end method
```

**Methods to implement:**
- `parse_smali_file(filepath: str) -> List[SmaliMethod]` — Returns list of `(class_name, method_name, method_signature, method_body, instruction_count, register_count)`
- `parse_smali_directory(dirpath: str) -> Dict[str, SmaliMethod]` — Parses all `.smali` files, returns dict keyed by `class_name->method_name(signature)`
- `normalize_smali_body(body: str) -> str` — Strip comments, normalize whitespace, remove line numbers (`.line N` directives) for consistent hashing
- `count_instructions(body: str) -> int` — Count actual Dalvik instructions (skip directives like `.registers`, `.param`, `.line`, `.local`, `.annotation`)

**Key parsing rules:**
- Method starts with `.method` line, ends with `.end method`
- Access modifiers: `public`, `private`, `protected`, `static`, `final`, `abstract`, `native`
- Skip `abstract` and `native` methods (no body)
- Dalvik instructions start with a verb: `invoke-*`, `const*`, `move*`, `if-*`, `goto*`, `return*`, `new-*`, `iget*`, `iput*`, `sget*`, `sput*`, `aget*`, `aput*`, etc.
- Directives start with `.` — these are metadata, not instructions

**Tests:** Parse sample smali files, verify method boundaries, instruction counts, normalization.

---

### Task 1.3: JADX Wrapper (`jadx_wrapper.py`)

**What to build:** Subprocess wrapper for JADX decompilation, following the CAPA extractor pattern.

**Reference implementation:** `redb/extractors/capa.py` — uses `subprocess.Popen` with process group management and timeout escalation (SIGTERM → wait 3s → SIGKILL).

**Methods to implement:**
- `__init__(jadx_path, timeout, log)` — Configure from env vars `JADX_PATH` and `JADX_TIMEOUT`
- `decompile(apk_path: str, output_dir: str) -> bool` — Run JADX, return success/failure
- `parse_java_methods(output_dir: str) -> Dict[str, str]` — Parse JADX output into per-method Java source, keyed by `class_name.method_name(param_types)`
- `cleanup(output_dir: str)` — Remove temp directory

**JADX command:**
```bash
jadx --no-res --no-imports --threads-count 2 --output-dir <tmpdir>/jadx <apk_path>
```

**Output structure from JADX:**
```
<tmpdir>/jadx/sources/com/example/MyClass.java
```

Each `.java` file contains one class. Methods must be extracted by parsing Java source (find method signatures via regex or simple brace-counting parser).

**Java method extraction approach:**
- Use regex to find method declarations: `(public|private|protected|static|...)*\s+\w+\s+\w+\s*\([^)]*\)\s*\{`
- Track brace depth to find method end
- Key each method by `fully.qualified.ClassName.methodName(ParamType1, ParamType2)`
- Normalize: strip leading/trailing whitespace, normalize indentation

**Tests:** Mock subprocess, test output parsing with sample Java files, test timeout handling, test error recovery.

---

### Task 1.4: Apktool Wrapper (`apktool_wrapper.py`)

**What to build:** Subprocess wrapper for apktool disassembly, same pattern as JADX wrapper.

**Methods to implement:**
- `__init__(apktool_path, timeout, log)` — Configure from env vars `APKTOOL_PATH` and `APKTOOL_TIMEOUT`
- `disassemble(apk_path: str, output_dir: str) -> bool` — Run apktool, return success/failure
- `get_smali_directory(output_dir: str) -> str` — Return path to smali output (handles multi-dex: `smali/`, `smali_classes2/`, `smali_classes3/`, etc.)
- `cleanup(output_dir: str)` — Remove temp directory

**apktool command:**
```bash
apktool d --no-res --force --output <tmpdir>/apktool <apk_path>
```

**Multi-DEX handling:** apktool creates `smali/` for `classes.dex`, `smali_classes2/` for `classes2.dex`, etc. The wrapper must iterate all `smali*` directories.

**Tests:** Mock subprocess, test multi-dex directory detection, test timeout handling.

---

### Task 1.5: APK Code Analyzer (`analyzer.py`)

**What to build:** Orchestrator that combines androguard, JADX, and apktool into a single analysis engine. This is the APK equivalent of `BinaryNinjaDecompiler` in `redb/extractors/decompiler/bninja/decompiler.py`.

**Constructor:**
- `__init__(filepath, timeout, log, decompile_modules={"all"})` — Same module selection pattern as `BinaryNinjaDecompiler`

**Main method — `extract() -> dict`:**

1. Create temp directories for JADX and apktool output
2. Run JADX and apktool in parallel (use `concurrent.futures.ThreadPoolExecutor` with 2 workers)
3. While JADX/apktool run, start androguard analysis: `AnalyzeAPK(filepath)` → get `(apk, dexs, analysis)`
4. Wait for JADX and apktool to complete
5. Parse smali output → `Dict[method_key, SmaliMethod]`
6. Parse Java output → `Dict[method_key, str]`
7. Get method list from androguard `analysis.get_methods()`
8. For each method from androguard:
   a. Check if library → skip content if yes (but keep in xref arrays)
   b. Look up smali body from parsed smali output
   c. Look up Java source from parsed Java output
   d. Get xrefs: `method.get_xref_from()` (callers) and `method.get_xref_to()` (callees)
   e. Compute hashes (SHA-256 of normalized smali, SHA-256 of normalized Java)
   f. Apply minimum instruction filter (`APK_MIN_METHOD_INSTRUCTIONS`, default 5)
   g. Build content and reference records
9. Return results dict with keys: `decompiled_content`, `decompiled_refs`, `smali_content`, `smali_refs`, `similarity_metrics`, `analysis_errors`

**Method key matching:**
The critical challenge is matching methods across three tools that use different naming conventions:
- **Androguard:** `Lcom/example/MyClass;->onCreate(Landroid/os/Bundle;)V` (Dalvik descriptor format)
- **Smali (apktool):** Same Dalvik descriptor format (same as androguard)
- **Java (JADX):** `com.example.MyClass.onCreate(Bundle)` (Java format)

Build a `method_key_converter` utility that normalizes between these formats:
- `dalvik_to_java(descriptor: str) -> str` — Convert `Lcom/example/MyClass;->onCreate(Landroid/os/Bundle;)V` to `com.example.MyClass.onCreate(Bundle)`
- `java_to_dalvik(java_sig: str) -> str` — Reverse mapping (may not be needed)

**Reference pattern:** Study `BinaryNinjaDecompiler.analyze_binary()` in `redb/extractors/decompiler/bninja/decompiler.py` — it follows the same per-function loop with content extraction, hash computation, and cross-linking.

**Tests:** Mock all three tools, test method key matching, test library filtering integration, test parallel execution, test error handling when one tool fails.

---

### Phase 1 Quality Gate

After completing Tasks 1.1–1.5:

```bash
# Run all existing tests — must pass
pytest tests/unit/ -v

# Run new tests
pytest tests/unit/test_apk_code_analyzer.py tests/unit/test_apk_jadx_wrapper.py \
       tests/unit/test_apk_apktool_wrapper.py -v

# Linting
black redb/extractors/decompiler/apk/ tests/unit/test_apk_*.py
isort redb/extractors/decompiler/apk/ tests/unit/test_apk_*.py
flake8 redb/extractors/decompiler/apk/
```

Update `TEST_INDEX.md` with new test file entries. Commit.

---

## 4. Phase 2 — Extractor and Data Export

### Task 2.1: Tag Enum Update

**File:** `redb/extractors/enum.py`

Add after the existing APK tags:

```python
APK_DECOMPILED = "apk_decompiled"
```

---

### Task 2.2: Dataclasses

**File:** `redb/models/dataclasses.py`

Add the 6 dataclasses defined in PDD §4.1:
- `APKDecompiledMethodContent`
- `APKDecompiledMethodReference`
- `APKSmaliMethodContent`
- `APKSmaliMethodReference`
- `APKMethodSimilarityMetrics`
- `APKCodeAnalysisError`

Follow the exact style of existing dataclasses in the file. Use `Optional` for nullable fields, `field(default_factory=list)` for list defaults.

---

### Task 2.3: DecompileAPK Extractor (`DecompileAPK.py`)

**What to build:** The main extractor class, following `DecompileBinja.py` exactly.

**File:** `redb/extractors/decompiler/DecompileAPK.py`

**Pattern to follow (from `DecompileBinja.py`):**
- Extends `Extractor` directly
- Constructor: filepath, log, exporters, index_prefix, filetype
- Timeouts from env vars: `APK_DECOMPILE_TIMEOUT` (default 1800s)
- `extract()`: Run `APKCodeAnalyzer` in a daemon thread with timeout (same pattern as DecompileBinja)
- `tag()`: Return `Tag.APK_DECOMPILED.value`
- `prepare_export_data()`: Build `multi_table` dict for ClickHouse export
- `cleanup_run()`: Remove temp directories, force GC
- `get_clickhouse_table()`: Return `None` (multi-table export, no single table)

**Multi-table export structure** (from `prepare_export_data` — follow the exact pattern in `DecompileBinja.prepare_export_data()`):

```python
export_data = {"multi_table": True}

# Table 1: Decompiled method content
if self.analysis_results.get("decompiled_content"):
    export_data["code_apk_decompiled_methods_content"] = {
        "data": [...],
        "column_names": [...],
        "column_type_names": [...]
    }

# Table 2: Decompiled method references
# Table 3: Smali method content
# Table 4: Smali method references
# Table 5: Similarity metrics
# Table 6: Analysis errors
```

**Column names and types** must match the ClickHouse schema in §7. Study `DecompileBinja.prepare_export_data()` for the exact tuple format.

---

### Task 2.4: ClickHouse Schema Functions

**File:** `docs/apk-code-schema.md`

Create table creation functions following `docs/new-code-binja-schema.md` pattern. Full schema in §7 below.

---

### Task 2.5: Workers Integration

**File:** `redb/workers.py`

In the APK processing section (search for `apk` filetype handling), add `DecompileAPK` after the existing APK extractors.

Follow the same conditional pattern used for `DecompileBinja`:
- Check if module is selected
- Instantiate with filepath, logger, exporters
- Call `export_data()`

Also update `get_module_by_name()` to register `"DecompileAPK"` in the modules map.

---

### Phase 2 Quality Gate

```bash
pytest tests/unit/ -v                          # All existing + new tests pass
pytest tests/unit/test_apk_decompile_extractor.py -v   # New extractor tests
pytest tests/unit/test_apk_dataclasses.py -v           # Updated dataclass tests
```

Update `TEST_INDEX.md`. Commit.

---

## 5. Phase 3 — Similarity and Obfuscation

### Task 3.1: Method-Level Similarity Hashes

**File:** Extend `method_extractor.py`

Compute for each user method's smali body:
- **ssdeep** — Use `ppdeep` (already in requirements): `ppdeep.hash(smali_body.encode())`
- **TLSH** — Use `tlsh` (already in requirements): `tlsh.hash(smali_body.encode())`
- **MinHash** — Use existing `MinHashCustom` from `redb/extractors/decompiler/bninja/similarity.py`. Generate n-grams from smali instructions, compute MinHash signature.

**Reference:** Study `BinaryNinjaDecompiler.analyze_binary()` where it computes `ssdeep_disassembly`, `tlsh_disassembly`, and calls `MinHasher`.

**Minimum size for fuzzy hashes:** ssdeep and TLSH both require minimum input sizes. Skip if smali body is too short (ssdeep < 4096 bytes returns empty, TLSH < 50 bytes returns empty).

---

### Task 3.2: Obfuscation Indicators (per method)

**File:** Extend `method_extractor.py`

For each method's smali body, compute:
- `short_method_name`: method name length <= 2
- `short_class_name`: class simple name (after last `/`) length <= 2
- `has_string_encryption`: presence of `const-string` followed within 3 instructions by `invoke-*` to known decryptor patterns
- `has_reflection_calls`: presence of `invoke-*` targeting `Ljava/lang/reflect/*` or `Ljava/lang/Class;->forName`
- `excessive_goto_count`: count of `goto` / `goto/16` / `goto/32` instructions exceeds `max(5, instruction_count * 0.15)`

---

### Task 3.3: Populate Similarity Metrics Table

Extend `APKCodeAnalyzer.extract()` to populate the `similarity_metrics` results key with `APKMethodSimilarityMetrics` records.

---

### Phase 3 Quality Gate

```bash
pytest tests/unit/ -v
pytest tests/unit/test_apk_method_extractor.py -v   # Hashing + obfuscation tests
```

Update `TEST_INDEX.md`. Commit.

---

## 6. Phase 4 — Integration Testing

### Task 4.1: Integration Test Suite

**File:** `tests/integration/test_apk_code_analysis.py`

**Requirements:** JADX, apktool, and Java must be installed.

**Test cases:**
1. **Simple APK** — Known benign APK with 5-10 user classes. Verify: methods extracted, smali and Java content populated, xrefs present, hashes computed.
2. **Multi-DEX APK** — APK with `classes.dex` + `classes2.dex`. Verify: methods from both DEX files analyzed, cross-DEX xrefs work.
3. **Obfuscated APK** — ProGuard/R8 obfuscated APK. Verify: short name indicators detected, library filter still works on renamed packages.
4. **Empty DEX** — APK with only framework calls, no user code after filtering. Verify: extractor returns gracefully with zero methods.
5. **Packed APK** — APK with encrypted DEX (e.g., Qihoo 360). Verify: error logged, no crash, analysis_errors table populated.
6. **JADX failure** — Simulate JADX timeout/crash. Verify: smali analysis still completes, decompiled content tables empty, error logged.
7. **apktool failure** — Same for apktool. Verify: Java analysis still completes.
8. **Large APK** — APK with 1000+ user methods. Verify: completes within timeout, memory stays bounded.

**Markers:**
```python
@pytest.mark.integration
@pytest.mark.apk
@pytest.mark.decompile
```

---

### Task 4.2: Performance Profiling

Run the integration tests with timing:
```bash
pytest tests/integration/test_apk_code_analysis.py -v --durations=0
```

Verify:
- apktool + JADX subprocess total < 60s for a typical APK
- Androguard analysis < 30s for a typical APK
- Per-method processing < 5ms
- Total pipeline < `APK_DECOMPILE_TIMEOUT` (1800s) for even the largest APKs

---

### Task 4.3: Final TEST_INDEX.md Update

Add all new test files, update counts, add integration test descriptions.

Final commit.

---

## 7. ClickHouse Schema

All tables follow the `ReplacingMergeTree(analysis_date)` pattern from `docs/new-code-binja-schema.md`,
except `code_apk_analysis_errors` which uses `MergeTree()`.

Full DDL is also available in `docs/apk-code-schema.md`.

### Table overview

| # | Table | Analog (Binja) | Engine | Key |
|---|-------|----------------|--------|-----|
| 1 | `code_apk_decompiled_methods_content` | `code_binja_decompiled_functions_content` | ReplacingMergeTree | `decompiled_method_hash` |
| 2 | `code_apk_decompiled_methods_references` | `code_binja_decompiled_functions_references` | ReplacingMergeTree | `(sha256, decompiled_method_hash)` |
| 3 | `code_apk_smali_methods_content` | `code_binja_disassembled_functions_content` | ReplacingMergeTree | `smali_method_hash` |
| 4 | `code_apk_smali_methods_references` | `code_binja_disassembled_functions_references` | ReplacingMergeTree | `(sha256, smali_method_hash)` |
| 5 | `code_apk_method_similarity_metrics` | `code_binja_function_similarity_metrics` | ReplacingMergeTree | `smali_method_hash` |
| 6 | `code_apk_cfg_methods` | `code_binja_cfg_functions` | ReplacingMergeTree | `smali_method_hash` |
| 7 | `code_binja_strings_raw` *(shared)* | — | Null (→ MV) | — |
| 8 | `code_apk_analysis_errors` | `function_analysis_errors_binja` | MergeTree | `(sha256, error_location, error_hash)` |

### Table 1: `code_apk_decompiled_methods_content`

**Analog:** `code_binja_decompiled_functions_content`

```sql
CREATE TABLE IF NOT EXISTS code_apk_decompiled_methods_content (
    decompiled_method_hash FixedString(64),          -- SHA-256 of normalized Java source
    decompiled_method String CODEC(ZSTD(3)),          -- Full Java method source
    decompiled_method_lower String MATERIALIZED lower(decompiled_method) CODEC(ZSTD(3)),
    method_type Enum8('USER'=1, 'LIBRARY'=2, 'UNKNOWN'=5) DEFAULT 'UNKNOWN',
    has_string_encryption UInt8 DEFAULT 0,
    has_reflection_calls UInt8 DEFAULT 0,
    excessive_goto_count UInt8 DEFAULT 0,
    analysis_date DateTime64(3, 'UTC'),

    INDEX idx_method_content_token lower(decompiled_method) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 1,
    INDEX idx_method_hash decompiled_method_hash TYPE bloom_filter GRANULARITY 1
) ENGINE = ReplacingMergeTree(analysis_date)
PRIMARY KEY decompiled_method_hash
ORDER BY decompiled_method_hash;
```

### Table 2: `code_apk_decompiled_methods_references`

**Analog:** `code_binja_decompiled_functions_references`

```sql
CREATE TABLE IF NOT EXISTS code_apk_decompiled_methods_references (
    sha256 FixedString(64),
    sha1 FixedString(40),
    md5 FixedString(32),
    decompiled_method_hash FixedString(64),
    smali_method_hash Nullable(FixedString(64)),
    class_name LowCardinality(String),
    method_name LowCardinality(String),
    method_signature String,                          -- Dalvik descriptor: (Landroid/os/Bundle;)V
    method_prototype String,                          -- Java-style: void onCreate(Bundle)
    functions_caller Array(String),
    functions_call Array(String),
    analysis_date DateTime64(3, 'UTC'),

    INDEX idx_sha256 sha256 TYPE bloom_filter GRANULARITY 1,
    INDEX idx_method_hash decompiled_method_hash TYPE bloom_filter GRANULARITY 1,
    INDEX idx_class_name class_name TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 1,
    INDEX idx_method_name method_name TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 1
) ENGINE = ReplacingMergeTree(analysis_date)
ORDER BY (sha256, decompiled_method_hash);
```

### Table 3: `code_apk_smali_methods_content`

**Analog:** `code_binja_disassembled_functions_content`

```sql
CREATE TABLE IF NOT EXISTS code_apk_smali_methods_content (
    smali_method_hash FixedString(64),                -- SHA-256 of normalized smali body
    smali_method String CODEC(ZSTD(3)),               -- Full smali method body
    method_type Enum8('USER'=1, 'LIBRARY'=2, 'UNKNOWN'=5) DEFAULT 'UNKNOWN',
    instructions_count UInt32,
    register_count UInt16,
    has_string_encryption UInt8 DEFAULT 0,
    has_reflection_calls UInt8 DEFAULT 0,
    excessive_goto_count UInt8 DEFAULT 0,
    analysis_date DateTime64(3, 'UTC'),

    INDEX idx_smali_ngram smali_method TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 1,
    INDEX idx_method_type method_type TYPE set(10) GRANULARITY 1,
    INDEX idx_instr_count instructions_count TYPE minmax GRANULARITY 4
) ENGINE = ReplacingMergeTree(analysis_date)
ORDER BY smali_method_hash;
```

### Table 4: `code_apk_smali_methods_references`

**Analog:** `code_binja_disassembled_functions_references`

```sql
CREATE TABLE IF NOT EXISTS code_apk_smali_methods_references (
    sha256 FixedString(64),
    sha1 FixedString(40),
    md5 FixedString(32),
    smali_method_hash FixedString(64),
    decompiled_method_hash Nullable(FixedString(64)),
    class_name LowCardinality(String),
    method_name LowCardinality(String),
    method_signature String,
    ssdeep_smali Nullable(String),
    tlsh_smali Nullable(FixedString(72)),
    analysis_date DateTime64(3, 'UTC'),

    INDEX idx_sha256 sha256 TYPE bloom_filter GRANULARITY 1,
    INDEX idx_smali_hash smali_method_hash TYPE bloom_filter GRANULARITY 1,
    INDEX idx_tlsh_smali tlsh_smali TYPE bloom_filter GRANULARITY 1,
    INDEX idx_ssdeep_smali ssdeep_smali TYPE bloom_filter GRANULARITY 1
) ENGINE = ReplacingMergeTree(analysis_date)
ORDER BY (sha256, smali_method_hash);
```

### Table 5: `code_apk_method_similarity_metrics`

**Analog:** `code_binja_function_similarity_metrics`

Content-based fuzzy matching table. Structural/CFG scalars (block_count, edge_count, etc.)
are in `code_apk_cfg_methods` (Table 6) — this table holds only fuzzy hashes and MinHash.

```sql
CREATE TABLE IF NOT EXISTS code_apk_method_similarity_metrics (
    smali_method_hash FixedString(64),
    cyclomatic_complexity Nullable(UInt16),
    ssdeep_smali Nullable(String),
    tlsh_smali Nullable(FixedString(72)),
    minhash Array(UInt8),
    analysis_date DateTime64(3, 'UTC'),

    INDEX idx_complexity cyclomatic_complexity TYPE minmax GRANULARITY 4,
    INDEX idx_ssdeep ssdeep_smali TYPE bloom_filter GRANULARITY 1,
    INDEX idx_tlsh tlsh_smali TYPE bloom_filter GRANULARITY 1
) ENGINE = ReplacingMergeTree(analysis_date)
ORDER BY smali_method_hash;
```

### Table 6: `code_apk_cfg_methods`

**Analog:** `code_binja_cfg_functions` (see `db_migration/cfg_functions_ddl.sql`)

Structural and topological features computed from smali CFG. All fields mirror
the Binja table with APK-appropriate naming (`smali_method_hash` instead of
`disassembled_function_hash`, `instructions_count` instead of `llil_total_operations`,
`prime_product_smali` instead of `prime_product_llil`).

```sql
CREATE TABLE IF NOT EXISTS code_apk_cfg_methods (
    -- Identity
    smali_method_hash FixedString(64),

    -- Tier 0: Exact structural match
    cfg_topology_hash FixedString(16),

    -- Tier 1: Structural pre-filtering
    block_count UInt16,
    edge_count UInt16,
    instructions_count UInt32,               -- Dalvik instruction count (analog: llil_total_operations)
    call_count UInt16,
    cyclomatic_complexity UInt16,
    loop_count UInt8,
    max_depth UInt16,
    max_fan_out UInt8,
    md_index_topdown UInt64,
    md_index_bottomup UInt64,
    prime_product_smali UInt64,              -- Dalvik semantic primes (analog: prime_product_llil)

    -- Tier 2: Fuzzy matching
    cfg_feature_tlsh Nullable(FixedString(72)),
    wl_minhash Array(UInt8),

    -- Embedding-ready storage
    bb_features Array(Array(UInt16)),        -- ACFG block feature vectors
    cfg_adjacency Array(UInt32),             -- Packed (src << 16 | tgt)

    analysis_date DateTime64(3, 'UTC'),

    -- Indexes
    INDEX idx_topology cfg_topology_hash TYPE bloom_filter GRANULARITY 1,
    INDEX idx_complexity cyclomatic_complexity TYPE minmax GRANULARITY 4,
    INDEX idx_block_count block_count TYPE minmax GRANULARITY 4,
    INDEX idx_edge_count edge_count TYPE minmax GRANULARITY 4,
    INDEX idx_call_count call_count TYPE minmax GRANULARITY 4,
    INDEX idx_instr_count instructions_count TYPE minmax GRANULARITY 4,
    INDEX idx_md_topdown md_index_topdown TYPE bloom_filter GRANULARITY 1,
    INDEX idx_md_bottomup md_index_bottomup TYPE bloom_filter GRANULARITY 1,
    INDEX idx_prime prime_product_smali TYPE bloom_filter GRANULARITY 1,
    INDEX idx_cfg_tlsh cfg_feature_tlsh TYPE bloom_filter GRANULARITY 1
) ENGINE = ReplacingMergeTree(analysis_date)
ORDER BY smali_method_hash
SETTINGS index_granularity = 8192;
```

**Column mapping (APK → Binja):**

| APK column | Binja column | Notes |
|------------|-------------|-------|
| `smali_method_hash` | `disassembled_function_hash` | Both SHA-256 of normalized code |
| `instructions_count` | `llil_total_operations` | Dalvik instructions vs LLIL operations |
| `prime_product_smali` | `prime_product_llil` | Same prime algorithm, Dalvik semantic categories |
| All others | Same name | Identical algorithms from shared `cfg_features.py` |

### Table 7: `code_binja_strings_raw` (shared)

APK strings are inserted into the existing `code_binja_strings_raw` Null-engine table,
which feeds materialized views (`code_binja_strings_by_binary`, `mv_string_popularity_public`).
This enables cross-format string correlation between PE/ELF/Mach-O/APK samples.

No new table creation needed — see `docs/new-code-binja-schema.md` for the existing DDL.

### Table 8: `code_apk_analysis_errors`

**Analog:** `function_analysis_errors_binja`

```sql
CREATE TABLE IF NOT EXISTS code_apk_analysis_errors (
    sha256 FixedString(64),
    class_name Nullable(String) CODEC(ZSTD(3)),
    method_name Nullable(String) CODEC(ZSTD(3)),
    error_location LowCardinality(String) CODEC(ZSTD(3)),   -- 'jadx', 'apktool', 'androguard', 'analysis'
    error_message Nullable(String) CODEC(ZSTD(3)),
    error_type Nullable(String) CODEC(ZSTD(3)),
    error_hash FixedString(32),                               -- MD5 for dedup
    status Enum8('new' = 1, 'investigating' = 2, 'fixed' = 3, 'wontfix' = 4) DEFAULT 'new',
    analysis_date DateTime64(3, 'UTC'),
    PRIMARY KEY (sha256, error_location, error_hash)
) ENGINE = MergeTree()
ORDER BY (sha256, error_location, error_hash);
```

---

## 8. Androguard API Quick Reference

Key API calls used by `APKCodeAnalyzer`:

```python
from androguard.misc import AnalyzeAPK

# Parse APK — returns (APK object, list of DEX objects, Analysis object)
apk, dexs, analysis = AnalyzeAPK("sample.apk")

# Enumerate all methods
for method in analysis.get_methods():
    # method is a MethodAnalysis object

    # Check if it's an external method (no code body)
    if method.is_external():
        continue

    # Get the underlying EncodedMethod
    encoded = method.get_method()
    class_name = encoded.get_class_name()    # "Lcom/example/MyClass;"
    method_name = encoded.get_name()          # "onCreate"
    descriptor = encoded.get_descriptor()     # "(Landroid/os/Bundle;)V"

    # Cross-references: who calls this method
    for ref_class, ref_method, offset in method.get_xref_from():
        caller = f"{ref_method.get_class_name()}->{ref_method.get_name()}"

    # Cross-references: what this method calls
    for ref_class, ref_method, offset in method.get_xref_to():
        callee = f"{ref_method.get_class_name()}->{ref_method.get_name()}"

# Call graph (networkx MultiDiGraph)
call_graph = analysis.get_call_graph()
# Nodes are MethodAnalysis objects
# Edges represent caller → callee relationships
```

**Important notes:**
- `AnalyzeAPK()` loads ALL DEX files (handles multi-DEX automatically)
- `is_external()` returns True for methods declared but not defined (framework methods)
- `get_xref_from()` and `get_xref_to()` return tuples of `(ClassAnalysis, MethodAnalysis, int_offset)`
- The Analysis object is the single source of truth for method enumeration and xrefs

---

## 9. Smali Format Reference

### Method declaration syntax

```smali
.method <access_flags> <name>(<param_descriptors>)<return_type>
    .registers <N>              # Total register count
    .param p1, "paramName"      # Parameter name (debug info, may be missing)
    .line <N>                   # Source line number (debug info)

    # Instructions (Dalvik opcodes)
    invoke-virtual {p0, v0}, Lcom/example/Foo;->bar(I)V
    const-string v1, "hello"
    if-eqz v0, :cond_0
    goto :goto_0

    :cond_0                     # Label
    return-void
.end method
```

### Type descriptors

| Descriptor | Java Type |
|-----------|-----------|
| `V` | void |
| `Z` | boolean |
| `B` | byte |
| `I` | int |
| `J` | long |
| `F` | float |
| `D` | double |
| `Lcom/example/Foo;` | com.example.Foo |
| `[I` | int[] |
| `[Ljava/lang/String;` | String[] |

### Instructions to count (not directives)

All lines that are NOT labels (`:label_N`) and NOT directives (`.something`) are instructions. Common instruction prefixes:
`invoke-*`, `const*`, `move*`, `return*`, `if-*`, `goto*`, `new-*`, `iget*`, `iput*`, `sget*`, `sput*`, `aget*`, `aput*`, `check-cast`, `instance-of`, `throw`, `monitor-*`, `fill-*`, `packed-switch`, `sparse-switch`, `cmp*`, `add-*`, `sub-*`, `mul-*`, `div-*`, `rem-*`, `and-*`, `or-*`, `xor-*`, `shl-*`, `shr-*`, `ushr-*`, `neg-*`, `not-*`, `int-to-*`, `long-to-*`, `float-to-*`, `double-to-*`, `array-length`, `nop`

---

## Appendix A — Phase 5: CFG Feature Parity with Binary Ninja Pipeline

**Date:** 2026-03-13
**Depends on:** Phases 1–3 complete
**Reference:** APK_CODE_ANALYSIS_PDD.md, Appendix A

This appendix provides the implementation details for adding a dedicated `code_apk_cfg_methods` table and refactoring `code_apk_method_similarity_metrics` to match the Binja pipeline's separation of concerns.

---

### A.1 Rationale

The existing `smali_cfg.py` already builds full adjacency lists and computes per-block ACFG feature vectors, but this data is either reduced to scalars (block_count, edge_count, etc.) or silently dropped (bb_features). Meanwhile, `cfg_features.py` contains generic graph algorithms (topology hash, MD-index, WL-MinHash, etc.) that operate on adjacency lists — they have no Binary Ninja dependency. Connecting these two modules requires minimal glue code.

---

### A.2 File Changes

#### Modified files:

```
redb/extractors/decompiler/apk/smali_cfg.py       # Add predecessors, wire cfg_features
redb/extractors/decompiler/apk/method_extractor.py # Add call_count, prime_product_smali
redb/extractors/decompiler/apk/analyzer.py         # Populate new CFG fields in results
redb/extractors/decompiler/DecompileAPK.py         # New table export, slim similarity table
docs/apk-code-schema.md                            # Add code_apk_cfg_methods schema
tests/unit/test_apk_method_extractor.py            # Tests for new fields
```

#### No new files needed.

---

### A.3 Task A.1: Extend `smali_cfg.py`

**A.1.1 — Build predecessors from successors**

Add to `compute_cfg_metrics()`, after the adjacency list is built:

```python
predecessors = [[] for _ in range(n)]
for src, targets in enumerate(successors):
    for tgt in targets:
        predecessors[tgt].append(src)
```

**A.1.2 — Compute advanced CFG features**

Import and call `cfg_features.py` functions after building the graph:

```python
from redb.extractors.decompiler.bninja.analysis import cfg_features

bfs = cfg_features.bfs_order(successors, n)
topology_hash = cfg_features.compute_topology_hash(successors, bfs, n)
md_topdown = cfg_features.compute_md_index_topdown(successors, predecessors, bfs)
md_bottomup = cfg_features.compute_md_index_bottomup(successors, predecessors, n)
cfg_tlsh = cfg_features.compute_cfg_feature_tlsh(block_features, bfs)
wl_minhash = cfg_features.compute_wl_minhash(successors, predecessors, block_features, n)
adjacency = cfg_features.pack_adjacency(successors)
```

**A.1.3 — Extend `SmaliCFGMetrics` dataclass**

Add fields to the existing dataclass:

```python
@dataclass
class SmaliCFGMetrics:
    # Existing fields (unchanged)
    block_count: int = 0
    edge_count: int = 0
    cyclomatic_complexity: int = 1
    loop_count: int = 0
    max_depth: int = 0
    max_fan_out: int = 0
    block_features: List[List[int]] = field(default_factory=list)

    # New fields
    cfg_topology_hash: bytes = field(default_factory=lambda: b'\x00' * 16)
    md_index_topdown: int = 0
    md_index_bottomup: int = 0
    cfg_feature_tlsh: Optional[str] = None
    wl_minhash: List[int] = field(default_factory=lambda: [255] * 128)
    cfg_adjacency: List[int] = field(default_factory=list)
```

---

### A.4 Task A.2: Smali Prime Product (`method_extractor.py`)

Define a Dalvik-to-prime mapping using the semantic categories already in `smali_normalization.py`. Each category maps to the same prime its LLIL equivalent uses in `cfg_features.py`:

```python
SMALI_OP_PRIMES = {
    "ALU":    37,   # ADD/SUB → same prime as LLIL_ADD
    "CONV":   131,  # Type conversions → same as LLIL_SX
    "CMP":    103,  # Comparisons → same as LLIL_CMP_E
    "MOV":    2,    # Register moves → same as LLIL_SET_REG
    "CONST":  2,    # Constants → SET_REG equivalent
    "LOAD":   5,    # Field/array reads → same as LLIL_LOAD
    "STORE":  7,    # Field/array writes → same as LLIL_STORE
    "CALL":   17,   # invoke-* → same as LLIL_CALL
    "BRANCH": 29,   # if-* → same as LLIL_IF
    "JMP":    31,   # goto → same as LLIL_GOTO
    "SWITCH": 151,  # switch → same as LLIL_JUMP_TO
    "RET":    23,   # return → same as LLIL_RET
    "ALLOC":  5,    # new-instance/new-array → LOAD-adjacent (heap access)
    "TYPE":   1,    # check-cast/instance-of → identity (metadata)
    "ARR":    5,    # array-length/fill-array → LOAD-adjacent
    "EXC":    23,   # throw → RET-adjacent (control transfer out)
    "SYNC":   1,    # monitor → identity (no LLIL equivalent)
    "OTHER":  1,    # Unknown → identity
}


def compute_prime_product_smali(smali_body: str) -> int:
    """Multiplicative hash of normalized Dalvik opcodes. Mod 2^64."""
    product = 1
    for line in smali_body.splitlines():
        stripped = line.strip()
        if not stripped or stripped.startswith(('.', ':', '#')):
            continue
        category = classify_instruction(stripped)  # existing function
        prime = SMALI_OP_PRIMES.get(category, 1)
        product = (product * prime) % (2**64)
    return product
```

**`call_count`** — count lines where `classify_instruction()` returns `"CALL"`.

---

### A.5 Task A.3: Populate CFG Results (`analyzer.py`)

In `_process_method()`, after `compute_cfg_metrics()`, add the new fields to the results dict:

```python
cfg_metrics = compute_cfg_metrics(smali_body)

# Build cfg entry (separate from similarity_metrics)
cfg_entry = {
    "smali_method_hash": sha256_smali,
    "cfg_topology_hash": cfg_metrics.cfg_topology_hash,
    "block_count": cfg_metrics.block_count,
    "edge_count": cfg_metrics.edge_count,
    "instructions_count": instruction_count,
    "call_count": call_count,
    "cyclomatic_complexity": cfg_metrics.cyclomatic_complexity,
    "loop_count": cfg_metrics.loop_count,
    "max_depth": cfg_metrics.max_depth,
    "max_fan_out": cfg_metrics.max_fan_out,
    "md_index_topdown": cfg_metrics.md_index_topdown,
    "md_index_bottomup": cfg_metrics.md_index_bottomup,
    "prime_product_smali": prime_product,
    "cfg_feature_tlsh": cfg_metrics.cfg_feature_tlsh,
    "wl_minhash": cfg_metrics.wl_minhash,
    "bb_features": cfg_metrics.block_features,
    "cfg_adjacency": cfg_metrics.cfg_adjacency,
}
```

Add `"cfg"` as a new top-level key in the results dict returned by `extract()`.

---

### A.6 Task A.4: Export Tables (`DecompileAPK.py`)

**New table — `code_apk_cfg_methods`:**

```python
if self.analysis_results.get("cfg"):
    export["cfg_methods"] = {
        "table": "code_apk_cfg_methods",
        "data": [
            [
                cfg["smali_method_hash"],
                cfg["cfg_topology_hash"],
                cfg["block_count"],
                cfg["edge_count"],
                cfg.get("instructions_count", 0),
                cfg.get("call_count", 0),
                cfg["cyclomatic_complexity"],
                cfg.get("loop_count", 0),
                cfg.get("max_depth", 0),
                cfg.get("max_fan_out", 0),
                cfg.get("md_index_topdown", 0),
                cfg.get("md_index_bottomup", 0),
                cfg.get("prime_product_smali", 0),
                cfg.get("cfg_feature_tlsh"),
                cfg.get("wl_minhash", []),
                cfg.get("bb_features", []),
                cfg.get("cfg_adjacency", []),
                now,
            ]
            for cfg in self.analysis_results["cfg"]
            if cfg is not None
        ],
        "column_names": [
            "smali_method_hash", "cfg_topology_hash",
            "block_count", "edge_count", "instructions_count",
            "call_count", "cyclomatic_complexity", "loop_count",
            "max_depth", "max_fan_out",
            "md_index_topdown", "md_index_bottomup",
            "prime_product_smali", "cfg_feature_tlsh",
            "wl_minhash", "bb_features", "cfg_adjacency",
            "analysis_date",
        ],
        "column_type_names": [
            "FixedString(64)", "FixedString(16)",
            "UInt16", "UInt16", "UInt32",
            "UInt16", "UInt16", "UInt8",
            "UInt16", "UInt8",
            "UInt64", "UInt64",
            "UInt64", "Nullable(FixedString(72))",
            "Array(UInt8)", "Array(Array(UInt16))", "Array(UInt32)",
            "DateTime64(3, 'UTC')",
        ],
    }
```

**Slim down `code_apk_method_similarity_metrics`** — remove `block_count`, `edge_count`, `loop_count`, `max_depth`, `max_fan_out` from its column lists and data arrays.

---

### A.7 Task A.5: ClickHouse Schema

```sql
CREATE TABLE IF NOT EXISTS code_apk_cfg_methods (
    smali_method_hash FixedString(64),
    cfg_topology_hash FixedString(16),
    block_count UInt16,
    edge_count UInt16,
    instructions_count UInt32,
    call_count UInt16,
    cyclomatic_complexity UInt16,
    loop_count UInt8,
    max_depth UInt16,
    max_fan_out UInt8,
    md_index_topdown UInt64,
    md_index_bottomup UInt64,
    prime_product_smali UInt64,
    cfg_feature_tlsh Nullable(FixedString(72)),
    wl_minhash Array(UInt8),
    bb_features Array(Array(UInt16)),
    cfg_adjacency Array(UInt32),
    analysis_date DateTime64(3, 'UTC'),

    INDEX idx_topology cfg_topology_hash TYPE bloom_filter GRANULARITY 1,
    INDEX idx_complexity cyclomatic_complexity TYPE minmax GRANULARITY 4,
    INDEX idx_block_count block_count TYPE minmax GRANULARITY 4,
    INDEX idx_md_topdown md_index_topdown TYPE bloom_filter GRANULARITY 1,
    INDEX idx_md_bottomup md_index_bottomup TYPE bloom_filter GRANULARITY 1,
    INDEX idx_prime prime_product_smali TYPE bloom_filter GRANULARITY 1,
    INDEX idx_cfg_tlsh cfg_feature_tlsh TYPE bloom_filter GRANULARITY 1
) ENGINE = ReplacingMergeTree(analysis_date)
ORDER BY smali_method_hash;
```

---

### A.8 Task A.6: Tests

Add to `tests/unit/test_apk_method_extractor.py`:

- **`test_predecessors_from_successors`** — verify reverse mapping is correct
- **`test_cfg_topology_hash_identical_graphs`** — two methods with same control flow produce same hash
- **`test_cfg_topology_hash_different_graphs`** — different structure produces different hash
- **`test_md_index_topdown_bottomup`** — verify non-zero values for multi-block methods
- **`test_prime_product_smali`** — known input produces expected product
- **`test_prime_product_smali_position_independent`** — reordering blocks doesn't change result
- **`test_call_count`** — count invoke-* instructions
- **`test_wl_minhash_length`** — verify 128-element signature
- **`test_cfg_feature_tlsh_small_method`** — returns None when < 50 bytes
- **`test_bb_features_exported`** — verify block features appear in results
- **`test_cfg_adjacency_packed`** — verify (src << 16) | tgt encoding
- **`test_similarity_table_slimmed`** — verify block_count etc. removed from similarity export

---

### A.9 Phase 5 Quality Gate

```bash
# All existing tests pass
pytest tests/unit/ -v

# New CFG tests
pytest tests/unit/test_apk_method_extractor.py -k "cfg or prime_product or call_count or wl_minhash" -v

# Linting
black redb/extractors/decompiler/apk/ tests/unit/test_apk_method_extractor.py
isort redb/extractors/decompiler/apk/ tests/unit/test_apk_method_extractor.py
flake8 redb/extractors/decompiler/apk/
```

Update `TEST_INDEX.md`. Commit.