Ingmar Kallfass

32 papers A 1C 3Journal 9Unranked 19
YearRankTypeTitle / Venue / Authors
2024 J jnl
CoRR
Daniel Lindenschmitt, Christoph Fischer, Simon Haussmann, Marc Kalter, Ingmar Kallfass, Hans D. Schotten
2024 conf
RWS
Dominik Wrana, Christopher M. Grötsch, Benjamin Schoch, Lukas Gebert, Thomas Ufschlag, Arnulf Leuther, Robert C. Lozar, Ingmar Kallfass
2023 conf
RWS
Benjamin Schoch, Dominik Wrana, Laura Manoliu, Michael Kuri, Sandrine Wagner, Axel Tessmann, Ingmar Kallfass
2023 conf
EUROCON
Anqi Chu, Chris Manuel Hermann, Johannes Silz, Johannes Pfau, Kevin Muñoz Barón, Nidhi Anantharajaiah, Patrick Schmidt, Tim Hotfilter, Xiang Xie, Jürgen Becker, Ingmar Kallfass, Jörg Roth-Stielow, Wilhelm Stork
2023 conf
EUROCON
Oleksandr Solomakha, Valentyna Afanasenko, Ingmar Kallfass
2022 A conf
DATE
Hussam Amrouch, Jens Anders, Steffen Becker, Maik Betka, Gerd Bleher, Peter Domanski, Nourhan Elhamawy, Thomas Ertl, Athanasios Gatzastras, Paul R. Genssler, Sebastian Hasler, Martin Heinrich, André van Hoorn, Hanieh Jafarzadeh, Ingmar Kallfass, Florian Klemme, Steffen Koch, Ralf Küsters, Andrés Lalama, Raphaël Latty, Yiwen Liao, Natalia Lylina, Zahra Paria Najafi-Haghi, Dirk Pflüger, Ilia Polian, Jochen Rivoir, Matthias Sauer, Denis Schwachhofer, Steffen Templin, Christian Volmer, Stefan Wagner, Daniel Weiskopf, Hans-Joachim Wunderlich, Bin Yang, Martin Zimmermann
2022 conf
ESSDERC
Mamta Pradhan, Matthias Moser, Mohammed Alomari, Joachim N. Burghartz, Ingmar Kallfass
2022 conf
RWS
Dominik Wrana, Yigal Leiba, Laurenz John, Benjamin Schoch, Axel Tessmann, Ingmar Kallfass
2022 C conf
ETFA
Simon Kamm, Sandra Bickelhaupt, Kanuj Sharma, Nasser Jazdi, Ingmar Kallfass, Michael Weyrich
2021 J jnl
CoRR
Cristino Salcines, Sourabh Khandelwal, Ingmar Kallfass
2021 J jnl
IET Circuits Devices Syst.
Christopher M. Grötsch, Iulia Dan, Laurenz John, Sandrine Wagner, Ingmar Kallfass
2021 conf
CASE
Kanuj Sharma, Simon Kamm, Valentyna Afanasenko, Kevin Muñoz Barón, Ingmar Kallfass
2019 J jnl
IET Circuits Devices Syst.
Aleksey Dyskin, Ingmar Kallfass
2017 conf
RWS
Iulia Dan, Sebastian Rey, Thomas Merkle, Thomas Kürner, Ingmar Kallfass
2017 conf
RWS
Ingmar Kallfass, Jochen Antes, Axel Tessmann, Thomas Zwick, Ralf Henneberger
2015 conf
RWS
Florian Boes, Jochen Antes, Dominik Meier, Tobias Messinger, Daniel Müller, Ralf Henneberger, Axel Tessmann, Ingmar Kallfass
2015 conf
ICUWB
Karthik KrishneGowda, Tobias Messinger, Andreas C. Wolf, Rolf Kraemer, Ingmar Kallfass, Johann-Christoph Scheytt
2015 J jnl
IEICE Trans. Electron.
Ingmar Kallfass, Iulia Dan, Sebastian Rey, Parisa Harati, Jochen Antes, Axel Tessmann, Sandrine Wagner, Michael Kuri, Rainer Weber, Hermann Massler, Arnulf Leuther, Thomas Merkle, Thomas Kürner
2014 J jnl
JOCN
Swen Koenig, Jochen Antes, Daniel Lopez-Diaz, René Schmogrow, Thomas Zwick, Christian Koos, Wolfgang Freude, Juerg Leuthold, Ingmar Kallfass
2014 conf
RWS
Ulrich J. Lewark, Jochen Antes, Michael Kuri, Ralf Henneberger, Axel Tessmann, Sandrine Wagner, Arnulf Leuther, Thomas Zwick, Ingmar Kallfass
2014 conf
RWS
Jochen Antes, Florian Boes, Dominik Meier, Ulrich J. Lewark, Axel Tessmann, Arnulf Leuther, Ralf Henneberger, Ingmar Kallfass
2014 conf
ICTON
Wolfgang Freude, Swen Koenig, Daniel Lopez-Diaz, Jochen Antes, Florian Boes, Ralf Henneberger, Arnulf Leuther, Axel Tessmann, René Schmogrow, David Hillerkuss, Robert Palmer, Thomas Zwick, Christian Koos, Oliver Ambacher, Juerg Leuthold, Ingmar Kallfass
2013 conf
OFC/NFOEC
Swen Koenig, Florian Boes, Daniel Lopez-Diaz, Jochen Antes, Ralf Henneberger, René Schmogrow, David Hillerkuss, Robert Palmer, Thomas Zwick, Christian Koos, Wolfgang Freude, Oliver Ambacher, Ingmar Kallfass, Juerg Leuthold
2012 C conf
IGARSS
Ernst Weissbrodt, Axel Tessmann, Michael Schlechtweg, Ingmar Kallfass, Oliver Ambacher
2012 conf
EW
Ingmar Kallfass, Jochen Antes, Daniel Lopez-Diaz, Sandrine Wagner, Axel Tessmann, Arnulf Leuther
2012 conf
Future Security
Michael Schlechtweg, Axel Tessmann, Axel Hülsmann, Ingmar Kallfass, Arnulf Leuther, Rolf Aidam, Christian Zech, Ulrich J. Lewark, Hermann Massler, Markus Riessle, Martin Zink, Josef Rosenzweig, Oliver Ambacher
2012 conf
ISSSE
Jochen Antes, Tobias Mahler, Thomas Zwick, Axel Tessmann, Oliver Ambacher, Ingmar Kallfass
2012 conf
ISSSE
Aleksey Dyskin, Dan Ritter, Ingmar Kallfass
2011 J jnl
IEEE J. Solid State Circuits
Axel Tessmann, Arnulf Leuther, Volker Hurm, Ingmar Kallfass, Hermann Massler, Michael Kuri, Markus Riessle, Martin Zink, Rainer Lösch, Matthias Seelmann-Eggebert, Michael Schlechtweg, Oliver Ambacher
2011 C conf
IGARSS
Ernst Weissbrodt, Ingmar Kallfass, Axel Hülsmann, Axel Tessmann, Arnulf Leuther, Hermann Massler, Oliver Ambacher
2010 J jnl
IEEE J. Solid State Circuits
Stefan Koch, Marc Guthoerl, Ingmar Kallfass, Arnulf Leuther, Shin Saito
2008 J jnl
IEEE J. Solid State Circuits
Axel Tessmann, Ingmar Kallfass, Arnulf Leuther, Hermann Massler, Michael Kuri, Markus Riessle, Martin Zink, Rainer Sommer, Alfred Wahlen, Helmut Essen, Volker Hurm, Michael Schlechtweg, Oliver Ambacher
APK_CODE_ANALYSIS_PDD.md
← Index APK_CODE_ANALYSIS_PDD.md markdown
# APK Code Analysis — Product Design Document

**Author:** Engineering Team
**Date:** 2026-03-07
**Status:** Draft
**Target:** redb ingestor pipeline
**Depends on:** APK_FEATURES_PDD.md (APK static analysis extractors — implemented)

---

## 1. Overview

This document describes the design for adding **DEX code analysis** (decompilation, disassembly, call graphs, cross-references, and function similarity) to the redb ingestor pipeline. This is the Android equivalent of the Binary Ninja code analysis pipeline that exists for PE and ELF binaries.

### 1.1 Goals

- Decompile and disassemble APK DEX bytecode at the **method level**, producing per-method content and reference records analogous to the Binary Ninja `code_binja_*` tables
- Extract **call graphs and cross-references** (caller/callee relationships) for each method
- Compute **function similarity hashes** (SHA-256, ssdeep, TLSH, MinHash) for method-level clustering and hunting
- **Filter out library/framework code** to focus on user-written application logic — same philosophy as the `is_lib_or_thunk()` filter in Binary Ninja analysis
- Produce **decompiled Java source** (via JADX) and **smali disassembly** (via apktool) for each method, following the content/reference split pattern used by Binary Ninja tables
- Integrate with the existing APK extractor pipeline (runs after the Phase 1 APK extractors from `APK_FEATURES_PDD.md`)

### 1.2 Non-Goals

- **Native .so library analysis** — These are equivalent to external DLLs/shared libraries in PE/ELF. They are catalogued by `APKNativeLibExtractor` but not decompiled. If deep native analysis is needed, the existing ELF pipeline can be used on extracted `.so` files in a future phase.
- **Dynamic analysis / emulation** — Out of scope
- **Full APK repackaging / patching** — We use apktool for disassembly only, not rebuild
- **Inter-procedural data-flow analysis** (e.g., FlowDroid taint tracking) — Future consideration

### 1.3 Relationship to Existing Work

| Existing | New (this PDD) |
|----------|----------------|
| `APK_FEATURES_PDD.md` — APK metadata, manifest, permissions, certificates, DEX summary, resources, native libs | DEX **code-level** analysis: per-method decompilation, disassembly, call graphs, similarity hashes |
| `DecompileBinja` — PE/ELF code analysis via Binary Ninja | `DecompileAPK` — APK/DEX code analysis via androguard + JADX + apktool |
| `code_binja_*` ClickHouse tables | `code_apk_*` ClickHouse tables (same content/reference split pattern) |

---

## 2. Background

### 2.1 DEX Bytecode vs Native Code

| Aspect | PE/ELF (Binary Ninja) | APK/DEX (This PDD) |
|--------|----------------------|---------------------|
| Code format | Machine code (x86, ARM) | Dalvik bytecode (register-based VM) |
| Basic unit | Function (by address) | Method (by class + signature) |
| Disassembly | x86/ARM mnemonics | Smali (Dalvik assembly) |
| Decompilation | Pseudo-C (HLIL) | Java source code |
| Library filtering | `is_lib_or_thunk()` — symbol type | Package prefix filtering (e.g., `android.*`, `androidx.*`, `com.google.*`) |
| Similarity hashing | SHA-256 of normalized disassembly | SHA-256 of normalized smali |

### 2.2 Tool Selection

Three tools are combined to replicate the Binary Ninja analysis depth:

| Tool | Role | Integration | License |
|------|------|-------------|---------|
| **Androguard** (Python library) | Method enumeration, call graphs, cross-references, bytecode access, permissions analysis | Direct Python import — `from androguard.misc import AnalyzeAPK` | Apache 2.0 |
| **JADX** (Java CLI) | High-quality Java decompilation (equivalent to Binary Ninja HLIL) | subprocess (following CAPA/DIE pattern) | Apache 2.0 |
| **apktool** (Java CLI) | Smali disassembly with resource decoding (equivalent to Binary Ninja disassembly) | subprocess (following CAPA/DIE pattern) | Apache 2.0 |

**Why all three:**
- **Androguard** is the analysis engine — it provides call graphs, xrefs, and method enumeration natively in Python. However, its decompiler (DAD) produces lower-quality Java than JADX.
- **JADX** produces the best Java decompilation available. It is the industry standard for Android reverse engineering (47k+ GitHub stars).
- **apktool** produces canonical smali output with decoded resources. While androguard can access bytecode, apktool's smali output is the standard interchange format for Android RE.

### 2.3 Library Filtering Strategy

Native `.so` libraries are **not reverse-engineered** — they are equivalent to external DLLs in PE or shared libraries in ELF, and are already inventoried by `APKNativeLibExtractor`.

For DEX code, we filter out **framework/library packages** to focus on user-written code. This is the Android equivalent of `is_lib_or_thunk()` in the Binary Ninja pipeline.

**Default filter list** (configurable via environment variable `APK_LIBRARY_PREFIXES`):

```
android.*              # Android SDK
androidx.*             # AndroidX support libraries
com.google.android.*   # Google Play Services, Firebase
com.google.firebase.*  # Firebase
com.google.gson.*      # Gson JSON library
com.google.protobuf.*  # Protocol Buffers
kotlin.*               # Kotlin stdlib
kotlinx.*              # Kotlin extensions
org.apache.*           # Apache Commons
com.squareup.*         # OkHttp, Retrofit, Moshi
io.reactivex.*         # RxJava
org.reactivestreams.*  # Reactive Streams
com.facebook.*         # Facebook SDK
com.crashlytics.*      # Crashlytics
io.fabric.*            # Fabric
org.junit.*            # Test frameworks
org.mockito.*          # Test frameworks
```

Methods in filtered packages are still counted in call graph edges (caller/callee arrays) but their content is not stored in content tables. This mirrors how Binary Ninja records calls to library functions in `functions_caller`/`functions_call` arrays without decompiling the library functions themselves.

### 2.4 Packer Detection

Packer/protector detection for APKs uses **DetectItEasy (DIE)**, consistent with how packer detection works for PE/ELF/Mach-O in the existing pipeline. DIE already has signatures for common Android packers (Qihoo 360, Bangcle, Ijiami, Tencent Legu, Baidu, etc.).

The existing `DIEExtractor` runs as a format-agnostic extractor before format-specific analysis and requires no changes.

---

## 3. Architecture

### 3.1 Extractor Class Hierarchy

```
Extractor (redb/extractors/extractor.py)
└── DecompileAPK (NEW — redb/extractors/decompiler/DecompileAPK.py)
    ├── Uses: APKCodeAnalyzer (NEW — redb/extractors/decompiler/apk/analyzer.py)
    │   ├── AndroguardAnalysis — call graphs, xrefs, method enumeration
    │   ├── JADXDecompiler — Java decompilation (subprocess)
    │   └── ApktoolDisassembler — smali extraction (subprocess)
    └── Produces: multi_table ClickHouse export (same pattern as DecompileBinja)
```

**Design rationale:** `DecompileAPK` extends `Extractor` directly (not `APKExtractor`) because it follows the `DecompileBinja` pattern — a standalone extractor with its own analysis engine, rather than an APK metadata extractor that shares a parsed `APK` object. The APK parsing object from androguard is used internally but not shared with other extractors.

### 3.2 Analysis Pipeline Flow

```
APK file
  │
  ├─[1]─► apktool d <apk> ─► smali files on disk (temp dir)
  │
  ├─[2]─► jadx <apk> --no-res ─► Java source files on disk (temp dir)
  │
  └─[3]─► androguard AnalyzeAPK() ─► Analysis object (in-memory)
              │
              ├── Method enumeration ──► filter library packages
              │
              ├── For each user method:
              │     ├── Read smali from apktool output [1]
              │     ├── Read Java source from JADX output [2]
              │     ├── Get xrefs from Analysis object [3]
              │     ├── Compute content hashes (SHA-256 of smali, SHA-256 of Java)
              │     ├── Compute similarity hashes (ssdeep, TLSH of smali)
              │     └── Emit content + reference records
              │
              └── Call graph export ──► caller/callee arrays per method
```

Steps [1], [2], and [3] run in parallel (apktool and JADX as subprocess, androguard in-process). All three must complete before per-method analysis begins.

### 3.3 Content/Reference Split Pattern

Following the Binary Ninja schema pattern exactly:

- **Content tables** — Keyed by `function_hash` (SHA-256 of the method content). Deduplicated: if two APKs share identical method code, only one content record exists.
- **Reference tables** — Keyed by `(sha256, method_hash)`. Links a specific binary to its methods. Contains per-binary metadata (method name, class, address, callers, callees, fuzzy hashes).

This is the same pattern as `code_binja_decompiled_functions_content` / `code_binja_decompiled_functions_references`.

### 3.4 Method-Level Hashing

Hashing is computed at the **method level** for consistency with the Binary Ninja pipeline:

| Hash | Input | Purpose |
|------|-------|---------|
| `decompiled_method_hash` | SHA-256 of decompiled Java source (whitespace-normalized) | Content deduplication, exact match |
| `smali_method_hash` | SHA-256 of smali body (instructions only, no `.method`/`.end method` directives) | Content deduplication, exact match |
| `ssdeep_smali` | ssdeep of smali body | Fuzzy similarity search |
| `tlsh_smali` | TLSH of smali body | Fuzzy similarity search |
| `minhash_smali` | MinHash signature of smali instruction n-grams | LSH-based similarity clustering |

### 3.5 Obfuscation Indicators (per method)

Computed from the smali representation:

- `short_method_name` — Method name is <= 2 characters (a, b, c — typical R8/ProGuard output)
- `short_class_name` — Enclosing class has a single-letter name
- `has_string_encryption` — Method contains `const-string` followed by decryption-pattern calls
- `has_reflection_calls` — Method uses `java.lang.reflect.*` APIs
- `excessive_goto_count` — Number of `goto` instructions exceeds threshold (control flow flattening indicator)

---

## 4. Data Model

### 4.1 New Dataclasses

```python
@dataclass
class APKDecompiledMethodContent:
    """Decompiled Java source for a single method (content table — deduplicated by hash)."""
    decompiled_method_hash: str          # SHA-256 of normalized Java source
    decompiled_method: str               # Full Java method source
    method_type: str                     # "USER" or "LIBRARY"
    has_string_encryption: bool = False
    has_reflection_calls: bool = False
    excessive_goto_count: bool = False


@dataclass
class APKDecompiledMethodReference:
    """Links a specific APK to one of its decompiled methods (reference table)."""
    sha256: str                          # APK hash
    sha1: str
    md5: str
    decompiled_method_hash: str          # FK to content table
    smali_method_hash: Optional[str]     # FK to smali content table
    class_name: str                      # e.g., "com.example.MainActivity"
    method_name: str                     # e.g., "onCreate"
    method_signature: str                # e.g., "(Landroid/os/Bundle;)V"
    method_prototype: str                # e.g., "void onCreate(Bundle)"
    functions_caller: List[str]          # Methods that call this method
    functions_call: List[str]            # Methods called by this method


@dataclass
class APKSmaliMethodContent:
    """Smali disassembly for a single method (content table — deduplicated by hash)."""
    smali_method_hash: str               # SHA-256 of normalized smali body
    smali_method: str                    # Full smali method body
    method_type: str                     # "USER" or "LIBRARY"
    instructions_count: int = 0
    register_count: int = 0
    has_string_encryption: bool = False
    has_reflection_calls: bool = False
    excessive_goto_count: bool = False


@dataclass
class APKSmaliMethodReference:
    """Links a specific APK to one of its smali methods (reference table)."""
    sha256: str
    sha1: str
    md5: str
    smali_method_hash: str               # FK to content table
    decompiled_method_hash: Optional[str] # FK to decompiled content table
    class_name: str
    method_name: str
    method_signature: str
    ssdeep_smali: Optional[str] = None
    tlsh_smali: Optional[str] = None


@dataclass
class APKMethodSimilarityMetrics:
    """Similarity hashes for method-level clustering (keyed by smali hash)."""
    smali_method_hash: str
    cyclomatic_complexity: Optional[int] = None
    ssdeep_smali: Optional[str] = None
    tlsh_smali: Optional[str] = None
    minhash: Optional[List[int]] = None


@dataclass
class APKCodeAnalysisError:
    """Error encountered during method analysis."""
    sha256: str
    class_name: Optional[str] = None
    method_name: Optional[str] = None
    error_location: str = ""             # "jadx", "apktool", "androguard", "analysis"
    error_message: Optional[str] = None
    error_type: Optional[str] = None
```

### 4.2 Tag Enum Addition

```python
# In redb/extractors/enum.py
APK_DECOMPILED = "apk_decompiled"
```

### 4.3 ClickHouse Tables

| Table | Key | Pattern | Analog |
|-------|-----|---------|--------|
| `code_apk_decompiled_methods_content` | `decompiled_method_hash` | Content (deduplicated) | `code_binja_decompiled_functions_content` |
| `code_apk_decompiled_methods_references` | `(sha256, decompiled_method_hash)` | Reference (per-binary) | `code_binja_decompiled_functions_references` |
| `code_apk_smali_methods_content` | `smali_method_hash` | Content (deduplicated) | `code_binja_disassembled_functions_content` |
| `code_apk_smali_methods_references` | `(sha256, smali_method_hash)` | Reference (per-binary) | `code_binja_disassembled_functions_references` |
| `code_apk_method_similarity_metrics` | `smali_method_hash` | Similarity | `code_binja_function_similarity_metrics` |
| `code_apk_analysis_errors` | `(sha256, class_name, method_name)` | Errors | `function_analysis_errors_binja` |

All tables use `ReplacingMergeTree(analysis_date)` engine, consistent with existing schema.

---

## 5. External Tool Management

### 5.1 JADX

- **Invocation:** `jadx --no-res --no-imports --threads-count 2 --output-dir <tmpdir> <apk_path>`
- **Flags:**
  - `--no-res` — Skip resource decompilation (androguard handles resources)
  - `--no-imports` — Omit import statements for cleaner per-method extraction
  - `--threads-count 2` — Limit threads (same as Binary Ninja worker thread limit)
- **Output:** Java source files in `<tmpdir>/<package>/<Class>.java`
- **Timeout:** Configurable via `JADX_TIMEOUT` env var (default: 600s)
- **Path:** Configurable via `JADX_PATH` env var (default: `jadx`)
- **Error handling:** If JADX fails for a specific APK, the decompiled content tables are skipped but smali analysis continues. Error logged to `code_apk_analysis_errors`.

### 5.2 apktool

- **Invocation:** `apktool d --no-res --force --output <tmpdir> <apk_path>`
- **Flags:**
  - `--no-res` — Skip resource decoding (only want smali)
  - `--force` — Overwrite output directory if exists
- **Output:** Smali files in `<tmpdir>/smali/com/example/ClassName.smali` (one per class, containing all methods)
- **Timeout:** Configurable via `APKTOOL_TIMEOUT` env var (default: 600s)
- **Path:** Configurable via `APKTOOL_PATH` env var (default: `apktool`)
- **Error handling:** Same as JADX — if apktool fails, smali content tables are skipped but decompiled Java analysis continues. Error logged.

### 5.3 Androguard

- **Invocation:** Direct Python API — `AnalyzeAPK(filepath)` returns `(APK, list[DEX], Analysis)`
- **The `Analysis` object provides:**
  - `get_methods()` — All `MethodAnalysis` objects
  - `get_call_graph()` — networkx `MultiDiGraph` of method calls
  - `MethodAnalysis.get_xref_from()` — Who calls this method
  - `MethodAnalysis.get_xref_to()` — What this method calls
  - `MethodAnalysis.get_method()` — Access to `EncodedMethod` for bytecode
- **No timeout needed** — runs in-process, same Python process

---

## 6. Ingestor Integration

In `workers.py:process_binary_file()`, the APK branch will be extended to run `DecompileAPK` after the existing APK extractors:

```python
# Existing APK extractors (from APK_FEATURES_PDD.md)
for module in apk_modules:
    extractor = module(filepath, logger, exporters=exporters, ...)
    extractor.export_data()

# NEW: Code analysis (this PDD)
if "DecompileAPK" in selected_modules or "all" in selected_modules:
    decompiler = DecompileAPK(
        filepath, logger, exporters=exporters,
        index_prefix=index_prefix, filetype="apk",
    )
    decompiler.export_data()
```

The `DecompileAPK` extractor runs with its own timeout (configurable via `APK_DECOMPILE_TIMEOUT`, default: 1800s) using the same daemon-thread pattern as `DecompileBinja`.

---

## 7. New Dependencies

### 7.1 Required (system-level)

| Tool | Installation | Version | License | Purpose |
|------|-------------|---------|---------|---------|
| **JADX** | System package or download from GitHub releases | >= 1.5 | Apache 2.0 | Java decompilation |
| **apktool** | System package or download from GitHub releases | >= 2.9 | Apache 2.0 | Smali disassembly |
| **Java Runtime** | System package (`openjdk-17-jre` or similar) | >= 11 | GPL+CE | Required by JADX and apktool |

### 7.2 Required (Python — already installed)

| Library | Current Version | Usage in this PDD |
|---------|----------------|-------------------|
| `androguard` | >=4.1 | Call graphs, xrefs, method enumeration (already in requirements.txt) |
| `ppdeep` | installed | ssdeep fuzzy hashing (already used by Binary Ninja pipeline) |
| `py-tlsh` | installed | TLSH fuzzy hashing (already used by Binary Ninja pipeline) |
| `mmh3` | installed | MinHash computation (already used by Binary Ninja pipeline) |
| `networkx` | installed via androguard | Call graph representation (transitive dependency) |

### 7.3 No new Python dependencies required

All Python libraries needed are already in `requirements.txt`. The only new system-level dependencies are JADX, apktool, and a Java runtime.

---

## 8. Implementation Phases

### Phase 1 — Core Infrastructure

1. `APKCodeAnalyzer` class — androguard integration (method enumeration, call graph, xrefs, library filtering)
2. `JADXDecompiler` wrapper — subprocess management with timeout, output parsing
3. `ApktoolDisassembler` wrapper — subprocess management with timeout, smali parsing
4. Method-level content extraction and hashing logic
5. Unit tests for all Phase 1 components

### Phase 2 — Extractor and Data Export

6. `DecompileAPK` extractor class (following `DecompileBinja` pattern)
7. ClickHouse table creation functions
8. Multi-table export (`prepare_export_data`) for all 6 tables
9. Integration with `workers.py` dispatch
10. Unit tests for extractor, schema, and export
11. Update `TEST_INDEX.md`

### Phase 3 — Similarity and Obfuscation Analysis

12. Method-level similarity hash computation (ssdeep, TLSH, MinHash on smali)
13. Obfuscation indicator computation per method
14. `code_apk_method_similarity_metrics` table population
15. Unit tests for similarity and obfuscation
16. Update `TEST_INDEX.md`

### Phase 4 — Integration Testing and Hardening

17. End-to-end integration tests with real APK samples (benign + malicious + obfuscated)
18. Edge case handling: multi-DEX, empty DEX, packed APKs, APKs with no user code
19. Performance profiling and timeout tuning
20. Final `TEST_INDEX.md` update

---

## 9. Testing Strategy

### 9.1 Unit Tests

All unit tests mock external tools (JADX, apktool, androguard) and require no system dependencies:

- **Analyzer tests** — Method enumeration, library filtering, call graph extraction, xref parsing
- **JADX wrapper tests** — Subprocess invocation, output parsing, timeout handling, error recovery
- **Apktool wrapper tests** — Same as JADX
- **Hashing tests** — SHA-256 normalization, ssdeep/TLSH computation, MinHash signature generation
- **Extractor tests** — `DecompileAPK.extract()`, `prepare_export_data()`, multi-table schema validation
- **Smali parsing tests** — Method boundary detection, instruction extraction, register counting

### 9.2 Integration Tests

Require JADX, apktool, and Java installed:

- Full pipeline run on known APK samples
- Cross-validate decompiled output against known method signatures
- Verify ClickHouse export column counts and types
- Test with obfuscated APKs (ProGuard/R8 output)

### 9.3 Markers

```python
@pytest.mark.apk           # All APK tests
@pytest.mark.decompile      # All decompiler tests
@pytest.mark.unit           # No external deps
@pytest.mark.integration    # Requires JADX/apktool/Java
```

---

## 10. Configuration

All configuration via environment variables, consistent with existing extractors:

| Variable | Default | Description |
|----------|---------|-------------|
| `JADX_PATH` | `jadx` | Path to JADX binary |
| `JADX_TIMEOUT` | `600` | JADX subprocess timeout (seconds) |
| `APKTOOL_PATH` | `apktool` | Path to apktool binary |
| `APKTOOL_TIMEOUT` | `600` | apktool subprocess timeout (seconds) |
| `APK_DECOMPILE_TIMEOUT` | `1800` | Overall decompilation timeout (seconds) |
| `APK_LIBRARY_PREFIXES` | (see §2.3) | Comma-separated package prefixes to filter |
| `APK_MIN_METHOD_INSTRUCTIONS` | `5` | Minimum smali instruction count to analyze a method |

---

## 11. Open Questions / Future Work

1. **ProGuard/R8 mapping file support** — If mapping files are bundled (rare in malware, common in crash reports), JADX can use them to restore original names. Deferred.
2. **Kotlin-specific analysis** — Kotlin metadata annotations could provide richer type information. Deferred.
3. **Cross-DEX analysis** — Multi-DEX APKs may have cross-DEX method calls. Androguard handles this via `AnalyzeAPK()` which loads all DEX files into a single `Analysis` object. No special handling needed.
4. **JADX as Java library via JPype** — Could eliminate subprocess overhead. Deferred in favor of the proven subprocess pattern, but may be revisited if performance is an issue.

---

## Appendix A — CFG Feature Parity with Binary Ninja Pipeline

**Date:** 2026-03-13
**Status:** Planned (Phase 5)
**Depends on:** Phases 13 complete

### A.1 Motivation

The Binary Ninja pipeline produces a dedicated `code_binja_cfg_functions` table with 17 fields capturing graph topology, structural hashes, and per-block feature vectors. The current APK pipeline computes only basic graph scalars (`block_count`, `edge_count`, `cyclomatic_complexity`, `loop_count`, `max_depth`, `max_fan_out`) and bundles them into the `code_apk_method_similarity_metrics` table alongside fuzzy hashes.

Analysis shows that **all advanced CFG features can be computed from smali** — this is not a limitation of Java bytecode. The existing APK infrastructure already:

- Builds `successors[]` adjacency lists from smali control flow (`smali_cfg.py`)
- Computes per-block ACFG feature vectors using the same 8-category schema as Binary Ninja (`smali_cfg.py:_build_block_features`)
- Normalizes Dalvik opcodes into semantic categories equivalent to LLIL categories (`smali_normalization.py`)

The generic functions in `cfg_features.py` (`compute_topology_hash`, `compute_md_index_topdown/bottomup`, `compute_wl_minhash`, `compute_cfg_feature_tlsh`, `pack_adjacency`) operate on adjacency lists and block feature arrays — they have no Binary Ninja dependency and can be called directly from the APK pipeline.

### A.2 Table Restructuring

Split the current single table into two, mirroring the Binja pattern:

#### `code_apk_method_similarity_metrics` (content-based fuzzy matching)

Retains only fuzzy hashes and instruction-sequence similarity data:

| Column | Type | Change |
|--------|------|--------|
| `smali_method_hash` | FixedString(64) | Unchanged |
| `cyclomatic_complexity` | Nullable(UInt16) | Stays (duplicated in both, same as Binja) |
| `ssdeep_smali` | Nullable(String) | Unchanged |
| `tlsh_smali` | Nullable(FixedString(72)) | Unchanged |
| `minhash` | Array(UInt8) | Unchanged |
| `analysis_date` | DateTime64(3, 'UTC') | Unchanged |

Removed from this table: `block_count`, `edge_count`, `loop_count`, `max_depth`, `max_fan_out` — these move to the CFG table.

#### `code_apk_cfg_methods` (NEW — structural/topological similarity)

Mirrors `code_binja_cfg_functions`:

| Column | Type | Source | Analog in Binja |
|--------|------|--------|-----------------|
| `smali_method_hash` | FixedString(64) | Existing | `disassembled_function_hash` |
| `cfg_topology_hash` | FixedString(16) | NEW — `cfg_features.compute_topology_hash()` | Same |
| `block_count` | UInt16 | Moved from similarity table | Same |
| `edge_count` | UInt16 | Moved from similarity table | Same |
| `instructions_count` | UInt32 | Existing (total Dalvik instructions) | `llil_total_operations` |
| `call_count` | UInt16 | NEW — count of `invoke-*` instructions | Same |
| `cyclomatic_complexity` | UInt16 | Moved from similarity table | Same |
| `loop_count` | UInt8 | Moved from similarity table | Same |
| `max_depth` | UInt16 | Moved from similarity table | Same |
| `max_fan_out` | UInt8 | Moved from similarity table | Same |
| `md_index_topdown` | UInt64 | NEW — `cfg_features.compute_md_index_topdown()` | Same |
| `md_index_bottomup` | UInt64 | NEW — `cfg_features.compute_md_index_bottomup()` | Same |
| `prime_product_smali` | UInt64 | NEW — Dalvik opcode → prime mapping | `prime_product_llil` |
| `cfg_feature_tlsh` | Nullable(FixedString(72)) | NEW — `cfg_features.compute_cfg_feature_tlsh()` | Same |
| `wl_minhash` | Array(UInt8) | NEW — `cfg_features.compute_wl_minhash()`, 128 elements | Same |
| `bb_features` | Array(Array(UInt16)) | Existing (computed, not exported) | Same |
| `cfg_adjacency` | Array(UInt32) | NEW — `cfg_features.pack_adjacency()` | Same |
| `analysis_date` | DateTime64(3, 'UTC') | | Same |

### A.3 Naming Differences from Binja

Two columns are intentionally renamed to reflect what the data actually represents:

- **`prime_product_smali`** (not `prime_product_llil`) — the prime mapping is applied to normalized Dalvik opcodes, not LLIL. Semantically equivalent for APK-to-APK comparison but not numerically comparable to Binja values.
- **`instructions_count`** (not `llil_total_operations`) — counts Dalvik instructions, not LLIL operations. LLIL decomposes machine instructions into sub-operations; Dalvik bytecode is already at a higher abstraction level where one instruction ≈ one operation.

### A.4 Implementation Requirements

| Task | Effort | Notes |
|------|--------|-------|
| Build `predecessors[]` from `successors[]` in `smali_cfg.py` | ~5 lines | Trivial reverse mapping |
| Define `SMALI_OP_PRIMES` mapping | ~30 lines | Map semantic categories from `smali_normalization.py` to same primes used in `cfg_features.py` |
| Wire `cfg_features.py` functions into `smali_cfg.py` | ~40 lines | Call `compute_topology_hash`, `compute_md_index_*`, `compute_wl_minhash`, `compute_cfg_feature_tlsh`, `pack_adjacency` |
| Export `bb_features` (already computed, not exported) | ~5 lines | Add to results dict |
| Count `invoke-*` instructions for `call_count` | ~5 lines | Filter in instruction loop |
| New `code_apk_cfg_methods` table export in `DecompileAPK.py` | ~60 lines | Follow existing export pattern |
| Slim down `code_apk_method_similarity_metrics` export | ~10 lines | Remove moved columns |
| ClickHouse schema for new table | ~30 lines | Mirror `code_binja_cfg_functions` |
| Unit tests | ~100 lines | Test new fields, reuse patterns from `test_cfg_features.py` |

**Total estimated: ~285 lines of code changes.**

### A.5 What Cannot Be Identical Cross-Platform

The `prime_product_smali` values are **not numerically comparable** to `prime_product_llil` from the Binja pipeline. LLIL decomposes native instructions into sub-operations (e.g., one x86 `push` becomes `STORE` + `SET_REG`), while Dalvik bytecode maps 1:1 to semantic categories. The prime products are valid for APK-vs-APK similarity and APK-vs-APK clustering, which is the intended use case.

All other fields (`cfg_topology_hash`, `md_index_*`, `wl_minhash`, `cfg_feature_tlsh`, `bb_features`, `cfg_adjacency`) are computed from the same generic algorithms and are structurally equivalent across platforms.