Vasilis Syrgkanis

# APK Extractor — Product Design Document

**Author:** Engineering Team
**Date:** 2026-02-21
**Status:** Draft
**Target:** redb ingestor pipeline

---

## 1. Overview

This document describes the design for adding APK (Android Package) static analysis to the redb ingestor pipeline. The APK extractor will follow the same architecture used by the existing PE, ELF, and Mach-O extractors: a format-specific base class (`APKExtractor`) with specialized sub-extractors for each analysis dimension.

### 1.1 Goals

- Extract comprehensive static metadata from Android APK files, comparable in depth to our PE/ELF/Mach-O analysis
- Follow the established extractor architecture (base class, sub-extractors, dataclasses, dual-export to Elasticsearch and ClickHouse)
- Enable clustering, hunting, and pivoting on APK-specific fields (permissions, certificates, DEX API usage, package names)
- Integrate with the existing format-agnostic extractors (BasicProperties, Hashes, DIE, CAPA, YARA, Strings, IOC)

### 1.2 Non-Goals

- Dynamic analysis / sandbox execution (out of scope)
- Full DEX decompilation to Java/smali (out of scope; may be a future extension)
- Deep analysis of embedded native `.so` libraries (inventory only; full ELF pipeline deferred)
- Recursive ingestion of APKs embedded inside other APKs (flag only; full recursive ingestion deferred)

---

## 2. Background

### 2.1 What Is an APK

An APK is a ZIP archive containing an Android application. Its internal structure:

| Path | Contents |
|------|----------|
| `AndroidManifest.xml` | Binary Android XML — package name, permissions, components, SDK versions, intent filters |
| `classes.dex` (+ `classes2.dex`, ...) | Dalvik bytecode — compiled Java/Kotlin code |
| `resources.arsc` | Compiled resource table (strings, dimensions, styles) |
| `res/` | Layouts, drawables, raw resources |
| `lib/<abi>/` | Native shared libraries (`.so`) per CPU architecture |
| `META-INF/` | JAR signing (v1 scheme), CERT.RSA/DSA certificates |
| `assets/` | Arbitrary files bundled by the developer |

### 2.2 Current State

The ingestor already detects APK files via Magika (`ingestor.py:1668`), but the handler is a no-op — it logs `"APK file detected"` and returns without running any extractors. All infrastructure for registration, dispatch, and export is already in place.

### 2.3 Industry Reference

This design was informed by analysis of existing platforms:

- **VirusTotal** — Extracts: hashes (MD5, SHA-1, SHA-256, SSDEEP, TLSH, Permhash), Android metadata (package name, SDK versions, main activity), certificate attributes, permissions with danger flags, full component lists (activities, services, receivers, providers), intent filters, and capability indicators ("performs reflection calls", "makes use of telephony related APIs")
- **Koodous** (https://docs.koodous.com/) — Powered by Androguard for static analysis. Extracts: package name, app name, activities, services, receivers, providers, permissions, intent filters, certificate (SHA-1, issuer, subject), hardcoded URLs, SDK versions. Supports YARA rules with an Androguard module for matching on all these fields
- **APKiD** (https://github.com/rednaga/APKiD) — "PEiD for Android". Signature-based identification of compilers, packers, obfuscators, protectors, anti-VM/debug/root techniques. Uses YARA rules on DEX/APK/ELF. Available under GPL or commercial perpetual license (https://github.com/rednaga/APKiD/blob/master/LICENSE.COMMERCIAL)
- **APKDetect** (https://www.apkdetect.com/) — Malware family identification (30+ families), configuration extraction, loader recognition, shared code detection

---

## 3. Architecture

### 3.1 Existing Extractor Pattern

All extractors in redb follow this hierarchy:

```
Extractor (abstract base — redb/extractors/extractor.py)
├── PEExtractor (redb/extractors/pe_extractor.py)
│   ├── PEFeaturesExtractor
│   ├── PEImportExtractor
│   ├── PESectionExtractor
│   └── ...
├── ELFExtractor (redb/extractors/elf_extractor.py)
│   ├── ELFFeaturesExtractor
│   ├── ELFImportExtractor
│   └── ...
├── MachOExtractor (redb/extractors/macho_extractor.py)
│   ├── MachOFeaturesExtractor
│   ├── MachOImportExtractor
│   └── ...
└── [Format-agnostic extractors]
    ├── BasicPropertiesExtractor
    ├── HashExtractor
    ├── DIEExtractor
    ├── CAPAExtractor
    ├── YaraExtractor
    └── StringsExtractor
```

Each concrete extractor implements:
- `tag()` — returns a `Tag` enum value identifying the extraction category
- `extract()` — performs the analysis, returns a dataclass instance or `None`
- `prepare_export_data(exporter_type)` — formats data for Elasticsearch or ClickHouse
- `get_clickhouse_table()` — returns the target ClickHouse table name

The format-specific base class (e.g., `PEExtractor`) handles:
- Accepting a pre-parsed object (e.g., `pe=`) to avoid redundant parsing
- Providing shared helper methods used by multiple sub-extractors
- Passing `precomputed_hashes` to the parent `Extractor.__init__()` when available

The ingestor dispatches extractors by filetype detected via Magika, running format-agnostic extractors (BasicProperties, Hashes, DIE, CAPA, YARA) followed by the format-specific list.

### 3.2 APK Extractor Architecture

```
Extractor
└── APKExtractor (NEW — redb/extractors/apk_extractor.py)
    ├── APKFeaturesExtractor      (redb/extractors/apk_extractors/apk_features.py)
    ├── APKManifestExtractor      (redb/extractors/apk_extractors/apk_manifest.py)
    ├── APKPermissionsExtractor   (redb/extractors/apk_extractors/apk_permissions.py)
    ├── APKSignatureExtractor     (redb/extractors/apk_extractors/apk_signature.py)
    ├── APKDexExtractor           (redb/extractors/apk_extractors/apk_dex.py)
    ├── APKResourceExtractor      (redb/extractors/apk_extractors/apk_resources.py)
    ├── APKNativeLibExtractor     (redb/extractors/apk_extractors/apk_native_libs.py)
    └── APKInconsistencyTestsExtractor (redb/extractors/apk_extractors/apk_inconsistency_tests.py)
```

The `APKExtractor` base class will:
- Accept an optional pre-parsed `androguard.core.apk.APK` object (`apk=`) to share across sub-extractors
- Parse the APK once in `__init__` if not provided
- Provide shared helpers: `_get_manifest()`, `_get_certificates()`, `_list_files()`, `_is_valid_apk()`

### 3.3 Ingestor Integration

In `ingestor.py:process_binary_file()`, the `elif filetype == "apk"` branch will be expanded to:

1. Parse the APK once using Androguard (`APK(filepath)`)
2. Run format-agnostic extractors (BasicProperties, Hashes, DIE, YARA) — same as PE/ELF/Mach-O
3. Run APK-specific extractors with the shared `apk` object
4. Run Strings + IOC extractors if applicable

---

## 4. Extractor Specifications

### 4.1 APKFeaturesExtractor

**Purpose:** Core APK metadata — the equivalent of `PEFeaturesExtractor` or `ELFFeaturesExtractor`.

**Extracted fields:**
- `package_name` — Android package identifier (e.g., `com.example.app`)
- `app_name` — Human-readable application name
- `version_code` — Internal integer version
- `version_name` — Display version string (e.g., `"1.2.3"`)
- `min_sdk_version` — Minimum Android API level
- `target_sdk_version` — Target Android API level
- `compile_sdk_version` — Compile SDK (if available)
- `main_activity` — Launcher activity class name
- `is_debuggable` — Whether `android:debuggable="true"`
- `allow_backup` — Whether `android:allowBackup="true"`
- `uses_cleartext_traffic` — Whether `android:usesCleartextTraffic="true"`
- `supported_abis` — List of ABIs from `lib/` directory (e.g., `["arm64-v8a", "armeabi-v7a"]`)
- `dex_count` — Number of DEX files
- `total_dex_size` — Combined DEX file size in bytes
- `total_file_count` — Total number of files in the APK archive
- `has_native_code` — Whether `lib/` contains `.so` files
- `has_assets` — Whether `assets/` directory is non-empty
- `uses_libraries` — Declared `<uses-library>` entries
- `earliest_content_modification` — Earliest timestamp from ZIP entry metadata
- `latest_content_modification` — Latest timestamp from ZIP entry metadata
- `contains_embedded_apk` — Whether the archive contains nested APK files (flag only)

**Tag:** `APK_FEATURES`
**ClickHouse table:** `redb_apk_features`

### 4.2 APKManifestExtractor

**Purpose:** Full AndroidManifest.xml component enumeration, mirroring what VirusTotal and Koodous display.

**Extracted fields:**
- `activities` — List of Activity class names with `exported` flag
- `services` — List of Service class names with `exported` flag
- `receivers` — List of BroadcastReceiver class names with `exported` flag
- `providers` — List of ContentProvider class names with `exported` flag
- `intent_filters_by_action` — Aggregated list of all intent filter actions
- `intent_filters_by_category` — Aggregated list of all intent filter categories
- `uses_features` — Declared hardware/software features (e.g., `android.hardware.camera`)
- `meta_data` — Key-value pairs from `<meta-data>` elements
- `manifest_xml` — Full decompiled AndroidManifest.xml as plain text

**Tag:** `APK_MANIFEST`
**ClickHouse table:** `redb_apk_manifest` (one row per APK for aggregated data), `redb_apk_components` (one row per component)

### 4.3 APKPermissionsExtractor

**Purpose:** Dedicated permission analysis with protection-level classification and permhash computation.

**Extracted fields:**
- `permissions` — List of requested permissions with protection level (`normal`, `dangerous`, `signature`, `signatureOrSystem`)
- `dangerous_permissions` — Filtered list of dangerous permissions only
- `dangerous_permission_count` — Count of dangerous permissions
- `custom_permissions` — Permissions defined by the app itself (`<permission>` declarations)
- `total_permission_count` — Total number of requested permissions
- `permhash` — SHA-256 of sorted permission list (Mandiant/Google permhash — https://github.com/google/permhash)

The `permhash` value will also be added to the `Hashes` dataclass in `redb/models/dataclasses.py` so it appears alongside `imphash`, `symhash`, etc. in the unified hash record.

**Tag:** `APK_PERMISSIONS`
**ClickHouse table:** `redb_apk_permissions` (one row per permission per APK)

### 4.4 APKSignatureExtractor

**Purpose:** Certificate and signing scheme analysis, analogous to `PESignatureExtractor` and `MachOSignatureExtractor`.

**Extracted fields:**
- `signature_scheme_versions` — Which signing schemes are present (v1 JAR, v2, v3, v4)
- `is_signed` — Whether the APK has a valid signature
- `number_of_certificates` — Certificate count in the chain
- `x509_certificates` — List of certificate details:
  - `subject` (Distinguished Name)
  - `issuer` (Distinguished Name)
  - `serial_number`
  - `valid_from` / `valid_to`
  - `thumbprint_sha1`
  - `thumbprint_sha256`
  - `algorithm`
  - `key_size`
  - `is_self_signed`
- `signer_subject` — Primary signer's subject DN (convenience field)
- `signer_issuer` — Primary signer's issuer DN

**Tag:** `APK_SIGNATURE`
**ClickHouse table:** `redb_apk_signature`

### 4.5 APKDexExtractor

**Purpose:** DEX file analysis — summarized with categorized API usage (not full method enumeration).

**Output structure (per DEX file):**

- `filename` — DEX filename (e.g., `classes.dex`)
- `sha256` — SHA-256 of the DEX file
- `class_count` — Total number of classes
- `method_count` — Total number of methods
- `string_count` — Total number of string constants
- `top_packages` — List of `{package, class_count}` for top-level Java packages
- `api_usage` — Categorized sensitive API calls:
  - `reflection` — `java.lang.reflect.*`, `Class.forName`, etc.
  - `crypto` — `javax.crypto.*`, `java.security.*`
  - `dynamic_loading` — `DexClassLoader`, `PathClassLoader`, `InMemoryDexClassLoader`
  - `telephony` — `TelephonyManager` methods
  - `sms` — `SmsManager`, `SmsReceiver`
  - `network` — `HttpURLConnection`, `OkHttp`, `Volley`, `Retrofit`
  - `native` — `System.loadLibrary`, `Runtime.exec`
  - `device_info` — `Build.*`, `Settings.Secure.ANDROID_ID`, IMEI/IMSI access
  - `file_io` — `FileOutputStream`, `SharedPreferences`, `SQLiteDatabase`
  - `ipc` — `ContentResolver`, `BroadcastReceiver`, `Binder`
- `obfuscation_indicators`:
  - `short_class_names_pct` — Percentage of classes with names <= 2 chars
  - `short_method_names_pct` — Percentage of methods with names <= 2 chars
  - `non_ascii_identifiers` — Count of identifiers with non-ASCII characters
  - `avg_class_name_length` — Average class name length

**Tag:** `APK_DEX`
**ClickHouse table:** `redb_apk_dex` (one row per DEX file), `redb_apk_dex_api_usage` (one row per API category per DEX)

### 4.6 APKResourceExtractor

**Purpose:** Inventory of embedded resources with filetype detection for suspicious content.

**Extracted fields:**
- `total_resource_count` — Total files in `res/` and `assets/`
- `total_resource_size` — Combined size
- `resource_inventory` — List of `{path, size, sha256, filetype_magika}` for each file
- `suspicious_files` — Files detected as ELF, PE, DEX, APK, ZIP, script, or other executable types
- `suspicious_file_count` — Count of suspicious files

**Tag:** `APK_RESOURCES`
**ClickHouse table:** `redb_apk_resources`

### 4.7 APKNativeLibExtractor

**Purpose:** Inventory of native `.so` libraries per ABI. No deep ELF analysis (inventory only).

**Extracted fields:**
- `native_lib_count` — Total `.so` file count
- `abis` — List of ABI directories present (e.g., `["arm64-v8a", "x86"]`)
- `native_libs` — List of `{abi, filename, size, sha256}` per `.so` file
- `known_packer_libs` — Any `.so` files matching known packer/protector library names (e.g., `libjiagu.so`, `libsecexe.so`, `libDexHelper.so`, `libprotectClass.so`)

**Tag:** `APK_NATIVE_LIBS`
**ClickHouse table:** `redb_apk_native_libs`

### 4.8 APKInconsistencyTestsExtractor

**Purpose:** Anomaly and anti-analysis detection, analogous to `PEInconsistencyTestsExtractor`.

**Extracted fields:**
- `test_zip_bomb` — Compression ratio exceeds threshold
- `test_zip_duplicate_entries` — ZIP contains duplicate filenames
- `test_zip_path_traversal` — ZIP entries with `../` path traversal
- `test_zip_suspicious_timestamps` — Timestamps in the future or at epoch (1980-01-01)
- `test_hidden_dex_files` — DEX files outside standard `classes*.dex` naming or in unexpected locations
- `test_manifest_component_mismatch` — Declared components that don't exist in DEX, or undeclared code
- `test_debuggable_release` — `android:debuggable="true"` combined with a release signature
- `test_emulator_detection_strings` — Presence of Build.FINGERPRINT/MANUFACTURER emulator check strings in DEX
- `test_debugger_detection` — Presence of `Debug.isDebuggerConnected()` calls
- `test_root_detection` — Presence of root detection patterns (su binary checks, Superuser.apk)

**Tag:** `APK_INCONSISTENCY_TESTS`
**ClickHouse table:** `redb_apk_inconsistency_tests`

---

## 5. New Dependencies

### 5.1 Required

| Library | Version | License | Purpose |
|---------|---------|---------|---------|
| **androguard** | >=4.1 | Apache 2.0 | Core APK parsing: binary XML manifest, DEX analysis, certificate extraction, permissions |
| **permhash** | latest | Apache 2.0 | Compute permhash (SHA-256 of sorted permissions) for APK clustering |

### 5.2 Already Available (no changes)

| Library | Current Version | Usage |
|---------|----------------|-------|
| `zipfile` (stdlib) | — | APK archive traversal, ZIP anomaly detection |
| `pyelftools` | 0.32 | Could be used for native .so analysis if scope expands |
| `lief` | 0.17.3 | Has `lief.DEX` module; complement to androguard for DEX class/method enumeration |
| `cryptography` | 43.0.3 | Certificate chain validation (used by PE/Mach-O signature extractors) |
| `Pillow` | 10.4.0 | Icon extraction if needed |
| `magika` | 1.0.1 | Filetype detection for embedded resources |

### 5.3 Future Consideration

| Library | License | Purpose | Notes |
|---------|---------|---------|-------|
| **APKiD** | GPL-3.0 or Commercial (perpetual, royalty-free) | Packer/protector/obfuscator identification | "PEiD for Android". Detects compilers (dx, dexlib, r8), packers (AppGuard, DingXiang, JiaguK), obfuscators (DexGuard, BlackObfuscator), protectors (DexProtector, DxShield), anti-VM/debug/root. Commercial license is perpetual and allows binary redistribution: https://github.com/rednaga/APKiD/blob/master/LICENSE.COMMERCIAL |
| **apksigtool** | MIT | APK Signature Scheme v2/v3/v4 verification | androguard handles v1 well; apksigtool provides more thorough v2+ support |
| **quark-engine** | GPL-3.0 | Behavioral analysis scoring | Similar to CAPA but for Android. Stretch goal |

---

## 6. Data Model Changes

### 6.1 New Dataclasses

Add to `redb/models/dataclasses.py`:
- `APKFeatures`
- `APKManifestComponent`
- `APKPermission`
- `APKCertificate`
- `APKCodeSigningInfo`
- `APKDexFile`
- `APKDexApiUsage`
- `APKResource`
- `APKNativeLib`
- `APKInconsistencyTests`

See Tech Annex for full field definitions.

### 6.2 Existing Dataclass Changes

**`Hashes` dataclass** — add:
```python
permhash: Optional[str] = None  # APK: SHA-256 of sorted permissions (Mandiant/Google)
```

### 6.3 Tag Enum Additions

Add to `redb/extractors/enum.py`:
```python
# APK
APK_FEATURES = "apk_features"
APK_MANIFEST = "apk_manifest"
APK_PERMISSIONS = "apk_permissions"
APK_SIGNATURE = "apk_signature"
APK_DEX = "apk_dex"
APK_RESOURCES = "apk_resources"
APK_NATIVE_LIBS = "apk_native_libs"
APK_INCONSISTENCY_TESTS = "apk_inconsistency_tests"
```

---

## 7. Implementation Phases

### Phase 1 — Core Extractors

1. `APKExtractor` base class + `APKFeaturesExtractor`
2. `APKManifestExtractor`
3. `APKPermissionsExtractor` (including permhash)
4. `APKSignatureExtractor`
5. `APKDexExtractor`
6. `APKResourceExtractor`
7. `APKNativeLibExtractor`
8. Ingestor integration (wire up dispatch in `process_binary_file()`)
9. Hashes dataclass update (add `permhash`)

### Phase 2 — Detection and Anomalies

10. `APKInconsistencyTestsExtractor`
11. Wire up existing format-agnostic extractors for APK (YARA, Strings, IOC)

### Phase 3 — Future (out of scope for this PDD)

- APKiD integration for packer/protector detection
- Deep native library analysis (run ELF pipeline on extracted `.so` files)
- Recursive APK ingestion
- androguard-yara module integration for YARA rules matching on APK metadata

---

## 8. Testing Strategy

- Unit tests per extractor using known APK samples (benign + malicious)
- Validate output against VirusTotal reports for the same samples (cross-reference fields)
- Edge cases: split APKs, obfuscated APKs, packed APKs, APKs with no native code, APKs with no DEX, corrupted APKs
- Integration test: full pipeline run (ingest APK, verify all extractors produce output, verify ClickHouse/ES export)