Mamoru Sawahashi

# REDB Test Suite

## Running Tests

```bash
# All unit tests (no external dependencies, fast)
pytest tests/unit/                    # 784 tests, ~4s

# Integration tests (requires pefile, elftools, magic, etc.)
pytest tests/integration/             # ~209 tests

# By binary format
pytest -m pe                          # PE-specific tests
pytest -m elf                         # ELF-specific tests
pytest -m macho                       # Mach-O-specific tests
pytest -m apk                         # APK-specific tests

# By category
pytest -m dataclass                   # Dataclass construction tests
pytest -m exporters                   # Database exporter tests

# Decompiler / Binary Ninja tests only (mocked, no license needed)
pytest tests/unit/test_decompile_*.py # 227 tests, ~1s
pytest -k decompile                   # Same, by keyword

# Individual decompiler module
pytest tests/unit/test_decompile_utils.py        # Hashes, JSON encoder, normalization
pytest tests/unit/test_decompile_similarity.py   # MinHash
pytest tests/unit/test_decompile_ioc_extractor.py # IOC scraping & extraction
pytest tests/unit/test_decompile_analysis.py     # CFG, disassembly, LLIL analysis
pytest tests/unit/test_decompile_binja_extractor.py # DecompileBinja + ClickHouse schemas
pytest tests/unit/test_decompile_decompiler.py   # BinaryNinjaDecompiler orchestration
pytest tests/unit/test_decompile_arch.py         # x86 architecture, opcode categories
pytest tests/unit/test_decompile_scores.py       # Obfuscation scores
pytest tests/unit/test_decompile_strings.py      # String analysis
pytest tests/unit/test_decompile_function_type.py # Function type classification

# Combinations
pytest -m "pe and unit"               # PE unit tests only
pytest -m "elf and integration"       # ELF integration tests
pytest -m "unit and not exporters"    # Unit tests excluding exporters

# Run everything
pytest                                # All tests (requires full deps)

# Verbose with short traceback
pytest -v --tb=short

# Stop on first failure
pytest -x
```

---

## Directory Structure

```
tests/
├── conftest.py                        # Shared fixtures for all tests
├── TEST_INDEX.md                      # This file
├── DECOMPILE_TEST_PLAN.md             # Detailed test plan for decompiler modules
├── __init__.py
│
├── unit/                              # Tests with NO external dependencies
│   ├── __init__.py
│   ├── conftest.py                    # Stubs for heavy imports (magic, pefile, etc.)
│   ├── conftest_binja_stubs.py        # Binary Ninja stubs (enums, mock factories)
│   ├── test_database_exporters.py     # 28 tests — ClickHouse, Elasticsearch, Print
│   ├── test_dataclasses.py            # 46 tests — All data model dataclasses
│   ├── test_macho_dataclasses.py      # 20 tests — Mach-O dataclasses
│   ├── test_apk_dataclasses.py       # 24 tests — APK dataclasses + Tag enum
│   ├── test_apk_extractors.py        # 61 tests — APK extractors, base class, worker dispatch
│   ├── test_apk_code_analyzer.py     # 48 tests — APKCodeAnalyzer, LibraryFilter, SmaliParser
│   ├── test_apk_jadx_wrapper.py      # 13 tests — JADX subprocess, output parsing, timeouts
│   ├── test_apk_apktool_wrapper.py   # 12 tests — Apktool subprocess, smali dir detection
│   ├── test_apk_method_extractor.py  # 32 tests — Hashing, obfuscation, type conversion
│   ├── test_apk_process_method.py   # 29 tests — _process_method androguard integration, xrefs
│   ├── test_apk_strings_ioc.py      # 30 tests — APK string extraction, IOC integration
│   ├── test_apk_decompile_extractor.py # 40 tests — DecompileAPK extractor, export, schema
│   ├── test_decompile_utils.py        # 20 tests — Hashes, JSON encoder, normalization
│   ├── test_decompile_similarity.py   # 27 tests — MinHashCustom, MinHasher
│   ├── test_decompile_ioc_extractor.py # 34 tests — IOCScraper, IOCExtractorFromResults
│   ├── test_decompile_analysis.py     # 44 tests — CFG, disassembly, LLIL analysis
│   ├── test_decompile_binja_extractor.py # 36 tests — DecompileBinja + export schemas
│   ├── test_decompile_decompiler.py   # 21 tests — BinaryNinjaDecompiler orchestration
│   ├── test_decompile_arch.py         # 18 tests — ArchitectureCreator, Arch_x86
│   ├── test_decompile_scores.py       # 11 tests — ObfuscationScores
│   ├── test_decompile_strings.py      #  9 tests — StringAnalysis
│   ├── test_decompile_function_type.py #  7 tests — FunctionType enum + analysis
│   └── test_force_reprocessing.py     # 77 tests — --force flag, dedup bypass, YARA dedup
│
├── integration/                       # Tests requiring real binaries & libs
│   ├── __init__.py
│   ├── test_base_extractor.py
│   ├── test_basicproperties.py
│   ├── test_elf_extractor.py
│   ├── test_elf_extractors.py
│   ├── test_hashes.py
│   ├── test_pe_extractor.py
│   └── test_pe_extractors.py
│
└── scripts/                           # Manual CLI tools (not pytest)
    ├── test_linking.py
    └── test_macho_extractors_local.py
```

---

## Unit Tests (`tests/unit/`)

Unit tests run without any external dependencies. The `tests/unit/conftest.py`
stubs out heavy imports (`elasticsearch`, `clickhouse_connect`, `magic`, `pefile`,
etc.) so these tests execute in any environment.

### `test_database_exporters.py`
**Markers:** `unit`, `exporters` | **28 tests**

Tests for the database exporter classes in `redb/extractors/database_exporters.py`:

- **PrintExporter**: Initialization, exporting dataclasses/dicts/lists, handling None/empty data
- **ElasticsearchExporter**: Initialization, export with mocked client, error handling, custom IDs
- **ClickHouseExporter**: Initialization, tuple/multi-table exports, null/array handling, connection context manager
- **DatabaseExporter ABC**: Verifies abstract base class cannot be instantiated directly
- **Edge cases**: Nested data, datetime serialization, type conversions

### `test_dataclasses.py`
**Markers:** `unit`, `dataclass` | **46 tests**

Comprehensive tests for all dataclasses in `redb/models/dataclasses.py` (excluding Mach-O):

- **Hash / Hashes**: Required fields, optional PE/ELF/Mach-O hash fields, defaults
- **BasicProperties**: Required fields, FAT Mach-O fields (`is_fat`, `child_sha256`, etc.)
- **DIEinfo / CAPA**: Creation with default and populated fields
- **String**: String metadata with hash associations
- **PE dataclasses**: PE, PEImport, PESection, PEResource, PEOverlay, PESigner, PECertificate, PECodeSigningInfo, PEExtraFinding, PEDotNet, PEInconsistencyTests, DotNetInconsistencyTests
- **ELF dataclasses**: ELFFeatures, ELFDependency, ELFImport, ELFExport, ELFSection, ELFSegment, ELFSymbol, ELFRelocation, ELFNote
- **YARA dataclasses**: YaraRule, YaraMatch
- **Decompiled**: Decompiled function metadata

### `test_macho_dataclasses.py`
**Markers:** `unit`, `dataclass`, `macho` | **20 tests**

Tests for Mach-O dataclasses in `redb/models/dataclasses.py`:

- **MachO**: Required/optional fields, architecture lists, flags
- **MachOSegment**: Segment properties, entropy, hashes
- **MachOImport / MachOExport**: Import/export totals and lists
- **MachODylib**: Dynamic library dependencies
- **MachOLoadCommand**: Load command types and sizes
- **MachOUniversal**: FAT binary detection, architecture info
- **MachOCertificate**: Code signing certificate fields
- **MachOCodeDirectory**: Bundle ID, team ID, CD hash
- **MachOCodeSigningInfo**: Signing type (certificate/ad-hoc/unsigned), entitlements
- **MachOEntitlement**: Boolean/string/array entitlement values

### `test_apk_dataclasses.py`
**Markers:** `unit`, `dataclass`, `apk` | **24 tests**

Tests for APK dataclasses in `redb/models/dataclasses.py`:

- **APKFeatures**: Required fields (package_name, app_name, version_code, version_name), optional field defaults (SDK versions, debuggable, backup, ABIs, DEX counts), fully populated creation, asdict roundtrip
- **APKManifestComponent**: Creation with intent filters, default empty lists
- **APKManifest**: Required counts, optional list defaults, fully populated with XML
- **APKPermission / APKPermissions**: Permission creation, custom permissions, dangerous classification, permission details with permhash
- **APKCertificate / APKCodeSigningInfo**: Certificate fields, optional algorithm/key_size, signing info with scheme versions and certificate list
- **APKDexFile**: Required fields, optional TLSH/API usage/obfuscation indicators, TLSH populated
- **APKResource**: Creation, suspicious flag
- **APKNativeLib**: Creation, known packer detection
- **APKInconsistencyTests**: All 10 test fields default to None, fully populated, field count validation (exactly 10)
- **Hashes.permhash**: Default None, populated 64-char hash
- **Tag enum**: All 8 APK tags exist with correct values

### `test_apk_extractors.py`
**Markers:** `unit`, `apk` | **62 tests**

Tests for all 8 APK extractors, base class, module registry, and worker dispatch:

- **APKFeaturesExtractor**: tag(), get_clickhouse_table(), extract() returns APKFeatures dataclass, detects native code/assets, handles missing version_code, prepare_export_data() for ClickHouse (tuple format with column names/types), Elasticsearch (dataclass), and None when no extract
- **APKManifestExtractor**: tag(), table, extract() returns APKManifest with component counts
- **APKPermissionsExtractor**: tag(), table, extract() returns APKPermissions, classifies dangerous permissions, handles no-permission APKs
- **APKSignatureExtractor**: tag(), table, detects v1/v2/v3 signature schemes, handles unsigned APKs, certificate subject/issuer uses RFC 4514 short names (CN=, O=) not long names
- **APKDexExtractor**: tag(), table, extract() returns list of APKDexFile with mocked androguard DEX parser, TLSH computation for sufficient data, TLSH None for small DEX (<50 bytes), ClickHouse export includes dex_tlsh column, returns None for no DEX
- **APKResourceExtractor**: tag(), table, extract() returns dict with resources list
- **APKNativeLibExtractor**: tag(), table, finds native libs in ZIP, detects known packer libs (libjiagu.so), returns None for no libs
- **APKInconsistencyTestsExtractor**: tag(), table, extract() returns APKInconsistencyTests dataclass
- **APKModuleRegistry**: All 8 APK modules registered in workers.py get_module_by_name(), correct class mappings
- **APKWorkerDispatch**: APK excluded from DIE/BasicProperties/Hash general guards (source inspection), APK dispatch branch exists with all 8 extractors, APK branch runs BasicPropertiesExtractor and HashExtractor
- **APKExtractorBase**: binary property returns file content, _is_valid_apk() with/without APK object, _list_files(), _get_zip_file(), SHA256 matches file content

### `test_apk_code_analyzer.py`
**Markers:** `unit`, `apk`, `decompile` | **48 tests**

Tests for APK code analysis core components:

- **LibraryFilter**: Default prefixes loaded, user classes not filtered, Dalvik descriptor format, custom prefixes, env var override, filter stats, class name normalization, Google/Kotlin/third-party libs filtered, empty class name
- **SmaliParser**: Parse smali content, method names/class names/signatures extracted, instruction/register counts, abstract/native skipped, normalization (strips .line directives, comments), count_instructions, make_method_key, parse_smali_file, parse_smali_directory, empty/class-only files
- **MethodExtractor**: SHA-256 determinism/length/different inputs, ssdeep/TLSH for short/long input, Dalvik-to-Java class/type conversion, Java prototype generation, obfuscation detection (short names, string encryption, reflection, excessive goto), Dalvik param parsing
- **APKCodeAnalyzer**: Init defaults/custom timeout/env var, extract returns correct dict keys, androguard failure handling, cleanup removes temp dirs, Java normalization

### `test_apk_jadx_wrapper.py`
**Markers:** `unit`, `apk`, `decompile` | **13 tests**

Tests for JADX subprocess wrapper:

- **Init**: Default/custom path, env var override
- **Decompile**: Success, non-zero exit, timeout handling (SIGTERM/SIGKILL), not found, command args verification
- **Output parsing**: Empty directory, Java file parsing, method extraction, method declaration detection, control flow exclusion, static methods

### `test_apk_apktool_wrapper.py`
**Markers:** `unit`, `apk`, `decompile` | **12 tests**

Tests for apktool subprocess wrapper:

- **Init**: Default/custom path, env var override
- **Disassemble**: Success, failure, timeout, not found, command args
- **Smali directories**: Single DEX, multi-DEX (smali_classes2, smali_classes3), empty dir, nonexistent path

### `test_apk_method_extractor.py`
**Markers:** `unit`, `apk`, `decompile` | **32 tests**

Tests for method-level hashing, obfuscation detection, and type conversion:

- **Hashing**: SHA-256 determinism/length/uniqueness, ssdeep/TLSH null for short input
- **Obfuscation**: Short method/class names, string encryption detection, reflection calls, excessive goto (including goto/16)
- **Type conversion**: All Dalvik primitives, object types, arrays, empty
- **Prototypes**: Simple methods, multi-param, object return, empty signature
- **Param parsing**: Empty, single/multiple primitives, objects, mixed, arrays

### `test_apk_process_method.py`
**Markers:** `unit`, `apk`, `decompile` | **29 tests**

Mock-based tests for `APKCodeAnalyzer._process_method` and androguard integration:

- **XrefExtraction**: androguard 4.x xref format (MethodAnalysis wrapper unwrap), 3.x fallback (direct EncodedMethod), empty xrefs, xref exception resilience, multiple callers/callees
- **MethodFiltering**: External method skipping, library method filtering, min instruction threshold (below/at threshold)
- **SmaliLookup**: Apktool smali used when available, androguard fallback disassembly, no-code skip, bytecode exception handling
- **OutputPopulation**: All 6 result dict keys populated (smali_content, smali_refs, decompiled_content, decompiled_refs, similarity_metrics with block_features), content deduplication (smali and decompiled)
- **DisassembleWithAndroguard**: No code returns None, no bytecode returns None, basic disassembly, invoke operand normalization (brace wrapping), bytecode exception handling

### `test_apk_strings_ioc.py`
**Markers:** `unit`, `apk` | **30 tests**

Tests for APK string extraction and IOC extraction integration:

- **StringEntropy**: Empty/single/repeated chars, two-char entropy = 1.0, higher variety = higher entropy, URL string entropy
- **ExtractStrings**: Basic extraction, field schema (string, encoding, offset, length, entropy), deduplication across DEX files, empty/null string filtering, no DEX files, DEX returning None, DEX exception handling, entropy computed, incremental offsets, URL preservation
- **ExtractStringsIntegration**: Strings included in extract() output, string extraction error is non-fatal (recorded in analysis_errors)
- **DecompileAPKStringsExport**: Strings table present in export, schema validation (7 columns), SHA256 propagation, empty strings not exported
- **IOCExtractorAPKFormat**: IOCs from APK strings, IOCs from decompiled_content (APK key), LIBRARY methods skipped, combined Binja+APK sources, APK-only results (no 'decompiled' key), source_type correctness (STRING vs DECOMPILED_FUNCTION)
- **WorkerAPKIOCWiring**: Verify workers.py has IOCExtractorFromResults in APK branch

### `test_apk_decompile_extractor.py`
**Markers:** `unit`, `apk`, `decompile` | **40 tests**

Tests for DecompileAPK extractor and ClickHouse export schemas:

- **DecompileAPK**: tag(), get_clickhouse_table(), timeout defaults/env/invalid, calculate_md5(), context manager, cleanup_run()
- **Export schemas**: Returns None when no results, multi_table flag, all 6 table keys present, per-table schema validation (column names, types, data row length), SHA-256 propagation to refs, empty tables excluded, non-ClickHouse returns None
- **Dataclasses**: APKDecompiledMethodContent, APKDecompiledMethodReference, APKSmaliMethodContent, APKSmaliMethodReference, APKMethodSimilarityMetrics, APKCodeAnalysisError — defaults, populated, asdict roundtrip
- **Tag enum**: APK_DECOMPILED exists with correct value
- **Worker integration**: DecompileAPK registered in modules, dispatch source includes DecompileAPK

---

## Decompiler Unit Tests (`tests/unit/test_decompile_*.py`)

These 227 tests cover the `redb/extractors/decompiler/` subtree and the
`redb/extractors/ioc_extractor/` module. Binary Ninja is **not required** --
all BN dependencies are mocked via `conftest_binja_stubs.py`.

**Source modules tested:** `bninja/utils/`, `bninja/similarity/`, `bninja/arch/`,
`bninja/analysis/`, `bninja/decompiler.py`, `bninja/function_type.py`,
`DecompileBinja.py`, `ioc_extractor/`.

### Binary Ninja Stubs (`conftest_binja_stubs.py`)

Not a test file -- provides the mocking infrastructure used by all `test_decompile_*.py` files.

- **Stub enums**: `BranchType`, `InstructionTextTokenType`, `SymbolType`, `HighLevelILOperation`, `LowLevelILOperation` as real `IntEnum` classes matching Binary Ninja's values
- **Mock factories**: `MockInstructionToken`, `MockBasicBlock`, `MockFunction`, `MockBinaryView`, `MockILInstruction`, `MockLLILFunction`, `MockHLILInstruction`, `MockEdge`, `MockSymbol`
- **Module stubs**: Injects mock `binaryninja`, `binaryninja.enums`, `binaryninja.lowlevelil`, `binaryninja.highlevelil`, `binaryninja.mediumlevelil` into `sys.modules`

### `test_decompile_utils.py`
**20 tests** | Source: `bninja/utils/hashes.py`, `bninja/utils/json_encoder.py`, `bninja/analysis/low_level_normalization.py`

- **TestCalculateMD5**: Known values, empty string
- **TestCalculateSHA256**: Known values, empty string, 64-char hex output
- **TestCalculateTLSH**: Long data (>=50 bytes), short data returns None, determinism
- **TestCalculateSSDeep**: Valid data, short data returns None, determinism
- **TestBinaryNinjaEncoder**: Objects with `.value`/`.confidence`, `__str__` fallback, standard types, unsupported types raise TypeError
- **TestLowLevelNormalization**: Single instruction, nested operands, empty operands, list operands, None input, deep nesting

### `test_decompile_similarity.py`
**27 tests** | Source: `bninja/similarity/minhashcustom.py`, `bninja/similarity/minhasher.py`

- **TestMinHashCustom**: Default construction, init from signature/bytes, both-raises ValueError, set/get roundtrip, truncation to `2^MINHASH_BITS`, `hashData()` for string/bytes/list/unsupported types, identical/different scores, 8-bit vs 32-bit modes, `scoreAgainst()` instance method
- **TestMinHasher**: n-gram generation (basic, short input, exact-n), minhash calculation (empty function, determinism, output length, value range), shingle hash determinism, seed reproducibility

### `test_decompile_ioc_extractor.py`
**34 tests** | Source: `ioc_extractor/standalone_ioc_extractor.py`, `ioc_extractor/ioc_extractor.py`

- **TestIOCScraper**: IPv4 (excluded private/reserved IPs, valid public), IPv6, URLs, defanged URLs (`hxxps://`, `[.]`), emails, FQDNs (valid + excluded domains), `.onion` addresses, MD5/SHA1/SHA256 hashes, hash deduplication (SHA256 not also emitted as MD5/SHA1 substring), invalid hashes (all zeros/low entropy), CVE/CWE identifiers, BTC/ETH crypto addresses, Linux/Windows file paths, server `ip:port` patterns, deduplication within text, empty text
- **TestIOCExtractorFromResults**: Extract from strings, extract from decompiled functions (Binja + APK formats), skip LIBRARY/THUNK functions, handle bytes-encoded strings, empty results, ClickHouse export format, print export format, no-IOCs returns None, `get_clickhouse_table()`, `tag()`

### `test_decompile_analysis.py`
**44 tests** | Source: `bninja/analysis/cfg.py`, `bninja/analysis/disassembly.py`, `bninja/analysis/low_level.py`

- **TestCFGAnalysis**: Cyclomatic complexity (linear, branch, loop), CFG structure extraction, block JSON schema (14 fields), branch types (RETURN, CONDITIONAL, DIRECT, CALL, FALLTHROUGH, UNKNOWN), block types (CODE, THUNK, DATA), BFS depth map, sequential block IDs, block flags (EntryBlock, ExitBlock, LoopBlock), dominators, post-dominators
- **TestDisassemblyAnalysis**: `get_json()` structure (12+ keys), hash determinism, instruction type collection, memory patterns (MEM_STACK, MEM_DIRECT, MEM_SCALED_INDEX, MEM_BASE_OFFSET), register usage (GPR, SIMD, FPU), data reference counting, max block size, CALL counting, stack size estimation, opcode normalization
- **TestLowLevelAnalysis**: Return type `(dict, list)`, JSON schema (15 keys), control flow counting, memory patterns, register usage reads/writes, data references, CALL+TAILCALL counting, instruction type collection, normalized instruction lists, negative offset clamping

### `test_decompile_binja_extractor.py`
**36 tests** | Source: `DecompileBinja.py`

- **TestDecompileBinja**: `calculate_md5()`, timeout defaults (BINJA=1200, DECOMPILE=2580), timeout from env vars, invalid env fallback, `.NET` detection (non-PE, with/without COM descriptor), Go detection (ELF with `.gopclntab`, ELF without, PE with Go strings, non-PE/ELF), `tag()` returns `"decompiled"`, context manager protocol, `cleanup_run()` state clearing, goresym file cleanup, thread timeout handling, hash propagation on successful extract, .NET binary skipping, goresym invocation for Go
- **TestPrepareExportData (ClickHouse schemas)**: Returns None when no results, `multi_table: True` flag, all 10 table keys present, schema validation for each table (decompiled_content, decompiled_refs, disassembled_content, disassembled_refs, llil_content, llil_refs, cfg_blocks_content, cfg_blocks_refs, function_similarity_metrics, strings_raw, function_analysis_errors), golang_metadata presence/absence, array field None-to-empty, list pass-through, register usage map conversion

### `test_decompile_decompiler.py`
**21 tests** | Source: `bninja/decompiler.py`

- **TestBinaryNinjaDecompiler**: `log_error()` appending and schema (7 keys), `is_too_few_blocks()` for None/empty/small-single/large-single/multiple blocks, `is_lib_or_thunk()` for USER/THUNK/EXTERNAL functions, `analyze_binary()` cross-linkage (HLIL-disasm, LLIL-disasm, CFG-disasm bidirectional hashes), cyclomatic complexity propagation, fuzzy hash calculation (ssdeep/tlsh), `extract()` success/failure, `tag()`, GoReSym application (user functions, std functions, invalid JSON handling)

### `test_decompile_arch.py`
**18 tests** | Source: `bninja/arch/creator.py`, `bninja/arch/x86.py`

- **TestArchitectureCreator**: Create x86, case-insensitive lookup, unsupported architecture raises ValueError
- **TestArchX86**: Register classification (`is_register`, `is_general_purpose_register`, `is_stack_register`, `is_xmm_register`, `is_simd_register`), opcode categories populated and correct (MOV->DATA_MOVEMENT, CALL->CONTROL_FLOW, ADD->ARITHMETIC, JNZ->CONDITIONAL_JUMP), `is_control_flow_instruction_by_mnemonic()` for JMP/CALL/RET/LOOP/MOV/empty

### `test_decompile_scores.py`
**11 tests** | Source: `bninja/analysis/scores.py`

- **TestObfuscationScores**: `flattened_score()` with no back edges (0.0), with loop (>0.0), fully flattened CFG (~1.0); `MBA_score()` with no mixed ops (0.0), some mixed (>0.0), all mixed (1.0); `get_dominated_by()` BFS; `uses_mba()` for arithmetic-only, logic-only, mixed

### `test_decompile_strings.py`
**9 tests** | Source: `bninja/analysis/strings.py`

- **TestStringAnalysis**: Entropy edge cases (empty string->0.0, single char->0.0, uniform distribution->2.0, binary->1.0), deduplication by (string, encoding), sorted by lowest offset, empty binary view, output schema validation (7 required fields)

### `test_decompile_function_type.py`
**7 tests** | Source: `bninja/function_type.py`

- **TestFunctionType**: Enum values (USER, LIBRARY, EXTERNAL, THUNK, UNKNOWN)
- **TestFunctionTypeAnalysis**: Regular function->USER, ImportedFunctionSymbol->EXTERNAL, LibraryFunctionSymbol->LIBRARY, `is_thunk=True` overrides to THUNK, exception in symbol access->UNKNOWN

### `test_force_reprocessing.py`
**77 tests** | Source: `start.py`, `redb/ingestor.py`, `redb/queries.py`

- **TestForceCliArgument**: `--force` flag default/explicit, `--modules` does not implicitly enable force, explicit `--force` required with specific modules
- **TestPerFileDedupBypass**: Default calls `is_in_db` and skips, `force=True` skips `is_in_db` check, decompile mode skips `is_in_db` (existing behavior)
- **TestBulkDedupBypass**: `fetch_s3_objects_by_repository` and `fetch_s3_objects_by_date_range` call/skip `is_in_db_bulk` based on force flag, force preserves all rows
- **TestIngestorForceParam**: Ingestor defaults `force=False`, stores `force=True` when passed
- **TestForceThreading**: `_process_file_internal`, `process_file`, `process_s3_file`, `process_zip_file`, `process_7zip_file` correctly thread `force` parameter through to `process_binary_file`
- **TestAnalyzedCliArgument**: `--analyzed` flag default/explicit, mutual exclusivity with `--path`/`--s3`, combinability with `--decompile`/`--modules`/`--magika`/`--force`, no `--repo` required
- **TestFetchAnalyzedSamples**: Queries `basic_properties` table, magika filter, decompile mode filters already-decompiled, force skips decompile check, analysis mode no filtering, missing S3_BUCKET, empty results, bytes SHA256 handling, exception handling, yara_scan bypass, CLICKHOUSE_CODE_PREFIX env var
- **TestIngestorAnalyzedParam**: Ingestor defaults `analyzed=False`, stores `analyzed=True`, combined with `decompile`
- **TestAnalyzedReExport**: `fetch_analyzed_samples` importable from `redb.ingestor`
- **TestYaraForceDedup**: `is_in_db_bulk` YARA path queries `yara_matches` with `unhex()`, handles empty table/bytes response/DB errors, `fetch_s3_objects_by_repository` and `fetch_analyzed_samples` respect `--force` for YARA dedup, `_check_yara_matches_bulk` batching

---

## Integration Tests (`tests/integration/`)

Integration tests require external dependencies (`pefile`, `elftools`, `magic`,
`signify`, `ppdeep`, `tlsh`, etc.) and use real test binaries from `test_files/`.

### `test_base_extractor.py`
**Markers:** `integration`

Tests for the base `Extractor` class in `redb/extractors/extractor.py`:

- **Static methods**: `process_binary_string()`, `remove_non_utf8()`
- **Entropy calculation**: High/low entropy data, edge cases
- **Initialization**: With/without exporters, logger setup
- **Export pipeline**: `export_data()` with PrintExporter

### `test_basicproperties.py`
**Markers:** `integration`

Tests for `BasicPropertiesExtractor` in `redb/extractors/basicproperties.py`:

- **Initialization**: With ELF/PE binaries, parent_sha256 parameter
- **Extraction**: Filename, filesize, filetype detection (magic/magika), entropy
- **FAT Mach-O support**: `is_fat`, `child_sha256`, `child_architecture`, `child_filetype`
- **ClickHouse export**: Column names and types for database schema
- **Dataclass validation**: BasicProperties field structure

### `test_hashes.py`
**Markers:** `integration`

Tests for `HashExtractor` in `redb/extractors/hashes.py`:

- **Initialization**: With ELF/PE/Mach-O objects
- **Hash extraction**: MD5, SHA1, SHA256, ssdeep, TLSH
- **PE-specific hashes**: Authentihash, imphash, richhash
- **Mach-O hashes**: dylib_hash, import_hash, export_hash, entitlement_hash, symhash
- **ClickHouse export**: Hashes dataclass columns
- **Consistency**: Same file produces same hashes, lowercase hex, correct lengths

### `test_pe_extractor.py`
**Markers:** `integration`, `pe`

Tests for base `PEExtractor` class in `redb/extractors/pe_extractor.py`:

- **Initialization**: With file path or provided pefile object
- **PE parsing**: Headers, architecture detection, .NET detection
- **Compilation timestamp**: Extraction and UTC conversion
- **Helper methods**: Section/import/export counts

### `test_pe_extractors.py`
**Markers:** `integration`, `pe`

Tests for specialized PE extractors:

- **PEFeaturesExtractor**: Headers, .NET detection, counts, rich header, version info
- **PEImportExtractor**: Import libraries and functions, ClickHouse format
- **PESectionExtractor**: Section properties, entropy ranges, hashes
- **PEResourceExtractor**: Resource types, languages, entropy
- **PEOverlayExtractor**: Overlay detection, size, hashes
- **PESignatureExtractor**: Signature presence, certificate chain
- **PEDotNetExtractor**: .NET metadata, streams
- **PEInconsistencyTestsExtractor**: Rich header anomalies, anti-analysis checks
- **PEExtraFindings**: Additional findings and anomalies

### `test_elf_extractor.py`
**Markers:** `integration`, `elf`

Tests for base `ELFExtractor` class in `redb/extractors/elf_extractor.py`:

- **Initialization**: With file path, binary content validation
- **ELF parsing**: Header fields, architecture, endianness
- **Security features**: PIE, NX, RELRO, stack canary detection
- **Helper methods**: Section/segment iteration

### `test_elf_extractors.py`
**Markers:** `integration`, `elf`

Tests for specialized ELF extractors:

- **ELFFeaturesExtractor**: Header fields, security properties, counts
- **ELFSectionExtractor**: Section names, types, flags, entropy
- **ELFSegmentExtractor**: Segment types, permissions, sizes
- **ELFSymbolExtractor**: Symbol names, types, bindings, visibility
- **ELFImportExtractor**: Imported libraries and functions
- **ELFExportExtractor**: Exported symbols
- **ELFDependencyExtractor**: DT_NEEDED, SONAME, RPATH, RUNPATH
- **ELFRelocationExtractor**: Relocation types, symbols, addends
- **ELFNoteExtractor**: Build ID, ABI tags, GNU notes

---

## Manual Scripts (`tests/scripts/`)

These are **not** part of the pytest suite. They're standalone CLI tools for
manual testing and debugging, run directly with `python <script>.py`.

### `test_linking.py`

Manual test script for verifying function linking between decompiled and
disassembled code in Binary Ninja integration. Tests that `DecompileBinja`
correctly cross-references decompiled functions with their disassembled
counterparts.

### `test_macho_extractors_local.py`

Manual test script for Mach-O extractors without database connections.
Used for local development and debugging of Mach-O feature extraction
with mocked settings.

---

## Test Fixtures (conftest.py)

### Root `tests/conftest.py`

Shared fixtures available to all tests:

- `mock_logger` / `real_logger`: Logger instances for testing
- `elf_binary_path` / `pe_binary_path`: Paths to test binaries
- `elf_binary_content` / `pe_binary_content`: Binary file contents
- `temp_binary_file` / `temp_text_file`: Temporary file factories
- `mock_print_exporter`: Mocked PrintExporter
- `known_hashes` / `pe_known_hashes`: Expected hash values for test files
- `pe_object` / `elf_object`: Parsed pefile/ELF objects
- `sample_hash_dataclass` / `sample_hashes_dataclass`: Pre-populated dataclasses
- `entropy_calculator`: Entropy computation helper
- Skip markers: `requires_die`, `requires_capa`, `requires_floss`

### Unit `tests/unit/conftest.py`

Stubs heavy dependencies so unit tests run without external packages:

- Stubs `elasticsearch`, `clickhouse_connect`, `dotenv` in `sys.modules`
- Replaces `redb.settings` with MagicMock
- Replaces `redb.extractors` package to prevent eager imports

### Binary Ninja Stubs `tests/unit/conftest_binja_stubs.py`

Provides mock infrastructure for all decompiler tests:

- Stub `IntEnum` classes matching Binary Ninja's enum values
- Mock factory classes for BN objects (functions, blocks, IL instructions, binary views)
- Injected into `sys.modules` before any `binaryninja` import

---

## Markers Reference

| Marker | Description |
|--------|-------------|
| `unit` | Pure unit tests - no external deps, no disk I/O, fast (<1s each) |
| `integration` | Integration tests - use real test binaries on disk |
| `slow` | Tests that take >5s (entropy on large files, full pipelines) |
| `pe` | Tests specific to PE binary analysis |
| `elf` | Tests specific to ELF binary analysis |
| `macho` | Tests specific to Mach-O binary analysis |
| `apk` | Tests specific to APK binary analysis |
| `exporters` | Tests for database exporters (ClickHouse, Elasticsearch, Print) |
| `dataclass` | Tests for dataclass construction and field validation |
| `binja` | Tests for Binary Ninja decompiler extractors (mocked, no license needed) |

---

## Test Count Summary

| File | Tests | Source Modules |
|------|------:|----------------|
| `test_dataclasses.py` | 46 | `redb/models/dataclasses.py` |
| `test_decompile_analysis.py` | 44 | `bninja/analysis/cfg.py`, `disassembly.py`, `low_level.py` |
| `test_decompile_binja_extractor.py` | 36 | `DecompileBinja.py` |
| `test_decompile_ioc_extractor.py` | 34 | `ioc_extractor/` |
| `test_database_exporters.py` | 28 | `database_exporters.py` |
| `test_decompile_similarity.py` | 27 | `bninja/similarity/` |
| `test_decompile_decompiler.py` | 21 | `bninja/decompiler.py` |
| `test_decompile_utils.py` | 20 | `bninja/utils/`, `bninja/analysis/low_level_normalization.py` |
| `test_macho_dataclasses.py` | 20 | `redb/models/dataclasses.py` (Mach-O) |
| `test_apk_dataclasses.py` | 24 | `redb/models/dataclasses.py` (APK), `redb/extractors/enum.py` |
| `test_apk_extractors.py` | 62 | `redb/extractors/apk_extractors/`, `redb/workers.py` |
| `test_apk_code_analyzer.py` | 48 | `redb/extractors/decompiler/apk/` (analyzer, filter, parser) |
| `test_apk_jadx_wrapper.py` | 13 | `redb/extractors/decompiler/apk/jadx_wrapper.py` |
| `test_apk_apktool_wrapper.py` | 12 | `redb/extractors/decompiler/apk/apktool_wrapper.py` |
| `test_apk_method_extractor.py` | 32 | `redb/extractors/decompiler/apk/method_extractor.py` |
| `test_apk_process_method.py` | 29 | `redb/extractors/decompiler/apk/analyzer.py` (_process_method) |
| `test_apk_strings_ioc.py` | 30 | `apk/analyzer.py` (strings), `DecompileAPK.py`, `ioc_extractor/` |
| `test_apk_decompile_extractor.py` | 40 | `DecompileAPK.py`, `redb/models/dataclasses.py` (APK code) |
| `test_decompile_arch.py` | 18 | `bninja/arch/` |
| `test_decompile_scores.py` | 11 | `bninja/analysis/scores.py` |
| `test_decompile_strings.py` | 9 | `bninja/analysis/strings.py` |
| `test_decompile_function_type.py` | 7 | `bninja/function_type.py` |
| `test_force_reprocessing.py` | 77 | `start.py`, `redb/ingestor.py`, `redb/queries.py` |
| **Unit total** | **784** | |

---

## Test Binaries

Located in `test_files/`:

| File | Type | Description |
|------|------|-------------|
| `hello` | ELF | Simple ELF executable |
| `hello_advanced` | ELF | ELF with more features (symbols, sections) |
| `d8637bdbcfc9112fcb1f0167b398e771` | PE | .NET PE executable |