Haiwen Cai

# redb
RationalEdge Samples DB

A malware analysis framework that extracts features from binary files (PE, ELF, Mach-O, APK) and stores them in ClickHouse for analysis.

## Quick Start

```bash
# Setup
source venv/bin/activate
pip install -r requirements.txt

# Process local files
python start.py --path /path/to/samples --repo test --index_prefix redb
```

## Usage Modes

### Local Mode
Process files from local filesystem:

```bash
# Single file or directory
python start.py --path /path/to/binary --repo test --index_prefix redb

# From a text file with paths (one per line)
python start.py --path /path/to/filelist.txt --repo test --index_prefix redb
```

### S3 Mode
Process samples from S3 storage based on catalog queries:

```bash
# By repository
python start.py --s3 --repo bazaar --index_prefix redb

# By repository with notes filter
python start.py --s3 --repo vx-itw --s3-notes "ITW.0138" --index_prefix redb

# By filetype (magika) - all ELF samples across all repos
python start.py --s3 --magika elf --index_prefix redb

# By filetype with repository filter
python start.py --s3 --repo bazaar --magika elf --index_prefix redb
```

### Date-Based Mode
Process samples by first_seen date from catalog:

```bash
# Single date (all samples first seen on Jan 15, 2025)
python start.py --date 2025-01-15 --index_prefix redb

# Date with repository filter
python start.py --date 2025-01-15 --repo bazaar --index_prefix redb

# Date range (inclusive)
python start.py --range 2025-01-01 2025-01-31 --index_prefix redb

# Date range with repository and notes filters
python start.py --range 2025-01-01 2025-01-31 --repo malshare --s3-notes "batch1" --index_prefix redb

# Date range with filetype filter
python start.py --range 2025-01-01 2025-01-31 --magika pebin --index_prefix redb
```

### S3-Solo Mode
Process a single sample by S3 key:

```bash
python start.py --s3-solo "09/f7/09f7d02a...hash.zip" --index_prefix redb
```

## Analysis Options

### Feature Extraction (default)
Runs all extractors to extract features from binaries:

```bash
python start.py --s3 --repo bazaar --index_prefix redb
```

### Specific Modules
Run only specific extractors:

```bash
python start.py --path /path/to/binary --repo test --index_prefix redb \
    --modules "BasicPropertiesExtractor,PEFeaturesExtractor,HashExtractor"
```

Available modules:
- **General**: `BasicPropertiesExtractor`, `HashExtractor`, `DIEExtractor`, `CAPAExtractor`
- **PE**: `PEFeaturesExtractor`, `PEImportExtractor`, `PEResourceExtractor`, `PEOverlayExtractor`, `PESectionExtractor`, `PESignatureExtractor`, `PEDotNetExtractor`, `PEInconstistencyTestsExtractor`, `PEExtraFindings`
- **ELF**: `ELFFeaturesExtractor`, `ELFSegmentExtractor`, `ELFSectionExtractor`, `ELFDependencyExtractor`, `ELFSymbolExtractor`, `ELFImportExtractor`, `ELFExportExtractor`, `ELFRelocationExtractor`, `ELFNotesExtractor`
- **Mach-O**: `MachOFeaturesExtractor`, `MachOSegmentExtractor`, `MachOImportExtractor`, `MachOExportExtractor`, `MachODylibExtractor`, `MachOSignatureExtractor`, `MachOSimilarityHashExtractor`
- **APK**: `APKFeaturesExtractor`, `APKManifestExtractor`, `APKPermissionsExtractor`, `APKSignatureExtractor`, `APKDexExtractor`, `APKResourceExtractor`, `APKNativeLibExtractor`, `APKInconsistencyTestsExtractor`
- **JavaScript**: `JSFeaturesExtractor`, `JSSuspiciousAPIsExtractor`, `JSStringsExtractor`, `JSDeobfuscationExtractor`, `JSContentExtractor`

**Note:** Using `--modules` with specific extractors respects the normal deduplication check. Add `--force` to reprocess samples already in the database.

### Analyzed Samples Mode
Process samples that are already in the database (from `basic_properties`). Useful for decompiling or re-running specific modules on previously analyzed samples:

```bash
# Decompile all already-analyzed samples that haven't been disassembled yet
python start.py --analyzed --index_prefix redb --decompile

# Decompile only ELF samples that were already analyzed
python start.py --analyzed --magika elf --index_prefix redb --decompile

# Re-run a specific extractor on already-analyzed samples
python start.py --analyzed --index_prefix redb --modules "MachOFeaturesExtractor"

# Force decompile ALL analyzed samples (even already-disassembled ones)
python start.py --analyzed --index_prefix redb --decompile --force

# Re-run a specific decompiler module on only already-disassembled samples
python start.py --analyzed --index_prefix redb --decompile --rerun --decompile-modules cfg
```

When combined with `--decompile`, the `--analyzed` flag has three behaviors:

| Flags | Source | Description |
|-------|--------|-------------|
| `--analyzed --decompile` | `basic_properties` minus `disassembled` | New samples only (first-time decompilation) |
| `--analyzed --decompile --force` | All of `basic_properties` | Re-run everything from scratch (e.g., new binja version) |
| `--analyzed --decompile --rerun` | Only `disassembled` table | Re-run on already-disassembled samples only (e.g., updated CFG module) |

The `--rerun` flag is particularly useful with `--decompile-modules` to selectively re-run a single module without reprocessing the full pipeline.

### Force Reprocessing
By default, samples already in the database are skipped. Use `--force` to reprocess them:

```bash
# Force full reprocessing of all samples
python start.py --s3 --repo bazaar --index_prefix redb --force

# Re-run a specific extractor on already-processed samples
python start.py --s3 --repo bazaar --index_prefix redb --modules "MachOFeaturesExtractor" --force

# Force YARA rescan (e.g., after updating rules)
python start.py --s3 --magika elf --index_prefix redb --yara --force
```

`--force` works across all modes: feature extraction, decompilation, and YARA scanning. ReplacingMergeTree handles deduplication, so reprocessed data cleanly replaces existing rows.

### Decompilation Mode
Run Binary Ninja decompilation only:

```bash
python start.py --s3 --repo bazaar --index_prefix redb --decompile
```

#### Selective Decompiler Modules
Run only specific decompiler sub-modules instead of the full pipeline:

```bash
# Run only strings extraction (fastest - skips per-function analysis)
python start.py --s3 --repo bazaar --index_prefix redb --decompile --decompile-modules strings

# Run disassembly and CFG analysis only
python start.py --s3 --repo bazaar --index_prefix redb --decompile --decompile-modules disassembly,cfg

# Run multiple modules
python start.py --s3 --repo bazaar --index_prefix redb --decompile --decompile-modules decompilation,disassembly,llil
```

Available decompiler modules:
- **decompilation** — High-level IL (HLIL) decompiled output → `code_binja_decompiled_functions_*` tables
- **disassembly** — Low-level assembly representation → `code_binja_disassembled_functions_*` tables
- **cfg** — Control flow graph analysis → `code_binja_cfg_functions` table
- **llil** — Low-level intermediate language → `code_binja_llil_functions_*` tables
- **strings** — Binary string extraction → `code_binja_strings_raw` table

**IOC extraction** runs automatically when `decompilation` or `strings` is selected (it consumes their in-memory results). It is skipped for modules like `cfg` or `disassembly` that don't produce IOC-relevant data.

Default is `all` (runs every module). Requires `-d/--decompile` flag.

### YARA Scanning
Run YARA rules against samples:

```bash
# YARA scanning only (skips already-scanned samples by default)
python start.py --s3 --magika elf --index_prefix redb --yara

# Force rescan all samples (e.g., after updating YARA rules)
python start.py --s3 --magika elf --index_prefix redb --yara --force

# Feature extraction + YARA scanning combined
python start.py --s3 --repo bazaar --index_prefix redb --with-yara
```

By default, `--yara` skips samples that already have matches in the `yara_matches` table. Use `--force` to rescan everything (e.g., after updating YARA rules).

### Dry Run Mode
Print results instead of uploading to database:

```bash
python start.py --path /path/to/binary --repo test --index_prefix redb --dry-run
```

## Environment Variables

See `.env.example` for all configuration options:

| Variable | Description |
|----------|-------------|
| `CLICKHOUSE_HOST` | ClickHouse server host |
| `CLICKHOUSE_PORT` | ClickHouse server port (default: 8123) |
| `CLICKHOUSE_USER` | ClickHouse username |
| `CLICKHOUSE_PASSWORD` | ClickHouse password |
| `S3_ENDPOINT` | S3/MinIO endpoint |
| `S3_ACCESS_KEY` | S3 access key |
| `S3_SECRET_KEY` | S3 secret key |
| `S3_BUCKET` | S3 bucket name |
| `INDEX_PREFIX` | Table prefix for ClickHouse (default: redb) |
| `SUPPORTED_FORMATS` | File formats to query (default: `['pebin']`) |
| `BATCH_SIZE` | Files per batch (default: 1000) |
| `REDB_TIMEOUT` | Worker timeout in seconds (default: 600) |
| `DECOMPILE_WORKER_TIMEOUT` | Decompile timeout (default: 2700) |

## Filtering Options Summary

| Option | Description | Standalone | With --repo | With --date/--range |
|--------|-------------|------------|-------------|---------------------|
| `--repo` | Filter by repository | Required for --s3 (unless --magika) | - | Optional |
| `--s3-notes` | Filter by notes field | No | Yes | Yes |
| `--magika` | Filter by filetype | Yes (queries all repos) | Yes | Yes |
| `--date` | Filter by single date | Yes | Yes | - |
| `--range` | Filter by date range | Yes | Yes | - |
| `--analyzed` | Process already-analyzed samples | Yes | N/A | N/A |