Ichiro Sakuma

# JavaScript Malware Analysis

REDB extracts features from JavaScript files using five dedicated extractors plus two shared extractors (IOCs and strings). Magika detects the file as `javascript`; the file must be listed in `SUPPORTED_FORMATS` in `.env` to be processed.

## Configuration

Add `javascript` to `SUPPORTED_FORMATS` in `.env`:

```
SUPPORTED_FORMATS=['pebin', 'elf', 'macho', 'apk', 'javascript']
```

| Variable | Default | Required | Description |
|----------|---------|----------|-------------|
| `SUPPORTED_FORMATS` | `['pebin']` | Yes | Must include `javascript` for JS files to be processed |
| `JS_DEOBFUSCATOR_PATH` | `webcrack` | No | Path or name of an external JS deobfuscator. If not installed, falls back to `jsbeautifier` (Python library, always available) |
| `JS_DEOBFUSCATE_TIMEOUT` | `60` | No | Timeout in seconds for the external deobfuscator subprocess |
| `JS_XRAY_RUNNER_PATH` | bundled `redb/extractors/js_extractors/scripts/js-xray-runner.js` | No | Node bridge that runs `@nodesecure/js-x-ray` and emits JSON. Falls back to heuristic-only when the bridge or its `node_modules` are missing |
| `JS_XRAY_NODE_BIN` | `node` | No | Node binary to invoke the bridge with |
| `JS_XRAY_TIMEOUT` | `30` | No | Timeout in seconds for the js-x-ray subprocess |

### Python dependencies

Installed via `requirements.txt`:
- `jsbeautifier` — code normalization and fallback deobfuscation
- `chardet` — source encoding detection
- `pyjsparser` — ES5.1 AST parser. The obfuscation heuristic's `avg_identifier_length<2` strong signal depends on AST identifier walking, so without pyjsparser the JS pipeline runs in a degraded "regex-only" mode that misses a key obfuscator.io tell. Listed as required, not optional.

### External Node tools

The Docker image bundles everything below; host CLI installs need to be done once.

- **webcrack** — reverses webpack bundling, obfuscator.io output, and common packing patterns. Significantly better than jsbeautifier for real-world obfuscated malware. Pinned to **2.16.0** in the `Dockerfile` and installed globally inside the container; on the host run `npm install -g webcrack@2.16.0` (or set `JS_DEOBFUSCATOR_PATH` to a non-default path).
- **@nodesecure/js-x-ray** — static AST analyser used by the NodeSecure project (and npm's package scanning) that recognises specific obfuscator families (`jsfuck`, `obfuscator.io`, `morse`, `jjencode`, `freejsobfuscator`, ...) and emits structured warnings. We invoke it via the bundled Node bridge at `redb/extractors/js_extractors/scripts/js-xray-runner.js`. The Dockerfile runs `npm install --omit=dev` in that directory at build time; on the host run the same once: `cd redb/extractors/js_extractors/scripts && npm install --omit=dev`. When `node_modules/@nodesecure/js-x-ray` is absent, the Python wrapper short-circuits without forking a subprocess and the pipeline falls back to heuristic-only obfuscation detection (no error, just a `debug` log line). A local patch (see *Patches* below) is auto-applied by `patch-package` during `npm install` to fix a Node 22 compatibility regression.

#### Patches

`redb/extractors/js_extractors/scripts/patches/` holds local patches applied to `node_modules/` after every `npm install` via the `postinstall: patch-package` hook in `package.json`. There's currently one:

| File | Upstream | What it fixes |
|---|---|---|
| `@nodesecure+js-x-ray+7.3.0.patch` | [@nodesecure/js-x-ray#???](https://github.com/NodeSecure/js-x-ray) | Changes `import { builtinModules } from "repl"` to `from "module"` in `src/probes/isLiteral.js`. `repl.builtinModules` was a deprecated re-export that Node 22.x stopped exposing as a named ESM export somewhere between 22.10 and 22.22; `module.builtinModules` is the canonical location and works on every Node ≥9.3. Without the patch, importing js-x-ray throws `SyntaxError: The requested module 'repl' does not provide an export named 'builtinModules'` and the bridge falls back to heuristic-only. |

Patches apply automatically — no manual step required. They're regenerated with `npx patch-package <package-name>` after editing the file in `node_modules/`. Drop a patch by deleting its file in `patches/` once upstream ships a fix.

### Host CLI vs Docker

| | Host CLI | Docker (SaaS) |
|---|---|---|
| Python deps | `pip install -r requirements.txt` | done at image build |
| Node 22 LTS | install once on host (see below) | bundled in image |
| webcrack | `sudo npm install -g webcrack@2.16.0`* | bundled in image |
| @nodesecure/js-x-ray | `cd redb/extractors/js_extractors/scripts && npm install --omit=dev` | bundled in image |

\* Global `npm install -g` writes into `/usr/lib/node_modules/` on a system-installed Node (apt / NodeSource), which is root-owned — so `sudo` is required. Skip the `sudo` if you installed Node via `nvm` or a user-owned prefix. The js-x-ray install is local to the repo so it does *not* need root; running it under `sudo` only makes `node_modules/` root-owned (harmless, the Python wrapper only reads, but tidier without).

Both paths produce the same fully-equipped pipeline. The Docker image is self-contained — unlike Binary Ninja (which is mounted from the host because of size and licensing), the JS Node tools are small enough to bundle.

#### Installing Node 22 LTS on the host

Pick whichever matches your OS — all paths land you on `node --version` reporting `v22.x`.

**macOS (Homebrew).** Most REDB developers run macOS; `brew` is the path of least resistance:

```bash
brew install node@22
brew link --overwrite node@22
node --version  # v22.x
```

**Linux (Debian / Ubuntu via NodeSource).** Same recipe the Dockerfile uses, so behaviour matches the container exactly:

```bash
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt-get install -y nodejs
node --version
```

**Linux/macOS via `nvm` (multiple Node versions on one host).** Useful if other projects on the same machine want different majors:

```bash
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
nvm install 22 --lts
nvm use 22
```

After Node is in place, run the two `npm install` commands from the table above. Verify the toolchain with these commands (run them from the repo root — adjust the path if your repo lives elsewhere):

```bash
# webcrack on PATH (global install)
which webcrack && webcrack --version                  # 2.16.0

# js-x-ray installed locally next to the bridge
ls -d redb/extractors/js_extractors/scripts/node_modules/@nodesecure/js-x-ray

# end-to-end smoke test — should print one line of JSON
node redb/extractors/js_extractors/scripts/js-xray-runner.js test_files/test_malicious.js
```

If either of the first two checks fails, the JS pipeline still runs — webcrack falls back to `jsbeautifier` and js-x-ray short-circuits to heuristic-only obfuscation detection — but you lose the obfuscator-family identification and most semantic deobfuscation. The Python side never raises on a missing tool; it logs at `debug` and moves on.

---

## Pipeline architecture

For every JS sample, `workers.py` builds **one `JSContext`** (`redb/extractors/js_extractors/js_context.py`) and threads it into every JS extractor that runs. The context owns all per-sample shared state:

| `JSContext` field | Computed | Consumed by |
|---|---|---|
| `raw_bytes` | Single `open(...).read()` at construction | `BasicPropertiesExtractor`/`HashExtractor` go through their own paths; `self.binary` on each JS extractor delegates here |
| `source` | Decoded once at construction (BOM → UTF-8 → chardet → latin-1 fallback) | `self.js_source` on every JS extractor |
| `lines` | `source.splitlines()`, cached on first access | `self.lines` on every JS extractor |
| `text_entropy` | Shannon entropy over `source`, cached | `JSFeaturesExtractor` (stored as `text_entropy` column), `JSDeobfuscationExtractor` (`original_entropy`) |
| `scan` | One `scan_source()` pass producing `{pattern_name: {count, lines}}` for every regex in `js_patterns.PATTERNS` and `js_patterns.FEATURE_PATTERNS`, cached | `JSFeaturesExtractor` (per-pattern counts + obfuscation score + technique detection), `JSSuspiciousAPIsExtractor` (every finding), `JSDeobfuscationExtractor` (original-side `new_apis_found` set) |
| `ast` | `pyjsparser.parse(source)` lazily on first access, returns `None` if pyjsparser is absent or parsing fails | `JSFeaturesExtractor` for `total_function_count` / `total_variable_count` / `max_nesting_depth` / `avg_identifier_length` |
| `deobfuscated` | External JS deobfuscator (default `webcrack`) with `jsbeautifier` fallback, run lazily once per sample. Returns `(text, normalizer_used)` or `(None, None)` when neither produced output | `JSDeobfuscationExtractor` (metrics row), `JSContentExtractor` (persisted text) — both read the same cached value, so the subprocess runs at most once |
| `xray` | `@nodesecure/js-x-ray` invoked via the bundled Node bridge, run lazily once per sample. Returns `XRayResult(obfuscator, warnings)`; empty when the bridge or its `node_modules` are missing, when Node is absent, or when the subprocess errors out | `JSFeaturesExtractor` reads `obfuscator` for the `obfuscator_name` column and uses it as the authoritative signal in the obfuscation verdict |
| `content_type` | The magika label workers.py dispatched on (`"javascript"`), carried through so `JSContentExtractor` can record it without re-running magika | `JSContentExtractor` |

The shape eliminates the per-extractor disk reads, source decodes, scan passes, AST parses, deobfuscation runs, and entropy computations the pipeline used to do independently for each extractor instance.

### Shared regex catalogue

All compiled regexes live in `redb/extractors/js_extractors/js_patterns.py`:

- `PATTERNS` — 46 named entries that double as suspicious-API row labels and as count sources for the features extractor (the 8 patterns shared across both extractors are defined exactly once here).
- `CATEGORIES` — pattern name → category (`code_execution` / `network` / `filesystem` / `process` / `registry` / `crypto_encoding` / `dom_manipulation`).
- `FEATURE_PATTERNS` — 11 additional regexes used only by `JSFeaturesExtractor` (hex/unicode escapes, base64 strings, comments, string concatenation, etc.).
- `STRING_PATTERNS` — 6 regexes used only by `JSStringsExtractor` for encoded-string discovery (`hex_escape_seq`, `unicode_escape_seq`, `charcode_call`, `base64_quoted`, `long_quoted`, `concat_chain`). Distinct from the look-alike entries in `FEATURE_PATTERNS` (e.g. `STRING_PATTERNS["hex_escape_seq"]` matches 4+ consecutive `\xHH` while `FEATURE_PATTERNS["hex_escape"]` matches a single one). Not folded into `JSContext.scan` because the strings extractor needs the match objects (capture groups, raw text) and is the sole consumer.
- `scan_source(source, patterns=...)` — runs every compiled pattern against `source` once, with O(log N) line lookup via a precomputed line-offset table, and returns `{name: {"count": int, "lines": [unique_sorted]}}` for any pattern that matched.

All `PATTERNS` are compiled with `re.IGNORECASE`. JS is case-sensitive at runtime, but the patterns themselves match literal-case identifiers (`eval`, `atob`, `WScript.Shell`, etc.) that real-world JS spells exactly as written, so IGNORECASE produces no extra matches in normal code while making the catalogue easier to share. Two pinned tests (`test_pattern_match_is_case_insensitive`, `test_pattern_counts_are_case_insensitive`) guard against an accidental flag regression.

---

## JSFeaturesExtractor

**Table:** `redb_js_features` (1 row per sample)

Extracts structural metadata and obfuscation indicators from JavaScript source code. No external tools required — pure regex (via the shared `JSContext.scan`) and optional AST parsing.

### Fields

File size and byte-level entropy are not stored here — they are written by `BasicPropertiesExtractor` (`redb_basic_properties.filesize`, `redb_basic_properties.file_entropy`) and joinable on `sha256`. Character-level entropy is stored separately as `text_entropy` because it differs meaningfully from byte entropy on non-ASCII sources (e.g. UTF-16 inflates byte counts and depresses byte entropy).

| Field | How it is extracted |
|-------|-------------------|
| `line_count` | `source.splitlines()` count |
| `char_count` | Length of decoded text (distinct from `filesize` for non-ASCII sources) |
| `text_entropy` | Shannon entropy over the character distribution of the decoded source text. Distinct from `redb_basic_properties.file_entropy`, which is over raw bytes. Obfuscated/packed JS typically scores above 5.0; clean code is usually 4.0–4.8. The obfuscation-score thresholds are tuned on this value |
| `max_line_length` | Longest line in characters. Values above 5–10K suggest minification or single-line obfuscation |
| `avg_line_length` | Mean line length across all lines |
| `is_minified` | True when the file has fewer than 5 lines but more than 500 characters, or when `avg_line_length` exceeds 500. These thresholds come from observing webpack/uglify output vs hand-written code |
| `is_likely_obfuscated` | True when `@nodesecure/js-x-ray` recognised the obfuscator family, OR when the heuristic score reaches 60 *and* at least one strong signal fired (encoding density >5%, single line >10K chars, avg identifier length <2, or text entropy >5.0). The two-tier check stops mid-band entropy + single eval + handful of `\xHH` escapes from masquerading as a verdict — the failure mode of the original score-only threshold |
| `obfuscator_name` | Family name reported by js-x-ray (`jsfuck`, `obfuscator.io`, `morse`, `jjencode`, `freejsobfuscator`, ...) or empty when js-x-ray didn't flag the sample / wasn't installed. When this is non-empty, `is_likely_obfuscated` is forced True regardless of the heuristic |
| `obfuscation_score` | Weighted heuristic score 0–100 (see section below). Kept as the explainability layer even when the verdict comes from js-x-ray |
| `obfuscation_techniques` | Array of detected technique labels (see section below) |
| `eval_count` | Regex `\beval\s*\(` — direct eval calls, the most common JS code execution vector |
| `function_constructor_count` | Regex `\bnew\s+Function\s*\(` — `new Function("code")` is equivalent to eval but harder to grep for |
| `settimeout_setinterval_count` | Regex `\b(setTimeout\|setInterval)\s*\(` — when called with a string argument these execute code after a delay, commonly used to evade sandbox timeouts |
| `document_write_count` | Regex `\bdocument\.write(ln)?\s*\(` — injects HTML/script into the page, used by exploit kits |
| `innerhtml_count` | Regex `\.innerHTML\s*=` — DOM injection, common in XSS and skimmers |
| `unescape_count` | Regex `\bunescape\s*\(` — deprecated decoding function, almost exclusively found in malware |
| `fromcharcode_count` | Regex `String\.fromCharCode\s*\(` — converts integer arrays to strings, used to hide payloads from static string matching |
| `atob_count` | Regex `\batob\s*\(` — base64 decode, commonly wraps encoded payloads |
| `decodeuri_count` | Regex `\b(decodeURI\|decodeURIComponent)\s*\(` — URL decoding used to unpack percent-encoded payloads |
| `total_function_count` | AST: counts `FunctionDeclaration`, `FunctionExpression`, `ArrowFunctionExpression` nodes. Falls back to regex `\bfunction\s+\w+\s*\(\|\bfunction\s*\(` when pyjsparser is not installed |
| `total_variable_count` | AST: counts declarations inside `VariableDeclaration` nodes. Regex fallback: `\b(var\|let\|const)\s+` |
| `max_nesting_depth` | AST: tracks depth through `BlockStatement` and function nodes. 0 when AST is unavailable. Deep nesting (>5) correlates with obfuscation wrappers |
| `avg_identifier_length` | AST: mean character length of all `Identifier` node names. Obfuscators like javascript-obfuscator produce 1–2 character names (`_0x4a2f`, `a`, `b`); clean code averages 6–10. Computed by pyjsparser when the source is ES5.1; on ES2015+ sources (destructuring, classes, optional chaining, etc.) pyjsparser fails parse and the value falls back to `idsLengthAvg` from `@nodesecure/js-x-ray`, which uses a modern parser. Equals `0.0` only when both paths are unavailable |
| `hex_string_count` | Count of `\xHH` escape sequences via regex `\\x[0-9a-fA-F]{2}`. High counts indicate hex-encoded string literals |
| `unicode_escape_count` | Count of `\uHHHH` escape sequences. Same reasoning as hex — used to hide readable strings |
| `long_string_count` | String literals longer than 256 chars inside quotes. Long strings often contain encoded payloads |
| `base64_string_count` | Sequences of 40+ base64 characters. Matches `[A-Za-z0-9+/]{40,}={0,2}` |
| `comment_ratio` | Ratio of characters inside `//` and `/* */` comments to total characters. Obfuscated code rarely has comments; a ratio near 0 combined with large file size is suspicious |
| `script_type` | First-match file-format classification (see *Script type values* below). Distinct from `detected_environment`, which classifies the runtime API surface — an HTA, for example, is `script_type=hta` *and* `detected_environment=wscript` |
| `detected_environment` | First-match runtime classification by API presence (see *Environment values* below) |

#### Script type values

Checked in this order; first match wins. The ordering encodes specificity — encoded JScript can only be `jse`, a WSF wrapper can only be `wsf`, etc.

| Value | Trigger |
|-------|---------|
| `jse` | Source starts with `#@~^` (JScript.Encode marker). Body is unanalysable until decoded |
| `wsf` | First 4KB contains `<job`/`<package` *and* `<script` — Windows Script File XML wrapper |
| `hta` | First 4KB contains `<hta:application` or the `application/hta` MIME hint — runs under mshta.exe |
| `embedded_html` | Starts with `<!`/`<html` or contains `<script` in first 2000 chars (generic HTML host) |
| `wscript` | Contains `WScript.` or `WSH.` (loose `.js` invoked via `wscript.exe` / `cscript.exe`) |
| `esm` | Line-anchored `import …from "…"` / bare side-effect `import "…"` / top-level `export …` |
| `node_module` | Contains `require(` or `module.exports` (CommonJS) |
| `standalone` | Fallback when nothing above matches |
| `unknown` | Empty source |

#### Environment values

Checked in this order; first match wins.

| Value | Trigger |
|-------|---------|
| `wscript` | `WScript.`, `WSH.`, `ActiveXObject`, `Scripting.FileSystemObject`, `WScript.Shell`, `ADODB.Stream` |
| `browser_extension` | `chrome.runtime`, `chrome.tabs`, `chrome.storage`, `chrome.webRequest`, `browser.runtime`, `browser.tabs` (MV2/MV3 extension APIs) |
| `service_worker` | `self.addEventListener('fetch'`, `self.importScripts`, `self.skipWaiting`, `caches.match`, `caches.open` (worker-only APIs not present in regular pages) |
| `deno` | `Deno.` (Deno runtime global) |
| `node` | `require(`, `module.exports`, `process.env`, `__dirname`, `__filename`, `Buffer.`, `child_process` |
| `browser` | `document.`, `window.`, `navigator.`, `localStorage`, `sessionStorage`, `XMLHttpRequest`, `addEventListener` |
| `unknown` | Fallback |

### Two-tier obfuscation verdict

The `is_likely_obfuscated` boolean is the answer to "should an analyst treat this file as obfuscated." It comes from two sources, in priority order:

1. **js-x-ray hit (authoritative).** When `@nodesecure/js-x-ray` recognises the obfuscator family, the verdict is `True` and `obfuscator_name` carries the family label. js-x-ray catches `jsfuck`, `obfuscator.io`, `morse`, `jjencode`, and `freejsobfuscator` by AST shape, which is far more precise than any heuristic.
2. **Heuristic with strong-signal corroboration.** When js-x-ray either didn't flag the sample or isn't installed, the heuristic decides: `obfuscation_score >= 40` AND at least one *strong* signal fired. Strong signals are unambiguous on their own; weak signals are commonly seen in legitimate code and only count toward the score, not toward the strong-signal gate. The strong-signal gate (not the score threshold) is what does the heavy lifting against false positives — a clean file with multiple weak ticks but no strong signal cannot be flagged regardless of the score.

The two-tier check is a deliberate response to the score-only threshold's failure mode: a non-obfuscated file with mid-band entropy, a single `eval`, and a handful of `\xHH` escapes used to clear `>= 40` and show up as `is_obfuscated: Yes` even though it was just legitimate code with one or two ambient indicators. With strong-signal corroboration, three weak ticks alone no longer cross the line.

### Obfuscation score breakdown

The score is a sum of weighted indicators, capped at 100:

| Indicator | Tier | Weight | Rationale |
|-----------|------|--------|-----------|
| Hex/unicode escape density > 5% of source | strong | +20 | Encoded payload — at this density the source is mostly escape sequences |
| Hex/unicode escape density > 1% | weak | +8 | Notable encoding but could also be a few hex literals in legitimate code |
| Avg identifier length < 2 chars | strong | +15 | Obfuscators shorten everything to single chars; clean code averages 6+ |
| Avg identifier length < 3 chars | weak | +6 | Slightly longer but still suspicious |
| Max line > 10K chars | strong | +15 | Single enormous line — hallmark of packer output |
| Max line > 5K chars | weak | +8 | Long single line |
| `text_entropy` > 5.0 | strong | +15 | Encoded payload range. The old 4.5–4.8 weak band caught jQuery and is dropped |
| Each `eval()` call (capped at +12) | weak | +4 each | One eval is normal in templating / AngularJS / polyfills; only piles of them count |
| `String.fromCharCode` present | weak | +6 | Common in legacy escapers but worth a tick |
| String concat density > 20 per 100 lines | weak | +8 | Excessive `"a" + "b" + "c"` rebuild of greppable strings |
| Comment ratio < 1% + few lines + size > 1 KB | weak | +5 | Minifier/packer tell |
| Non-ASCII codepoint density > 30% | strong | +20 | Unicode-codepoint payload (e.g. WSH droppers building a runtime string of >0x7f chars). Real-world JS averages <5% non-ASCII; >30% is almost always obfuscation. The strong-signal gate prevents the corner-case false-positive on heavy-localization files (which can cross 30% legitimately) — a localization file scoring only this signal can't reach the threshold |
| Non-ASCII codepoint density > 10% | weak | +8 | Notable non-ASCII presence — could be substantial i18n in legitimate code, or the start of a Unicode-codepoint obfuscation pattern |
| Line-uniqueness ratio < 10% (line_count > 100) | strong | +15 | Junk-padded bulk: thousands of duplicate lines burying the actual logic. Hand-written code has near-1 uniqueness even in repetitive sections (CSS-in-JS, fixture data, etc.) |
| Line-uniqueness ratio < 30% (line_count > 100) | weak | +6 | Significant repetition; could be a packer working from a small template, or padding warming up |

### Obfuscation techniques detected

Each technique is flagged when its threshold is exceeded. Density-based tags use the same bar as the score's strong-signal threshold so the displayed tags reflect what the score actually credited:

| Technique label | Detection rule |
|----------------|---------------|
| `eval_usage` | `eval(` present |
| `function_constructor` | Function-constructor invocation present (`new Function(...)`) |
| `hex_encoding` | More than 5 `\xHH` sequences AND density > 0.1% of source |
| `unicode_encoding` | More than 5 `\uHHHH` sequences AND density > 0.1% of source |
| `charcode_encoding` | More than 3 `String.fromCharCode(` calls |
| `string_concatenation` | More than 10 `"..." + "..."` patterns |
| `base64_decoding` | `atob(` present |
| `unescape_usage` | `unescape(` present |
| `array_function_calls` | Pattern `[0xNN](` or `[N](` — calling functions via array index lookup, typical of javascript-obfuscator output |
| `short_identifiers` | `0 < avg_identifier_length < 3.0` — identifiers averaging under 3 chars, typical of obfuscator.io's `_0xNNNN` renaming. Sourced from pyjsparser when the file parses as ES5.1, falling back to js-x-ray's `idsLengthAvg` on ES2015+ sources |
| `packed_single_line` | `max_line_length > 5000` — single enormous line, hallmark of packer/minifier output |
| `high_entropy` | `text_entropy > 5.0` — character distribution in encoded-payload range; distinct from `redb_basic_properties.file_entropy` (byte entropy) |
| `non_ascii_payload` | Non-ASCII codepoint density > 10%. Catches Unicode-codepoint stuffing (e.g. `this.x += "<U+1184><U+159b>..."` repeated thousands of times) — a pattern the per-escape `unicode_encoding` tag misses because the source contains the actual codepoints, not literal `\uHHHH` escape sequences |
| `repetitive_padding` | Line-uniqueness ratio < 30% with line_count > 100. Junk-filled bulk burying the actual payload; the line-count floor prevents false positives on tiny files that happen to repeat a few lines |

---

## JSSuspiciousAPIsExtractor

**Table:** `redb_js_suspicious_apis` (multi-row per sample, one row per detected API)

Reads `JSContext.scan` and emits one row per `js_patterns.PATTERNS` entry that matched the source. Findings are emitted in the canonical insertion order of `PATTERNS` (`code_execution` → `network` → `filesystem` → `process` → `registry` → `crypto_encoding` → `dom_manipulation`) so output ordering is deterministic. Each pattern matches a specific API call or object instantiation known to be used in malicious JavaScript.

### Categories and patterns

**code_execution** — APIs that execute arbitrary code:
`eval()`, `new Function()`, `execScript()`, `document.write()`, `.innerHTML =`, `.outerHTML =`, `.insertAdjacentHTML()`

**network** — APIs that make network requests:
`new XMLHttpRequest`, `fetch()`, `new WebSocket()`, `navigator.sendBeacon()`, `ActiveXObject("MSXML2.XMLHTTP")`, `require("http"/"https"/"net"/"dgram")`, `axios`

**filesystem** — APIs that access the filesystem:
`require("fs")`, `require("path")`, `Scripting.FileSystemObject`, `ADODB.Stream`, `Shell.Application`, `WScript.CreateObject`

**process** — APIs that spawn processes:
`require("child_process")`, `child_process.exec/spawn/execFile/fork`, `WScript.Shell`, `.Run()`, `.Exec()`, `ShellExecute`, `"powershell"`, `"cmd.exe"`, `require("os")`

**registry** — Windows registry access:
`.RegRead()`, `.RegWrite()`, `.RegDelete()`, `StdRegProv`

**crypto_encoding** — Encoding/decoding/crypto operations:
`atob()`, `btoa()`, `String.fromCharCode()`, `unescape()`, `decodeURIComponent()`, `Buffer.from()`, `crypto.createCipher/Decipher/Hash/Hmac`

**dom_manipulation** — DOM operations typical of skimmers/injectors:
`document.forms`, `document.cookie`, `querySelector` targeting password/credit/card/cvv/ssn inputs, `addEventListener("submit")`, `createElement("script"/"iframe")`, `.src = "http://..."`

### Output fields

| Field | Description |
|-------|-------------|
| `api_name` | Human-readable name of the matched API |
| `api_category` | One of the 7 categories above |
| `call_count` | Number of lines where the pattern matched |
| `line_numbers` | Array of line numbers (1-indexed) where the API was found |
| `context_snippet` | Up to 3 truncated source lines where the API appears (max 200 chars each, joined by ` \| `) |

---

## JSStringsExtractor

**Table:** `code_binja_strings_raw` (shared with binary string extraction)

Finds encoded strings in JS source, decodes them, and writes both the decoded value and the original encoded form to the same table used by DecompileBinja and DecompileAPK. This means `string:"powershell"` queries return results from all formats.

The 6 detection regexes live in `js_patterns.STRING_PATTERNS` (compiled once at module load); the per-match concat tokeniser is also compiled once. Line numbers for each finding (`string_offset`) are looked up in O(log L) via `bisect` against a newline-offset table built once per `extract()` call — the historical O(N·M) `source[:start].count('\n')` pass is gone.

### Decoding methods

Scope: only *hidden* strings — values whose decoded form is not visible to a substring search over the raw text. Plain long literals are not extracted here because they're already preserved in `code_text_content.text_raw` and scraped by the IOC pipeline over the same `text_raw` / `text_normalized` surfaces (column names match the `redb_iocs.source_type` enum values, so a join across the two tables doesn't have to translate names).

| `string_encoding` value | What it decodes | Example input | Example output |
|------------------------|----------------|---------------|----------------|
| `hex` | `\xHH` escape sequences (4+ consecutive) | `\x68\x74\x74\x70` | `http` |
| `unicode` | `\uHHHH` escape sequences (3+ consecutive) | `WScript` | `WScript` |
| `charcode` | `String.fromCharCode(N, N, ...)` calls | `String.fromCharCode(112, 111, 119)` | `pow` |
| `base64` | Base64 strings (40+ chars) inside quotes. Only kept if decoding produces >80% printable UTF-8 text | `"cG93ZXJzaGVsbA=="` | `powershell` |
| `concat` | Reassembled `"a" + "b" + "c"` concatenation (3+ parts) | `"ht" + "tp" + "://" + "evil" + ".com"` | `http://evil.com` |

### Field mapping to shared table

| Shared column | JS value |
|--------------|----------|
| `string` | Decoded/reconstructed string value |
| `string_raw` | Original encoded form as it appeared in source |
| `string_encoding` | One of: hex, unicode, charcode, base64, concat, plaintext |
| `string_offset` | Line number in the JS source file (1-indexed) |
| `string_length` | Length of the decoded string |
| `string_raw_length` | Length of the original encoded form |
| `string_entropy` | Shannon entropy of the decoded string |

---

## JSDeobfuscationExtractor

**Table:** `redb_js_deobfuscation` (1 row per sample)

Attempts to deobfuscate the JS source using external tools, then compares pre/post metrics to measure how much was hidden.

### Tool chain

1. **Primary: webcrack** (or any tool at `JS_DEOBFUSCATOR_PATH` env var). Run as a subprocess with `JS_DEOBFUSCATE_TIMEOUT` seconds timeout (default 60). The tool receives the source file path and its stdout is captured as the deobfuscated output. Process group management handles cleanup on timeout (SIGTERM then SIGKILL).

2. **Fallback: jsbeautifier** (Python library). Used when the primary tool is not installed. Normalizes formatting (indentation, line breaks) but does not perform semantic deobfuscation. Still useful because it makes minified code readable and can reveal strings that were hidden by formatting tricks.

### Output fields

| Field | Description |
|-------|-------------|
| `deobfuscator_used` | Name of the tool that produced the output (`webcrack`, `jsbeautifier`, etc.) |
| `deobfuscation_successful` | 1 if the tool produced non-empty output |
| `original_size` | Character count of the input source |
| `deobfuscated_size` | Character count of the deobfuscated output |
| `size_change_ratio` | `deobfuscated_size / original_size`. Values significantly different from 1.0 indicate the tool transformed the code |
| `original_entropy` | Shannon entropy of the input. Reused from `JSContext.text_entropy` so the same Shannon computation is not redone here |
| `deobfuscated_entropy` | Shannon entropy of the output. A drop in entropy after deobfuscation suggests encoded content was unpacked into readable text |
| `new_strings_found` | Count of string literals (4+ chars) present in the deobfuscated output but absent in the original. These are strings that were hidden by the obfuscation |
| `new_apis_found` | Count of `PATTERNS` entries that matched the deobfuscated output but did not match the original. The original-side pattern set is read from `JSContext.scan` (already computed once for this sample); only the deobfuscated text triggers an additional `scan_source()` pass since that text is unique to this extractor. Reveals API calls that were concealed |
| `deobfuscated_sha256` | SHA-256 of the deobfuscated output, for deduplication and cross-referencing |

---

## JSContentExtractor

**Table:** `code_text_content` (1 row per sample, shared with future text-content extractors)

Persists the actual text of the sample (raw + normalised) so analysts can re-query the source content directly and so future improvements to IOC extraction or pattern matching can be re-applied without re-running the deobfuscator. The same table is intended to host any text-based artefact in the future (PowerShell, Python, plain text, email bodies, extracted PDF/Office text); the `content_type` column carries the magika label so callers can filter without joining other tables.

The deobfuscation pass is computed once per sample and shared with `JSDeobfuscationExtractor` (which writes the metrics row), so this extractor adds no extra subprocess cost.

| Field | Description |
|-------|-------------|
| `content_type` | The magika label for the artefact (`"javascript"` for JS samples). Lets a single table hold heterogeneous text content without per-format tables |
| `text_raw` | The decoded source as it sits on disk. Column name matches the `redb_iocs.source_type='text_raw'` enum value, so an analyst tracing an IOC back to its surface lands on the column with the same identifier |
| `text_normalized` | Output of the deobfuscator (or jsbeautifier fallback). `NULL` when neither produced output, distinguishing "we tried and got nothing" from a successful normalisation. Same naming alignment with `redb_iocs.source_type='text_normalized'` |
| `normalizer_used` | Name of the tool that produced the normalised text (`"webcrack"`, `"jsbeautifier"`, etc.). `NULL` when `text_normalized` is `NULL` |

Both `text_raw` and `text_normalized` are stored with ClickHouse `CODEC(ZSTD(3))` to keep storage cost reasonable across millions of samples.

---

## IOC extraction

**Table:** `redb_iocs` (shared with all formats)

JavaScript IOC extraction uses the same `IOCExtractorFromResults` class as DecompileBinja and DecompileAPK. JS samples get the same 22 IOC types (IPv4, IPv6, FQDN, URL, email, crypto addresses, CVEs, file paths, registry keys, etc.) with defanging support and IANA TLD validation. Called automatically in `workers.py` after the JS-specific extractors complete.

### Windows paths & registry keys in source-code form

`WINDOWS_PATH_PATTERN` and `REGISTRY_KEY_PATTERN` accept both the runtime form (`C:\Windows\Temp`, `HKLM\SYSTEM\...`) and the source-escaped form (`C:\\Windows\\Temp`, `HKLM\\SYSTEM\\...`) that appears inside JS / JSON / PowerShell string literals. Doubled backslashes are normalised to single before storage so an analyst querying for `C:\Users\Public` sees both forms collapsed to one IOC. Wildcard segments (e.g. `C:\Users\*\AppData\Local\Temp`) are preserved.

Registry hives recognised: `HKLM`, `HKCU`, `HKCR`, `HKU`, `HKCC`, `HKPD`, `HKEY_LOCAL_MACHINE`, `HKEY_CURRENT_USER`, `HKEY_CLASSES_ROOT`, `HKEY_USERS`, `HKEY_CURRENT_CONFIG`, `HKEY_PERFORMANCE_DATA`. A bare hive mention with no path component does not match (avoids prose false positives).

Three surfaces are scraped for every JS sample, each tagged with its own `redb_iocs.source_type` value so analysts can tell where an IOC was first visible:

| `source_type` | Surface | Catches |
|---|---|---|
| `text_raw` | The decoded source as it sits on disk | URLs, IPs, emails, etc. that aren't hidden by encoding or wrapping |
| `text_normalized` | The deobfuscated/beautified form (only added when it differs from raw) | IOCs unwrapped by webcrack from `eval(atob(...))` payloads, identifiers exposed by jsbeautifier on minified code |
| `string` | The decoded strings produced by `JSStringsExtractor` (hex/unicode/charcode/base64/concat unpacked into plaintext) | URLs and FQDNs hidden behind `String.fromCharCode(...)`, base64-wrapped tokens, concatenated `"a" + "b" + ...` chains, etc. |

The `text_raw` and `text_normalized` values are universal across text-based artefacts — the same two `SourceType` values are intended to host PowerShell, Python, email body, and extracted PDF/Office text in the future.