Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,35 @@ cum inspect --text "C​laude" # ZWSP between C and l
curl -s https://example.com | cum inspect --stdin --media html
```

### `cum enhance` - Stochastic synonym replacement

```
cum enhance [OPTIONS] [FILE]
```

Applies a best-effort stochastic synonym replacement to plain-text input to defeat Layer B statistical watermarks (SynthID-Text, KGW). By default, each non-stop word is replaced with a 50% probability using a curated synonym table and the system dictionary.

| Argument | Default | Description |
| ----------------------- | ------- | ------------------------------------------------ |
| `[FILE]` | - | Path to the plain-text file to enhance. |
| `-t`, `--text <TEXT>` | - | Inline text to enhance. |
| `--stdin` | - | Read plain-text input from stdin. |
| `-o`, `--output <OUT>` | stdout | Write the enhanced output to this path. |
| `-p`, `--probability` | `0.5` | Per-word substitution probability `[0.0, 1.0]`. |

#### Examples

```bash
# Enhance inline text with a 80% substitution chance
cum enhance --text "The chaos governs the universe" -p 0.8

# Enhance a text file and save the result
cum enhance report.txt --output enhanced_report.txt

# Pipe via stdin
cat essay.md | cum enhance --stdin -p 0.6
```

## Supported Formats

| Extension | Detection | Layer |
Expand Down
126 changes: 116 additions & 10 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ rustdoc-args = ["--cfg", "docsrs"]

[dependencies]
thiserror = "2.0.20"
rand = "0.10.2"
phf = { version = "0.14.0", features = ["macros"] }
unicode-general-category = "1.1.0"
unicode_names2 = "3.1.0"
unicode-normalization = "0.1.25"
Expand Down
9 changes: 0 additions & 9 deletions NODE.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,3 @@ import { cleanText, inspectText, CleanTextResult, TextInspectReport } from ".";

const result: CleanTextResult = cleanText("Hello\u200b!");
```

## 🔍 See Also: Core Logic

The napi-rs hooks expose standard JavaScript strings and `Uint8Array` primitives directly into the Rust parser.

For details on the engine architecture and detection heuristics:

- The rust core leverages zero-copy buffer modifications where possible.
- Text strings are O(1) matching against pre-computed static codepoint slices in `clean_text`.
9 changes: 0 additions & 9 deletions PYTHON.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,3 @@ cleaned = cum_rs.clean_bytes(data)
with open("photo.cleaned.png", "wb") as f:
f.write(cleaned)
```

## 🔍 See Also: Core Logic

The PyO3 Python hooks seamlessly route queries into the Rust core, ensuring no Python loops block processing. String inspection is zero-copy until modification is detected.

For architecture and heuristic details on _how_ cleaning works natively, see:

- `cum_rs::unicode` for text algorithms.
- `cum_rs::image_meta` for PNG/JPEG headers.
21 changes: 18 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<div align="center">

<img src="assets/logo.webp" alt="cum-rs logo" width="220"/>
<img src="https://raw.githubusercontent.com/wiseaidotdev/cum/main/assets/logo.webp" alt="cum-rs logo" width="220"/>

# CUM

Expand All @@ -18,7 +18,7 @@
> Works regardless of provider (Claude, OpenAI, Gemini, Grok, open-LLM).
> All processing is **100% local**: no data leaves your machine.

![crab dancing](assets/crabby-dance.gif)
![crab dancing](https://raw.githubusercontent.com/wiseaidotdev/cum/main/assets/crabby-dance.gif)

_The `cum` binary, cheerfully evicting zero-width gremlins from your prose._

Expand All @@ -36,7 +36,7 @@ The good news: we have the specialized equipment. And it is written in Rust, so
| ------------------ | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| **A: Unicode** | ZWSP, bidi controls, tag chars, variation selectors, private-use codepoints, basically a Unicode horror movie | Deterministic, lossless exorcism 🧹 |
| **File: Metadata** | C2PA manifests, EXIF, XMP, document properties, the digital equivalent of a tracking ankle bracelet | Stripped from PNG, JPEG, WebP, SVG, PDF, DOCX, ODT, HTML, Markdown |
| **B: Statistical** | Token-sampling watermarks (SynthID-Text, KGW), watermarks baked into the actual word choices | Best-effort; the only real fix is to rewrite the text yourself, sorry |
| **B: Statistical** | Token-sampling watermarks (SynthID-Text, KGW), watermarks baked into the actual word choices | Best-effort via stochastic synonym replacement: `enhance` command / `enhance_text` API |

> **Fun fact:** some of those invisible characters are technically in the Unicode "Tag" block, which was originally designed for plane tickets in 1997 and then deprecated. AI providers found a new use for them. The Unicode Consortium is presumably very proud.

Expand Down Expand Up @@ -127,6 +127,21 @@ console.log(result.removedCount); // 2

See **[NODE.md](NODE.md)** for the full binding reference.

## 🎲 Stochastic Enhancer

CUM includes a best-effort countermeasure against Layer B statistical watermarks (SynthID, KGW) by stochastically replacing eligible words with semantically equivalent synonyms. This modifies the raw byte pairs chosen by the LLM's token-sampler, disrupting the periodic watermark signal.

```rust
use cum_rs::stochastic::StochasticEnhancer;

let enhancer = StochasticEnhancer::new(0.5); // 50% substitution chance
let output = enhancer.enhance("The chaos governs the universe.");
println!("Substituted {} words", output.words_substituted);
println!("{}", output.text);
```

The synonyms table relies on a curated perfect-hash map combined with the host's `/usr/share/dict/` system wordlist.

## 🚨 Disclaimer _(the responsible adult part)_

**Layer A** (Unicode scrubbing) and **file metadata** stripping are fully deterministic and lossless: every modification is logged in `stats`. You can see exactly what changed.
Expand Down
27 changes: 13 additions & 14 deletions RUST.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,19 @@ for hit in &report.hits {
}
```

## Stochastic Enhancement (Layer B)

```rust
use cum_rs::stochastic::StochasticEnhancer;

// Create an enhancer with 70% substitution probability
let enhancer = StochasticEnhancer::new(0.7);

let output = enhancer.enhance("The chaos governs the universe.");
println!("Enhanced text: {}", output.text);
println!("Words substituted: {}", output.words_substituted);
```

## Image Metadata Stripping

```rust
Expand Down Expand Up @@ -59,17 +72,3 @@ println!("Chunks removed: {}", output.stats.metadata_chunks_removed);
| `python` | PyO3 extension module |
| `node` | napi-rs Node.js add-on |
| `wasm` | wasm-bindgen WASM bindings |

## 🔍 See Also: Core Logic & Reasoning

The core implementation works identically regardless of binding (Rust, Python, Node, WASM).

1. **`src/unicode.rs`** — **Layer A (Text)**
- Text sweeps process string characters asynchronously against a static known list of Unicode categories (`STRIP_CODEPOINTS`, `EMOJI_GLUE`, etc.).
- This operates at `O(1)` per-character space/time complexity via static pattern matching and binary search over codepoint slices.

2. **`src/image_meta.rs` & `src/container_meta.rs`** — **Metadata layer**
- Media cleaners rely on byte boundary scanning or deterministic structural formats (e.g., zip streams for docx). We don't read entire payloads into graphics stacks, we do zero-allocation sub-slice byte patching where possible to ensure robust minimal modifications without risk of arbitrary-code execution on maliciously formed headers.

3. **`src/cleaner.rs`** — **Format Auto-Detection**
- Implements a magic-byte sniffer prioritizing fast chunk detection. If it cannot identify a header (like a PNG `89 50 4E 47`), it falls back to parsing as UTF-8 plaintext.
Loading