# unicode-segmentation-cj

**Repository Path**: iJetLi/unicode-segmentation-cj

## Basic Information

- **Project Name**: unicode-segmentation-cj
- **Description**: Rust unicode-segmentation库的仓颉cangjie实现
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-07-08
- **Last Updated**: 2026-07-12

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# unicode-segmentation-cj

A [Cangjie](https://cangjie-lang.cn/) implementation of the Unicode text boundary segmentation algorithms, ported from the Rust [unicode-segmentation](https://github.com/rust-lang/unicode-segmentation) crate v1.13.3.

Implements the text boundary segmentation algorithms defined by Unicode Standard Annex #29:

- **Grapheme Cluster segmentation** — split by grapheme clusters (user-perceived characters)
- **Word segmentation** — split by word boundaries
- **Sentence segmentation** — split by sentence boundaries

## Unicode Version

Unicode 17.0.0

## Features

### Grapheme Cluster Segmentation

```cangjie
let s = "a̐éö̲\r\n"

// Forward iteration
for (g in s.graphemes(true)) {
    println(g)  // "a̐", "é", "ö̲", "\r\n"
}

// Reverse iteration
var iter = s.graphemes(true)
var g = iter.next_back()
while (g.isSome()) {
    println(g.getOrThrow())
    g = iter.next_back()
}

// Offset iteration
for (pair in s.grapheme_indices(true)) {
    let (offset, grapheme) = (pair[0], pair[1])
    println("${offset}: ${grapheme}")
}
```

### Word Segmentation

```cangjie
let s = "The quick (\"brown\") fox can't jump 32.3 feet, right?"

// Alphanumeric words only
for (w in s.unicode_words()) {
    println(w)  // "The", "quick", "brown", "fox", "can't", "jump", "32.3", "feet", "right"
}

// Full boundaries including non-alphanumeric
for (w in s.split_word_bounds()) {
    println(w)  // "The", " ", "quick", " ", "(", ...
}
```

### Sentence Segmentation

```cangjie
let s = "Mr. Fox jumped. The dog was too lazy."

// Alphanumeric sentences only
for (sent in s.unicode_sentences()) {
    println(sent)
}

// Full boundaries (including spaces and punctuation)
for (sent in s.split_sentence_bounds()) {
    println(sent)
}
```

### GraphemeCursor Chunked Processing

```cangjie
// For rope/buffer non-contiguous memory scenarios
var cursor = GraphemeCursor(0, full_len, true)

try {
    let is_break = cursor.is_boundary(chunk, chunk_start)
} catch (e: GraphemeIncomplete) {
    // Need to provide more context
    cursor.provide_context(prev_chunk, prev_chunk_start)
    let is_break = cursor.is_boundary(chunk, chunk_start)
}
```

## Public API

### UnicodeSegmentation Interface

All methods are added to `String` via `extend String <: UnicodeSegmentation`:

| Method | Description |
|--------|-------------|
| `graphemes(is_extended)` | Grapheme Cluster iterator |
| `grapheme_indices(is_extended)` | Grapheme Cluster + byte offsets |
| `split_word_bounds()` | Word full boundary iterator |
| `split_word_bound_indices()` | Word full boundaries + offsets |
| `unicode_words()` | Alphanumeric-only word iterator |
| `unicode_word_indices()` | Words + offsets |
| `split_sentence_bounds()` | Sentence full boundary iterator |
| `split_sentence_bound_indices()` | Sentence boundaries + offsets |
| `unicode_sentences()` | Alphanumeric-only sentence iterator |

### Public Classes/Enums

- `GraphemeCursor` — chunked grapheme boundary detection
- `GraphemeIncomplete` — chunked processing error enum
- `UNICODE_VERSION` — (17, 0, 0)

## Build & Test

```bash
cjpm build
cjpm test
```

## License

MIT