By Gapipro


2012-05-25 17:31:40 8 Comments

I have this Unicode string: Ааа́Ббб́Ввв́Г㥴Дд

And I want to it split by chars. Right now if I try to loop truth all chars I get something like this:
A a a ' Б ...

Is there a way to properly split this string to chars: А а а́ ?

5 comments

@Zach Bloomquist 2019-08-13 20:18:30

If you're writing an application that needs to consume chunks of data from a Node.js stream, then you can probably just pipe through utf8-stream to prevent this:

https://github.com/substack/utf8-stream

@madcampos 2016-10-04 07:17:38

A little update on this.

As ES6 came by, there are new string methods and ways of dealing with strings. There are solutions for two problems present in this.

1) Emoji and surrogate pairs

Emoji and other Unicode characters that fall above the Basic Multilingual Plane (BMP) (Unicode "code points" in the range 0x0000 - 0xFFFF) can be worked out as the strings in ES6 adhere to the iterator protocol, so you can do like this:

let textWithEmoji = '\ud83d\udc0e\ud83d\udc71\u2764'; //horse, happy face and heart
[...textWithEmoji].length //3
for (char of textWithEmoji) { console.log(char) } //will log 3 chars

2) Diacritics

A harder problem to solve, as you start to work with "grapheme clusters" (a character and it's diacritics). In ES6 there is a method that simplify working with this, but it's still hard to work. The String.prototype.normalize method eases the work, but as Mathias Bynens puts:

(A) code points with multiple combining marks applied to them always result in a single visual glyph, but may not have a normalized form, in which case normalization doesn’t help.

More insight can be found here:

https://ponyfoo.com/articles/es6-strings-and-unicode-in-depth https://mathiasbynens.be/notes/javascript-unicode

@ygormutti 2018-07-10 19:07:05

This is the best answer since ES6 came out. Could mention Array.from which also uses String iterator for completeness sake.

@ygormutti 2018-07-10 19:14:28

Now I see this is not exactly what the OP asked for, but perfect for the issue that brought me here (surrogate pairs). The question title needs an improvement.

@Vitaly Domnikov 2016-08-19 18:53:26

This package might help you: https://www.npmjs.com/package/runes

const runes = require('runes')

const example = 'Emoji 🤖'
example.split('') // ["E", "m", "o", "j", "i", " ", "�", "�"] 
runes(example)    // ["E", "m", "o", "j", "i", " ", "🤖"] 

@bobince 2012-05-27 11:46:51

To do this properly, what you want is the algorithm for working out the grapheme cluster boundaries, as defined in UAX 29. Unfortunately this requires knowledge of which characters are members of which classes, from the Unicode Character Database, and JavaScript doesn't make that information available(*). So you'd have to include a copy of the UCD with your script, which would make it pretty bulky.

An alternative if you only need to worry about the basic accents used by Latin or Cyrillic would be to take only the Combining Diacritical Marks block (U+0300-U+036F). This would fail for other languages and symbols, but might be enough for what you want to do.

function findGraphemesNotVeryWell(s) {
    var re= /.[\u0300-\u036F]*/g;
    var match, matches= [];
    while (match= re.exec(s))
        matches.push(match[0]);
    return matches;
}

findGraphemesNotVeryWell('Ааа́Ббб́Ввв́Г㥴Дд');
["А", "а", "а́", "Б", "б", "б́", "В", "в", "в́", "Г", "г", "Ґ", "ґ", "Д", "д"]

(*: there might be a way to extract the information by letting the browser render the string, and measuring the positions of selections in it... but it would surely be very messy and difficult to get working cross-browser.)

@Aleš Kotnik 2012-05-25 17:36:27

The problem of your string are surogate pairs ("a" "́) which get combined to signle character only when displayed by browser. For your case, it is enough if you attach \u0301 to the previous character but this is by no means a general solution.

var a="Ааа́Ббб́Ввв́Г㥴Дд",
    i =0,
    chars=[];

while(a.charAt(i)) {
  if (a.charAt(i+1) == "\u0301") {
    chars.push(a.charAt(i++)+a.charAt(i++));
  } else {
    chars.push(a.charAt(i++));}}

To clarify the issue, go and read Mathias Bynens's blog post.

@dda 2012-05-25 18:05:28

Your code is deeply flawed -- and besides having a bug, a.fromCharCode(i), really? -- it doesn't do composition, so you're back to square 1...

@Aleš Kotnik 2012-05-25 18:23:00

Thanx for the warning. Corrected.

@bames53 2012-05-25 18:30:14

Doesn't charCodeAt(index) work in terms of UTF-16 code units? So this wouldn't work for anything outside the BMP.

@Aleš Kotnik 2012-05-25 18:33:53

The question was how to split unicode string to array of single unicode characters and the code does just this. Check the chars array.

@Gapipro 2012-05-26 07:33:41

chars array still returns every separate char and doesn't combine "а" + "́" === "а́"

@hippietrail 2012-06-07 05:43:19

Surrogate pairs are a totally different thing to combining characters. Surrogates are when, in UTF-16, two successive 16-bit values combine to make one 32-bit codepoint. Combining characters are full codepoints which combine with a previous base codepoint to form one user-perceived character called a "grapheme cluster".

Related Questions

Sponsored Content

3 Answered Questions

63 Answered Questions

[SOLVED] What is the difference between String and string in C#?

84 Answered Questions

[SOLVED] How do I make the first letter of a string uppercase in JavaScript?

37 Answered Questions

[SOLVED] How do I loop through or enumerate a JavaScript object?

76 Answered Questions

[SOLVED] How do I iterate over the words of a string?

  • 2008-10-25 08:58:21
  • Ashwin Nanjappa
  • 2119241 View
  • 2851 Score
  • 76 Answer
  • Tags:   c++ string split

47 Answered Questions

[SOLVED] Convert form data to JavaScript object with jQuery

56 Answered Questions

[SOLVED] How to replace all occurrences of a string?

60 Answered Questions

[SOLVED] How do you split a list into evenly sized chunks?

34 Answered Questions

[SOLVED] How do I split a string on a delimiter in Bash?

18 Answered Questions

[SOLVED] Why is char[] preferred over String for passwords?

Sponsored Content