By Pekka

2010-04-30 07:07:54 8 Comments

In 2010, would you serve URLs containing UTF-8 characters in a large web portal?

Unicode characters are forbidden as per the RFC on URLs (see here). They would have to be percent encoded to be standards compliant.

My main point, though, is serving the unencoded characters for the sole purpose of having nice-looking URLs, so percent encoding is out.

All major browsers seem to be parsing those URLs okay no matter what the RFC says. My general impression, though, is that it gets very shaky when leaving the domain of web browsers:

  • URLs getting copy+pasted into text files, E-Mails, even Web sites with a different encoding
  • HTTP Client libraries
  • Exotic browsers, RSS readers

Is my impression correct that trouble is to be expected here, and thus it's not a practical solution (yet) if you're serving a non-technical audience and it's important that all your links work properly even if quoted and passed on?

Is there some magic way of serving nice-looking URLs in HTMLüsseldorf?neighbourhood=Lörick

that can be copy+pasted with the special characters intact, but work correctly when re-used in older clients?


@Ciro Santilli 郝海东冠状病六四事件法轮功 2014-08-29 14:09:24

Not sure if it is a good idea, but as mentioned in other comments and as I interpret it, many Unicode chars are valid in HTML5 URLs.

E.g., href docs say

The href attribute on a and area elements must have a value that is a valid URL potentially surrounded by spaces.

Then the definition of "valid URL" points to, which defines URL code points as:

ASCII alphanumeric, "!", "$", "&", "'", "(", ")", "*", "+", ",", "-", ".", "/", ":", ";", "=", "?", "@", "_", "~", and code points in the ranges U+00A0 to U+D7FF, U+E000 to U+FDCF, U+FDF0 to U+FFFD, U+10000 to U+1FFFD, U+20000 to U+2FFFD, U+30000 to U+3FFFD, U+40000 to U+4FFFD, U+50000 to U+5FFFD, U+60000 to U+6FFFD, U+70000 to U+7FFFD, U+80000 to U+8FFFD, U+90000 to U+9FFFD, U+A0000 to U+AFFFD, U+B0000 to U+BFFFD, U+C0000 to U+CFFFD, U+D0000 to U+DFFFD, U+E1000 to U+EFFFD, U+F0000 to U+FFFFD, U+100000 to U+10FFFD.

The term "URL code points" is then used in a few parts of the parsing algorithm, e.g. for the relative path state:

If c is not a URL code point and not "%", parse error.

Also the validator passes for URLs like "你好", and does not pass for URLs with characters like spaces "a b"

Related: Which characters make a URL invalid?

@Utku 2016-08-08 04:53:29

But both URLs ("你好" and "a b") have to be percent encoded when making the HTTP request right?

@Ciro Santilli 郝海东冠状病六四事件法轮功 2016-08-08 07:00:07

@Utku for "a b" I'm pretty sure yes since space is not in the allowed list above. For "你好", it is definitely the better idea to percent encode, but I don't know if it is just a question of "the implementations are not good enough" or the "standard says so". The HTML standard seems to allows those characters. But I think this is specified by the HTTP standard, not HTML. See also:…

@Utku 2016-08-08 07:51:08

Yes, I was thinking of the HTTP standard, not HTML.

@EKons 2015-11-18 13:11:03

Use percent-encoded form. Some (mainly old) computers running Windows XP for example do not support Unicode, but rather ISO encodings. That is the reason percent-encoded URLs were invented. Also, if you give a URL printed on paper to a user, containing characters that cannot be easily typed, that user may have a hard time typing it (or just ignore it). Percent-encoded form can even be used in many of the oldest machines that ever existed (although they don't support internet of course).

There is a downside though, as percent-encoded characters are longer than the original ones, thus possibly resulting in really long URLs. But just try to ignore it, or use a URL shortener (I would recommend in this case, which makes a 13-character long URL). Also, if you don't want to register for a Google account, try ( makes slightly longer URLs, with the length being 14 characters).

@Mateus Felipe 2020-04-16 12:44:52

Why would I want to support obsolete computers that still uses Windows XP?

@Dean Harding 2010-04-30 07:29:06

Depending on your URL scheme, you can make the UTF-8 encoded part "not important". For example, if you look at Stack Overflow URLs, they're of the following form:

However, the server doesn't actually care if you get the part after the identifier wrong, so this also works:これは、これを日本語のテキストです

So if you had a layout like this, then you could potentially use UTF-8 in the part after the identifier and it wouldn't really matter if it got garbled. Of course this probably only works in somewhat specialised circumstances...

@Pekka 2010-04-30 07:34:08

Hmmm, very clever thinking! It could still be that some clients choke on the characters no matter where they are located in the string, but it would eliminate all the problems with ordinary garbling when copy+pasting a URL, which I think is the most important part. Hadn't looked at SO's URL that way yet. Thanks!

@Evgeny 2010-04-30 16:39:05

well, this still leaves word "questions" untranslated, plus there is stuff after hash #, which follows entire url, very nice trick though!!

@Glutexo 2016-08-12 11:58:20


@Peter Manoukian 2014-09-03 08:44:47

For me this is the correct way, This just worked:

    $linker = rawurldecode("$link");
    <a href="<?php echo $link;?>"   target="_blank"><?php echo $linker ;?></a>

This worked, and now links are displayed properly:معرض--جوزف-حرب-في-غاليري-جانين-ربيز-لوحاته-الجدية-تبحث-وتكتشف-وتفرض-الاحترام

Link found on:

@MrWhite 2015-10-30 12:18:55

"links are displayed properly" - except that the StackOverflow markdown parser doesn't interpret URLs as intended!

@Nasser Hadjloo 2010-05-03 07:30:44

As all of these comments are true, you should note that as far as ICANN approved Arabic (Persian) and Chinese characters to be registered as Domain Name, all of the browser-making companies (Microsoft, Mozilla, Apple, etc.) have to support Unicode in URLs without any encoding, and those should be searchable by Google, etc.

So this issue will resolve ASAP.

@Pekka 2010-05-03 07:33:42

@Nasser: True - we have special characters in german domains now, too - but those are encoded into ASCII characters using Punycode. While they are sure to work in major browsers, it will be a long time before every HTTP client library and exotic application will be able to deal with unencoded Unicode characters.

@Nasser Hadjloo 2010-05-03 07:44:05

@Pekka, I'm not sure but as I heard, all of browsers have to support Unicode URL at 4th quarter of 2010. (I'm Not Sure)

@Cornelius 2014-01-23 12:41:33

The issue is complicated by the fact that not every user agent is a web browser. Largest example is google itself: It does not use common web browsers to do it's crawling. So would many libraries for API interaction etc. etc. — URLs are almost literally everywhere, not just in the WWW. Probably even on your file system right now.

@bobince 2010-04-30 11:37:01

What Tgr said. Background:üsseldorf?neighbourhood=Lörick

That's not a URI. But it is an IRI.

You can't include an IRI in an HTML4 document; the type of attributes like href is defined as URI and not IRI. Some browsers will handle an IRI here anyway, but it's not really a good idea.

To encode an IRI into a URI, take the path and query parts, UTF-8-encode them then percent-encode the non-ASCII bytes:

If there are non-ASCII characters in the hostname part of the IRI, eg. http://例え.テスト/, they have be encoded using Punycode instead.

Now you have a URI. It's an ugly URI. But most browsers will hide that for you: copy and paste it into the address bar or follow it in a link and you'll see it displayed with the original Unicode characters. Wikipedia have been using this for years, eg.:ɸ

The one browser whose behaviour is unpredictable and doesn't always display the pretty IRI version is...

...well, you know.

@Pekka 2010-04-30 11:48:34

I know. One day, somebody has to take a big club and smack those Lynx developers on the head. Thanks for the excellent background info.

@Tom Harrison 2013-05-28 20:54:28

@bobince And the one bot (fast forward to 2013) that also cannot handle non IRI URIs is... ...well, you know: bingbot! Go figure.

@Oliver 2013-10-25 12:46:02

HTML5 finally supports IRIs. More info on the subject can be found in this answer to a related question.

@codingoutloud 2014-02-15 14:40:28

Re: IE not always displaying pretty IRIs - they are protecting users from homograph-based phishing attacks. Check out (specifically the section 'Domain names-and phishing') and

@Tgr 2015-07-04 00:38:56

Domain names have nothing to do with this. All browsers disallow a wide range of characters to prevent phishing. Displaying non-ASCII characters in the path or query string part does not create a similar vilnerability. IE simply didn't bother to implement it. (And Firefox is the only one that implemented it for the fragment part as well.)

@Tgr 2010-04-30 07:39:22

Use percent encoding. Modern browsers will take care of display & paste issues and make it human-readable. E. g.위키백과:대문

Edit: when you copy such an url in Firefox, the clipboard will hold the percent-encoded form (which is usually a good thing), but if you copy only a part of it, it will remain unencoded.

@Dean Harding 2010-04-30 07:44:02

Wow, actually you're right! If you cut'n'paste a %-encoded URL Firefox will turn it into the correct thing for display.

@Pekka 2010-04-30 07:45:28

Wow, I wasn't aware of this. Chances are this is the best solution!

@Roman Starkov 2011-01-09 14:48:10

@Dean that's a fairly recent change - in 2005 all international wikipedias looked like a real %6D%65%73%73.

@Oliver 2013-10-23 12:54:34

You can use the unencoded UTF-8 URLs, namely IRIs, in HTML5 documents by now. If you do that, all major browsers will understand it and display it correctly in their address bar.

@Flimm 2015-09-11 16:34:39

What bytes do modern browsers send to to servers in the request line GET /images/logo.png HTTP/1.1? Do they always percent-encode the URL?

@Tgr 2015-09-11 17:50:49

RfC 3986 has the details but basically alphanumeric and _.- are never encoded, !$'*,:[email protected] might or might not be encoded (they don't need to be but some implementations do it anyway), /?#[]&=+ might or might not be encoded and (depending on which part of the URL it happens in) it might change the meaning of the URI when they are (e.g. a web server might interpret images/logo.png as the logo.png file in the images directory and images%2Flogo.png as a file called images/logo.png in the root directory), everything else should always be encoded.

@Tgr 2015-09-11 17:52:05

Browsers usually get the URI from the link you click on so they don't need to encode anything, but if you type the address in manually, I believe they usually do the least amount of encoding that's necessary.

@Neme 2017-08-28 13:57:21

Is there a way to guess if a browser supports this human-readable form to fall back to de-accented URLs for browsers that don't?

@Tgr 2017-08-28 21:21:10

Probably not short of looking it up in some kind of browser support database, but pretty much everything supports it these days. (Percent-encoded URL fragments are a different matter altogether, though.)

@palswim 2018-11-05 21:40:24

To clarify, the URL in the answer would look like:‌​B3%BC:%EB%8C%80%EB%A‌​C%B8

Related Questions

Sponsored Content

24 Answered Questions

[SOLVED] Get the current URL with JavaScript?

  • 2009-06-23 19:26:45
  • dougoftheabaci
  • 2843607 View
  • 3027 Score
  • 24 Answer
  • Tags:   javascript url

18 Answered Questions

[SOLVED] What is the maximum length of a URL in different browsers?

  • 2009-01-06 16:14:30
  • Sander Versluys
  • 1265212 View
  • 4870 Score
  • 18 Answer
  • Tags:   http url browser

40 Answered Questions

[SOLVED] Path.Combine for URLs?

  • 2008-12-16 21:42:26
  • Brian MacKay
  • 317044 View
  • 1255 Score
  • 40 Answer
  • Tags:   c# .net url path

18 Answered Questions

[SOLVED] Encode URL in JavaScript?

17 Answered Questions

[SOLVED] How do I modify the URL without reloading the page?

23 Answered Questions

[SOLVED] How to change the URI (URL) for a remote Git repository?

  • 2010-03-12 12:48:47
  • e-satis
  • 1675821 View
  • 3960 Score
  • 23 Answer
  • Tags:   git url git-remote

32 Answered Questions

[SOLVED] What is the difference between a URI, a URL and a URN?

  • 2008-10-06 21:26:58
  • Sean McMains
  • 1152818 View
  • 4398 Score
  • 32 Answer
  • Tags:   http url uri urn rfc3986

32 Answered Questions

[SOLVED] Get current URL with jQuery?

  • 2009-01-02 06:42:54
  • venkatachalam
  • 2160895 View
  • 1842 Score
  • 32 Answer
  • Tags:   javascript jquery url

19 Answered Questions

[SOLVED] Is it possible to apply CSS to half of a character?

  • 2014-05-09 16:16:57
  • Mathew MacLean
  • 243286 View
  • 2830 Score
  • 19 Answer
  • Tags:   javascript html css

7 Answered Questions

Sponsored Content