When you see issues with Character Encoding, it’s traditionally in the form of text on your page that looks like this: in Firefox or in IE.

Usually, those characters mean that the character encoding used on the page is either ambiguous (not specified), or incorrect. We can use Firefox to determine that Character Encoding of a web page (Right Click and go to View Page Info; or use the “Character Encoding” entry in the View menu). Check to make sure that the encoding reported by Firefox is the same encoding used in your IDE. For example, Eclipse 3.5 has a “Set Encoding” option in the Edit menu.

The reason most English alphabetic and numeric characters are consistent independent of character encoding is due to consistency in the lower characters in each encoding. The characters making up the ASCII character set (0-127) are the same as the lowest 128 characters of ISO-8859-1, UTF-8, and others.

Managing your character encodings gets trickier as you add more architectural layers to your application. For example, character encodings may differ in your database, the properties files used to configure your application (java.util.Properties uses ISO-8859-1 by default), or maybe the XML or JSON file you’re loading from an external API.

Ever heard of HTML character entities? That’s the primary reason they exist — as a sort of encoding independent reference to a particular character. So, for example, the Œ character does not exist in the ISO-8859-1 character set. To display this character in a document with ISO-8859-1 encoding, use the equivalent HTML character entity: Œ. For an easier reference, check out this full table of HTML character entities. If using ISO-8859-1 for your HTML document, any entity above Unicode index 255 will need to be escaped. If you’re using UTF-8 encoding, HTML character entities shouldn’t be required.

Setting the Character Encoding

To specify the character encoding for any file, you can set a Content-Type header by configuring your web server or application. Apache lets you easily set different default character encodings for each individual file extension (.js for example). Using the Content-Type header is the most full proof and efficient ¹ method to serve content.

But without access to the Apache configuration, how do we specify the character encoding?

For external JavaScript Files

In the HTML File

Just add the charset attribute. If not specified, the HTML document’s character encoding is used by default (specified in the Content-Type header or the appropriate meta tag, for example: <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>).

<script type="text/javascript" src="script.js" charset="utf-8"></script>

In the JavaScript File

To my knowledge, there is no way for a JavaScript file to report its own character encoding. To me, this seems like an omission. Each individual document should be able to report its own character encoding without a header. CSS files can do it (@charset at-rule). HTML files can do it (<meta> tag). Why not JavaScript files?

For Dynamically Created Script Tags

var s = document.createElement('script');
s.src = 'script.js';
s.type = 'text/javascript';
s.charset = 'utf-8';

If you’re using jQuery’s Ajax functions to load external JavaScript files, perhaps you might be inclined to use the script dataType. jQuery even provides a scriptCharset option for wrapping the above method for changing the charset on a dynamic script tag. Be warned, the jQuery Ajax function uses two different methods to load external script files (as of version 1.4.2). If a same-domain request, it uses an XMLHttpRequest. If a cross-domain request, it uses a dynamic script tag. So the scriptCharset jQuery option only applies to cross-domain requests. We’ll need some other method to mitigate our character encoding issues (or just use dynamic script tags).

For XMLHttpRequest Objects

Our saving grace would be the overrideMimeType method, if it weren’t poetically unavailable in Internet Explorer. Using this method, we can override the mime type and character encoding.

var xhr = new XMLHttpRequest();
// Not available in Internet Explorer (up to version 8 at time of writing)
if (xhr.overrideMimeType) {
    xhr.overrideMimeType('application/x-javascript; charset=utf-8');
}

Portable non-ASCII JavaScript

The best way to make non-ASCII characters in JavaScript files portable is to escape the characters properly. If the character is destined for HTML, use an HTML character entity (if available, not all Unicode or ISO-8559-1 characters have entities). Or, escape the characters using the proper Latin or Unicode escape sequence.

// Raw characters
var string = "ñó";

// HTML character entities
var string = "ñó";

// Escaped to Latin
var string = "\xf1\xf3";

// Escaped to Unicode
var string = "\u00f1\u00f3";

If you use the Google Closure Compiler, you’ll get the Unicode escape sequences for free (see issues 24 and 68). Make sure to read the tickets for more benefits of serving your JavaScript files using Unicode escape sequences to output only ASCII characters.

Summary

The easiest way to preemptively solve a lot of character encoding issues is to use UTF-8 for everything, and configure your web server/application to serve the UTF-8 Content-Type header. If you’re writing JavaScript code that you’re going to distribute to the masses, convert any non-ASCII characters using the proper escape sequences. Your JavaScript will be more portable, and will work out of the box on more server configurations.

Sources

1. Performance Implications of Charset, an article by Kyle Scholz

Generally when exploring a development opportunity, I tend to prioritize my adventures towards things that are supported in Internet Explorer first. This often has the biggest cross-browser payoff, since the other browser vendors tend to have a quicker draw than the Microsoft Team. However, surprisingly enough, this writing-mode study proved the opposite to be true. It seems very interesting that Microsoft has decided to implement a portion of the CSS 3 specification, given its general stance of moving slower than an iceberg to avoid “breaking the web.” But I, for one, welcome our new choose-your-own-adventure standards loving overlords.

As far as my tests go, the only browser to support the writing-mode property at all is Internet Explorer, which was very surprising. At it’s heart, though, writing-mode is just shorthand for two other properties: direction and block-progression. Luckily, Firefox, Safari, Chrome and IE back through at least version 6 support the direction property and have proprietary options for rotation, which allows for emulation of a few of the unsupported writing-mode‘s, but not all of them. The missing piece of writing-mode emulation belongs to the block-progression property, which isn’t supported by anyone, and would allow elements to flow reverse vertically (start at the bottom of a block and flow upwards).

It’s important to note that while IE8 has really set the bar for implementation here and has chosen to support writing-mode: lr-bt and writing-mode: rl-bt, they aren’t used to display any known language text. They’re just included for completeness, and aren’t a part of the W3C CSS 3 Text Module specification.

View the Demo / Test Page

Compatibility Table

CSS Code for Emulation

// Default

direction: rtl;

// Not possible using W3C spec

// Not possible using W3C spec

// Not possible using W3C spec

-webkit-transform: rotate(90deg);
-moz-transform: rotate(90deg);
filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=1);

// Not possible using W3C spec

direction: rtl;
-webkit-transform: rotate(90deg);
-moz-transform: rotate(90deg);
filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=1);

Conclusion

Given that 4 of the 6 known writing modes are available or available through CSS emulation means we’re in pretty good shape on the internationalization front. Consulting the Microsoft provided table for common use cases, we’re only in trouble when trying to use the “Mongolian script writing system” and an “Arabic script block quote embedded in Mongolian script document.”

In some far fetched fantasy-world legacy application where a page may use tables for layout, I could see an application team possibly using the direction property to redistribute the tables for a print stylesheet. But that certainly wouldn’t be a common use case, since using CSS for layouts is going to give you much more flexibility in that regard. If you can think of any other off the wall uses for writing-mode or direction, I’d love to hear them!

Related

• Very complete article on CSS 3 writing-mode, including some direction properties that don’t exist in the specification (like ttb, ltr-ttb, and ltr-btt)
• Bugzilla Bug for writing-mode in Firefox
• As of the time of this writing, I was unable to find any results for writing-mode on the Webkit bug tracker.

We must be resilient to resist the transformation from development pragmatism to user abandonment, especially considering a developer’s trend of being insulated with the latest equipment. We use the newest hardware and download the most sophisticated browser software because we want the best experiences. We stack the cards in our favor. The difficulty lies with the populace that want a good experience (they can tell a fast site from a slow site), but they don’t want to buy new equipment or install a better browser. So the question has become: Where do you place your chips on the spectrum balancing modern hardware/software and the user experience?

As web developers, it has become increasingly easy to become disenfranchised with the lay majority’s computing environment. I’ve witnessed this personally after purchasing my first Apple computer, a device curiously immune to the plague of IE 6/7 (8 as well, but I would hesitate to label it as a plague, more like a normal run of the mill flesh eating virus). It’s an interesting phenomenon, being cutoff from the Windows environment (save virtualization) during development. It’s delightfully tempting to declare damnation upon the plague ridden Internet Explorer Zombies, and develop your web presence to standards alone, allowing less focus on the harshest development environment imaginable and more on actual problem solving.

When developers engage in quasi-web development, perhaps in the form of an iPhone specific web application or an Adobe AIR desktop application, it fosters similar feelings. The work shares many languages and technologies with real (wide-open) web development, but is targeted towards a single web browser. This benefits of this sort of tunnel vision device targeting must be carefully weighed against the penalties you’re paying in narrowed default-install client compatibility. At that point, it’s just friends with benefits.

Google Wave, in fact, requires the use of Safari, Firefox, Chrome, or the Google Chrome Frame plugin (which doesn’t count as actual browser support). As of the time this article was written, as many as 64% of all users (source) are met with the above screen when trying to access Google Wave. Let’s be clear: Ease of development is not an excuse to abandon users. Whatever happened to Device Independence?

Not surprisingly, Google Wave’s stand against Internet Explorer has not started a trend. Instead of all out war, the community at large has decided that subtlety is what’s needed to win this fight. Let’s hold back the sexual healing of rounded corners, even though perhaps users will call our design harsh or cold. Let’s give them plain buttons instead of nice three dimensional buttons with text and box shadows, even though they may be less actionable.

Let’s make it easier on ourselves at the expense of the user experience of The Zombie Majority. We make this sacrifice because we assume that Zombies won’t be able to tell the difference between eating a big juicy Einstein brain and a smaller but nonetheless still tasty brush clearing George W. And you know what, I think we’re right.

Updated to include a few more thoughts on Adobe AIR and the iPhone.

• Prototype and Dojo are the only major JavaScript frameworks that correctly time the DOMContentLoaded inside of an iframe in Internet Explorer. MooTools comes close and both YUI and jQuery both exhibit incorrect behavior.
• Usually, browsers that have a native DOMContentLoaded event will fire it after both JavaScript and CSS (external script, link tags) have loaded. But Opera fires prior to CSS link tags loading.

We all know the problem, but we may not all call it the same thing. Every popular JavaScript library has its own name for the DOMContentLoaded event.

But the naming scheme isn’t the only part about this event that’s inconsistent. There are a few hacks strewn around the net that have been bastardized (read: modified to the will of library authors) into each individual framework. These have been posted time and time again, but for post completeness are the following:

Benchmark for Library DOMContentLoaded Implementations

I created a benchmark trying to answer the simple question: when does the DOMContentLoaded event fire? Each implementation has different effects, especially in iframes and with stylesheets.

What Works

The Internet Explorer hacks presented above are reliable when used on a standalone document (not inside of an iframe).

Mozilla Firefox, Safari, Chrome, and Opera all contain a native DOMContentLoaded event that each framework uses to consistently fire at the correct time. So, it isn’t even worth summarizing the results for tests in every browser with a native DOMContentLoaded event, except Opera. Opera took a different design approach with their DOMContentLoaded event (certainly not wrong, just different), which we’ll analyze below.

Internet Explorer inside an Iframe

All libraries below did no branching based on IE versions, all used the same method for IE6, IE7, or IE8. I performed no tests in IE8, however.

* For some reason, not only did the MooTools library not wait for the body script to trigger DOMContentLoaded, the body script never executed at all.

The most important takeaway from this blog post are the successful methods of faking DOMContentLoaded in Internet Explorer. The obvious library winners here are Prototype and Dojo, with the addDOMLoadEvent script also exhibiting correct behavior. MooTools is a close runner up, and I would be interested to find out why it didn’t execute the body script (that’s left for another day, or perhaps a generous commenter).

Opera 9.6

As you can see above, Opera typically will fire DOMContentLoaded prior to stylesheets loading successfully (ignoring the jQuery specific code to provide consistency cross-browser)

Thanks to Adam Koch for first pointing me in the direction of jQuery in an iframe ignoring DOMContentLoaded in IE.

Update: There is an open ticket for YUI on Sourceforge, and I’ve opened a ticket for jQuery on their Trac site.

The most common usage of a relative URL is linked from the root (note the slash at the beginning), like so:

<img src=”/web/wp-content/themes/hemingway/images/title.png“/>

You can also use ../ to navigate up a directory in your path, but that’s boring. The interesting question is: what if I wanted to load content from a separate domain, while at the same time transparently using the protocol used on my page?

Do what Google Calendar does and use the following style:

<img src=”//calendar.google.com/googlecalendar/images/calendar_sm2_en.gif“/>

Note the lack of http: or https: from the URL. If this page were hosted on https (we don’t pay for certificates around here, so you’ll have to put up with the security warning), the last image source will load from https as well at no additional development cost.

This seems like it would be especially useful if you were using the YDN Performance tip to Split Components Across Domains. Remember, we’re not just talking images. This involves how the browser resolves URLs, so could be used inside href tags, css urls (as Google Calendar did), etc.

Another use for the onbeforeunload event to allow the user to cancel the action that initiated the user leaving in the first place. Gmail uses this technique when the user is in the middle of writing a draft of an e-mail and attempts to leave the page.

Gmail’s pops up this prompt when the user attempts to leave the page while drafting an email.

Worthy to note, however, is that Opera doesn’t fire the unload event when the browser refreshes the page, or uses the back/forward buttons to browse off of the page (I had no success with the fix posted in the comments on that page). What’s worse, Opera never fires the onbeforeunload event. This creates a serious problem with attempting to save page state prior to a user leaving your page.

Browser support aside, I believe that the onbeforeunload prompt is not an ideal way to protect the user from lost work (or unsaved page state). Humanized has argued, and I agree, that an undo operation is much easier on the end user than a warning message. The strange thing is, Gmail could save the draft in a synchronous Ajax request in the onunload event. They aren’t using the prompt to save Opera users from losing their drafts, since the Opera web browser doesn’t even fire the onbeforeunload event. (Interestingly enough, they are using some sort of browser history management to fire a warning to the user when they press back, or forward, in Opera — but Reload can’t be caught using this method, so your could draft email be lost).

From a User Interface design standpoint, I would recommend just sticking with onunload. You can still perform your synchronous Ajax call in the method to save the state of your page, so that the user can later resume their state or undo the operation. (Except for Back/Forward/Refresh in Opera, until they support a better onunload or any onbeforeunload). The onbeforeunload prompt is an unnecessary evil, and doesn’t do much besides annoy the end user with another warning message and a mouse click.

Use the Firefox Plug-In called (humorously enough) JohnnyCache. I added the pattern ‘http://localhost/’ to my preferences and now I don’t have to worry about cache on my local machine anymore. Simple, easy, effective.

Now only if I knew how to disable the Firefox preference that autopopulates form fields with default values when F5 is hit to refresh the page. Try creating a page with a single text field, type a value into the form, and then hit F5 to refresh the page. Your value will be prefilled into the form.

I find myself always hitting “CTRL+L” (goes to the Location Bar) and “Enter” to avoid that one. I looked through about:config, but only found: signon.prefillForms (related to usernames and passwords), and browser.formfill.enable (related to autocomplete dropdowns for forms, not default values).

Anyone know an easier method?

var d = new Date();
var r = [];
for(var j=0,k=2;j<k ;j++) {
    window.setInterval(function()
    {
        var next = new Date();
        r.push(next.getTime() - d.getTime());
    },1000);
}

Every 1000 milliseconds (1 second), the r array should have three delta timestamps added into it (one for each interval set in the for loop). In FireFox, however, there are exactly nine delta timestamps added per second. If you set k=10, the result is one hundred delta timestamps per second. If anyone has the answer, I am curious, even if the result is that I'm an idiot. It seems to work correctly in Internet Explorer 7. Thanks for your help, internet.