Nick >Interesting – how does it 'break' such things? A stray BOM in the page should just be interpreted as a ZWNJ. If it were in the middle of a long word that happened to fall near the end of a line, you’d get an unexpected word break, but I don’t see how that could be thought of as being particularly 'broken'. Simple Programming Language For Beginners. Moreover, I don’t see how you’d end up with two parts of a word in different files. Or does it output the 'no glyph' box symbol for ZWNJs? Can you expand on this?

UTF stands for “Universal character set Transformation Format”. Usually a UTF text file is identical to an ASCII text file. ASCII is the standard that most computers have used up until now. A file created with NotePad is an ASCII text file. However, a UTF text file can include non-ASCII characters such as Cyrillic, Greek, and. ASCII chars have code-points between 0 and 127 (included), so anything higher is certainly not an ASCII character. Computer Programming. While it's certainly possible that the text file could be UTF-16 or one 32-bit Unicode character at a time, these have the disadvantage that they are two and four times the size of.

But, the general problem is one of the main reasons why BOMs are discouraged. It breaks the semantics of being able to concatenate two text files together simply by appending one stream of bytes to another. You now have an extra character in the resultant stream that was not 'in' the original files. Pcooper — True, but depending on the dialog, they might not have to know anything about character encodings to answer it, either. If it’s possible to narrow the (large) number of possible encodings down to a smaller number (say two to five), it may be possible to design a dialog that would allow the user to choose an encoding from this smaller list. Show them a preview of the text as it would display under the currently-selected encoding; then they can just switch between encodings until they find one that makes the text look right.