Newline

Newline (frequently called line ending, end of line (EOL), line feed, or line break) is a control character or sequence of control characters in a character encoding specification (e.g. ASCII or EBCDIC) that is used to signify the end of a line of text and the start of a new one. Text editors set this special character when pressing the Enter key.

When displaying (or printing) a text file, this control character causes the text editor to show the following characters in a new line.

Illustration of a newline
Newline inserted between the words "Hello" and "world"

History

In the mid-1800s, long before the advent of teleprinters and teletype machines, Morse code operators or telegraphists invented and used Morse code prosigns to encode white space text formatting in formal written text messages. In particular the Morse prosign represented by the concatenation of two literal textual Morse code "A" characters sent without the normal inter-character spacing is used in Morse code to encode and indicate a new line in a formal text message.

Later in the age of modern teleprinters standardized character set control codes were developed to aid in white space text formatting. ASCII was developed simultaneously by the International Organization for Standardization (ISO) and the American Standards Association (ASA), the latter being the predecessor organization to American National Standards Institute (ANSI). During the period of 1963 to 1968, the ISO draft standards supported the use of either CR+LF or LF alone as a newline, while the ASA drafts supported only CR+LF.

The sequence CR+LF was commonly used on many early computer systems that had adopted Teletype machines—typically a Teletype Model 33 ASR—as a console device, because this sequence was required to position those printers at the start of a new line. The separation of newline into two functions concealed the fact that the print head could not return from the far right to the beginning of the next line in time to print the next character. Any character printed after a CR would often print as a smudge in the middle of the page while the print head was still moving the carriage back to the first position. "The solution was to make the newline two characters: CR to move the carriage to column one, and LF to move the paper up."[1] In fact, it was often necessary to send extra characters—extraneous CRs or NULs—which are ignored but give the print head time to move to the left margin. Many early video displays also required multiple character times to scroll the display.

On such systems, applications had to talk directly to the Teletype machine and follow its conventions since the concept of device drivers hiding such hardware details from the application was not yet well developed. Therefore, text was routinely composed to satisfy the needs of Teletype machines. Most minicomputer systems from DEC used this convention. CP/M also used it in order to print on the same terminals that minicomputers used. From there MS-DOS (1981) adopted CP/M's CR+LF in order to be compatible, and this convention was inherited by Microsoft's later Windows operating system.

The Multics operating system began development in 1964 and used LF alone as its newline. Multics used a device driver to translate this character to whatever sequence a printer needed (including extra padding characters), and the single byte was more convenient for programming. What seems like a more obvious choice—CR—was not used, as CR provided the useful function of overprinting one line with another to create boldface and strikethrough effects. Perhaps more importantly, the use of LF alone as a line terminator had already been incorporated into drafts of the eventual ISO/IEC 646 standard. Unix followed the Multics practice, and later Unix-like systems followed Unix.

Representation

The concepts of line feed (LF) and carriage return (CR) are closely associated, and can be considered either separately or together. In the physical media of typewriters and printers, two axes of motion, "down" and "across", are needed to create a new line on the page. Although the design of a machine (typewriter or printer) must consider them separately, the abstract logic of software can combine them together as one event. This is why a newline in character encoding can be defined as LF and CR combined into one (commonly called CR+LF or CRLF).

Some character sets provide a separate newline character code. EBCDIC, for example, provides an NL character code in addition to the CR and LF codes. Unicode, in addition to providing the ASCII CR and LF control codes, also provides a "next line" (NEL) control code, as well as control codes for "line separator" and "paragraph separator" markers.

Software applications and operating systems usually represent a newline with one or two control characters:

Operating system Character encoding Abbreviation hex value dec value Escape sequence
Multics, Unix and Unix-like systems (Linux, macOS, FreeBSD, AIX, Xenix, etc.), BeOS, Amiga, RISC OS, and others[2] ASCII LF 0A 10 \n
Atari TOS, Microsoft Windows, DOS (MS-DOS, PC DOS, etc.), DEC TOPS-10, RT-11, CP/M, MP/M, OS/2, Symbian OS, Palm OS, Amstrad CPC, and most other early non-Unix and non-IBM operating systems CR LF 0D 0A 13 10 \r\n

Commodore 8-bit machines (C64, C128), Acorn BBC, ZX Spectrum, TRS-80, Apple II family, Oberon, the classic Mac OS, MIT Lisp Machine and OS-9

CR 0D 13 \r
QNX pre-POSIX implementation (version < 4) RS 1E 30
Acorn BBC[3] and RISC OS spooled text output.[4] LF+CR 0A 0D 10 13 \n\r
Atari 8-bit machines ATASCII 9B 155
IBM mainframe systems, including z/OS (OS/390) and i5/OS (OS/400) EBCDIC NL 15 21 \025
ZX80 and ZX81 (Home computers from Sinclair Research Ltd) used a specific non-ASCII character set NEWLINE 76 118
  • EBCDIC systems—mainly IBM mainframe systems, including z/OS (OS/390) and i5/OS (OS/400)—use NL (New Line, 0x15)[5] as the character combining the functions of line-feed and carriage-return. The equivalent UNICODE character(0x85) is called NEL (Next Line). EBCDIC also has control characters called CR and LF, but the numerical value of LF (0x25) differs from the one used by ASCII (0x0A). Additionally, some EBCDIC variants also use NL but assign a different numeric code to the character. However, those operating systems use a record-based file system, which stores text files as one record per line. In most file formats, no line terminators are actually stored.
  • Operating systems for the CDC 6000 series defined a newline as two or more zero-valued six-bit characters at the end of a 60-bit word. Some configurations also defined a zero-valued character as a colon character, with the result that multiple colons could be interpreted as a newline depending on position.
  • RSX-11 and OpenVMS also use a record-based file system, which stores text files as one record per line. In most file formats, no line terminators are actually stored, but the Record Management Services facility can transparently add a terminator to each line when it is retrieved by an application. The records themselves could contain the same line terminator characters, which could either be considered a feature or a nuisance depending on the application. RMS not only stored records, but also stored metadata about the record separators in different bits for the file to complicate matters even more (since files could have fixed length records, records that were prefixed by a count or records that were terminated by a specific character). The bits weren't generic, so while they could specify that CRLF or LF or even CR was the line terminator, it couldn't substitute some other code.
  • Fixed line length was used by some early mainframe operating systems. In such a system, an implicit end-of-line was assumed every 72 or 80 characters, for example. No newline character was stored. If a file was imported from the outside world, lines shorter than the line length had to be padded with spaces, while lines longer than the line length had to be truncated. This mimicked the use of punched cards, on which each line was stored on a separate card, usually with 80 columns on each card, often with sequence numbers in columns 73–80. Many of these systems added a carriage control character to the start of the next record; this could indicate whether the next record was a continuation of the line started by the previous record, or a new line, or should overprint the previous line (similar to a CR). Often this was a normal printing character such as "#" that thus could not be used as the first character in a line. Some early line printers interpreted these characters directly in the records sent to them.

Unicode

The Unicode standard defines a number of characters that conforming applications should recognize as line terminators:[6]

 LF:    Line Feed, U+000A
 VT:    Vertical Tab, U+000B
 FF:    Form Feed, U+000C
 CR:    Carriage Return, U+000D
 CR+LF: CR (U+000D) followed by LF (U+000A)
 NEL:   Next Line, U+0085
 LS:    Line Separator, U+2028
 PS:    Paragraph Separator, U+2029

This may seem overly complicated compared to an approach such as converting all line terminators to a single character, for example LF. However, Unicode was designed to preserve all information when converting a text file from any existing encoding to Unicode and back. Therefore, Unicode should contain characters included in existing encodings. NEL is included in EBCDIC with code (0x15). NEL is also a control character in the C1 control set.[7] As such, it is defined by ECMA 48,[8] and recognized by encodings compliant with ISO/IEC 2022 (which is equivalent to ECMA 35).[9] C1 control set is also compatible with ISO-8859-1. The approach taken in the Unicode standard allows round-trip transformation to be information-preserving while still enabling applications to recognize all possible types of line terminators.

Recognizing and using the newline codes greater than 0x7F (NEL, LS and PS) is not often done. They are multiple bytes in UTF-8, and the code for NEL has been used as the ellipsis ('…') character in Windows-1252. For instance:

  • ECMAScript[10] accepts LS and PS as line breaks, but considers U+0085 (NEL) white space, not a line break.
  • Windows 10 does not treat any of NEL, LS, or PS as line-break in the default text editor Notepad
  • On Linux, a popular editor, gedit, treats LS and PS as newlines but does not for NEL.
  • YAML[11] no longer recognizes them as special, in order to be compatible with JSON.
  • JSON treats LS and PS inside a String as letter while ECMAScript/Javascript treats them as new line and shows an error

The Unicode characters U+2424 (SYMBOL FOR NEWLINE, ␤), U+23CE (RETURN SYMBOL, ⏎), U+240D (SYMBOL FOR CARRIAGE RETURN, ␍) and U+240A (SYMBOL FOR LINE FEED, ␊) are intended for presenting a user-visible character to the reader of the document, and are thus not recognized themselves as a newline.

Escape sequences

An escape sequence is a combination of characters which represents no text; instead of being displayed (as text) it is supposed to be intercepted by the program and a special function is supposed to be performed. Escape sequences are also used to handle (set, search, replace, etc.) special characters.

Special character Escape sequence Used by … Examples
line feed \n Perl, Vim, … Vim: :%s/}/}\r\t/g = replace each '}' character with '} newline tabulator' in the entire file
carriage return \r
tabulator \t

In programming languages

To facilitate the creation of portable programs, programming languages provide some abstractions to deal with the different types of newline sequences used in different environments.

The C programming language provides the escape sequences '\n' (newline) and '\r' (carriage return). However, these are not required to be equivalent to the ASCII LF and CR control characters. The C standard only guarantees two things:

  1. Each of these escape sequences maps to a unique implementation-defined number that can be stored in a single char value.
  2. When writing to a file, device node, or socket/fifo in text mode, '\n' is transparently translated to the native newline sequence used by the system, which may be longer than one character. When reading in text mode, the native newline sequence is translated back to '\n'. In binary mode, no translation is performed, and the internal representation produced by '\n' is output directly.

On Unix platforms, where C originated, the native newline sequence is ASCII LF (0x0A), so '\n' was simply defined to be that value. With the internal and external representation being identical, the translation performed in text mode is a no-op, and Unix has no notion of text mode or binary mode. This has caused many programmers who developed their software on Unix systems simply to ignore the distinction completely, resulting in code that is not portable to different platforms.

The C library function fgets() is best avoided in binary mode because any file not written with the Unix newline convention will be misread. Also, in text mode, any file not written with the system's native newline sequence (such as a file created on a Unix system, then copied to a Windows system) will be misread as well.

Another common problem is the use of '\n' when communicating using an Internet protocol that mandates the use of ASCII CR+LF for ending lines. Writing '\n' to a text mode stream works correctly on Windows systems, but produces only LF on Unix, and something completely different on more exotic systems. Using "\r\n" in binary mode is slightly better.

Many languages, such as C++, Perl,[12] and Haskell provide the same interpretation of '\n' as C. C++ has an alternative I/O model where the manipulator std::endl can be used to output a newline (and flushes the stream buffer).

Java, PHP,[13] and Python[14] provide the '\r\n' sequence (for ASCII CR+LF). In contrast to C, these are guaranteed to represent the values U+000D and U+000A, respectively.

The Java I/O libraries do not transparently translate these into platform-dependent newline sequences on input or output. Instead, they provide functions for writing a full line that automatically add the native newline sequence, and functions for reading lines that accept any of CR, LF, or CR+LF as a line terminator (see BufferedReader.readLine()). The System.lineSeparator() method can be used to retrieve the underlying line separator.

Example:

   String eol = System.lineSeparator();
   String lineColor = "Color: Red" + eol;

Python permits "Universal Newline Support" when opening a file for reading, when importing modules, and when executing a file.[15]

Some languages have created special variables, constants, and subroutines to facilitate newlines during program execution. In some languages such as PHP and Perl, double quotes are required to perform escape substitution for all escape sequences, including '\n' and '\r'. In PHP, to avoid portability problems, newline sequences should be issued using the PHP_EOL constant.[16]

Example in C#:

   string eol = Environment.NewLine;
   string lineColor = "Color: Red" + eol;
   
   string eol2 = "\n";
   string lineColor2 = "Color: Blue" + eol2;

Issues with different newline formats

Newline hex 0A
A text file created with gedit and viewed with a hex editor. Besides the text objects, there are only EOL markers with the hexadecimal value 0A.

Even though the control characters are unambiguously defined in the corresponding character encoding table used by a text file, there still is an issue: there are different conventions to set and display a line break.

To denote a single line break, Unix programs use line feed, whose hexadecimal value in ASCII is 0a, while most programs common to MS-DOS and Microsoft Windows use carriage return+line feed, whose hexadecimal value in ASCII is 0d 0a. In ASCII, carriage return is a distinct control character.

The different newline conventions cause text files that have been transferred between systems of different types to be displayed incorrectly.

Text in files created with programs which are common on Unix-like or classic Mac OS, appear as a single long line on most programs common to MS-DOS and Microsoft Windows because these do not display a single line feed or a single carriage return as a line break.

Conversely, when viewing a file originating from a Windows computer on a Unix-like system, the extra CR may be displayed as a second line break, as ^M, or as <cr> at the end of each line.

Furthermore, programs other than text editors may not accept a file, e.g. some configuration file, encoded using the foreign newline convention, as a valid file.

The problem can be hard to spot because some programs handle the foreign newlines properly while others do not. For example, a compiler may fail with obscure syntax errors even though the source file looks correct when displayed on the console or in an editor. On a Unix-like system, the command cat -v myfile.txt will send the file to stdout (normally the terminal) and make the ^M visible, which can be useful for debugging. Modern text editors generally recognize all flavours of CR+LF newlines and allow users to convert between the different standards. Web browsers are usually also capable of displaying text files and websites which use different types of newlines.

Even if a program supports different newline conventions, these features are often not sufficiently labeled, described, or documented. Typically a menu or combo-box enumerating different newline conventions will be displayed to users without an indication if the selection will re-interpret, temporarily convert, or permanently convert the newlines. Some programs will implicitly convert on open, copy, paste, or save—often inconsistently.

Most textual Internet protocols (including HTTP, SMTP, FTP, IRC, and many others) mandate the use of ASCII CR+LF ('\r\n', 0x0D 0x0A) on the protocol level, but recommend that tolerant applications recognize lone LF ('\n', 0x0A) as well. Despite the dictated standard, many applications erroneously use the C newline escape sequence '\n' (LF) instead of the correct combination of carriage return escape and newline escape sequences '\r\n' (CR+LF) (see section Newline in programming languages above). This accidental use of the wrong escape sequences leads to problems when trying to communicate with systems adhering to the stricter interpretation of the standards instead of the suggested tolerant interpretation. One such intolerant system is the qmail mail transfer agent that actively refuses to accept messages from systems that send bare LF instead of the required CR+LF.[17]

The standard Internet Message Format[18] for eMail states: CR and LF MUST only occur together as CRLF; they MUST NOT appear independently in the body.

The File Transfer Protocol can automatically convert newlines in files being transferred between systems with different newline representations when the transfer is done in "ASCII mode". However, transferring binary files in this mode usually has disastrous results: any occurrence of the newline byte sequence—which does not have line terminator semantics in this context, but is just part of a normal sequence of bytes—will be translated to whatever newline representation the other system uses, effectively corrupting the file. FTP clients often employ some heuristics (for example, inspection of filename extensions) to automatically select either binary or ASCII mode, but in the end it is up to users to make sure their files are transferred in the correct mode. If there is any doubt as to the correct mode, binary mode should be used, as then no files will be altered by FTP, though they may display incorrectly.[19]

Conversion between newline formats

Text editors are often used for converting a text file between different newline formats; most modern editors can read and write files using at least the different ASCII CR/LF conventions. The standard Windows editor Notepad is not one of them (although WordPad and the MS-DOS Editor are).

Editors are often unsuitable for converting larger files. For larger files (on Windows NT/2000/XP) the following command is often used:

D:\>TYPE unix_file | FIND /V "" > dos_file

On many Unix systems, the dos2unix (sometimes named fromdos or d2u) and unix2dos (sometimes named todos or u2d) utilities are used to translate between ASCII CR+LF (DOS/Windows) and LF (Unix) newlines. Different versions of these commands vary slightly in their syntax. However, the tr command is available on virtually every Unix-like system and can be used to perform arbitrary replacement operations on single characters. A DOS/Windows text file can be converted to Unix format by simply removing all ASCII CR characters with

$ tr -d '\r' < inputfile > outputfile

or, if the text has only CR newlines, by converting all CR newlines to LF with

$ tr '\r' '\n' < inputfile > outputfile

The same tasks are sometimes performed with awk, sed, or in Perl if the platform has a Perl interpreter:

$ awk '{sub("$","\r\n"); printf("%s",$0);}' inputfile > outputfile  # UNIX to DOS  (adding CRs on Linux and BSD based OS that haven't GNU extensions)
$ awk '{gsub("\r",""); print;}' inputfile > outputfile              # DOS to UNIX  (removing CRs on Linux and BSD based OS that haven't GNU extensions)
$ sed -e 's/$/\r/' inputfile > outputfile              # UNIX to DOS  (adding CRs on Linux based OS that use GNU extensions)
$ sed -e 's/\r$//' inputfile > outputfile              # DOS  to UNIX (removing CRs on Linux based OS that use GNU extensions)
$ cat inputfile | tr -d "\r" > outputfile              # DOS  to UNIX (removing CRs using tr(1). Not Unicode compliant.)
$ perl -pe 's/\r?\n|\r/\r\n/g' inputfile > outputfile  # Convert to DOS
$ perl -pe 's/\r?\n|\r/\n/g'   inputfile > outputfile  # Convert to UNIX
$ perl -pe 's/\r?\n|\r/\r/g'   inputfile > outputfile  # Convert to old Mac

To identify what type of line breaks a text file contains, the file command can be used. Moreover, the editor Vim can be convenient to make a file compatible with the Windows notepad text editor. For example:

 $ file myfile.txt
 myfile.txt: ASCII English text
 $ vim myfile.txt

within vim

 :set fileformat=dos
 :wq
 $ file myfile.txt
 myfile.txt: ASCII English text, with CRLF line terminators

The following grep commands echo the filename (in this case myfile.txt) to the command line if the file is of the specified style:

$ grep -PL $'\r\n' myfile.txt # show UNIX style file (LF terminated)
$ grep -Pl $'\r\n' myfile.txt # show DOS style file (CRLF terminated)

For systems with egrep (extended grep) such as Debian (Linux) based systems and many other Unix systems, these commands can be used:

$ egrep -L $'\r\n' myfile.txt # show UNIX style file (LF terminated)
$ egrep -l $'\r\n' myfile.txt # show DOS style file (CRLF terminated)

The above grep commands work under Unix systems or in Cygwin under Windows. Note that these commands make some assumptions about the kinds of files that exist on the system (specifically it's assuming only Unix and DOS-style files—no Mac OS 9-style files).

This technique is often combined with find to list files recursively. For instance, the following command checks all "regular files" (e.g. it will exclude directories, symbolic links, etc.) to find all Unix-style files in a directory tree, starting from the current directory (.), and saves the results in file unix_files.txt, overwriting it if the file already exists:

$ find . -type f -exec grep -PL '\r\n' {} \; > unix_files.txt

This example will find C files and convert them to LF style line endings:

$ find -name '*.[ch]' -exec fromdos {} \;

The file command also detects the type of EOL used:

$ file myfile.txt
myfile.txt: ASCII text, with CRLF line terminators

Other tools permit the user to visualise the EOL characters:

$ od -a myfile.txt
$ cat -e myfile.txt
$ hexdump -c myfile.txt

dos2unix, unix2dos, mac2unix, unix2mac, mac2dos, dos2mac can perform conversions. The flip[20] command is often used.

Interpretation

Two ways to view newlines, both of which are self-consistent, are that newlines either separate lines or that they terminate lines. If a newline is considered a separator, there will be no newline after the last line of a file. Some programs have problems processing the last line of a file if it is not terminated by a newline. On the other hand, programs that expect newline to be used as a separator will interpret a final newline as starting a new (empty) line. Conversely, if a newline is considered a terminator, all text lines including the last are expected to be terminated by a newline. If the final character sequence in a text file is not a newline, the final line of the file may be considered to be an improper or incomplete text line, or the file may be considered to be improperly truncated.

In text intended primarily to be read by humans using software which implements the word wrap feature, a newline character typically only needs to be stored if a line break is required independent of whether the next word would fit on the same line, such as between paragraphs and in vertical lists. Therefore, in the logic of word processing and most text editors, newline is used as a paragraph break and is known as a "hard return", in contrast to "soft returns" which are dynamically created to implement word wrapping and are changeable with each display instance. In many applications a separate control character called "manual line break" exists for forcing line breaks inside a single paragraph. The glyph for the control character for a hard return is usually a pilcrow (¶), and for the manual line break is usually a carriage return arrow (↵).

Reverse and partial line feeds

RI, (U+008D REVERSE LINE FEED,[21] ISO/IEC 6429 8D, decimal 141) is used to move the printing position back one line (by reverse feeding the paper, or by moving a display cursor up one line) so that other characters may be printed over existing text. This may be done to make them bolder, or to add underlines, strike-throughs or other characters such as diacritics.

Similarly, PLD (U+008B PARTIAL LINE FORWARD, decimal 139) and PLU (U+008C PARTIAL LINE BACKWARD, decimal 140) can be used to advance or reverse the text printing position by some fraction of the vertical line spacing (typically, half). These can be used in combination for subscripts (by advancing and then reversing) and superscripts (by reversing and then advancing), and may also be useful for printing diacritics.

See also

References

  1. ^ Qualline, Steve (2001). Vi Improved - Vim (PDF). Sams. p. 120. ISBN 9780735710016.
  2. ^ "ASCII Chart".
  3. ^ Bray, Andrew C.; Dickens, Adrian C.; Holmes, Mark A. The Advanced User Guide for the BBC Microcomputer (PDF). pp. 103, 104. ISBN 978-0946827008. Retrieved 30 January 2019.
  4. ^ "RISC OS 3 Programmers' Reference Manual". Retrieved 18 July 2018.
  5. ^ IBM System/360 Reference Data Card, Publication GX20-1703, IBM Data Processing Division, White Plains, NY
  6. ^ "UAX #14: Unicode Line Breaking Algorithm". www.unicode.org.
  7. ^ "C1 Control Character Set of ISO 6429" (PDF). October 1, 1983.
  8. ^ "Control Functions for Coded Character Sets" (PDF). June 1991.
  9. ^ "Character Code Structure and Extension Techniques, 6th edition" (PDF). December 1994.
  10. ^ "ECMAScript Language Specification 5th edition" (PDF). ECMA International. December 2009. p. 15. Retrieved 4 April 2010.
  11. ^ "YAML Ain't Markup Language (YAML™) Version 1.2". yaml.org.
  12. ^ "binmode - perldoc.perl.org". perldoc.perl.org.
  13. ^ "PHP: Strings - Manual". www.php.net.
  14. ^ "Lexical analysis — Python v3.0.1 documentation". docs.python.org.
  15. ^ "What's new in Python 2.3".
  16. ^ "PHP: Predefined Constants - Manual". www.php.net.
  17. ^ "cr.yp.to".
  18. ^ "RFC 2822 - Internet Message Format". The Internet Engineering Task Force.
  19. ^ "File Transfer". When in doubt, transfer in binary mode.
  20. ^ "ASCII text converstion between UNIX, Macintosh, MS-DOS". Archived from the original on 2009-02-09.
  21. ^ "C1 Controls and Latin-1 Supplement" (PDF). unicode.org. Retrieved 13 February 2016.

External links

Aldor

Aldor is a programming language. It is the successor of A# as the extension language of the Axiom computer algebra system.

Aldor combines imperative, functional, and object-oriented features. It has an elaborate type system,"Aldor Programming Language". Aldor.org. Retrieved 12 February 2017. allowing types to be used as first-class values. Aldor's syntax is heavily influenced by Pascal, but it is optionally indentation-sensitive, using whitespace characters and the off-side rule, like Python. In its current implementation, it is compiled, but an interactive listener is provided.

Aldor is distributed as free and open-source software, under the Apache License 2.0.

Canonical XML

Canonical XML is a normal form of XML, intended to allow relatively simple comparison of pairs of XML documents for equivalence; for this purpose, the Canonical XML transformation removes non-meaningful differences between the documents. Any XML document can be converted to Canonical XML.

For example, XML permits whitespace to occur at various points within start-tags, and attributes to be specified in any order. Such differences are seldom if ever used to convey meaning, and so these forms are generally considered equivalent:

class='a' >

In converting an arbitrary XML document to Canonical XML, attributes are encoded in a normative order (alphabetical by name), and with normative spacing and quoting (though with all namespace declarations placed ahead of regular attributes, and namespaced attributes sorted by namespace rather than prefix or qualified name). Thus, the second form above would be converted to the first.

Canonical XML specifies a number of other details, some of which are:

the UTF-8 encoding is used

line-ends are represented using the newline character 0x0A

whitespace in attribute values is normalized

entity references and non-special character references are expanded

CDATA sections are replaced with their character content

empty elements are encoded as start/end pairs, not using the special empty-element syntax

default attributes are made explicit

superfluous namespace declarations are deletedAccording to the W3C, if two XML documents have the same canonical form, then the two documents are logically equivalent within the given application context (except for limitations regarding a few unusual cases).

However, in a special context users might care about special semantics beyond the generic logical equivalence with which Canonical XML is associated. For example, a steganography system could conceal information in an XML document by varying whitespace, attribute quoting and order, the use of hexadecimal vs. decimal numeric character references, and so on. Obviously converting such a file to Canonical XML would lose those specialized semantics. On the other hand, XML files that differ in their use of upper- vs. lower-case, or that use archaic versus modern spelling, and so on, might be considered equivalent for certain purposes. Such contexts are beyond the scope of Canonical XML.

Carriage return

A carriage return, sometimes known as a cartridge return and often shortened to CR, or return, is a control character or mechanism used to reset a device's position to the beginning of a line of text. It is closely associated with the line feed and newline concepts, although it can be considered separately in its own right.

Comparison of programming languages (syntax)

This comparison of programming languages compares the features of language syntax (format) for over 50 computer programming languages.

EBCDIC code pages

EBCDIC code pages are based on the original EBCDIC character encoding; there are a variety of EBCDIC code pages intended for use in different parts of the world, including code pages for non-Latin scripts such as Chinese, Japanese (e.g., EBCDIC 930, JEF, and KEIS), Korean, and Greek (EBCDIC 875).

The most common newline convention used with EBCDIC code pages is to use a NEL (NEXT LINE) code between lines. Converters to other encodings often replace NEL with LF or CR/LF, even if there is a NEL in the target encoding.

Electronic document

An electronic document is any electronic media content (other than computer programs or system files) that are intended to be used in either an electronic form or as printed output. Originally, any computer data were considered as something internal — the final data output was always on paper. However, the development of computer networks has made it so that in most cases it is much more convenient to distribute electronic documents than printed ones. The improvements in electronic visual display technologies made it possible to view documents on screen instead of printing them (thus saving paper and the space required to store the printed copies).

However, using electronic documents for final presentation instead of paper has created the problem of multiple incompatible file formats. Even plain text computer files are not free from this problem — e.g. under MS-DOS, most programs could not work correctly with UNIX-style text files (see newline), and for non-English speakers, the different code pages always have been a source of trouble.

Even more problems are connected with complex file formats of various word processors, spreadsheets, and graphics software. To alleviate the problem, many software companies distribute free file viewers for their proprietary file formats (one example is Adobe's Acrobat Reader). The other solution is the development of standardized non-proprietary file formats (such as HTML and OpenDocument), and electronic documents for specialized uses have specialized formats – the specialized electronic articles in physics use TeX or PostScript.

Enriched text

Enriched text is a formatted text format for e-mail, defined by the IETF in RFC 1896 and associated with the text/enriched MIME type which is defined in RFC 1563. It is "intended to facilitate the wider interoperation of simple enriched text across a wide variety of hardware and software platforms". As of 2012, enriched text remained almost unknown in e-mail traffic, while HTML e-mail is widely used. Enriched text, or at least the subset of HTML that can be transformed into enriched text, is seen as superior to full HTML for use with e-mail (mainly because of security considerations).A predecessor of this MIME type was called text/richtext in RFC 1341 and RFC 1521. Neither should be confused with Rich Text Format (MIME type text/rtf or application/rtf) which are unrelated specifications, devised by Microsoft.

A single newline in enriched text is treated as a space. Formatting commands are in the same style as SGML and HTML. They must be balanced and nested.

Enriched text is a supported format of Emacs, Mutt, Mulberry and Netscape Communicator.

Escape sequences in C

Escape sequences are used in the programming languages C and C++, and their design was copied in many other languages such as Java and C#. An escape sequence is a sequence of characters that does not represent itself when used inside a character or string literal, but is translated into another character or a sequence of characters that may be difficult or impossible to represent directly.

In C, all escape sequences consist of two or more characters, the first of which is the backslash, \ (called the "Escape character"); the remaining characters determine the interpretation of the escape sequence. For example, \n is an escape sequence that denotes a newline character.

Image Cytometry Standard

The Image Cytometry Standard (ICS) is a digital multidimensional image file format used in life sciences microscopy. It stores not only the image data, but also the microscopic parameters describing the optics during the acquisition.

ICS was first proposed in: P. Dean, L. Mascio, D. Ow, D. Sudar, J. Mullikin, Proposed standard for image cytometry data files, Cytometry, n. 11, pp. 561-569, 1990 [1].

The original ICS file format actually uses two separate files: a text header file with .ics extension and other, much bigger and with the actual image data, with .ids extension. This allows the compression of the data while leaving the header file accessible.

On the other hand, the newer ICS2 file format uses only one single .ics file with both the header and the data together.

The .ics in the two-file format is a text file with fields separated by tabs, and lines ending with a newline character. In the newer ICS2 format this text header precedes the binary data.

The ICS format is capable of storing:

multidimensional and multichannel data

images in 8, 16 or 32 bit integer, 32 or 64 bit floating point and floating point complex data

all microscopic parameters directly relevant to the image formation

free-form comments

Internal field separator

For many command line interpreters (“shell”) of Unix operating systems, the internal field separator (abbreviated IFS) refers to a variable which defines the character or characters used to separate a pattern into tokens for some operations.

IFS typically includes the space, tab, and the newline.

From the bash man page:

The shell treats each character of $IFS as a delimiter, and splits the results of the other expansions into words on these characters.

If IFS is unset, or its value is exactly , the default, then sequences of , , and at the beginning and end of the results of the previous expansions are ignored, and any sequence of IFS characters not at the beginning or end serves to delimit words.

If IFS has a value other than the default, then sequences of the whitespace characters space and tab are ignored at the beginning and end of the word, as long as the whitespace character is in the value of IFS (an IFS whitespace character).

Any character in IFS that is not IFS whitespace, along with any adjacent IFS whitespace characters, delimits a field. A sequence of IFS whitespace characters is also treated as a delimiter. If the value of IFS is null, no word splitting occurs.

IFS was usable as an exploit in some versions of Unix. A program with root permissions could be fooled into executing user-supplied code if it ran (for instance) system("/bin/mail") and was called with $IFS set to "/", in which case it would run the program "bin" (in the current directory and thus writable by the user) with root permissions. This has been fixed by making the shells not inherit the IFS variable.

JSON streaming

JSON streaming comprises communications protocols to delimit JSON objects built upon lower-level stream-oriented protocols (such as TCP), that ensures individual JSON objects are recognized, when the server and clients use the same one (e.g. implicitly coded in).

New Line Cinema

New Line Productions Inc., doing business as New Line Cinema, is an American film production studio of Warner Bros. It was founded in 1967 by Robert Shaye as an independent film distribution company, later becoming a film studio. It was acquired by Turner Broadcasting System in 1994; Turner later merged with Time Warner (now WarnerMedia) in 1996, and New Line was merged with Warner Bros. Pictures in 2008. Currently, its films are distributed by Warner Bros. Pictures.

Notepad2

Notepad2 is a free and open-source text editor for Microsoft Windows, released under a BSD software license. It was written by Florian Balmer using the Scintilla editor component, and it was first publicly released in April 2004. Balmer based Notepad2 on the principles of Notepad: small, fast, and usable.

It features syntax highlighting for many programming languages: ASP, assembly language, C, C++, C#, Common Gateway Interface (CGI), Cascading Style Sheets (CSS), HTML, Java, JavaScript, NSIS, Pascal, Perl, PHP, Python, SQL, Visual Basic (VB), VBScript, XHTML, and XML. It also features syntax highlighting for the following file formats: BAT, DIFF, INF, INI, REG, and configuration files (.properties).Notepad2 also has several other features:

Auto indentation

Bracket matching

Character encoding conversion between ASCII, UTF-8, and UTF-16 formats

Multiple undo/redo; rectangular block selection

Newline format conversion, between DOS (CR/LF), Unix (LF), and Macintosh (CR) formats

Regular expression-based find and replaceBalmer has stated that some features will probably never be implemented in Notepad2, as they are beyond his design goal of a simple Notepad-like application. These include Notepad2's most requested feature, a multiple document interface. Other missing features such as code folding, file associations and bookmarks are available through modifications, linked from the homepage. Starting with version 4.2.25-rc6, an x64 build is also offered.

Paragraph

A paragraph (from the Ancient Greek παράγραφος paragraphos, "to write beside" or "written beside") is a self-contained unit of a discourse in writing dealing with a particular point or idea. A paragraph consists of one or more sentences. Though not required by the syntax of any language, paragraphs are usually an expected part of formal writing, used to organize longer prose.

Perl Compatible Regular Expressions

Perl Compatible Regular Expressions (PCRE) is a library written in C, which implements a regular expression engine, inspired by the capabilities of the Perl programming language. Philip Hazel started writing PCRE in summer 1997. PCRE's syntax is much more powerful and flexible than either of the POSIX regular expression flavors and than that of many other regular-expression libraries.

While PCRE originally aimed at feature-equivalence with Perl, the two implementations are not fully equivalent. During the PCRE 7.x and Perl 5.9.x phase, the two projects have coordinated development, with features being ported between them in both directions.A number of prominent open-source programs, such as the Apache and Nginx HTTP Servers, and the PHP and R scripting languages, incorporate the PCRE library; proprietary software can do likewise, as the library is BSD licensed. As of Perl 5.10, PCRE is also available as a replacement for Perl's default regular expression engine through the re::engine::PCRE module.

The library can be built on Unix, Windows, and several other environments. PCRE is distributed with a POSIX C wrapper, a native C++ wrapper, several test programs, and the utility program pcregrep built in tandem with the library.

Pseudocode

Pseudocode is an informal high-level description of the operating principle of a computer program or other algorithm.

It uses the structural conventions of a normal programming language, but is intended for human reading rather than machine reading. Pseudocode typically omits details that are essential for machine understanding of the algorithm, such as variable declarations, system-specific code and some subroutines. The programming language is augmented with natural language description details, where convenient, or with compact mathematical notation. The purpose of using pseudocode is that it is easier for people to understand than conventional programming language code, and that it is an efficient and environment-independent description of the key principles of an algorithm. It is commonly used in textbooks and scientific publications that are documenting various algorithms, and also in planning of computer program development, for sketching out the structure of the program before the actual coding takes place.

No standard for pseudocode syntax exists, as a program in pseudocode is not an executable program. Pseudocode resembles, but should not be confused with, skeleton programs which can be compiled without errors. Flowcharts, drakon-charts and Unified Modeling Language (UML) charts can be thought of as a graphical alternative to pseudocode, but are more spacious on paper. Languages such as HAGGIS bridge the gap between pseudocode and code written in programming languages. Its main use is to introduce students to high level languages through use of this hybrid language.

Tr (Unix)

tr is a command in Unix and Unix-like operating systems. It is an abbreviation of translate or transliterate, indicating its operation of replacing or removing specific characters in its input data set.

Wc (Unix)

wc (short for word count) is a command in Unix and Unix-like operating systems.

The program reads either standard input or a list of files and generates one or more of the following statistics: newline count, word count, and byte count. If a list of files is provided, both individual file and total statistics follow.

Words (Unix)

words is a standard file on Unix and Unix-like operating systems, and is simply a newline-delimited list of dictionary words. It is used, for instance, by spell-checking programs.The words file is usually stored in /usr/share/dict/words or /usr/dict/words.

On Debian and Ubuntu, the words file is provided by the wordlist package, or its provider packages wbritish, wamerican, etc. On Fedora and Arch, the words file is provided by the words package.

This page is based on a Wikipedia article written by authors (here).
Text is available under the CC BY-SA 3.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.