4.1 Introduction to Encoding Schemes

Computers operate using binary digits (bits – 0s and 1s). However, humans communicate using characters, symbols, and languages. To bridge this gap, encoding schemes are used to represent these human-readable characters as binary numbers that computers can understand and process. An encoding scheme defines a unique numeric code for each character.

4.2 ASCII (American Standard Code for Information Interchange)

• Purpose: ASCII was one of the earliest and most widely used encoding standards, primarily designed for the English language.
• Structure:
  • Standard ASCII: Uses 7 bits to represent 128 characters (codes 0-127). This includes:
    • Uppercase English letters (A-Z)
    • Lowercase English letters (a-z)
    • Digits (0-9)
    • Punctuation symbols (!, @, #, etc.)
    • Control characters (like newline, tab, backspace).
  • Extended ASCII: Uses 8 bits (1 byte) to represent 256 characters (codes 0-255). The first 128 characters are the same as standard ASCII, while the additional 128 characters were used for graphical symbols and letters with accents (e.g., é, ñ).
• Limitations:
  • Limited character set: It cannot represent characters from languages other than English effectively.
  • Lack of support for symbols and characters used in many other languages.

4.3 ISCII (Indian Standard Code for Information Interchange)

• Purpose: Developed in India to address the need for representing characters from various Indian scripts (Devanagari, Bengali, Tamil, Telugu, etc.).
• Structure: ISCII is an 8-bit encoding standard. The first 128 codes are the same as ASCII. The remaining codes are used to represent characters from Indian scripts.
• Key Feature: ISCII allows for transliteration – the conversion of text from one Indian script to another. This is useful for multilingual applications and data exchange.

4.4 Unicode

• Purpose: Unicode is a modern, universal character encoding standard designed to represent every character from every language in the world. It aims to unify all other encoding schemes into one.
• Concept: Unicode assigns a unique number, called a code point, to every character. For example, the code point for the letter 'A' is U+0041, and for the Devanagari letter 'क' is U+0915.
• Advantages:
  • Universal character support: Can represent characters from all languages.
  • Solves compatibility issues: Eliminates the need for multiple encoding schemes.

Unicode Encoding Formats

The Unicode standard itself doesn't define how to store the code points in memory; that job is done by different Unicode encodings.

a) UTF-8 (Unicode Transformation Format - 8-bit)

• Mechanism: UTF-8 is a variable-width encoding. This means it uses a variable number of bytes to represent each character.
  • For characters that are also in ASCII (like 'A', 'b', '7'), it uses only 1 byte. This makes it fully backward compatible with ASCII.
  • For other characters (like 'é', '€', 'क', '😂'), it uses 2, 3, or 4 bytes as needed.
• Advantages:
  • Space-efficient: For text that is primarily English, a UTF-8 file is roughly the same size as an ASCII file.
  • Compatibility: Its backward compatibility with ASCII is a huge advantage.
  • Widely Used: It is the dominant character encoding for the World Wide Web.

b) UTF-16 (Unicode Transformation Format - 16-bit)

• Mechanism: UTF-16 uses 2 or 4 bytes to represent each character. Most commonly used characters are represented using 2 bytes.
• Advantages:
  • Efficient for languages with a large number of characters that fall within the Basic Multilingual Plane (BMP).
• Disadvantages:
  • Less space-efficient for English text compared to UTF-8.

c) UTF-32 (Unicode Transformation Format - 32-bit)

• Mechanism: UTF-32 is a fixed-width encoding. It uses exactly 4 bytes (32 bits) to store every single character, regardless of what that character is.
• Advantages:
  • Simplicity: Since every character has the same length, finding the Nth character in a string is very simple and fast.
• Disadvantages:
  • Inefficient Storage: It is very wasteful of space. A text file containing only the word "Hello" would take 5 * 4 = 20 bytes in UTF-32, whereas it would only take 5 bytes in UTF-8 or ASCII.
• Usage: UTF-32 is rarely used for storing or transmitting data but may be used internally by some programs for easier processing.

Summary Table:

---

ASCII Character Table (Standard 7-bit)