YanoSec
Progress0 of 152 lessons complete

Course / 04 Code / Lesson 02 of 11

ASCII and Unicode

Estimated time: 15 min reading · 15 min activity

An editor may show one accented letter while a file reader counts two bytes for it. Both can be correct because a character and its stored representation are different things. This distinction matters when you later compare a Python text length with a file size.

Unicode assigns numbers, called code points, to characters. A code point is commonly written as U+ followed by hex digits. U+0041 identifies the capital letter A. UTF-8 is an encoding: a rule for representing code points with bytes. It uses one to four bytes for a valid Unicode scalar value, which is a code point allowed in this encoding.

ASCII is the older set of 128 characters from U+0000 through U+007F. It includes English letters, digits, punctuation, and control characters such as a newline. Each ASCII code point uses one byte in UTF-8. Other code points can need more bytes even when they appear as one symbol.

Before you begin

Estimated time
15 min reading, 15 min activity. Setup and optional practice may take longer.
Where you work
Use the printed text and byte tables in this lesson, with paper or a text editor for notes. No account, installation, or terminal is needed. Each table supplies the complete evidence for its activity.
Without a lab
Work from the complete pair lists and source notes, make your prediction in notes, and compare with the activity’s stated result. The tables are authored models. Reading works without JavaScript. Answer checking needs JavaScript and a secure context, normally HTTPS or a local preview; saving additionally needs usable browser storage.

After this lesson you can:

  • Name a character's code point separately from its UTF-8 bytes.
  • Explain why one character can occupy more than one byte.
  • Count bytes from hex pairs instead of from the letters on screen.

01Example

Start with A, a character in the ASCII range. Its code point is U+0041. UTF-8 stores it as the single byte 41. The binary and hex entries below describe that same byte; they do not add extra copies of the character. This table is a hand grouping.

Character: A
Code point: U+0041
UTF-8 bit pattern: 01000001
Hex byte: 41
Bytes: 1
Code points: 1

02The code point is not the byte list

The code point identifies the character; the byte list describes its encoding. For €, the code point is U+20AC and its UTF-8 bytes are e2 82 ac. Count the three pairs to find its byte size. The single character takes more space than the ASCII A in the worked example.

The UTF-8 specification gives a pattern for each allowed length. In the two-byte pattern, 110xxxxx 10xxxxxx, the first byte signals that another byte follows. The x positions carry bits from the code point. You do not need to calculate those bits for this activity; the table supplies the complete bytes.

A symbol on screen can also be built from several code points. For example, a base e followed by a combining accent may look like composed é. The visible result can look the same while the code points and bytes differ. Here, character counts use the explicitly stated code points, including the newline; they do not count how many shapes a font draws.

Keep character, code point and UTF-8 bytes aligned: A needs one byte, while € needs three.Model. A teaching model drawn by hand. It shows the idea, not real tool output.

Read the full explanation below.

Playback is optional. Step through at your own pace.

Declared UTF-8 examples

Character

Aone character
€one character

Code point

U+0041one code point
U+20ACone code point

UTF-8 bytes

41one byteByte count 1; code-point count 1.
e2 82 acthree bytes

A → U+0041A has code point U+0041

U+0041 → 41UTF-8 encodes U+0041 as 41

A is one code point and one UTF-8 byte.

Follow the explanation

  1. A is one code point and one UTF-8 byte.
  2. € is one code point but three UTF-8 bytes.
Read every state and value

1. A is one code point and one UTF-8 byte.

Example
Declared UTF-8 examples
Visible Item
a · U+0041 · 41
Current mappings
A has code point U+0041 · UTF-8 encodes U+0041 as 41
Annotations
Item: 41; Text: Byte count 1; code-point count 1.

2. € is one code point but three UTF-8 bytes.

Example
Declared UTF-8 examples
Visible Item
a · U+0041 · 41 · euro · U+20AC · e2 82 ac
Current mappings
€ has code point U+20AC · UTF-8 encodes U+20AC as three bytes
Annotations
Item: e2 82 ac; Text: Byte count 3; code-point count 1.
Keep character, code point and UTF-8 bytes aligned: A needs one byte, while € needs three.

03Count a word that includes an accented character

The download contains an accented-word reference, Café, that you will reuse in the later file-reading lesson. For a fresh guided count here, use the price marker €! followed by LF. LF means line feed, the newline character represented by byte 0a. The complete display is e2 82 ac 21 0a.

YanoSec / Code
UTF-8 bytes below were grouped by hand from RFC 3629. You can finish this card without running Python.

A code point is the integer that identifies a character. U+0041 means hex 41.
ASCII names the characters U+0000 through U+007F.
UTF-8 stores each of those as one byte equal to the code point.

A | U+0041 LATIN CAPITAL LETTER A | 41
é | U+00E9 LATIN SMALL LETTER E WITH ACUTE | c3 a9
€ | U+20AC EURO SIGN | e2 82 ac | three pairs

Guided activity: €! followed by LF (line feed).
Complete bytes: e2 82 ac 21 0a
Code points: 3 (€ then ! then LF)
Byte count: 5
The euro sign contributes three pairs; the other two entries contribute one each.

Accented-word reference for the later note.txt file, including one LF byte 0a:
Café
Code points, counting the newline: 5
Bytes: 43 61 66 c3 a9 0a
Byte count: 6
For a shorter pair list, count only the pairs that are actually present.

Optional, ungraded:
composed U+00E9 is c3 a9
decomposed U+0065 then U+0301 is 65 cc 81
A matching shape on screen is not matching bytes.

Download the code-point and UTF-8 table

  1. Write €, !, and LF as three entries. Each entry in this case is one code point.
  2. Copy e2 82 ac 21 0a. Group e2 82 ac under €, 21 under !, and 0a under LF. Count the complete pairs.
  3. Compare with the guided check: 3 code points and 5 bytes. The euro sign contributes three bytes while ! and LF contribute one each. Explain why visible entries and byte pairs are different counts.
  4. Stop after recording the two counts and their reason. Running an encoder can wait; changing the file to remove the accent would change the example.

04Optional challenge

For about ten extra minutes, compare the two spellings of an accented letter on the download: one composed code point and a base letter followed by a combining accent. Your explanation should identify why similar appearance does not guarantee identical code points or bytes. Keep the two supplied sequences separate.

05Fresh cases for the checks

Use these fictional cases for the checks below. They have different inputs from the worked examples. Read them here or make notes; no installation or external service is needed.

Unicode case AOwned fixture. An original file made for this course. Follow the activity's instructions in your own lab.

The string is ©B followed by one LF newline.
© is U+00A9, UTF-8 c2 a9.
B is U+0042, UTF-8 42.
LF is U+000A, UTF-8 0a.
Complete bytes: c2 a9 42 0a. Each listed entry is one code point.

Unicode case BOwned fixture. An original file made for this course. Follow the activity's instructions in your own lab.

Two supplied representations of an accented letter:
Version P: U+00F1, bytes c3 b1.
Version Q: U+006E then U+0303, bytes 6e cc 83.
They may look alike on screen; compare the supplied sequences.

06Explain your prediction

Optional written self-check: Distinguish code-point counts from UTF-8 pair counts, include explicit newlines, and explain why a matching appearance may conceal different sequences.

07Common mistakes

If the byte count matches the number of visible entries, check the supplied pairs for €. It contributes three bytes even though it appears as one symbol.

If you equate code points with visible symbols in every language, return to the combining-accent example. Code points are numbered units; a displayed symbol can combine several of them.

If a later byte sequence contains ff, do not assign it a guessed UTF-8 character. That byte is outside the valid UTF-8 start patterns; keep a decoding problem visible.

References