From TAP to curl

The life of a curl request

curl only ever talks to the kernel. What the kernel does between those calls is what this talk builds.

The networking stack lives in the kernel

Use the HTTP standard library, as a client or a server, in Go, Ruby or Node, and you hand the hard part to the operating system.

• Your code builds a request. The kernel puts it on the wire: Ethernet, IP, TCP, the retransmits.
• A userland process only reaches that stack through system calls: socket connect send recv
• Those calls hide the complexity. They are a narrow API; the bytes on the wire are on the other side of them.

This talk is that other side.

Syscalls hide the complexity

strace -e trace=network curl -4 -sI --max-redirs 0 http://google.com/ · inside Docker

socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
connect(5, {sin_port=htons(80),
            sin_addr=inet_addr("142.251.30.113")}, 16) = 0
# ↑ curl said “connect”. The kernel did ARP (maybe cached)
#   and the whole TCP handshake. No syscall per packet.

sendto(5, "HEAD / HTTP/1.1\r\nHost: google.com\r\n"..., 75) = 75
recvfrom(5, "HTTP/1.1 301 Moved Permanently\r\n"
            "Location: http://www.google.com/\r\n"..., ...) = 554
close(5) = 0

curl only sent and received HTTP text. The handshake, the headers and any resends all happened inside the kernel, where curl can’t see them. Next: what that looks like on the wire.

Behind those few syscalls

ARP finds the next hop, TCP opens a connection, HTTP rides inside it.

It is all one string of bits

From the CPU, through the operating system, down into the networking stack. The same stuff, the whole way.

• Ones and zeros, arranged so they mean something. A port. A promise to resend. A line of HTTP.
• Look at the endless stream through a window of fixed length. Computation is that window stepping from one arrangement to the next.
• The path of those arrangements is the work. One trajectory is a hard problem. Another is just hello world, drawn on a screen.

I think of that as a fundamental theorem of computation.

How I got here

I wanted to own the bytes myself, from the wire up.

Why rebuild something that already works?

• Curiosity — the fields you see in Wireshark become structs you wrote
• Debugging — timeouts and resets stop being mysteries
• Vocabulary — cloud networking, eBPF and QUIC all assume these layers
• Joy — watching ping get answered by your code

Not competing with the kernel. Learning beside it.

What this project is, and isn’t

Keeping the scope small is how it got finished.

What you’ll leave with

1. Why — a reason to take apart something that already works
2. Scope — a teaching stack, not production TCP
3. The onion — every layer has the same shape
4. A demo — from ping to HTTPS, answered by our code
5. Giants — great systems are built from many people’s work
6. Foresight — protocols that shipped and left room for what came later
7. Where to dig — each layer is a whole field of its own

What “onion interface” means

Each layer is a header plus a payload.
In code it’s a struct with Decode (bytes → struct) and Encode (struct → bytes).
It supports only a few operations.
Its payload is the next layer’s whole message.

Layers nest inside each other. A type field in each header says which layer comes next.

Each layer is an RFC. Its payload is the next one.

A new layer is an agreement about how to read someone else’s payload. The layers outside do not change.

The lab setup

A TAP device hands us whole Ethernet frames, so we handle ARP ourselves. It runs in a Linux container, because macOS has no TAP.

The stack we’re going to build

Every layer does the same job: read bytes, decide, write bytes.

Bottom layer: the wire

TAP hands us every Ethernet frame sent to 10.0.0.2, raw.

Opening a TAP device

internal/tap/tap_linux.go

// Linux creates TUN and TAP devices through one special file
f, err := os.OpenFile("/dev/net/tun", os.O_RDWR, 0)

// ask for a TAP device (whole Ethernet frames) named e.g. "tap0"
var req ifreq
req.flags = iffTap | iffNoPI   // no extra packet-info header
copy(req.name[:], name)
syscall.Syscall(SYS_IOCTL, fd, tunSetIff, …)

// from now on, each read returns one frame and each write sends one

Rabbit holes kernel networking internals · virtual NICs (veth, TAP, macvlan) · XDP / eBPF · NIC offloads

Next: the link layer

Ethernet: machines on the same network talk using MAC addresses.

An Ethernet frame, as a Go struct

internal/eth/ethernet.go

// One Ethernet frame: a 14-byte header, then the payload
type Frame struct {
    Dst     net.HardwareAddr // who it's for (6-byte MAC address)
    Src     net.HardwareAddr // who sent it
    Type    uint16           // what's inside: 0x0806 ARP, 0x0800 IPv4
    Payload []byte           // ← the next layer's whole message
}

// bytes off the wire → struct
func Decode(b []byte) (Frame, error) { /* … */ }

// struct → bytes ready to send
func (f Frame) Encode() []byte     { /* … */ }

Addresses, a type field saying what’s inside, and the payload. Every layer follows this pattern.

Decoding means reading bytes at fixed offsets

internal/eth/ethernet.go

func Decode(b []byte) (Frame, error) {
    // anything shorter than the 14-byte header can't be a frame
    if len(b) < HeaderLen {
        return Frame{}, fmt.Errorf("ethernet: frame too short")
    }
    // every field sits at a fixed position, so just copy it out
    return Frame{
        Dst:     append(net.HardwareAddr(nil), b[0:6]...),  // bytes 0–5
        Src:     append(net.HardwareAddr(nil), b[6:12]...), // bytes 6–11
        Type:    binary.BigEndian.Uint16(b[12:14]),         // bytes 12–13
        Payload: append([]byte(nil), b[14:]...),            // the rest
    }, nil
}

An Ethernet frame, as real bytes

captures/talk-http.jsonl · frame 0 (an ARP request)

# mytcp logs every frame to captures/<name>.jsonl (and a .pcap for Wireshark)
# pick frame 0, turn its hex back into bytes, and dump them
jq -r 'select(.n==0).data' captures/talk-http.jsonl | xxd -r -p | hexdump -C

00000000  ff ff ff ff ff ff ee ea  f3 0f 57 32 08 06 00 01  |..........W2....|
00000010  08 00 06 04 00 01 ee ea  f3 0f 57 32 0a 00 00 01  |..........W2....|
00000020  00 00 00 00 00 00 0a 00  00 02                    |..........|
0000002a

Each line: where it starts in hex (00000010 = byte 16), 16 bytes in hex, then the same bytes as text (. = not printable).

Numbers longer than a byte are big-endian, biggest byte first (“network byte order”): 08 06 is 0x0806. Go reads them with binary.BigEndian.Uint16.

Choosing the next layer by its type field

internal/stack/stack.go · HandleFrame

f, err := eth.Decode(b)
// …

// the EtherType field says what the payload is
switch f.Type {
case eth.TypeARP:  // 0x0806: "who has this IP?"
    return s.handleARP(f)
case eth.TypeIPv4: // 0x0800: an IP packet, for later
    return s.handleIPv4(f)
default:           // usually IPv6 chatter from the host: drop it
    return nil
}

Each protocol only decodes its own header. The stack package reads the type field and hands the payload to the right decoder. Frame 0 said 08 06, so next: ARP.

An ARP packet, as a Go struct

internal/arp/arp.go

// "Who has this IP?" (request) or "I do, here's my MAC" (reply)
type Packet struct {
    Op  uint16           // 1 = request, 2 = reply
    SHA net.HardwareAddr // sender MAC
    SPA net.IP           // sender IP
    THA net.HardwareAddr // target MAC (blank in a request)
    TPA net.IP           // target IP: the one being asked about
}

func Decode(b []byte) (Packet, error) { /* … */ } // bytes → struct
func (p Packet) Encode() []byte     { /* … */ } // struct → bytes

ARP: ask the local network which MAC address owns this IP. The owner replies with its MAC.

Answering “who has 10.0.0.2?”

internal/stack/stack.go · handleARP

p, err := arp.Decode(f.Payload)
// ignore replies, and questions about someone else's IP
if p.Op != arp.OpRequest || !p.TPA.Equal(s.cfg.IP) {
    return nil
}
// build the answer: "10.0.0.2 is at <our MAC>"
reply := arp.ReplyFor(p, s.cfg.MAC, s.cfg.IP)
// wrap it in Ethernet, addressed straight back to whoever asked
out := eth.Frame{
    Dst: p.SHA, Src: s.cfg.MAC, Type: eth.TypeARP,
    Payload: reply.Encode(),
}
return s.writeFrame(out)

Rabbit holes switching and MAC learning · VLANs · spanning tree · ARP spoofing · the Wi-Fi MAC layer

A real ARP request and reply: the header

captures/talk-http.jsonl · the first two frames of the capture

Frame 0 · kernel → everyone · the question: “who has 10.0.0.2?”

00000000  ff ff ff ff ff ff ee ea  f3 0f 57 32 08 06 00 01  |..........W2....|
00000010  08 00 06 04 00 01 ee ea  f3 0f 57 32 0a 00 00 01  |..........W2....|
00000020  00 00 00 00 00 00 0a 00  00 02                    |..........|
0000002a

Frame 1 · mytcp → kernel · our answer: “10.0.0.2 is at 02:00:00:00:00:02”

00000000  ee ea f3 0f 57 32 02 00  00 00 00 02 08 06 00 01  |....W2..........|
00000010  08 00 06 04 00 02 02 00  00 00 00 02 0a 00 00 02  |................|
00000020  ee ea f3 0f 57 32 0a 00  00 01                    |....W2....|
0000002a

One packet format for both directions: the header fields match byte for byte, except the operation.

A real ARP request and reply: the addresses

captures/talk-http.jsonl · the first two frames of the capture

Frame 0 · kernel → everyone · the question: “who has 10.0.0.2?”

00000000  ff ff ff ff ff ff ee ea  f3 0f 57 32 08 06 00 01  |..........W2....|
00000010  08 00 06 04 00 01 ee ea  f3 0f 57 32 0a 00 00 01  |..........W2....|
00000020  00 00 00 00 00 00 0a 00  00 02                    |..........|
0000002a

Frame 1 · mytcp → kernel · our answer: “10.0.0.2 is at 02:00:00:00:00:02”

00000000  ee ea f3 0f 57 32 02 00  00 00 00 02 08 06 00 01  |....W2..........|
00000010  08 00 06 04 00 02 02 00  00 00 00 02 0a 00 00 02  |................|
00000020  ee ea f3 0f 57 32 0a 00  00 01                    |....W2....|
0000002a

The reply swaps sender and target, and fills in the one thing the request didn’t know: our MAC.

Next: the network layer

IPv4: addresses that work across networks, so routers can carry a packet hop by hop.

An IPv4 packet, as a Go struct

internal/ip4/ipv4.go

// one hop-by-hop packet: where from, where to, and what's inside
type Packet struct {
    TOS       uint8  // type of service; 0 means ordinary traffic
    ID        uint16 // identification, used to reassemble fragments
    FlagsFrag uint16 // flags (e.g. Don't Fragment) + fragment offset
    TTL       uint8  // hop limit: every router takes one off
    Proto     uint8  // what's in Payload: 1 = ICMP, 6 = TCP
    Src       net.IP // source address
    Dst       net.IP // destination address
    Payload   []byte // the ICMP or TCP message
}

func Decode(b []byte) (Packet, error) { /* … */ } // bytes → struct
func (p Packet) Encode() []byte     { /* … */ } // struct → bytes

MAC addresses only reach the next machine. An IP address names the final destination, so the packet keeps it the whole way while the Ethernet header is rewritten at every hop.

An IPv4 header, on a real packet

captures/talk-http.jsonl · frame 2 (from ping -c 1 10.0.0.2)

00000000  02 00 00 00 00 02 ee ea  f3 0f 57 32 08 00 45 00  |..........W2..E.|
00000010  00 54 c6 d3 40 00 40 01  5f d3 0a 00 00 01 0a 00  |.T..@.@._.......|
00000020  00 02 08 00 40 fe 00 07  00 01 80 39 bc 6a 00 00  |....@......9.j..|
00000030  00 00 b9 82 02 00 00 00  00 00 10 11 12 13 14 15  |................|
00000040  16 17 18 19 1a 1b 1c 1d  1e 1f 20 21 22 23 24 25  |.......... !"#$%|
00000050  26 27 28 29 2a 2b 2c 2d  2e 2f 30 31 32 33 34 35  |&'()*+,-./012345|
00000060  36 37                                             |67|
00000062

IPv4’s protocol field picks the next layer

internal/stack/stack.go · handleIPv4

// after ip4.Decode, the protocol field says what the payload is
switch pkt.Proto {
case ip4.ProtoICMP: // 1 = ICMP, e.g. ping
    return s.handleICMP(f.Src, pkt)
case ip4.ProtoTCP:  // 6 = TCP: peel one more layer
    seg, err := tcp.Decode(pkt.Payload)
    // …
    return s.tcp.Handle(f.Src, pkt.Src, seg)
}

ICMP and TCP both ride on IPv4, side by side. The ping said 01, so ICMP first.

Rabbit holes routing (OSPF, BGP) · IPv6 · NAT · MTU and fragmentation

On top of IPv4: ICMP

ICMP is the network’s own messages. Ping asks “can a packet get there and back?”, which proves the IP path works before we trust it with TCP.

Answering a ping

internal/stack/stack.go · handleICMP

// the IPv4 payload is an ICMP message
echo, err := icmp.Decode(pkt.Payload)
// only answer echo requests (type 8)
if echo.Type != icmp.TypeEcho {
    return nil
}
// same ID, sequence number and data, but type 0 = echo reply
reply := icmp.ReplyFrom(echo)
// Encode adds a fresh checksum; send it back in IPv4, then Ethernet
return s.sendIPv4(srcMAC, pkt.Src, ip4.ProtoICMP, reply.Encode())

Rabbit holes how traceroute uses TTL and “time exceeded” · “destination unreachable” · path MTU discovery · why some firewalls drop ping

Next: the transport layer

TCP sits on IPv4 next to ICMP. It turns lost and reordered packets into a reliable, ordered stream between two programs. Ours connects, sends, receives and closes; no congestion control.

A TCP segment, as a Go struct

internal/tcp/segment.go

type Segment struct {
    SrcPort, DstPort uint16 // which app on each end (a number 1–65535)
    Seq, Ack         uint32 // "my bytes start at N", "I got yours up to M"
    Flags            uint8  // SYN (open), ACK, FIN (close), RST (abort)
    Window           uint16 // how many more bytes the sender can take
    Payload          []byte // app bytes, if any; a SYN has none
}

func Decode(b []byte) (Segment, error) { /* … */ }
// the checksum also covers both IP addresses, so Encode needs them
func (s Segment) Encode(srcIP, dstIP net.IP) []byte { /* … */ }

Rabbit holes congestion control (Reno, CUBIC, BBR) · flow control and windows · SACK · QUIC · SRD

Three messages to open, then bytes can flow

An ack means “I have every byte before this one”. SYN and FIN each count as one.

A real TCP segment: opening a connection

captures/talk-http.jsonl · frame 4 · client → mytcp

00000000  02 00 00 00 00 02 ee ea  f3 0f 57 32 08 00 45 00  |..........W2..E.|
00000010  00 3c 56 ea 40 00 40 06  cf cf 0a 00 00 01 0a 00  |.<V.@.@.........|
00000020  00 02 c6 8a 00 50 13 e8  5a 66 00 00 00 00 a0 02  |.....P..Zf......|
00000030  fa f0 2f 00 00 00 02 04  05 b4 04 02 08 0a a1 a8  |../.............|
00000040  33 3b 00 00 00 00 01 03  03 07                    |3;........|
0000004a

No application data. A SYN only opens the connection; bytes for an app come later, after both sides agree.

The SYN’s options, and why we ignore them

captures/talk-http.jsonl · frame 4 · same SYN

00000000  02 00 00 00 00 02 ee ea  f3 0f 57 32 08 00 45 00  |..........W2..E.|
00000010  00 3c 56 ea 40 00 40 06  cf cf 0a 00 00 01 0a 00  |.<V.@.@.........|
00000020  00 02 c6 8a 00 50 13 e8  5a 66 00 00 00 00 a0 02  |.....P..Zf......|
00000030  fa f0 2f 00 00 00 02 04  05 b4 04 02 08 0a a1 a8  |../.............|
00000040  33 3b 00 00 00 00 01 03  03 07                    |3;........|
0000004a

Options are offers. Our SYN+ACK sends none back, so the kernel on curl’s side turns off SACK, timestamps and window scaling, and falls back to the default 536-byte segments. Everything still works, just less efficiently.

The three-way handshake, in bytes

captures/talk-http.jsonl · frames 4, 5 and 6

Frame 4 · curl → mytcp · SYN

00000000  02 00 00 00 00 02 ee ea  f3 0f 57 32 08 00 45 00  |..........W2..E.|
00000010  00 3c 56 ea 40 00 40 06  cf cf 0a 00 00 01 0a 00  |.<V.@.@.........|
00000020  00 02 c6 8a 00 50 13 e8  5a 66 00 00 00 00 a0 02  |.....P..Zf......|
00000030  fa f0 2f 00 00 00 02 04  05 b4 04 02 08 0a a1 a8  |../.............|
00000040  33 3b 00 00 00 00 01 03  03 07                    |3;........|
0000004a

Frame 5 · mytcp → curl · SYN+ACK

00000000  ee ea f3 0f 57 32 02 00  00 00 00 02 08 00 45 00  |....W2........E.|
00000010  00 28 00 02 00 00 40 06  66 cc 0a 00 00 02 0a 00  |.(....@.f.......|
00000020  00 01 00 50 c6 8a 00 00  03 e8 13 e8 5a 67 50 12  |...P........ZgP.|
00000030  ff ff 62 be 00 00                                 |..b...|
00000036

Frame 6 · curl → mytcp · ACK

00000000  02 00 00 00 00 02 ee ea  f3 0f 57 32 08 00 45 00  |..........W2..E.|
00000010  00 28 56 eb 40 00 40 06  cf e2 0a 00 00 01 0a 00  |.(V.@.@.........|
00000020  00 02 c6 8a 00 50 13 e8  5a 67 00 00 03 e9 50 10  |.....P..Zg....P.|
00000030  fa f0 67 ce 00 00                                 |..g...|
00000036

Ethernet IPv4 TCP ports Seq Ack header length + flags window, checksum, urgent options (SYN only)

The first segment that carries data

captures/talk-http.jsonl · frame 7 · same connection, next segment from the client

00000000  02 00 00 00 00 02 ee ea  f3 0f 57 32 08 00 45 00  |..........W2..E.|
00000010  00 70 56 ec 40 00 40 06  cf 99 0a 00 00 01 0a 00  |.pV.@.@.........|
00000020  00 02 c6 8a 00 50 13 e8  5a 67 00 00 03 e9 50 18  |.....P..Zg....P.|
00000030  fa f0 91 b1 00 00 47 45  54 20 2f 20 48 54 54 50  |......GET / HTTP|
00000040  2f 31 2e 31 0d 0a 48 6f  73 74 3a 20 31 30 2e 30  |/1.1..Host: 10.0|
00000050  2e 30 2e 32 0d 0a 55 73  65 72 2d 41 67 65 6e 74  |.0.2..User-Agent|
00000060  3a 20 63 75 72 6c 2f 37  2e 38 38 2e 31 0d 0a 41  |: curl/7.88.1..A|
00000070  63 63 65 70 74 3a 20 2a  2f 2a 0d 0a 0d 0a        |ccept: */*....|
0000007e

The letters in the margin are the application’s. TCP does not know what they mean.

TCP is a stream. Segment edges mean nothing upstairs.

Give the app a net.Conn and a Listener, and both fit: HTTP reads until the blank line, TLS drives the handshake itself. Our server, Go’s net/http, and TLS all plug in the same way.

From segments to sockets: four types

internal/tcp

Segment     one packet: ports, seq, ack, flags, payload       segment.go
   │  Stack.Handle looks up its connection by (their IP, their port, our port)
   ▼
Conn        one connection's TCP state, inside the stack      stack.go
            state SYN_RECEIVED → ESTABLISHED, next seq / ack,
            sent-but-unacked segments to resend
   │  reaches ESTABLISHED: wrapped once
   ▼
StreamConn  the app's view of that Conn: Read, Write, Close   stream.go
   │  handed out by
   ▼
Listener    one per port: Accept returns the next StreamConn  listener.go

Segments carry bytes; Conn puts them in order; StreamConn lets an app read them; Listener is how the app gets one. Going out, it runs the other way: Write → Conn → new Segments.

AWS replaced TCP underneath: SRD

Apps still open TCP sockets. Underneath, AWS can carry the traffic with its own protocol.

• SRD (Scalable Reliable Datagram) runs on AWS’s Nitro network card
• Sprays one flow across many network paths
• Resends lost packets within microseconds, in hardware
• Delivers everything, but out of order; the layer above puts it back in order

TCP isn’t mandatory everywhere. Neither is TLS.

Where SRD sits under your TCP connection

your app     ── TCP socket          (API unchanged)
                     │
Nitro card   ── SRD                 multipath · fast retransmit
                     │              reliable, out-of-order
fabric       ── many paths at once
                     │
peer Nitro   ── reorder → hand TCP bytes up

With ENA Express, ordinary TCP traffic between instances rides on SRD without the app noticing. Because the layers are separate, AWS could replace the middle and leave both ends alone.

Next: the application

HTTP: the application layer, which gives the stream meaning: a request and a response.

Our http1 vs Go’s net/http

Ours shows what HTTP/1 is: text on a stream. Go’s is the one you’d ship.

Same question to strace, this time for mytcp

strace -f -e trace=%network mytcp -get https://www.google.com/ · inside Docker

socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 3
sendto(3, [{nlmsg_type=RTM_NEWLINK, ...}], 32, 0, ...) = 32
# ↑ the ip tool bringing tap0 up. Setup, not traffic.

# No socket(AF_INET). No connect. No sendto(5, "GET / …").
# ARP, DNS, TCP, TLS and HTTP all left as read and write
# on /dev/net/tun.

curl fetching the same page: 41 socket and connect lines. mytcp: none. The kernel is just the wire.

Optional: the security layer

TLS: secure transport. Nobody in between can read or change the bytes. It runs on the same TCP code, unchanged.

TLS takes a net.Conn and returns a net.Conn

http1.ServeConn(c)               the same HTTP code, with or without TLS
   │  Read / Write: plaintext
   ▼
TLS conn         net.Conn        tls.Conn (Go’s) or mintls.Conn (ours)
   │  Read / Write: records, a 5-byte header + encrypted payload
   ▼
tcp.StreamConn   net.Conn        our TCP
   │  segments
   ▼
tcp.Listener     net.Listener

The interface is the same above and below, so TLS is optional: slot it in or leave it out. HTTP can’t tell.

Go’s crypto/tls vs our mintls

curl needs --tls-max 1.2 to talk to mintls. Small on purpose, so you can read all of it.

TLS has layers of its own

┌──────────┬─────────────┬────────────┬──────────────────────┐
│ type (1) │ version (2) │ length (2) │ payload              │
└──────────┴─────────────┴────────────┴──────────────────────┘
  type: 20 ChangeCipherSpec  21 Alert  22 Handshake  23 AppData

internal/mintls/record.go

// every TLS record starts with a 5-byte header
hdr := make([]byte, 5)
io.ReadFull(r, hdr)                      // block until all 5 arrive
typ  = hdr[0]                            // which kind of TLS message
vers = binary.BigEndian.Uint16(hdr[1:3]) // 0x0303 means TLS 1.2
n   := binary.BigEndian.Uint16(hdr[3:5]) // how long the payload is

// then read exactly that many bytes
payload = make([]byte, n)
io.ReadFull(r, payload)                  // ← the next layer's message

The TLS 1.2 handshake: agreeing on keys

That’s two round trips before the first byte of HTTP. TLS 1.3 cuts it to one.

The four jobs cryptography does in TLS

Each row is a field of study. We only glue them together.

Both sides compute a secret without sending it

internal/mintls/conn.go · Handshake

// a fresh key pair, used for this connection only
ecdhe, _ := ecdh.P256().GenerateKey(rand.Reader)
// send our public half, signed with the certificate's key so the
// client knows it's really us (ch = the decoded ClientHello)
ske, _ := encodeServerKeyExchange(ch.random, serverRandom,
    ecdhe.PublicKey().Bytes(), cert.PrivateKey)
// the client sends its public half back; combining it with our
// private half gives a secret only the two of us can compute
clientPub, _ := ecdh.P256().NewPublicKey(clientPubBytes)
preMaster, _ := ecdhe.ECDH(clientPub)
// mix in both randoms to get the 48-byte master secret
master := masterSecret(preMaster, ch.random, serverRandom)

The key pair is thrown away after each connection, so stealing the certificate key later can’t decrypt recorded traffic. This is called forward secrecy.

Turning one secret into four keys

internal/mintls/prf.go

// prf12 = TLS 1.2's pseudo-random function: HMAC-SHA256 in a loop,
// stretching (secret, label, seed) into as many bytes as we ask for
// step 1: shared secret → 48-byte master secret
func masterSecret(preMaster, clientRandom, serverRandom []byte) []byte {
    seed := append(clientRandom, serverRandom...)
    return prf12(preMaster, "master secret", seed, 48)
}
// step 2: master secret → 40 bytes, cut into four pieces
func keyBlock(master, clientRandom, serverRandom []byte) (...) {
    seed := append(serverRandom, clientRandom...)
    block := prf12(master, "key expansion", seed, 40)
    return block[0:16], block[16:32], // AES key: client→server, server→client
        block[32:36], block[36:40]    // IV prefix: same two directions
}

The Finished message is derived the same way from a hash of the whole handshake, so any tampering is caught.

Encrypting a record with AES-GCM

internal/mintls/record.go · seal (paraphrased)

// ("||" means "followed by")
// the nonce must never repeat under one key; seq counts records 0, 1, 2…
nonce := fixedIV(4) || seq(8)
// extra data the tag protects, even though it stays readable
aad := seq(8) || type(1) || version(2) || length(2)
// encrypt, and append a 16-byte tag covering ciphertext and aad
ciphertext := h.aead.Seal(nil, nonce, plaintext, aad)
h.seq++
// what actually goes into the record
record payload = explicitNonce(8) || ciphertext || tag(16)

The header isn’t encrypted, but it is checked. Reordering, replaying or changing a single bit makes decryption fail.

The same GET, plain and encrypted

curl http://10.0.0.2/ · captures/talk-http.jsonl frame 7

00000000  02 00 00 00 00 02 ee ea  f3 0f 57 32 08 00 45 00  |..........W2..E.|
00000010  00 70 56 ec 40 00 40 06  cf 99 0a 00 00 01 0a 00  |.pV.@.@.........|
00000020  00 02 c6 8a 00 50 13 e8  5a 67 00 00 03 e9 50 18  |.....P..Zg....P.|
00000030  fa f0 91 b1 00 00 47 45  54 20 2f 20 48 54 54 50  |......GET / HTTP|
00000040  2f 31 2e 31 0d 0a 48 6f  73 74 3a 20 31 30 2e 30  |/1.1..Host: 10.0|
00000050  2e 30 2e 32 0d 0a 55 73  65 72 2d 41 67 65 6e 74  |.0.2..User-Agent|
00000060  3a 20 63 75 72 6c 2f 37  2e 38 38 2e 31 0d 0a 41  |: curl/7.88.1..A|
00000070  63 63 65 70 74 3a 20 2a  2f 2a 0d 0a 0d 0a        |ccept: */*....|
0000007e

curl -k https://10.0.0.2/ through mintls · captures/talk-https-diy.jsonl frame 31

00000000  02 00 00 00 00 02 aa 0b  aa 94 e9 f3 08 00 45 00  |..............E.|
00000010  00 8d 13 f3 40 00 40 06  12 76 0a 00 00 01 0a 00  |....@.@..v......|
00000020  00 02 ce 9e 01 bb 6f 3a  1b 39 00 00 06 54 50 18  |......o:.9...TP.|
00000030  f9 5a c9 13 00 00 17 03  03 00 60 91 a2 0a d9 8a  |.Z........`.....|
00000040  04 d7 9f 38 10 ff fe b5  c3 77 07 da 77 69 85 3c  |...8.....w..wi.<|
00000050  a0 87 8e 2c 7c e8 54 56  b3 a1 80 0b b3 b9 99 08  |...,|.TV........|
00000060  98 a4 48 41 01 4f 6e d8  1e d1 92 9b 37 44 2d 7a  |..HA.On.....7D-z|
00000070  f6 8a 88 66 45 c2 08 88  6b 41 60 40 27 02 1a a3  |...fE...kA`@'...|
00000080  73 52 91 8a 84 3c 50 cd  88 7e 9a ad 26 5f 2a 1c  |sR...<P..~..&_*.|
00000090  e5 ab d3 8a 80 39 34 71  bf 80 c9                 |.....94q...|
0000009b

Ethernet IPv4 TCP HTTP (72) TLS header: 17 = app data, 03 03 = TLS 1.2, 00 60 = 96 nonce (8) ciphertext (72) tag (16)

Bytes 0–53 have the same layout; only lengths, IDs, ports, sequence numbers, checksums and tap0’s MAC differ. 96 = 8 + 72 + 16: the ciphertext is as long as the plain GET, so encryption hides the content, not the size.

What we wrote vs what we borrowed

Build the protocol to learn it. Never write your own cipher code.

What mintls leaves out, and what TLS 1.3 changed

Rabbit holes elliptic-curve math · AEAD design · PKI, cert chains, Certificate Transparency · side channels · post-quantum key exchange (hybrid ML-KEM)

Every stage ended with a working demo

We didn’t start a layer until the one below it worked.

Where each layer lives in the repo

Demo commands

# start the Linux lab container and build mytcp inside it
make lab-up && make lab-build

# shell A: run our stack on tap0 serving HTTP, and save a packet capture
/tmp/mytcp -i tap0 -app http -dump=false -pcap captures/latest
# same Listener, Go’s server instead:  -app http-go

# shell B: talk to it with ordinary tools
ping -c 3 10.0.0.2       # exercises ARP + ICMP
curl http://10.0.0.2/    # ARP + IP + TCP + HTTP

# HTTPS: restart shell A with -app https (Go’s TLS), -app https-go
# (all stdlib above TCP) or -app https-diy (our mintls), then
# (-k accepts our self-signed cert; mintls only speaks TLS 1.2)
curl -k --tlsv1.2 --tls-max 1.2 https://10.0.0.2/
/tmp/mytcp -i tap0 -get https://www.google.com/   # other way: Dial, via NAT

What you just saw

• ping = ARP + ICMP
• curl = ARP + IP + TCP + HTTP
• curl -k = the same path, with TLS between TCP and HTTP
• mytcp -get = the same layers, with us asking first

Great systems are built by many people

Every layer we lean on is the sum of decades of shared work.

• Standards (RFCs) agreed between rival vendors, often after failures
• One web request passes through a NIC, a kernel, a TLS library, curl and a CDN
• Our few thousand lines stand on millions of hours

That isn’t a caveat. It’s the point.

Good protocols leave room for what comes next

They didn’t predict the future. They drew clean boundaries that later ideas could plug into.

What could plug in next

• Real congestion control for our TCP
• A toy SRD: reliable datagrams over several paths, delivered out of order
• UDP, and QUIC on top of it, next to TCP
• Tracing each packet where one layer hands off to the next
• TLS 1.3, or keep mintls as a simple TLS 1.2 example

None of these need changes to Ethernet, ARP or IP.

Every layer is a discipline

Peeling the onion once gives you a map. Pick a layer and keep digging.

Takeaways

1. Build it once — the “magic” turns into headers you can read
2. Stand on giants — every layer is decades of shared work
3. Clean boundaries beat prediction — they let SRD, TLS and QUIC plug in
4. Same shape everywhere — header, Decode / Encode, payload
5. Pick a layer and dig — each one is a whole discipline