Add initial TCP reconstruction implementation

changelog.md

@@ -2,6 +2,21 @@
 
 ## Unreleased
 
+* Added TCP stream reassembly support (in the `tcp_reassembly` module, contains allocations):
+  * `TcpStreamReassemblyBuf` (requires the `alloc` feature), a re-usable buffer that reconstructs the payload byte stream of a single direction of a TCP connection. It is robust against retransmits, re-ordered & duplicated/overlapping segments, stale or out-of-window FINs (which a real receiver would reject, e.g. blindly injected segments), big offset jumps and 32 bit sequence number wrap-around (all bookkeeping is done in a monotonic 64 bit "absolute stream offset" space). Data is exposed in-place via `contiguous` and freed via `consume` ("collect then clean", amortized O(1) via deferred compaction). Streams whose start was never observed (capture started mid-connection) are reconstructed from the first seen segment and can be identified via `syn_observed`.
+  * `TcpStreamReassemblyBuf::add_segment` additionally interprets the `SYN` flag (establishing the stream start, recognizing duplicated, re-transmitted & late SYNs and reporting the SYN of a new connection re-using the same ports via `TcpSegmentOutcome::NewConnection`), while `add` just takes the payload of a segment.
+  * Bytes that are not part of the stream are dropped instead of being handed out: data before the read cursor, data behind a received `FIN` and data in front of a *known* stream start. Overlapping segments carrying differing content are resolved deterministically ("first writer wins") so a reassembly cannot be de-synchronized from the actual receiver by overlapping re-transmits.
+  * `TcpStreamReassemblyBuf::skip_gap` gives up on segments that were lost and never re-transmitted (e.g. dropped by the capture) by advancing the read cursor to the next received data section, which would otherwise stall the stream forever.
+  * `TcpStreamReassemblyBuf::drain` (and `drain_unacked`) hand the available data to a closure and consume the number of bytes it reports as processed, so a parser that can only handle complete messages can keep the remainder buffered. The raw buffer behind the reconstruction (including the zero filled gaps of not yet received ranges) stays accessible via `raw_buffer` together with `sections`.
+  * By default (`TcpAckPolicy::Required`) only data that the receiver acknowledged is handed out by `contiguous`, so segments that never reached the receiver (lost or rejected after the point of capture) do not end up in the reconstructed stream. The acknowledgment numbers travel in the opposite direction of the data and are fed via `TcpStreamReassemblyBuf::add_ack` (stale & nonsensical acknowledgments are ignored). As a capture usually ends before the last data is acknowledged, the trailing unacknowledged bytes stay reachable via `contiguous_unacked` & `consume_unacked`, `ack_observed` identifies streams that never saw an acknowledgment (e.g. one-way captures) and `TcpAckPolicy::Ignore` disables the behavior entirely.
+  * `TcpStreamReassemblyPool<Timestamp, CustomChannelId>` (requires the `std` feature), a pool that reassembles many connections in parallel while re-using the underlying buffers to minimize allocations (mirrors `defrag::IpDefragPool`). Both directions of a connection are tracked together as a `TcpConnection`, which is what allows the acknowledgment numbers of one direction to release the data of the other one. It interprets the TCP control flags automatically (`SYN` establishes/replaces a connection — duplicated & late SYNs are recognized and keep the buffered state, `ACK` releases the data of the reverse direction, `FIN` marks the stream end, `RST` ends the whole connection) and reports the outcome per packet via `TcpReassemblyEvent` (`Ignored` / `Segment` / `Closed`, where `Segment` exposes both the stream the payload was added to and the stream the segment acknowledged, and `Closed` still exposes never consumed leftover data; both carry the `TcpConnectionId` the connection is tracked under, which requires `Clone` on the custom channel id for the `process_*` methods). Connections can be enumerated & drained via `iter_mut` and evicted via the id, connection & timestamp aware `retain`.
+  * Connections whose end is never observed (no `FIN`, no `RST`, e.g. because one side crashed or the capture does not contain the rest of them) would occupy memory forever, so `TcpStreamReassemblyPool::evict_older_than` (and `evict_older_than_with`, which hands the discarded connections to a closure so their leftover data can be drained first) discards the connections that have been inactive for longer than a caller chosen timeout, based on the `Timestamp` the segments were processed with (`last_activity` exposes it per connection). Every segment counts as activity, also the ones that carry no data (keep alive probes, the acknowledgments answering them, window updates & re-transmits), so a connection that is idle on the application layer but alive on the wire is not discarded. `DEFAULT_TCP_CONNECTION_TIMEOUT` (10 minutes) documents how to choose the timeout.
+  * A `RST` carrying a sequence number outside of the tracked window is ignored, so a blindly injected `RST` cannot tear down a connection.
+  * `TcpConnection::is_closed` reports connections whose both directions received everything up to their `FIN` and `TcpConnection::is_bidirectional` identifies connections for which only one direction was observed (which under `TcpAckPolicy::Required` never hand out data). Packets parsed without length & consistency checks can be fed via `TcpStreamReassemblyPool::process_lax_sliced_packet` / `TcpSegmentInfo::from_lax_sliced_packet`.
+  * Connections are identified by a `TcpConnectionId` (VLAN ids, the two `TcpEndpoint`s in a canonical order and a custom channel id) together with a `TcpDirection`, so both directions of a connection map to the same identifier. `TcpSegmentInfo::from_sliced_packet` extracts everything the reassembly needs from a parsed packet and can be modified before being passed to `process_tcp` (e.g. clearing the VLAN ids and moving them into the channel id if the two directions of a connection are tagged differently). For segments that do not arrive as one parsable packet there are `TcpSegmentInfo::from_tcp_slice` (assembles the info from a `TcpSlice` plus the IP addresses & VLAN ids) and `TcpSegmentInfo::from_defragmented_payload` (takes the re-assembled IP payload of a fragmented packet together with any of its fragments, for combining `defrag::IpDefragPool` with the TCP reassembly).
+  * Per-stream memory is bounded via a configurable `max_capacity` (buffered bytes ahead of the read cursor) and `max_sections` (tracked non-contiguous ranges), returning `TcpReassembleError` when a limit would be exceeded (the buffer is left unchanged in that case). The buffers kept around for re-use are bounded in count & size via `TcpStreamReassemblyPool::with_pool_limits` (so a burst of connections or a single big stream does not make the pool hold on to the memory forever) and the number of parallel connections can be limited via `TcpStreamReassemblyPool::with_max_connections` (`TcpReassembleError::TooManyConnections`).
+  * Supporting types `TcpConnection`, `TcpConnectionId`, `TcpDirection`, `TcpEndpoint`, `TcpSegmentInfo`, `TcpStreamRange`, `TcpSegmentOutcome`, `TcpAckPolicy`, `TcpReassembleError` and the `DEFAULT_MAX_TCP_STREAM_CAPACITY`, `DEFAULT_MAX_TCP_STREAM_SECTIONS`, `DEFAULT_MAX_TCP_POOLED_BUFS`, `DEFAULT_MAX_TCP_POOLED_BUF_CAPACITY` & `DEFAULT_TCP_CONNECTION_TIMEOUT` constants, plus a `tcp_reassembly` example and a `pcap_tcp_reassembly` example that reads a classic pcap file and reconstructs the TCP streams of all connections including fragmented IP packets (combining `defrag::IpDefragPool` & `TcpStreamReassemblyPool`), with a `--metadata` option to print a per connection summary instead of the stream contents (the stream data itself is printed as text if it decodes as UTF-8 - with terminal control characters escaped - and as a hex dump otherwise) and a `--timeout` option controlling when inactive connections are discarded.
+
 * Added zero-copy IGMP support (IGMPv1, IGMPv2 & IGMPv3):
   * `IgmpSlice`, an enum with one zero-copy variant per message type (`MembershipQuery`, `MembershipQueryWithSources`, `MembershipReportV1`, `MembershipReportV2`, `MembershipReportV3`, `LeaveGroup` & `Unknown`), plus common accessors (`header`, `payload`, `slice`, `checksum`, `is_checksum_valid`).
   * Per-variant slice types (`MembershipQuerySlice`, `MembershipQueryWithSourcesSlice`, `MembershipReportV1Slice`, `MembershipReportV2Slice`, `MembershipReportV3Slice`, `LeaveGroupSlice` & `IgmpUnknownSlice`) with typed field accessors. Variable-length data is exposed where it exists: `MembershipReportV3Slice::group_records` and `MembershipQueryWithSourcesSlice::source_addresses` / `source_addrs_bytes` (no longer `Option`-returning).

etherparse/Cargo.toml

@@ -28,7 +28,9 @@ std = ["alloc", "arrayvec/std"]
 arrayvec = { version = "0.7.2", default-features = false }
 
 [dev-dependencies]
+clap = { version = "4.5.61", features = ["derive"] }
 proptest = "1.4.0"
+rpcap = "1.0.0"
 
 [package.metadata.docs.rs]
 all-features = true