codex review and fixes
This commit is contained in:
+93
-184
@@ -1,204 +1,113 @@
|
||||
# lanspread-peer
|
||||
|
||||
A peer-to-peer networking component for the Lanspread system that enables distributed game sharing and discovery across local networks using QUIC protocol and mDNS service discovery.
|
||||
`lanspread-peer` is the networking runtime that lets Lanspread nodes find each
|
||||
other on the local network, exchange library metadata, and transfer game files.
|
||||
It is designed to run headless – other crates (most notably
|
||||
`lanspread-tauri-deno-ts`) embed it and drive it through a channel-based API.
|
||||
|
||||
## Overview
|
||||
## Runtime Overview
|
||||
|
||||
The `lanspread-peer` crate implements a peer-to-peer networking layer that allows multiple instances of Lanspread to discover each other on the local network, share game libraries, and distribute game files efficiently. It operates as both a server (providing local games to other peers) and a client (discovering and downloading games from other peers).
|
||||
- `start_peer(game_dir, tx_events)` boots the asynchronous runtime in the
|
||||
background and returns an `UnboundedSender<PeerCommand>` that the caller uses
|
||||
for control. The function immediately forwards the supplied game directory via
|
||||
`PeerCommand::SetGameDir`.
|
||||
- `PeerCommand` represents the small control surface exposed to the UI layer:
|
||||
`ListGames`, `GetGame`, `DownloadGameFiles`, and `SetGameDir`.
|
||||
- `PeerEvent` enumerates everything the peer runtime reports back to the UI:
|
||||
library snapshots, download lifecycle updates, and peer membership changes.
|
||||
- `PeerGameDB` collects remote peer metadata. It aggregates discovered peers’
|
||||
`Game` definitions, tracks the latest ETI version per title, and keeps the
|
||||
last seen list of `GameFileDescription` entries for each peer.
|
||||
|
||||
## Core Architecture
|
||||
Internally the peer runtime owns three long-lived tasks that run for the
|
||||
lifetime of the process:
|
||||
|
||||
### Main Components
|
||||
1. **Server component** (`run_server_component`) – listens for QUIC connections,
|
||||
advertises via mDNS, and serves `Request::ListGames`, `Request::GetGame`,
|
||||
`Request::GetGameFileData`, and `Request::GetGameFileChunk` by reading from
|
||||
the local game directory.
|
||||
2. **Discovery loop** (`run_peer_discovery`) – uses the `lanspread-mdns`
|
||||
helper to discover other peers. The blocking mDNS work is executed on a
|
||||
dedicated thread via `tokio::task::spawn_blocking` so that the Tokio runtime
|
||||
remains responsive.
|
||||
3. **Ping service** (`run_ping_service`) – periodically issues QUIC ping requests
|
||||
to keep peer liveness up to date and prunes stale entries from `PeerGameDB`.
|
||||
|
||||
1. **Peer System (`run_peer`)**: The main orchestrator that manages all peer operations
|
||||
2. **Server Component (`run_server_component`)**: Handles incoming connections from other peers
|
||||
3. **Peer Discovery (`run_peer_discovery`)**: Continuously discovers new peers via mDNS
|
||||
4. **Ping Service (`run_ping_service`)**: Maintains peer connectivity through health checks
|
||||
5. **Download Manager**: Handles parallel file downloads from multiple peers
|
||||
`load_local_game_db` scans the configured game directory (looking for folders
|
||||
with a `version.ini`) and hydrates a `GameDB`. That database is used to respond
|
||||
to incoming metadata requests (`Request::ListGames` / `Request::GetGame`).
|
||||
|
||||
### Key Data Structures
|
||||
## Networking and File Transfer
|
||||
|
||||
- **`PeerGameDB`**: Central database managing all discovered peers and their game collections
|
||||
- **`PeerInfo`**: Contains information about a single peer including address, last seen time, and available games
|
||||
- **`PeerEvent`**: Events sent to the UI layer (peer discovery, connection status, download progress)
|
||||
- **`PeerCommand`**: Commands received from the UI layer (list games, download, etc.)
|
||||
- Transport is handled by [`s2n-quic`](https://github.com/aws/s2n-quic); TLS
|
||||
cert/key material is compiled in from the repository root.
|
||||
- Protocol messages are JSON-encoded structures defined in
|
||||
`lanspread-proto::{Request, Response}`.
|
||||
- File transfers stream raw bytes over dedicated bidirectional QUIC streams.
|
||||
`peer::send_game_file_data` sends entire files, while
|
||||
`peer::send_game_file_chunk` services ranged requests.
|
||||
|
||||
## Network Protocol
|
||||
### Download Pipeline
|
||||
|
||||
### Communication Layer
|
||||
When the UI asks to download a game:
|
||||
|
||||
The crate uses **QUIC** (via `s2n-quic`) for all peer-to-peer communication with TLS encryption:
|
||||
1. The UI first issues `PeerCommand::GetGame`. Each peer that still reports the
|
||||
game is queried via `request_game_details_from_peer`, and their file
|
||||
manifests are merged inside `PeerGameDB`.
|
||||
2. Once the UI receives `PeerEvent::GotGameFiles`, it forwards the selected file
|
||||
list back with `PeerCommand::DownloadGameFiles`.
|
||||
3. `download_game_files` prepares the filesystem (creating directories and
|
||||
pre-sizing files where possible), emits `PeerEvent::DownloadGameFilesBegin`,
|
||||
and builds a per-peer plan (`build_peer_plans`) that round-robins file chunks
|
||||
across the available peers that advertise the latest version.
|
||||
4. Each plan is executed in its own task (`download_from_peer`). Chunk requests
|
||||
use per-chunk QUIC streams and write into pre-created files. The chunk writer
|
||||
keeps existing data intact and only truncates when we intentionally fall back
|
||||
to a full file transfer, which prevents corruption when multiple peers fill
|
||||
different regions of the same file.
|
||||
5. Failures are accumulated and retried (up to `MAX_RETRY_COUNT`) via
|
||||
`retry_failed_chunks`. If everything succeeds,
|
||||
`PeerEvent::DownloadGameFilesFinished` is emitted; otherwise the UI receives
|
||||
`PeerEvent::DownloadGameFilesFailed`.
|
||||
|
||||
- **Certificate-based security**: Uses embedded TLS certificates for secure communication
|
||||
- **Bidirectional streams**: Each peer interaction uses separate QUIC streams
|
||||
- **Connection management**: Built-in connection pooling and keep-alive mechanisms
|
||||
## Integration with `lanspread-tauri-deno-ts`
|
||||
|
||||
### Message Protocol
|
||||
The Tauri application embeds this crate in
|
||||
`crates/lanspread-tauri-deno-ts/src-tauri/src/lib.rs`:
|
||||
|
||||
Messages are serialized using JSON and follow the request/response pattern defined in `lanspread-proto`:
|
||||
- `LanSpreadState` holds onto the peer control channel, the latest aggregated
|
||||
`GameDB`, per-game download state, and the user-selected game directory.
|
||||
- The Tauri commands (`request_games`, `install_game`, `update_game`, and
|
||||
`update_game_directory`) translate UI actions into `PeerCommand`s. In
|
||||
particular, `update_game_directory` records the filesystem path, kicks off the
|
||||
peer runtime on first use, and mirrors the installed/uninstalled state into
|
||||
the UI-facing database.
|
||||
- A background task consumes `PeerEvent`s and fans them out to the front-end via
|
||||
Tauri publish/subscribe events (`games-list-updated`, `game-download-*`,
|
||||
`peer-*`). Successful downloads trigger an `unrar` sidecar to unpack ETI
|
||||
archives and clean up the temporary backup folders that are created when
|
||||
updates begin.
|
||||
- When downloads fail the Tauri layer restores the on-disk backup, keeping the
|
||||
previous installation consistent even after partial transfers.
|
||||
|
||||
**Request Types:**
|
||||
- `Ping`: Health check
|
||||
- `ListGames`: Request available games from a peer
|
||||
- `GetGame`: Request detailed file list for a specific game
|
||||
- `GetGameFileData`: Request complete file download
|
||||
- `GetGameFileChunk`: Request specific file chunk for parallel downloads
|
||||
## Security & Operational Notes
|
||||
|
||||
**Response Types:**
|
||||
- `Pong`: Health check response
|
||||
- `ListGames(Vec<Game>)`: List of available games
|
||||
- `GetGame`: Game file descriptions
|
||||
- `GameNotFound`: Game not available
|
||||
- Error responses for various failure conditions
|
||||
- All QUIC connections are TLS encrypted; the shipped certificates are suitable
|
||||
for local-network trust but should be rotated for production deployments.
|
||||
- Peer discovery is restricted to the local link via mDNS.
|
||||
- Long-running blocking mDNS calls are isolated on dedicated threads which keeps
|
||||
the async runtime responsive even when discovery takes a long time.
|
||||
- File writes are chunk-safe: partial chunk downloads now open files without
|
||||
truncating existing data, avoiding the corruption that occurred previously
|
||||
when multiple peers collectively filled a file.
|
||||
|
||||
## Peer Discovery
|
||||
## Known Limitations
|
||||
|
||||
### mDNS Integration
|
||||
- `PeerGameDB` currently models the latest metadata that other peers advertise.
|
||||
If the UI needs to surface titles that only exist locally, additional merging
|
||||
with the locally scanned `GameDB` will be required.
|
||||
- The download planner uses a simple round-robin and does not yet take per-peer
|
||||
throughput or failures into account when distributing work.
|
||||
|
||||
The crate uses mDNS (multicast DNS) for automatic peer discovery:
|
||||
|
||||
- **Service Type**: `_lanspread._udp.local.`
|
||||
- **Instance Naming**: `{hostname}-{uuid}` format to ensure uniqueness
|
||||
- **Continuous Discovery**: Scans every 10 seconds for new peers
|
||||
- **Automatic Advertising**: Each peer advertises its availability via mDNS
|
||||
|
||||
### Discovery Process
|
||||
|
||||
1. **mDNS Browser**: Continuously scans for `_lanspread._udp.local.` services
|
||||
2. **Peer Registration**: New peers are added to `PeerGameDB`
|
||||
3. **Game Synchronization**: Automatically requests game lists from newly discovered peers
|
||||
4. **UI Notification**: Notifies the UI layer about peer discovery events
|
||||
|
||||
## Game Distribution System
|
||||
|
||||
### File Management
|
||||
|
||||
The system supports efficient game file distribution with these features:
|
||||
|
||||
**File Description:**
|
||||
- `GameFileDescription`: Metadata for each file including path, size, and directory structure
|
||||
- **Version Tracking**: Reads `version.ini` files for ETI game version management
|
||||
- **Directory Structure**: Preserves complete game directory hierarchies
|
||||
|
||||
**Download Strategies:**
|
||||
1. **Chunked Downloads**: Large files are split into 512KB chunks for parallel processing
|
||||
2. **Multi-peer Downloads**: Chunks are distributed across available peers for maximum throughput
|
||||
3. **Retry Mechanism**: Failed chunks are retried up to 3 times with different peers
|
||||
4. **Integrity Verification**: File integrity is verified after download completion
|
||||
|
||||
### Download Process
|
||||
|
||||
```rust
|
||||
// Simplified download flow
|
||||
1. Prepare local storage (create directories and pre-allocate files)
|
||||
2. Build download plan (distribute chunks across available peers)
|
||||
3. Execute parallel downloads from multiple peers
|
||||
4. Retry failed chunks with alternative peers
|
||||
5. Verify file integrity and notify completion
|
||||
```
|
||||
|
||||
## Concurrency Model
|
||||
|
||||
### Async Architecture
|
||||
|
||||
The system uses Tokio's async runtime with these concurrent tasks:
|
||||
|
||||
1. **Main Peer Loop**: Handles UI commands and orchestrates operations
|
||||
2. **Server Task**: Accepts incoming peer connections
|
||||
3. **Discovery Task**: Continuously scans for new peers
|
||||
4. **Ping Service**: Maintains peer health monitoring
|
||||
5. **Connection Handlers**: Spawns per-connection tasks for request handling
|
||||
6. **Download Tasks**: Parallel download workers for file transfers
|
||||
|
||||
### Thread Safety
|
||||
|
||||
- **Arc<RwLock<T>>**: Used for shared data structures to enable concurrent reads
|
||||
- **Message Passing**: Uses Tokio channels (`UnboundedSender/Receiver`) for task communication
|
||||
- **Connection Isolation**: Each peer connection is handled in an independent task
|
||||
|
||||
## Integration Points
|
||||
|
||||
### UI Interface
|
||||
|
||||
The crate communicates with the UI layer through two channels:
|
||||
|
||||
**Events to UI (`PeerEvent`):**
|
||||
- Game list updates
|
||||
- Peer discovery/connection events
|
||||
- Download progress and completion
|
||||
- Error notifications
|
||||
|
||||
**Commands from UI (`PeerCommand`):**
|
||||
- Set game directory
|
||||
- List available games
|
||||
- Request game details
|
||||
- Initiate game downloads
|
||||
|
||||
### Dependency Integration
|
||||
|
||||
- **`lanspread-db`**: Game metadata and file description structures
|
||||
- **`lanspread-proto`**: Message serialization and protocol definitions
|
||||
- **`lanspread-mdns`**: Service discovery functionality
|
||||
- **`lanspread-utils`**: Common utilities and macros
|
||||
|
||||
## Configuration Constants
|
||||
|
||||
```rust
|
||||
const CHUNK_SIZE: u64 = 512 * 1024; // 512KB file chunks
|
||||
const MAX_RETRY_COUNT: usize = 3; // Maximum download retries
|
||||
const PING_INTERVAL: Duration = 10s; // Peer health check interval
|
||||
const DISCOVERY_INTERVAL: Duration = 10s; // mDNS scan interval
|
||||
const STALE_TIMEOUT: Duration = 30s; // Peer inactivity timeout
|
||||
```
|
||||
|
||||
## Usage Example
|
||||
|
||||
```rust
|
||||
// Start the peer system
|
||||
let (tx_notify_ui, rx_notify_ui) = tokio::sync::mpsc::unbounded_channel();
|
||||
let tx_control = lanspread_peer::start_peer(
|
||||
"/path/to/games".to_string(),
|
||||
tx_notify_ui,
|
||||
)?;
|
||||
|
||||
// Send commands to the peer
|
||||
tx_control.send(PeerCommand::ListGames)?;
|
||||
tx_control.send(PeerCommand::DownloadGameFiles {
|
||||
id: "game_id".to_string(),
|
||||
file_descriptions: vec![/* file descriptions */],
|
||||
})?;
|
||||
|
||||
// Receive events from the peer
|
||||
while let Some(event) = rx_notify_ui.recv().await {
|
||||
match event {
|
||||
PeerEvent::ListGames(games) => println!("Available games: {:?}", games),
|
||||
PeerEvent::DownloadGameFilesFinished { id } => println!("Downloaded: {}", id),
|
||||
// Handle other events...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
The crate implements comprehensive error handling:
|
||||
|
||||
- **Connection Errors**: Automatic reconnection and peer removal
|
||||
- **Download Failures**: Retry with alternative peers and fallback strategies
|
||||
- **Protocol Errors**: Graceful degradation and error reporting
|
||||
- **File System Errors**: Proper cleanup and rollback on failures
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **TLS Encryption**: All peer communication is encrypted using QUIC with TLS
|
||||
- **Certificate Validation**: Uses embedded certificates for peer authentication
|
||||
- **Network Isolation**: Operates only on local network via mDNS discovery
|
||||
- **File Access**: Restricted to configured game directory boundaries
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
- **Parallel Downloads**: Maximizes bandwidth by downloading from multiple peers simultaneously
|
||||
- **Chunked Transfer**: Large files are split for parallel processing and resume capability
|
||||
- **Connection Reuse**: Maintains persistent connections to reduce handshake overhead
|
||||
- **Incremental Discovery**: Only synchronizes changed game metadata
|
||||
|
||||
This crate provides the foundation for Lanspread's distributed game sharing capabilities, enabling efficient peer-to-peer game distribution across local networks.
|
||||
Refer to the source (particularly `src/lib.rs`) for the exact message shapes and
|
||||
state machines.
|
||||
|
||||
@@ -453,10 +453,14 @@ async fn download_chunk(
|
||||
let path = base_dir.join(&chunk.relative_path);
|
||||
let mut file = OpenOptions::new()
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.write(true)
|
||||
.truncate(false)
|
||||
.open(&path)
|
||||
.await?;
|
||||
if chunk.length == 0 && chunk.offset == 0 {
|
||||
// fallback-to-whole-file path replaces any existing partial data
|
||||
file.set_len(0).await?;
|
||||
}
|
||||
file.seek(std::io::SeekFrom::Start(chunk.offset)).await?;
|
||||
|
||||
let mut remaining = chunk.length;
|
||||
@@ -1252,8 +1256,11 @@ async fn run_peer_discovery(
|
||||
log::info!("Starting peer discovery task");
|
||||
|
||||
loop {
|
||||
match discover_service(LANSPREAD_SERVICE_TYPE) {
|
||||
Ok(peer_addr) => {
|
||||
let discovery_result =
|
||||
tokio::task::spawn_blocking(|| discover_service(LANSPREAD_SERVICE_TYPE)).await;
|
||||
|
||||
match discovery_result {
|
||||
Ok(Ok(peer_addr)) => {
|
||||
log::info!("Discovered peer at: {peer_addr}");
|
||||
|
||||
// Add peer to database
|
||||
@@ -1290,12 +1297,15 @@ async fn run_peer_discovery(
|
||||
});
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
Ok(Err(e)) => {
|
||||
log::debug!("Peer discovery error: {e}");
|
||||
tokio::time::sleep(Duration::from_secs(5)).await;
|
||||
}
|
||||
Err(e) => {
|
||||
log::error!("Peer discovery join error: {e}");
|
||||
tokio::time::sleep(Duration::from_secs(5)).await;
|
||||
}
|
||||
}
|
||||
|
||||
// Wait before next discovery cycle
|
||||
tokio::time::sleep(Duration::from_secs(10)).await;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user