This is an automated archive made by the Lemmit Bot.

The original was posted on /r/rust by /u/ActiveStress3431 on 2025-12-21 22:29:43+00:00.

---

Hi r/rust,

I’m sharing a project I’ve been working on called Parcode.

Parcode is a persistence library for Rust designed for true lazy access to data structures. The goal is simple: open a large persisted object graph and access any specific field, record, or asset without deserializing the rest of the file.

The problem

Most serializers (Bincode, Postcard, etc.) are eager by nature. Even if you only need a single field, you pay the cost of deserializing the entire object graph. This makes cold-start latency and memory usage scale with total file size.

The idea

Parcode uses Compile-Time Structural Mirroring:

• The Rust type system itself defines the storage layout
• Structural metadata is loaded eagerly (very small)
• Large payloads (Vecs, HashMaps, assets) are stored as independent chunks
• Data is only materialized when explicitly requested

No external schemas, no IDLs, no runtime reflection.

What this enables

• Sub-millisecond cold starts
• Constant memory usage during traversal
• Random access to any field inside the file
• Explicit control over what gets loaded

Example benchmark (cold start + targeted access)

Serializer	Cold Start	Deep Field	Map Lookup	Total
Parcode	~1.4 ms	~0.00002 ms	~0.00016 ms	~1.4 ms + p-t
Cap’n Proto	~60 ms	~0.00005 ms	~4.3 µs	~60 ms + p-t
Postcard	~80 ms	~0.00002 ms	~0.00002 ms	~80 ms + p-t
Bincode	~299 ms	~0.00001 ms	~0.000002 ms	~299 ms + p-t

p-t: per-target

The key difference is that Parcode avoids paying the full deserialization cost when accessing small portions of large files.

Quick example

use parcode::{Parcode, ParcodeObject};
use serde::{Serialize, Deserialize};
use std::collections::HashMap;

// The ParcodeObject derive macro analyzes this struct at compile-time and 
// generates a "Lazy Mirror" (shadow struct) that supports deferred I/O.
#[derive(Serialize, Deserialize, ParcodeObject)]
struct GameData {
    // Standard fields are stored "Inline" within the parent chunk.
    // They are read eagerly during the initial .root() call.
    version: u32,

    // #[parcode(chunkable)] tells the engine to store this field in a 
    // separate physical node. The mirror will hold a 16-byte reference 
    // (offset/length) instead of the actual data.
    #[parcode(chunkable)]
    massive_terrain: Vec<u8>,

    // #[parcode(map)] enables "Database Mode". The HashMap is sharded 
    // across multiple disk chunks based on key hashes, allowing O(1) 
    // lookups without loading the entire collection.
    #[parcode(map)]
    player_db: HashMap<u64, String>,
}

fn main() -> parcode::Result<()> {
    // Opens the file and maps only the structural metadata into memory.
  // Total file size can be 100GB+; startup cost remains O(1).
    let file = Parcode::open("save.par")?;

    // .root() projects the structural skeleton into RAM.
    // It DOES NOT deserialize massive_terrain or player_db yet.
    let mirror = file.root::<GameData>()?;

    // Instant Access (Inline data):
    // No disk I/O triggered; already in memory from the root header.
    println!("File Version: {}", mirror.version);

    // Surgical Map Lookup (Hash Sharding):
    // Only the relevant ~4KB shard containing this specific ID is loaded.
    // The rest of the player_db (which could be GBs) is NEVER touched.
    if let Some(name) = mirror.player_db.get(&999)? {
        println!("Player found: {}", name);
    }

    // Explicit Materialization:
    // Only now, by calling .load(), do we trigger the bulk I/O 
    // to bring the massive terrain vector into RAM.
    let terrain = mirror.massive_terrain.load()?;

    Ok(())
}

Trade-offs

• Write throughput is currently lower than pure sequential formats
• The design favors read-heavy and cold-start-sensitive workloads
• This is not a replacement for a database

Repo

Parcode

Whis whitepaper explain the Compile-Time Structural Mirroring (CTSM) architecture.

Also you can add and test using cargo add parcode.

I’d love feedback, questions, or criticism — especially around the design, trade-offs or any.