Contact
Research project · under heavy development

A data-to-GPU highway.

Kylix is research into a data-to-GPU highway for Rust. Model weights and data move from storage and the network straight into GPU memory, with no proxy and no extra copy, and Tokio applications drive NVIDIA CUDA directly. One design, from small edge devices to multi-GPU clusters.

See the bottlenecks Get in touch
  • Storage to GPU
  • Network to GPU
  • Tokio to CUDA
Designed for Edge devices from Jetson Orin Nano class Multi-GPU servers GPU clusters
The problem

Where loading time goes today.

On the way from storage to the GPU, model weights and inference data usually make several stops. Each stop adds a copy, CPU work or waiting, and the GPU waits until the data has arrived. These are the bottlenecks this research goes after.

Loading a model

  1. Object storeremote bucketBottlenecks12
  2. Local diskstaging copyBottleneck3
  3. Page cachekernel memoryBottleneck4
  4. CPUparse, convert, sliceBottlenecks56
  5. Pinned bufferhost stagingBottleneck7
  6. GPU memoryfinally usable

Feeding data to a running model

  1. Storage or networkfiles, servicesBottlenecks18
  2. Socket bufferskernel memoryBottleneck9
  3. CPUdecode, re-packBottleneck5
  4. Pinned bufferhost stagingBottleneck7
  5. GPU memorycontext fills upBottleneck10

Serving at scale

  1. Stored modelsversions, replicasBottlenecks1112
  2. One central storeshared by everyoneBottleneck13
  3. Many GPU nodesstarting at once

The research target

  1. Storage or networklocal disk or other nodes
  2. GPU memoryone hop, no proxy, no host copy
  1. A proxy in the pathBytes pass through a gateway or proxy server, which adds a hop, a copy and that server's CPU limit.
  2. One stream at a timeA single connection or sequential reads often use only a fraction of a fast network card or an array of SSDs.
  3. Download, then loadThe whole file is written to local disk before loading starts. Nothing overlaps, and every byte is written once and read again.
  4. Kernel copyOrdinary reads copy every byte from the operating system's page cache into the application's memory.
  5. CPU unpackingTensors and inputs are parsed, converted or re-laid out on the CPU, often on one thread, while the GPU waits.
  6. Slicing at load timeCheckpoints are not laid out per GPU, so each GPU's share is cut out while loading, from slices that are not aligned for direct reads.
  7. Bounce bufferThe GPU's copy engines read only from page-locked memory, so ordinary memory is first copied into a staging buffer before it crosses PCIe.
  8. Small filesDatasets and retrieval corpora kept as millions of small files spend their time on lookups, not on moving bytes.
  9. Socket copyData arriving over the network is copied out of kernel buffers into the application before it can go to the GPU.
  10. Context overflowWhen the cached context (the KV cache) of long or many conversations outgrows GPU memory, it is dropped and recomputed, or reloaded through the same slow path.
  11. Copies of the same bytesModel versions that share identical weights are stored and moved again in full.
  12. Three full replicasData at rest is often protected by three complete copies, which triples the raw capacity it needs.
  13. Everyone pulls at onceWhen many GPUs start the same model, they all download it from one central store, which becomes the choke point.
The research

Remove the stops, not just speed them up.

Everything below is a design target of an active research project. The numbers are goals the design is measured against.

The data-to-GPU path

  • Storage to GPU

    Weights are stored the way GPUs read them, so each GPU loads only its own share as a plain read: no page-cache copy and no unpacking on the CPU. On servers it is designed for GPUDirect Storage, from NVMe straight into GPU memory. On edge devices, where CPU and GPU share one memory, the read is designed to land where the GPU can use it directly.

    Targets4567

  • Network to GPU

    Data that is not on the machine arrives from several nodes in parallel, with no server in between. On GPU servers it is designed to go straight from the network card into GPU memory with GPUDirect RDMA, with every byte still verified, so loading runs while the data arrives and a local copy is kept for the next start. Without RDMA it takes one copy through host memory. The goal is at least 70% of the slower of the network card and the SSDs.

    Targets12313and9over RDMA

  • No silent detours

    A machine that cannot put bytes straight into GPU memory says so, instead of quietly copying through host memory. What the system reports about its data path stays true on every machine.

    Targets7

Storage built for AI

  • One engine for AI data

    Model weights, datasets and live data share one storage engine, with one approach to durability and repair. Datasets are kept in large objects in open formats, so reading them moves bytes instead of looking up millions of files.

    Targets8

  • No metadata bottleneck

    Coordination grows with the number of objects, not with their size, so it stays small and consistent as data grows. The same design is meant to work from a single edge device to petabytes on a GPU cluster.

  • Capacity that pays off

    Data moves to cheaper tiers as it ages. In clusters with enough nodes or racks to spread it across, data that no longer changes is protected with less than half the raw capacity of three full copies. Weights shared between model versions are stored only once.

    Targets1112

The end goal

Tokio to CUDA, without the seams.

The goal is a Rust application on Tokio that works with NVIDIA GPUs as naturally as with files and sockets. Data arrives in GPU memory through the same async runtime that serves requests, GPU work is awaited like any other task, and the kernels are Rust too.

The first experiments on the host-to-GPU path used CUDA C++ called from async Rust over FFI. The current work moves that code to NVIDIA's CUDA Rust.

  • GPU work as async tasksTransfers and kernels are started and awaited from Tokio like any other task, so loading, serving and I/O share one async runtime.
  • One language, end to endThe storage engine, the network fetch, the GPU kernels and the application are all Rust on Tokio, with io_uring for storage I/O and no C++ layer in between.
  • Mistakes caught at compile timeGPU memory and the transfers that use it are designed as Rust types, so mistakes such as freeing memory a transfer still uses are caught when compiling, not in production.
  • Built on CUDA RustNVIDIA's CUDA platform for Rust: GPU kernels and the host code that drives them, both written in Rust. NVIDIA introduced it in September 2026 and calls it early-stage, not yet production-ready. github.com/NVIDIA/cuda-rust · announcement
  • Under heavy developmentKylix is a private, non-commercial research project. Designs change as measurements come in.
Market trends

Why the data path matters now.

Models are growing faster than the paths that feed them, and inference is becoming the main workload: Deloitte forecasts it at roughly two-thirds of compute in 2026, up from a third in 2023 (source).

  • Checkpoints near a terabyte

    Large open-weight models now ship as checkpoints of hundreds of gigabytes to well over a terabyte: DeepSeek-V4-Pro is about 0.9 TB and Kimi K3 about 1.6 TB. Every new replica has to move that much before it serves its first token.

    Sources: DeepSeek-V4-Pro, Kimi K3 on Hugging Face

  • Cold start is a named problem

    Model loading now has its own research and tools. NVIDIA reports starting a new DeepSeek-V4-Pro replica on eight GPUs in 1 minute 44 seconds instead of 8 minutes by copying weights GPU to GPU over RDMA. IBM reports loads 4.8 to 7.5 times faster by building tensors directly in GPU memory instead of in host memory.

    Sources: NVIDIA, IBM, OSDI '24

  • Storage joins the inference path

    Long contexts and many concurrent users outgrow GPU memory, so cached context is moving to SSDs and network storage. NVIDIA Dynamo spreads it across GPU memory, CPU memory, local SSDs and remote storage, and in January 2026 NVIDIA announced a context-memory storage platform with a dozen storage vendors.

    Sources: Dynamo, NVIDIA, LMCache

  • The path sets the pace

    A B200 GPU reads its own memory at about 8 TB/s. A PCIe Gen5 x16 link carries about 64 GB/s each way, an 800 Gb/s network card about 100 GB/s and a fast Gen5 SSD about 14 GB/s. Every extra stop on the way is time the GPU spends waiting.

    Sources: DGX B200, ConnectX-8, Micron 9550

  • Edge devices share one memory

    Jetson Orin Nano Super has 8 GB of memory at 102 GB/s, and Jetson Thor 128 GB at 273 GB/s. On both, the CPU and GPU share the same DRAM, so every needless copy spends the bandwidth the model itself needs.

    Sources: Orin Nano Super, Jetson Thor, CUDA for Tegra

  • Data formats built for GPUs

    Vortex, hosted by the LF AI & Data Foundation since 2025, is designed for direct GPU decompression. Lance brings multimodal AI tables to the Hugging Face Hub, and Apache Iceberg is opening its table format to new file formats such as Vortex.

    Sources: Linux Foundation, Hugging Face, Apache Iceberg

Get in touch

Let's talk.

Kylix is a research project under heavy development. We welcome conversations with researchers, infrastructure engineers and hardware makers who care about how fast data reaches the GPU.

Please do not include confidential information in your first email.

info@kylix.ai

What would you like to talk about?

Choosing a topic opens your email program with the subject filled in. We use your details only to reply. See the privacy policy.