Zero Heap Allocations • 120 FPS Frame Budget Safe

QuantizedLab SDK Documentation

QuantizedLab exposes a panic-isolated pure C-ABI (quantizedlab_engine.h) wrapped in first-class native packages for Android Kotlin, Apple Swift 6, Unity C#, and Unreal Engine 5.

1. Native Architecture & Memory Paradigms

Unlike conventional LLM wrappers that allocate temporary garbage-collected strings on every token or query, QuantizedLab enforces a zero-heap, zero-copy architecture. Model weights (.lev2) are memory-mapped directly from APK assets or app bundles via OS kernel pages. Game engine queries execute in caller-allocated scratchpad buffers within a <50 µs frame budget.

5-MINUTE NATIVE INTEGRATION

Drop-In Client SDKs

Zero garbage collection, zero JVM overhead. Choose your target runtime to inspect runnable starter code.

🤖Android Kotlin Quick Start
// app/build.gradle.kts
dependencies {
    implementation("com.quantizedlabs:leviathan-sdk:1.0.0")
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.10.2")
}
// 1. Initialize client with pre-flight RAM guard & core affinity
val client = LeviathanClient.builder(context)
    .setModelPath(modelFile.absolutePath)
    .setTokenizerPath(tokenizerFile.absolutePath)
    .setLoraAdapter(godModeLoraFile.absolutePath) // Optional God Mode LoRA
    .setExecutionAffinity(ExecutionAffinity.BIG_CORES_ONLY)
    .build()

// 2. Stream tokens in real time (18-21 TPS on $180 Moto G Stylus)
client.generateStream("Explain quantum cryptography simply.").collect { token ->
    uiState.appendToken(token)
}

// 3. Guaranteed JSON structured output & on-device tool calling
val toolCall = client.generateToolCall("Can you split a $140 bill with 4 people?", availableTools)
if (toolCall != null) {
    executeLocalTool(toolCall.name, toolCall.parseArguments())
}

2. Android Kotlin SDK (Coroutines + Zero-Copy JNI)

Mount compressed foundation models directly from APK assets with zero disk duplication. Run streaming token generation and dynamic LoRA swapping without blocking the Android UI thread.

// 1. In your Android Activity or ViewModel
import com.quantizedlab.engine.QuantizedLabEngine
import androidx.lifecycle.viewModelScope
import kotlinx.coroutines.launch
import kotlinx.coroutines.Dispatchers

class LocalAssistantViewModel(application: Application) : AndroidViewModel(application) {
    private val engine = QuantizedLabEngine.getInstance(application)

    init {
        viewModelScope.launch(Dispatchers.IO) {
            // Direct zero-copy memory mapping out of APK assets (<18ms boot)
            val mounted = engine.mountAssetModel(application, "llama_8b_2bit.lev2")
            if (mounted) {
                // 11.6 TPS sustained on Snapdragon 6 Gen 1 (Moto G 5G)
                engine.generateStream("Explain quantum superposition in 2 sentences:") { token ->
                    print(token)
                }
            }
        }
    }
}

3. Apple Swift 6 SDK (Actor Isolation & Zero-Copy mmap)

Thread-safe, actor-isolated Swift 6 interface for iOS, macOS, and visionOS apps with direct Metal GPU and native vector acceleration.

import Foundation
import QuantizedLabCore

public actor SovereignBrain {
    private var engine: QuantizedLabEngineHandle?

    public init() {
        self.engine = quantizedlab_engine_init()
    }

    public func loadModel(bundleResource: String) async -> Bool {
        guard let path = Bundle.main.path(forResource: bundleResource, ofType: "lev2") else {
            return false
        }
        let code = quantizedlab_mount_model(self.engine, path)
        return code == 0
    }

    public func generateStream(prompt: String) async -> AsyncStream<String> {
        return AsyncStream { continuation in
            guard let engine = self.engine else {
                continuation.finish()
                return
            }
            quantizedlab_stream_callback(engine, prompt) { tokenPtr in
                if let tokenPtr = tokenPtr {
                    let token = String(cString: tokenPtr)
                    continuation.yield(token)
                } else {
                    continuation.finish()
                }
            }
        }
    }
}

4. Unity C# Package (Burst & Job System Compatible)

Expose microsecond knowledge retrieval and on-device NPC dialogue to Unity MonoBehaviour and DOTS systems using raw struct pinvokes.

using System;
using UnityEngine;
using QuantizedLab;

public class NPCAgent : MonoBehaviour
{
    private IntPtr engineHandle;
    private byte[] scratchBuffer = new byte[2048];

    void Start()
    {
        // Mount local 2-bit model directly from StreamingAssets
        engineHandle = QuantizedLabNative.quantizedlab_engine_init();
        string modelPath = Application.streamingAssetsPath + "/qwen_1.5b_2bit.lev2";
        QuantizedLabNative.quantizedlab_mount_model(engineHandle, modelPath);
    }

    public void OnPlayerDialogue(string playerSpeech)
    {
        // Zero-allocation frame query executed in <45 µs
        string reply = QuantizedLabNative.GenerateResponse(engineHandle, playerSpeech);
        Debug.Log($"[NPC Reply]: {reply}");
    }

    void OnDestroy()
    {
        if (engineHandle != IntPtr.Zero) {
            QuantizedLabNative.quantizedlab_free(engineHandle);
        }
    }
}

5. Unreal Engine 5 Integration (Zero-Allocation 120 FPS Queries)

Call QuantizedLab directly inside the game loop or NPC behavior trees using caller-allocated scratchpad buffers. Zero garbage collection, zero dynamic memory allocations per frame.

// Source/MyGame/NPCBrainComponent.cpp
#include "NPCBrainComponent.h"
#include "quantizedlab_engine.h"

void UNPCBrainComponent::TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
    Super::TickComponent(DeltaTime, TickType, ThisTickFunction);

    // Caller-allocated scratchpad buffer on stack (0 heap allocations)
    char Buffer[1024];
    uint32_t Written = 0;

    // Sub-45 µs decision query (<50 µs 120 FPS safe)
    int32_t Status = quantizedlab_query_buffered(
        EngineHandle,
        "Evaluate tactical stance",
        Buffer, 1024, &Written
    );
}

6. Core C-ABI Reference (quantizedlab_engine.h)

quantizedlab_engine_init() -> *mut QuantizedLabHandle

Allocates an isolated inference engine state. Thread-safe, panic-isolated across FFI.

quantizedlab_mount_model(handle, path) -> i32

Memory-maps a .lev2 container via OS kernel pages. Returns 0 on success.

quantizedlab_query_buffered(handle, prompt, buf, cap, written) -> i32

Caller-allocated scratchpad query API. Zero heap allocations. Guarantees <50 µs response time for 120 FPS game tick safety.

quantizedlab_attach_lora(handle, lora_path) -> i32

Hot-swaps task-specific LoRA adapters in 0 milliseconds latency via decoupled memory pointer update during decode.

7. Zero-JVM TokenRingBuffer Streaming

Eliminate Android garbage collection pauses and ANRs during high-throughput token generation. Pass a 64KB Direct ByteBuffer directly from Java/Kotlin into native Rust, and poll new tokens with zero String allocations:

// Direct-memory 64KB shared ring buffer for 0-allocation token streaming
val directBuffer = ByteBuffer.allocateDirect(65536).order(ByteOrder.nativeOrder())
QuantizedLabJNI.attachTokenRingBuffer(handle, directBuffer)

// Native inference writes directly into shared memory
// UI thread reads token offsets in <0.5 µs without triggering GC sweeps
lifecycleScope.launch(Dispatchers.Default) {
    while (isGenerating) {
        val token = pollNextTokenFromDirectBuffer(directBuffer)
        if (token != null) {
            tokenFlow.emit(token)
        }
    }
}

8. Standalone Model Compiler CLI (quantizedlab-pack)

Convert Hugging Face .safetensors models (LLaMA 3.1, Qwen 2.5, Mamba-2) directly into memory-mapped .lev2 containers with zero Python dependencies and zero post-quantization retraining:

# 1. Install pure-Rust conversion binary
cargo install quantizedlab-pack

# 2. Convert LLaMA 3.1 8B with full error recovery compression (<8 mins)
quantizedlab-pack convert \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quant tripartite-2bit \
  --out llama_3.1_8b_2bit.lev2

# 3. Convert Qwen 2.5 1.5B with INT8 embeddings & active vocab slicing
quantizedlab-pack convert \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --quant 2bit-optimized \
  --vocab-slice 16384 \
  --out qwen_2.5_1.5b_int8embed_890m.lev2