QuantizedLab SDK Documentation
QuantizedLab exposes a panic-isolated pure C-ABI (quantizedlab_engine.h) wrapped in first-class native packages for Android Kotlin, Apple Swift 6, Unity C#, and Unreal Engine 5.
1. Native Architecture & Memory Paradigms
Unlike conventional LLM wrappers that allocate temporary garbage-collected strings on every token or query, QuantizedLab enforces a zero-heap, zero-copy architecture. Model weights (.lev2) are memory-mapped directly from APK assets or app bundles via OS kernel pages. Game engine queries execute in caller-allocated scratchpad buffers within a <50 µs frame budget.
Drop-In Client SDKs
Zero garbage collection, zero JVM overhead. Choose your target runtime to inspect runnable starter code.
// app/build.gradle.kts
dependencies {
implementation("com.quantizedlabs:leviathan-sdk:1.0.0")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.10.2")
}// 1. Initialize client with pre-flight RAM guard & core affinity
val client = LeviathanClient.builder(context)
.setModelPath(modelFile.absolutePath)
.setTokenizerPath(tokenizerFile.absolutePath)
.setLoraAdapter(godModeLoraFile.absolutePath) // Optional God Mode LoRA
.setExecutionAffinity(ExecutionAffinity.BIG_CORES_ONLY)
.build()
// 2. Stream tokens in real time (18-21 TPS on $180 Moto G Stylus)
client.generateStream("Explain quantum cryptography simply.").collect { token ->
uiState.appendToken(token)
}
// 3. Guaranteed JSON structured output & on-device tool calling
val toolCall = client.generateToolCall("Can you split a $140 bill with 4 people?", availableTools)
if (toolCall != null) {
executeLocalTool(toolCall.name, toolCall.parseArguments())
}2. Android Kotlin SDK (Coroutines + Zero-Copy JNI)
Mount compressed foundation models directly from APK assets with zero disk duplication. Run streaming token generation and dynamic LoRA swapping without blocking the Android UI thread.
// 1. In your Android Activity or ViewModel
import com.quantizedlab.engine.QuantizedLabEngine
import androidx.lifecycle.viewModelScope
import kotlinx.coroutines.launch
import kotlinx.coroutines.Dispatchers
class LocalAssistantViewModel(application: Application) : AndroidViewModel(application) {
private val engine = QuantizedLabEngine.getInstance(application)
init {
viewModelScope.launch(Dispatchers.IO) {
// Direct zero-copy memory mapping out of APK assets (<18ms boot)
val mounted = engine.mountAssetModel(application, "llama_8b_2bit.lev2")
if (mounted) {
// 11.6 TPS sustained on Snapdragon 6 Gen 1 (Moto G 5G)
engine.generateStream("Explain quantum superposition in 2 sentences:") { token ->
print(token)
}
}
}
}
}3. Apple Swift 6 SDK (Actor Isolation & Zero-Copy mmap)
Thread-safe, actor-isolated Swift 6 interface for iOS, macOS, and visionOS apps with direct Metal GPU and native vector acceleration.
import Foundation
import QuantizedLabCore
public actor SovereignBrain {
private var engine: QuantizedLabEngineHandle?
public init() {
self.engine = quantizedlab_engine_init()
}
public func loadModel(bundleResource: String) async -> Bool {
guard let path = Bundle.main.path(forResource: bundleResource, ofType: "lev2") else {
return false
}
let code = quantizedlab_mount_model(self.engine, path)
return code == 0
}
public func generateStream(prompt: String) async -> AsyncStream<String> {
return AsyncStream { continuation in
guard let engine = self.engine else {
continuation.finish()
return
}
quantizedlab_stream_callback(engine, prompt) { tokenPtr in
if let tokenPtr = tokenPtr {
let token = String(cString: tokenPtr)
continuation.yield(token)
} else {
continuation.finish()
}
}
}
}
}4. Unity C# Package (Burst & Job System Compatible)
Expose microsecond knowledge retrieval and on-device NPC dialogue to Unity MonoBehaviour and DOTS systems using raw struct pinvokes.
using System;
using UnityEngine;
using QuantizedLab;
public class NPCAgent : MonoBehaviour
{
private IntPtr engineHandle;
private byte[] scratchBuffer = new byte[2048];
void Start()
{
// Mount local 2-bit model directly from StreamingAssets
engineHandle = QuantizedLabNative.quantizedlab_engine_init();
string modelPath = Application.streamingAssetsPath + "/qwen_1.5b_2bit.lev2";
QuantizedLabNative.quantizedlab_mount_model(engineHandle, modelPath);
}
public void OnPlayerDialogue(string playerSpeech)
{
// Zero-allocation frame query executed in <45 µs
string reply = QuantizedLabNative.GenerateResponse(engineHandle, playerSpeech);
Debug.Log($"[NPC Reply]: {reply}");
}
void OnDestroy()
{
if (engineHandle != IntPtr.Zero) {
QuantizedLabNative.quantizedlab_free(engineHandle);
}
}
}5. Unreal Engine 5 Integration (Zero-Allocation 120 FPS Queries)
Call QuantizedLab directly inside the game loop or NPC behavior trees using caller-allocated scratchpad buffers. Zero garbage collection, zero dynamic memory allocations per frame.
// Source/MyGame/NPCBrainComponent.cpp
#include "NPCBrainComponent.h"
#include "quantizedlab_engine.h"
void UNPCBrainComponent::TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
Super::TickComponent(DeltaTime, TickType, ThisTickFunction);
// Caller-allocated scratchpad buffer on stack (0 heap allocations)
char Buffer[1024];
uint32_t Written = 0;
// Sub-45 µs decision query (<50 µs 120 FPS safe)
int32_t Status = quantizedlab_query_buffered(
EngineHandle,
"Evaluate tactical stance",
Buffer, 1024, &Written
);
}6. Core C-ABI Reference (quantizedlab_engine.h)
Allocates an isolated inference engine state. Thread-safe, panic-isolated across FFI.
Memory-maps a .lev2 container via OS kernel pages. Returns 0 on success.
Caller-allocated scratchpad query API. Zero heap allocations. Guarantees <50 µs response time for 120 FPS game tick safety.
Hot-swaps task-specific LoRA adapters in 0 milliseconds latency via decoupled memory pointer update during decode.
7. Zero-JVM TokenRingBuffer Streaming
Eliminate Android garbage collection pauses and ANRs during high-throughput token generation. Pass a 64KB Direct ByteBuffer directly from Java/Kotlin into native Rust, and poll new tokens with zero String allocations:
// Direct-memory 64KB shared ring buffer for 0-allocation token streaming
val directBuffer = ByteBuffer.allocateDirect(65536).order(ByteOrder.nativeOrder())
QuantizedLabJNI.attachTokenRingBuffer(handle, directBuffer)
// Native inference writes directly into shared memory
// UI thread reads token offsets in <0.5 µs without triggering GC sweeps
lifecycleScope.launch(Dispatchers.Default) {
while (isGenerating) {
val token = pollNextTokenFromDirectBuffer(directBuffer)
if (token != null) {
tokenFlow.emit(token)
}
}
}8. Standalone Model Compiler CLI (quantizedlab-pack)
Convert Hugging Face .safetensors models (LLaMA 3.1, Qwen 2.5, Mamba-2) directly into memory-mapped .lev2 containers with zero Python dependencies and zero post-quantization retraining:
# 1. Install pure-Rust conversion binary
cargo install quantizedlab-pack
# 2. Convert LLaMA 3.1 8B with full error recovery compression (<8 mins)
quantizedlab-pack convert \
--model meta-llama/Llama-3.1-8B-Instruct \
--quant tripartite-2bit \
--out llama_3.1_8b_2bit.lev2
# 3. Convert Qwen 2.5 1.5B with INT8 embeddings & active vocab slicing
quantizedlab-pack convert \
--model Qwen/Qwen2.5-1.5B-Instruct \
--quant 2bit-optimized \
--vocab-slice 16384 \
--out qwen_2.5_1.5b_int8embed_890m.lev2