native inference, retrieval, and agents for real-time engines
A C++17 inference control plane for native applications. Run local or remote models, batch active conversations, search compact native memory, and stream bounded results through one C ABI.
pre-1.0 · Apache-2.0 · read the runtime architecture
Provider selection happens when a model loads. Handles, cancellation, request states, grammar, sampling, and streaming remain in Astral, so the application keeps one lifecycle when model execution moves between those providers.
Astral gives games and native applications one inference API. Local GGUF models, remote providers, continuous batching, embeddings, structured output, and cancellation share the same request model. Engine code polls or waits at its own boundary. Token hot paths never call into application code.
Agents keep prompts, summaries, retrieved context, and chat history in native memory. When a turn completes, its decode slot can be released without destroying the agent handle, prompt state, summary, retrieved context, or history. The embedded index supplies memory without adding a separate database service to the application.
Astral ships no model weights. It is the native layer between an engine and the inference backends the application chooses. The feature matrix lists the exact build support and restrictions for each surface.
Local backends and remote providers share handles, request states, cancellation, streaming, and structured generation.
Unity, Unreal, and native callers poll bounded streams and marshal results on their own threads. Runtime hot paths do not invoke callbacks.
Agent handles retain history, summaries, prompt-cache state, and retrieved context after a turn releases its decode slot.
Flat and bounded graph search share f32, q8, Float6, and E5M2 storage. Filtered queries scan every matching stored score.
Runtime architecture follows one request across the C ABI, retained agent state, scheduling, provider execution, sampling, and engine-paced delivery. The retrieval architecture carries a query through compact storage, flat or graph traversal, and optional f32 reranking.
The engineering notes keep the measurements and failed approaches alongside the implementation that survived them. Start with a prompt crossing the runtime, then follow the retrieval work into its scoring kernels.
recall / graph index
A graph index that looked finished was returning wildly different result quality from one query to the next, and more search budget only hid it. The write-up follows the diagnosis from symptom to a ground-up rewrite, including three competing fixes that lost on measurement first.
read the articlesimd / e5m2
Astral's AVX2 and F16C lane has no native E5M2 load-and-dot primitive, so five conversion paths competed for the kernel over one day. The write-up keeps the losers in the story, because two of them still serve machines the winner cannot.
read the articleretrieval / flat search
A graph budget sweep could not meet the fixed latency and recall target at the same point. The write-up follows the exhaustive E5M2 scanner from one serial loop to private record shards and the retained 384-component kernel.
read the article