FreeToken Wiki
Run frontier open-weight MoE models locally across consumer NVIDIA GPUs, with adaptive CPU-GPU execution, efficient caching, quantization, and OpenAI or Anthropic compatible APIs.
FreeToken Wiki Guides
Everything you need to install, configure, and run large MoE models locally with FreeToken
Latest Updates
Discover the newest guides, tips, and content
FreeToken 290b model: 2026 Edge MoE Setup Guide
Learn how FreeToken serves frontier-scale MoE models on consumer hardware through adaptive caching, bandwidth scheduling, and elastic memory.
FreeToken 4gb vram: Setup Guide & Hardware Limits
Learn how FreeToken handles limited VRAM, CPU offload, expert caching, and realistic hardware requirements for local MoE serving.
FreeToken 753b model: Setup Guide and MoE Tuning Tips
Learn how FreeToken serves 753B GLM-5.2 locally, including setup, memory paging, MoE caching, benchmarks, and practical tuning advice.
FreeToken anthropic api: Step-by-Step Setup Guide
Set up Anthropic API access for a FreeToken project with secure keys, environment variables, request testing, and troubleshooting tips.
FreeToken api: Local MoE Serving Setup Guide 2026
Learn how FreeToken serves frontier-scale MoE models locally with expert caching, adaptive CPU-GPU execution, and elastic memory management.
FreeToken api server: Setup Guide & Runtime Tips
Learn how the FreeToken api server handles MoE caching, CPU-GPU execution, hardware checks, and practical local serving decisions.
FreeToken benchmark: Local MoE Serving Setup Guide
Review the FreeToken benchmark, local MoE serving design, hardware requirements, installation flow, and measured performance results.
FreeToken claude code: Setup Guide for Model Routing
Configure FreeToken claude code with a local gateway, model routing, fallback providers, and safe troubleshooting practices.
FreeToken codex: Setup Guide, Benchmarks & Limits
FreeToken codex explained: learn how its expert-aware memory system works, review benchmarks, hardware limits, setup considerations, and local AI tradeoffs.
FreeToken deepseek: Local AI Setup Guide & Benchmarks
Learn how FreeToken serves DeepSeek locally with MoE caching, CPU-GPU offload, hardware guidance, setup steps, and performance notes.
FreeToken deepseek v4 flash: Local Setup Guide 2026
Set up DeepSeek V4 Flash with FreeToken for local AI inference, including RAM planning, GPU expectations, Open WebUI access, and troubleshooting.
FreeToken edge native moe serving: Architecture Guide
Learn how FreeToken uses bandwidth-adaptive execution, caching, and CPU-GPU coordination to serve large MoE models on edge hardware.
FreeToken Install Guide
Set up FreeToken with the desktop app, uv, PyPI, or source code, then launch your first local model server. NVIDIA GPU users can install the accelerated Python package, check system bandwidth, and start an OpenAI-compatible server with the ft CLI.
Check the host system
Use a Windows or Linux PC with a supported NVIDIA GPU. FreeToken targets RTX 30, RTX 40, and RTX 50 series systems and can combine GPU memory with system RAM for large-model execution.
Install the desktop app
For the simplest setup, download and install the FreeToken desktop application for Windows or Linux. The desktop workflow provides a graphical path for configuring and serving local models.
Install with uv
Install the accelerated FreeToken package into your Python environment with uv.
uv pip install "freetoken[accel]"Install from PyPI
The same accelerated package can be installed through a standard Python package workflow when uv is not being used.
pip install "freetoken[accel]"Install from source
Clone the FreeToken repository when you need the development version or want to work directly with the source tree, then install the project and its acceleration dependencies.
git clone https://github.com/FlashML-org/FreeToken.gitMeasure host bandwidth
Run the built-in bandwidth benchmark before serving very large MoE checkpoints. FreeToken uses the result to help determine how effectively CPU memory and the GPU can work together.
ft bench bwStart the model server
Launch the FreeToken serving workflow with ft serve. The server exposes local inference through OpenAI- and Anthropic-compatible interfaces for applications and coding agents.
ft serveVerify the first run
Confirm that the selected model loads, the server remains active, and a local API request can return generated output before connecting external clients such as Codex, Claude Code, or OpenCode.
Quick Tips
- The freetoken[accel] extra installs the GPU acceleration stack needed for RTX-class hardware.
- Run ft bench bw once before your first large model so FreeToken can pick an efficient execution strategy.
- The desktop app and the ft CLI serve the same engine, so you can switch between them freely.
- Keep CUDA and NVIDIA drivers up to date to avoid kernel and compatibility errors inside the accelerated package.
FreeToken Supported Models
FreeToken is designed around large sparse Mixture-of-Experts models whose total parameter counts can exceed the memory of a single consumer GPU. Its execution backends combine GPU compute, system memory, expert caching, and model-specific checkpoint handling to make these models practical on local machines.
| Model | Scale | Checkpoint Format | Quantization | Backend | Best For |
|---|---|---|---|---|---|
| GLM-5.2 | 753B total parameters | GLM MoE checkpoint | MXFP4 / NVFP4 / FP8 / BF16 | Hybrid MoE / expert offload | High-capacity workstation serving |
| DeepSeek-V4-Flash | Large sparse MoE | DeepSeek MoE checkpoint | MXFP4 / NVFP4 / FP8 / BF16 | Hybrid MoE / expert offload | Local reasoning and agent workloads |
| Qwen3.6 | 35B-A3B-class sparse MoE | Qwen MoE checkpoint | MXFP4 / NVFP4 / FP8 / BF16 | MoE serving backend | Local chat, coding, and agents |
| Qwen3 MoE | Multiple sparse sizes | Qwen MoE checkpoints | MXFP4 / NVFP4 / FP8 / BF16 | MoE serving backend | General-purpose local inference |
| gpt-oss | Multiple checkpoint sizes | gpt-oss checkpoint | MXFP4 / NVFP4 / FP8 / BF16 | Sparse-model serving backend | OpenAI-compatible app workflows |
| Gemma-4 | Model-dependent | Gemma checkpoint | MXFP4 / NVFP4 / FP8 / BF16 | Model-appropriate backend | Local general-purpose inference |
| MiniMax | Large sparse MoE | MiniMax checkpoint | MXFP4 / NVFP4 / FP8 / BF16 | Hybrid MoE / expert offload | Large-context agent workloads |
Quick Tips
- All major families support MXFP4 and NVFP4 low-precision checkpoints alongside FP8 and BF16.
- Pick quantization by matching checkpoint size against your combined VRAM plus system RAM budget.
- Hybrid MoE execution suits the largest checkpoints; smaller sparse models can stay more GPU-resident.
- New model families are added regularly, so check the models documentation with each engine release.
FreeToken Hardware Requirements
FreeToken does not require the complete model to fit inside GPU VRAM. Its bandwidth-adaptive CPU-GPU execution, expert caching, and elastic memory management allow sparse MoE checkpoints to use both GPU memory and host RAM, making system memory and PCIe bandwidth important alongside raw GPU capacity.
| Component | Requirement | Why It Matters |
|---|---|---|
| Operating system | Windows or Linux | Both platforms are supported by the desktop app and local serving workflow. |
| GPU family | NVIDIA RTX 30 / 40 / 50 series | Newer GPUs provide more VRAM and compute headroom for larger active expert sets. |
| GPU VRAM | Model, precision, and cache dependent | More VRAM keeps active weights and cached experts on the GPU, reducing transfers. |
| System RAM | High capacity for large MoE checkpoints | Host RAM holds weights that exceed VRAM, enabling far larger checkpoints. |
| CPU memory bandwidth | Higher is better for offload | Bandwidth-adaptive execution uses measured host-memory performance. |
| PCIe bandwidth | Fast transfers recommended | Determines how quickly expert weights move between RAM and VRAM. |
| CUDA and drivers | Compatible NVIDIA runtime | Required for accelerated GPU execution through freetoken[accel]. |
| 290B+ class models | Substantial host RAM plus RTX GPU | Hybrid CPU-GPU execution replaces the need to fit in VRAM. |
| Largest checkpoints | Workstation RAM, storage, bandwidth | Serves frontier-scale sparse models such as 753B-parameter GLM-5.2. |
Quick Tips
- RTX 30 series works well with mid-size checkpoints and heavier quantization.
- RTX 40 series offers stronger inference performance and memory efficiency for hybrid MoE workloads.
- RTX 50 series suits NVFP4 low-precision formats and larger high-throughput deployments.
- Dual-channel or better system memory directly improves offloaded expert delivery speed.
FreeToken Benchmarks
FreeToken performance depends on more than GPU compute because large MoE checkpoints can continuously move experts between host memory and the GPU. Its benchmark tooling measures the bandwidth available on a specific machine and helps the runtime choose between heavier offload and more GPU-resident hybrid execution.
| Benchmark | Command | Measures | Why It Matters |
|---|---|---|---|
| CPU memory bandwidth | ft bench bw | Host-memory throughput | Offloaded expert weights are read from system RAM, so bandwidth directly improves expert delivery. |
| CPU-to-GPU bandwidth | ft bench bw | PCIe transfer throughput | Determines how fast weights reach the GPU when not already in VRAM. |
| Decode throughput | ft serve | Generated tokens per second | Interactive generation speed after prompt processing. |
| Prefill throughput | ft serve | Prompt-processing throughput | Double-buffered prefill streaming overlaps movement and computation for long prompts. |
| GPU memory use | ft serve | VRAM for weights, caches, state | Elastic VRAM management keeps the model resident within the memory budget. |
| System memory use | ft serve | RAM for checkpoint and experts | Large host capacity serves checkpoints far larger than GPU VRAM. |
| Expert cache efficiency | ft serve | GPU-resident expert reuse | The global LRU cache reduces repeated CPU-to-GPU transfers. |
| Offload execution | ft bench bw | Host-resident expert share | Works best with strong CPU-memory and PCIe bandwidth even with limited VRAM. |
| Hybrid MoE execution | ft bench bw | GPU + host expert mix | Balances VRAM capacity, cache locality, and transfer bandwidth. |
| Consumer laptop profile | ft bench bw | Mobile bandwidth limits | Helps pick smaller or more quantized checkpoints for laptop memory. |
| Gaming PC profile | ft bench bw | Desktop RTX + RAM + PCIe | The common FreeToken configuration on RTX 30, 40, and 50 systems. |
| Workstation profile | ft bench bw | High-capacity everything | Supports frontier-scale sparse checkpoints with CPU-GPU cooperation. |
Quick Tips
- Run ft bench bw on every new machine before choosing an execution strategy.
- Decode speed follows host bandwidth closely when most experts are offloaded.
- Hybrid execution usually beats pure offload once enough experts fit in VRAM.
- Compare prefill and decode together, since long-context workloads stress both.
FreeToken API Guide
FreeToken exposes local model inference through familiar API formats. After loading a supported model with ft serve, applications can discover models, send chat or Responses API requests, use Anthropic-style messages, and stream generated tokens from the local server.
Launch the FreeToken serving process with a supported model. The server handles model loading, CPU-GPU execution, memory allocation, and HTTP API requests.
ft serve <model>
OpenAI Chat Completions
OpenAI-compatibleUse the OpenAI-compatible chat completions endpoint with existing SDKs or HTTP clients.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<model>",
"messages": [
{"role": "user", "content": "Explain mixture-of-experts models."}
]
}'OpenAI Responses API
OpenAI-compatibleApplications built around the newer OpenAI Responses API can send requests to the corresponding local endpoint.
curl http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "<model>",
"input": "Write a short Python function that reverses a list."
}'List Available Models
OpenAI-compatibleQuery the OpenAI-compatible models endpoint to see models exposed by the running FreeToken server.
curl http://localhost:8000/v1/models
Anthropic Messages API
Anthropic-compatibleFreeToken also provides an Anthropic-compatible messages interface for clients and coding tools designed around the Anthropic API.
curl http://localhost:8000/v1/messages \
-H "Content-Type: application/json" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "<model>",
"max_tokens": 512,
"messages": [
{"role": "user", "content": "Summarize this repository architecture."}
]
}'Streaming Responses
SSEEnable streaming when the client supports server-sent incremental output so generated tokens can be consumed as they arrive.
{
"model": "<model>",
"messages": [
{"role": "user", "content": "Write a REST API example."}
],
"stream": true
}Use ft serve configuration to select the model and control how the local HTTP service is exposed, including host, port, and memory settings.
ft serve --help
Inspect CLI Options
CLIFreeToken keeps current serving and runtime options available directly from command-line help, including options added by newer engine releases.
ft --help ft serve --help
FreeToken Claude Code and Codex Setup
FreeToken can act as the local inference backend for agentic coding tools that understand OpenAI- or Anthropic-compatible APIs. The ft launch workflow helps configure supported agents, while dry-run mode lets you inspect the planned setup before changing or starting anything.
Install and verify FreeToken
Make sure the FreeToken CLI is installed and a supported model can be loaded locally before connecting an external coding agent.
ft --helpCheck agent launch options
Use the launch command help to view the currently supported agent integrations and their available configuration options.
ft launch --helpPreview with dry run
Use dry-run mode before launching an agent when you want to inspect the generated command and environment configuration without starting the final workflow.
ft launch <agent> --dry-runLaunch the coding agent
Start the selected coding agent through FreeToken so its API configuration points to the local inference backend.
ft launch <agent>Claude Code
Claude Code can use FreeToken's Anthropic-compatible interface. The local configuration redirects supported Anthropic message traffic to the FreeToken server while preserving the agent workflow.
ft launch claudeCodex
Codex-style clients can connect through FreeToken's OpenAI-compatible endpoints, allowing a locally served model to handle agent requests.
ft launch codexOpenCode, Hermes and OpenClaw
FreeToken's launch workflow also targets additional coding and autonomous-agent clients such as OpenCode, Hermes, and OpenClaw. Check the launch help for current integration names and options.
ft launch --helpPrevent cloud API fallback
Verify the agent's provider and base URL configuration before beginning a long task. Remove or disable unwanted cloud-provider configuration when the goal is fully local inference, and use dry-run mode to inspect the environment FreeToken will pass to the agent.
ft launch <agent> --dry-runQuick Tips
- Always run the dry-run preview once before the first real launch of a new agent.
- Keep the FreeToken server running while agents work, since long sessions reconnect to the same endpoint.
- Check the provider base URL after setup to confirm requests reach the local server, not a hosted API.
- Integration names evolve with engine releases, so re-check ft launch --help after upgrading.
FreeToken Architecture
FreeToken is designed around the sparse structure of mixture-of-experts models. Instead of requiring every expert weight to remain in GPU memory, it coordinates CPU and GPU execution, caches frequently used experts, overlaps transfers with computation, and adapts memory use to the machine running the model.
FreeToken Updates and Releases
FreeToken is evolving across both its inference engine and desktop experience. The release tracker focuses on changes that directly affect installation, model compatibility, local serving, GPU support, performance, and agent workflows.
Follow FreeToken Development
Track new releases and join the community through the official channels: