02. Server options

Server Options

Note

If you want the latest config.yml docs, please look at the comments in config_sample.yml

TabbyAPI primarily uses a config.yml file to adjust various options. This is the preferred way and has the ability to adjust all options of TabbyAPI.

CLI arguments are also included, but those serve to override the options set in config.yml. Therefore, they act a bit differently compared to other programs, especially with booleans. Example: A user sets gpu_split_auto to True. The CLI arg will then be --gpu_split-auto False to override that previous config.yml setting.

In addition, some config.yml options are too complex to represent as command args, so those are not included with the argparser.

All of these options have descriptive comments above them. You should not need to reference this documentation page unless absolutely necessary.

Networking Options

Config Option	Type (Default)	Description
host	String (127.0.0.1)	Set the IP address used for hosting TabbyAPI
port	Int (5000)	Set the TCP Port use for TabbyAPI. If the port is taken, TabbyAPI exits with an error rather than switching to another port
disable_auth	Bool (False)	Disables API authentication. Web pages open in a browser on the machine count as local callers, so restrict allowed_origins when this is on.
allowed_origins	List[String] (["*"])	CORS allowlist: origins a browser lets read this API's responses. "*" allows any site; list your own frontends (e.g. ["http://localhost:8000"]) or use [] to block all cross-origin browser callers. Not an authentication mechanism.
disable_fetch_requests	Bool (False)	Disables fetching external content when responding to requests (ex. fetching images from URLs)
send_tracebacks	Bool (False)	Send server tracebacks to client. Note: It's not recommended to enable this if sharing the instance with others.
api_servers	List[String] (["OAI"])	API servers to enable. Possible values "OAI", "Kobold"
sse_ping_interval	Int (15)	Interval in seconds between SSE keep-alive pings on streaming responses. Set to 0 to disable pings.
access_log	Bool (False)	Log every HTTP request with client address, method, path and status code. Generation requests are logged in detail regardless; this adds the rest, such as model list and health polls.
web_ui	Bool (True)	Serve the built-in chat client at /ui. It is a static page that uses the regular API, so it costs nothing until opened.

Logging Options

Config Option	Type (Default)	Description
log_prompt	Bool (False)	Logs prompts to the console
log_generation_params	Bool (False)	Logs request generation options to the console
log_requests	Bool (False)	Logs a request's URL, Body, and Headers to the console
log_live_status	Bool (True)	Shows a live status line below the log with cache usage, hit rate and in-flight jobs (prefill progress, generation speed). Only on an interactive terminal.
log_timestamps	Bool (True)	Prefix console log lines with the time of day. Log files under logs/ always carry full timestamps.
log_chat_completion_requests	Bool (False)	Writes each /v1/chat/completions request to logs/debug as JSON, along with the fully templated prompt (the exact text sent to the tokenizer) as a .txt file with the same basename. Note: this includes full message history and generation parameters, with only API keys redacted.

Sampling Options

Note: This block is for sampling overrides, not samplers themselves.

Config Option	Type (Default)	Description
override_preset	String (None)	Load the given sampler override preset from the sampler_overrides folder at startup (with or without the .yml extension). Startup fails if the preset doesn't exist, and warns if no overrides are set at all. A model's own sampling section (model options) layers on top of it.
(sampler name)	Mapping	Any sampler name may appear directly in this section as an inline override with the preset-file syntax, e.g. temperature: {override: 0.5, force: false}. Applies on top of override_preset, or on its own.
reasoning_override	Mapping (None)	Inline overrides that apply only while a model is reasoning, nested with the same syntax, e.g. reasoning_override: {temperature: {override: 1.5, force: true}}. Unlisted settings are inherited. See Sampler Overrides.

Developer Options

Note: These are experimental flags that may be removed at any point.

Config Option	Type (Default)	Description
unsafe_launch	Bool (False)	Skips dependency checks on startup. Only recommended for debugging.
disable_request_streaming	Bool (False)	Forcefully disables streaming requests
realtime_process_priority	Bool (False)	Set the process priority to "Realtime". Administrator/sudo access required, otherwise the priority is set to the highest it can go in userland.
seqlog	Bool (False)	Enable logging to Seq server
seqlog_server_url	String (http://localhost:5341)	Seq server URL and port
seqlog_api_key	String (None)	API key for Seq server, if required by server

Model Options

Note: Most of the options here will only apply on initial model load/startup (ephemeral). They will not persist unless you add the option name to use_as_default.

Config Option	Type (Default)	Description
model_dir	String ("models")	Directory to look for models. Note: Persisted across subsequent load requests
inline_model_loading	Bool (False)	Enables ability to switch models using the model argument in a generation request. More info in Usage
use_dummy_models	Bool (False)	Send a dummy OAI model card when calling the /v1/models endpoint. Used for clients which enforce specific OAI models. Note: Persisted across subsequent load requests
dummy_model_names	List[String] (["gpt-3.5-turbo"])	List of dummy names to send on model endpoint requests
model_name	String (None)	Folder name of a model to load. The below parameters will not apply unless this is filled out.
use_as_default	List[String] ([])	Keys to use by default when loading models. For example, putting cache_mode in this array will make every model load with that value unless specified by the API request. Note: Also applies to the draft sub-block
max_seq_len	Float (None)	Maximum sequence length of the model. Uses the value from config.json if not specified here. Also called the max context length.
tensor_parallel	Bool (False)	Enables tensor parallelism. Automatically falls back to autosplit if GPU split isn't provided. Note: gpu_split_auto is ignored when this is enabled.
gpu_split_auto	Bool (True)	Automatically split the model across multiple GPUs. Manual GPU split isn't used if this is enabled.
autosplit_reserve	List[Int] ([96])	Amount of empty VRAM to reserve when loading with autosplit. Represented as an array of MB per GPU used.
gpu_split	List[Float] ([])	Float array of GBs to split a model between GPUs.
cpu_moe_offload_layers	Int (0)	Number of mixture-of-expert layers to run on the CPU, with expert weights in system RAM. Only affects MoE models; set a large value such as 999 to offload all layers. Requires layer-split mode (not tensor parallel).
cpu_moe_split_experts	Int (0)	Number of routed experts per MoE layer to run on the CPU, with the coldest experts' weights in system RAM. Unlike cpu_moe_offload_layers this splits every MoE layer instead of offloading whole layers, overlapping CPU and GPU compute, with dynamic placement keeping hot experts in VRAM. Mutually exclusive with cpu_moe_offload_layers; not supported with tensor parallelism.
cpu_moe_threads	Int (None)	Worker thread count for CPU MoE inference; applies to both cpu_moe_offload_layers and cpu_moe_split_experts. When unset, defers to the EXL3_MOE_CPU_THREADS environment variable, then half the CPU core count.
cpu_moe_pinned_arena	Bool (None)	Experimental: keep the CPU MoE worker's expert weights in shared, page-locked memory. Streamed prefill then copies experts straight from it to the GPU, and cpu_moe_split_experts can promote frequently used experts into spare VRAM slots on the fly. Costs about 0.2 s per GiB of offloaded experts at load. Only has an effect with cpu_moe_offload_layers or cpu_moe_split_experts. When unset, defers to the EXL3_MOE_PINNED_ARENA environment variable.
ngram_ram	Bool (False)	Load a model's n-gram embedding table fully into system RAM. Only affects PLE models with n-gram embeddings (e.g. Qwen3.8-Flash-Next). By default the table is streamed from disk during inference; loading it into RAM avoids per-token disk reads at the cost of tens of GB of system memory.
embed_stream_from_disk	Bool (False)	Stream token embeddings from disk on demand instead of holding the whole embedding table in system RAM. Saves memory on large-vocabulary models at the cost of small disk reads during inference.
rope_scale	Float (1.0)	Adjustment for rope scale (or compress_pos_emb) Note: If the model has YaRN support, this option will not apply.
rope_alpha	Float (None)	Adjustment for rope alpha. Leave blank to automatically calculate based on the max_seq_len. Note: If the model has YaRN support, this option will not apply.
cache_mode	String ("FP16")	Cache mode for the model. Specify the pair k_bits,v_bits where each is an integer from 2-8 (e.g. 8,8). The legacy values FP16, Q8, Q6 and Q4 are also accepted.
cache_size	Int (max_seq_len)	Size of the K/V cache Note: If using CFG, the cache size should be 2 * max_seq_len.
chunk_size	Int (2048)	Amount of tokens per chunk with ingestion. A lower value reduces VRAM usage at the cost of ingestion speed.
max_batch_size	Int (None)	The absolute maximum amount of prompts to process at one time. This value is automatically adjusted based on cache size.
recurrent_checkpoint_interval	Int (None)	Tokens between recurrent state checkpoints near the end of the prompt and during generation, for models with recurrent (linear or sliding attention) layers. Must be a multiple of 256. Blank uses the engine's per-architecture default (2048 for most models).
recurrent_checkpoint_interval_pp	Int (None)	Tokens between recurrent state checkpoints during prompt ingestion, more than 2 * chunk_size from the end of the prompt. Must be a multiple of 256; blank uses the engine default of 32768. Recurrent states cannot be rolled back, so editing an earlier part of a cached prompt replays from the last checkpoint before the edit. A denser grid makes that cost proportional to the distance from the edit to the end of the prompt, for one recurrent state of system RAM per checkpoint (bounded by sysmem_recurrent_cache) and a few percent slower cold prefill.
prompt_template	String (None)	Name of a jinja2 chat template to apply for this model. Must be located in the templates directory.
vision	Bool (False)	Enable vision support for the provided model (if it exists).
sampling	Mapping (None)	Sampler overrides for this model, with the same syntax as the top-level sampling section: an optional override_preset plus inline sampler overrides. Applied on top of the global section while the model is loaded; put it in the model folder's tabby_config.yml.
warmup	Bool (False)	Warm up the model after loading: runs a short schedule of forward passes so kernel compilation, autotuning and CUDA graph capture happen at load time rather than on the first requests. Sized from the cache, batch and chunk settings in effect.
vision_offload	Bool (False)	Keep the vision model's weights in pinned system RAM and stream them to the GPU during inference, trading vision speed for VRAM. Only applies when vision is enabled.
template_vars_default	Dict ({})	Default chat template variables, merged into every chat completion request. Values sent by the client (template_vars/chat_template_kwargs, or the top-level reasoning_effort field) take precedence. Example: {reasoning_effort: high}
template_vars_force	Dict ({})	Like template_vars_default, but these override any values sent by the client.
force_enable_thinking	Bool (False)	DEPRECATED: alias for template_vars_force: {enable_thinking: true}.
reasoning	Bool (True)	Enable the reasoning parser: splits the response into reasoning_content and content. With the tokens below at auto, only active when the model's template or tokenizer shows which tags it uses.
reasoning_start_token	String ("auto")	Start token for the reasoning parser. auto takes the tags from the detected tool format or the chat template; set both tokens to override.
reasoning_end_token	String ("auto")	End token for the reasoning parser.
start_in_reasoning	String ("auto")	Whether generation starts inside a reasoning block. Options: auto, always, never. auto scans the end of the templated prompt for an unclosed reasoning start token.
tool_calls_in_reasoning	Bool (True)	Parse tool calls that occur inside reasoning content. If False, tool call tags inside a reasoning block are treated as plain reasoning text.
reasoning_budget_tokens	Int (None)	Default reasoning token budget. When a request's reasoning content exceeds it, the server forces the end of the reasoning phase by injecting reasoning_budget_message followed by the model's end-of-reasoning tokens. 0 ends reasoning as soon as it starts; None or negative disables. Overridable per request via reasoning_budget_tokens (aliases: reasoning_budget, thinking_budget, thinking_token_budget) or reasoning.max_tokens. Requires a reasoning format: reasoning tags, Harmony or Muse Glimmer.
reasoning_budget_message	String (None)	Text injected before the end-of-reasoning tokens when the reasoning budget is exhausted (default: none, matching llama.cpp — only the end-of-reasoning tokens are forced). Overridable per request via reasoning_budget_message.
tool_format	String ("auto")	Tool calling format. auto picks a parser from the model's chat template, tokenizer and architecture and warns if none matches; set a name such as qwen3_5 to override, or blank to disable parsing. See Tool Calling.
harmony	Bool (None)	Parse responses in the Harmony message format (gpt-oss models). Auto-detected from the model's special tokens by default; set to true or false to override. tool_format: harmony is equivalent to setting this to true. When active, supersedes the reasoning and tool format settings.
muse_glimmer	Bool (None)	Parse responses in the Muse Glimmer message format. Auto-detected from the model's special tokens by default; set to true or false to override. tool_format: muse_glimmer is equivalent to setting this to true. When active, supersedes the reasoning and tool format settings.

Draft Model Options

Note: Sub-block of Model Options. Same rules apply.

Config Option	Type (Default)	Description
draft_mode	String ("model")	Exllamav3 drafting mode. Options: model, disabled, mtp, ngram
draft_model_dir	String ("models")	Directory to look for draft models. Note: Persisted across subsequent load requests
draft_model_name	String (None)	String: Folder name of a draft model to load. Only used by model mode.
draft_rope_scale	Float (1.0)	String: RoPE scale value for the draft model.
draft_rope_alpha	Float (1.0)	RoPE alpha value for the draft model. Leave blank for auto-calculation.
draft_cache_mode	String ("FP16")	Cache mode for the draft model. Specify the pair k_bits,v_bits where each is an integer from 2-8 (e.g. 8,8). The legacy values FP16, Q8, Q6 and Q4 are also accepted.
draft_gpu_split	List[Float] ([])	Float array of GBs to split a draft model between GPUs.
draft_num_tokens	Integer (None)	Number of tokens to draft per iteration. Exllamav3 selects the mode-specific default when blank.
dynamic_draft	Bool (False)	Adjust the number of draft tokens dynamically based on observed acceptance rates. Ceiling is given by draft_num_tokens.
ngram_match_min	Integer (2)	Minimum match length for exllamav3 n-gram drafting. Only used by ngram mode.
hybrid_draft_tokens	Integer (0)	Hybrid drafting: when a response is repeating a passage of its context verbatim (reciting a drafted reply, editing source), the earlier continuation is drafted this many tokens at a time and verified in one pass. Works on top of any draft mode, or on its own; 0 disables. Longer values advance more tokens at prefill speed on a hit, at the cost of a wider verification pass.

Lora Options

Note: Sub-block of Mode Options. Same rules apply.

Config Option	Type (Default)	Description
lora_dir	String ("loras")	Directory to look for loras. Note: Persisted across subsequent load requests
loras	List[loras] ([])	List of lora objects to apply to the model. Each object contains a name and scaling.
name	String (None)	Folder name of a lora to load. Note: An element of the loras key
scaling	Float (1.0)	"Weight" to apply the lora on the parent model. For example, applying a lora with 0.9 scaling will lower the amount of application on the parent model. Note: An element of the loras key

Memory Options

Config Option	Type (Default)	Description
sysmem_recurrent_cache	Int (4096)	Max size of the recurrent cache in system memory, in MB. Only used by models with recurrent (linear or sliding attention) layers.
sysmem_kv_cache	Int (0)	Size of a second-tier K/V page cache in system memory, in MB. Complete pages evicted from the GPU cache are stored there and restored on prompt-cache hits instead of being recomputed. Set to 0 to disable.
disk_cache_dir	String (None)	Directory for a persistent third-tier cache on disk (experimental, ExLlamaV3 1.6.2 or later). K/V pages entering the system memory tier and recurrent checkpoints are written through to disk and read back when a prompt's pages are in no other tier, so cached contexts survive a model reload or a server restart. Requires sysmem_kv_cache > 0; ignored with a warning otherwise. Each model and cache configuration (quantization, draft cache, engine version) gets its own subdirectory, so one directory serves every model.
disk_cache_size	Int (51200)	Size budget of the whole disk cache directory, in MB, shared by every model and cache configuration stored under disk_cache_dir. The least recently used entries are removed to stay within it.
disk_cache_flush_on_idle	Bool (True)	Also write every complete page still in VRAM to the disk cache whenever the generator goes idle, so a context is persisted without having to be evicted from the GPU first.
disk_cache_checkpoint_interval	Int (4096)	Minimum spacing, in tokens, between the recurrent checkpoints the disk cache keeps along a chain of cached pages. A resumed prompt replays at most this many tokens past the nearest checkpoint. Checkpoints take far more space than pages on hybrid models; 0 keeps every checkpoint. Only matters for models with recurrent layers.
sysmem_multimodal_cache	Int (1024)	Size of the image embedding cache in system memory, in MB. Encoded images are reused across turns; images a request already uses are never evicted, and a context that exceeds the budget is cached only partially (with a warning). Only applies when vision is enabled.
cuda_malloc_async	Bool (False)	Force Torch's cudaMallocAsync allocator backend. Off by default: unless PYTORCH_CUDA_ALLOC_CONF is set in the environment, ExLlamaV3 enables expandable segments in Torch's native allocator, which performs better.

Embeddings Options

Note: Most of the options here will only apply on initial embedding model load/startup (ephemeral).

Config Option	Type (Default)	Description
embedding_model_dir	String ("models")	Directory to look for embedding models. Note: Persisted across subsequent load requests
embeddings_device	String ("cpu")	Device to load an embedding model on. Options: cpu, cuda, auto Note: Persisted across subsequent load requests
embedding_model_name	String (None)	Folder name of an embedding model to load using infinity-emb.