How does the Python Backend create multiple instances with instance_kind: count setting?

[IMG-PRCSNG]

Does the python backend load the model and fork to many instances? Can we configure it to use the equivalent of multiprocessing spawn mode?

I am using a custom backend to run some old caffe models but finding the throughput remaining the same when I increase the instance count, when testing with perf_analyzer. CPU and GPU are not at 100% utilisation.

I think the caffe is being imported in the parent process and somehow is shared with the child, thus only allowing sequential access.

[demirciberk]

Confirmed facts (from Triton python_backend docs/source):

1. Each Python-backend model instance runs in its own OS process, not a thread. Per the python_backend README, "Multiple Model Instance Support":

   "In order to work around this issue [Python objects serializing all operations], Python backend spawns a separate process for each model instance."

   This is exec-based process creation of the triton_python_backend_stub binary (see src/pb_stub.cc), driven by the C++ core per TRITONBACKEND_ModelInstance (see backend/README.md — Backend API). Each stub process independently imports your model.py and runs its own initialize(). There is no fork() of an already-initialized parent interpreter and no shared GIL/interpreter state between instances — this is architecturally different from Python's multiprocessing fork mode; it is closer in spirit to spawn mode already (fresh process, fresh import), so there is no spawn-mode config knob because you already get separate-process isolation by default.

   • Docs: https://github.com/triton-inference-server/python_backend#multiple-model-instance-support
   • config.pbtxt instance_group semantics: https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups

2. This behavior is specific to the Python backend. Backends like ONNX Runtime, TensorRT, PyTorch libtorch create additional threads, not processes, when count is increased (same README section). So "python backend" vs "custom backend" matters: if you're actually using a custom C++/other backend (not python_backend), instance creation semantics depend entirely on how that backend implements TRITONBACKEND_ModelInstanceInitialize — Triton core just calls it once per configured instance; whether it threads or forks is up to the backend's own code, Triton doesn't fork/spawn Python interpreters for non-Python backends at all.

3. Related upstream issues confirm the same design and known limitations:

   • python backend model instances as threads instead of separate processes? #3560 (separate CUDA context per Python instance — confirms per-instance process isolation, including duplicated CUDA context cost)
   • Python backend: enable sharing memory between multiple instances of a model #4975 (feature request to fork worker processes from an already-initialized master instead of independent initialize() per instance — not implemented, discussion suggests decoupled/BLS-based custom multiplexing as a workaround)

Hypothesis (not confirmed, needs more info from you):

• Since each Python-backend instance is a genuinely separate OS process with its own interpreter, "the caffe module being imported in a parent process and shared with children via fork" as described in the discussion is unlikely to be the root cause of flat throughput if you are truly using python_backend — there's no shared parent interpreter to inherit from. If you're instead writing a custom (non-Python) backend, the flat-throughput/no-100%-utilization symptom is more likely one of:
  • The custom backend's TRITONBACKEND_ModelInstanceExecute internally serializing work (e.g., a global lock/mutex, or Caffe/BLAS libraries pinned to a single thread pool shared across instances).
  • Caffe (or its underlying BLAS: OpenBLAS/MKL) spawning its own thread pool that is shared/oversubscribed across instances, causing contention rather than parallelism — check OMP_NUM_THREADS/OPENBLAS_NUM_THREADS and whether Caffe was built with a threading backend that doesn't parallelize well across independent instances.
  • perf_analyzer concurrency/request settings not actually issuing concurrent requests (verify with --concurrency-range and check -v output / request counts, not just CPU%).

What would help narrow this down:

• Confirm whether you're using python_backend (with a Python wrapper calling into Caffe) or a genuinely custom C++ backend.
• Your config.pbtxt instance_group block.
• perf_analyzer command line and concurrency settings used.
• Whether GPU/CPU utilization is checked per-process (e.g., nvidia-smi/htop showing N separate triton_python_backend_stub processes each partially utilized, vs one process pegged).