Allow use of templates without local models

[clemens-zauchner]

💡 Feature Description

As a user without spyre cards, I would like to test the ai-services with watsonx so that I do not have to deploy models using local compute, but still can evaluate future accuracy of a spyre based system

❗ Why This Matters

Spyre cards are new and not a lot of references are available. The success of data and AI related projects depends a lot on the available data. Building a solid business case requires an understanding of the potential of a Spyre based system that includes the models that can be run on the hardware.
Since watsonx can be integrated, I would like to use the models that are available for spyre in watsonx to quickly showcase the capabilities and to build a solid business case.

🧩 Requirements

• only use watsonx as an llm component for all templates

📚 Additional Details

n/a

[dharaneeshvrd]

@clemens-zauchner
Support for WatsonX is already added on latest release.
Can you please go through the docs https://www.ibm.com/docs/en/aiservices/2026.06.0?topic=podman-configuring-powervs-watsonx?
Please let us know if you face any issues while trying this out!

[clemens-zauchner]

yes, it is added, but still vllm containers for embedding, reranker and llm models are started. I would like to have the option to only use watsonx here.

when I try to use watsonx only

ai-services application create test -t rag --runtime podman --params llm.watsonx.model=ibm/granite-4-h-small,llm.watsonx.watsonxApiKey=<apikey>,llm.watsonx.watsonxProjectId=<id>,llm.watsonx.watsonxUrl=https://<region>.ml.cloud.ibm.com --skip-validation spyre,power,rhn

There is an error:

Error: failed to create application after 3 retries: retry failed after 3 attempts with err: create application: server returned HTTP 500: Failed to create application: failed to create deployment plan: failed to allocate Spyre cards: insufficient Spyre cards: required 5, available 0

I have tried to configure additional params like reranker.vllm-spyre=false,llm.vllm-spyre=false,reranker.vllm-cpu=false,llm.vllm-cpu=false and combinations, but it would either try to deploy using spyre, or use the cpu based vllm.

[dharaneeshvrd]

Ok currently support for running embedding and reranker on WatsonX is not added. We will consider this scenario. Thankyou!

[clemens-zauchner]

but with the configuration above, the llm container is also created. or is there a parameter that is not documented that prevents it?

[dharaneeshvrd]

@yussufsh Can you please help here?

[yussufsh]

@clemens-zauchner Please try using just reranker.vllm-cpu to run the reranker on the CPU and llm.watsonx.* to use watsonx.ai.
Example:

ai-services application create test -t rag --runtime podman --params reranker.vllm-cpu=true,llm.watsonx.model=ibm/granite-4-h-small,llm.watsonx.watsonxApiKey=<apikey>,llm.watsonx.watsonxProjectId=<id>,llm.watsonx.watsonxUrl=https://<region>.ml.cloud.ibm.com --skip-validation spyre,power,rhn

I see there is an issue when you provide multiple parameters at once due to Golang map iteration. Will try to provide a fix for it. Thanks for reporting the issue.

I also suggest trying the Catalog GUI for deployment, which provides better usability and proper input validation.

[clemens-zauchner]

With the command above, there are 2 vllm containers created, one for reranker and one for embeddings.

In the catalog GUI, I can only choose Spyre or CPU based deployments, but not watsonx.

[yussufsh]

With the command above, there are 2 vllm containers created, one for reranker and one for embeddings.

Hope it is what is expected. 😄

In the catalog GUI, I can only choose Spyre or CPU based deployments, but not watsonx.

You need to select the granite-4-h-small model, which is supported on watsonx (it will show in the dropdown when you select).

[yussufsh]

@clemens-zauchner did you try exploring watsonx support? Let us know if you are having any issues.
I have also added warnings so the user is aware we use spyre by default and a fix for this...

I see there is an issue when you provide multiple parameters at once due to Golang map iteration. Will try to provide a fix for it. Thanks for reporting the issue.

[clemens-zauchner]

yes, but since i cannot prevent vllm containers to start, I cannot roll the stack out properly and so far I do not have the option to configure watsonx from the UI.