OpenAI-Compatible LLM

Send prompts to any OpenAI-compatible API endpoint.

Send a prompt to any OpenAI-compatible API endpoint (e.g. local Qwen, vLLM, Ollama, LM Studio, or any service that implements the OpenAI chat completions API).

How this block works

1. You provide a Base URL (e.g. http://localhost:8000/v1) and a Model Name.

2. Write an Instruction - the text the model receives.

3. Add rows under Inputs to feed step outputs (images, detections, text) into the request. Image inputs are base64-encoded and sent as vision content parts. A list of images becomes one vision part per image.

4. Non-image inputs are converted to strings. To splice them into the instruction text, reference the input by name with the placeholder syntax shown in the Instruction field's help text.

5. Optionally apply UQL operations to transform input values before insertion.

Image handling

• A WorkflowImageData value is JPEG-encoded and sent as an image_url part.

• Raw JPEG bytes (e.g. from the Image Stack block) are sent directly.

• A list of either is fanned out into multiple image_url parts.

If an image input is also referenced in the instruction text by name, the placeholder is removed from the text - the image only travels as a vision part.

Type identifier

Use the following identifier in step "type" field: roboflow_core/openai_compatible@v1 to add the block as a step in your workflow.

Properties

Name	Type	Description	Refs
name	str	Enter a unique identifier for this step..	❌
base_url	str	URL of the OpenAI-compatible server, including /v1..	✅
model_name	str	Model identifier sent to the server..	✅
api_key	str	API key, if the endpoint requires one..	✅
system_prompt	str	Optional system message that sets model behavior..	✅
prompt	str	Text sent to the model..	✅
prompt_parameters	Dict[str, Union[bool, float, int, str]]	Step outputs to include in the request (images or text)..	✅
prompt_parameters_operations	Dict[str, List[Union[ClassificationPropertyExtract, ConvertDictionaryToJSON, ConvertImageToBase64, ConvertImageToJPEG, DetectionsFilter, DetectionsOffset, DetectionsPropertyExtract, DetectionsRename, DetectionsSelection, DetectionsShift, DetectionsToDictionary, Divide, ExtractDetectionProperty, ExtractFrameMetadata, ExtractImageProperty, LookupTable, Multiply, NumberRound, NumericSequenceAggregate, PickDetectionsByParentClass, RandomNumber, SequenceAggregate, SequenceApply, SequenceElementsCount, SequenceLength, SequenceMap, SortDetections, StringMatches, StringSubSequence, StringToLowerCase, StringToUpperCase, TimestampToISOFormat, ToBoolean, ToNumber, ToString]]]	Optional UQL operations applied to inputs before use..	❌
max_tokens	int	Maximum tokens the model may generate..	❌
temperature	float	Sampling temperature, 0.0 to 2.0..	✅
extra_body	Dict[Any, Any]	Extra JSON forwarded as the OpenAI SDK extra_body argument..	❌

The Refs column marks possibility to parametrise the property with dynamic values available in workflow runtime. See Bindings for more info.

Runtime compatibility

requires_internet - air-gapped / offline deployments : This block depends on a service that is not reachable from fully offline / air-gapped deployments.

Input and Output Bindings

The available connections depend on its binding kinds. Check what binding kinds OpenAI-Compatible LLM in version v1 has.

Input and output bindings

• input

  • base_url (string): URL of the OpenAI-compatible server, including /v1..

  • model_name (string): Model identifier sent to the server..

  • api_key (Union[secret, string]): API key, if the endpoint requires one..

  • system_prompt (string): Optional system message that sets model behavior..

  • prompt (string): Text sent to the model..

  • prompt_parameters (*): Step outputs to include in the request (images or text)..

  • temperature (float): Sampling temperature, 0.0 to 2.0..

• output

  • output (Union[string, language_model_output]): String value if string or LLM / VLM output if language_model_output.

  • error_status (string): String value.

Example JSON definition

{
	    "name": "<your_step_name_here>",
	    "type": "roboflow_core/openai_compatible@v1",
	    "base_url": "http://localhost:8000/v1",
	    "model_name": "Qwen/Qwen2.5-VL-7B-Instruct",
	    "api_key": "xxx-xxx",
	    "system_prompt": "You are a helpful assistant.",
	    "prompt": "Describe what you see in the image.",
	    "prompt_parameters": {
	        "detections": "$steps.model.predictions",
	        "frames": "$steps.image_stack.frames"
	    },
	    "prompt_parameters_operations": {
	        "detections": [
	            {
	                "property_name": "class_name",
	                "type": "DetectionsPropertyExtract"
	            }
	        ]
	    },
	    "max_tokens": "<block_does_not_provide_example>",
	    "temperature": "<block_does_not_provide_example>",
	    "extra_body": {
	        "chat_template_kwargs": {
	            "enable_thinking": false
	        },
	        "guided_choice": [
	            "A",
	            "B",
	            "C",
	            "D"
	        ]
	    }
	}