Skip to content

Distinguish unmet required options from unsupported capabilities in model selection errors #254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
saarnilauri wants to merge 3 commits into
base: trunk
Choose a base branch
from
Open

Distinguish unmet required options from unsupported capabilities in model selection errors #254

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a063070
Add ModelRequirements::getUnmetRequiredOptions()
saarnilauri Jun 12, 2026
188bf29
Distinguish unmet options from unsupported capabilities in model sele…
saarnilauri Jun 12, 2026
6d582f1
Correct stale docblocks about explicit model validation
saarnilauri Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions src/Builders/PromptBuilder.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1126,14 +1126,15 @@ protected function appendPartToMessages(MessagePart $part): void
/**
* Gets the model to use for generation.
*
* If a model has been explicitly set, validates it meets requirements and returns it.
* If a model has been explicitly set, it is used as-is without validating it against the prompt
* requirements; unsupported parameters surface as errors from the provider's API.
* Otherwise, finds a suitable model based on the prompt requirements.
*
* @since 0.1.0
*
* @param CapabilityEnum $capability The capability the model will be using.
* @return ModelInterface The model to use.
* @throws InvalidArgumentException If no suitable model is found or set model doesn't meet requirements.
* @throws InvalidArgumentException If no suitable model is found.
*/
private function getConfiguredModel(CapabilityEnum $capability): ModelInterface
{
Expand Down
186 changes: 135 additions & 51 deletions src/Providers/ModelResolver.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
use WordPress\AiClient\Providers\Models\DTO\ModelConfig;
use WordPress\AiClient\Providers\Models\DTO\ModelMetadata;
use WordPress\AiClient\Providers\Models\DTO\ModelRequirements;
use WordPress\AiClient\Providers\Models\DTO\RequiredOption;

/**
* Resolves the concrete AI model to use based on selection preferences.
Expand Down Expand Up @@ -195,17 +196,18 @@ public function setRequestOptions(RequestOptions $requestOptions): void
/**
* Resolves the model to use for the given requirements.
*
* If a model has been explicitly set, updates its config, binds dependencies,
* and returns it. Otherwise, finds a suitable model based on the requirements,
* honoring any configured model preferences and provider constraint.
* If a model has been explicitly set, it is used as-is without validating it against the
* requirements; its config is updated, dependencies are bound, and unsupported parameters
* surface as errors from the provider's API. Otherwise, finds a suitable model based on the
* requirements, honoring any configured model preferences and provider constraint.
*
* @since 1.4.0
*
* @param ModelRequirements $requirements The requirements the model must satisfy.
* @param ModelConfig $modelConfig The model configuration to apply.
* @param string|null $subject Optional caller-specific subject for failure messages.
* @return ModelInterface The model to use.
* @throws InvalidArgumentException If no suitable model is found or set model doesn't meet requirements.
* @throws InvalidArgumentException If no suitable model is found.
*/
public function resolve(
ModelRequirements $requirements,
Expand All @@ -225,53 +227,9 @@ public function resolve(
$candidateMap = $this->getCandidateModelsMap($requirements);

if (empty($candidateMap)) {
// The primary capability is always the first required capability (see
// ModelRequirements::fromPromptData()/fromEmbeddingData()).
$requiredCapabilities = $requirements->getRequiredCapabilities();
$primaryCapability = reset($requiredCapabilities);
if ($primaryCapability === false) {
$message = 'No models found that meet the requested requirements.';

if ($this->providerIdOrClassName !== null) {
$message = sprintf(
'No models found for provider "%s" that meet the requested requirements.',
$this->providerIdOrClassName
);
}

throw new InvalidArgumentException($message);
}

$capabilityValue = $primaryCapability->value;

$message = sprintf('No models found that support %s.', $capabilityValue);

if ($subject !== null) {
$message = sprintf(
'No models found that support %s for this %s.',
$capabilityValue,
$subject
);
}

if ($this->providerIdOrClassName !== null) {
$message = sprintf(
'No models found for provider "%s" that support %s.',
$this->providerIdOrClassName,
$capabilityValue
);

if ($subject !== null) {
$message = sprintf(
'No models found for provider "%s" that support %s for this %s.',
$this->providerIdOrClassName,
$capabilityValue,
$subject
);
}
}

throw new InvalidArgumentException($message);
throw new InvalidArgumentException(
$this->getNoSuitableModelsMessage($requirements, $subject)
);
}

// Check if any preferred models match the candidates, in priority order.
Expand Down Expand Up @@ -326,6 +284,132 @@ public function isSupported(ModelRequirements $requirements): bool
}
}

/**
* Builds the exception message for when no models satisfy the given requirements.
*
* Distinguishes between no model supporting the required capability at all and models supporting
* the capability but not the required options. This avoids misleading messages where an
* unsupported option (e.g. a sampling parameter such as temperature) would otherwise be reported
* as the capability itself being unsupported.
*
* @since n.e.x.t
*
* @param ModelRequirements $requirements The requirements no model satisfied.
* @param string|null $subject Optional caller-specific subject for failure messages.
* @return string The exception message.
*/
private function getNoSuitableModelsMessage(ModelRequirements $requirements, ?string $subject): string
{
$scope = '';
if ($this->providerIdOrClassName !== null) {
$scope = sprintf(' for provider "%s"', $this->providerIdOrClassName);
}

$subjectSuffix = '';
if ($subject !== null) {
$subjectSuffix = sprintf(' for this %s', $subject);
}

// The primary capability is always the first required capability (see
// ModelRequirements::fromPromptData()/fromEmbeddingData()).
$requiredCapabilities = $requirements->getRequiredCapabilities();
$primaryCapability = reset($requiredCapabilities);
if ($primaryCapability === false) {
return sprintf('No models found%s that meet the requested requirements.', $scope);
}

$capabilityValue = $primaryCapability->value;

$capabilityCandidates = [];
if (count($requirements->getRequiredOptions()) > 0) {
// Check whether models would qualify based on the required capabilities alone.
$capabilityCandidates = $this->findCandidateModelsMetadata(
new ModelRequirements($requirements->getRequiredCapabilities(), [])
);
}

if (count($capabilityCandidates) === 0) {
return sprintf(
'No models found%s that support %s%s.',
$scope,
$capabilityValue,
$subjectSuffix
);
}

/*
* Models support the required capabilities, so the required options are what excluded them.
* Determine which option names are unmet by every candidate (definite blockers) and which
* are unmet by at least one candidate (relevant when only the combination is unsupported).
*/
$unmetByAll = null;
$unmetByAny = [];
foreach ($capabilityCandidates as $modelMetadata) {
$unmetNames = array_map(
static function (RequiredOption $option): string {
return $option->getName()->value;
},
$requirements->getUnmetRequiredOptions($modelMetadata)
);

$unmetByAny = array_merge($unmetByAny, $unmetNames);
$unmetByAll = $unmetByAll === null ? $unmetNames : array_intersect($unmetByAll, $unmetNames);
}
$unmetByAll = array_values(array_unique($unmetByAll));
$unmetByAny = array_values(array_unique($unmetByAny));

if (count($unmetByAll) > 0) {
$detail = sprintf(
'Models supporting %s are available, but none of them support the following required options: %s.',
$capabilityValue,
implode(', ', $unmetByAll)
);
} else {
$detail = sprintf(
'Models supporting %s are available, ' .
'but no single model supports all of the following required options together: %s.',
$capabilityValue,
implode(', ', $unmetByAny)
);
}

return sprintf(
'No models found%s that support %s with the required options%s. %s',
$scope,
$capabilityValue,
$subjectSuffix,
$detail
);
}

/**
* Finds the metadata of all models that satisfy the given requirements.
*
* Honors the configured provider restriction, if any.
*
* @since n.e.x.t
*
* @param ModelRequirements $requirements The requirements to match against.
* @return list<ModelMetadata> The metadata of the matching models.
*/
private function findCandidateModelsMetadata(ModelRequirements $requirements): array
{
if ($this->providerIdOrClassName === null) {
$modelsMetadata = [];
foreach ($this->registry->findModelsMetadataForSupport($requirements) as $providerModelsMetadata) {
foreach ($providerModelsMetadata->getModels() as $modelMetadata) {
$modelsMetadata[] = $modelMetadata;
}
}
return $modelsMetadata;
}

return $this->registry->findProviderModelsMetadataForSupport(
$this->providerIdOrClassName,
$requirements
);
}

/**
* Binds configured request options to the model if present and supported.
*
Expand Down
44 changes: 29 additions & 15 deletions src/Providers/Models/DTO/ModelRequirements.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,17 +102,12 @@ public function getRequiredOptions(): array
*/
public function areMetBy(ModelMetadata $metadata): bool
{
// Create lookup maps for better performance (instead of nested foreach loops)
// Create lookup map for better performance (instead of nested foreach loops)
$capabilitiesMap = [];
foreach ($metadata->getSupportedCapabilities() as $capability) {
$capabilitiesMap[$capability->value] = $capability;
}

$optionsMap = [];
foreach ($metadata->getSupportedOptions() as $option) {
$optionsMap[$option->getName()->value] = $option;
}

// Check if all required capabilities are supported using map lookup
foreach ($this->requiredCapabilities as $requiredCapability) {
if (!isset($capabilitiesMap[$requiredCapability->value])) {
Expand All @@ -121,21 +116,40 @@ public function areMetBy(ModelMetadata $metadata): bool
}

// Check if all required options are supported with the specified values
return count($this->getUnmetRequiredOptions($metadata)) === 0;
}

/**
* Returns the required options that the given model metadata does not support.
*
* A required option is unmet when the model does not list it as a supported
* option at all, or when the required value is not among the option's
* supported values.
*
* @since n.e.x.t
*
* @param ModelMetadata $metadata The model metadata to check against.
* @return list<RequiredOption> The required options the model does not support.
*/
public function getUnmetRequiredOptions(ModelMetadata $metadata): array
{
// Create lookup map for better performance (instead of nested foreach loops)
$optionsMap = [];
foreach ($metadata->getSupportedOptions() as $option) {
$optionsMap[$option->getName()->value] = $option;
}

$unmetOptions = [];
foreach ($this->requiredOptions as $requiredOption) {
// Use map lookup instead of linear search
if (!isset($optionsMap[$requiredOption->getName()->value])) {
return false;
}
$supportedOption = $optionsMap[$requiredOption->getName()->value] ?? null;

$supportedOption = $optionsMap[$requiredOption->getName()->value];

// Check if the required value is supported by this option
if (!$supportedOption->isSupportedValue($requiredOption->getValue())) {
return false;
if ($supportedOption === null || !$supportedOption->isSupportedValue($requiredOption->getValue())) {
$unmetOptions[] = $requiredOption;
}
}

return true;
return $unmetOptions;
}

/**
Expand Down
Loading
Loading