# Scores ## Retrieve Prompt `models.scores.retrieve_prompt(strmodel_id, ScoreRetrievePromptParams**kwargs) -> ScoreRetrievePromptResponse` **get** `/models/{modelId}/scores/prompt` Get the prompt scores for the given `modelId` ### Parameters - `model_id: str` - `project_id: Optional[str]` The projectId used for ownership resource management. Either to assert ownership or to set the owner of the resource(s) - `prompt: Optional[str]` The prompt to score against the model. Required — the call returns 400 without it. ### Returns - `class ScoreRetrievePromptResponse: …` - `details: Details` Different components used to compute the final score - `captions_similarity: DetailsCaptionsSimilarity` The Dice-Sørensen similarity between the prompt and each captions from the model's training images The values are between 0 and 1. The higher the value, the more similar the prompt is to the training images' captions - `mean: float` - `median: float` - `min: float` - `embedding_similarity: float` The cosine similarity between the prompt and the model's caption embedding The value is a number between -1 and 1. The higher the value, the more similar the prompt is to the model's caption embedding - `prompts_similarity: Optional[DetailsPromptsSimilarity]` The Dice-Sørensen similarity between the prompt and each prompts from the model's examples The values are between 0 and 1. The higher the value, the more similar the prompt is to the prompts' examples - `mean: float` - `median: float` - `min: float` - `model_id: str` The model ID (example: "model_eyVcnFJcR92BxBkz7N6g5w") - `score: float` The prompt's score for the input modelId The score is a number between 0 and 1. The higher the score, the better the prompt is for the model - `status: Literal["complete", "incomplete", "unknown"]` The status of the prompt's score Details if all the necessary data is present to compute the score - `"complete"` - `"incomplete"` - `"unknown"` - `syntax_error: Optional[Literal["Prompt weight should be between 0 and 2", "Prompt weight symbol should be - or + with a maximum length of 5. Ex: --, ++, ---, +++++", "Unsupported prompt syntax: '[to:when], [from::when]'", 4 more]]` Present only if the prompt has one or more syntax errors The API checks several syntax errors in the prompt - `"Prompt weight should be between 0 and 2"` - `"Prompt weight symbol should be - or + with a maximum length of 5. Ex: --, ++, ---, +++++"` - `"Unsupported prompt syntax: '[to:when], [from::when]'"` - `"Unsupported prompt syntax: 'word:' or '(word):'. Please use 'word' or '(word)' instead"` - `"Unsupported prompt syntax: --(s|c)ref"` - `"Unsupported prompt syntax: or "` - `"Unsupported url in prompt"` ### Example ```python import os from scenario_sdk import Scenario client = Scenario( api_key=os.environ.get("SCENARIO_SDK_API_KEY"), # This is the default and can be omitted api_secret=os.environ.get("SCENARIO_SDK_API_SECRET"), # This is the default and can be omitted ) response = client.models.scores.retrieve_prompt( model_id="modelId", ) print(response.details) ``` #### Response ```json { "details": { "captionsSimilarity": { "mean": 0, "median": 0, "min": 0 }, "embeddingSimilarity": 0, "promptsSimilarity": { "mean": 0, "median": 0, "min": 0 } }, "modelId": "modelId", "score": 0, "status": "complete", "syntaxError": "Prompt weight should be between 0 and 2" } ``` ## Retrieve Training Dataset `models.scores.retrieve_training_dataset(strmodel_id, ScoreRetrieveTrainingDatasetParams**kwargs) -> ScoreRetrieveTrainingDatasetResponse` **get** `/models/{modelId}/scores/training-dataset` Get the training dataset scores for the given `modelId` ### Parameters - `model_id: str` - `project_id: Optional[str]` The projectId used for ownership resource management. Either to assert ownership or to set the owner of the resource(s) ### Returns - `class ScoreRetrieveTrainingDatasetResponse: …` - `details: Details` Different components used to compute the final score - `asset_couples_captions_distance: List[DetailsAssetCouplesCaptionsDistance]` For each couple of assets in the training dataset, the distance between the captions - `couple: List[str]` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `distance: float` The Dice-Sørensen distance between the captions of the assetIds of the couple. - `asset_couples_similarity: List[DetailsAssetCouplesSimilarity]` For each couple of assets in the training dataset, the similarity between the assets - `couple: List[str]` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `similarity: float` The cosine similarity between the assetIds of the couple. The value is a number between -1 and 1. The higher the value, the more similar the assets are to each other - `assets_embedding_similarity: List[DetailsAssetsEmbeddingSimilarity]` For each asset, the similarity with the model's embedding (excluding the current asset) - `asset_id: str` The AssetId associated with their similarity (example: "asset_GTrL3mq4SXWyMxkOHRxlpw") - `similarity: float` The cosine similarity between the assetId and the model's embedding minus the asset's embedding The value is a number between -1 and 1. The higher the value, the more similar the asset is to the model's embedding - `assets_similarity: DetailsAssetsSimilarity` The min, mean, median of the assetsEmbeddingSimilarity - `mean: float` - `median: float` - `min: float` - `couples_captions_distance: DetailsCouplesCaptionsDistance` The min, mean, median of the assetCouplesCaptionsDistance - `mean: float` - `median: float` - `min: float` - `couples_similarity: DetailsCouplesSimilarity` The min, mean, median of the assetCouplesSimilarity - `mean: float` - `median: float` - `min: float` - `metadata_score: float` Takes into account all missing data and warnings to compute a metadata score - `model_id: str` The model ID (example: "model_eyVcnFJcR92BxBkz7N6g5w") - `score: float` The training dataset's score for the input modelId The score is a number between 0 and 1. The higher the score, the most chance the model has to train well - `status: Literal["complete", "incomplete", "unknown"]` The status of the training dataset's score Details if all the necessary data is present to compute the score - `"complete"` - `"incomplete"` - `"unknown"` - `asset_couples_warnings: Optional[List[AssetCouplesWarning]]` Present only if their is a warning between the assets of the couples - `couple: List[str]` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `message: List[Literal["Captions are identical", "Captions are too close", "Empty caption", 6 more]]` An indicative message concerning the couple of assets - `"Captions are identical"` - `"Captions are too close"` - `"Empty caption"` - `"Missing caption"` - `"Missing metadata"` - `"Same asset"` - `"Similarity is too high"` - `"Similarity with other assets is too low"` - `"Width and height must be larger than 1024"` - `assets_warnings: Optional[List[AssetsWarning]]` Present only if the training image assets of the modelId have one or more warnings - `asset_id: str` The AssetId which has a warning (example: "asset_GTrL3mq4SXWyMxkOHRxlpw") - `message: List[Literal["Captions are identical", "Captions are too close", "Empty caption", 6 more]]` An indicative message concerning the asset - `"Captions are identical"` - `"Captions are too close"` - `"Empty caption"` - `"Missing caption"` - `"Missing metadata"` - `"Same asset"` - `"Similarity is too high"` - `"Similarity with other assets is too low"` - `"Width and height must be larger than 1024"` ### Example ```python import os from scenario_sdk import Scenario client = Scenario( api_key=os.environ.get("SCENARIO_SDK_API_KEY"), # This is the default and can be omitted api_secret=os.environ.get("SCENARIO_SDK_API_SECRET"), # This is the default and can be omitted ) response = client.models.scores.retrieve_training_dataset( model_id="modelId", ) print(response.details) ``` #### Response ```json { "details": { "assetCouplesCaptionsDistance": [ { "couple": [ "string", "string" ], "distance": 0 } ], "assetCouplesSimilarity": [ { "couple": [ "string", "string" ], "similarity": -1 } ], "assetsEmbeddingSimilarity": [ { "assetId": "assetId", "similarity": -1 } ], "assetsSimilarity": { "mean": -1, "median": -1, "min": -1 }, "couplesCaptionsDistance": { "mean": 0, "median": 0, "min": 0 }, "couplesSimilarity": { "mean": -1, "median": -1, "min": -1 }, "metadataScore": -1 }, "modelId": "modelId", "score": 0, "status": "complete", "assetCouplesWarnings": [ { "couple": [ "string", "string" ], "message": [ "Captions are identical" ] } ], "assetsWarnings": [ { "assetId": "assetId", "message": [ "Captions are identical" ] } ] } ``` ## Domain Types ### Score Retrieve Prompt Response - `class ScoreRetrievePromptResponse: …` - `details: Details` Different components used to compute the final score - `captions_similarity: DetailsCaptionsSimilarity` The Dice-Sørensen similarity between the prompt and each captions from the model's training images The values are between 0 and 1. The higher the value, the more similar the prompt is to the training images' captions - `mean: float` - `median: float` - `min: float` - `embedding_similarity: float` The cosine similarity between the prompt and the model's caption embedding The value is a number between -1 and 1. The higher the value, the more similar the prompt is to the model's caption embedding - `prompts_similarity: Optional[DetailsPromptsSimilarity]` The Dice-Sørensen similarity between the prompt and each prompts from the model's examples The values are between 0 and 1. The higher the value, the more similar the prompt is to the prompts' examples - `mean: float` - `median: float` - `min: float` - `model_id: str` The model ID (example: "model_eyVcnFJcR92BxBkz7N6g5w") - `score: float` The prompt's score for the input modelId The score is a number between 0 and 1. The higher the score, the better the prompt is for the model - `status: Literal["complete", "incomplete", "unknown"]` The status of the prompt's score Details if all the necessary data is present to compute the score - `"complete"` - `"incomplete"` - `"unknown"` - `syntax_error: Optional[Literal["Prompt weight should be between 0 and 2", "Prompt weight symbol should be - or + with a maximum length of 5. Ex: --, ++, ---, +++++", "Unsupported prompt syntax: '[to:when], [from::when]'", 4 more]]` Present only if the prompt has one or more syntax errors The API checks several syntax errors in the prompt - `"Prompt weight should be between 0 and 2"` - `"Prompt weight symbol should be - or + with a maximum length of 5. Ex: --, ++, ---, +++++"` - `"Unsupported prompt syntax: '[to:when], [from::when]'"` - `"Unsupported prompt syntax: 'word:' or '(word):'. Please use 'word' or '(word)' instead"` - `"Unsupported prompt syntax: --(s|c)ref"` - `"Unsupported prompt syntax: or "` - `"Unsupported url in prompt"` ### Score Retrieve Training Dataset Response - `class ScoreRetrieveTrainingDatasetResponse: …` - `details: Details` Different components used to compute the final score - `asset_couples_captions_distance: List[DetailsAssetCouplesCaptionsDistance]` For each couple of assets in the training dataset, the distance between the captions - `couple: List[str]` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `distance: float` The Dice-Sørensen distance between the captions of the assetIds of the couple. - `asset_couples_similarity: List[DetailsAssetCouplesSimilarity]` For each couple of assets in the training dataset, the similarity between the assets - `couple: List[str]` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `similarity: float` The cosine similarity between the assetIds of the couple. The value is a number between -1 and 1. The higher the value, the more similar the assets are to each other - `assets_embedding_similarity: List[DetailsAssetsEmbeddingSimilarity]` For each asset, the similarity with the model's embedding (excluding the current asset) - `asset_id: str` The AssetId associated with their similarity (example: "asset_GTrL3mq4SXWyMxkOHRxlpw") - `similarity: float` The cosine similarity between the assetId and the model's embedding minus the asset's embedding The value is a number between -1 and 1. The higher the value, the more similar the asset is to the model's embedding - `assets_similarity: DetailsAssetsSimilarity` The min, mean, median of the assetsEmbeddingSimilarity - `mean: float` - `median: float` - `min: float` - `couples_captions_distance: DetailsCouplesCaptionsDistance` The min, mean, median of the assetCouplesCaptionsDistance - `mean: float` - `median: float` - `min: float` - `couples_similarity: DetailsCouplesSimilarity` The min, mean, median of the assetCouplesSimilarity - `mean: float` - `median: float` - `min: float` - `metadata_score: float` Takes into account all missing data and warnings to compute a metadata score - `model_id: str` The model ID (example: "model_eyVcnFJcR92BxBkz7N6g5w") - `score: float` The training dataset's score for the input modelId The score is a number between 0 and 1. The higher the score, the most chance the model has to train well - `status: Literal["complete", "incomplete", "unknown"]` The status of the training dataset's score Details if all the necessary data is present to compute the score - `"complete"` - `"incomplete"` - `"unknown"` - `asset_couples_warnings: Optional[List[AssetCouplesWarning]]` Present only if their is a warning between the assets of the couples - `couple: List[str]` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `message: List[Literal["Captions are identical", "Captions are too close", "Empty caption", 6 more]]` An indicative message concerning the couple of assets - `"Captions are identical"` - `"Captions are too close"` - `"Empty caption"` - `"Missing caption"` - `"Missing metadata"` - `"Same asset"` - `"Similarity is too high"` - `"Similarity with other assets is too low"` - `"Width and height must be larger than 1024"` - `assets_warnings: Optional[List[AssetsWarning]]` Present only if the training image assets of the modelId have one or more warnings - `asset_id: str` The AssetId which has a warning (example: "asset_GTrL3mq4SXWyMxkOHRxlpw") - `message: List[Literal["Captions are identical", "Captions are too close", "Empty caption", 6 more]]` An indicative message concerning the asset - `"Captions are identical"` - `"Captions are too close"` - `"Empty caption"` - `"Missing caption"` - `"Missing metadata"` - `"Same asset"` - `"Similarity is too high"` - `"Similarity with other assets is too low"` - `"Width and height must be larger than 1024"`