# Scores ## Retrieve Prompt **get** `/models/{modelId}/scores/prompt` Get the prompt scores for the given `modelId` ### Path Parameters - `modelId: string` ### Query Parameters - `projectId: optional string` The projectId used for ownership resource management. Either to assert ownership or to set the owner of the resource(s) - `prompt: optional string` The prompt to score against the model. Required — the call returns 400 without it. ### Returns - `details: object { captionsSimilarity, embeddingSimilarity, promptsSimilarity }` Different components used to compute the final score - `captionsSimilarity: object { mean, median, min }` The Dice-Sørensen similarity between the prompt and each captions from the model's training images The values are between 0 and 1. The higher the value, the more similar the prompt is to the training images' captions - `mean: number` - `median: number` - `min: number` - `embeddingSimilarity: number` The cosine similarity between the prompt and the model's caption embedding The value is a number between -1 and 1. The higher the value, the more similar the prompt is to the model's caption embedding - `promptsSimilarity: optional object { mean, median, min }` The Dice-Sørensen similarity between the prompt and each prompts from the model's examples The values are between 0 and 1. The higher the value, the more similar the prompt is to the prompts' examples - `mean: number` - `median: number` - `min: number` - `modelId: string` The model ID (example: "model_eyVcnFJcR92BxBkz7N6g5w") - `score: number` The prompt's score for the input modelId The score is a number between 0 and 1. The higher the score, the better the prompt is for the model - `status: "complete" or "incomplete" or "unknown"` The status of the prompt's score Details if all the necessary data is present to compute the score - `"complete"` - `"incomplete"` - `"unknown"` - `syntaxError: optional "Prompt weight should be between 0 and 2" or "Prompt weight symbol should be - or + with a maximum length of 5. Ex: --, ++, ---, +++++" or "Unsupported prompt syntax: '[to:when], [from::when]'" or 4 more` Present only if the prompt has one or more syntax errors The API checks several syntax errors in the prompt - `"Prompt weight should be between 0 and 2"` - `"Prompt weight symbol should be - or + with a maximum length of 5. Ex: --, ++, ---, +++++"` - `"Unsupported prompt syntax: '[to:when], [from::when]'"` - `"Unsupported prompt syntax: 'word:' or '(word):'. Please use 'word' or '(word)' instead"` - `"Unsupported prompt syntax: --(s|c)ref"` - `"Unsupported prompt syntax: or "` - `"Unsupported url in prompt"` ### Example ```http curl https://api.cloud.scenario.com/v1/models/$MODEL_ID/scores/prompt \ -u "$SCENARIO_SDK_API_KEY:SCENARIO_SDK_API_SECRET" ``` #### Response ```json { "details": { "captionsSimilarity": { "mean": 0, "median": 0, "min": 0 }, "embeddingSimilarity": 0, "promptsSimilarity": { "mean": 0, "median": 0, "min": 0 } }, "modelId": "modelId", "score": 0, "status": "complete", "syntaxError": "Prompt weight should be between 0 and 2" } ``` ## Retrieve Training Dataset **get** `/models/{modelId}/scores/training-dataset` Get the training dataset scores for the given `modelId` ### Path Parameters - `modelId: string` ### Query Parameters - `projectId: optional string` The projectId used for ownership resource management. Either to assert ownership or to set the owner of the resource(s) ### Returns - `details: object { assetCouplesCaptionsDistance, assetCouplesSimilarity, assetsEmbeddingSimilarity, 4 more }` Different components used to compute the final score - `assetCouplesCaptionsDistance: array of object { couple, distance }` For each couple of assets in the training dataset, the distance between the captions - `couple: array of string` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `distance: number` The Dice-Sørensen distance between the captions of the assetIds of the couple. - `assetCouplesSimilarity: array of object { couple, similarity }` For each couple of assets in the training dataset, the similarity between the assets - `couple: array of string` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `similarity: number` The cosine similarity between the assetIds of the couple. The value is a number between -1 and 1. The higher the value, the more similar the assets are to each other - `assetsEmbeddingSimilarity: array of object { assetId, similarity }` For each asset, the similarity with the model's embedding (excluding the current asset) - `assetId: string` The AssetId associated with their similarity (example: "asset_GTrL3mq4SXWyMxkOHRxlpw") - `similarity: number` The cosine similarity between the assetId and the model's embedding minus the asset's embedding The value is a number between -1 and 1. The higher the value, the more similar the asset is to the model's embedding - `assetsSimilarity: object { mean, median, min }` The min, mean, median of the assetsEmbeddingSimilarity - `mean: number` - `median: number` - `min: number` - `couplesCaptionsDistance: object { mean, median, min }` The min, mean, median of the assetCouplesCaptionsDistance - `mean: number` - `median: number` - `min: number` - `couplesSimilarity: object { mean, median, min }` The min, mean, median of the assetCouplesSimilarity - `mean: number` - `median: number` - `min: number` - `metadataScore: number` Takes into account all missing data and warnings to compute a metadata score - `modelId: string` The model ID (example: "model_eyVcnFJcR92BxBkz7N6g5w") - `score: number` The training dataset's score for the input modelId The score is a number between 0 and 1. The higher the score, the most chance the model has to train well - `status: "complete" or "incomplete" or "unknown"` The status of the training dataset's score Details if all the necessary data is present to compute the score - `"complete"` - `"incomplete"` - `"unknown"` - `assetCouplesWarnings: optional array of object { couple, message }` Present only if their is a warning between the assets of the couples - `couple: array of string` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `message: array of "Captions are identical" or "Captions are too close" or "Empty caption" or 6 more` An indicative message concerning the couple of assets - `"Captions are identical"` - `"Captions are too close"` - `"Empty caption"` - `"Missing caption"` - `"Missing metadata"` - `"Same asset"` - `"Similarity is too high"` - `"Similarity with other assets is too low"` - `"Width and height must be larger than 1024"` - `assetsWarnings: optional array of object { assetId, message }` Present only if the training image assets of the modelId have one or more warnings - `assetId: string` The AssetId which has a warning (example: "asset_GTrL3mq4SXWyMxkOHRxlpw") - `message: array of "Captions are identical" or "Captions are too close" or "Empty caption" or 6 more` An indicative message concerning the asset - `"Captions are identical"` - `"Captions are too close"` - `"Empty caption"` - `"Missing caption"` - `"Missing metadata"` - `"Same asset"` - `"Similarity is too high"` - `"Similarity with other assets is too low"` - `"Width and height must be larger than 1024"` ### Example ```http curl https://api.cloud.scenario.com/v1/models/$MODEL_ID/scores/training-dataset \ -u "$SCENARIO_SDK_API_KEY:SCENARIO_SDK_API_SECRET" ``` #### Response ```json { "details": { "assetCouplesCaptionsDistance": [ { "couple": [ "string", "string" ], "distance": 0 } ], "assetCouplesSimilarity": [ { "couple": [ "string", "string" ], "similarity": -1 } ], "assetsEmbeddingSimilarity": [ { "assetId": "assetId", "similarity": -1 } ], "assetsSimilarity": { "mean": -1, "median": -1, "min": -1 }, "couplesCaptionsDistance": { "mean": 0, "median": 0, "min": 0 }, "couplesSimilarity": { "mean": -1, "median": -1, "min": -1 }, "metadataScore": -1 }, "modelId": "modelId", "score": 0, "status": "complete", "assetCouplesWarnings": [ { "couple": [ "string", "string" ], "message": [ "Captions are identical" ] } ], "assetsWarnings": [ { "assetId": "assetId", "message": [ "Captions are identical" ] } ] } ``` ## Domain Types ### Score Retrieve Prompt Response - `ScoreRetrievePromptResponse object { details, modelId, score, 2 more }` - `details: object { captionsSimilarity, embeddingSimilarity, promptsSimilarity }` Different components used to compute the final score - `captionsSimilarity: object { mean, median, min }` The Dice-Sørensen similarity between the prompt and each captions from the model's training images The values are between 0 and 1. The higher the value, the more similar the prompt is to the training images' captions - `mean: number` - `median: number` - `min: number` - `embeddingSimilarity: number` The cosine similarity between the prompt and the model's caption embedding The value is a number between -1 and 1. The higher the value, the more similar the prompt is to the model's caption embedding - `promptsSimilarity: optional object { mean, median, min }` The Dice-Sørensen similarity between the prompt and each prompts from the model's examples The values are between 0 and 1. The higher the value, the more similar the prompt is to the prompts' examples - `mean: number` - `median: number` - `min: number` - `modelId: string` The model ID (example: "model_eyVcnFJcR92BxBkz7N6g5w") - `score: number` The prompt's score for the input modelId The score is a number between 0 and 1. The higher the score, the better the prompt is for the model - `status: "complete" or "incomplete" or "unknown"` The status of the prompt's score Details if all the necessary data is present to compute the score - `"complete"` - `"incomplete"` - `"unknown"` - `syntaxError: optional "Prompt weight should be between 0 and 2" or "Prompt weight symbol should be - or + with a maximum length of 5. Ex: --, ++, ---, +++++" or "Unsupported prompt syntax: '[to:when], [from::when]'" or 4 more` Present only if the prompt has one or more syntax errors The API checks several syntax errors in the prompt - `"Prompt weight should be between 0 and 2"` - `"Prompt weight symbol should be - or + with a maximum length of 5. Ex: --, ++, ---, +++++"` - `"Unsupported prompt syntax: '[to:when], [from::when]'"` - `"Unsupported prompt syntax: 'word:' or '(word):'. Please use 'word' or '(word)' instead"` - `"Unsupported prompt syntax: --(s|c)ref"` - `"Unsupported prompt syntax: or "` - `"Unsupported url in prompt"` ### Score Retrieve Training Dataset Response - `ScoreRetrieveTrainingDatasetResponse object { details, modelId, score, 3 more }` - `details: object { assetCouplesCaptionsDistance, assetCouplesSimilarity, assetsEmbeddingSimilarity, 4 more }` Different components used to compute the final score - `assetCouplesCaptionsDistance: array of object { couple, distance }` For each couple of assets in the training dataset, the distance between the captions - `couple: array of string` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `distance: number` The Dice-Sørensen distance between the captions of the assetIds of the couple. - `assetCouplesSimilarity: array of object { couple, similarity }` For each couple of assets in the training dataset, the similarity between the assets - `couple: array of string` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `similarity: number` The cosine similarity between the assetIds of the couple. The value is a number between -1 and 1. The higher the value, the more similar the assets are to each other - `assetsEmbeddingSimilarity: array of object { assetId, similarity }` For each asset, the similarity with the model's embedding (excluding the current asset) - `assetId: string` The AssetId associated with their similarity (example: "asset_GTrL3mq4SXWyMxkOHRxlpw") - `similarity: number` The cosine similarity between the assetId and the model's embedding minus the asset's embedding The value is a number between -1 and 1. The higher the value, the more similar the asset is to the model's embedding - `assetsSimilarity: object { mean, median, min }` The min, mean, median of the assetsEmbeddingSimilarity - `mean: number` - `median: number` - `min: number` - `couplesCaptionsDistance: object { mean, median, min }` The min, mean, median of the assetCouplesCaptionsDistance - `mean: number` - `median: number` - `min: number` - `couplesSimilarity: object { mean, median, min }` The min, mean, median of the assetCouplesSimilarity - `mean: number` - `median: number` - `min: number` - `metadataScore: number` Takes into account all missing data and warnings to compute a metadata score - `modelId: string` The model ID (example: "model_eyVcnFJcR92BxBkz7N6g5w") - `score: number` The training dataset's score for the input modelId The score is a number between 0 and 1. The higher the score, the most chance the model has to train well - `status: "complete" or "incomplete" or "unknown"` The status of the training dataset's score Details if all the necessary data is present to compute the score - `"complete"` - `"incomplete"` - `"unknown"` - `assetCouplesWarnings: optional array of object { couple, message }` Present only if their is a warning between the assets of the couples - `couple: array of string` The couple of AssetIds (example: ['asset_GTrL3mq4SXWyMxkOHRxlpw', 'asset_6VrLs8nC1DHWCfKGaVne0gBn']) - `message: array of "Captions are identical" or "Captions are too close" or "Empty caption" or 6 more` An indicative message concerning the couple of assets - `"Captions are identical"` - `"Captions are too close"` - `"Empty caption"` - `"Missing caption"` - `"Missing metadata"` - `"Same asset"` - `"Similarity is too high"` - `"Similarity with other assets is too low"` - `"Width and height must be larger than 1024"` - `assetsWarnings: optional array of object { assetId, message }` Present only if the training image assets of the modelId have one or more warnings - `assetId: string` The AssetId which has a warning (example: "asset_GTrL3mq4SXWyMxkOHRxlpw") - `message: array of "Captions are identical" or "Captions are too close" or "Empty caption" or 6 more` An indicative message concerning the asset - `"Captions are identical"` - `"Captions are too close"` - `"Empty caption"` - `"Missing caption"` - `"Missing metadata"` - `"Same asset"` - `"Similarity is too high"` - `"Similarity with other assets is too low"` - `"Width and height must be larger than 1024"`