Composing Models with LoRAs

Composing models with LoRAs (Low-Rank Adaptations) allows you to blend the capabilities of multiple specialized models into a single, powerful new model. This technique is particularly useful for combining different styles, objects, or concepts learned by individual LoRAs to achieve unique and nuanced image generation results. This guide will walk you through the process of composing your own models using LoRAs, detailing the necessary steps, API endpoints, and parameters. We will also provide code examples to help you integrate model composition into your applications.

🚀 Key Concepts

To effectively compose models, it's helpful to understand these concepts:

• LoRA (Low-Rank Adaptation): As discussed in the Training Models article, LoRA is an efficient fine-tuning method that allows for the adaptation of pre-trained models to new tasks or styles with minimal computational overhead. Each LoRA typically specializes in a particular aspect, such as a specific artistic style, character, or object.

• Composed Model: A new model created by combining the strengths of several individual LoRAs. This allows for highly customized and versatile image generation.

• Concepts: In the context of model composition, concepts refer to the individual LoRAs that you want to combine. Each concept includes the modelId of the LoRA and a scale parameter, which determines the weight or influence of that LoRA in the final composed model.

⚡️ Composition Workflow

The process of composing a model using LoRAs generally follows these steps:

1. Get LoRA Models: Identify and retrieve the modelIds of the LoRA models you wish to combine.
2. Create a Composed Model: Send a request to the API to create a new model, specifying the selected LoRAs and their respective scales as concepts.
3. Train the Composed Model: Initiate the training process for your newly composed model.

Let's delve into each step.

1. Get LoRA Models

Before you can compose a model, you need to know which LoRA models are available and their corresponding modelIds. You can retrieve a list of LoRA models by making a GET request to the /v1/models endpoint and filtering by the type parameter set to flux.1-lora or sd-xl-lora depending on your needs .

Endpoint:

GET https://api.cloud.scenario.com/v1/models

Query Parameters:

Example Request (Python):

import requests

api_key = "YOUR_API_KEY"
api_secret = "YOUR_API_SECRET"

url = "https://api.cloud.scenario.com/v1/models"
headers = {"Authorization": f"Basic {requests.utils.b64encode(f'{api_key}:{api_secret}'.encode()).decode()}"}

params = {
    "type": "flux.1-lora"
}

response = requests.get(url, headers=headers, params=params)

if response.status_code == 200:
    lora_models = response.json()
    print("Available LoRA Models:")
    for model in lora_models:
        print(f"  Name: {model.get('name')}, ID: {model.get('id')}")
else:
    print(f"Error retrieving LoRA models: {response.status_code} - {response.text}")

From the returned list, identify the modelIds of the LoRAs you wish to combine.

2. Create a Composed Model

With the modelIds of your desired LoRAs in hand, you can now create a new composed model. This is achieved by making a POST request to the /v1/models endpoint, similar to creating a new model for training, but with a crucial difference: you will include a concepts array in the request body and set the type to flux.1-composition or sd-xl-composition.

Each object within the concepts array should contain:

• modelId: The ID of the LoRA model you want to include.
• scale: A float value between 0 and 1, representing the weight or influence of this LoRA in the final composition. A higher value means more influence.

Endpoint:

POST https://api.cloud.scenario.com/v1/models

Request Body Parameters:

Example Request (Python):

import requests

api_key = "YOUR_API_KEY"
api_secret = "YOUR_API_SECRET"

url = "https://api.cloud.scenario.com/v1/models"
headers = {"Authorization": f"Basic {requests.utils.b64encode(f'{api_key}:{api_secret}'.encode()).decode()}"}

payload = {
    "name": "My Blended LoRA Model",
    "type": "sd-xl-composition",
    "concepts": [
        {"modelId": "V2anp75qTuKmmHOeInzQhg", "scale": 0.65}, # Replace with your LoRA model IDs and desired scales
        {"modelId": "I3fUkYcTSY-PEkhvYMDvSQ", "scale": 0.8}
    ]
}

response = requests.post(url, headers=headers, json=payload)

if response.status_code == 200:
    composed_model_data = response.json()
    composed_model_id = composed_model_data["id"]
    print(f"Composed model created successfully! Model ID: {composed_model_id}")
else:
    print(f"Error creating composed model: {response.status_code} - {response.text}")

Upon successful creation, the response will include the modelId of your new composed model.

3. Train the Composed Model

After creating your composed model, you need to initiate its training. This step is similar to training a regular LoRA model, using the PUT /v1/models/{modelId}/train endpoint. This process finalizes the blending of the specified LoRAs into a cohesive model.

Endpoint:

PUT https://api.cloud.scenario.com/v1/models/{modelId}/train

Path Parameters:

Request Body Parameters:

Example Request (Python):

import requests

# Assuming composed_model_id is obtained from the previous step
# composed_model_id = "your_composed_model_id"

api_key = "YOUR_API_KEY"
api_secret = "YOUR_API_SECRET"

url = f"https://api.cloud.scenario.com/v1/models/{composed_model_id}/train"
headers = {"Authorization": f"Basic {requests.utils.b64encode(f'{api_key}:{api_secret}'.encode()).decode()}"}

payload = {
    "parameters": {
        # You can add specific training parameters here if needed
    }
}

response = requests.put(url, headers=headers, json=payload)

if response.status_code == 200:
    job_data = response.json()
    job_id = job_data["job"]["jobId"]
    print(f"Composed model training initiated successfully! Job ID: {job_id}")
else:
    print(f"Error initiating composed model training: {response.status_code} - {response.text}")

Similar to regular model training, this is an asynchronous process. You will need to monitor the training status using the GET /v1/models/{modelId} endpoint (as it is an SDXL-based composition) until the model status is trained.

📚 References

• Scenario API Reference: POST /v1/models
• Scenario API Reference: PUT /v1/models/{modelId}/train