Multipart Upload API Documentation

Overview

The Multipart Upload API allows you to efficiently upload large files by splitting them into smaller parts and uploading them concurrently. This approach is particularly useful for files that are several hundred megabytes in size. The API provides functionalities to initiate an upload, create pre-signed URLs for individual parts, upload the parts, and finally, complete the upload.

Why Multipart Upload?

Uploading large files as a single unit can be challenging due to network constraints and potential interruptions. Multipart uploads offer several advantages:

Resumable Uploads: In case of failure, only the failed part needs to be re-uploaded, minimizing data transfer.
Parallelism: Multiple parts can be uploaded concurrently, improving upload speed.
Scalability: Multipart uploads facilitate the handling of large files without overwhelming system resources.

Pre-requirements

Let supposed for the following examples we have access to:

const baseApiUri = 'api.scenario.com/v1';
async function getAuthenticationHeader() {
  return `Basic ${btoa('apiKey:apiSecret')}`;
}
const file = inputHtmlElement.files[0];

Calculate the appropriate number of parts regarding your upload:

function calculateOptimalNumberOfParts(fileSize) {
    // You may implement your own logic to determine the optimal number of parts
    // For simplicity, let's say we want approximately 7 MB per part
    const targetPartSize = 7 * 1024 * 1024;
    return Math.ceil(fileSize / targetPartSize);
}

We decided to report the part number calculation on API user side because we want you to be able to optimise the number of part regarding external configurations (such as network, device, ...).

API Endpoints

1. Create Multipart Presigned URLs

Endpoint

POST /uploads

Request Body

{
    "fileName": "test-model.zip",
    "contentType": "application/zip",
    "fileSize": 31092875,
    "parts": 5,
    "kind": "model"
}

Description

Initiates the multipart upload process by providing metadata about the file to be uploaded, including the file name, content type, number of parts, and the kind of upload.

Response Example

{
  "id": "uploadId",
  "parts": [
    {
      "expires": "2023-12-29T15:31:49.432Z",
      "url": "presignedUrlForPut",
      "number": 1
    },
    // ... additional parts
  ],
  // ... additional upload informations 
}

Example (Node.js)

function calculateOptimalNumberOfParts(fileSize) {
    // You may implement your own logic to determine the optimal number of parts
    // For simplicity, let's say we want approximately 7 MB per part
    const targetPartSize = 7 * 1024 * 1024;
    return Math.ceil(fileSize / targetPartSize);
}

async function createMultipartUpload(fileName, contentType, fileSize, numberOfParts) {
    const response = await fetch(`https://${baseApiUri}/uploads`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': await getAuthenticationHeader(),
        },
        body: JSON.stringify({
            fileName,
            contentType,
            fileSize,
            parts: numberOfParts,
            kind: 'model',
        }),
    });

    return await response.json();
}

// from your main script you can now call:
// Create Multipart Presigned URLs
const { upload } = await createMultipartUpload(file.name, file.type, file.size, calculateOptimalNumberOfParts(file.size));

2. Upload Parts

From the previous request you should get access to upload.parts which contains for each parts an url where to send (with PUT method), each part of your upload.

Example (Node.js)

async function uploadPart(partNumber, presignedUrl, filePart) {
    await fetch(presignedUrl, {
        method: 'PUT',
        body: filePart,
    });
    console.log(`Part ${partNumber} uploaded successfully.`);
}

async function uploadParts(uploadId, parts, file) {
    const maxConcurrentPartsUpload = 5;
    const uploadPartPromises = [];

    const partSize = Math.ceil(file.size / parts.length);

    let i = 0;
    for (const part of parts) {
        const { number, url } = part;

        const filePart = file.slice(i * partSize, (i + 1) * partSize);
        uploadPartPromises.push(uploadPart(number, url, filePart));

        if (uploadPartPromises.length >= maxConcurrentPartsUpload) {
            await Promise.all(uploadPartPromises);
            uploadPartPromises.length = 0;
        }

        i++;
    }

    if (uploadPartPromises.length > 0) {
        await Promise.all(uploadPartPromises);
    }
}

// from your main script you can now call:

// Step 2: Upload Parts
await uploadParts(upload.id, upload.parts, file);

3. Complete upload after all parts sent

Endpoint

POST /uploads/{uploadId}/action

Request Body

{
    "action": "complete",
    "config": {
      "modelType": "sd_xl"
    }
}

Description

At this point you need to call a last API to inform Scenario you have finished to upload all parts. On our side, scenario will check the validity of each part and try to recreate the original file. If it success, we will start a validator, regarding the "kind" of upload originally sent when creating the upload to be sure of the content we recieved.

Here you are able to send additionnal configuration for given file. For example with the kind "model", you can send a config node with: modelType = "sd_1-5", "sd_xl", ... to inform Scenario which kind of model it is and optimize the validation step.

Example (Node.js)

async function completeMultipartUpload(uploadId) {
    await fetch(`https://${baseApiUri}/uploads/${uploadId}/action`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': await getAuthenticationHeader(),
        },
        body: JSON.stringify({
            action: 'complete',
        }),
    });
}

// Step 4: Complete Multipart Upload
await completeMultipartUpload(upload.id);

At this point the upload is complete. You can call the API:

GET /uploads/{uploadId}

To see the status of your upload. Just after the call to "complete" the upload the status should be "complete". But it's not the final state of the upload.

Short time after this status you should get for an upload of kind "model":

validating: The worker to validate your upload (anti-virus or corrupted file) is running
validated: The worker as validated your upload, should be available very soon
imported: Your upload as been imported in scenario and is now available for uses

Complete HTML page for test purposes

Description

With an access to scenario, the endpoint for APIs and valides API Key and API Secret, you can check the behavior with this full autonomous HTML page:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Multipart Upload Example</title>
</head>
<body>

    <h1>Multipart Upload Example</h1>

    <form id="uploadForm">
        <label for="fileInput">Select a file:</label>
        <input type="file" id="fileInput" accept=".zip">

        <button type="button" onclick="startMultipartUpload()">Start Upload</button>
    </form>

    <script>
        const baseApiUri = 'api.xxxxxx.scenario.xxx/v1';
        async function getAuthenticationHeader() {
            return `Basic ${btoa('api_XXXXXXXXXXXXX:XXXXXXXXXXXXX')}`;
        }

        async function startMultipartUpload() {
            const fileInput = document.getElementById('fileInput');
            const file = fileInput.files[0];

            if (!file) {
                alert('Please select a file');
                return;
            }

            // Step 1: Determine Optimal Number of Parts (You may implement your own logic here)
            const optimalNumberOfParts = calculateOptimalNumberOfParts(file.size);

            // Step 2: Create Multipart Presigned URLs
            const { upload } = await createMultipartUpload(file.name, file.type, file.size, optimalNumberOfParts);

            // Step 3: Upload Parts
            await uploadParts(upload.id, upload.parts, file);

            // Step 4: Complete Multipart Upload
            await completeMultipartUpload(upload.id);

            alert('Multipart upload completed successfully.');
        }

        function calculateOptimalNumberOfParts(fileSize) {
            // You may implement your own logic to determine the optimal number of parts
            // For simplicity, let's say we want approximately 7 MB per part
            const targetPartSize = 7 * 1024 * 1024;
            return Math.ceil(fileSize / targetPartSize);
        }

        async function createMultipartUpload(fileName, contentType, fileSize, numberOfParts) {
            const response = await fetch(`https://${baseApiUri}/uploads`, {
                method: 'POST',
                headers: {
                    'Content-Type': 'application/json',
                    'Authorization': await getAuthenticationHeader(),
                },
                body: JSON.stringify({
                    fileName,
                    contentType,
                    fileSize,
                    parts: numberOfParts,
                    kind: 'model',
                }),
            });

            return await response.json();
        }

        async function uploadPart(partNumber, presignedUrl, filePart) {
            await fetch(presignedUrl, {
                method: 'PUT',
                body: filePart,
            });
            console.log(`Part ${partNumber} uploaded successfully.`);
        }

        async function uploadParts(uploadId, parts, file) {
            const maxConcurrentPartsUpload = 5;
            const uploadPartPromises = [];

            const partSize = Math.ceil(file.size / parts.length);

            let i = 0;
            for (const part of parts) {
                const { number, url } = part;

                const filePart = file.slice(i * partSize, (i + 1) * partSize);
                uploadPartPromises.push(uploadPart(number, url, filePart));

                if (uploadPartPromises.length >= maxConcurrentPartsUpload) {
                    await Promise.all(uploadPartPromises);
                    uploadPartPromises.length = 0;
                }

                i++;
            }

            if (uploadPartPromises.length > 0) {
                await Promise.all(uploadPartPromises);
            }
        }

        async function completeMultipartUpload(uploadId) {
            await fetch(`https://${baseApiUri}/uploads/${uploadId}/action`, {
                method: 'POST',
                headers: {
                    'Content-Type': 'application/json',
                    'Authorization': await getAuthenticationHeader(),
                },
                body: JSON.stringify({
                    action: 'complete',
                }),
            });
        }
    </script>

</body>
</html>