File Upload Script Documentation¶

This standalone script uploads a folder of files to the server using SAS/Signed URLs. It supports Azure and Google Cloud Storage, concurrent uploads, checksum verification, recursive folder processing, and a file-level progress bar.

You do not need to clone the language-server repository.

Prerequisites¶

Python 3.10+ (standard library only — no pip install required)
GitHub access to the karya-inc organization (you must be able to open the private repo in the browser)

Download the script (no personal access token)¶

Because the repository is private, unauthenticated raw.githubusercontent.com links return 404. If you already have org GitHub access, use one of these — neither requires creating a token:

Option A — Browser (simplest)

Open dataset_upload_standalone.py on dev while logged into GitHub.
Click Raw, then save the page as dataset_upload.py (or use your browser’s Save As).

Option B — GitHub CLI (if you already use gh)

If you have already run gh auth login once for the org, this reuses that login — no new token:

gh api repos/karya-inc/language-server/contents/scripts/dataset_upload_standalone.py?ref=dev \
  -H "Accept: application/vnd.github.raw" > dataset_upload.py

Maintainer script

An asyncio-based variant remains in the repo at scripts/upload_script.py for maintainers who already have project dependencies installed. External teams should use dataset_upload_standalone.py.

Server Upload API¶

The upload script uses two internal API endpoints. If you need to integrate uploads programmatically (e.g., in your own application), you can call these endpoints directly.

Step 1: Request Upload URLs¶

Endpoint: POST /v1/task/upload Content-Type: multipart/form-data

Form Fields:

Field	Description
`payload_data`	JSON string with upload metadata (see below)
`file`	A JSON file named `upload_payload.json` containing the file list

payload_data structure:

{
  "provider": "Google",
  "user_email": "user@example.com",
  "project_name": "Individual"
}

upload_payload.json (the file form field) structure:

{
  "files": [
    {
      "name": "audio_file.wav",
      "content": "audio/wav",
      "checksum": "<md5hex>"
    }
  ],
  "metadata": {
    "title": "My Dataset",
    "description": "Description here",
    "number_of_files": 1
  },
  "expiry": "7"
}

Response:

{
  "task_id": "upload_task_id_123",
  "files": [
    {
      "filename": "audio_file.wav",
      "sas_url": "https://storage.googleapis.com/bucket/audio_file.wav?X-Goog-Algorithm=...&X-Goog-Signature=..."
    }
  ]
}

Upload each file directly to its sas_url using a PUT request.

Step 2: Finalize Upload¶

After all files have been uploaded to their SAS/Signed URLs, finalize the upload to create the dataset:

Endpoint: PUT /v1/upload/completed?task_id=<task_id>

Response:

{
  "dataset_id": "abc123-def456-ghi789"
}

Use the returned dataset_id in subsequent task requests.

How it Works¶

The upload process follows these steps:

File Scanning: The script scans your specified folder for files
Checksum Calculation: MD5 checksums are calculated for each file
MIME Type Detection: File types are automatically detected
URL Generation: The server generates SAS/Signed URLs for each file
Concurrent Upload: Files are uploaded to cloud storage (max 5 concurrent uploads) with a progress bar
Completion: The upload process is finalized and a dataset ID is returned

Key Features¶

No repo clone / no pip: Single downloadable script using the Python standard library
Concurrent Uploads: Up to 5 files uploaded simultaneously for faster processing
Progress Bar: File-level progress while uploading to cloud storage
Automatic Retries: Transient upload failures get up to 3 total attempts (1 initial + 2 retries) with exponential backoff
Checksum Verification: MD5 checksums help detect accidental transfer corruption (not tamper protection or authenticity)
Recursive Processing: Option to process files in nested folders
Multiple Providers: Support for both Azure and Google Cloud Storage
Error Handling: Comprehensive error reporting
Progress Logging: Detailed logging with optional verbose mode
File Count Limit: For Sarvam ASR, upload no more than 20 audio files per dataset (service limit, not enforced by the script).

Command Line Arguments¶

Required Arguments¶

Argument	Description
`folder_path`	Path to the folder containing files to upload
`--api-key`	API key for server authentication
`--user-email`	User email address
`--server-url`	Base server URL (e.g., https://dev-server.com)

Optional Arguments¶

Argument	Default	Description
`--provider`	`Google`	Storage provider (`Google` or `Azure`)
`--project-name`	`Individual`	Project name for the upload
`--title`	Auto-generated	Custom title for the upload
`--description`	Auto-generated	Custom description for the upload
`--recursive`	`False`	Process files recursively from nested folders. Filenames in the upload payload are relative POSIX paths (e.g. `sub/a.wav`), so duplicate basenames in different folders stay distinct.
`--allow-partial`	`False`	Complete the dataset even if some files failed to upload. By default any failed file aborts before completion.
`--verbose`, `-v`	`False`	Enable verbose logging (debug level)

Sarvam ASR File Limit

When uploading files for Sarvam ASR transcription, upload no more than 20 audio files per dataset. This limit is enforced by the Sarvam ASR service. The upload script itself does not enforce this limit — you are responsible for keeping batches within this size.

Usage Examples¶

Download once, then upload¶

Download dataset_upload.py via the browser or gh (see Prerequisites), then:

python3 dataset_upload.py /path/to/files \
  --api-key YOUR_KEY \
  --user-email user@example.com \
  --server-url https://dev-server.com

Recursive Upload (Include nested folders)¶

python3 dataset_upload.py /path/to/files \
  --api-key YOUR_KEY \
  --user-email user@example.com \
  --server-url https://dev-server.com \
  --recursive

Azure Blob Storage Upload¶

python3 dataset_upload.py /path/to/files \
  --api-key YOUR_KEY \
  --user-email user@example.com \
  --server-url https://dev-server.com \
  --provider Azure

Complete Example with All Options¶

python3 dataset_upload.py ./my-dataset/ \
  --api-key "your-api-key-here" \
  --user-email "user@example.com" \
  --server-url "https://api.yourcompany.com" \
  --provider Azure \
  --project-name "ML-Project" \
  --title "Audio Dataset v2" \
  --description "Updated audio files for ML training" \
  --recursive \
  --allow-partial \
  --verbose

Output and Results¶

Success Output¶

When the upload completes successfully, you'll see:

🎉 SUCCESS! Dataset ID: abc123-def456-ghi789

Progress Bar¶

During cloud uploads:

Uploading 12/25 [██████████████░░░░░░░░░░░░░░] 48%  current: clip_03.wav

Progress Logging¶

The script provides detailed logging:

2024-01-15 10:30:00 - INFO - === File Upload Configuration ===
2024-01-15 10:30:00 - INFO - Folder Path: ./my-dataset/
2024-01-15 10:30:00 - INFO - Server URL: https://api.yourcompany.com
2024-01-15 10:30:00 - INFO - Provider: Google
2024-01-15 10:30:00 - INFO - User Email: user@example.com
2024-01-15 10:30:00 - INFO - Project Name: ML-Project
2024-01-15 10:30:00 - INFO - Recursive: Yes
2024-01-15 10:30:00 - INFO - ===================================
2024-01-15 10:30:01 - INFO - Step 1: Scanning files...
2024-01-15 10:30:01 - INFO - Found 25 files
2024-01-15 10:30:02 - INFO - Step 2: Creating JSON payload...
2024-01-15 10:30:02 - INFO - Step 3: Getting upload URLs...
2024-01-15 10:30:03 - INFO - Received task ID: task_12345
2024-01-15 10:30:03 - INFO - Step 4: Uploading files...
2024-01-15 10:30:05 - INFO - ✓ Successfully uploaded file1.wav (1024000 bytes)
2024-01-15 10:30:06 - INFO - ✓ Successfully uploaded file2.wav (2048000 bytes)
2024-01-15 10:30:10 - INFO - Upload Results: 25/25 files uploaded successfully
2024-01-15 10:30:10 - INFO - Step 5: Completing upload...
2024-01-15 10:30:11 - INFO - ✓ Upload process completed! Dataset ID: abc123-def456-ghi789

Error Handling¶

Common Error Scenarios¶

Dataset Name Conflict (409 Error)

❌ ERROR: Dataset name already exists. Please use a different name or delete the existing dataset first.

Authentication Failure

❌ ERROR: Server error 401: Unauthorized

File Not Found

❌ ERROR: Folder path does not exist: /invalid/path

Upload Failures

❌ FAILED: 3 files failed to upload; re-run with --allow-partial to complete the dataset with successful files only

With --allow-partial, the same situation logs a warning and still completes the dataset:

⚠️ WARNING: 3 files failed to upload

Transient failures (network errors, HTTP 408/429, and HTTP 5xx responses) make up to 3 total attempts (1 initial + 2 retries), waiting 1 second and then 2 seconds between attempts. Other HTTP 4xx errors are not retried because they usually require a configuration or authentication change.

Troubleshooting¶

Check API Key: Ensure your API key is valid and has upload permissions
Verify Server URL: Make sure the server URL is correct and accessible
File Permissions: Ensure the script has read access to your files
Network Connectivity: Check your internet connection for large file uploads
Use Verbose Mode: Add --verbose flag for detailed debugging information

Best Practices¶

File Organization: Keep related files in the same folder for easier management
Naming Conventions: Use descriptive filenames and avoid special characters
File Sizes: Large files may take longer to upload; consider splitting very large files
Recursive Uploads: Use --recursive only when you need files from subdirectories. Nested files are identified by relative paths (e.g. folder/clip.wav), not basename alone.
Verbose Logging: Use --verbose for troubleshooting upload issues
Backup: Always keep backups of your original files before uploading

Next Steps¶

After successful upload, you can:

Use the returned dataset_id to create batch transcription tasks (see Transcription Guide)
Use the returned dataset_id to create batch completion tasks (see Completion Guide)
Share the dataset with team members
Create additional datasets using the same process

Note

The script automatically generates titles and descriptions if not provided. For better organization, consider providing custom titles and descriptions that clearly identify your datasets.