Managed inference

Run fast text inference without managing a container

Submit an asynchronous text-generation job using a stable Jungle Grid model alias, let the platform select the serving backend, and retrieve the completed response as a result artifact.

At a glance

•Use a canonical alias such as jg/text/general-instruct or jg/text/fast-instruct instead of a vendor-specific model ID.
•Set execution_mode to managed_model, provider to auto, and serving_mode to serverless.
•No image, command, GPU, environment, uploaded-file, or container fields are accepted on this route.
•The initial route is asynchronous, non-streaming text chat. Completed output is stored as result.json.

Submit a managed inference job

POST /v1/jobs accepts the managed-model contract alongside the existing container workload contract. The model field is a Jungle Grid alias; backend selection and provider credentials remain platform-managed.

•Use jg/text/general-instruct for general instruction following and jg/text/fast-instruct when latency is the stronger priority.
•Supported message roles are user, assistant, and system.
•Supported parameters are temperature, top_p, max_output_tokens, stop, and seed.
•The selected alias must be enabled by Jungle Grid. Unsupported or unavailable aliases return MODEL_NOT_SUPPORTED.

Request

POST /v1/jobs

{
  "name": "fast-text-response",
  "workload_type": "inference",
  "model": "jg/text/fast-instruct",
  "input": {
    "messages": [
      {
        "role": "user",
        "content": "Summarize why low-latency inference matters."
      }
    ]
  },
  "parameters": {
    "temperature": 0.2,
    "max_output_tokens": 256
  },
  "routing": {
    "execution_mode": "managed_model",
    "provider": "auto",
    "serving_mode": "serverless"
  }
}

Accepted response

{
  "job_id": "job_123",
  "status": "queued",
  "execution_route": "managed_model",
  "route_reason": "explicit managed-model semantic execution"
}

Output limits and non-streaming behavior

max_output_tokens is the maximum number of tokens the model may generate for that request. If generation reaches the limit, result.json reports finish_reason as length. Omitting the field uses the platform default, bounded by the selected alias.

•A small value reduces latency and usage but can truncate an answer.
•The configured alias also has context and maximum-output limits; the API rejects requests that exceed them.
•Non-streaming means generated tokens are not sent incrementally. Poll the job until terminal state, then retrieve result.json.
•Job lifecycle status is still asynchronous: queued, running, completed, failed, or cancelled.

Retrieve the generated result

When the job completes, GET /v1/jobs/{job_id}/artifacts lists a ready result.json artifact. POST its download endpoint to receive the inline JSON response.

List and retrieve

GET /v1/jobs/{job_id}/artifacts

POST /v1/jobs/{job_id}/artifacts/{artifact_id}/download

Inline result

{
  "content": {
    "type": "text_completion",
    "model": "jg/text/fast-instruct",
    "usage": {
      "input_tokens": 24,
      "output_tokens": 80,
      "total_tokens": 104
    },
    "output": {
      "role": "assistant",
      "content": "..."
    },
    "finish_reason": "stop"
  }
}

Managed inference produces an inline result artifact rather than a container filesystem artifact. Treat model output as untrusted application data and validate it before using it in tools or downstream automation.

Troubleshooting

Managed inference uses structured API errors. A rejected request does not call the serving backend. A job accepted with HTTP 202 should be monitored through the normal job status and artifact endpoints.

•MANAGED_MODEL_INFERENCE_DISABLED: the managed route is temporarily unavailable.
•MODEL_NOT_SUPPORTED: the requested Jungle Grid alias is not enabled.
•INVALID_REQUEST: required semantic fields are missing, routing is not provider-neutral, or container-specific fields were mixed into the request.
•REQUEST_TOO_LARGE: message count, message bytes, aggregate input, stop sequences, or output-token limits were exceeded.
•Provider availability is confirmed during execution, so a valid accepted job can still fail with a redacted runtime reason.