Rate limits

Concurrent requests

Every account has a limit on how many requests run at the same time. There is no default limit: an account without a limit may run as many requests at once as it needs. sivcloud can set a limit for one account. If you need a limit changed, contact support: through the Support page in the console. A request over the limit is answered with rate_limit_exceeded (HTTP 429).

A request counts until it ends, fails or is cut off. A stuck request does not lock the account forever, because its slot is released automatically.

Limits on plans

A plan allowance can carry limits of its own:

• requests per minute,
• concurrent requests,
• requests per day.

A request that hits one of these is rejected with rate_limit_exceeded (HTTP 429) and is not moved to your balance, because a rate limit is a property of the plan, not a sign that the allowance is used up. An allowance that runs out is a different case: see Plans.

A request is paid by the plan only if its whole estimate (prompt + max_tokens, or 4,096 tokens when not set; at most 8,192) fits the remaining allowance; otherwise the whole request is paid from your balance. See Plans.

How to respond

• Retry after longer and longer pauses (1, 2, 4 seconds and so on).
• Cap the parallel requests in your application at the account limit.
• A per-key limit in the console only limits balance spending, in rupiah. It is not a rate limit.