NVIDIA NIM: Llama 3.1 70B Instruct

stable

Llama 3.1 70B Instruct is available through Everstack's OpenAI-compatible gateway with routing, fallback, retries, and observability.

meta__llama-3.1-70b-instructllamareleased Jul 16, 2024

Provider Quick start

Context

128K

tokens

Max output

4.1K

tokens

Input

per 1M

Output

per 1M

Activity

Daily period usage

Aggregated managed Everstack traffic. Buckets below 5 tenants or 100 requests are withheld.

Providers

ProviderContextInput /MOutput /M

NVIDIA NIM

128K--

Pricing

Item	Price	Unit
Input	-	per 1M tokens
Output	-	per 1M tokens

Performance

Reliability

Compare

Compare usage and performance

Place this model beside up to three other catalog models.

Capabilities

Modalities

Input

Output

Details

Model ID: meta__llama-3.1-70b-instruct
Family: llama
Status: stable
Knowledge: -
Released: Jul 16, 2024

Gateway Config

API: 2024-01-01
Auth: NVIDIA_API_KEY
Discovery: static
Provider type: meta

Quick Start

index.ts

import Everstack from "@everstack/node";const client = new Everstack({  apiKey: process.env.EVERSTACK_API_KEY!,  baseUrl: "https://{instance}.{region}.everstack.ai",});const completion = await client.chat.completions.create({  model: "@nvidia-nim/meta__llama-3.1-70b-instruct",  messages: [{ role: "user", content: "Explain vector cache routing." }],});console.log(completion.choices[0]?.message?.content);

Endpoint

Base URL

https://integrate.api.nvidia.com/v1

Chat completions

https://integrate.api.nvidia.com/v1/chat/completions

Model

meta__llama-3.1-70b-instruct

Auth env

NVIDIA_API_KEY

Llama 3.1 70B Instruct FAQ

What is the context window of Llama 3.1 70B Instruct?

Llama 3.1 70B Instruct accepts up to 128K tokens of context and returns up to 4.1K output tokens on NVIDIA NIM.

What can Llama 3.1 70B Instruct do?

Llama 3.1 70B Instruct supports chat and function calling. It accepts text input.

How do I call Llama 3.1 70B Instruct through Everstack?

Point any OpenAI-compatible SDK at your Everstack gateway and set the model to @nvidia-nim/meta__llama-3.1-70b-instruct. Everstack adds retries, provider fallback, caching, per-tenant budgets and OpenTelemetry traces for every request.