NVIDIA Nemotron 3.5 Lightning in Ollama: Using Local AI Agents

Architecture and Local Execution

Nemotron 3.5 Lightning uses a hybrid Mixture of Experts (MoE) architecture. Although the model contains a total of 30 billion parameters, only 3 billion are active per token. This design significantly reduces the computational overhead for inference, making it suitable for on-device systems and eliminating the need for data center-scale resources.

The model is compatible with various hardware, including NVIDIA GeForce RTX cards, RTX Pro workstations, and DGX Spark or Station GPUs. For users with Apple Silicon, a special MLX version is available via Ollama to optimize performance on these specific chips. Since the model runs locally, all processed data remains on the user’s device.

Technical Specifications and Performance

The model has a context window of 1 million tokens, enabling it to process large amounts of information in a single session. To increase efficiency, NVIDIA has implemented speculative decoding using Multi-Token Prediction (MTP) in addition to DFlash and DSpark technologies. These optimizations can result in up to four times higher throughput compared to similar open-source models.

As an open model trained on open datasets, Nemotron 3.5 Lightning can be further customized. Developers can fine-tune the model for specific tasks and deploy the resulting version in various environments, from edge devices to cloud infrastructure.

Applications for Agent-Based Workloads

The architecture is optimized for tasks that involve reading files, invoking external tools, sorting results, and executing retry logic in case of errors. Key use cases include:

• Personal assistants: Managing emails, calendars, and appointments using local contextual data without transferring data externally.
• Sub-agents for programming: Integration into existing development environments to search codebases, run tests, and perform refactoring.
• Security operations: Classifying incidents, querying logs, and correlating indicators to generate structured reports for analysts.
• Hybrid Deployment: Functions as a local layer that handles simple, high-volume tasks before forwarding complex queries to a larger hosted model via the same API.

Deploying Ollama Nemotron 3.5 Lightning

Users can deploy Nemotron 3.5 Lightning via Ollama using standard CLI commands. The model is compatible with several agent frameworks, including Claude Code, OpenClaw, Hermes Agent, and OpenCode. This integration allows agents to execute individual steps locally or forward specific queries to larger models in the cloud without modifying the primary application logic.

General chat

ollama run nemotron-3.5-lightning

Claude Code

ollama launch claude --model nemotron-3.5-lightning

OpenClaw

ollama launch openclaw --model nemotron-3.5-lightning

Hermes Agent

ollama launch hermes --model nemotron-3.5-lightning

OpenCode

ollama launch opencode --model nemotron-3.5-lightning

[end of page]