Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Mitigating cloud LLM risks, including data retention, input usage for training, and foreign jurisdiction issues.
- Understanding Ollama’s architecture, comprising the model server, registry, and OpenAI-compatible API.
- Evaluating Ollama against alternatives like vLLM, llama.cpp, and Text Generation Inference.
- Navigating model licensing agreements for Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Setting up Ollama on Linux with support for both CUDA and ROCm.
- Implementing CPU-only fallback strategies and optimizing for AVX/AVX2.
- Managing Docker deployments and persistent volume mappings.
- Configuring multi-GPU setups and defining VRAM allocation strategies.
Model Administration
- Retrieving models from the Ollama registry using commands like ollama pull llama3.
- Importing GGUF models from HuggingFace and TheBloke.
- Assessing quantization levels (Q4_K_M, Q5_K_M, Q8_0) and their respective tradeoffs.
- Managing model switching and understanding concurrent loading limitations.
Creating Custom Modelfiles
- Mastering Modelfile syntax, including FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Adjusting generation parameters such as temperature, top_p, and repeat_penalty.
- Crafting system prompts to define role-specific behaviors.
- Developing and publishing custom models to a local registry.
API Integration Strategies
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enforcing JSON output modes.
- Integrating with frameworks like LangChain, LlamaIndex, and custom applications.
- Implementing authentication and rate limiting via reverse proxies.
Performance Tuning
- Optimizing context window sizes and managing KV cache resources.
- Executing batch inference and handling parallel requests efficiently.
- Allocating CPU threads with consideration for NUMA architecture.
- Monitoring GPU utilization and memory pressure to prevent bottlenecks.
Security and Compliance Protocols
- Establishing network isolation for model serving endpoints.
- Implementing input filtering and output moderation pipelines.
- Maintaining comprehensive audit logs of prompts and completions.
- Verifying model provenance and integrity through hash validation.
Requirements
- Proficiency in intermediate Linux system administration and containerization.
- A solid conceptual understanding of machine learning principles and transformer architectures.
- Working knowledge of RESTful APIs and JSON data formats.
Target Audience
- AI engineers and developers looking to migrate away from cloud-based LLM APIs.
- Organizations with strict data sensitivity requirements that preclude the use of cloud models.
- Government and defense units that mandate the use of air-gapped language models.
