Skip to content

02: Server Setup

Kaggle Time Level

Overview

Phase: Foundation Difficulty: Beginner Duration: 15 min

Learn server setup for LLM inference on Kaggle dual Tesla T4 GPUs.

Learning Objectives

  • ✅ Understand server configuration options
  • ✅ Configure for Kaggle T4
  • ✅ Test different configurations
  • ✅ Monitor server health
  • ✅ Benchmark performance

Topics Covered

  • 📚 Server configuration
  • 📚 Performance optimization
  • 📚 Health monitoring

Prerequisites

  • llamatelemetry v0.1.0 installed
  • Kaggle dual Tesla T4 environment (30GB VRAM)
  • Basic Python knowledge
  • Completed Tutorial 01

Quick Start

# Install llamatelemetry
!pip install -q --no-cache-dir git+https://github.com/llamatelemetry/llamatelemetry.git@v0.1.0

# Verify GPU environment
!nvidia-smi --query-gpu=index,name,memory.total --format=csv

Key Concepts

Server Configuration

llamatelemetry uses llama.cpp's llama-server for high-performance GGUF model inference.

from llamatelemetry.server import ServerManager

server = ServerManager()
server.start_server(
    model_path=model_path,
    gpu_layers=99,
    ctx_size=4096,
    tensor_split="0.5,0.5"  # Dual GPU split
)

Step-by-Step Guide

Step 1: Environment Setup

Verify your Kaggle environment has dual Tesla T4 GPUs.

Step 2: Installation

Install llamatelemetry v0.1.0 with all dependencies.

Step 3: Configuration

Configure the server for optimal performance on Kaggle T4.

Step 4: Implementation

Implement the tutorial objectives step by step.

Step 5: Verification

Test and verify the implementation works correctly.

Expected Output

After completing this tutorial, you should be able to:

  • ✅ Understand server configuration options
  • ✅ Configure for Kaggle T4
  • ✅ Test different configurations
  • ✅ Monitor server health
  • ✅ Benchmark performance

Common Issues

Issue: Server Fails to Start

Solution: Check GPU memory and ensure no other processes are using the GPUs.

nvidia-smi

Issue: Out of Memory

Solution: Reduce context size or use lower quantization.

ctx_size=2048  # Instead of 4096

Performance Benchmarks

Expected performance on Kaggle dual Tesla T4:

Model Quantization Speed VRAM
Gemma-3 1B Q4_K_M ~85 tok/s ~1 GB
Gemma-3 4B Q4_K_M ~42 tok/s ~2.5 GB
Llama-3.1 8B Q4_K_M ~25 tok/s ~5 GB

Next Steps

Resources

Full Notebook

View and run the complete notebook on Kaggle:

Kaggle

Tutorial {num}/{len(NOTEBOOKS)} | llamatelemetry v0.1.0 | Back to Tutorials