Introduction

Modern Python applications often rely on configuration files to manage settings like database connections, application features, and environment-specific parameters. Popular formats include YAML and TOML because they are human-readable and easy to maintain. However, these files are prone to errors, like missing fields or invalid types, which can cause runtime crashes.

This article demonstrates how to use Pydantic, a Python data validation library, to validate YAML and TOML configurations — from basic usage to advanced production-ready techniques.

Understanding the Problem

Suppose we have a configuration file for the application:

app:
  name: MyApp
  version: 1.0
database:
  host: localhost
  port: 5432
  enabled: true

Without validation:

Goal: Automatically verify that all required fields exist, are of the correct type, and meet any constraints.

Introduction to Pydantic

Pydantic provides:

Basic Pydantic example:

from pydantic import BaseModel

class AppConfig(BaseModel):
    name: str
    version: float

app = AppConfig(name="MyApp", version=1.0)

Loading YAML and TOML in Python

Python libraries:

import yaml
import tomllib

Defining Pydantic Models for Configuration

Suppose your config has two sections: add and database

from pydantic import BaseModel, Field

class AppConfig(BaseModel):
    name: str
    version: float

class DatabaseConfig(BaseModel):
    host: str
    port: int = Field(gt=0, lt=65536)  # port must be between 1-65535
    enabled: bool

class Settings(BaseModel):
    app: AppConfig
    database: DatabaseConfig

Loading and Validating YAML

def load_yaml(file_path: str) -> Settings:
    with open(file_path, "r") as f:
        data = yaml.safe_load(f)
    return Settings.model_validate(data)  # Pydantic v2

Example usage:

settings = load_yaml("config.yaml")
print(settings.app.name)         # MyApp
print(settings.database.port)    # 5432

Loading and Validating TOML

def load_toml(file_path: str) -> Settings:
    with open(file_path, "rb") as f:
        data = tomllib.load(f)
    return Settings.model_validate(data)

TOML example (config.toml):

[app]
name = "MyApp"
version = 1.0

[database]
host = "localhost"
port = 5432
enabled = true

Unified Loader for YAML and TOML

To simplify usage:

import os
import sys

def load_config(file_path: str) -> Settings:
    if not os.path.exists(file_path):
        raise FileNotFoundError(f"Config file not found: {file_path}")
    try:
        if file_path.endswith((".yaml", ".yml")):
            return load_yaml(file_path)
        elif file_path.endswith(".toml"):
            return load_toml(file_path)
        else:
            raise ValueError("Unsupported config format")
    except ValidationError as e:
        print("Config validation failed!")
        print(e)
        sys.exit(1)

Basic Usage in Your Application

from config import load_config
import os

def main():
    config_file = os.path.join(os.path.dirname(__file__), "config.yaml")
    settings = load_config(config_file)
    print("App Name:", settings.app.name)
    print("Database Host:", settings.database.host)
if __name__ == "__main__":
    main()

Advanced Features

Environment Variable Overrides

With Pydantic v2, you can use BaseSettings to automatically override fields from environment variables:

from pydantic import BaseSettings

class Settings(BaseSettings):
    app_name: str
    database_port: int

    class Config:
        env_prefix = "MYAPP_"

Custom Validators

from pydantic import field_validator

class DatabaseConfig(BaseModel):
    host: str
    port: int
    enabled: bool

    @field_validator("host")
    @classmethod
    def host_not_empty(cls, v):
        if not v.strip():
            raise ValueError("Host cannot be empty")
        return v

Default Values and Optional Fields

class DatabaseConfig(BaseModel):
    host: str = "localhost"
    port: int = 5432
    enabled: bool = True

Fail-Fast and User-Friendly Errors

try:
    settings = load_config("config.yaml")
except ValidationError as e:
    print("Invalid config:", e)
    sys.exit(1)

Benefits of Using Pydantic

Full working example is available at my github Jayant0516 (Jayant Kumar)