pydantic-file-secrets 0.3.0

Description:

pydanticfilesecrets 0.3.0

pydantic-file-secrets 🔑

Use file secrets in nested Pydantic Settings models, drop-in replacement for SecretsSettingsSource.









This project is inspired by discussions in Pydantic Settings and solves problems in issues #3, #30, #154.
This package unties secrets from environment variables config options and implements other long waited features.
Features

Use secret file source in nested settings models
Drop-in replacement of standard SecretsSettingsSource (up to monkey patching)
Plain or nested directory layout: /run/secrets/dir__key or /run/secrets/dir/key
Respects env_prefix, env_nested_delimiter and other config options
Has secrets_prefix, secrets_nested_delimiter, etc. to configure secrets and env vars separately
Use multiple secrets_dir directories
Pure Python thin wrapper over standard EnvSettingsSource
No third party dependencies except pydantic-settings
100% test coverage

Motivation
Nested Pydantic config can contain nested models with secret entries, as well as secrets in top level config. In dockerized environment, these entries may be read from file system, e.g. /run/secrets when using Docker Secrets:
from pydantic import BaseModel, Secret
from pydantic_settings import BaseSettings, SettingsConfigDict

class DbSettings(BaseModel):
user: str
password: Secret[str] # secret in nested model

class Settings(BaseSettings):
db: DbSettings
app_key: Secret[str] # secret in root config

model_config = SettingsConfigDict(
secrets_dir='/run/secrets',
)

Pydantic Settings has a corresponding data source, SecretsSettingsSource, but it does not load secrets in nested models. For things that DO NOT work in original Pydantic Settings, see test_pydantic_motivation.py.
Solution
The new FileSecretsSettingsSource is a drop-in replacement of stock SecretsSettingsSource.
Installation
$ pip install pydantic-file-secrets

Plain secrets directory layout
# /run/secrets/app_key
secret1

# /run/secrets/db__password
secret2

from pydantic import BaseModel, Secret
from pydantic_file_secrets import FileSecretsSettingsSource
from pydantic_settings import BaseSettings, SettingsConfigDict

class DbSettings(BaseModel):
user: str
password: Secret[str]

class Settings(BaseSettings):
db: DbSettings
app_key: Secret[str]

model_config = SettingsConfigDict(
secrets_dir='/run/secrets',
env_nested_delimiter='__',
)

@classmethod
def settings_customise_sources(
cls,
settings_cls,
init_settings,
env_settings,
dotenv_settings,
file_secret_settings,
):
return (
init_settings,
env_settings,
dotenv_settings,
FileSecretsSettingsSource(file_secret_settings),
)

Nested secrets directory layout
Config option secrets_nested_delimiter overrides env_nested_delimiter for files. In particular, this allows to use nested directory layout along with environmemt variables for other non-secret settings:
# /run/secrets/app_key
secret1

# /run/secrets/db/password
secret2

...
model_config = SettingsConfigDict(
secrets_dir='/run/secrets',
secrets_nested_subdir=True,
)
...

Multiple secrets_dir
When passing list to secrets_dir, last match wins.
...
model_config = SettingsConfigDict(
secrets_dir=['/run/configs/', '/run/secrets'],
)
...

Configuration options
secrets_dir
Path to secrets directory. Same as SecretsSettingsSource.secrets_dir if str or Path. If list, the last match wins. If secrets_dir is passed in both source constructor and model config, values are not merged (constructor takes priority).
secrets_dir_missing
If secrets_dir does not exist, original SecretsSettingsSource issues a warning. However, this may be undesirable, for example if we don't mount Docker Secrets in e.g. dev environment. Now you have a choice:

'ok' — do nothing if secrets_dir does not exist
'warn' (default) — print warning, same as SecretsSettingsSource
'error' — raise SettingsError

If multiple secrets_dir passed, the same secrets_dir_missing action applies to each of them.
secrets_dir_max_size
Limit the size of secrets_dir for security reasons, defaults to 8 MiB.
FileSecretsSettingsSource is a thin wrapper around EnvSettingsSource, which loads all potential secrets on initialization. This could lead to MemoryError if we mount a large file under secrets_dir.
If multiple secrets_dir passed, the limit applies to each directory independently.
secrets_case_sensitive
Same as case_sensitive, but works for secrets only. If not specified, defaults to case_sensitive.
secrets_nested_delimiter
Same as env_nested_delimiter, but works for secrets only. If not specified, defaults to env_nested_delimiter. This option is used to implement nested secrets directory layout and allows to do even nastier things like /run/secrets/model/delim/nested1/delim/nested2.
secrets_nested_subdir
Boolean flag to turn on nested secrets directory mode, False by default. If True, sets secrets_nested_delimiter to os.sep. Raises SettingsError if secrets_nested_delimiter is already specified.
secrets_prefix
Secret path prefix, similar to env_prefix, but works for secrets only. Defaults to env_prefix if not specified. Works in both plain and nested directory modes, like '/run/secrets/prefix_model__nested' and '/run/secrets/prefix_model/nested'.
Not supported config options
Some config options that are declared in SecretsSettingsSource interface are actually not working and are not supported in FileSecretsSettingsSource:

env_ignore_empty
env_parse_none_str
env_parse_enums

However, we make sure that the behaviour of FileSecretsSettingsSource matches SecretsSettingsSource to provide a drop-in replacement, although it is somewhat wierd (e.g. env_parse_enums is always True).
Testing
We ensure 100% test coverage for latest Python release (3.12).
We test all minor Pydantic Settings v2 versions and all minor Python 3 versions supported by Pydantic Settings:

Python 3.{8,9,10,11,12,13} + pydantic-settings 2.{0,1,2,3,4}

Roadmap

Support _FILE environment variables to set secret file name.
Per-field secret file name override.

Authors

Michael Makukha

License
MIT License