Configuration

Setting up configuration in your app

Create a ConfigManager and specify sources

Configuration is handled by a ConfigManager. When you instantiate the ConfigManager, you pass it a list of sources that it should look at when resolving configuration requests. The list of sources are consulted in the order you specify.

For example:

import os
from everett.ext.inifile import ConfigIniEnv
from everett.manager import (
    ConfigDictEnv,
    ConfigManager,
    ConfigOSEnv
)

config = ConfigManager([
    # Pull from the OS environment first
    ConfigOSEnv(),

    # Fall back to the file specified by the FOO_INI OS environment
    # variable if such file exists
    ConfigIniEnv(os.environ.get('FOO_INI')),

    # Fall back to this dict of defaults
    ConfigDictEnv({
        'FOO_VAR': 'bar'
    })
])

assert config('foo_var') == 'bar'

Specify pointer to configuration errors docs

In addition to a list of sources, you can provide a doc. You can use this to guide users trying to use your software and hitting configuration errors to documentation for your configuration.

Here’s a trivial program:

from everett.manager import ConfigManager, ConfigOSEnv


def main():
    config = ConfigManager(
        environments=[ConfigOSEnv()],
        doc='For configuration help, see https://example.com/configuration'
    )

    debug_mode = config(
        'debug', default='false', parser=bool,
        doc='True to put the app in debug mode. Don\'t use this in production!'
    )

    if debug_mode:
        print('Debug mode is on!')
    else:
        print('Debug mode off.')


if __name__ == '__main__':
    main()

Let’s configure the program wrong and run it in Python 3 and see what it tells us:

$ python trivial.py
Debug mode off.

$ DEBUG=true python trivial.py
Debug mode is on!

$ DEBUG=omg python trivial.py
Traceback (most recent call last):
  File "/home/willkg/mozilla/everett/everett/manager.py", line 908, in __call__
    return parser(val)
  File "/home/willkg/mozilla/everett/everett/manager.py", line 109, in parse_bool
    raise ValueError('"%s" is not a valid bool value' % val)
ValueError: "omg" is not a valid bool value

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "configuration_doc.py", line 22, in <module>
    main()
  File "configuration_doc.py", line 12, in main
    doc='True to put the app in debug mode. Don\'t use this in production!'
  File "/home/willkg/mozilla/everett/everett/manager.py", line 936, in __call__
    raise InvalidValueError(msg)
everett.InvalidValueError: ValueError: "omg" is not a valid bool value
namespace=None key=debug requires a value parseable by everett.manager.parse_bool
True to put the app in debug mode. Don't use this in production!
For configuration help, see https://example.com/configuration

Here, we see the documentation for the configuration item, the documentation from the ConfigManager and the specific Python exception information.

Where to put ConfigManager

You can create the ConfigManager when instantiating the app–that works fine. You could also create it as a global singleton.

ConfigManager should be thread-safe and re-entrant with the provided sources. If you implement your own configuration environments, then thread-safety and re-entrantcy depend on whether your configuration environments are safe in these ways.

Configuration sources

Dict (ConfigDictEnv)

class everett.manager.ConfigDictEnv(cfg)

Source for pulling configuration out of a dict

This is handy for testing. You might also use it if you wanted to move all your defaults values into one centralized place.

Keys are prefixed by namespaces and the whole thing is uppercased.

For example, namespace “bar” for key “foo” becomes BAR_FOO in the dict.

For example:

from everett.manager import ConfigDictEnv, ConfigManager

config = ConfigManager([
    ConfigDictEnv({
        'FOO_BAR': 'someval',
        'BAT': '1',
    })
])

Keys are not case sensitive. This also works:

from everett.manager import ConfigDictEnv, ConfigManager

config = ConfigManager([
    ConfigDictEnv({
        'foo_bar': 'someval',
        'bat': '1',
    })
])

print config('foo_bar')
print config('FOO_BAR')
print config.with_namespace('foo')('bar')

Also, ConfigManager has a convenience classmethod for creating a ConfigManager with just a dict environment:

from everett.manager import ConfigManager

config = ConfigManager.from_dict({
    'FOO_BAR': 'bat'
})

Changed in version 0.3: Keys are no longer case-sensitive.

Process environment (ConfigOSEnv)

class everett.manager.ConfigOSEnv

Source for pulling configuration out of the environment

This source lets you specify configuration in the environment. This is useful for infrastructure related configuration like usernames and ports and secret configuration like passwords.

Keys are prefixed by namespaces and the whole thing is uppercased.

For example, key “foo” will be FOO in the environment.

For example, namespace “bar” for key “foo” becomes BAR_FOO in the environment.

Key and namespace can consist of alphanumeric characters and _.

Note

Unlike other config environments, this one is case sensitive in that keys defined in the environment must be all uppercase.

For example, these are good:

FOO=bar
FOO_BAR=bar
FOO_BAR1=bar

This is bad:

foo=bar

To use, instantiate and toss in the source list:

from everett.manager import ConfigOSEnv, ConfigManager

config = ConfigManager([
    ConfigOSEnv()
])

ENV files (ConfigEnvFileEnv)

class everett.manager.ConfigEnvFileEnv(possible_paths)

Source for pulling configuration out of .env files

This source lets you specify configuration in an .env file. This is useful for local development when in production you use values in environment variables.

Keys are prefixed by namespaces and the whole thing is uppercased.

For example, key “foo” will be FOO in the file.

For example, namespace “bar” for key “foo” becomes BAR_FOO in the file.

Key and namespace can consist of alphanumeric characters and _.

To use, instantiate and toss in the source list:

from everett.manager import ConfigEnvFileEnv, ConfigManager

config = ConfigManager([
    ConfigEnvFileEnv('.env')
])

For multiple paths:

from everett.manager import ConfigEnvFileEnv, ConfigManager

config = ConfigManager([
    ConfigEnvFileEnv([
        '.env',
        'config/prod.env'
    ])
])

Here’s an example .env file:

DEBUG=true

# secrets
SECRET_KEY=ou812

# database setup
DB_HOST=localhost
DB_PORT=5432

Python objects (ConfigObjEnv)

class everett.manager.ConfigObjEnv(obj, force_lower=True)

Source for pulling configuration values out of a Python object

This is handy for a few weird situations. For example, you can use this to “bridge” Everett configuration with command line arguments. The argparse Namespace works fine here.

Namespace (the Everett one–not the argparse one) is prefixed. So key “foo” in namespace “bar” is “foo_bar”.

For example:

import argparse

from everett.manager import ConfigObjEnv, ConfigManager

parser = argparse.ArgumentParser()
parser.add_argument(
    '--debug', help='to debug or not to debug'
)
parsed_vals = parser.parse_known_args()[0]

config = ConfigManager([
    ConfigObjEnv(parsed_vals)
])

print config('debug', parser=bool)

Keys are not case-sensitive–everything is converted to lowercase before pulling it from the object.

Note

ConfigObjEnv has nothing to do with the library configobj.

New in version 0.6.

INI files (ConfigIniEnv)

class everett.ext.inifile.ConfigIniEnv(possible_paths)

Source for pulling configuration from INI files

This requires optional dependencies. You can install them with:

$ pip install everett[ini]

Takes a path or list of possible paths to look for a INI file. It uses the first INI file it can find.

If it finds no INI files in the possible paths, then this configuration source will be a no-op.

This will expand ~ as well as work relative to the current working directory.

This example looks just for the INI file specified in the environment:

from everett.manager import ConfigManager
from everett.ext.inifile import ConfigIniEnv

config = ConfigManager([
    ConfigIniEnv(os.environ.get('FOO_INI'))
])

If there’s no FOO_INI in the environment, then the path will be ignored.

Here’s an example that looks for the INI file specified in the environment variable FOO_INI and failing that will look for .antenna.ini in the user’s home directory:

from everett.manager import ConfigManager
from everett.ext.inifile import ConfigIniEnv

config = ConfigManager([
    ConfigIniEnv([
        os.environ.get('FOO_INI'),
        '~/.antenna.ini'
    ])
])

This example looks for a config/local.ini file which overrides values in a config/base.ini file both are relative to the current working directory:

from everett.manager import ConfigManager
from everett.ext.inifile import ConfigIniEnv

config = ConfigManager([
    ConfigIniEnv('config/local.ini'),
    ConfigIniEnv('config/base.ini')
])

Note how you can have multiple ConfigIniEnv files and this is how you can set Everett up to have values in one INI file override values in another INI file.

INI files must have a “main” section. This is where keys that aren’t in a namespace are placed.

Minimal INI file:

[main]

In the INI file, namespace is a section. So key “user” in namespace “foo” is:

[foo]
user=someval

Everett uses configobj, so it supports nested sections like this:

[main]
foo=bar

[namespace]
foo2=bar2

  [[namespace2]]
  foo3=bar3

Which gives you these:

  • FOO
  • NAMESPACE_FOO2
  • NAMESPACE_NAMESPACE2_FOO3

See more details here: http://configobj.readthedocs.io/en/latest/configobj.html#the-config-file-format

YAML files (ConfigYamlEnv)

class everett.ext.yamlfile.ConfigYamlEnv(possible_paths)

Source for pulling configuration from YAML files

This requires optional dependencies. You can install them with:

$ pip install everett[yaml]

Takes a path or list of possible paths to look for a YAML file. It uses the first YAML file it can find.

If it finds no YAML files in the possible paths, then this configuration source will be a no-op.

This will expand ~ as well as work relative to the current working directory.

This example looks just for the YAML file specified in the environment:

from everett.manager import ConfigManager
from everett.ext.yamlfile import ConfigYamlEnv

config = ConfigManager([
    ConfigYamlEnv(os.environ.get('FOO_YAML'))
])

If there’s no FOO_YAML in the environment, then the path will be ignored.

Here’s an example that looks for the YAML file specified in the environment variable FOO_YAML and failing that will look for .antenna.yaml in the user’s home directory:

from everett.manager import ConfigManager
from everett.ext.yamlfile import ConfigYamlEnv

config = ConfigManager([
    ConfigYamlEnv([
        os.environ.get('FOO_YAML'),
        '~/.antenna.yaml'
    ])
])

This example looks for a config/local.yaml file which overrides values in a config/base.yaml file both are relative to the current working directory:

from everett.manager import ConfigManager
from everett.ext.yamlfile import ConfigYamlEnv

config = ConfigManager([
    ConfigYamlEnv('config/local.yaml'),
    ConfigYamlEnv('config/base.yaml')
])

Note how you can have multiple ConfigYamlEnv files. This is how you can set Everett up to have values in one YAML file override values in another YAML file.

Everett looks for keys and values in YAML files. YAML files can be split into multiple documents, but Everett only looks at the first one.

Keys are case-insensitive. You can do namespaces either in the key itself using _ as a separator or as nested mappings.

All values should be double-quoted.

Here’s an example:

foo: "bar"
FOO2: "bar"
namespace_foo: "bar"
namespace:
    namespace2:
        foo: "bar"

Giving you these namespaced keys:

  • FOO
  • FOO2
  • NAMESPACE_FOO
  • NAMESPACE_NAMEPSACE2_FOO

Implementing your own configuration environments

You can implement your own configuration environments. For example, maybe you want to pull configuration from a database or Redis or a post-it note on the refrigerator.

They just need to implement the .get() method. A no-op implementation is this:

from everett import NO_VALUE
from everett.manager import listify


class NoOpEnv(object):
    def get(self, key, namespace=None):
        # The namespace is either None, a string or a list of
        # strings. This converts it into a list.
        namespace = listify(namespace)

        # FIXME: Your code to extract the key in namespace here.

        # At this point, the key doesn't exist in the namespace
        # for this environment, so return a ``NO_VALUE``.
        return NO_VALUE

Generally, environments should return a value if the key exists in that environment and should return NO_VALUE if and only if the key does not exist in that environment.

For exceptions, it depends on what you want to have happen. It’s ok to let exceptions go unhandled–Everett will wrap them in a ConfigurationError. If your environment promises never to throw an exception, then you should handle them all and return NO_VALUE since with that promise all exceptions would indicate the key is not in the environment.

Extracting values

Once you have a configuration manager set up with sources, you can pull configuration values from it.

Configuration must have a key. Other than that, everything is optionally specified.

ConfigManager.__call__(key, namespace=None, default=NO_VALUE, alternate_keys=NO_VALUE, doc='', parser=<class 'str'>, raise_error=True, raw_value=False)

Returns a parsed value from the environment

Parameters:
  • key – the key to look up
  • namespace – the namespace for the key–different environments use this differently
  • default – the default value (if any); this must be a string that is parseable by the specified parser; if no default is provided, this will raise an error or return everett.NO_VALUE depending on the value of raise_error
  • alternate_keys

    the list of alternate keys to look up; supports a root: key prefix which will cause this to look at the configuration root rather than the current namespace

    New in version 0.3.

  • doc

    documentation for this config option

    New in version 0.6.

  • parser – the parser for converting this value to a Python object
  • raise_error – True if you want a lack of value to raise a everett.ConfigurationError
  • raw_value – True if you want the raw unparsed value, False otherwise
Raises:
  • everett.ConfigurationMissingError – if the required bit of configuration is missing from all the environments
  • everett.InvalidKeyError – if the configuration key doesn’t exist for that component
  • everet.InvalidValueError – (Python 3-only) if the configuration value is invalid in some way (not an integer, not a bool, etc)
  • Exception subclass – (Python 2-only) parser code can raise anything and since this is Python 2, we can’t do much about it without stomping on the traceback so we change the message and raise the same exception

Examples:

config = ConfigManager([])

# Use the special bool parser
DEBUG = config('DEBUG', default='false', parser=bool)
DEBUG = config('DEBUG', default='True', parser=bool)
DEBUG = config('DEBUG', default='true', parser=bool)
DEBUG = config('DEBUG', default='yes', parser=bool)
DEBUG = config('DEBUG', default='y', parser=bool)

# Use the list of parser
from everett.manager import ListOf
ALLOWED_HOSTS = config('ALLOWED_HOSTS', default='localhost',
                       parser=ListOf(str))

# Use alternate_keys for backwards compatibility with an
# older version of this software
PASSWORD = config('PASSWORD', alternate_keys=['SECRET'])

The default value should always be a string that is parseable by the parser. This simplifies thinking about values since all values are strings that are parsed by the parser rather than default values do one thing and non-default values doa nother. Further, it simplifies documentation for the user since the default value is an example value.

The parser can be any callable that takes a string value and returns a parsed value.

Some more examples:

config('password')

The key is “password”.

The value is parsed as a string.

There is no default value provided so if “password” isn’t provided in any of the configuration sources, then this will raise a everett.ConfigurationError.

This is what you want to do to require that a configuration value exist.

config('name', raise_error=False)

The key is “name”.

The value is parsed as a string.

There is no default value provided and raise_error is set to False, so if this configuration variable isn’t set anywhere, the result of this will be everett.NO_VALUE.

Note

everett.NO_VALUE is a falsy value so you can use it in comparative contexts:

debug = config('DEBUG', parser=bool, raise_error=False)
if not debug:
    pass
config('debug', default='false', parser=bool)

The key is “debug”.

The value is parsed using the special Everett bool parser.

There is a default provided, so if this configuration variable isn’t set in the specified sources, the default will be false.

config('username', namespace='db')

The key is “username”.

The namespace is “db”.

There’s no default, so if there’s no “username” in namespace “db” configuration variable set in the sources, this will raise a everett.ConfigurationError.

config('password', namespace='postgres', alternate_keys=['db_password', 'root:postgres_password'])

The key is “password”.

The namespace is “postgres”.

If there is no key “password” in namespace “postgres”, then it looks for “db_password” in namespace “postgres”. This makes it possible to deprecate old key names, but still support them.

If there is no key “password” or “db_password” in namespace “postgres”, then it looks at “postgres_password” in the root namespace. This allows you to have multiple components that share configuration like credentials and hostnames.

config('port', parser=int, doc='The port you want this to listen on.')

You can provide a doc argument which will give users users who are trying to configure your software a more helpful error message when they hit a configuration error.

Example of error message with doc:

everett.InvalidValueError: ValueError: invalid literal for int() with base 10:
'bar'; namespace=None key=foo requires a value parseable by int
The port you want this to listen on.

That last line comes directly from the doc argument you provide.

class everett.ConfigurationError

Configuration error base class

class everett.InvalidValueError(*args, **kwargs)

Indicates that the value is not valid

class everett.ConfigurationMissingError(*args, **kwargs)

Indicates that required configuration is missing

class everett.InvalidKeyError

Indicates the key is not valid for this component

Handling exceptions when extracting values

Getting configuration should always return a subclass of everett.ConfigurationError. This makes it easier to programmatically figure out what happened.

For example:

#!/usr/bin/env python3

import logging

from everett import InvalidValueError
from everett.manager import ConfigManager

logging.basicConfig()

config = ConfigManager.from_dict({
    'debug_mode': 'monkey'
})

try:
    some_val = config('debug_mode', parser=bool)
except InvalidValueError:
    # The "debug_mode" configuration value is incorrect--alert
    # user in the logs.
    logging.exception('gah!')

That logs this:

ERROR:root:gah!
Traceback (most recent call last):
  File "/home/willkg/mozilla/everett/everett/manager.py", line 908, in __call__
    return parser(val)
  File "/home/willkg/mozilla/everett/everett/manager.py", line 109, in parse_bool
    raise ValueError('"%s" is not a valid bool value' % val)
ValueError: "monkey" is not a valid bool value

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "configuration_handling_exceptions.py", line 13, in <module>
    some_val = config('debug_mode', parser=bool)
  File "/home/willkg/mozilla/everett/everett/manager.py", line 936, in __call__
    raise InvalidValueError(msg)
everett.InvalidValueError: ValueError: "monkey" is not a valid bool value
namespace=None key=debug_mode requires a value parseable by everett.manager.parse_bool

Namespaces

Everett has namespaces for grouping related configuration values.

For example, say you had database code that required a username, password and port. You could do something like this:

def open_db_connection(config):
    username = config('username', namespace='db')
    password = config('password', namespace='db')
    port = config('port', namespace='db', default=5432, parser=int)


conn = open_db_connection(config)

These variables in the environment would be DB_USERNAME, DB_PASSWORD and DB_PORT.

This is helpful when you need to create two of the same thing, but using separate configuration. Extending this example, you could pass the namespace as an argument.

For example, say you wanted to use open_db_connection for a source db and for a dest db:

def open_db_connection(config, namespace):
    username = config('username', namespace=namespace)
    password = config('password', namespace=namespace)
    port = config('port', namespace=namespace, default=5432, parser=int)


source = open_db_connection(config, 'source_db')
dest = open_db_connection(config, 'dest_db')

Then you end up with SOURCE_DB_USERNAME and friends and DEST_DB_USERNAME and friends.

Parsers

Python types like str, int, float, pathlib.Path

Python types can convert strings to Python values. You can use these as parsers:

  • str
  • int
  • float
  • pathlib.Path

bools

Everett provides a special bool parser that handles more explicit values for “true” and “false”:

  • true: t, true, yes, y, on, 1 (and uppercase versions)
  • false: f, false, no, n, off, 0 (and uppercase versions)
everett.manager.parse_bool(val)

Parses a bool

Handles a series of values, but you should probably standardize on “true” and “false”.

>>> parse_bool('y')
True
>>> parse_bool('FALSE')
False

classes

Everett provides a everett.manager.parse_class that takes a string specifying a module and class and returns the class.

everett.manager.parse_class(val)

Parses a string, imports the module and returns the class

>>> parse_class('hashlib.md5')
<built-in function openssl_md5>

ListOf(parser)

Everett provides a special everett.manager.ListOf parser which parses a list of some other type. For example:

ListOf(str)  # comma-delimited list of strings
ListOf(int)  # comma-delimited list of ints
everett.manager.ListOf(parser, delimiter=', ')

Parses a comma-separated list of things

>>> ListOf(str)('')
[]
>>> ListOf(str)('a,b,c,d')
['a', 'b', 'c', 'd']
>>> ListOf(int)('1,2,3,4')
[1, 2, 3, 4]

Note: This doesn’t handle quotes or backslashes or any complicated string parsing.

For example:

>>> ListOf(str)('"a,b",c,d')
['"a', 'b"', 'c', 'd']

dj_database_url

Everett works great with dj-database-url.

For example:

import dj_database_url
from everett.manager import ConfigManager, ConfigOSEnv

config = ConfigManager([ConfigOSEnv()])
DATABASE = {
    'default': config('DATABASE_URL', parser=dj_database_url.parse)
}

That’ll pull the DATABASE_URL value from the environment (it throws an error if it’s not there) and runs it through dj_database_url which parses it and returns what Django needs.

With a default:

import dj_database_url
from everett.manager import ConfigManager, ConfigOSEnv

config = ConfigManager([ConfigOSEnv()])
DATABASE = {
    'default': config('DATABASE_URL', default='sqlite:///my.db',
                      parser=dj_database_url.parse)
}

Note

To use dj-database-url, you’ll need to install it separately. Everett doesn’t require it to be installed.

django-cache-url

Everett works great with django-cache-url.

For example:

import django_cache_url
from everett.manager import ConfigManager, ConfigOSEnv

config = ConfigManager([ConfigOSEnv()])
CACHES = {
    'default': config('CACHE_URL', parser=django_cache_url.parse)
}

That’ll pull the CACHE_URL value from the environment (it throws an error if it’s not there) and runs it through django_cache_url which parses it and returns what Django needs.

With a default:

import django_cache_url
from everett.manager import ConfigManager, ConfigOSEnv

config = ConfigManager([ConfigOSEnv()])
CACHES = {
    'default': config('CACHE_URL', default='locmem://myapp',
                      parser=django_cache_url.parse)
}

Note

To use django-cache-url, you’ll need to install it separately. Everett doesn’t require it to be installed.

Implementing your own parsers

It’s easy to implement your own parser. You just need to build a callable that takes a string and returns the Python value you want.

If the value is not parseable, then it should raise a ValueError.

For example, say we wanted to implement a parser that returned yes/no/no-answer:

from everett.manager import ConfigManager


def parse_ynm(val):
    """Returns True, False or None (empty string)"""
    val = val.strip().lower()
    if not val:
        return None

    return val[0] == 'y'


config = ConfigManager.from_dict({
    'NO_ANSWER': '',
    'YES': 'yes',
    'ALSO_YES': 'y',
    'NO': 'no'
})

assert config('no_answer', parser=parse_ynm) is None
assert config('yes', parser=parse_ynm) is True
assert config('also_yes', parser=parse_ynm) is True
assert config('no', parser=parse_ynm) is False

Say you wanted to make a parser class that’s line delimited:

from everett.manager import ConfigManager, get_parser


class Pairs(object):
    def __init__(self, val_parser):
        self.val_parser = val_parser

    def __call__(self, val):
        val_parser = get_parser(self.val_parser)
        out = []
        for part in val.split(','):
            k, v = part.split(':')
            out.append((k, val_parser(v)))
        return out


config = ConfigManager.from_dict({
    'FOO': 'a:1,b:2,c:3'
})

assert (
    config('FOO', parser=Pairs(int)) ==
    [('a', 1), ('b', 2), ('c', 3)]
)

Trouble-shooting and logging what happened

If you have a non-trivial Everett configuration, it might be difficult to figure out exactly why a key lookup failed.

Everett logs to the everett logger at the logging.DEBUG level. You can enable this logging and get a clearer idea of what’s going on.

See Python logging documentation for details on enabling logging.