Configuration¶
Contents
Setting up configuration in your app¶
Create a ConfigManager and specify sources¶
Configuration is handled by a ConfigManager
. When you instantiate the
ConfigManager
, you pass it a list of sources that it should look at when
resolving configuration requests. The list of sources are consulted in the order
you specify.
For example:
import os
from everett.ext.inifile import ConfigIniEnv
from everett.manager import (
ConfigDictEnv,
ConfigManager,
ConfigOSEnv
)
config = ConfigManager([
# Pull from the OS environment first
ConfigOSEnv(),
# Fall back to the file specified by the FOO_INI OS environment
# variable if such file exists
ConfigIniEnv(os.environ.get('FOO_INI')),
# Fall back to this dict of defaults
ConfigDictEnv({
'FOO_VAR': 'bar'
})
])
assert config('foo_var') == 'bar'
Specify pointer to configuration errors docs¶
In addition to a list of sources, you can provide a doc
. You can use this to
guide users trying to use your software and hitting configuration errors to
documentation for your configuration.
Here’s a trivial program:
from everett.manager import ConfigManager, ConfigOSEnv
def main():
config = ConfigManager(
environments=[ConfigOSEnv()],
doc='For configuration help, see https://example.com/configuration'
)
debug_mode = config(
'debug', default='false', parser=bool,
doc='True to put the app in debug mode. Don\'t use this in production!'
)
if debug_mode:
print('Debug mode is on!')
else:
print('Debug mode off.')
if __name__ == '__main__':
main()
Let’s configure the program wrong and run it in Python 3 and see what it tells us:
$ python trivial.py
Debug mode off.
$ DEBUG=true python trivial.py
Debug mode is on!
$ DEBUG=omg python trivial.py
Traceback (most recent call last):
File "/home/willkg/mozilla/everett/everett/manager.py", line 908, in __call__
return parser(val)
File "/home/willkg/mozilla/everett/everett/manager.py", line 109, in parse_bool
raise ValueError('"%s" is not a valid bool value' % val)
ValueError: "omg" is not a valid bool value
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "configuration_doc.py", line 22, in <module>
main()
File "configuration_doc.py", line 12, in main
doc='True to put the app in debug mode. Don\'t use this in production!'
File "/home/willkg/mozilla/everett/everett/manager.py", line 936, in __call__
raise InvalidValueError(msg)
everett.InvalidValueError: ValueError: "omg" is not a valid bool value
namespace=None key=debug requires a value parseable by everett.manager.parse_bool
True to put the app in debug mode. Don't use this in production!
For configuration help, see https://example.com/configuration
Here, we see the documentation for the configuration item, the documentation
from the ConfigManager
and the specific Python exception information.
Where to put ConfigManager¶
You can create the ConfigManager
when instantiating the app–that works
fine. You could also create it as a global singleton.
ConfigManager
should be thread-safe and re-entrant with the provided
sources. If you implement your own configuration environments, then
thread-safety and re-entrantcy depend on whether your configuration environments
are safe in these ways.
Configuration sources¶
Dict (ConfigDictEnv)¶
-
class
everett.manager.
ConfigDictEnv
(cfg) Source for pulling configuration out of a dict
This is handy for testing. You might also use it if you wanted to move all your defaults values into one centralized place.
Keys are prefixed by namespaces and the whole thing is uppercased.
For example, namespace “bar” for key “foo” becomes
BAR_FOO
in the dict.For example:
from everett.manager import ConfigDictEnv, ConfigManager config = ConfigManager([ ConfigDictEnv({ 'FOO_BAR': 'someval', 'BAT': '1', }) ])
Keys are not case sensitive. This also works:
from everett.manager import ConfigDictEnv, ConfigManager config = ConfigManager([ ConfigDictEnv({ 'foo_bar': 'someval', 'bat': '1', }) ]) print config('foo_bar') print config('FOO_BAR') print config.with_namespace('foo')('bar')
Also,
ConfigManager
has a convenience classmethod for creating aConfigManager
with just a dict environment:from everett.manager import ConfigManager config = ConfigManager.from_dict({ 'FOO_BAR': 'bat' })
Changed in version 0.3: Keys are no longer case-sensitive.
Process environment (ConfigOSEnv)¶
-
class
everett.manager.
ConfigOSEnv
Source for pulling configuration out of the environment
This source lets you specify configuration in the environment. This is useful for infrastructure related configuration like usernames and ports and secret configuration like passwords.
Keys are prefixed by namespaces and the whole thing is uppercased.
For example, key “foo” will be
FOO
in the environment.For example, namespace “bar” for key “foo” becomes
BAR_FOO
in the environment.Key and namespace can consist of alphanumeric characters and
_
.Note
Unlike other config environments, this one is case sensitive in that keys defined in the environment must be all uppercase.
For example, these are good:
FOO=bar FOO_BAR=bar FOO_BAR1=bar
This is bad:
foo=bar
To use, instantiate and toss in the source list:
from everett.manager import ConfigOSEnv, ConfigManager config = ConfigManager([ ConfigOSEnv() ])
ENV files (ConfigEnvFileEnv)¶
-
class
everett.manager.
ConfigEnvFileEnv
(possible_paths) Source for pulling configuration out of .env files
This source lets you specify configuration in an .env file. This is useful for local development when in production you use values in environment variables.
Keys are prefixed by namespaces and the whole thing is uppercased.
For example, key “foo” will be
FOO
in the file.For example, namespace “bar” for key “foo” becomes
BAR_FOO
in the file.Key and namespace can consist of alphanumeric characters and
_
.To use, instantiate and toss in the source list:
from everett.manager import ConfigEnvFileEnv, ConfigManager config = ConfigManager([ ConfigEnvFileEnv('.env') ])
For multiple paths:
from everett.manager import ConfigEnvFileEnv, ConfigManager config = ConfigManager([ ConfigEnvFileEnv([ '.env', 'config/prod.env' ]) ])
Here’s an example .env file:
DEBUG=true # secrets SECRET_KEY=ou812 # database setup DB_HOST=localhost DB_PORT=5432
Python objects (ConfigObjEnv)¶
-
class
everett.manager.
ConfigObjEnv
(obj, force_lower=True) Source for pulling configuration values out of a Python object
This is handy for a few weird situations. For example, you can use this to “bridge” Everett configuration with command line arguments. The argparse Namespace works fine here.
Namespace (the Everett one–not the argparse one) is prefixed. So key “foo” in namespace “bar” is “foo_bar”.
For example:
import argparse from everett.manager import ConfigObjEnv, ConfigManager parser = argparse.ArgumentParser() parser.add_argument( '--debug', help='to debug or not to debug' ) parsed_vals = parser.parse_known_args()[0] config = ConfigManager([ ConfigObjEnv(parsed_vals) ]) print config('debug', parser=bool)
Keys are not case-sensitive–everything is converted to lowercase before pulling it from the object.
Note
ConfigObjEnv has nothing to do with the library configobj.
New in version 0.6.
INI files (ConfigIniEnv)¶
-
class
everett.ext.inifile.
ConfigIniEnv
(possible_paths) Source for pulling configuration from INI files
This requires optional dependencies. You can install them with:
$ pip install everett[ini]
Takes a path or list of possible paths to look for a INI file. It uses the first INI file it can find.
If it finds no INI files in the possible paths, then this configuration source will be a no-op.
This will expand
~
as well as work relative to the current working directory.This example looks just for the INI file specified in the environment:
from everett.manager import ConfigManager from everett.ext.inifile import ConfigIniEnv config = ConfigManager([ ConfigIniEnv(os.environ.get('FOO_INI')) ])
If there’s no
FOO_INI
in the environment, then the path will be ignored.Here’s an example that looks for the INI file specified in the environment variable
FOO_INI
and failing that will look for.antenna.ini
in the user’s home directory:from everett.manager import ConfigManager from everett.ext.inifile import ConfigIniEnv config = ConfigManager([ ConfigIniEnv([ os.environ.get('FOO_INI'), '~/.antenna.ini' ]) ])
This example looks for a
config/local.ini
file which overrides values in aconfig/base.ini
file both are relative to the current working directory:from everett.manager import ConfigManager from everett.ext.inifile import ConfigIniEnv config = ConfigManager([ ConfigIniEnv('config/local.ini'), ConfigIniEnv('config/base.ini') ])
Note how you can have multiple
ConfigIniEnv
files and this is how you can set Everett up to have values in one INI file override values in another INI file.INI files must have a “main” section. This is where keys that aren’t in a namespace are placed.
Minimal INI file:
[main]
In the INI file, namespace is a section. So key “user” in namespace “foo” is:
[foo] user=someval
Everett uses configobj, so it supports nested sections like this:
[main] foo=bar [namespace] foo2=bar2 [[namespace2]] foo3=bar3
Which gives you these:
FOO
NAMESPACE_FOO2
NAMESPACE_NAMESPACE2_FOO3
See more details here: http://configobj.readthedocs.io/en/latest/configobj.html#the-config-file-format
YAML files (ConfigYamlEnv)¶
-
class
everett.ext.yamlfile.
ConfigYamlEnv
(possible_paths) Source for pulling configuration from YAML files
This requires optional dependencies. You can install them with:
$ pip install everett[yaml]
Takes a path or list of possible paths to look for a YAML file. It uses the first YAML file it can find.
If it finds no YAML files in the possible paths, then this configuration source will be a no-op.
This will expand
~
as well as work relative to the current working directory.This example looks just for the YAML file specified in the environment:
from everett.manager import ConfigManager from everett.ext.yamlfile import ConfigYamlEnv config = ConfigManager([ ConfigYamlEnv(os.environ.get('FOO_YAML')) ])
If there’s no
FOO_YAML
in the environment, then the path will be ignored.Here’s an example that looks for the YAML file specified in the environment variable
FOO_YAML
and failing that will look for.antenna.yaml
in the user’s home directory:from everett.manager import ConfigManager from everett.ext.yamlfile import ConfigYamlEnv config = ConfigManager([ ConfigYamlEnv([ os.environ.get('FOO_YAML'), '~/.antenna.yaml' ]) ])
This example looks for a
config/local.yaml
file which overrides values in aconfig/base.yaml
file both are relative to the current working directory:from everett.manager import ConfigManager from everett.ext.yamlfile import ConfigYamlEnv config = ConfigManager([ ConfigYamlEnv('config/local.yaml'), ConfigYamlEnv('config/base.yaml') ])
Note how you can have multiple
ConfigYamlEnv
files. This is how you can set Everett up to have values in one YAML file override values in another YAML file.Everett looks for keys and values in YAML files. YAML files can be split into multiple documents, but Everett only looks at the first one.
Keys are case-insensitive. You can do namespaces either in the key itself using
_
as a separator or as nested mappings.All values should be double-quoted.
Here’s an example:
foo: "bar" FOO2: "bar" namespace_foo: "bar" namespace: namespace2: foo: "bar"
Giving you these namespaced keys:
FOO
FOO2
NAMESPACE_FOO
NAMESPACE_NAMEPSACE2_FOO
Implementing your own configuration environments¶
You can implement your own configuration environments. For example, maybe you want to pull configuration from a database or Redis or a post-it note on the refrigerator.
They just need to implement the .get()
method. A no-op implementation is
this:
from everett import NO_VALUE
from everett.manager import listify
class NoOpEnv(object):
def get(self, key, namespace=None):
# The namespace is either None, a string or a list of
# strings. This converts it into a list.
namespace = listify(namespace)
# FIXME: Your code to extract the key in namespace here.
# At this point, the key doesn't exist in the namespace
# for this environment, so return a ``NO_VALUE``.
return NO_VALUE
Generally, environments should return a value if the key exists in that
environment and should return NO_VALUE
if and only if the key does not
exist in that environment.
For exceptions, it depends on what you want to have happen. It’s ok to let
exceptions go unhandled–Everett will wrap them in a ConfigurationError
.
If your environment promises never to throw an exception, then you should
handle them all and return NO_VALUE
since with that promise all exceptions
would indicate the key is not in the environment.
Extracting values¶
Once you have a configuration manager set up with sources, you can pull configuration values from it.
Configuration must have a key. Other than that, everything is optionally specified.
-
ConfigManager.
__call__
(key, namespace=None, default=NO_VALUE, alternate_keys=NO_VALUE, doc='', parser=<class 'str'>, raise_error=True, raw_value=False)¶ Returns a parsed value from the environment
Parameters: - key – the key to look up
- namespace – the namespace for the key–different environments use this differently
- default – the default value (if any); this must be a string that is
parseable by the specified parser; if no default is provided, this
will raise an error or return
everett.NO_VALUE
depending on the value ofraise_error
- alternate_keys –
the list of alternate keys to look up; supports a
root:
key prefix which will cause this to look at the configuration root rather than the current namespaceNew in version 0.3.
- doc –
documentation for this config option
New in version 0.6.
- parser – the parser for converting this value to a Python object
- raise_error – True if you want a lack of value to raise a
everett.ConfigurationError
- raw_value – True if you want the raw unparsed value, False otherwise
Raises: - everett.ConfigurationMissingError – if the required bit of configuration is missing from all the environments
- everett.InvalidKeyError – if the configuration key doesn’t exist for that component
- everet.InvalidValueError – (Python 3-only) if the configuration value is invalid in some way (not an integer, not a bool, etc)
- Exception subclass – (Python 2-only) parser code can raise anything and since this is Python 2, we can’t do much about it without stomping on the traceback so we change the message and raise the same exception
Examples:
config = ConfigManager([]) # Use the special bool parser DEBUG = config('DEBUG', default='false', parser=bool) DEBUG = config('DEBUG', default='True', parser=bool) DEBUG = config('DEBUG', default='true', parser=bool) DEBUG = config('DEBUG', default='yes', parser=bool) DEBUG = config('DEBUG', default='y', parser=bool) # Use the list of parser from everett.manager import ListOf ALLOWED_HOSTS = config('ALLOWED_HOSTS', default='localhost', parser=ListOf(str)) # Use alternate_keys for backwards compatibility with an # older version of this software PASSWORD = config('PASSWORD', alternate_keys=['SECRET'])
The default value should always be a string that is parseable by the parser. This simplifies thinking about values since all values are strings that are parsed by the parser rather than default values do one thing and non-default values doa nother. Further, it simplifies documentation for the user since the default value is an example value.
The parser can be any callable that takes a string value and returns a parsed value.
Some more examples:
config('password')
The key is “password”.
The value is parsed as a string.
There is no default value provided so if “password” isn’t provided in any of the configuration sources, then this will raise a
everett.ConfigurationError
.This is what you want to do to require that a configuration value exist.
config('name', raise_error=False)
The key is “name”.
The value is parsed as a string.
There is no default value provided and raise_error is set to False, so if this configuration variable isn’t set anywhere, the result of this will be
everett.NO_VALUE
.Note
everett.NO_VALUE
is a falsy value so you can use it in comparative contexts:debug = config('DEBUG', parser=bool, raise_error=False) if not debug: pass
config('debug', default='false', parser=bool)
The key is “debug”.
The value is parsed using the special Everett bool parser.
There is a default provided, so if this configuration variable isn’t set in the specified sources, the default will be false.
config('username', namespace='db')
The key is “username”.
The namespace is “db”.
There’s no default, so if there’s no “username” in namespace “db” configuration variable set in the sources, this will raise a
everett.ConfigurationError
.config('password', namespace='postgres', alternate_keys=['db_password', 'root:postgres_password'])
The key is “password”.
The namespace is “postgres”.
If there is no key “password” in namespace “postgres”, then it looks for “db_password” in namespace “postgres”. This makes it possible to deprecate old key names, but still support them.
If there is no key “password” or “db_password” in namespace “postgres”, then it looks at “postgres_password” in the root namespace. This allows you to have multiple components that share configuration like credentials and hostnames.
config('port', parser=int, doc='The port you want this to listen on.')
You can provide a
doc
argument which will give users users who are trying to configure your software a more helpful error message when they hit a configuration error.Example of error message with
doc
:everett.InvalidValueError: ValueError: invalid literal for int() with base 10: 'bar'; namespace=None key=foo requires a value parseable by int The port you want this to listen on.
That last line comes directly from the
doc
argument you provide.
-
class
everett.
ConfigurationError
¶ Configuration error base class
-
class
everett.
InvalidValueError
(*args, **kwargs)¶ Indicates that the value is not valid
-
class
everett.
ConfigurationMissingError
(*args, **kwargs)¶ Indicates that required configuration is missing
-
class
everett.
InvalidKeyError
¶ Indicates the key is not valid for this component
Handling exceptions when extracting values¶
Getting configuration should always return a subclass of
everett.ConfigurationError
. This makes it easier to
programmatically figure out what happened.
For example:
#!/usr/bin/env python3
import logging
from everett import InvalidValueError
from everett.manager import ConfigManager
logging.basicConfig()
config = ConfigManager.from_dict({
'debug_mode': 'monkey'
})
try:
some_val = config('debug_mode', parser=bool)
except InvalidValueError:
# The "debug_mode" configuration value is incorrect--alert
# user in the logs.
logging.exception('gah!')
That logs this:
ERROR:root:gah!
Traceback (most recent call last):
File "/home/willkg/mozilla/everett/everett/manager.py", line 908, in __call__
return parser(val)
File "/home/willkg/mozilla/everett/everett/manager.py", line 109, in parse_bool
raise ValueError('"%s" is not a valid bool value' % val)
ValueError: "monkey" is not a valid bool value
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "configuration_handling_exceptions.py", line 13, in <module>
some_val = config('debug_mode', parser=bool)
File "/home/willkg/mozilla/everett/everett/manager.py", line 936, in __call__
raise InvalidValueError(msg)
everett.InvalidValueError: ValueError: "monkey" is not a valid bool value
namespace=None key=debug_mode requires a value parseable by everett.manager.parse_bool
Namespaces¶
Everett has namespaces for grouping related configuration values.
For example, say you had database code that required a username, password and port. You could do something like this:
def open_db_connection(config):
username = config('username', namespace='db')
password = config('password', namespace='db')
port = config('port', namespace='db', default=5432, parser=int)
conn = open_db_connection(config)
These variables in the environment would be DB_USERNAME
, DB_PASSWORD
and DB_PORT
.
This is helpful when you need to create two of the same thing, but using separate configuration. Extending this example, you could pass the namespace as an argument.
For example, say you wanted to use open_db_connection
for a source
db and for a dest db:
def open_db_connection(config, namespace):
username = config('username', namespace=namespace)
password = config('password', namespace=namespace)
port = config('port', namespace=namespace, default=5432, parser=int)
source = open_db_connection(config, 'source_db')
dest = open_db_connection(config, 'dest_db')
Then you end up with SOURCE_DB_USERNAME
and friends and
DEST_DB_USERNAME
and friends.
Parsers¶
Python types like str, int, float, pathlib.Path¶
Python types can convert strings to Python values. You can use these as parsers:
str
int
float
pathlib.Path
bools¶
Everett provides a special bool parser that handles more explicit values for “true” and “false”:
- true: t, true, yes, y, on, 1 (and uppercase versions)
- false: f, false, no, n, off, 0 (and uppercase versions)
-
everett.manager.
parse_bool
(val) Parses a bool
Handles a series of values, but you should probably standardize on “true” and “false”.
>>> parse_bool('y') True >>> parse_bool('FALSE') False
classes¶
Everett provides a everett.manager.parse_class
that takes a
string specifying a module and class and returns the class.
-
everett.manager.
parse_class
(val) Parses a string, imports the module and returns the class
>>> parse_class('hashlib.md5') <built-in function openssl_md5>
ListOf(parser)¶
Everett provides a special everett.manager.ListOf
parser which
parses a list of some other type. For example:
ListOf(str) # comma-delimited list of strings
ListOf(int) # comma-delimited list of ints
-
everett.manager.
ListOf
(parser, delimiter=', ') Parses a comma-separated list of things
>>> ListOf(str)('') [] >>> ListOf(str)('a,b,c,d') ['a', 'b', 'c', 'd'] >>> ListOf(int)('1,2,3,4') [1, 2, 3, 4]
Note: This doesn’t handle quotes or backslashes or any complicated string parsing.
For example:
>>> ListOf(str)('"a,b",c,d') ['"a', 'b"', 'c', 'd']
dj_database_url¶
Everett works great with dj-database-url.
For example:
import dj_database_url
from everett.manager import ConfigManager, ConfigOSEnv
config = ConfigManager([ConfigOSEnv()])
DATABASE = {
'default': config('DATABASE_URL', parser=dj_database_url.parse)
}
That’ll pull the DATABASE_URL
value from the environment (it throws an error
if it’s not there) and runs it through dj_database_url
which parses it and
returns what Django needs.
With a default:
import dj_database_url
from everett.manager import ConfigManager, ConfigOSEnv
config = ConfigManager([ConfigOSEnv()])
DATABASE = {
'default': config('DATABASE_URL', default='sqlite:///my.db',
parser=dj_database_url.parse)
}
Note
To use dj-database-url, you’ll need to install it separately. Everett doesn’t require it to be installed.
django-cache-url¶
Everett works great with django-cache-url.
For example:
import django_cache_url
from everett.manager import ConfigManager, ConfigOSEnv
config = ConfigManager([ConfigOSEnv()])
CACHES = {
'default': config('CACHE_URL', parser=django_cache_url.parse)
}
That’ll pull the CACHE_URL
value from the environment (it throws an error if
it’s not there) and runs it through django_cache_url
which parses it and
returns what Django needs.
With a default:
import django_cache_url
from everett.manager import ConfigManager, ConfigOSEnv
config = ConfigManager([ConfigOSEnv()])
CACHES = {
'default': config('CACHE_URL', default='locmem://myapp',
parser=django_cache_url.parse)
}
Note
To use django-cache-url, you’ll need to install it separately. Everett doesn’t require it to be installed.
Implementing your own parsers¶
It’s easy to implement your own parser. You just need to build a callable that takes a string and returns the Python value you want.
If the value is not parseable, then it should raise a ValueError
.
For example, say we wanted to implement a parser that returned yes/no/no-answer:
from everett.manager import ConfigManager
def parse_ynm(val):
"""Returns True, False or None (empty string)"""
val = val.strip().lower()
if not val:
return None
return val[0] == 'y'
config = ConfigManager.from_dict({
'NO_ANSWER': '',
'YES': 'yes',
'ALSO_YES': 'y',
'NO': 'no'
})
assert config('no_answer', parser=parse_ynm) is None
assert config('yes', parser=parse_ynm) is True
assert config('also_yes', parser=parse_ynm) is True
assert config('no', parser=parse_ynm) is False
Say you wanted to make a parser class that’s line delimited:
from everett.manager import ConfigManager, get_parser
class Pairs(object):
def __init__(self, val_parser):
self.val_parser = val_parser
def __call__(self, val):
val_parser = get_parser(self.val_parser)
out = []
for part in val.split(','):
k, v = part.split(':')
out.append((k, val_parser(v)))
return out
config = ConfigManager.from_dict({
'FOO': 'a:1,b:2,c:3'
})
assert (
config('FOO', parser=Pairs(int)) ==
[('a', 1), ('b', 2), ('c', 3)]
)
Trouble-shooting and logging what happened¶
If you have a non-trivial Everett configuration, it might be difficult to figure out exactly why a key lookup failed.
Everett logs to the everett
logger at the logging.DEBUG
level. You
can enable this logging and get a clearer idea of what’s going on.
See Python logging documentation for details on enabling logging.