Include One Yaml File Inside Another

Question

I want to have a base config file which is used by other config files to share common config.

E.g if I have one file base.yml with

foo: 1

bar:
  - 2
  - 3

And then a second file some_file.yml with

foo: 2

baz: "baz"

What I'd want to end up with a merged config file with

foo: 2

bar:
  - 2
  - 3

baz: "baz"

It's easy enough to write a custom loader that handles an !include tag.

class ConfigLoader(yaml.SafeLoader):

    def __init__(self, stream):
        super().__init__(stream)
        self._base = Path(stream.name).parent

    def include(self, node):
        file_name = self.construct_scalar(node)
        file_path = self._base.joinpath(file_name)

        with file_path.open("rt") as fh:
            return yaml.load(fh, IncludeLoader)

Then I can parse an !include tag. So if my file is

inherit:
   !include base.yml

foo: 2

baz: "baz"

But now the base config is a mapping. I.e. if I load the the file I'll end up with

config = {'a': [42], 'c': [3.6, [1, 2, 3]], 'include': [{'a': 1, 'b': [1.43, 543.55]}]}

But if I don't make the tag part of a mapping, e.g.

!include base.yml

foo: 2

baz: "baz"

I get an error. yaml.scanner.ScannerError: mapping values are not allowed here.

But I know that the yaml parser can parse tags without needing a mapping. Because I can do things like

!!python/object:foo.Bar
x: 1.0   
y: 3.14

So how do I write a loader and/or structure my YAML file so that I can include another file in my configuration?

Answer 1

I recommend these 2 steps:

1. write or plug-in a custom load-constructor for including other files
2. structure the YAML for merging other keys (within the same file) or including other files

Similar to what Anthon, author and maintainer of ruamel.yaml answered ... but with pyyaml.

How to merge in YAML keys (from within same file)

See the YAML merge syntax <<: is for merging in keys:

---
- &OTHER { foo: 1, bar: [2, 3] }

# merge it in from above
<< : OTHER 

# the base
foo: 1

bar:
  - 2
  - 3

How to include YAML files

YAML markup syntax has no include-directive or similar. But each YAML-parser implementation can offer this feature.

For example, see another answer of Josh which has does so:

PyYAML allows you to attach custom constructors (such as !include) to the YAML loader.

This constructor can be plugged-in by pyyaml-include does:

import yaml
from yamlinclude import YamlIncludeConstructor

YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='/your/conf/dir')  # or specify another dir relatively or absolutely
# default is: include YAML files from current working directory

with open('base.yaml') as f:
    data = yaml.load(f, Loader=yaml.FullLoader)

print(data)

To include (not merging in) a second file some_file.yaml (located in same directory) given as:

foo: 1

bar:
  - 2
  - 3

within the base.yaml add:

!include some_file.yaml    # includes the file on top-level (relative path!)

foo: 1

bar:
  - 2
  - 3

See also:

• Are there specifiers to merge keys in yaml?
• How can I include a YAML file inside another?
• merge two yaml files in python

Recommended filename extension

From Wikipedia, YAML:

The official recommended filename extension for YAML files has been .yaml since 2006.12

(sourced from the official YAML-FAQ: "YAML Ain't Markup Language". September 24, 2006. Archived from the original on 2006-09-24.)