How can I include a YAML file inside another

Question

So I have two YAML files   A  and  B  and I want the contents of A to be inserted inside B  either spliced into the existing data structure  like an array  or as a child of an element  like the value for a certain hash key   Is this possible at all  How  If not  any pointers to a normative reference

User · Answer

Expanding on  Josh Bode s answer  here s my own PyYAML solution  which has the advantage of being a self-contained subclass of yaml Loader  It doesn t depend on any module-level globals  or on modifying the global state of the yaml module   import yaml  os  class IncludeLoader yaml Loader                                                                                                                                           yaml Loader subclass handles   include path to foo yml  directives in config       files   When constructed with a file object  the root path for includes            defaults to the directory containing the file  otherwise to the current            working directory  In either case  the root path can be overridden by the           root  keyword argument                                                             When an included file F contain its own  include directive  the path is            relative to F s location                                                            Example                                                                                YAML file  home frodo one-ring yml                                                     ---                                                                                Name  The One Ring                                                                 Specials                                                                               - resize-to-wearer                                                             Effects                   -  include path to invisibility yml                                      YAML file  home frodo path to invisibility yml                                         ---                                                                                Name  invisibility                                                                 Message  Suddenly you disappear                                                 Loading                                                                                data   IncludeLoader open   home frodo one-ring yml    r    get data            Result                                                                                   Effects      Message    Suddenly you disappear     Name                                invisibility      Name    The One Ring    Specials                                   resize-to-wearer                                                                                                                                        def   init   self   args    kwargs                                                     super IncludeLoader  self    init    args    kwargs                                self add constructor   include   self  include                                     if  root  in kwargs                                                                    self root   kwargs  root                                                       elif isinstance self stream  file                                                      self root   os path dirname self stream name                                   else                                                                                   self root   os path curdir                                                  def  include self  loader  node                                               oldRoot   self root                                                       filename   os path join self root  loader construct scalar node           self root   os path dirname filename                                     data   yaml load open filename   r                                        self root   oldRoot                                                       return data

User · Answer

With Symfony  its handling of yaml will indirectly allow you to nest yaml files  The trick is to make use of the parameters option  eg   common yml  parameters      yaml to repeat          option   value          foo              -  bar              -  baz    config yml  imports      -   resource  common yml   whatever      thing    yaml to repeat       other thing    yaml to repeat     The result will be the same as   whatever      thing          option   value          foo              -  bar              -  baz      other thing          option   value          foo              -  bar              -  baz

User · Answer

Your question does not ask for a Python solution  but here is one using PyYAML   PyYAML allows you to attach custom constructors  such as  include  to the YAML loader  I ve included a root directory that can be set so that this solution supports relative and absolute file references   Class-Based Solution  Here is a class-based solution  that avoids the global root variable of my original response   See this gist for a similar  more robust Python 3 solution that uses a metaclass to register the custom constructor   import yaml import os  class Loader yaml SafeLoader        def   init   self  stream            self  root   os path split stream name  0           super Loader  self    init   stream       def include self  node            filename   os path join self  root  self construct scalar node            with open filename   r   as f              return yaml load f  Loader   Loader add constructor   include   Loader include    An example   foo yaml  a  1 b      - 1 43     - 543 55 c   include bar yaml   bar yaml  - 3 6 -  1  2  3    Now the files can be loaded using    gt  gt  gt  with open  foo yaml    r   as f   gt  gt  gt     data   yaml load f  Loader   gt  gt  gt  data   a   1   b    1 43  543 55    c    3 6   1  2  3

User · Answer

For Python users  you can try pyyaml-include   Install  pip install pyyaml-include   Usage  import yaml from yamlinclude import YamlIncludeConstructor  YamlIncludeConstructor add to loader class loader class yaml FullLoader  base dir   your conf dir    with open  0 yaml   as f      data   yaml load f  Loader yaml FullLoader   print data    Consider we have such YAML files    -- 0 yaml  -- include d      -- 1 yaml      -- 2 yaml    1 yaml  s content    name   1     2 yaml  s content    name   2    Include files by name   On top level   If 0 yaml was     include include d 1 yaml   We ll get     name    1      In mapping   If 0 yaml was    file1   include include d 1 yaml file2   include include d 2 yaml   We ll get     file1      name   1    file2      name   2     In sequence   If 0 yaml was    files    -  include include d 1 yaml   -  include include d 2 yaml   We ll get   files    - name   1    - name   2         Note       File name can be either absolute  like  usr conf 1 5 Make yml  or relative  like       cfg img yml     Include files by wildcards  File name can contain shell-style wildcards  Data loaded from the file s  found by wildcards will be set in a sequence   If 0 yaml was   files   include include d   yaml   We ll get   files    - name   1    - name   2         Note          For Python gt  3 5  if recursive argument of  include YAML tag is true  the pattern          will match any files and zero or more directories and subdirectories    Using the          pattern in large directory trees may consume an inordinate amount of time because of recursive search       In order to enable recursive argument  we shall write the  include tag in Mapping or Sequence mode    Arguments in Sequence mode     include  tests data include d      yaml  true     Arguments in Mapping mode     include  pathname  tests data include d      yaml  recursive  true

User · Answer

Unfortunately YAML doesn t provide this in its standard   But if you are using Ruby  there is a gem providing the functionality you are asking for by extending the ruby YAML library  https   github com entwanderer yaml extend

User · Answer

No  YAML does not include any kind of  import  or  include  statement

User · Answer

The YML standard does not specify a way to do this  And this problem does not limit itself to YML  JSON has the same limitations   Many applications which use YML or JSON based configurations run into this problem eventually  And when that happens  they make up their own convention   e g  for swagger API definitions    ref   file yml    e g  for docker compose configurations   services    app      extends        file  docker-compose base yml   Alternatively  if you want to split up the content of a yml file in multiple files  like a tree of content  you can define your own folder-structure convention and use an  existing  merge script

User · Answer

If you re using Symfony s version of YAML  this is possible  like this   imports      -   resource  sub-directory file yml       -   resource  sub-directory another-file yml

User · Answer

Standard YAML 1 2 doesn t include natively this feature  Nevertheless many implementations provides some extension to do so   I present a way of achieving it with Java and snakeyaml 1 24  Java library to parse emit YAML files  that allows creating a custom YAML tag to achieve the following goal  you will see I m using it to load test suites defined in several YAML files and that I made it work as a list of includes for a target test  node          yaml prev stuff  tests   include   -  1 hello-test-suite yaml    -  3 foo-test-suite yaml    -  2 bar-test-suite yaml         more yaml document   Here is the one-class Java that allows processing the  include tag  Files are loaded from classpath  Maven resources directory           Custom YAML loader  It adds support to the custom  include tag which allows splitting a YAML file across several    files for a better organization of YAML tests       Slf4j       lt -- This is a Lombok annotation to auto-generate logger public class MyYamlLoader        private static final Constructor CUSTOM CONSTRUCTOR   new MyYamlConstructor         private MyYamlLoader                           Parse the only YAML document in a stream and produce the Java Map  It provides support for the custom  include        YAML tag to split YAML contents across several files              public static Map lt String  Object gt  load InputStream inputStream            return new Yaml CUSTOM CONSTRUCTOR                   load inputStream                          Custom SnakeYAML constructor that registers custom tags              private static class MyYamlConstructor extends Constructor            private static final String TAG INCLUDE     include            MyYamlConstructor                    Register custom tags             yamlConstructors put new Tag TAG INCLUDE   new IncludeConstruct                                       The actual include tag construct                      private static class IncludeConstruct implements Construct                 Override             public Object construct Node node                    List lt Node gt  inclusions   castToSequenceNode node                   return parseInclusions inclusions                               Override             public void construct2ndStep Node node  Object object                       do nothing                            private List lt Node gt  castToSequenceNode Node node                    try                       return   SequenceNode  node  getValue                       catch  ClassCastException e                        throw new IllegalArgumentException String format  The  import value must be a sequence node  but                                    s  found    node                                                 private Object parseInclusions List lt Node gt  inclusions                     List lt InputStream gt  inputStreams   inputStreams inclusions                    try  final SequenceInputStream sequencedInputStream                                new SequenceInputStream Collections enumeration inputStreams                           return new Yaml CUSTOM CONSTRUCTOR                               load sequencedInputStream                      catch  IOException e                        log error  Error closing the stream    e                       return null                                               private List lt InputStream gt  inputStreams List lt Node gt  scalarNodes                    return scalarNodes stream                            map this  inputStream                           collect toList                                private InputStream inputStream Node scalarNode                    String filePath   castToScalarNode scalarNode  getValue                    final InputStream is   getClass   getClassLoader   getResourceAsStream filePath                   Assert notNull is  String format  Resource file  s not found    filePath                    return is                             private ScalarNode castToScalarNode Node scalarNode                    try                       return   ScalarNode  scalarNode                      catch  ClassCastException e                        throw new IllegalArgumentException String format  The value must be a scalar node  but   s  found                                     scalarNode

User · Answer

Includes are not directly supported in YAML as far as I know  you will have to provide a mechanism yourself however  this is generally easy to do    I have used YAML as a configuration language in my python apps  and in this case often define a convention like this    gt  gt  gt  main yml  lt  lt  lt  includes    wibble yml  wobble yml    Then in my  python  code I do   import yaml cfg   yaml load open  main yml    for inc in cfg get  includes           cfg update yaml load open inc      The only down side is that variables in the includes will always override the variables in main  and there is no way to change that precedence by changing where the  includes  statement appears in the main yml file    On a slightly different point  YAML doesn t support includes as its not really designed as as exclusively   as a file based mark up  What would an include mean if you got it in a response to an AJAX request

User · Answer

Maybe this could inspire you  try to align to jbb conventions   https   docs openstack org infra jenkins-job-builder definition html inclusion-tags   - job      name  test-job-include-raw-1     builders        - shell             include-raw  include-raw001-hello-world sh

User · Answer

I think the solution used by  maxy-B looks great  However  it didn t succeed for me with nested inclusions  For example if config 1 yaml includes config 2 yaml  which includes config 3 yaml there was a problem with the loader  However  if you simply point the new loader class to itself on load  it works  Specifically  if we replace the old  include function with the very slightly modified version   def  include self  loader  node                                            oldRoot   self root                                                    filename   os path join self root  loader construct scalar node        self root   os path dirname filename                                  data   yaml load open filename   r    loader   IncludeLoader                                   self root   oldRoot                                                    return data   Upon reflection I agree with the other comments  that nested loading is not appropriate for yaml in general as the input stream may not be a file   but it is very useful

User · Answer

With Yglu  you can import other files like this   A yaml  foo      import  B yaml     B yaml  bar  Hello     yglu A yaml foo    bar  Hello   As  import is a function  you can also pass an expression as argument     dep   - b   foo      import    dep toUpper       yaml     This would give the same output as above   Disclaimer  I am the author of Yglu

User · Answer

I make some examples for your reference   import yaml  main yaml       Package   -  include  shape yaml      -  include  path yaml       shape yaml         Define Rectangle   amp id Rectangle     name  Rectangle     width   amp Rectangle width 20     height   amp Rectangle height 10     area   product   Rectangle width   Rectangle height   Circle   amp id Circle     name  Circle     radius   amp Circle radius 5     area   product   Circle radius   Circle radius  pi     Setting Shape      property   id Rectangle     color  red       path yaml         Define Root   amp BASE  path src   Paths       a   amp id path a  join   BASE  a      b   amp id path b  join   BASE  b     Setting Path      input file   id path a         define custom tag handler def yaml import loader  node       other yaml file   loader construct scalar node      return yaml load eval other yaml file   Loader yaml SafeLoader    def yaml product loader  node       import math     list data   loader construct sequence node      result   1     pi   math pi     for val in list data          result    eval val  if isinstance val  str  else val     return result   def yaml join loader  node       seq   loader construct sequence node      return    join  str i  for i in seq     def yaml ref loader  node       ref   loader construct sequence node      return ref 0    def yaml dict ref loader  yaml loader SafeLoader  node       dict data  key  const value   loader construct sequence node      return dict data key    str const value    def main          register the tag handler     yaml SafeLoader add constructor tag   include   constructor yaml import      yaml SafeLoader add constructor tag   product   constructor yaml product      yaml SafeLoader add constructor tag   join   constructor yaml join      yaml SafeLoader add constructor tag   ref   constructor yaml ref      yaml SafeLoader add constructor tag   dict ref   constructor yaml dict ref       config   yaml load main yaml  Loader yaml SafeLoader       pk shape  pk path   config  Package       pk shape  pk path   pk shape  Shape    pk path  Path       print f shape name   pk shape  property    name          print f shape area   pk shape  property    area          print f shape color   pk shape  color           print f input file   pk path  input file        if   name         main         main      output  shape name  Rectangle shape area  200 shape color  red input file   path src a   Update 2  and you can combine it  like this    xxx yaml CREATE FONT PICTURE    PROJECTS      SUNG   amp id SUNG       name  SUNG       work dir  SUNG       output dir  temp       font pixel  24     DEFINE   amp id define  ref   id SUNG     you can use config  CREATE FONT PICTURE    DEFINE   name  work dir      font pixel    AUTO INIT      basename suffix   dict ref   id define  name   product  5  3  2      SUNG30      This is not correct    basename suffix   dict ref   id define  name   product  5  3  2      It will build by Deep-level  id define is Deep-level  2  So you must put it after 2  otherwise  it can t refer to the correct value

User · Answer

Probably it was not supported when question was asked but you can import other YAML file into one   imports    your location to yaml file Util area yaml    Though I don t have any online reference but this works for me

[yaml] How can I include a YAML file inside another?

Examples related to yaml

Examples related to transclusion