Fork me on github:
yaml-test-suite
All test cases and expected event/JSON output
yaml-test-matrix
Code to run tests for all supported processors and create matrix HTML
yaml-runtimes
Code to create Docker image for all supported processors

YAML Test Matrix

This Matrix combines all tests from yaml-test-suite and all processors from yaml-runtimes.

The YAML Test Suite currently targets YAML Version 1.2. Note that this list is not meant as a "Best Of" list, as some processors implement 1.1 or 1.0 only, and the test suite contains a lot of edge cases that don't play such an important role in real world examples. It is suited to identify which problems each processors has and what to avoid.

Test case sheet Valid Invalid
Test results per processor Valid Invalid

Valid (248) Invalid (74)
Processors ok n/i diff error n/a correct incorrect
event - Compare parsing events
c-libfyaml-event C libfyaml      248 74
c-libyaml-event C libyaml      214 7 1 26 64 10
cpp-yamlcpp-event C++ yaml-cpp      227 12 9 45 29
dotnet-yamldotnet-event C# YamlDotNet      246 2 73 1
hs-hsyaml-event Haskell HsYAML      247 1 74
java-snakeyaml-event Java SnakeYAML      211 7 1 29 62 12
js-yaml-event Javascript yaml      246 1 1 72 2
nim-nimyaml-event Nim NimYAML      248 65 9
perl-pp-event Perl YAML::PP      241 7 70 4
perl-pplibyaml-event Perl YAML::PP::LibYAML      214 7 1 26 64 10
perl-refparser-event Perl RefParser      248 74
py-pyyaml-event Python PyYAML      211 7 1 29 64 10
py-ruamel-event Python ruamel.yaml      225 2 21 61 13
json - Compare loaded data with JSON
c-libfyaml-json C libfyaml       223 1 24 74
dotnet-yamldotnet-json C# YamlDotNet       151 16 34 23 24 74
hs-hsyaml-json Haskell HsYAML       223 1 24 74
java-snakeyaml-json Java SnakeYAML       167 13 15 29 24 65 9
js-jsyaml-json Javascript js-yaml       176 12 20 16 24 65 9
js-yaml-json Javascript yaml       217 7 24 72 2
lua-lyaml-json Lua lyaml       176 17 31 24 65 9
perl-pp-json Perl YAML::PP       222 2 24 70 4
perl-pplibyaml-json Perl YAML::PP::LibYAML       198 1 25 24 64 10
perl-syck-json Perl YAML::Syck       141 47 36 24 40 34
perl-tiny-json Perl YAML::Tiny       39 96 31 58 24 64 10
perl-xs-json Perl YAML::XS (libyaml)       184 8 1 31 24 65 9
perl-yaml-json Perl YAML.pm       82 13 129 24 66 8
py-pyyaml-json Python PyYAML       186 4 34 24 64 10
py-ruamel-json Python ruamel.yaml       199 2 23 24 61 13
ruby-psych-json Ruby psych       184 8 32 24 65 9
Generated with yaml-test-suite/data Commit 80f28943 2021-10-09 Regenerated data from main v2021-10-09

Legend

Columns

events
 okEvents matched
 n/iThis feature is not implemented in this library
 diffEvent output differs
 errorParsing was aborted with an error
 n/aNo data to compare for this test
 invalid_correctInput was correctly detected as invalid
 invalid_incorrectInput was accepted although it is invalid
json
 okLoaded data is equal to JSON
 n/iThis feature is not implemented in this library
 diffLoaded data differs
 errorParsing/Loading was aborted with an error
 n/aNo data to compare for this test
 invalid_correctInput was correctly detected as invalid
 invalid_incorrectInput was accepted although it is invalid

Which processors don't implement which features?




---
c-libyaml.event:        [ empty-key ]
dotnet-yamldotnet.json: [ local-tag, unknown-tag ]
java-snakeyaml.event:   [ empty-key ]
java-snakeyaml.json:    [ empty-key, local-tag, unknown-tag ]
js-jsyaml.json:         [ local-tag, unknown-tag ]
js-yaml.event:          [ duplicate-key ]
lua-lyaml-json:         [ empty-key ]
perl-pplibyaml.event:   [ empty-key ]
perl-pplibyaml.json:    [ empty-key ]
perl-pplibyaml.perl:    [ empty-key ]
perl-tiny.json:         &tiny [ alias, anchor, complex-key, empty-key,
                          explicit-key, flow, tag ]
perl-tiny.perl:         *tiny
perl-xs.json:           [ empty-key, unknown-tag ]
perl-xs.perl:           [ empty-key, unknown-tag ]
py-pyyaml.event:        [ empty-key ]
py-pyyaml.json:         [ empty-key ]
py-pyyaml.py:           [ empty-key ]

 

n/i - Not implemented


complex-key:
The test uses non-scalar keys, for example a mapping or a sequence.
---
? key: value
: value
---
[a, list]: value
duplicate-key:
The test has a duplicate key, which is valid YAML syntax, so the parser should accept it, but it should be an error when loaded.
a: 1
a: 2
empty-key:
The test uses an empty node as a key. For example in libyaml/PyYAML this wasn't implemented deliberately, although it was already part of YAML 1.1. And many implementations used or ported from those libraries. Use of empty keys is discouraged and might be removed in the next YAML version.
key: 1
: 2
explicit-key:
The test uses '?' to mark a key as explicit.
---
? |
  some multiline
  key
: value
---
? a: mapping
: as a key
flow:
The test uses flow style collections.
---
mapping: { a: 1, b: 2 }
sequence: [ a, b, c ]
local-tag:
The test uses a local tag with one exclamation mark. This is accepted by most parsers, but some libraries cannot load it into an object, or have no option to ignore such tags.
---
some: !local tag
unknown-tag:
The test uses a tag with the standard namespace '!!', but it is not part of the YAML 1.2 Core schema. A library should ideally be able to optionally ignore them or abort loading. Parsing should pass without an error.
---
some: !!nonsense tag