Fork me on github:
yaml-test-suite
All test cases and expected event/JSON output yaml-test-matrix
Code to run tests for all supported processors and create matrix HTML yaml-runtimes
Code to create Docker image for all supported processors

YAML Test Matrix

This Matrix combines all tests from yaml-test-suite and all processors from yaml-runtimes.

The YAML Test Suite currently targets YAML Version 1.2. Note that this list is not meant as a "Best Of" list, as some processors implement 1.1 or 1.0 only, and the test suite contains a lot of edge cases that don't play such an important role in real world examples. It is suited to identify which problems each processors has and what to avoid.

Test case sheet Valid Invalid
Test results per processor Valid Invalid

Valid (308) Invalid (94) Total (402)
Processors ok n/i diff error n/a pass fail pass fail
event - Compare parsing events
c-libfyaml-event C libfyaml      308 94 402
c-libyaml-event C libyaml      252 9 5 42 78 16 330 72
cpp-rapidyaml-event C++ rapidyaml      274 26 8 32 62 306 96
cpp-yamlcpp-event C++ yaml-cpp      278 15 15 60 34 338 64
dotnet-yamldotnet-event C# YamlDotNet      293 5 10 83 11 376 26
hs-hsyaml-event Haskell HsYAML      298 1 9 94 392 10
js-yaml-event Javascript yaml      307 1 92 2 399 3
nim-nimyaml-event Nim NimYAML      302 1 5 76 18 378 24
perl-pp-event Perl YAML::PP      300 8 90 4 390 12
perl-refparser-event Perl Generated RefParser      308 94 402
py-pyyaml-event Python PyYAML      249 9 5 45 80 14 329 73
py-ruamel-event Python ruamel.yaml      265 6 37 80 14 345 57
rust-yamlrust-event Rust yaml      251 28 29 66 28 317 85
json - Compare loaded data with JSON
c-libfyaml-json C libfyaml       278 1 29 94 372 1
dotnet-yamldotnet-json C# YamlDotNet       173 16 60 30 29 84 10 257 116
go-yaml-json Go go-yaml       224 6 49 29 79 15 303 70
hs-hsyaml-json Haskell HsYAML       269 1 9 29 94 363 10
js-jsyaml-json Javascript js-yaml       224 12 24 19 29 82 12 306 67
js-yaml-json Javascript yaml       276 3 29 92 2 368 5
lua-lyaml-json Lua lyaml       210 29 40 29 78 16 288 85
perl-pp-json Perl YAML::PP       277 2 29 90 4 367 6
perl-pplibyaml-json Perl YAML::PP::LibYAML       234 5 40 29 78 16 312 61
perl-syck-json Perl YAML::Syck       166 77 36 29 42 52 208 165
perl-tiny-json Perl YAML::Tiny       44 104 41 90 29 80 14 124 249
perl-xs-json Perl YAML::XS (libyaml)       220 8 5 46 29 79 15 299 74
perl-yaml-json Perl YAML.pm       98 15 166 29 79 15 177 196
py-pyyaml-json Python PyYAML       222 8 49 29 80 14 302 71
py-ruamel-json Python ruamel.yaml       234 6 39 29 80 14 314 59
ruby-psych-json Ruby psych       226 12 41 29 78 16 304 69
Generated with yaml-test-suite/data Commit 6e6c296a 2022-01-17 Regenerated data from main v2022-01-17

Legend

Columns

events
 okEvents matched
 n/iThis feature is not implemented in this library
 diffEvent output differs
 errorParsing was aborted with an error
 n/aNo data to compare for this test
 invalid_correctInput was correctly detected as invalid
 invalid_incorrectInput was accepted although it is invalid
json
 okLoaded data is equal to JSON
 n/iThis feature is not implemented in this library
 diffLoaded data differs
 errorParsing/Loading was aborted with an error
 n/aNo data to compare for this test
 invalid_correctInput was correctly detected as invalid
 invalid_incorrectInput was accepted although it is invalid

Which processors don't implement which features?




---
c-libyaml.event:        [ empty-key ]
dotnet-yamldotnet.json: [ local-tag, unknown-tag ]
java-snakeyaml.event:   [ empty-key ]
java-snakeyaml.json:    [ empty-key, local-tag, unknown-tag ]
js-jsyaml.json:         [ local-tag, unknown-tag ]
js-yaml.event:          [ duplicate-key ]
lua-lyaml-json:         [ empty-key ]
perl-pplibyaml.event:   [ empty-key ]
perl-pplibyaml.json:    [ empty-key ]
perl-pplibyaml.perl:    [ empty-key ]
perl-tiny.json:         &tiny [ alias, anchor, complex-key, empty-key,
                          explicit-key, flow, tag ]
perl-tiny.perl:         *tiny
perl-xs.json:           [ empty-key, unknown-tag ]
perl-xs.perl:           [ empty-key, unknown-tag ]
py-pyyaml.event:        [ empty-key ]
py-pyyaml.json:         [ empty-key ]
py-pyyaml.py:           [ empty-key ]

n/i - Not implemented


complex-key:
The test uses non-scalar keys, for example a mapping or a sequence. 
---
? key: value
: value
---
[a, list]: value
duplicate-key:
The test has a duplicate key, which is valid YAML syntax, so the parser should accept it, but it should be an error when loaded. 
a: 1
a: 2
empty-key:
The test uses an empty node as a key. For example in libyaml/PyYAML this wasn't implemented deliberately, although it was already part of YAML 1.1. And many implementations used or ported from those libraries. Use of empty keys is discouraged and might be removed in the next YAML version. 
key: 1
: 2
explicit-key:
The test uses '?' to mark a key as explicit. 
---
? |
  some multiline
  key
: value
---
? a: mapping
: as a key
flow:
The test uses flow style collections. 
---
mapping: { a: 1, b: 2 }
sequence: [ a, b, c ]
local-tag:
The test uses a local tag with one exclamation mark. This is accepted by most parsers, but some libraries cannot load it into an object, or have no option to ignore such tags. 
---
some: !local tag
unknown-tag:
The test uses a tag with the standard namespace '!!', but it is not part of the YAML 1.2 Core schema. A library should ideally be able to optionally ignore them or abort loading. Parsing should pass without an error. 
---
some: !!nonsense tag