yamlfix

A simple opinionated yaml formatter that keeps your comments!

Installing¶

pip install yamlfix

Usage¶

Imagine we've got the following source code:

book_library:
- title: Why we sleep
  author: Matthew Walker
- title: Harry Potter and the Methods of Rationality
  author: Eliezer Yudkowsky

It has the following errors:

There is no --- at the top.
The indentation is all wrong.

After running yamlfix the resulting source code will be:

---
book_library:
  - title: Why we sleep
    author: Matthew Walker
  - title: Harry Potter and the Methods of Rationality
    author: Eliezer Yudkowsky

yamlfix can be used both as command line tool and as a library.

To fix individual files:

yamlfix file.yaml file2.yml

Apply recursively:

yamlfix .

As a library:

from yamlfix import fix_files

fix_files(["file.py"])

If instead of reading from a file you want to fix the code saved into a variable, use fix_code:

from yamlfix import fix_code

source_code = "a: 1"

fixed_code = fix_code(source_code)

assert fixed_code == "---\na: 1\n"

Features¶

yamlfix will do the following changes in your yaml source code per default:

Add the header --- to your file.
Correct truthy strings: 'True' -> true, 'no' -> 'false'
Remove unnecessary apostrophes: title: 'Why we sleep' -> title: Why we sleep.
Correct comments
Ensure that there is exactly one newline at the end of each file, to comply with the POSIX standard.
Split long lines.
Respect Jinja2 syntax.
Convert short lists to flow-style list: [item, item]
Convert lists longer than line-width to block-style:
```
list:
  - item
  - item
```

Configuration¶

yamlfix uses the maison library to find and parse configuration from standard locations, and can additionally be configured through environment variables.

Any configuration found in the YamlfixConfig class can be set through your projects pyproject.toml, a custom toml-file or through the environment by providing an environment variable like {yamlfix_env_prefix}_{yamlfix_config_key}.

Configuration options that are provided through environment variables have higher priority than options provided through configuration files and will override those keys.

All provided configuration options, be it through pyproject.toml, config-files or env-vars, will be parsed by pydantic, so the target value type (str, bool, int, etc.) will be enforced, even if the provided value has the wrong type (for example all env vars in linux systems are strings, but pydantic will parse them to bools/numbers where necessary).

Auto-Configure through `pyproject.toml`¶

The maison library will automatically pick up your yamlfix configuration through your projects pyproject.toml. It will look in the section named tool.yamlfix and apply the provided configuration options. For example:

# pyproject.toml

[tool.yamlfix]
allow_duplicate_keys = true
line_length = 80
none_representation = "null"

Provide config-files¶

When running yamlfix as a standalone cli application it might be desireable to provide a config file containing just the configuration related to yamlfix. A cli-argument -c (--config-file) can be provided multiple times to read configuration values from toml formatted files. The rightmost value-files override the value-files preceding them, only trumped by environment variables. No section headers are necessary for these configuration files, as the expected behaviour is, that those files contain only configuration related to yamlfix. For example:

# run yamlfix with two config files
yamlfix -c base.toml --config-file environment.toml file.yaml

# base.toml
allow_duplicate_keys = false
line_length = 100

# environment.toml
allow_duplicate_keys = true

These provided configuration files would result in a merged runtime-configuration of:

# merged configuration
allow_duplicate_keys = true
line_length = 100

Configure environment prefix¶

Per default yamlfix, when run through cli, will read any environment variable that starts with YAMLFIX_ and apply it to the merged runtime-configuration object. This default value can be overridden with the cli-parameter --env-prefix. For example:

# set a configuration value with the default prefix
export YAMLFIX_LINE_LENGTH="300"

# set a configuration value with the custom prefix
export MY_PREFIX_NONE_REPRESENTATION="~"

# run yamlfix with a custom environment prefix
yamlfix --env-prefix "MY_PREFIX" file.yaml

These provided arguments and environment variables would result in a merged runtime-configuration of:

# merged configuration
# default value for line_length stays at: 80
none_representation = "~"

Configure include and exclude files¶

Per default yamlfix, when run through cli, will include all *.yaml and *.yml files from the directories passed via the CLI. With --exclude <glob> and --include <glob> you can include or exclude specific files within those directories.

Configuration Options¶

All fields configured in the YamlfixConfig class can be provided through the means mentioned in Configuration. Here are the currently available configuration options with short examples on their impact to provided yaml-files.

Allow Duplicate Keys¶

Default: allow_duplicate_keys: bool = False
Environment variable override:

export YAMLFIX_ALLOW_DUPLICATE_KEYS="true"

This option toggles the ruyaml duplicate keys check. With this setting set to False, yamlfix will throw an error if the same key is defined more than once in a mapping / dictionary. To allow using the same key, set this value to True. You might want to enable this option, if you want to use multiple yaml-anchor merge keys, instead of providing them as sequence / list elements - see: https://github.com/pycontribs/ruyaml/issues/43

Comments Min Spaces From Content¶

Default: comments_min_spaces_from_content: int = 2
Environment variable override:

export YAMLFIX_COMMENTS_MIN_SPACES_FROM_CONTENT="2"

This option enforces minimum spacing between the content of a line and the start of an inline-comment. It is the enforcement implementation to the yamllint rule rules.comments.min-spaces-from-content - see: https://yamllint.readthedocs.io/en/stable/rules.html#module-yamllint.rules.comments

Comments Require Starting Space¶

Default: comments_require_starting_space: bool = True
Environment variable override:

export YAMLFIX_COMMENTS_REQUIRE_STARTING_SPACE="true"

This option enforces a space between the comment indicator (#) and the first character in the comment. It implements the enforcement of the yamllint rule rules.comments.require-starting-space - see: https://yamllint.readthedocs.io/en/stable/rules.html#module-yamllint.rules.comments

Whitelines Adjusting¶

Default: whitelines: int = 0
Environment variable override:

export YAMLFIX_WHITELINES="0"

This option allows to keep a speficied number of whitelines between lines.

It's useful if, for one, you like to separate GitHub Actions job steps or Docker-Compose service definitions with a blank line.

Bear in mind that, like Comments Whitelines, it won't insert whitelines if there's no whitelines in the first place. It will only fix 1 or more whitelines to the desired amount (or remove them completely by default).

Comments Whitelines¶

Default: comments_whitelines: int = 1
Environment variable override:

export YAMLFIX_COMMENTS_WHITELINES="1"

This option allows to add a specific number of consecutive whitelines before a comment-only line.

A comment-only line is defined as a line that starts with a comment or with an indented comment.

Before a comment-only line, either:

0 whiteline is allowed
Exactly comments_whitelines whitelines are allowed

Section Whitelines¶

Default section_whitelines: int = 0
Environment variable override:

export YAMLFIX_SECTION_WHITELINES="1"

This option sets the number of whitelines before and after a section. A section is defined as a top-level key followed by at least one line with indentation. Section examples:

section1:
  key: value

list:
  - value

There are a few exceptions to this behaviour:

If there is a comment(s) on the line(s) preceding beginning of a section, comments_whitelines rules will be applied to whitelines before the section. e.g.
```
# Comment
section:
  key: value
```
Sections at the start and end of the document will have whitelines removed before and after respectively.
If whitelines parameter is higher than section_whitelines, the rule will preserve the amount of whitespaces specified by whitelines parameter.

Config Path¶

Default: config_path: Optional[str] = None
Environment variable override:

export YAMLFIX_CONFIG_PATH="/etc/yamlfix/"

Configure the base config-path that maison will look for a pyproject.toml configuration file. This path is traversed upwards until such a file is found.

Explicit Document Start¶

Default: explicit_start: bool = True
Environment variable override:

export YAMLFIX_EXPLICIT_START="true"

Add or remove the explicit document start (---) for yaml-files. For example:

Set to true:

---
project_name: yamlfix

Set to false:

project_name: yamlfix

Sequence (List) Style¶

Default: sequence_style: YamlNodeStyle = YamlNodeStyle.FLOW_STYLE
Environment variable override:

export YAMLFIX_SEQUENCE_STYLE="flow_style"

Available values: flow_style, block_style, keep_style

Transform sequences (lists) to either flow-style, block-style or leave them as-is. If enabled yamlfix will also ensure, that flow-style lists are automatically converted to block-style if the resulting key+list elements would breach the line-length. For example:

Set to true (flow-style):

---
list: [item, item, item]

Set to false (block-style):

---
list:
  - item
  - item
  - item

Indentation¶

Default: indent_mapping: int = 2 indent_offset: int = 2 indent_sequence: int = 4 Environment variable override:

export YAMLFIX_INDENT_MAPPING="2"
export YAMLFIX_INDENT_OFFSET="2"
export YAMLFIX_INDENT_SEQUENCE="4"

Provide the ruyaml configuration for indentation of mappings (dicts) and sequences (lists) and the indentation offset for elements. See the ruyaml configuration documentation: https://ruyaml.readthedocs.io/en/latest/detail.html#indentation-of-block-sequences

Line Length (Width)¶

Default: line_length: int = 80
Environment variable override:

export YAMLFIX_LINE_LENGTH="80"

Configure the line-length / width configuration for ruyaml. With this configuration long multiline-strings will be wrapped at that point and flow-style lists will be converted to block-style if they are longer than the provided value.

`None` Representation¶

Default: none_representation: str = ""
Environment variable override:

export YAMLFIX_LINE_LENGTH=""

In yaml-files an absence of a value can be described in multiple canonical ways. This configuration enforces a user-defined representation for None values. For example:

Valid None representation values are (empty string), null, Null, NULL, ~.

Provided the source yaml file looks like this:

none_value1:
none_value2: null
none_value3: Null
none_value4: NULL
none_value5: ~

The default behaviour (empty string) representation would look like this:

none_value1:
none_value2:
none_value3:
none_value4:
none_value5:

With this option set to none_representation="null" it would look like this:

none_value1: null
none_value2: null
none_value3: null
none_value4: null
none_value5: null

Quote Basic Values¶

Default: quote_basic_values: bool = False
Environment variable override:

export YAMLFIX_quote_basic_values="false"

Per default ruyaml will quote only values where it is necessary to explicitly define the type of a value. This is the case for numbers and boolean values for example. If your yaml-file contains a value of type string that would look like a number, then this value needs to be quoted.

This option allows for quoting of all simple values in mappings (dicts) and sequences (lists) to enable a homogeneous look and feel for string lists / simple key/value mappings. For example:

# option set to false
stringKey1: abc
stringKey2: "123"
stringList: [abc, "123"]

# option set to true
stringKey1: "abc"
stringKey2: "123"
stringList: ["abc", "123"]

Quote Keys and Basic Values¶

Default: quote_keys_and_basic_values: bool = False
Environment variable override:

export YAMLFIX_quote_keys_and_basic_values="false"

Similar to the quote basic values configuration option, this option, in addition to the values themselves, quotes the keys as well. For example:

# option set to false
key: value
list: [item, item]

# option set to true
"key": "value"
"list": ["item", "item"]

Quote Representation¶

Default: quote_representation: str = "'"
Environment variable override:

export YAMLFIX_quote_representation="'"

Configure which quotation string is used for quoting values. For example:

# Option set to: '
key: 'value'

# Option set to: "
key: "value"

Preserve Quotes¶

Default: preserve_quotes: bool = False
Environment variable override:

export YAMLFIX_preserve_quotes="false"

Keep quotes as they are

# original
key: [value, 'value with spaces']

# Option set to: false
key: [value, value with spaces]

# Option set to: true
key: [value, 'value with spaces']

Notes:

You may NOT want to use it in combination with quote basic values or quote keys and basic values, because the output may not be uniform

Example: quote_basic_values

# original
key: [value, 'value with spaces']

# preserve_quotes = true
# quote_basic_values = true
# quote_representation = "
key: ["value", 'value with spaces']

Example: quote_keys_and_basic_values

# original
key: [value, 'value with spaces']

# preserve_quotes = true
# quote_keys_and_basic_values = true
# quote_representation = "
"key": ["value", 'value with spaces']

References¶

As most open sourced programs, yamlfix is standing on the shoulders of giants, namely:

yamlfmt : Inspiration and alternative of this program. I created a new one because the pace of their pull requests is really slow, they don't have tests, CI pipelines or documentation.

ruyaml : A git based community maintained for of ruamel yaml parser.

Click : Used to create the command line interface.

maison : Used for finding, reading and parsing the configuration options.

Pytest : Testing framework, enhanced by the awesome pytest-cases library that made the parametrization of the tests a lovely experience.

Mypy : Python static type checker.

Flakeheaven : Python linter with lots of checks.

Black : Python formatter to keep a nice style without effort.

Autoimport : Python formatter to automatically fix wrong import statements.

isort : Python formatter to order the import statements.

PDM : Command line tool to manage the dependencies.

Mkdocs : To build this documentation site, with the Material theme.

Safety : To check the installed dependencies for known security vulnerabilities.

Bandit : To finds common security issues in Python code.

Contributing¶

For guidance on setting up a development environment, and how to make a contribution to yamlfix, see Contributing to yamlfix.

Donations¶

or

If you are using some of my open-source tools, have enjoyed them, and want to say "thanks", this is a very strong way to do it.

If your product/company depends on these tools, you can sponsor me to ensure I keep happily maintaining them.

If these tools are helping you save money, time, effort, or frustrations; or they are helping you make money, be more productive, efficient, secure, enjoy a bit more your work, or get your product ready faster, this is a great way to show your appreciation. Thanks for that!

And by sponsoring me, you are helping make these tools, that already help you, sustainable and healthy.