26th Week of 2023

Coding⚑

Languages⚑

Promql ⚑

New: Run operation only on the elements that match a condition.

Imagine we want to run the zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes operation only on the elements that match zfs_dataset_used_by_dataset_bytes > 200e3. You can do this with and:
```
zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes and zfs_dataset_used_by_dataset_bytes > 200e3
```
New: Substracting two metrics.

To run binary operators between vectors you need them to match. Basically it means that it will only do the operation on the elements that have the same labels. Sometimes you want to do operations on metrics that don't have the same labels. In those cases you can use the on operator. Imagine that we want to substract the next vectors:
```
zfs_dataset_used_bytes{type='filesystem'}
```
And
```
sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'})
```
That only have in common the labels hostname and filesystem`.

You can use the next expression then:
```
zfs_dataset_used_bytes{type='filesystem'} - on (hostname, filesystem) sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'})
```
To learn more on Vector matching read this article
New: Ranges only allowed for vector selectors.

You may need to specify a subquery range such as [1w:1d].

Python Snippets ⚑

New: Sort the returned paths of glob.

glob order is arbitrary, but you can sort them yourself.

If you want sorted by name:
```
sorted(glob.glob('*.png'))
```
sorted by modification time:
```
import os
sorted(glob.glob('*.png'), key=os.path.getmtime)
```
sorted by size:
```
import os
sorted(glob.glob('*.png'), key=os.path.getsize)
```

DevOps⚑

Infrastructure as Code⚑

Ansible Snippets ⚑

New: Start and enable a systemd service.

- name: Start the service
  become: true
  systemd:
    name: zfs_exporter
    enabled: true
    daemon_reload: true
    state: started

New: Download an decompress a tar.gz.

- name: Unarchive a file that needs to be downloaded (added in 2.0)
  ansible.builtin.unarchive:
    src: https://example.com/example.zip
    dest: /usr/local/bin
    remote_src: yes

If you want to only extract a file you can use the includes arg

- name: Download the zfs exporter
  become: true
  ansible.builtin.unarchive:
    src: https://github.com/pdf/zfs_exporter/releases/download/v{{ zfs_exporter_version }}/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64.tar.gz
    dest: /usr/local/bin
    include: zfs_exporter
    remote_src: yes
    mode: 0755

But that snippet sometimes fail, you can alternatively download it locally and copy it:

- name: Test if zfs_exporter binary exists
  stat:
    path: /usr/local/bin/zfs_exporter
  register: zfs_exporter_binary

- name: Install the zfs exporter
  block:
    - name: Download the zfs exporter
      delegate_to: localhost
      ansible.builtin.unarchive:
        src: https://github.com/pdf/zfs_exporter/releases/download/v{{ zfs_exporter_version }}/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64.tar.gz
        dest: /tmp/
        remote_src: yes

    - name: Upload the zfs exporter to the server
      become: true
      copy:
        src: /tmp/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64/zfs_exporter
        dest: /usr/local/bin
        mode: 0755
  when: not zfs_exporter_binary.stat.exists

Continuous Integration⚑

Drone ⚑

New: Create the administrators.

When you configure the Drone server you can create the initial administrative account by passing the below environment variable, which defines the account username (e.g. github handle) and admin flag set to true.
```
DRONE_USER_CREATE=username:octocat,admin:true
```
If you need to grant the primary administrative role to an existing user, you can provide an existing username. Drone will update the account and grant administrator role on server restart.

You can create administrator accounts using the command line tools. Please see the command line tools documentation for installation instructions.

Create a new administrator account:
```
$ drone user add octocat --admin
```
Or grant the administrator role to existing accounts:
```
$ drone user update octocat --admin
```
New: Linter: untrusted repositories cannot mount host volumes.

Thats because the repository is not trusted.

You have to set the trust as an admin of drone through the GUI or through the CLI with
```
drone repo update --trusted <your/repo>
```
If you're not an admin the above command returns a success but you'll see that the trust has not changed if you run
```
drone repo info <your/repo>
```

Storage⚑

OpenZFS ⚑

New: See how much space do your snapshots consume.

When a snapshot is created, its space is initially shared between the snapshot and the file system, and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot, and thus is counted in the snapshot’s used property.

Additionally, deleting snapshots can increase the amount of space that is unique for use by other snapshots.

Note: The value for a snapshot’s space referenced property is the same as that for the file system when the snapshot was created.

You can display the amount of space that is consumed by snapshots and descendant file systems by using the zfs list -o space command.
```
NAME                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool                            10.2G  5.16G         0   4.52M              0      5.15G
rpool/ROOT                       10.2G  3.06G         0     31K              0      3.06G
rpool/ROOT/solaris               10.2G  3.06G     55.0M   2.78G              0       224M
rpool/ROOT/solaris@install           -  55.0M         -       -              -          -
rpool/ROOT/solaris/var           10.2G   224M     2.51M    221M              0          0
rpool/ROOT/solaris/var@install       -  2.51M         -       -              -          -
```
From this output, you can see the amount of space that is:
- AVAIL: The amount of space available to the dataset and all its children, assuming that there is no other activity in the pool.
- USED: The amount of space consumed by this dataset and all its descendants. This is the value that is checked against this dataset's quota and reservation. The space used does not include this dataset's reservation, but does take into account the reservations of any descendants datasets.
  
  The used space of a snapshot is the space referenced exclusively by this snapshot. If this snapshot is destroyed, the amount of used space will be freed. Space that is shared by multiple snapshots isn't accounted for in this metric. * USEDSNAP: Space being consumed by snapshots of each data set * USEDDS: Space being used by the dataset itself * USEDREFRESERV: Space being used by a refreservation set on the dataset that would be freed if it was removed. * USEDCHILD: Space being used by the children of this dataset.
Other space properties are:
- LUSED: The amount of space that is "logically" consumed by this dataset and all its descendents. It ignores the effect of compression and copies properties, giving a quantity closer to the amount of data that aplication ssee. However it does include space consumed by metadata.
- REFER: The amount of data that is accessible by this dataset, which may or may not be shared with other dataserts in the pool. When a snapshot or clone is created, it initially references the same amount of space as the filesystem or snapshot it was created from, since its contents are identical.

ZFS Prometheus exporter ⚑

New: Introduce the ZFS exporter.

You can use a zfs exporter to create alerts on your ZFS pools, filesystems, snapshots and volumes.

It's not easy to match the exporter metrics with the output of zfs list -o space. Here is a correlation table:
- USED: zfs_dataset_used_bytes{type="filesystem"}
- AVAIL: zfs_dataset_available_bytes{type="filesystem"}
- LUSED: zfs_dataset_logical_used_bytes{type="filesystem"}
- USEDDS: zfs_dataset_used_by_dataset_bytes="filesystem"}
- USEDSNAP: Currently there is no published metric to get this data. You can either use zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes which will show wrong data if the dataset has children or try to do sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'}) which returns smaller sizes than expected.
It also covers the installation as well as some nice alerts.

Monitoring⚑

Blackbox Exporter ⚑

New: Check TCP with TLS.

If you want to test for example if an LDAP is serving the correct certificate on the port 636 you can use:

tcp_ssl_connect:
  prober: tcp
  timeout: 10s
  tls: true

- name: Ldap
  url: my-ldap-server:636
  module: tcp_ssl_connect

Operating Systems⚑

Linux⚑

Linux Snippets ⚑

New: Reset failed systemd services.

Use systemctl to remove the failed status. To reset all units with failed status:
```
systemctl reset-failed
```
or just your specific unit:

bash systemctl reset-failed openvpn-server@intranert.service