Skip to content

26th Week of 2023

Coding

Languages

Promql

  • New: Run operation only on the elements that match a condition.

    Imagine we want to run the zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes operation only on the elements that match zfs_dataset_used_by_dataset_bytes > 200e3. You can do this with and:

    zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes and zfs_dataset_used_by_dataset_bytes > 200e3
    
  • New: Substracting two metrics.

    To run binary operators between vectors you need them to match. Basically it means that it will only do the operation on the elements that have the same labels. Sometimes you want to do operations on metrics that don't have the same labels. In those cases you can use the on operator. Imagine that we want to substract the next vectors:

    zfs_dataset_used_bytes{type='filesystem'}
    

    And

    sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'})
    

    That only have in common the labels hostname and filesystem`.

    You can use the next expression then:

    zfs_dataset_used_bytes{type='filesystem'} - on (hostname, filesystem) sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'})
    

    To learn more on Vector matching read this article

  • New: Ranges only allowed for vector selectors.

    You may need to specify a subquery range such as [1w:1d].

Python Snippets

  • New: Sort the returned paths of glob.

    glob order is arbitrary, but you can sort them yourself.

    If you want sorted by name:

    sorted(glob.glob('*.png'))
    

    sorted by modification time:

    import os
    sorted(glob.glob('*.png'), key=os.path.getmtime)
    

    sorted by size:

    import os
    sorted(glob.glob('*.png'), key=os.path.getsize)
    

DevOps

Infrastructure as Code

Ansible Snippets

  • New: Start and enable a systemd service.

    - name: Start the service
      become: true
      systemd:
        name: zfs_exporter
        enabled: true
        daemon_reload: true
        state: started
    
  • New: Download an decompress a tar.gz.

    - name: Unarchive a file that needs to be downloaded (added in 2.0)
      ansible.builtin.unarchive:
        src: https://example.com/example.zip
        dest: /usr/local/bin
        remote_src: yes
    

    If you want to only extract a file you can use the includes arg

    - name: Download the zfs exporter
      become: true
      ansible.builtin.unarchive:
        src: https://github.com/pdf/zfs_exporter/releases/download/v{{ zfs_exporter_version }}/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64.tar.gz
        dest: /usr/local/bin
        include: zfs_exporter
        remote_src: yes
        mode: 0755
    

    But that snippet sometimes fail, you can alternatively download it locally and copy it:

    - name: Test if zfs_exporter binary exists
      stat:
        path: /usr/local/bin/zfs_exporter
      register: zfs_exporter_binary
    
    - name: Install the zfs exporter
      block:
        - name: Download the zfs exporter
          delegate_to: localhost
          ansible.builtin.unarchive:
            src: https://github.com/pdf/zfs_exporter/releases/download/v{{ zfs_exporter_version }}/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64.tar.gz
            dest: /tmp/
            remote_src: yes
    
        - name: Upload the zfs exporter to the server
          become: true
          copy:
            src: /tmp/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64/zfs_exporter
            dest: /usr/local/bin
            mode: 0755
      when: not zfs_exporter_binary.stat.exists
    

Continuous Integration

Drone

  • New: Create the administrators.

    When you configure the Drone server you can create the initial administrative account by passing the below environment variable, which defines the account username (e.g. github handle) and admin flag set to true.

    DRONE_USER_CREATE=username:octocat,admin:true
    

    If you need to grant the primary administrative role to an existing user, you can provide an existing username. Drone will update the account and grant administrator role on server restart.

    You can create administrator accounts using the command line tools. Please see the command line tools documentation for installation instructions.

    Create a new administrator account:

    $ drone user add octocat --admin
    

    Or grant the administrator role to existing accounts:

    $ drone user update octocat --admin
    
  • New: Linter: untrusted repositories cannot mount host volumes.

    Thats because the repository is not trusted.

    You have to set the trust as an admin of drone through the GUI or through the CLI with

    drone repo update --trusted <your/repo>
    

    If you're not an admin the above command returns a success but you'll see that the trust has not changed if you run

    drone repo info <your/repo>
    

Storage

OpenZFS

  • New: See how much space do your snapshots consume.

    When a snapshot is created, its space is initially shared between the snapshot and the file system, and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot, and thus is counted in the snapshot’s used property.

    Additionally, deleting snapshots can increase the amount of space that is unique for use by other snapshots.

    Note: The value for a snapshot’s space referenced property is the same as that for the file system when the snapshot was created.

    You can display the amount of space that is consumed by snapshots and descendant file systems by using the zfs list -o space command.

    NAME                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
    rpool                            10.2G  5.16G         0   4.52M              0      5.15G
    rpool/ROOT                       10.2G  3.06G         0     31K              0      3.06G
    rpool/ROOT/solaris               10.2G  3.06G     55.0M   2.78G              0       224M
    rpool/ROOT/solaris@install           -  55.0M         -       -              -          -
    rpool/ROOT/solaris/var           10.2G   224M     2.51M    221M              0          0
    rpool/ROOT/solaris/var@install       -  2.51M         -       -              -          -
    

    From this output, you can see the amount of space that is:

    • AVAIL: The amount of space available to the dataset and all its children, assuming that there is no other activity in the pool.
    • USED: The amount of space consumed by this dataset and all its descendants. This is the value that is checked against this dataset's quota and reservation. The space used does not include this dataset's reservation, but does take into account the reservations of any descendants datasets.

      The used space of a snapshot is the space referenced exclusively by this snapshot. If this snapshot is destroyed, the amount of used space will be freed. Space that is shared by multiple snapshots isn't accounted for in this metric. * USEDSNAP: Space being consumed by snapshots of each data set * USEDDS: Space being used by the dataset itself * USEDREFRESERV: Space being used by a refreservation set on the dataset that would be freed if it was removed. * USEDCHILD: Space being used by the children of this dataset.

    Other space properties are:

    • LUSED: The amount of space that is "logically" consumed by this dataset and all its descendents. It ignores the effect of compression and copies properties, giving a quantity closer to the amount of data that aplication ssee. However it does include space consumed by metadata.
    • REFER: The amount of data that is accessible by this dataset, which may or may not be shared with other dataserts in the pool. When a snapshot or clone is created, it initially references the same amount of space as the filesystem or snapshot it was created from, since its contents are identical.

ZFS Prometheus exporter

  • New: Introduce the ZFS exporter.

    You can use a zfs exporter to create alerts on your ZFS pools, filesystems, snapshots and volumes.

    It's not easy to match the exporter metrics with the output of zfs list -o space. Here is a correlation table:

    • USED: zfs_dataset_used_bytes{type="filesystem"}
    • AVAIL: zfs_dataset_available_bytes{type="filesystem"}
    • LUSED: zfs_dataset_logical_used_bytes{type="filesystem"}
    • USEDDS: zfs_dataset_used_by_dataset_bytes="filesystem"}
    • USEDSNAP: Currently there is no published metric to get this data. You can either use zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes which will show wrong data if the dataset has children or try to do sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'}) which returns smaller sizes than expected.

    It also covers the installation as well as some nice alerts.

Monitoring

Blackbox Exporter

  • New: Check TCP with TLS.

    If you want to test for example if an LDAP is serving the correct certificate on the port 636 you can use:

    tcp_ssl_connect:
      prober: tcp
      timeout: 10s
      tls: true
    
    - name: Ldap
      url: my-ldap-server:636
      module: tcp_ssl_connect
    

Operating Systems

Linux

Linux Snippets

  • New: Reset failed systemd services.

    Use systemctl to remove the failed status. To reset all units with failed status:

    systemctl reset-failed
    

    or just your specific unit:

    bash systemctl reset-failed openvpn-server@intranert.service