The Blue Book

2025

Lyz — 2026-01-01 07:00:46+00:00

Militancy

New: Add 38c3 talk on state of surveillance.

38C3 Talk: State of Surveillance: A year of digital threats to civil society
New: Anticolonialism in technology.

Art - Decolonizing technology

Articles - Shanzhai: An Opportunity to Decolonize Technology? by Sherry Liao - Técnicas autoritarias y técnicas democráticas by Lewis Mumford (pdf)

Books
Research
- Decolonization, Technology, and Online Research by Janet Salmons
Talks
- re:publica 2024: Christoph Hassler - Decolonize Tech (video)
New: Añadir poema sobre la falta de acción.

Maravilloso poema de alemán Martin Niemöller sobre la cobardía de los intelectuales alemanes tras el ascenso de los nazis al poder y la subsiguiente purga de sus objetivos escogidos, grupo tras grupo.
New: Apps para no saltarte el boycott.
- No Thanks
New: Oslo advertisement on the importance of the symbols.

link
New: Añadir pantube.

Pantube: YouTube de izquierdas
New: Add rich don't work representation.

The rich don't work: game to get a grasp of how much money they make without doing anything
New: Añadir podcast sobre la Operación Gladio.

No es el fin del mundo: 46. ¿Qué fue la Operación Gladio?: Para pasárselo a cualquier persona reformista
New: Añadir vídeos de David zona zero.

David zona zero y los distintos barrios

Antifascism

New: Añadir un par de artículos interestantes.
- Hitler Was Incompetent and Lazy—and His Nazi Government Was an Absolute Clown Show
- Bellincat Open-Source Extremist Identifications
New: Añadir web sistemapunk.
- Sistemapunk: hace investigaciones sobre grupos nazis

Hacktivism

Mobile Verification Toolkit

New: Chaos feminist convention.

Chaos feminist convention (looks like it will be mostly in german)
New: Add 38C3 talk from pegasus to predator.

38C3 talk: From Pegasus to Predator - The evolution of Commercial Spyware on iOS
New: Other tools for similar purpose.
- Rayhunter

Collectives

New: Remember these useful things.

Use c3lingo for german talks

There are many talks in german, this may deter you from attending them, but don't worry, if they are hosted in the main tracks you can hear the translation in english live.

You can access them here, they work best with the ccc wifi.

Chaos mentors

At Chaos Mentors, they connect experienced mentors with first-time visitors of Congress. Their goal is to make Congress more inclusive by supporting those who might not usually attend, including people with special needs.
New: First time angel tips.

If you're lost on what shifts you might want to take here is the ones I've found easy to do: - Talk access shifts: You will be on the access of the talks making sure that the doors are opened and closed when it should and that people do not linger in the corridors, whenever you want to see a talk in one of the main tracks take these shifts as you'll be able to watch it fine and also help the organisation. If you know for sure you'll be attending one of these talks, reserve the shift the day before as they usually fly away soon. - Standby angel: You just stay in heaven for a couple of hours waiting for someone to tell you what you need to do. It's a good shift to know people and to find your place in heaven. If you know for sure you'll have a free slot between talks and workshops, reserve the shift the day before as they usually fly away soon. - At teardown: If you can stay after the congress has ended, help in the tear down, I first go to the assemblies that have attracted me the most and then go to heaven to see what else is needed.
New: Add critical switch.

Critical Switch: una colectiva transhackfeminista no mixta1 interesades en la cultura libre, la privacidad y la seguridad digital. Promovemos la cultura de la seguridad para generar espacios más seguros en los movimientos sociales y activistas.
New: Add méxico collectives.
- Sursiendo
- Tecnoafecciones

Anticapacitismo

New: Añadir proyecto artefactos.

https://www.artefactos.org: son gente de tecnología y familias de críos con distintas necesidades, colaborando juntos. Hacen prototipos en 3D que te puedes imprimir

Palestine

New: Add Al Jazeera documenaries.

A la luz de los acontecimientos actuales en Palestina, un gran número de cineastas han puesto sus películas sobre Palestina a disposición en línea de forma gratuita.

Están en árabe y no tienen subtítulos así que no se puede bajar, pero si se pueden ver directamente en youtube con los subtítulos autogenerados.

Detección de estupas

New: Malditos estupas.

Los estupas (o infiltrados) son de los peores cuerpos de represión del estado. No sólo porque extraen información de los colectivos sino porque generan un ambiente de desconfianza y miedo que muchas veces es paralizante o incluso desarticula los propios colectivos. De la peor escoria de la sociedad...

Por desgracia, en el estado español está bastante de moda. Por suerte no estamos desamparadas, varias compañeras están generando materiales para familiarizarnos con este problema. Desde el documental Infiltrats hasta el Manual para destapar a un infiltrado (puedes ver el artículo del salto sobre el manual).

Probablemente estos desgraciados adaptarán sus modos de hacer para pasar estas detecciones, pero al menos los movimientos sociales ya tenemos una base formal sobre la que trabajar.

Referencias
- Manual para destapar a un infiltrado

Conflicto

New: Añadir notas sobre el conflicto desde un punto de vista antipunitivista.

Pensamientos sueltos sobre la visión del conflicto desde un punto de vista antipunitivista
- Dejar de ver los conflictos como una batalla, es una oportunidad de transformación
- Los conflictos deben de ser resueltos en colectivo siempre que se pueda
- Si se veta a un pavo por comportamientos machistas sólo estás trasladando el problema. Seguirá pululando por diferentes colectivos hasta que arraigue en uno más débil y lo torpedeará
- Es difícil de ver el límite entre lo terapéutico y lo transformativo
- Cuál es la responsabilidad colectiva de la transformación de una persona?
- Nos faltan herramientas para:
- la gestión de conflictos en general
- la gestión de conflictos físicos en particular
- el acompañamiento a ambas partes de un conflicto
- el acompañamiento a una persona agresora
- es todo lo que me produce malestar violencia?
- Cada situación es tan particular que los protocolos no sirven. Es mucho mejor someternos en colectivo y a menudo a situaciones de conflicto y generar desde esa práctica las herramientas que nos puedan servir, de manera que en el momento de la verdad salgan de manera intuitiva.
Referencias

Películas
- Ellas hablan (trailer)
- How to Have Sex (trailer)
- Promising young woman (trailer)
Libros
Series
- La Fièvre
Podcast
Artículos
Otras herramientas
- Guía para la prevención y actuación frente a las violencias patriarcales de la Cinètika.
- Guía Fil a l´agulla per a la gestió de conflictes a les cooperatives (en catalan)

Memoria histórica

Conflicto vasco

New: Añadir referencias sobre el conflicto vasco.
- Tercera temporada de (de eso no se habla): "Se llamaba como" yo es una serie documental sobre la memoria de la niña Begoña Urroz, sobre las cinco décadas de silencio de una familia… Y sobre el ruido que lo rompió.

Laboral

New: Añadir referencia a la novela Tierra de la luz.
- No hay negros en el Tíbet: Episodio 47 - Lucía Asué Mbomío: Presentan la novela "Tierra de la Luz" de Lucía Asué MbomÍo. Un relato que pone el foco en los temporeros del sur que trabajan en los invernaderos en unas condiciones durísimas, y las injusticias que se viven en el campo. Una novela “bajo plásticos”, cargada de emoción, crítica y toques de realismo mágico.

Trabajadoras del hogar

New: Introducir investigación sobre las trabajadoras del hogar.

Nota: para nada soy un experto en este tema, estas son las claves que he ido deduciendo a través de hablar del tema con trabajadoras y gestoras. Así que verificar todo antes de tomarlo por verdad!

Cuidado de personas mayores dependientes

El cuidado de las personas mayores es una movida, especialmente cuando empiezan a ser dependientes. Normalmente necesitan cuidados la mayor parte del tiempo del día. Actualmente existen las siguientes opciones para impartir dichos cuidados:
- La red cercana (normalmente las mujeres de la familia) se encargan de dichos cuidados.
- Parte o todos los cuidados se externalizan ya sea a una residencia de ancianos, centros de día o contratando a trabajadoras que acuden al hogar.
En este mundo podrido donde los servicios públicos están siendo desmantelados, la oferta pública de centros de día o residencias es insuficiente y generalmente en manos de políticas incompetentes (nunca olvidemos las 7291 muertes que pesan sobre los hombros de Ayuso {hija de puta!}).

Esto sumado a que las mujeres de la familia ahora trabajan y la precarización del sector de las empleadas del hogar hace que (sobre todo peña que tiene pasta) recurra a contratar trabajadoras en régimen de interna.

Trabajo del hogar en régimen de interna

En realidad este trabajo es esclavismo encubierto bajo una pátina legal. Aprovechandose de que la profesión está feminizada y generalmente por personas migrantes, se imponen unas condiciones laborales que no cumplen el estatuto de las trabajadoras.

Con un salario que normalmente no supera el mínimo estas trabajadoras:
- Trabajan muchas más horas que las 40 a la semana
- Hacen trabajos por fuera de su contrato como limpiar la casa o servir la comida.
- Se encuentran encerradas en su lugar de trabajo. Incluso cuando "dejan de trabajar" están dentro del control de sus empleadores.
- Al encontrarse solas con las personas a las que cuidan, 24 horas al día, es común encontrarse casos de violencia de género. Muchas cuentan testimonios de atrancar la puerta de su cuarto por la noche.
- Los espacios que se les ceden (habitaciones o cuartos de baño) no se respetan y generalmente son usados por otras personas de la familia cuando lo desean, arrebatándoles incluso su cuarto propio.
- Generalmente tienen unas dos horas al día para librar. Pero generalmente trabajan en barrios muy lejanos de su hogar, con precios y oferta de ocio muy lejana a sus posibilidades, así que normalmente usan esas horas para pasear. En invierno se pone más complicado con el frío y la lluvia.
- Las que libran los fines de semana tienen que pagar una habitación o piso que sólo pueden disfrutar unos pocos días a la semana.
- Tienen que soportar el maltrato y la tiranía propia de las personas mayores que ya empiezan a perder la cabeza. A esas edades se exacerban el clasismo y el racismo a la vez que desaparecen los mecanismos de control propio y filtro. Lo que genera situaciones muy desagradables que muchas veces desembocan en maltrato psicológico y físico.
Y aunque esto es harto conocido por la sociedad, es un modelo que se sigue usando con frecuencia.

Horario de trabajo

En algunos casos las trabajadoras libran el fin de semana, 36 horas consecutivas según la ley, lo que podría ser de sábado a las 9:00 hasta el domingo a las 21:00, además de 2 horas al día (no remuneradas) en los días de entre semana. Esto hace un total de 122 horas trabajadas a la semana, mucho mayor de las 40 horas establecidas.

Y aunque en teoría los empleadores están obligados a implementar un sistema para registrar la jornada de sus trabajadoras se torna difícil en la práctica

La ley además establece que además de las 40 horas semanales se pueden tener 20 horas extras de presencia. Las horas de presencia se pagan a precio de hora normal porque el régimen de empleadas de hogar lo establece así. No es como otros convenios.

La mayor parte de las trabajadoras no conoce que tienen derecho a estas 20 horas adicionales. Esas horas de presencia se pueden reclamar por los últimos 12 meses, las anteriores prescriben. Esto pueden ser unos 18.000 euros a reclamar. Para ello hacen falta pruebas de que la trabajadora está haciendo esa jornada. Una manera de pelearlo es pedir al empleador que justifique qué otras personas tiene contratadas para cuidar a la persona dependiente. Porque la prueba dentro de un domicilio es muy difícil. Si en el contrato no figura ningún horario se puede asumir que es 24h. También se puede preguntar a vecinos o si está empadronada.

Lo que si es claro es que la empresa tiene que definir el horario de trabajo en el contrato, lo que no siempre hacen.

Pernocta

En el contrato tiene que figurar si la trabajadora duerme en el lugar de trabajo.

Es muy difícil de regular las veces que se despiertan en la noche, así que pelear eso es aún complicado. Aunque se van haciendo avances.

Violencia y acoso en el empleo doméstico

Según el Real Decreto 893/2024 a ojos de noticias.juridicas.com:

El abandono del domicilio ante una situación de violencia o acoso sufrida por la persona trabajadora no podrá considerarse dimisión ni podrá ser causa de despido, sin perjuicio de la posibilidad de la persona trabajadora de solicitar la extinción del contrato en virtud del artículo 50 ET y de la solicitud de medidas cautelares en caso de formulación de demandas, de conformidad con la LRJS.

Salario

Lo normal es que se pague el salario mínimo, aunque Cuatro de cada diez ni llegan a eso. Hay que tener en cuenta que el salario mínimo ha sido actualizado desde enero de 2025 a 1383 euros. Es probable que a muchas ni se lo suban.

Si se cuentan las 20 horas de presencia, el salario serían aproximadamente 2000 euros al mes (2094 según la tabla salarial de senda de cuidados de 2024 para un régimen de 6 noches a la semana).

Pelear por sus derechos*

Normalmente aunque les cuentes todos los derechos que tienen, las trabajadoras no quieren reclamar ni ejercer sus derechos porque no se atreven. Por miedo a perder el trabajo u otras represalias.

Datos personales

Las asesorías pueden sacar el número de la seguridad social con un nombre y un DNI. Esto se hace para facilitar los trámites. Pero si no lo sabes puede rayarte.

Forma de pago

El empleador es el encargado de hacer la transferencia al trabajador como una nómina, no como una transferencia regular. Ya que si no el trabajador no obtiene las bonificaciones de tener domiciliada la nómina.

Aunque haya agencias de por medio, estas generalmente hacen de intermediarias y sólo un trabajo de asesoría, por lo tanto el contrato se suele hacer con la familia de la persona que es cuidada, y esta es la que ha de hacer el ingreso a la trabajadora. A no ser que la agencia sea una ETT de empleadas del hogar, que en ese caso es la agencia la que las contrata directamente.

Denuncias de inspección de trabajo

Las denuncias de inspección de trabajo en este régimen tienen un recorrido diferente dependiendo de a qué inspector le toque, porque como inspección de trabajo no puede entrar en los domicilios particulares por sorpresa aunque sea una empresa. Entonces hay inspectores que las denuncias por maltrato de las empleadas del hogar las meten en el cajón. Otros no, requieren y de más.

Impuestos

En las nóminas de las empleadas del hogar no hay IRPF. Normalmente el salario y la prorrata se desglosa, si está junto es una nómina cutre.

Papeleos

Una vez firmado el contrato, la empresa ha de entregar a la trabajadora la huella digital de su contrato comunicado al servicio público de empleo. Cuando haces un alta de una trabajadora hay que mandar dos ficheros, uno a la tesorería general con el alta y otro al servicio público de empleo estatal con el contrato. Si no lo hacen es un defecto formal, no es gravísimo.

Referencias

Empresas decentes: No todo es horrendo, existen cooperativas de trabajadoras que ofrecen estos servicios bajo unas condiciones que ellas han decidido:
- Senda de cuidados
- La Comala
Senda de cuidados publica su tabla salarial en la que te puedes hacer una idea del salario y de los diferentes tipos de régimen de trabajo.

Mejoras legales
- Real Decreto 893/2024, de 10 de septiembre, por el que se regula la protección de la seguridad y la salud en el ámbito del servicio del hogar familiar.
- Real Decreto-ley 16/2022, de 6 de septiembre, para la mejora de las condiciones de trabajo y de Seguridad Social de las personas trabajadoras al servicio del hogar.
Artículos sobre el trabajo interno

Collaborating tools

Aleph

Correction: Correct the releases url.

latest release
New: Add warning for new potential users.

WARNING: Check out the investigative journalism article before using Aleph
New: Compare investigative journalism tools.

After reviewing Aleph Pro, Open Aleph, DARC and Datashare I feel that the investigative reporting software environment, as of June 2025, is very brittle and under a huge crisis that will have a breaking point on October of 2025, day where OCCRP will do the switch from Aleph to Aleph Pro.

Given this scenario, I think the best thing to do, if you already have an Aleph instance, is to keep using it until things stabilise. As you won\'t have software updates since October 2025, I suggest that from then on you protect the service behind an VPN and/or SSL client certificate.

I also feel it\'s also a great moment to make a strategic decision on how you want to use an investigative reporting platform. Some key questions are:
- How much do you care of our data being leaked or lost by third parties?
- How much do you trust OCCRP, DARC or ICIJ?
- Do you want to switch from a self hosted platform to an external managed one? Will they be able to give a better service?
- If you want to stay on the self hosted solution. Shall you migrate to Datashare instead of Open Aleph?
- How dependent are you on open source software? How fragile are the teams that support that software? Can you help change that fragility?
- Shall you use the AI for your investigative processes? If so, where and when?
I hope the analysis below may help shed some light on some of these questions. The only one that is not addressed is the AI as it\'s a more political, philosophical one, that would make the document even longer.

Analysis of the present

Development dependent in US government

The two main software are developed by non-profits, Aleph by OCCRP and Datashare by ICIJ, that received part of their funding from the US government. This funding was lost after Naranjitler\'s administration funding cuts:
- OCCRP lost this year 38% of their operational funds: \"As a result, we had to lay off 40 people --- one fifth of our staff --- and have temporarily reduced some of the salaries of others. But there is more. OCCRP has also been funding a number of organizations across Europe, in some of the most difficult countries. Eighty percent of those sub-grants that we provide to other newsrooms have been cut as well.\"
- ICIJ lost this year 8.6% of their operational funds with no apparent effects on the software or the staff.
OCCRP decided to close the source code of Aleph triggering the team split up

With OCCRP decision to close the source of Aleph an important part of their team (the co-leader of the research and data team, the Chief Data Editor and a developer) decided to leave the project and fund DARC.

Although they look to be in good terms. They\'re collaborating, although it could be OCCRP trying to save their face on this dark turn.

These software have very few developers and no community behind them
- Aleph looks to be currently developed by 2 developers, and it\'s development has stagnated since the main developer (`pudo`) stopped developing and moved to Open Sanctions 4 years ago. 3 key members of their team have moved on to Open Aleph after the split. We can only guess if this is the same team that is developing Aleph Pro. If it\'s not then they are developers we know nothing about and can\'t audit.
- Open Aleph looks to be developed by 4 people, 3 of them were part of the Aleph development team until the break up. The other one created a company to host Aleph instances 4 years ago.
- Datashare seems to be developed by 6 developers.
In all projects pull requests by the community have been very scarce.

The community support is not that great

My experience requesting features, proposing fixes with Aleph before the split is that they answer well on their slack, but are slow on the issues and the pull requests that fall outside their planned roadmap. Even if they are bugs. I\'ve been running a script on each ingest to fix an UI bug for a year already. I tried to work with them in solving it without success.

I don\'t have experience with Datashare, but they do answer and fix the issues the people open.

Analysis of the available software

Aleph Pro

The analysis is based on their announcement and their FAQ.

Pros
- As long as you have less than 1TB of data and are a nonprofit it will, for now, cost you way less than hosting your solution
- OCCRP is behind the project
Cons
- They seem to have an unstable, small and broken development team
- They only offer 1TB of data which is enough for small to medium projects but doesn\'t give much space to grow
- They lost my trust
  
  There are several reasons that make me hesitant to trust them:
  - They don\'t want to publish their source code
  - They decided that the path to solve a complicated financial situation is to close their source code
  - They advocated in the past (and even now!) that being open-sourced was a corner part of the project and yet they close their code.
  - They hid that 52% of their funding came from the US government.
  With the next consequences:
  - I would personally not give them my data or host their software.
  - I wouldn\'t be surprised if in the future they retract on their promises, such as offering Aleph Pro for free forever for nonprofit journalism organizations.
- You loose sovereignty of your data
  
  Either if you upload your data to their servers or host a closed sourced program in yours, you have no way of knowing what are they doing with your data. Given their economical situation, doing business with the data could be an option.
  
  It could also be potentially difficult to extract your data in the future.
- You loose sovereignty of your service
  
  If they host the service you depend on them for any operations such as maintenance, upgrades, and keeping the service up.
  
  You\'ll also no longer be able to change the software to apply patches to problems and would depend on them for their implementation and application.
  
  You\'ll no longer have any easy way to know what does the program do. This is critical from a security point of view as introduced backdoors would go unnoticed. It\'s also worrying as we could not audit how they implement the AI. It is known that AI solutions tend to be biased and may thwart the investigative process.
  
  Finally, if they decide to shutdown or change the conditions you\'re sold.
- I looks like they are selling smoke
  
  Their development activity has dropped in the recent years, they have a weakened team and yet they are promising a complete rewrite to create a brand new software. In an announce that is filled with buzzwords such as AI without giving any solid evidence.
  
  I feel that the whole announcement is written to urge people to buy their product and to save their face. Its not written to the community or their users, is for those that can give them money.
  1. They offer significant performance upgrades and lower infrastructure costs at the same time that they incorporate the latest developments in data-analysis and AI
    
    Depending on how they materialise the data-analysis and AI new features it will mean a small to a great increase in infrastructure costs. Hosting these processes is very resource intensive and expensive.
    
    The decrease in infra costs may come from:
    - Hosting many aleph instances under the same infrastructure is more efficient than each organisation having their own.
    - They might migrate the code to a more efficient language like rust or go
    So even though Aleph Pro will require more resources, as they are all going to be hosted in OCCRP it will be cheaper overall.
    
    I\'m not sure how they want to implement the AI, I see two potential places:
    - To improve the ingest process.
    - To use LLM (like ChatGPT) to query the data.
    Both features are very, very, very expensive resource wise. The only way to give those features at the same time as lowering the infra costs is by outsourcing the AI services. If they do this, it will mean that your data will be processed by that third party, with all the awful consequences it entails.
  2. They are selling existent features as new or are part of other open source projects
    
    Such as:
    - Rebuilt the ingest pipeline: they recently released it in the latest versions of Aleph
    - Modular design: The source is already modular (although it can always be improved)
    - Enhanced data models for better linking, filtering, and insights. Their model is based on followthemoney which is open source.
- They are ditching part of their user base
  
  They only support self-hosted solutions to enterprise license clients. This leaves out small organisations or privacy minded individuals. Even this solution is said to be maintained in partnership with OCCRP.
- The new version benefits may not be worth the costs
  
  They say that Aleph Pro will deliver a faster, smarter, and more flexible platform, combining a modern technical foundation with user-centered design and major performance gains. But if you do not do a heavy use of the service you may not need some of these improvements. Although they for sure would be nice to have.
- It could be unstable for a while
  
  A complete platform rewrite is usually good in the long run but these kind of migrations tend to have an unstable period of time where some of the functionality might be missing
- You need to make the decision blindly
  
  Even though they are going to give a beta if you request it, I\'m not sure of this, before doing the switch, you need to make the decision beforehand. You may not even like the new software
Datashare

Pros
- ICIJ, a more reliable non profit, is behind it.
- Has the biggest and stable development team
- Is the most active project
- Better community support
- You can host it
- It\'s open source
Cons
- If you have an Aleph instance there is no documented way to migrate to Datashare. And there is still not an easy way to do the migration, as they don't yet use the followthemoney data schema.
- They won\'t host an instance for you.
Open Aleph

Pros
- You can host it
- It\'s open source
- The hosted solution will probably cost you way less than hosting your own solution (although they don\'t show prices)
- The people behind it have proven their ethic values
- I know one of their developers. She is a fantastic person which is very involved in putting technology in the service of society, and has been active at the CCC.
- They are actively reaching out to give support with the migration
Cons
- A new small organisation is behind the project
- A small development team with few recent activity. Since their creation (3 months ago) their development pace is slow (the contributors don\'t even load). It could be because they are still setting up the new organisation and doing the fork.
- It may not have all the features Aleph has. They started the fork on November of 2024 and are 137 commits ahead and 510 behind. But they could be squash commits.
- Their community forum doesn\'t have much activity
- The remote hosted solution has the same problems as Aleph Pro in terms of data and service sovereignty. Although I do trust more DARC than OCCRP.

Conference organisation

pretalx

New: Import a pretalx calendar in giggity.

Search the url similar to https://pretalx.com//schedule/export/schedule.xml
New: Add references for pretalx.
- Home
- Source

New: Install.

NOTE: it's probably too much for a small event.

Docker compose

The default docker compose doesn't work as it still uses mysql which was dropped. If you want to use sqlite just remove the database configuration.

---
services:
  pretalx:
    image: pretalx/standalone:v2024.3.0
    container_name: pretalx
    restart: unless-stopped
    depends_on:
      - redis
    environment:
      # Hint: Make sure you serve all requests for the `/static/` and `/media/` paths when debug is False. See [installation](https://docs.pretalx.org/administrator/installation/#step-7-ssl) for more information
      PRETALX_FILESYSTEM_MEDIA: /public/media
      PRETALX_FILESYSTEM_STATIC: /public/static
    ports:
      - "127.0.0.1:80:80"
    volumes:
      - ./conf/pretalx.cfg:/etc/pretalx/pretalx.cfg:ro
      - pretalx-data:/data
      - pretalx-public:/public

  redis:
    image: redis:latest
    container_name: pretalx-redis
    restart: unless-stopped
    volumes:
      - pretalx-redis:/data

volumes:
  pretalx-data:
  pretalx-public:
  pretalx-redis:

I was not able to find the default admin user so I had to create it manually. Get into the docker:

docker exec -it pretalx bash

When you run the commands by default it uses another database file /pretalx/src/data/db.sqlite3, so I removed it and created a symbolic link to the actual place of the database /data/db.sqlite

pretalxuser@82f886a58c57:/$ rm /pretalx/src/data/db.sqlite3
pretalxuser@82f886a58c57:/$ ln -s /data/db.sqlite3 /pretalx/src/data/db.sqlite3

Then you can create the admin user:

python -m pretalx createsuperuser

Org Mode

New: Footnotes.

A footnote is started by a footnote marker in square brackets in column 0, no indentation allowed. It ends at the next footnote definition, headline, or after two consecutive empty lines. The footnote reference is simply the marker in square brackets, inside text. Markers always start with ‘fn:’. For example:
```
The Org website[fn:1] now looks a lot better than it used to.
...
[fn:50] The link is: https://orgmode.org
```
Nvim-orgmode has some basic support for footnotes.

New: Custom agendas.

You an use custom agenda commands

Define custom agenda views that are available through the org_agenda mapping. It is possible to combine multiple agenda types into single view. An example:

require('orgmode').setup({
  org_agenda_files = {'~/org/**/*'},
  org_agenda_custom_commands = {
    -- "c" is the shortcut that will be used in the prompt
    c = {
      description = 'Combined view', -- Description shown in the prompt for the shortcut
      types = {
        {
          type = 'tags_todo', -- Type can be agenda | tags | tags_todo
          match = '+PRIORITY="A"', --Same as providing a "Match:" for tags view <leader>oa + m, See: https://orgmode.org/manual/Matching-tags-and-properties.html
          org_agenda_overriding_header = 'High priority todos',
          org_agenda_todo_ignore_deadlines = 'far', -- Ignore all deadlines that are too far in future (over org_deadline_warning_days). Possible values: all | near | far | past | future
        },
        {
          type = 'agenda',
          org_agenda_overriding_header = 'My daily agenda',
          org_agenda_span = 'day' -- can be any value as org_agenda_span
        },
        {
          type = 'tags',
          match = 'WORK', --Same as providing a "Match:" for tags view <leader>oa + m, See: https://orgmode.org/manual/Matching-tags-and-properties.html
          org_agenda_overriding_header = 'My work todos',
          org_agenda_todo_ignore_scheduled = 'all', -- Ignore all headlines that are scheduled. Possible values: past | future | all
        },
        {
          type = 'agenda',
          org_agenda_overriding_header = 'Whole week overview',
          org_agenda_span = 'week', -- 'week' is default, so it's not necessary here, just an example
          org_agenda_start_on_weekday = 1 -- Start on Monday
          org_agenda_remove_tags = true -- Do not show tags only for this view
        },
     }
    },
    p = {
      description = 'Personal agenda',
      types = {
        {
          type = 'tags_todo',
          org_agenda_overriding_header = 'My personal todos',
          org_agenda_category_filter_preset = 'todos', -- Show only headlines from `todos` category. Same value providad as when pressing `/` in the Agenda view
          org_agenda_sorting_strategy = {'todo-state-up', 'priority-down'} -- See all options available on org_agenda_sorting_strategy
        },
        {
          type = 'agenda',
          org_agenda_overriding_header = 'Personal projects agenda',
          org_agenda_files = {'~/my-projects/**/*'}, -- Can define files outside of the default org_agenda_files
        },
        {
          type = 'tags',
          org_agenda_overriding_header = 'Personal projects notes',
          org_agenda_files = {'~/my-projects/**/*'},
          org_agenda_tag_filter_preset = 'NOTES-REFACTOR' -- Show only headlines with NOTES tag that does not have a REFACTOR tag. Same value providad as when pressing `/` in the Agenda view
        },
      }
    }
  }
})

You can also define the org_agenda_sorting_strategy. The default value is { agenda = {'time-up', 'priority-down', 'category-keep'}, todo = {'priority-down', 'category-keep'}, tags = {'priority-down', 'category-keep'}}.

The available list of sorting strategies to apply to a given view are:

time-up: Sort entries by time of day. Applicable only in agenda view
time-down: Opposite of time-up
priority-down: Sort by priority, from highest to lowest
priority-up: Sort by priority, from lowest to highest
tag-up: Sort by sorted tags string, ascending
tag-down: Sort by sorted tags string, descending
todo-state-up: Sort by todo keyword by position (example: 'TODO, PROGRESS, DONE' has a sort value of 1, 2 and 3), ascending
todo-state-down: Sort by todo keyword, descending
clocked-up: Show clocked in headlines first
clocked-down: Show clocked in headines last
category-up: Sort by category name, ascending
category-down: Sort by category name, descending
category-keep: Keep default category sorting, as it appears in org-agenda-files

You can open the custom agendas with the API too. For example to open the agenda stored under t:

     keys = {
       {
        "gt",
         function()
          vim.notify("Opening today's agenda", vim.log.levels.INFO)
          require("orgmode.api.agenda").open_by_key("t")
        end,
        desc = "Open orgmode agenda for today",
      },
    },

In that case I'm configuring the keys section of the lazyvim plugin. Through the API you can also configure these options:

org_agenda_files
org_agenda_sorting_strategy
org_agenda_category_filter_preset
org_agenda_todo_ignore_deadlines: Ignore all deadlines that are too far in future (over org_deadline_warning_days). Possible values: all | near | far | past | future
org_agenda_todo_ignore_scheduled: Ignore all headlines that are scheduled. Possible values: past | future | all

New: Load different agendas with the same binding depending on the time.

I find it useful to bind gt to Today's agenda, but what today means is different between week days. Imagine that you want to load an agenda if you're from monday to friday before 18:00 (a work agenda) versus a personal agenda the rest of the time.

You could then configure this function:

    keys = {
     {
        "gt",
        function()
          local current_time = os.date("*t")
          local day = current_time.wday -- 1 = Sunday, 2 = Monday, etc.
          local hour = current_time.hour

          local agenda_key = "t"
          local agenda_name = "Today's" -- default

          -- Monday (2) through Friday (6)
          if day >= 2 and day <= 6 then
            if hour < 17 then
              agenda_key = "w"
              agenda_name = "Today + Work"
            end
          end

          vim.notify("Opening " .. agenda_name .. " agenda", vim.log.levels.INFO)
          require("orgmode.api.agenda").open_by_key(agenda_key)
        end,
        desc = "Open orgmode agenda for today",
      },
    }

New: Better handle indentations.

There is something called virtual indents that will prevent you from many indentation headaches. To enable them set the org_startup_indented = true configuration.

If you need to adjust the indentation of your document (for example after enabling the option on existent orgmode code), visually select the lines to correct the indentation (V) and then press =. You can do this with the whole file (╥﹏╥).

New: Remove some tags when the state has changed so DONE.

For example if you want to remove them for recurrent tasks

      local function remove_specific_tags(headline)
        local tagsToRemove = { "t", "w", "m", "q", "y" }
        local currentTags = headline:get_tags()
        local newTags = {}
        local needsUpdate = false

        -- Build new tags list excluding t, w, m
        for _, tag in ipairs(currentTags) do
          local shouldKeep = true
          for _, removeTag in ipairs(tagsToRemove) do
            if tag == removeTag then
              shouldKeep = false
              needsUpdate = true
              break
            end
          end
          if shouldKeep then
            table.insert(newTags, tag)
          end
        end
        -- Only update if we actually removed something
        if needsUpdate then
          headline:set_tags(table.concat(newTags, ":"))
          headline:refresh()
        end
      end

      local EventManager = require("orgmode.events")
      EventManager.listen(EventManager.event.TodoChanged, function(event)
        ---@cast event OrgTodoChangedEvent
        if event.headline then
          if type == "DONE" then
            remove_specific_tags(event.headline)
          end
        end
      end)

New: Register the todo changes in the logbook.

You can now register the changes with events. Add this to your plugin config. If you're using lazyvim:

return {
  {
    "nvim-orgmode/orgmode",
  config = function()
      require("orgmode").setup({...})

      local EventManager = require("orgmode.events")
      local Date = require("orgmode.objects.date")

      EventManager.listen(EventManager.event.TodoChanged, function(event)
        ---@cast event OrgTodoChangedEvent
        if event.headline then
         local current_todo, _, _ = event.headline:get_todo()
          local now = Date.now()

          event.headline:add_note({
            'State "' .. current_todo .. '" from "' .. event.old_todo_state .. '"  [' .. now:to_string() .. "]",
          })
        end
      end)
    end,
  },
}

New: API usage.

Get the headline under the cursor

Read and write files

You have information on how to do it in this pr

Create custom hyperlink types

Custom types can trigger functionality such as opening the terminal and pings the provided URL .

To add your own custom hyperlink type, provide a custom handler to hyperlinks.sources setting. Each handler needs to have a get_name() method that returns a name for the handler. Additionally, follow(link) and autocomplete(link) optional methods are available to open the link and provide the autocompletion. ## Refile a headline to another destination

Refile a headline to another destination

You can do this with the API.

Assuming you are in the filewhere your TODOs are:

local api = require('orgmode.api') local closest_headline = api.current():get_closest_headline() local destination_file = api.load('~/org/journal.org') ocal destination_headline = vim.tbl_filter(function(headline) return headline.title == 'My journal' end, destination_file.headlines)[1]

api.refile({ source = closest_headline, destination = destination_headline })

Use events
New: Introduce the time navigation abstract identity concept.

An identity is the set of qualities, beliefs, personality traits, appearance, and/or expressions that characterize a person or a group.

Identity serves multiple functions, acting as a "self-regulatory structure" that provides meaning, direction, and a sense of self-control. It fosters internal harmony and serves as a behavioral compass, enabling individuals to orient themselves towards the future and establish long-term goals. As an active process, it profoundly influences an individual's capacity to adapt to life events and achieve a state of well-being. However, identity originates from traits or attributes that you may have little or no control over, such as their family background or ethnicity.

Identities then will be the guide of my life. I've tried setting essential goals, answering big questions with no success so far. This approach however looks more interesting because:
- I can split myself in many identities each with it's definition and analyse life through the different lenses, identify identity conflicts, priorize identities...
- I can analyse each identity on it's own, decide how to change my roadmap to integrate the ones I want to adopt and get away from the ones I want to leave behind.
- It reminds me of the RPG character building and although it may seem silly, that motivates me.
- It fits quite well with what I've learnt regarding habit management
The identities archive

I'm using a new notebook page called identities.org where I plan to analyze, build and evolve my identities.

An identity section

Each identity may have the next sections. A simple heading with the name of the identity is just fine. You'll create the sections as long as you need them.

Analysis of the identity

Here we can develop our thoughts on what does the identity mean and what do you see yourself embodying the identity

Identity characteristics

List the values, habits, abilities, knowledge, capabilities and experiences that define the identity, and analyse each of them.

Identity plan

Here is where we can sketch the plan we want to follow to grow or shrink this identity. It contains a list of identity axis.

Children identities

Sometimes an identity can be refined in smaller more specific identities, here we'll add sections for each of them.

Identity analysis

Dump your thoughts on your identities

Before we get tainted with our past analysis imagine a fresh canvas and start painting yourself.

You can add a section in think.org to record your findings. I found that I needed some time to do this dump, working on the section through days before the actual analysis.

Do an initial list of values

I first created a global values or principles headline and listed all core principles such as:
- All creatures are beautiful
- Be excellent to each other
- Better done than perfect
You may refactor them into the identities once you start building them.

Do an initial list of identities

Empty your mind of the different identities that define you or that you want to be defined by. Create headlines for each using the sections defined above as you need them.

Do an initial list of axis

Dream of what axis you'll want to address. If you can order them into identities.

Do the identity analysis

Refactor the gathered thoughts into the identities.org file

Select the identities you want to prioritise in the year
- Skim over all identities and for the ones you want to focus in the year:
- add the identity tag
- assign a priority
- do not add a TODO keyword, we'll reserve them for the identity axis.
- Use the year custom agenda identities section to adjust the priority of the different identities following the next guides:
- Spread the identities over the different priorities so that each has more less the same number of elements.
- Compare an identity with the ones that are above or below and decide if you promote or demote it
- If you don't want one identity to bien the list add the backlog tag if you don't want that identity and it's subidentities to appear. Add the hide tag in case you want any of the subidentities
Refine the identities

Following the priority order of identities go one by one until you run out of time and:
- Read the identity analysis and the different axis.
- Order the different axis by the impact they can have in the identity
- Refine the projects of each axis
Correction: Use unison to sync.
New: Exporting to pdf or markdown.

To pdf

If you want to convert it with python you first need to install the dependencies: sudo apt install texlive-xetex

Then you can do
```
pandoc input.org -o output.pdf --pdf-engine=xelatex -V geometry:margin=1in -V fontsize=11pt -V colorlinks=true
```
To markdown
```
pandoc input.org -o output.md
```

Axis

New: Not adding a todo state when creating a new element by default.

The default state NOTE doesn't add any state.
New: Introduce the axis time navigation concept.

An identity axis is an abstract guide for action with an indefinite scope or a timeframe longer than one year. It serves as a high-level directional tool to materialise changes in your identities, helping to outline a general course without specifying exact destinations.

Limitations of an axis
- It is not suitable as a final destination on a roadmap because it is too ambiguous.
- Even minor progress could be considered sufficient, leading to a lack of clear endpoints.
- One could continue indefinitely without a sense of completion, such as continuously striving for improvement.
Axis orgmode representation

An axis is a headline or TODO headline (depending whether it's active or not) that is part of the plan section of an identity.

It can have none or many of the next sections:

Axis analysis section

To gather the thoughts regarding the study of the axis

Axis plan section

To gather the axis projects that will materialise the direction of the axis.

Axis projects

To prevent endless pursuit, an axis should be broken down into projects with defined scopes that indicate when to stop advancing in that direction.

Axis projects shall:
- Have a scope shorter than 11 months so they can be managed in the life stage review.
Depending on whether the axis projects will be acted upon in the current quarter they will have two possible representations:
- If you don't plan to it will be a project section with the análisis, and steps
- If you do, then it will be a link to the project section either in projects.org or in backlog.org. You can leave the analysis section below the link in identities.org that way it won't use precious space in your projects.org file.

Roadmap Adjustment

New: Small annotations throughout the document.

The idea of each adjustment is that it's not cumbersome, so I've decided to set a fixed time for each one, I assume that I'll do my best on both processes and wherever I reach is just fine (remember, better done than perfect).

I believe on the power of constant small changes, so the next review will be built on top of the current one, and thus it will be done better and faster.
New: Do the first sketch of the year review.

As the year comes to an end it can be a good idea to review stuff that has a year cadence of change and that people are reviewing, for example:
- How the world has changed
- Relevant content stuff
- Life experiences
- Your toolset
- Your economical year review
I'm not going to review the year roadmap or how I've changed or my tactics or strategies right now as I feel it makes more sense to do it in the life review just before spring comes. In fact, we'll be able to do a better life review once we have the output of the review of the year.

Year review timeline

As you can see the amount of stuff to review is not something that can be done in a day, my current plan is to prepare the review of the year throughout December and carry it out on the first two weeks of January.

So I've scheduled an action each 1^st of December with deadline on the 31^st to:
- Create a new entry in think.org with the next format:
```
* Year review (YYYY)
** Review adjustments
** Sections
*** How the world has changed
*** ...
```
Then I created another action from the 1^st of January to the 15^th to actually do the review.

Year review phases

How the world has changed

You don't live alone in a bubble, your life is affected by what is going on around you, so if you can it's always good to analyse it so that you can adjust your roadmap accordingly.

Doing this on January makes a lot of sense because most newspapers and important people do a year review which synthesizes the most important year events and how it has changed throughout the year.

I'm usually gathering the analysis on the Year reviews article.

How did my economy change this year

I take the chance to do a last review of the year through the lenses of my accounting system (beancount). I've made some fava dashboards that gather the most interesting information.

Review the relevant content I've consumed

With the use of mediatracker and other life logging tools I take a look at what content I've enjoyed the most. I want to share it also with all of you through these articles:
- Books
- Movies
- TV shows
- Podcasts
- Streaming channels
- Music
- Videogames
- Boardgames
New: First sketch of the life review.

Life reviews are meant to give you an idea of:
- How much have you and your workflows evolved
- What roadmap decisions were right, which ones were wrong
- With the context you have now, you can think of how you could have avoided the bad decisions.
If you have the year's planning you can analyze it against your task management tools and life logs and create a review document analyzing all.

Life review timeline

As you can see the amount of stuff to review is not something that can be done in a day, my current plan is to prepare the review from the 15^th of December till the 15^th of January and then carry it out until the 23^rd of February, to leave space to do the spring quarter and March month reviews.
New: Adjust the month review process.

To record the results of the review create the section in pages/reviews.org with the following template:
```
* winter
** january review
*** work
*** personal
**** month review
***** mental dump
****** What worries you right now?
****** What drained your energy or brought you down emotionally this last month?
****** What are the little things that burden you or slow you down?
****** What do you desire right now?
****** Where is your mind these days?
****** What did you enjoy most this last month?
****** What did help you most this last month?
****** What things would you want to finish throughout the month so you can carry them to the next?
****** What things do you feel you need to do?
****** What are you most proud of this month?
***** month checks
***** analyze
***** decide
```
I'm assuming it's the january's review and that you have two kinds of reviews, one personal and one for work.

Dump your mind

The first thing we want to do in the review is to dump all that's in our mind into our system to free up mental load.

Try not to, but if you think of decisions you want to make that address the elements you're discovering, write them down in the Decide section of your review document.

There are different paths to discover actionable items:
- Analyze what is in your mind: Take 10 minutes to answer to the questions of the template under the "mental dump" section (you don't need to answer them all). Notice that we do not need to review our life logging tools (diary, action manager, ...) to answer these questions. This means that we're doing an analysis of what is in our minds right now, not throughout the month. It's flawed but as we do this analysis often, it's probably fine. We add more importance to the latest events in our life anyway.
Clean your notebook
- Empty the elements you added to the review box. I have them in my inbox with the tag :review: (you have it in the month agenda view gM)
- Clean your life notebook by:
- Iterate over the areas of proyects.org only checking the first level of projects, don't go deeper and for each element:
  - Move the done elements either to archive.org or logbook.org.
  - Move to backlog.org the elements that don't make sense to be active anymore
- Check if you have any DONE element in calendar.org.
- Empty the inbox.org
- Empty the DONE elements of talk.org
- Clean the elements that don't make sense anymore from think.org
- Process your month checks. For each of them:
- If you need, add action elements in the mental dump section of the review.
- Think of whether you've met the check.
Refresh your idea of how the month go
- Open your bitácora.org agenda view to see what has been completed in the last month match = 'CLOSED>"<-30d>"-work-steps-done', ordered by name org_agenda_sorting_strategy = { "category-keep" }, and change the priority of the elements according to the impact. Open your recurrent.org agenda view to see what has been done the last month match = 'LAST_REPEAT>"<-30d>"-work'
- Check what has been left of your month objectives +m and refile the elements that don't make sense anymore.
- Check the reports of your weekly reviews of the month in the reviews.org document.
Check your close compromises

Check all your action management tools (in my case orgmode and ikhal) to identify: - Arranged compromises - trips
New: Life roadmap adjustment.

Create next stage's life notebook

After reading "The Bulletproof Journal", I was drawn to the idea of changing notebooks each year, carrying over only the necessary things.

I find this to be a powerful concept since you start each stage with a clean canvas. This brings you closer to desire versus duty as it removes the commitments you made to yourself, freeing up significant mental load. From this point, it's much easier to allow yourself to dream about what you want to do in this new stage.

I want to apply this concept to my digital life notebook as I see the following advantages:
- It lightens my files making them easier to manage and faster to process with orgmode
- It's a very easy way to clean up
- It's an elegant way to preserve what you've recorded without it becoming a hindrance
- In each stage, you can start with a different notebook structure, meaning new axes, tools, and structures. This helps avoid falling into the rigidity of a constrained system or artifacts defined by inertia rather than conscious decision
- It allows you to avoid maintaining files that follow an old scheme or having to migrate them to the new system
- Additionally, you get rid of all those actions you've been reluctant to delete in one fell swoop
The notebook change can be done in two phases:
- Notebook Construction
- Stage Closure
Notebook Construction

This phase spans from when you start making stage adjustments until you finally close the current stage. You can follow these steps:
- Create a directory with the name of the new stage. In my case, it's the number of my predominant age during the stage
- Create a directory for the current stage's notebook within "notebooks" in your references. Here we'll move everything that doesn't make sense to maintain. It's important that this directory isn't within your agenda files
- Quickly review the improvements you've noted that you want to implement in next year's notebook to keep them in mind. You can note the references in the "Create new notebook" action
As you review the stage, decide if it makes sense for the file you're viewing to exist as-is in the new notebook. Remember that the idea is to migrate minimal structure and data.
- If it makes sense:
- Create a symbolic link in the new notebook. When closing the stage, we'll replace the link with the file's final state
- If the file no longer makes sense, move it to references/notebooks
New: Life stage roadmap adjustment adjustments.

Review what you've done the last year

Read your logbook.org (o bitácora.org) adjusting the priorities of the áreas, projects and actions thinking of the impact the element has meant in your live.

Review what you've learn the last year

It's always interesting to look back and see what you've learned throughout the year. I have these sources of data:

Digital garden

If you happen to have a digital garden you can look at your git history to know what has changed since the last year. That's cumbersome and ugly though, it's better to review your newsletters, although you may need to use something like mkdocs-newsletter.

While you skim through the newsletters you can add to the analysis report the highlights of what you've learned.

You can also check your repository insights.

Anki

I use anki to record the knowledge that I need to have in my mind. The program has a "Stats" tab where you can see your insights of the last years to understand how are you learning. You can also go to the "Browse" tab to sort the cards by created and get an idea of which ones have been the most used decks.

Review what programs you have developed

Update your identities

Follow the steps of identity management.

Year reviews

New: Primera versión de la revisión del 2024.

Fuentes
Fascismo

En 2024, a nivel estatal la extrema derecha avanzó posiciones y cuotas de poder, pero las movilizaciones masivas por la vivienda, contra el modelo turístico y la gestión de la dana desafiaron abiertamente su relato del mundo y su reparto de culpas.

A nivel mundial la cosa va incluso peor:
- estados unidos, la principal potencia económica y militar del mundo ha vuelto a caer en manos de Trump, un multimillonario sádico, que ha conseguido todo su poder a través de su gigantesco poder mediático y un ejército de trolls, bulos y paramilitares supremacistas vestidos con pasamontañas y fusiles de asalto. Los principales países del mundo ya han caído o están a punto de caer en las redes de esta internacional de la desinformación, respaldada y promocionada a su vez por cuatro de los cinco hombres más ricos del mundo.
Uno de ellos es elon musk, un ser despreciable que tras cargarse twitter está haciendo campaña por las ultraderechas como la alemana afd. - en argentina ha estado gobernando javier milei desde el 10 de diciembre de 2023, básicamente se está cargando el estado de derecho y todos los avances en políticas feministas que se consiguieron con tanto esfuerzo.

Antifascismo

Las resistencias a las ideas que propugna esta coalición mundial neofascista, que culpa a la migración y a los pobres de todos los males, que niega el cambio climático, los derechos LGTBIQ+ y de las mujeres, pasan por otro lado.

Desde El Salto tenemos la convicción de que el resultado de esta contienda se dirimirá en gran parte en el campo de la información, de los medios y las redes sociales. Por eso creemos que es tan importante, ahora más que nunca, contar con medios propios, que no dependan de grandes poderes económicos o intereses partidarios.

Manifestación antiracista en las calles de Londres.

Vivienda

Las gigantescas movilizaciones por la vivienda y contra el modelo depredador del turismo que recorrieron 2024 desde las Islas Canarias, pasando por Baleares, Madrid, Barcelona y casi todas las grandes capitales españolas, son una buena demostración de que el final de esta película está abierto. Después de estas demostraciones de fuerza y significado, la vivienda no es más barata, pero son menos las personas que piensan que el problema de fondo son los “inquiokupas”.

Canarias se plantó en el mes de abril ante el turismo de masas en una movilización histórica simultánea en las ocho islas.

Manifestación contra la especulación inmobiliaria y por el derecho a la vivienda en octubre.

Los desahucios han continuado durante el 2024, al amparo de la cobertura jurídica ofrecida a los especuladores que trafican con el derecho a la vivienda digna, en contra de lo que establece la Constitución Española.

Un amanecer en el barrio de Lavapiés, mientras el vecindario espera un nuevo desahucio en su calle, que finalmente se ejecutó, dejando a otra familia en la calle.

Cambio climático

La Dana

Similar lucha por el discurso y reparto de culpas se produjo tras la dana en València, donde los intentos de la ultraderecha de dirigir el debate hacia los saqueos realizados supuestamente por migrantes fueron frustrados por la emocionante respuesta popular para hacer llegar una ayuda que ninguna institución, ni local ni estatal, estaba proporcionando.

Una senyera del Pais Valencià en una ventana de Paiporta, tras el paso de la dana.

El cuerpo de un perro flota junto a los restos de cañizo y arboles arrastrados por la Dana a la playa de Pinedo. La pérdida de biodiversidad, víctima y a la vez solución de esta crisis, también está detrás de lo que está ocurriendo. Restaurar la naturaleza es clave para protegernos frente a eventos extremos a la vez que es una oportunidad transformadora para tener unos entornos resilientes y sanos para la biodiversidad y las personas. Para esto se necesitan medidas desde el nivel local hasta el global.

No obstante la población valenciana (y la del resto del estado) ha hecho ver el descontento con la mala gestión que hizo el gobierno del PP en la comunidad como se puede ver en la manifestación multitudinaria del 30 de noviembre para pedir la dimisión del president Carlos Mazón.

Feminismo

Este 2024 también ha estado marcado por las luchas feministas: el terremoto mediático y social provocado por las denuncias anónimas de violencia sexual en redes ha trastocado el panorama político, ha revuelto el mundo del cine y del teatro, y sobre todo, ha resquebrajado el armazón de impunidad que recubre las agresiones machistas. Más allá de dónde acabe todo esto, el movimiento feminista ha vuelto a demostrar su enorme potencia de cambio en un sentido opuesto a la agenda de la extrema derecha.

El 25 de noviembre, Día Internacional de la Eliminación de la Violencia contra la Mujer, se celebró en medio de una intensa conversación en torno a las violencias sexuales.

Caso Pelicot

El 19 de diciembre de 2024 conocimos la sentencia del caso Pelicot. Dominique fue condenado a 20 años, la pena máxima. El Tribunal de Aviñón (Francia) que lo juzgaba lo consideró culpable de un delito de violación agravada contra ahora ya exesposa Gisèle Pelicot. Desde julio de 2011 y hasta octubre de 2020, el hombre utilizó webs de citas para invitar a desconocidos a agredir sexualmente a su entonces esposa, a la que previamente había suministrado una cantidad de ansiolíticos suficiente como para acercarla “al estado de coma”. Los otros casi cincuenta acusados han sido declarados culpables de violación con penas de entre tres y 15 años.

Gisèle también ha hecho alusión a la vergüenza cuando ha explicado por qué todo el mundo la conoce por el apellido de su exmarido y agresor: “Tengo nietos y no quiero que se avergüencen de usar ese apellido. Quiero que estén orgullosos de su abuela. A partir de hoy se recordará a la señora Pelicot y cada vez menos al señor Pelicot”, explicó esta mujer que aunque ha recuperado su apellido tras divorciarse no quiere regalarle el nombre de sus nietos a un violador.

La consigna “la vergüenza debe cambiar de bando” se ha repetido más allá de Francia y tuvo un lugar destacado en las convocatorias del 25 de noviembre, Día por la Eliminación de la Violencia contra las Mujeres.

Caso Errejón

El 23 de octubre, el testimonio de una mujer, publicado en la cuenta de la periodista Cristina Fallarás en Instagram corre por las redes. La mujer señala a “un político de Madrid” como un “maltratador psicológico”. En pocas horas, las redes identifican a Íñigo Errejón. El jueves 25 de octubre, Errejón anuncia que abandona la política y esa misma tarde la actriz Elisa Mouliaá presenta una denuncia ante la Policía en la que relata una agresión sexual.

Migración

El fin de año además nos arroja otra cifra, no por esperada menos demoledora: la estimación de más de diez mil personas muertas tratando de llegar a nuestro país. Personas que, por mucho que se empeñe una extrema derecha en auge mediático, solo buscaban mejorar en algo sus vidas y que la perdieron por un sistema cruel que denomina a este otro genocidio “crisis migratoria”.

Las fronteras españolas han marcado este 2024 un récord de la vergüenza: 10.457 personas han muerto tratando de alcanzar las costas de la península. Imagen de la manifestación antirracista el pasado 9 de noviembre, Madrid.

Movimientos sociales

Este año ha sido duro para las okupas en la ciudad de madríd. Se desalojaron el CSO La Ferroviaria y el CSO La Atalaya

Integrantes del CSO La Atalaya sacan materiales que formaban parte de las decenas de actividades que ofrecía el centro social, mientras la zona se encuentra acordonada por agentes de Policía Nacional.

Servicios públicos

Las asambleas de estudiantes piden seguir la reivindicación de tener unos presupuestos que permitan la supervivencia de la universidad pública en una autonomía donde se impulsa más a la enseñanza privada. Internacional

Palestina

En diciembre, Israel ha enfocado sus ataques sobre los hospitales del norte de Gaza.

ejército que ha llevado a cabo un genocidio que dura ya más de 440 días.

Comenzamos el año con los ojos puestos en Gaza y Cisjordania y pendientes de la vida de cientos de miles de palestinos amenazados en todo el mundo por el estado sionista de Israel. La cosa no ha ido a mejor en estos 12 meses y a este genocidio calculado le han acompañado otras desgracias, como la creciente crisis ambiental con efectos cada vez más recurrentes en todo el planeta, incluida nuestra españita. En Valencia 231 personas han perdido la vida y 4 más permanecen desaparecidas a causa de una dana que arrasó en pocas horas la comunidad, muertes que quizá pudieron haberse evitado si los responsables políticos hubieran actuado con algo de los que se les presupone, vocación de servicio público, en vez de esconder sus miserias y tratar a toda costa judicial de evitar responsabilidades.

Tras la elección de trump el panorama de palestina no ha hecho más que empeorar (aunque harris tampoco tenía pinta que se fuese a mojar lo más mínimo).

No obstante la sociedad está respondiendo como buenamente puede para presionar a sus gobiernos a que paren el genocidio.

En el mes de mayo, cientos de estudiantes tomaron la Universidad Humboldt de Berlín en apoyo a Gaza.

La batalla naval de Vallecas en 2024, con especial recuerdo para Palestina.

Siria

Miembros árabes de las Fuerzas Democráticas Sirias en el interior del estadio de Rakka, que el Estado Islámico había convertido en prisión. Honduras

Honduras heredó del narcogobierno de Juan Orlando Hernández la violencia que ejercen de forma estructural la Policía, el Ejército, las maras y los sicarios contratados por las empresas para matar a líderes ambientales. Corrupción

República

Este año la monarquía ha caído algo en el ideario popular, aunque no lo suficiente.

Momentos de tensión en la visita de los reyes y representantes políticos 5 días después del paso de la Dana en Valencia. La liaron parda porque impidieron que llegase ayuda voluntaria para que pasase l séquito real. No sentó nada bien, por lo que sea.
New: Cositas del 2025.

Fascismo

En el acto de toma de posesión del cargo de trump, elon musk hace el saludo nazi.

Feminismo
- trump nada más llegar al poder restablece el género de nacimiento a las personas trans y amenaza con terminar los programas de diversidad, inclusión e igualdad

Life chores management

aerc

New: Introduce aerc email command line client.

aerc is an email client that runs in your terminal.

Some of its more interesting features include:
- Editing emails in an embedded terminal tmux-style, allowing you to check on incoming emails and reference other threads while you compose your replies
- Render HTML emails with an interactive terminal web browser, highlight patches with diffs, and browse with an embedded less session
- Vim-style keybindings and ex-command system, allowing for powerful automation at a single keystroke
- First-class support for working with git & email
- Open a new tab with a terminal emulator and a shell running for easy access to nearby git repos for parallel work
- Support for multiple accounts, with IMAP, Maildir, Notmuch, Mbox and JMAP backends. Along with IMAP, JMAP, SMTP, and sendmail transfer protocols.
- Asynchronous IMAP and JMAP support ensures the UI never gets locked up by a flaky network.
- Efficient network usage - aerc only downloads the information which is necessary to present the UI, making for a snappy and bandwidth-efficient experience
- Email threading (with and/or without IMAP server support).
- PGP signing, encryption and verification using GNUpg.
- 100% free and open source software!
Installation

Source

Download the latest version Compile it with the repo instructions

Debian

The debian version is very old, compile it directly
```
sudo apt-get install aerc
```
Documentation

The docs are few and hard to read online, but there are throughout in local.

If you're lost you can always run again the tutorial with :help tutorial

Configuration

On its first run, aerc will copy the default config files to ~/.config/aerc on Linux. When you start the program for the first time a wizard will configure an account and start up the tutorial.

Read Bence post, it's a nice guideline.

notmuch

Notmuch can be used directly as a backend for several email clients, including alot, dodo, Emacs, vim and (more importantly for us) aerc. While it can be used on its own, we are going to use it for its search index, and ability to seamlessly operate over multiple accounts' maildir folder. This will provide us with the ability to search all of our email regardless of account, and to show a unified overview of certain folders, e.g. a unified inbox. If you are only setting this up for a single account, I still recommend using notmuch for its search capabilities.

Gmail

If these guidelines don't work, try this others.

Monitorization

If you are using the mailsync scripts proposed above or on Bence's post you can check if the service failed with:
```
groups:
  - name: email
    rules:
      - alert: MailsyncError
        expr: |
          count_over_time({user_service_name=~"mailsync-.*"} |= `Failed` [15m]) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Error syncing the email with service {{ $labels.user_service_name }} at {{ $labels.host}}"
```
It assumes that you have the user_service_name label defined in your logs. I create them with vector with the next config:
```
transforms:
  journald_labels:
    type: remap
    inputs:
      - journald_filter
    source: |
      .service_name = ._SYSTEMD_UNIT || "unknown"
      .user_service_name = .USER_UNIT || ._SYSTEMD_USER_UNIT || "unknown"
```
Usage

aerc has many commands that can be bound to keybindings, to see them all check man 1 aerc.

Main page
- , : Cycles to the previous or next tab
- k, j: Scrolls up and down between messages
- , : Scrolls half a page up or down
- g, G: Selects the first or last message, respectively
- K, J: Switches between folders in the sidebar
- : Opens the selected message
You can also search the selected folder with /, or filter with . When searching you can use n and p to jump to the next and previous result. Filtering hides any non-matching message.

Message viewer

Press to open a message. By default, the message viewer will display your message using less(1). This should also have familiar, vim-like keybindings for scrolling around in your message.

Multipart messages (messages with attachments, or messages with several alternative formats) show a part selector on the bottom of the message viewer.
- , : Cycle between parts of a multipart message
- q: Close the message viewer
- f: next message
- b: previous message
To show HTML messages, uncomment the text/html filter in your aerc.conf file (which is probably in ~/.config/aerc/) and install its dependencies: w3m and dante-utils.

You can also do many tasks you could do in the message list from here, like replying to emails, deleting the email, or view the next and previous message (J and K).

Some interesting commands are:
- :unsubscribe: Attempt to automatically unsubscribe the user from the mailing list through use of the List-Unsubscribe header. If supported, aerc may open a compose window pre-filled with the unsubscribe information or open the unsubscribe URL in a web browser.
Composing messages
- C: Compose a new message
- rr: Reply-all to a message
- rq: Reply-all to a message, and pre-fill the editor with a quoted version of the message being replied to
- Rr: Reply to a message
- Rq: Reply to a message, and pre-fill the editor with a quoted version of the message being replied to
The message composer will appear. You should see To, From, and Subject lines, as well as your $EDITOR. You can use or and to cycle between these fields (tab won't cycle between fields once you enter the editor, but and will).

References
- Home
- Source
- Docs
- Issues
Interesting configurations

himalaya

Correction: Writing an importer.

NOTE: since 3.0.0 the importers need to be done with beangulp. I've tried using it but found it confusing so I fell back to 2.x e

Check a list of already existing importers here

Once you have your importer built up you might want to spice it up with smart_importer
New: Configure GPG.

Himalaya relies on cargo features to enable gpg. You can see the default enabled features in the Cargo.toml file. As of 2025-01-27 the pgp-commands is enabled.

You only need to add the next section to your config:
```
pgp.type = "commands"
```
And then you can use both the cli and the vim plugin with gpg. Super easy

dawarich

New: Add interesting article to merge all protocols under matrix.

You can use bridges to merge all into matrix
New: Introduce dawarich.

Dawarich is a self-hostable alternative to Google Location History (Google Maps Timeline)

Installation

Tweak the official docker-compose keeping in mind:
- To configure the APPLICATION_HOST if you're using a reverse proxy
Then run docker compose up. You can now visit your Dawarich instance at http://localhost:3000 or http://:3000. The default credentials are demo@dawarich.app and password

Go to your account and change the default account and password.

Be careful not to upgrade with watchtower, the devs say that it's not safe yet to do so.

Not there yet - Immich photos are not well shown: This happens when opening the map, or selecting one of the buttons "Yesterday", "Last 7 Days" or "Last month". If you select the same date range through the date-pickers, the photos are shown. - Support import of OSMand+ favourites gpx - OpenID/Oauth support

References
- Home
- Docs
- Source
New: How to see the coordinates of a point.

You need to enable the "Points" layer, which is on the layers menu at the top right of the map.

Then if you click on one point you can get the coordinates

Signal

New: Signal bots.

To write signal bots you can use this library
New: How to set a master password.

You can't, it's not supported and it doesn't look that it will (1, 2)
New: Add note on AWS use.

It runs on AWS. On October 2025 there was an AWS outage and signal fell

Content Management

Jellyfin

New: Add note on apollo.

Also checkout apollo a sunshine fork.
New: How to rollback.

Copy the /srv/jellyfin/app directory to bk.app and then restore the directory from backup.

If the permissions have gone amiss you'll need to fix them before you start the server or you'll get an attempt to write a readonly database error feat(linux_snippets#Resize an image): Resize an image

To resize an image, you can use the convert command from ImageMagick. For example, to resize an image named input.jpg to have a width of 800 pixels while maintaining the aspect ratio, use the following command:
```
convert input.jpg -resize 800x input_resized.jpg
```
You can also specify both width and height if you want to resize the image to specific dimensions:
```
convert input.jpg -resize 800x600 input_resized_exact.jpg
```

Book Management

New: Convert images based pdf to epub.

NOTE: before proceeding inspect the next tools that use AI so it will probably give a better output:
- MinerU
- marker
- docling
- olmocr
If the pdf is based on images

Then you need to use OCR to extract the text.

First, convert the PDF to images:
```
pdftoppm -png input.pdf page
```
Apply OCR to your PDF

Use tesseract to extract text from each image:
```
for img in page-*.png; do
    tesseract "$img" "${img%.png}" -l eng
done
```
This produces page-1.txt, page-2.txt, etc.
New: Protect ombi behind authentik.

This option allows the user to select a HTTP header value that contains the desired login username.

Note that if the header value is present and matches an existing user, default authentication is bypassed - use with caution.

This is most commonly utilized when Ombi is behind a reverse proxy which handles authentication. For example, if using Authentik, the X-authentik-username HTTP header which contains the logged in user's username is set by Authentik's proxy outpost.

Book DRM

New: How to remove DRM from ebooks.

To remove the DRM from ebooks you can use DeDRM_tools.

Installation

You need to download the latest release, and follow the instructions of the README.md of the zip file.o

Usage

Once the plugin is installed, you can import the books you want to remove the DRM from. In theory the plugin only works on import and not on convert, but I've also used the convert tool to make sure it's a different file.

Beets

New: Add forensic architecture.
- Forensic architecture: Forensic Architecture (FA) is a research agency based at Goldsmiths, University of London. Our mandate is to develop, employ, and disseminate new techniques, methods, and concepts for investigating state and corporate violence. Our team includes architects, software developers, filmmakers, investigative journalists, scientists, and lawyers.
New: Add plugins.

Chromaprint

Programmatically interact with beets

There is the python api reference but I don't see how to import for example.

Knowledge Management

Anki

New: File location.

On Linux:
- recent Anki versions store your user data in ~/.local/share/Anki2, or $XDG_DATA_HOME/Anki2 if you have set a custom data path.
- Anki’s launcher is installed in /usr/local/share/anki
- When you install/update Anki with the launcher, it downloads support files and places them in ~/.local/share/AnkiProgramFiles
Removing that folder will cause the launcher to behave like a fresh install.

The AnkiProgramFiles contains all the files needed to run Anki aside from the launcher.

Torrent management

qBittorrent

New: Metrics monitorisation.

Health

Silence

New: Introduce the Right to quiet collective.
- Right to quiet: The Right to Quiet Society for Soundscape Awareness and Protection was founded in Vancouver, British Columbia in 1982 as a charitable organization with the mission of raising public awareness of the detrimental effects of noise on health; promoting awareness of noise pollution and the dangers of noise to our physical, emotional, and spiritual wellbeing; working for noise reduction through better regulation and enforcement; encouraging responsible behaviour regarding noise; advocating for manufacturing quieter products; and fostering recognition of the right to quiet as a basic human right, rather than as an amenity for the affluent.
- Right to quiet resources

Technology

Coding

Bash snippets

New: Add context switches column meanings.
- UID: The real user identification number of the task being monitored.
- USER: The name of the real user owning the task being monitored.
- PID: The identification number of the task being monitored.
- cswch/s: Total number of voluntary context switches the task made per second. A voluntary context switch occurs when a task blocks because it requires a resource that is unavailable.
- nvcswch/s: Total number of non voluntary context switches the task made per second. A involuntary context switch takes place when a task executes for the duration of its time slice and then is forced to relinquish the processor.
- Command: The command name of the task.

PDM

New: Load a plugin on startup.

You can define the lazy = false in your plugin spec
```
return {
  -- the colorscheme should be available when starting Neovim
  {
    "folke/tokyonight.nvim",
    lazy = false, -- make sure we load this during startup if it is your main colorscheme
  }
}
```
diff --git a/docs/linux/wireguard.md b/docs/linux/wireguard.md index d69f315a8e..6ccbaa011c 100644 --- a/docs/linux/wireguard.md +++ b/docs/linux/wireguard.md @@ -16,41 +16,42 @@ VPN solution in the industry.
Correction: Suggest to check uv.

Maybe use uv instead (although so far I'm still using pdm)

Vim Snippets

New: Search for different strings in the same search query.
```
* DONE\|* REJECTED\|* DUPLICATED
```
New: Upgrading python version of all your pipx packages.

If you upgrade the main python version and remove the old one (a dist upgrade) then you won't be able to use the installed packages.

If you're lucky enough to have the old one you can use:
```
pipx reinstall-all --python <the Python executable file>
```
Otherwise you need to export all the packages with pipx list --json > ~/pipx.json

Then reinstall one by one:
```
set -ux
if [[ -e ~/pipx.json ]]; then
    for p in $(cat ~/pipx.json | jq -r '.venvs[].metadata.main_package.package_or_url'); do
        pipx install $p
    done
fi
```
The problem is that this method does not respect the version constrains nor the injects, so you may need to debug each package a bit.
New: Push to a forgejo docker registry.

Login to the container registry

To push an image or if the image is in a private registry, you have to authenticate:
```
docker login forgejo.example.com
```
If you are using 2FA or OAuth use a personal access token instead of the password.

Image naming convention

Images must follow this naming convention:
```
{registry}/{owner}/{image}
```
When building your docker image, using the naming convention above, this looks like:
```
docker build -t {registry}/{owner}/{image}:{tag} .

docker tag {some-existing-image}:{tag} {registry}/{owner}/{image}:{tag}
```
Where your registry is the domain of your forgejo instance (e.g. forgejo.example.com). For example, these are all valid image names for the owner testuser:
- forgejo.example.com/testuser/myimage
- forgejo.example.com/testuser/my-image
- forgejo.example.com/testuser/my/image
NOTE: The registry only supports case-insensitive tag names. So image:tag and image:Tag get treated as the same image and tag.

Push an image

Push an image by executing the following command:
```
docker push forgejo.example.com/{owner}/{image}:{tag}
```
For example:
```
docker push forgejo.example.com/testuser/myimage:latest
```
New: Installing all the binaries the application installs.

If the package installs more than one binary (for example ansible), you need to use the --install-deps flag
```
pipx install --install-deps ansible
```
New: Create quickmarks across files.

Use capital letters for the mark.

http://vim.wikia.com/wiki/Using_marks

Marks can span across files. To use such marks one has to use upper-case registers i.e. A-Z. Lower-case registers are used only within files and do not span files. That's to say, if you were to set a mark in a file foo.c in register "a" and then move to another file and hit 'a, the cursor will not jump back to the previous location. If you want a mark which will take you to a different file then you will need to use an upper-case register. For example, use mA instead of ma.

Configure Docker to host the application

New: Syntax rules.

Boolean operations

Not equal operation In Lua, you can perform a "not equal" comparison using the ~= operator:
```
a ~= b  -- true if a is not equal to b
```
List operations

Length of a list
```
local current_slide = {}

if #current_slide > 0 then
  -- code
end
```
New: Configure gitea.

Check the configuration sheet or the default values
New: Upgrade the gitea actions runner.
- Check in the releases the last version and the changelog
- Deploy the new version
- Restart the service
New: Upgrade gitea.

Check the Changelog for breaking changes

To make Gitea better, some breaking changes are unavoidable, especially for big milestone releases. Before upgrading, please read the Changelog on Gitea blog and check whether the breaking changes affect your Gitea instance.

Verify there are no deprecated configuration options

New versions of Gitea often come with changed configuration syntax or options which are usually displayed for at least one release cycle inside at the top of the Site Administration panel. If these warnings are not resolved, Gitea may refuse to start in the following version.

Make a backup

Upgrade from docker
- docker pull the latest Gitea release.
- Stop the running instance, backup data.
- Use docker or docker-compose to start the newer Gitea Docker container.
Upgrade from binary

A script automating the following steps for a deployment on Linux can be found at contrib/upgrade.sh in Gitea's source tree.
- Download the latest Gitea binary to a temporary directory.
- Stop the running instance, backup data.
- Replace the installed Gitea binary with the downloaded one.
- Start the Gitea instance.
Read the script to see what it's going to do. To upgrade to 1.20.5 you can use:
```
./update.sh -v 1.20.5
```
If you have a different home directory for gitea you can set
```
giteahome=/var/gitea ./update.sh -v 1.20.5
```
New: Limit the access of a docker on a server to the access on the docker of another server.

WARNING: I had issues with this path and I ended up not using docker swarm networks.

If you want to restrict access to a docker (running on server 1) so that only another specific docker container running on another server (server 2) can access it. You need more than just IP-based filtering between hosts. The solution is then to:
1. Create a Docker network that spans both hosts using Docker Swarm or a custom overlay network.
2. Use Docker's built-in DNS resolution to allow specific container-to-container communication.
Here's a step-by-step approach:

1. Set up Docker Swarm (if not already done)

On server 1:
```
docker swarm init --advertise-addr <ip of server 1>
```
This will output a command to join the swarm. Run that command on server 2.

2. Create an overlay network
```
docker network create --driver overlay --attachable <name of the network>
```
3. Update the docker compose on server 1

magine for example that we want to deploy wg-easy.
```
services:
  wg-easy:
    image: ghcr.io/wg-easy/wg-easy:latest
    container_name: wg-easy
    networks:
      - wg
      - <name of the network> # Add the overlay network
    volumes:
      - wireguard:/etc/wireguard
      - /lib/modules:/lib/modules:ro
    ports:
      - "51820:51820/udp"
      # - "127.0.0.1:51821:51821/tcp" # Don't expose the http interface, it will be accessed from within the docker network
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.all.disable_ipv6=1

networks:
  wg:
    # Your existing network config
  <name of the network>:
    external: true # Reference the overlay network created above
```
4. On server 2, create a Docker Compose file for your client container
```
services:
  wg-client:
    image: your-client-image
    container_name: wg-client
    networks:
      - <name of the network>
    # Other configuration for your client container

networks:
  <name of the network>:
    external: true # Reference the same overlay network
```
5. Access the WireGuard interface from the client container

Now, from within the client container on server 2, you can access the WireGuard interface using the container name:
```
http://wg-easy:51821
```
This approach ensures that:
1. The WireGuard web interface is not exposed to the public (not even localhost on server 1)
2. Only containers on the shared overlay network can access it
3. The specific container on server 2 can access it using Docker's internal DNS
Testing the network is well set

You may be confused if the new network is not shown on server 2 when running docker network ls but that's normal. Server 2 is a swarm worker node. The issue with not seeing the overlay network on server 2 is actually expected behavior - worker nodes cannot list or manage networks directly. However, even though you can't see them, containers on the server 2 can still connect to the overlay network when properly configured.

To see that the swarm is well set you can use docker node ls on server 1 (you'll see an error on server 2 as it's a worker node)

Weird network issues with swarm overlays

I've seen cases where after a server reboot you need to remove the overlay network from the docker compose and then add it again.

After many hours of debugging I came with the patch of removing the overlay network from the docker-compose and attaching it with the systemd service
```
[Unit]
Description=wg-easy
Requires=docker.service
After=docker.service

[Service]
Restart=always
User=root
Group=docker
WorkingDirectory=/data/apps/wg-easy
TimeoutStartSec=100
RestartSec=2s
ExecStart=/usr/bin/docker compose -f docker-compose.yaml up

ExecStartPost=/bin/bash -c '\
    sleep 30; \
    /usr/bin/docker network connect wg-easy wg-easy; \
'
ExecStop=/usr/bin/docker compose -f docker-compose.yaml down

[Install]
WantedBy=multi-user.target
```
New: Configure triggers not to push to a branch.

There is now a branches-ignore option:
```
on:
  push:
    branches-ignore:
      - main
```
New: Not there yet.
- Being able to run two jobs on the same branch: It will be implemented with concurrency with this pr. This behavior didn't happen before 2023-07-25
New: Push an image with different architectures after building it in different instances.

To push both an ARM and AMD Docker image to Docker Hub, from two separate machines (e.g., an ARM-based and an AMD-based instance), follow these steps:

Tag the image correctly on each architecture

On each instance, build your image as normal, but tag it with a platform-specific suffix, like myuser/myimage:arm64 or myuser/myimage:amd64.

On the ARM machine:
```
docker build -t myuser/myimage:arm64 .
docker push myuser/myimage:arm64
```
On the AMD machine:
```
docker build -t myuser/myimage:amd64 .
ocker push myuser/myimage:amd64
```
Create a multi-architecture manifest (on one machine)

If you want users to pull the image without worrying about the platform (e.g., just docker pull myuser/myimage:latest), you can create and push a manifest list that combines the two:

Choose either machine to run this (after both images are pushed):
```
docker manifest create myuser/myimage:latest \
    --amend myuser/myimage:amd64 \
    --amend myuser/myimage:arm64

docker manifest push myuser/myimage:latest
```
Correction: Add note to run gitea actions runners in kubernetes.

Kubernetes act runner, there is a stale issue and an ugly implementation to run docker in docker inside the kubernetes node.
Correction: Push an image with different architectures after building it in different instances.

To push both an ARM and AMD Docker image to a Docker registry, from two separate machines (e.g., an ARM-based and an AMD-based instance), you have two options:
- Run two different pipelines and then build a manifest
- Use two buildx remotes
QEMU was discarted because it took too long to build the images.

New: Do a copy of a list of docker images in your private registry.

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
IMAGE_LIST_FILE="${1:-${SCRIPT_DIR}/bitnami-images.txt}"
TARGET_REGISTRY="${2:-}"

if [[ -z "$TARGET_REGISTRY" ]]; then
  echo "Usage: $0 <image_list_file> <target_registry>"
  echo "Example: $0 bitnami-images.txt your.docker.registry.org"
  exit 1
fi

if [[ ! -f "$IMAGE_LIST_FILE" ]]; then
  echo "Error: Image list file '$IMAGE_LIST_FILE' not found"
  exit 1
fi

log() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"
}

extract_image_name_and_tag() {
  local full_image="$1"

  # Remove registry prefix to get org/repo:tag
  # Examples:
  # docker.io/bitnami/discourse:3.4.7 -> bitnami/discourse:3.4.7
  # registry-1.docker.io/bitnami/os-shell:11-debian-11-r95 -> bitnami/os-shell:11-debian-11-r95

  if [[ "$full_image" =~ ^[^/]*\.[^/]+/ ]]; then
    # Contains registry with dot - remove everything up to first /
    echo "${full_image#*/}"
  else
    # No registry prefix
    echo "$full_image"
  fi
}

pull_and_push_multiarch() {
  local source_image="$1"
  local target_registry="$2"

  local image_name_with_tag
  image_name_with_tag=$(extract_image_name_and_tag "$source_image")
  local target_image="${target_registry}/${image_name_with_tag}"

  log "Processing: $source_image -> $target_image"

  local pushed_images=()
  local architectures=("linux/amd64" "linux/arm64")
  local arch_suffixes=("amd64" "arm64")

  # Try to pull and push each architecture
  for i in "${!architectures[@]}"; do
    local platform="${architectures[$i]}"
    local arch_suffix="${arch_suffixes[$i]}"

    log "Attempting to pull ${platform} image: $source_image"

    if sudo docker pull --platform "$platform" "$source_image" 2>/dev/null; then
      log "Successfully pulled ${platform} image"

      # Tag with architecture-specific tag for manifest creation
      local arch_specific_tag="${target_image}-${arch_suffix}"
      sudo docker tag "$source_image" "$arch_specific_tag"

      log "Pushing ${platform} image as ${arch_specific_tag}"
      if sudo docker push "$arch_specific_tag"; then
        log "Successfully pushed ${platform} image"
        pushed_images+=("$arch_specific_tag")
      else
        log "Failed to push ${platform} image"
        sudo docker rmi "$arch_specific_tag" 2>/dev/null || true
      fi
    else
      log "⚠️  ${platform} image not available for $source_image - skipping"
    fi
  done

  if [[ ${#pushed_images[@]} -eq 0 ]]; then
    log "❌ No images were successfully pushed for $source_image"
    return 1
  fi

  # Create the main tag with proper multi-arch manifest
  if [[ ${#pushed_images[@]} -gt 1 ]]; then
    log "Creating multi-arch manifest for $target_image"

    # Remove any existing manifest (in case of retry)
    sudo docker manifest rm "$target_image" 2>/dev/null || true

    if sudo docker manifest create "$target_image" "${pushed_images[@]}"; then
      # Annotate each architecture in the manifest
      for i in "${!pushed_images[@]}"; do
        local arch_tag="${pushed_images[$i]}"
        local arch="${arch_suffixes[$i]}"
        sudo docker manifest annotate "$target_image" "$arch_tag" --arch "$arch" --os linux
      done

      log "Pushing multi-arch manifest to $target_image"
      if sudo docker manifest push "$target_image"; then
        log "✅ Successfully pushed multi-arch image: $target_image"
      else
        log "❌ Failed to push manifest for $target_image"
        return 1
      fi
    else
      log "❌ Failed to create manifest for $target_image"
      return 1
    fi
  else
    # Only one architecture - tag and push directly
    log "Single architecture available, pushing as $target_image"
    sudo docker tag "${pushed_images[0]}" "$target_image"
    if sudo docker push "$target_image"; then
      log "✅ Successfully pushed single-arch image: $target_image"
    else
      log "❌ Failed to push $target_image"
      return 1
    fi
  fi

  # Clean up local images to save space
  sudo docker rmi "$source_image" "${pushed_images[@]}" 2>/dev/null || true
  if [[ ${#pushed_images[@]} -eq 1 ]]; then
    sudo docker rmi "$target_image" 2>/dev/null || true
  fi

  return 0
}

main() {
  log "Starting multi-architecture image pull and push"
  log "Source list: $IMAGE_LIST_FILE"
  log "Target registry: $TARGET_REGISTRY"

  # Enable experimental CLI features for manifest commands
  export DOCKER_CLI_EXPERIMENTAL=enabled

  total_images=$(wc -l "$IMAGE_LIST_FILE")
  local processed_images=0
  local successful_images=0
  local failed_images=()

  while IFS= read -r image_line; do
    # Skip empty lines and comments
    [[ -z "$image_line" || "$image_line" =~ ^[[:space:]]*# ]] && continue

    # Remove leading/trailing whitespace
    image_line=$(echo "$image_line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
    [[ -z "$image_line" ]] && continue

    echo $image_line
    processed_images=$((processed_images + 1))

    log "[$processed_images/$total_images] Processing: $image_line"

    if pull_and_push_multiarch "$image_line" "$TARGET_REGISTRY"; then
      successful_images=$((successful_images + 1))
      log "✓ Success: $image_line"
    else
      failed_images+=("$image_line")
      log "✗ Failed: $image_line"
    fi

    log "Progress: $processed_images/$total_images completed"
    echo "----------------------------------------"

  done <"$IMAGE_LIST_FILE"

  log "Final Summary:"
  log "Total images processed: $processed_images"
  log "Successful: $successful_images"
  log "Failed: ${#failed_images[@]}"

  if [[ ${#failed_images[@]} -gt 0 ]]; then
    log "Failed images:"
    printf '  %s\n' "${failed_images[@]}"
    exit 1
  fi

  log "🎉 All images processed successfully!"
}

main "$@"

New: Migrate away from bitnami.

Bitnami is changing their pull policy making it unfeasible to use their images (More info here: 1, 2, 3), so there is the need to migrate to other image providers.

Which alternative to use

The migration can be done to the official maintained images (although this has some disadvantages) or to any of the common docker image builders:

There is an effort to build a fork of bitnami images but it has not yet much inertia.

Regarding the alternatives of helm charts, a quick look shown this one.

Infrastructure archeology

First you need to know which images are you using, to do that you can:

Clone al git repositories of a series of organisations and do a local grep
Search all the container images in use in kubernetes that match a desired string
Recursively pull a copy of all helm charts used by an argocd repository and then do a grep.

Create a local copy of the images

It's wise to make a copy of the used images in your local registry to be able to pull the dockers once bitnami does not longer let you.

To do that you can save the used images in a bitnami-images.txt file and run the next script:

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
IMAGE_LIST_FILE="${1:-${SCRIPT_DIR}/bitnami-images.txt}"
TARGET_REGISTRY="${2:-}"

if [[ -z "$TARGET_REGISTRY" ]]; then
  echo "Usage: $0 <image_list_file> <target_registry>"
  echo "Example: $0 bitnami-images.txt registry.cloud.icij.org"
  exit 1
fi

if [[ ! -f "$IMAGE_LIST_FILE" ]]; then
  echo "Error: Image list file '$IMAGE_LIST_FILE' not found"
  exit 1
fi

log() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"
}

extract_image_name_and_tag() {
  local full_image="$1"

  # Remove registry prefix to get org/repo:tag
  # Examples:
  # docker.io/bitnami/discourse:3.4.7 -> bitnami/discourse:3.4.7
  # registry-1.docker.io/bitnami/os-shell:11-debian-11-r95 -> bitnami/os-shell:11-debian-11-r95
  # registry-proxy.internal.cloud.icij.org/bitnami/minideb:stretch -> bitnami/minideb:stretch

  if [[ "$full_image" =~ ^[^/]*\.[^/]+/ ]]; then
    # Contains registry with dot - remove everything up to first /
    echo "${full_image#*/}"
  else
    # No registry prefix
    echo "$full_image"
  fi
}

pull_and_push_multiarch() {
  local source_image="$1"
  local target_registry="$2"

  local image_name_with_tag
  image_name_with_tag=$(extract_image_name_and_tag "$source_image")
  local target_image="${target_registry}/${image_name_with_tag}"

  log "Processing: $source_image -> $target_image"

  local pushed_images=()
  local architectures=("linux/amd64" "linux/arm64")
  local arch_suffixes=("amd64" "arm64")

  # Try to pull and push each architecture
  for i in "${!architectures[@]}"; do
    local platform="${architectures[$i]}"
    local arch_suffix="${arch_suffixes[$i]}"

    log "Attempting to pull ${platform} image: $source_image"

    if sudo docker pull --platform "$platform" "$source_image" 2>/dev/null; then
      log "Successfully pulled ${platform} image"

      # Tag with architecture-specific tag for manifest creation
      local arch_specific_tag="${target_image}-${arch_suffix}"
      sudo docker tag "$source_image" "$arch_specific_tag"

      log "Pushing ${platform} image as ${arch_specific_tag}"
      if sudo docker push "$arch_specific_tag"; then
        log "Successfully pushed ${platform} image"
        pushed_images+=("$arch_specific_tag")
      else
        log "Failed to push ${platform} image"
        sudo docker rmi "$arch_specific_tag" 2>/dev/null || true
      fi
    else
      log "⚠️  ${platform} image not available for $source_image - skipping"
    fi
  done

  if [[ ${#pushed_images[@]} -eq 0 ]]; then
    log "❌ No images were successfully pushed for $source_image"
    return 1
  fi

  # Create the main tag with proper multi-arch manifest
  if [[ ${#pushed_images[@]} -gt 1 ]]; then
    log "Creating multi-arch manifest for $target_image"

    # Remove any existing manifest (in case of retry)
    sudo docker manifest rm "$target_image" 2>/dev/null || true

    if sudo docker manifest create "$target_image" "${pushed_images[@]}"; then
      # Annotate each architecture in the manifest
      for i in "${!pushed_images[@]}"; do
        local arch_tag="${pushed_images[$i]}"
        local arch="${arch_suffixes[$i]}"
        sudo docker manifest annotate "$target_image" "$arch_tag" --arch "$arch" --os linux
      done

      log "Pushing multi-arch manifest to $target_image"
      if sudo docker manifest push "$target_image"; then
        log "✅ Successfully pushed multi-arch image: $target_image"
      else
        log "❌ Failed to push manifest for $target_image"
        return 1
      fi
    else
      log "❌ Failed to create manifest for $target_image"
      return 1
    fi
  else
    # Only one architecture - tag and push directly
    log "Single architecture available, pushing as $target_image"
    sudo docker tag "${pushed_images[0]}" "$target_image"
    if sudo docker push "$target_image"; then
      log "✅ Successfully pushed single-arch image: $target_image"
    else
      log "❌ Failed to push $target_image"
      return 1
    fi
  fi

  # Clean up local images to save space
  sudo docker rmi "$source_image" "${pushed_images[@]}" 2>/dev/null || true
  if [[ ${#pushed_images[@]} -eq 1 ]]; then
    sudo docker rmi "$target_image" 2>/dev/null || true
  fi

  return 0
}

main() {
  log "Starting multi-architecture image pull and push"
  log "Source list: $IMAGE_LIST_FILE"
  log "Target registry: $TARGET_REGISTRY"

  # Enable experimental CLI features for manifest commands
  export DOCKER_CLI_EXPERIMENTAL=enabled

  total_images=$(wc -l "$IMAGE_LIST_FILE")
  local processed_images=0
  local successful_images=0
  local failed_images=()

  while IFS= read -r image_line; do
    # Skip empty lines and comments
    [[ -z "$image_line" || "$image_line" =~ ^[[:space:]]*# ]] && continue

    # Remove leading/trailing whitespace
    image_line=$(echo "$image_line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
    [[ -z "$image_line" ]] && continue

    echo $image_line
    processed_images=$((processed_images + 1))

    log "[$processed_images/$total_images] Processing: $image_line"

    if pull_and_push_multiarch "$image_line" "$TARGET_REGISTRY"; then
      successful_images=$((successful_images + 1))
      log "✓ Success: $image_line"
    else
      failed_images+=("$image_line")
      log "✗ Failed: $image_line"
    fi

    log "Progress: $processed_images/$total_images completed"
    echo "----------------------------------------"

  done <"$IMAGE_LIST_FILE"

  log "Final Summary:"
  log "Total images processed: $processed_images"
  log "Successful: $successful_images"
  log "Failed: ${#failed_images[@]}"

  if [[ ${#failed_images[@]} -gt 0 ]]; then
    log "Failed images:"
    printf '  %s\n' "${failed_images[@]}"
    exit 1
  fi

  log "🎉 All images processed successfully!"
}

main "$@"

Replace a bitnami image with a local one

If you for some reason need to pull bitnami/discourse:3.4.7 and get an error you need instead to pull it from {your_registry}/bitnami/discourse:3.4.7. If you want to pull a specific architecture you can append it at the end ({your_registry}/bitnami/discourse:3.4.7-amd64.

If you need to do the changes in an argocd managed kubernetes application search in the values.yaml or values-{environment}.yaml files for the image: string. If it's not defined you may need to look at the helm chart definition. To do that open the Chart.yaml file to find the chart and the version used. For example:

---
apiVersion: v2
name: discourse
version: 1.0.0
dependencies:
  - name: discourse
    version: 12.6.2
    repository: https://charts.bitnami.com/bitnami

You can pull a local copy of the chart with:

If the chart is using an oci url:

helm pull oci://registry-1.docker.io/bitnamicharts/postgresql --version 8.10.X --untar -d postgres8

If it's using an https url:

helm pull cost-analyzer --repo https://kubecost.github.io/cost-analyzer/ --version 2.7.2

And inspect the values.yaml file and all the templates until you find which key value you need to add.

feat(docker#Cannot invoke "jdk.internal.platform.CgroupInfo.getMountPoint()" because "anyController" is null) : Cannot invoke "jdk.internal.platform.CgroupInfo.getMountPoint()" because "anyController" is null

It's caused because the docker is not able to access the cgroups, this can be caused by a docker using the legacy cgroups v1 while the linux kernel (>6.12) is using the v2.

The best way to fix it is to upgrade the docker to use v2, if you can't you need to force the system to use the v1. To do that:

Edit /etc/default/grub to add the configuration GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0"
Then update GRUB: sudo update-grub
Reboot

New: Clone al git repositories of a series of organisations.

It assumes you have tea configured to interact with the desired gitea instance.

set -e

ORGANIZATIONS=("ansible-playbooks" "ansible-roles")

clone_org_repos() {
  local page=1
  local has_more=true

  while [ "$has_more" = true ]; do
    echo "Fetching page $page..."

    local csv_output
    csv_output=$(tea repo ls --output csv --page "$page" 2>/dev/null || true)

    if [ -z "$csv_output" ] || [ "$csv_output" = '"owner","name","type","ssh"' ] || [ "$(echo "$csv_output" | wc -l)" -lt 3 ]; then
      echo "No more repositories found on page $page"
      has_more=false
      break
    fi

    local repo_count=0
    while IFS=',' read -r owner name type ssh_url; do
      if [ "$owner" = '"owner"' ]; then
        continue
      fi

      owner=$(echo "$owner" | sed 's/"//g')
      name=$(echo "$name" | sed 's/"//g')
      ssh_url=$(echo "$ssh_url" | sed 's/"//g')

      # echo "owner: $owner name: $name ssh_url: $ssh_url"
      if [[ -n "$name" ]] && [[ -n "$ssh_url" ]] && [[ "${ORGANIZATIONS[*]}" =~ $owner ]]; then
        echo "Cloning repository: $name"
        if [ ! -d "$name" ]; then
          git clone "$ssh_url" "$owner/$name" || {
            echo "Failed to clone $name, skipping..."
            continue
          }
        else
          echo "Repository $name already exists, skipping..."
        fi
        repo_count=$((repo_count + 1))
      fi
    done <<<"$csv_output"

    ((page++))
  done

  cd ..
  echo "Finished processing $org"
  echo
}

main() {
  echo "Starting repository cloning process..."
  echo "Target organizations: ${ORGANIZATIONS[*]}"
  echo

  if ! command -v tea &>/dev/null; then
    echo "Error: 'tea' command not found. Please install gitea tea CLI."
    exit 1
  fi

  if ! command -v git &>/dev/null; then
    echo "Error: 'git' command not found. Please install git."
    exit 1
  fi

  for org in "${ORGANIZATIONS[@]}"; do
    if [ ! -d "$org" ]; then
      mkdir "$org"
    fi
  done

  clone_org_repos
  echo "Repository cloning process completed!"
  echo "Check the following directories:"
  for org in "${ORGANIZATIONS[@]}"; do
    if [ -d "$org" ]; then
      echo "  - $org/ ($(find "$org" -maxdepth 1 -type d | wc -l) repositories)"
    fi
  done
}

main "$@"

File management configuration

New: Introduce radicle.

Radicle is an open source, peer-to-peer code collaboration stack built on Git. Unlike centralized code hosting platforms, there is no single entity controlling the network. Repositories are replicated across peers in a decentralized manner, and users are in full control of their data and workflow.

References - Home - Support
New: Sum up the notes of the vim plugin development tutorial.

For the repository name, plugins usually finish with the .nvim extension. I'm going to call mine org-misc.nvim.

Let's go with the second, add to your init.lua the next code:
```
print("Hello from our plugin")
```
How to load our extension

We can have the code of our extension wherever we want in our filesystem, but we need to tell Neovim where our plugin’s code is, so it can load the files correctly. Since I use lazy.nvim this is the way to load a plugin from a local folder:
```
{
  dir = "~/projects/org-misc", -- Your path
}
```
Now if you restart your neovim you won't see anything until you load it with :lua require "org-misc" you'll see the message Hello from our plugin in the command line.

To automatically load the plugin when you open nvim, use the next lazy config:
```
{
  dir = "~/projects/org-misc", -- Your path
  config = function()
    require "org-misc"
  end
}
```
The plugin file structure

Usually init.lua starts with:
```
local M = {}

M.setup = function ()
  -- nothing yet
end

return M
```
Where: - M stands for module, and we'll start adding it methods. - M.setup will be the method we use to configure the plugin.

Let's start with a basic functionality to print some slides:
```
local M = {}

M.setup = function()
    -- nothing yet
end

---@class present.Slides
---@fields slides string[]: The slides of the file

--- Takes some lines and parses them
--- @param lines string
--- @return present.Slides
local parse_slides = function(lines)
    local slides = { slides = {} }
    for _, line in ipairs(lines) do
        print(line)
    end
    return slides
end

print(parse_slides({
    "# Hello",
    "this is something else",
    "# world",
    "this is something else",
}))

return M
```
You can run the code in the current buffer with :%lua. For quick access, I've defined the next binding:
```
keymap.set("n", "<leader>X", ":%lua<cr>", {desc = "Run the lua code in the current buffer"})
```
The print(parse_slides.. part it's temporal code so that you can debug your code easily. Once it's ready you'll remove them

Call a method of a module

To run the method of a module:
```
local M = {}

M.setup = function()
    -- nothing yet
end

return M
```
You can do require('org-misc').setup()

Set keymaps

Inside the code of the plugin

You can set keymaps into your plugins by using:
```
vim.keymap.set("n", "n", function()
  -- code
end)
```
The problem is that it will override the n key everywhere which is not a good idea, that's why we normally limit it to the current buffer.

You can get the current buffer with buffer = true
```
vim.keymap.set("n", "n", function()
  -- code
end,
  {
    buffer = true
  }
)
```
Continue till the end If you want to stop capturing the traffic flow and go to the end ignoring all breakpoints, remove all breakpoints and do .c

Reload the plugin without exiting nvim If you are using lazy.nvim, there is a feature that lazy.nvim provides for this purpose:
```
Lazy reload your_plugin your_plugin2
```
Neovim plugin testing

We're going to test it with plenary. We'll add a tests directory at the root of our repository.

Each of the test files need to end in _spec.lua, so if we want to test a parse_lines it will be called parse_lines_spec.lua.

Each test file has the following structure
```
local clockin = require('org-misc').clockin

describe("org-misc.clockin", function()
    it("should do clockin", function()
        assert.is.True(clock_in())
    end)
end)
```
These are all the tests for the clockin method,

Now you can run the test with :PlenaryBustedFile %

Configuring neotest to run the tests

Using :PlenaryBustedFile % is not comfortable, that's why we're going to use neotest

Configure it with:
```
return {
  {
    "nvim-neotest/neotest",
    dependencies = {
      "nvim-neotest/neotest-plenary",
    },
    config = function()
      require("neotest").setup({
        adapters = {
          require("neotest-plenary"),
        },
      })
    end,
  },
}
```
Now you can do:
- <leader>tT to run all test files
- <leader>tt to run the whole file
- <leader>tl to run the last test
- <leader>to to show the output
- <leader>tr to run the nearest
- <leader>ts to show the summary
Remove the Undefined global describe linter warnings

Add to the root of your repository a .luarc.json file with the next contents
```
{
  "$schema": "https://raw.githubusercontent.com/sumneko/vscode-lua/master/setting/schema.json",
  "diagnostics": {
    "globals": ["vim"]
  },
  "hint": {
    "enable": true
  },
  "runtime": {
    "path": ["?.lua", "?/init.lua"],
    "pathStrict": true,
    "version": "LuaJIT"
  },
  "telemetry": {
    "enable": false
  },
  "workspace": {
    "checkThirdParty": "Disable",
    "ignoreDir": [".git"],
    "library": [
      "./lua",
      "$VIMRUNTIME/lua",
      "${3rd}/luv/library",
      "./tests/.deps/plugins/plenary"
    ]
  }
}
```
Testing internal functions

If you have a function parse_lines in your module that you want to test, you can export it as an internal method
```
local parse_lines = function ()
  -- code
end

M._parse_lines = parse_lines
```
New: Control an existing nvim instance with dap.

Once you have all set up and assuming you're using the lazyvim keybindings for nvim-dap:
```
vim.api.nvim_set_keymap('n', '<leader>ds', [[:lua require"osv".launch({port = 8086})<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>dq', [[:lua require"osv".stop()<CR>]], { noremap = true })
```
You will debug the plugin by:
- Launch the server in the nvim instance where you're going to run the actions using <leader>ds.
- Open another Neovim instance with the source file (the debugger).
- Place breakpoint with <leader>db.
- On the debugger connect to the DAP client with <leader>dc.
- Optionally open the nvim-dap-ui with <leader>B in the debugger.
- Run your script/plugin in the debuggee
Now you can interact with the debugger in the window below the code. You have the next commands:
- help: Show all commands
- <enter>: run the same action as the previous one. For example if you do .n and then <enter> it will run .n again.
- .n or .next: next step
- .b or .back: previous step (if the debugger supports it)
- .c or .continue: Continue to the next breakpoint.
New: How to exclude some files from the search.

If anyone else comes here in the future and have the following setup
- Using fd as default command: export FZF_DEFAULT_COMMAND='fd --type file --hidden --follow'
- Using :Rg to grep in files
And want to exclude a specific path in a git project say path/to/exclude (but that should not be included in .gitignore) from both fd and rg as used by fzf.vim, then the easiest way I found to solve to create ignore files for the respective tool then ignore this file in the local git clone (as they are only used by me)
```
cd git_proj/
echo "path/to/exclude" > .rgignore
echo "path/to/exclude" > .fdignore
printf ".rgignore\n.fdignore" >> .git/info/exclude
```

Plugin System

Correction: Write python plugins with entrypoints.

Python Snippets

New: Convert a datetime into a date.
```
datetime.now().date()
```

New: Download book previews from google books.

You will only get some of the pages but it can help in the ending pdf

This first script gets the images data:

import asyncio
import os
import json
import re
from urllib.parse import urlparse, parse_qs
from playwright.async_api import async_playwright
import aiohttp
import aiofiles

async def download_image(session, src, output_path):
    """Download image from URL and save to specified path"""
    try:
       headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; rv:128.0) Gecko/20100101 Firefox/128.0",
            "Accept": "*/*",
            "Accept-Language": "en-US,en;q=0.5",
            "Referer": "https://books.google.es/",
            "DNT": "1",
            "Sec-GPC": "1",
            "Connection": "keep-alive",
        }

        async with session.get(src, headers=headers) as response:
            response.raise_for_status()
            async with aiofiles.open(output_path, "wb") as f:
                await f.write(await response.read())

        print(f"Downloaded: {output_path}")
        return True
    except Exception as e:
        print(f"Error downloading {src}: {e}")
        return False

def extract_page_number(pid):
    """Extract numeric page number from page ID"""
    match = re.search(r"PA(\d+)", pid)
    if match:
        return int(match.group(1))
    try:
        return int(pid.replace("PA", "").replace("PP", ""))
    except:
        return 9999

async def main():
    # Create output directory
    output_dir = "book_images"
    os.makedirs(output_dir, exist_ok=True)

    # Keep track of all pages found
    seen_pids = set()
    page_counter = 0
    download_tasks = []

    # Create HTTP session for downloads
    async with aiohttp.ClientSession() as session:
        async with async_playwright() as p:
            browser = await p.firefox.launch(headless=False)
            context = await browser.new_context(
                user_agent="Mozilla/5.0 (Windows NT 10.0; rv:128.0) Gecko/20100101 Firefox/128.0"
            )

            # Create a page and set up response handling
            page = await context.new_page()

            # Store seen URLs to avoid duplicates
            seen_urls = set()

            # Set up response handling for JSON data
            async def handle_response(response):
                nonlocal page_counter
                url = response.url

                # Only process URLs with jscmd=click3
                if "jscmd=click3" in url and url not in seen_urls:
                    try:
                        # Try to parse as JSON
                        json_data = await response.json()
                        seen_urls.add(url)

                       # Process and download page data immediately
                        if "page" in json_data and isinstance(json_data["page"], list):
                            for page_data in json_data["page"]:
                                if "src" in page_data and "pid" in page_data:
                                    pid = page_data["pid"]
                                    if pid not in seen_pids:
                                        seen_pids.add(pid)
                                       src = page_data["src"]

                                        # Create filename with sequential numbering
                                        formatted_index = (
                                            f"{int(pid.replace('PA', '')):03d}"
                                        )
                                        output_file = os.path.join(
                                            output_dir, f"page-{formatted_index}.png"
                                        )
                                        page_counter += 1

                                        print(
                                            f"Found new page: {pid}, scheduling download"
                                        )

                                        # Start download immediately
                                        task = asyncio.create_task(
                                            download_image(session, src, output_file)
                                        )
                                        download_tasks.append(task)

                        return len(seen_pids)
                   except Exception as e:
                        print(f"Error processing response from {url}: {e}")

            # Register response handler
            page.on("response", handle_response)

            # Navigate to the starting URL
            book_url = (
                "https://books.google.es/books?id=412loEMJA9sC&lpg=PP1&hl=es&pg=PA5"
            )
            await page.goto(book_url)

            # Wait for initial page load
            await page.wait_for_load_state("networkidle")

            # Scroll loop variables
            max_scroll_attempts = 500  # Safety limit
            scroll_count = 0
            pages_before_scroll = 0
            consecutive_no_new_pages = 0

            # Continue scrolling until we find no new pages for several consecutive attempts
            while scroll_count < max_scroll_attempts and consecutive_no_new_pages < 5:
                # Get current page count before scrolling
               pages_before_scroll = len(seen_pids)

                # Use PageDown key to scroll
                await page.keyboard.press("PageDown")
                scroll_count += 1

                # Wait for network activity
                await asyncio.sleep(2)

                # Check if we found new pages after scrolling
                if len(seen_pids) > pages_before_scroll:
                    consecutive_no_new_pages = 0
                    print(
                        f"Scroll {scroll_count}: Found {len(seen_pids) - pages_before_scroll} new pages"
                    )
                else:
                    consecutive_no_new_pages += 1
                    print(
                       f"Scroll {scroll_count}: No new pages found ({consecutive_no_new_pages}/5)"
                    )

            print(f"Scrolling complete. Found {len(seen_pids)} pages total.")
            await browser.close()

        # Wait for any remaining downloads to complete
        if download_tasks:
            print(f"Waiting for {len(download_tasks)} downloads to complete...")
            await asyncio.gather(*download_tasks)

        print(f"Download complete! Downloaded {page_counter} images.")

if __name__ == "__main__":
    asyncio.run(main())

New: Send keystrokes to an active window.

import subprocess

subprocess.run(['xdotool', 'type', 'Hello world!'])
subprocess.run(['xdotool', 'key', 'Return']) # press enter

subprocess.run(['xdotool', 'key', 'ctrl+c'])

window_id = subprocess.check_output(['xdotool', 'getactivewindow']).decode().strip()
subprocess.run(['xdotool', 'windowactivate', window_id])

New: Make temporal file.

import tempfile
with tempfile.NamedTemporaryFile(
    suffix=".tmp", mode="w+", encoding="utf-8"
) as temp:
    temp.write(
        "# Enter commit message body. Lines starting with '#' will be ignored.\n"
    )
    temp.write("# Leave file empty to skip the body.\n")
    temp.flush()

    subprocess.call([editor, temp.name])

    temp.seek(0)
    lines = temp.readlines()

New: Remove a directory with content.

import shutil

shutil.rmtree(Path('/path/to/directory'))

New: Recursively find files if you only want the files and directories of the first level.

from pathlib import Path

path = Path("/your/directory")
for item in path.iterdir():
    if item.is_file():
        print(f"File: {item.name}")
    elif item.is_dir():
        print(f"Directory: {item.name}")

feat(helm#Download a chart): Download a chart

If the chart is using an oci url:

helm pull oci://registry-1.docker.io/bitnamicharts/postgresql --version 8.10.X --untar -d postgres8

If it's using an https url:

helm pull cost-analyzer --repo https://kubecost.github.io/cost-analyzer/ --version 2.7.2

New: Get value of enum by value.
```
from enum import Enum

class Color(Enum):
    RED = 1
    GREEN = 2
    BLUE = 3
```
Sometimes it’s useful to access members in enumerations programmatically (i.e. situations where Color.RED won’t do because the exact color is not known at program-writing time). Enum allows such access:
```
Color(1)
<Color.RED: 1>

Color(3)
<Color.BLUE: 3>
```
If you want to access enum members by name, use item access:
```
Color['RED']
<Color.RED: 1>

Color['GREEN']
<Color.GREEN: 2>
```
```
New: Get the directory of a python script.

This can be useful to create relative paths between the script parts in a way that you can still run the script from another directory.
```
from pathlib import Path

script_dir = Path(__file__).parent
script_path = script_dir / '../another-file'
```

GitPython

New: Checking out an existing branch.

heads = repo.heads
develop = heads.develop
repo.head.reference = develop

Pandas

New: Try FireDuck!.

NOTE: you might as well use FireDucks as it has the same API interface and is waaay faster. The good thing is that you only need to add to the top of your code import fireducks.pandas as pd and everything should work (I haven't tried myself).

Elasticsearch

New: Delete documents from all indices in an elasticsearch cluster.

ES_HOST="${1:-http://localhost:9200}"
DEFAULT_SETTING="5"              # Target default value (5%)

INDICES=$(curl -s -XGET "$ES_HOST/_cat/indices?h=index")

for INDEX in $INDICES; do
  echo "Processing index: $INDEX"

  # Close the index to modify static settings
  curl -s -XPOST "$ES_HOST/$INDEX/_close" > /dev/null

 # Update expunge_deletes_allowed to 1%
  curl -s -XPUT "$ES_HOST/$INDEX/_settings" -H 'Content-Type: application/json' -d'
{
    "index.merge.policy.expunge_deletes_allowed": "0"
  }' > /dev/null

  # Reopen the index
  curl -s -XPOST "$ES_HOST/$INDEX/_open" > /dev/null

  # Trigger forcemerge (async)
  # curl -s -XPOST "$ES_HOST/$INDEX/_forcemerge?only_expunge_deletes=true&wait_for_completion=false" > /dev/null
  echo "Forcemerge triggered for $INDEX"
  curl -s -XPOST "$ES_HOST/$INDEX/_forcemerge?only_expunge_deletes=true" > /dev/null &
  echo "Waiting until all forcemerge tasks are done"
  while curl -s $ES_HOST/_cat/tasks\?v  | grep forcemerge > /dev/null ; do
    curl -s $ES_HOST/_cat/indices | grep $INDEX
    sleep 10
  done

  # Close the index again
  curl -s -XPOST "$ES_HOST/$INDEX/_close" > /dev/null

  # Update to the new default (5%)
  curl -s -XPUT "$ES_HOST/$INDEX/_settings" -H 'Content-Type: application/json' -d'
  {
    "index.merge.policy.expunge_deletes_allowed": "'"$DEFAULT_SETTING"'"
  }' > /dev/null

  # Reopen the index
  curl -s -XPOST "$ES_HOST/$INDEX/_open" > /dev/null
done

echo "Done! All indices updated."

Streamlit

New: Show a spinner while the data is loading.

st.title("Title")

with st.spinner("Loading data..."):
    data = long_process()
st.markdown('content shown once the data is loaded')

New: Deploy in docker.

Here's an example Dockerfile that you can add to the root of your directory

FROM python:3.11-slim

WORKDIR /app

RUN apt-get update && apt-get install -y \
  build-essential \
  curl \
  software-properties-common \
  && rm -rf /var/lib/apt/lists/*

COPY . .

RUN pip3 install -r requirements.txt

EXPOSE 8501

HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health

ENTRYPOINT ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]

While you debug you may want to replace the COPY . . to:

COPY requirements.txt .

RUN pip3 install -r requirements.txt

COPY app.py .

So that the build iterations are faster.

You can build it with docker build -t streamlit ., then test it with docker run -p 8501:8501 streamlit

Once you know it's working you can create a docker compose

services:
  streamlit:
    image: hm2025_nodos
    container_name: my_app
    env_file:
      - .env
    ports:
      - "8501:8501"
   healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8501/_stcore/health"]
      interval: 30s
      timeout: 10s
      retries: 5

If you use swag from linuxserver you can expose the service with the next nginx configuration:

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name my_app.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
       #auth_request /auth;
        #error_page 401 =200 /login;

        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_streamlit my_container_name;
        proxy_pass http://$upstream_streamlit:8501;
    }
}

And if you save your docker-compose.yaml file into /srv/streamlit you can use the following systemd service to automatically start it on boot.

``ini [Unit] Description=my_app Requires=docker.service After=docker.service

[Service] Restart=always User=root Group=docker WorkingDirectory=/srv/streamlit TimeoutStartSec=100 RestartSec=2s ExecStart=/usr/bin/docker compose -f docker-compose.yaml up ExecStop=/usr/bin/docker compose -f docker-compose.yaml down

[Install] WantedBy=multi-user.target ```

DevSecOps

Velero

New: Difference between sync and refresh.
- Sync: Reconciles the current cluster state with the target state in git.
- Refresh: Fetches the latest manifests from git and compares the diff with the live state.
- Hard Refresh: Clears any caches and does a refresh.
For more information read 1, 2, 3
New: Add api and library docs.

Using the API

There is a python library

New: ArgoCD commandline installation.

curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
install -m 555 argocd-linux-amd64 ~/.local/bin/argocd
rm argocd-linux-amd64

New: ArgoCD commandline usage.

Login into the server

The argocd login command is the first step in interacting with the Argo CD API. This command allows you to authenticate yourself, setting up a secure connection between your terminal and the Argo CD server. You’ll need to provide your server’s URL and your credentials. There are three different ways to login, I found that the --core is the most useful as it will use your kubernetes credentials.
```
argocd login your.argocd.url.com --core --name production
```
Be careful thought that you can't set different argocd context for different clusters using the --core even though you set the --kube-context flag. The config file ~/.config/argocd/config shows that it's using whatever kubernetes context you're using. So be careful that you're applying it in the correct one!

Set an argocd context

The argocd context command is used to manage your Argo CD contexts. A context is a configuration that represents a Kubernetes cluster, user, and namespace. You can use this command to switch Argo CD between different contexts, allowing you to manage multiple Kubernetes namespaces and clusters from a single terminal.

You can see the different contexts with argocd context

Get the list of applications
```
argocd app list
```
Refresh an application
```
argocd app get app_name --refresh
```
Show the diff of an application
```
argocd app diff app_name
```
Sync an application
```
argocd app sync app_name
```
New: More not there yet features.
- Python library: I have found none
- Argocd TUI: I have found none that is updated
New: Optimizing Kubernetes Cluster Node Count: A Strategic Approach.

Reducing the number of nodes in a Kubernetes cluster is a critical strategy for controlling cloud infrastructure costs without compromising system reliability. Here are key best practices to help organizations right-size their Kubernetes deployments:

1. Availability Zone Consolidation

Carefully evaluate the number of availability zones (AZs) used in your cluster. While multi-AZ deployments provide redundancy, using too many zones can: - Increase infrastructure complexity - Raise management overhead - Unnecessarily distribute resources - Increase cost without proportional benefit

Recommendation: Aim for a balanced approach, typically 3 AZs, which provides robust redundancy while allowing more efficient resource consolidation.

2. Intelligent Node Sizing and Management

Implement sophisticated node management strategies:

Node Provisioning Optimization - Use tools like Karpenter to dynamically manage node sizing - Continuously analyze and adjust node types based on actual workload requirements - Consolidate smaller nodes into fewer, more efficiently sized instances

Overhead Calculation Regularly assess system and Kubernetes overhead: - Calculate total system resource consumption - Identify underutilized resources - Understand the overhead percentage for different node types - Make data-driven decisions about node scaling

3. Advanced Pod Autoscaling Techniques

Horizontal Pod Autoscaling (HPA) - Implement HPA for workloads with variable load - Automatically adjust pod count based on CPU/memory utilization - Ensure efficient resource distribution across existing nodes

Vertical Pod Autoscaling (VPA) - Use VPA in recommendation mode initially - Carefully evaluate automated resource adjustments - Manually apply recommendations to prevent potential service disruptions

4. Workload Optimization Strategies

High Availability Considerations - Ensure critical workloads have robust high availability configurations - Design applications to tolerate node failures gracefully - Implement pod disruption budgets to maintain service reliability

Resource Right-Sizing - Conduct thorough analysis of actual resource utilization - Avoid over-provisioning by matching resource requests to actual usage - Use monitoring tools to gain insights into workload characteristics

5. Continuous Monitoring and Refinement
- Implement comprehensive monitoring of cluster performance
- Regularly review node utilization metrics
- Create feedback loops for continuous optimization
- Develop scripts or use tools to collect and analyze resource usage data
New: Upgrading.

You can no longer use the latest tag as it can lead to unintentional updates and potentially broken setups.

The tag will not be removed, however it will also not be updated past 2025.2.

They strongly recommend the use of a specific version tag for authentik instances' container images like :2025.2.

New: Get the values of a chart.

helm show values zammad --repo https://zammad.github.io/zammad-helm --version 14.0.1

New: Timestamp Issues ("entry too far behind").

The most frequent error shows log entries being rejected because their timestamps are too old.

This suggests either:
- Clock synchronization issues between your log sources and Loki
- Delayed log shipping/buffering
- Replay of old logs
To solve this:
- Check that your hosts' clocks are sync
- Adjust Loki's ingestion window in your config:
```
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h # Increase from default (usually 1h)
```
You can also prune the logs. For example in the case of a docker container (named dawarich_app) you can:
```
sudo truncate -s 0 /var/lib/docker/containers/$(docker inspect -f '{{.Id}}' dawarich_app)/$(docker inspect -f '{{.Id}}' dawarich_app)-json.log
```
New: Ingestion rate limit exceeded for user.

Increase rate limits in Loki config:
```
limits_config:
ingestion_rate_mb: 8 # Increase from 4MB default
  ingestion_burst_size_mb: 16
```
Also check which logs are triggering this rate limit because it may be the case that the amount of logs is too great due to an error.

New: Recursively pull a copy of all helm charts used by an argocd repository.

Including the dependencies of the dependencies.

import argparse
import logging
import subprocess
import sys
from pathlib import Path
from typing import Dict, List, Set

import yaml

class HelmChartPuller:
    def __init__(self):
        self.pulled_charts: Set[str] = set()
        self.setup_logging()

    def setup_logging(self):
        logging.basicConfig(
            level=logging.INFO,
            format="[%(asctime)s] %(levelname)s: %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S",
        )
        self.logger = logging.getLogger(__name__)

    def parse_chart_yaml(self, chart_file: Path) -> Dict:
        """Parse Chart.yaml file and return its contents."""
        try:
            with open(chart_file, "r", encoding="utf-8") as f:
                return yaml.safe_load(f) or {}
        except Exception as e:
            self.logger.error(f"Failed to parse {chart_file}: {e}")
            return {}

    def get_dependencies(self, chart_data: Dict) -> List[Dict]:
        """Extract dependencies from chart data."""
        return chart_data.get("dependencies", [])

    def is_chart_pulled(self, name: str, version: str) -> bool:
      """Check if chart has already been pulled."""
        chart_id = f"{name}-{version}"
        return chart_id in self.pulled_charts

    def mark_chart_pulled(self, name: str, version: str):
        """Mark chart as pulled to avoid duplicates."""
        chart_id = f"{name}-{version}"
        self.pulled_charts.add(chart_id)

    def pull_chart(self, name: str, version: str, repository: str) -> bool:
        """Pull a Helm chart using appropriate method (OCI or traditional)."""
        if self.is_chart_pulled(name, version):
            self.logger.info(f"Chart {name}-{version} already pulled, skipping")
            return True

        self.logger.info(f"Pulling chart: {name} version {version} from {repository}")

        try:
            if repository.startswith("oci://"):
                oci_url = f"{repository}/{name}"
                cmd = ["helm", "pull", oci_url, "--version", version, "--untar"]
            else:
                cmd = [
                    "helm",
                    "pull",
                    name,
                    "--repo",
                    repository,
                    "--version",
                    version,
                    "--untar",
                ]

            result = subprocess.run(cmd, capture_output=True, text=True, check=True)
            self.logger.info(f"Successfully pulled chart: {name}-{version}")
            self.mark_chart_pulled(name, version)
            return True

        except subprocess.CalledProcessError as e:
            self.logger.error(f"Failed to pull chart {name}-{version}: {e.stderr}")
            return False

    def process_chart_dependencies(self, chart_file: Path):
        """Process dependencies from a Chart.yaml file recursively."""
        self.logger.info(f"Processing dependencies from: {chart_file}")

        chart_data = self.parse_chart_yaml(chart_file)
        if not chart_data:
            return

        dependencies = self.get_dependencies(chart_data)
        if not dependencies:
            self.logger.info(f"No dependencies found in {chart_file}")
            return

        for dep in dependencies:
            name = dep.get("name", "")
           version = dep.get("version", "")
            repository = dep.get("repository", "")

            if not all([name, version, repository]):
                self.logger.warning(f"Incomplete dependency in {chart_file}: {dep}")
                continue

            if self.pull_chart(name, version, repository):
                pulled_chart_dir = Path.cwd() / name
                if pulled_chart_dir.is_dir():
                    dep_chart_file = pulled_chart_dir / "Chart.yaml"
                    if dep_chart_file.is_file():
                        self.logger.info(
                            f"Found Chart.yaml in pulled dependency: {dep_chart_file}"
                        )
                        self.process_chart_dependencies(dep_chart_file)

    def find_chart_files(self, search_dir: Path) -> List[Path]:
        """Find all Chart.yaml files in the given directory."""
        self.logger.info(f"Searching for Chart.yaml files in: {search_dir}")
        return list(search_dir.rglob("Chart.yaml"))

    def check_dependencies(self):
        """Check if required dependencies are available."""
        try:
            subprocess.run(["helm", "version"], capture_output=True, check=True)
        except (subprocess.CalledProcessError, FileNotFoundError):
            self.logger.error("helm command not found. Please install Helm.")
            sys.exit(1)

        try:
            import yaml
        except ImportError:
            self.logger.error(
                "PyYAML module not found. Install with: pip install PyYAML"
            )
            sys.exit(1)

    def run(self, target_dir: str):
        """Main execution method."""
        self.check_dependencies()

        target_path = Path(target_dir)
        if not target_path.is_dir():
            self.logger.error(f"Directory '{target_dir}' does not exist")
            sys.exit(1)

        self.logger.info(f"Starting to process Helm charts in: {target_path}")
        self.logger.info(f"Charts will be pulled to current directory: {Path.cwd()}")

        chart_files = self.find_chart_files(target_path)
        if not chart_files:
            self.logger.info("No Chart.yaml files found")
            return

        for chart_file in chart_files:
            self.logger.info(f"Found Chart.yaml: {chart_file}")
            self.process_chart_dependencies(chart_file)

        self.logger.info(
            f"Completed processing. Total unique charts pulled: {len(self.pulled_charts)}"
        )
        if self.pulled_charts:
            self.logger.info(f"Pulled charts: {', '.join(sorted(self.pulled_charts))}")

def main():
    parser = argparse.ArgumentParser(
        description="Recursively pull Helm charts and their dependencies from Chart.yaml files"
    )
    parser.add_argument("directory", help="Directory to search for Chart.yaml files")

    args = parser.parse_args()

    puller = HelmChartPuller()
    puller.run(args.directory)

if __name__ == "__main__":
    main()

New: Cordon all arm64 nodes.

kubectl get nodes -l kubernetes.io/arch=arm64 -o jsonpath='{.items[*].metadata.name}' | xargs kubectl cordon

New: Search all the container images in use that match a desired string.

set -e

log() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
}

usage() {
  echo "Usage: $0"
  echo "Describes all pods in all namespaces and greps for images containing 'bitnami'"
  exit 1
}

check_dependencies() {
  if ! command -v kubectl >/dev/null 2>&1; then
    log "Error: kubectl command not found"
    exit 1
  fi

  # Test kubectl connectivity
  if ! kubectl cluster-info >/dev/null 2>&1; then
    log "Error: Cannot connect to Kubernetes cluster"
    exit 1
  fi
}

find_bitnami_images() {
  log "Getting all pods from all namespaces..."

  # Get all pods from all namespaces and describe them
  kubectl get pods --all-namespaces -o wide --no-headers | while read -r namespace name ready status restarts age ip node nominated readiness; do
    log "Describing pod: $namespace/$name"

    # Describe the pod and grep for bitnami images
    description=$(kubectl describe pod "$name" -n "$namespace" 2>/dev/null)

    # Look for image lines containing bitnami
    bitnami_images=$(echo "$description" | grep -i "image:" | grep -i "bitnami" || true)

    if [[ -n "$bitnami_images" ]]; then
      echo "========================================="
      echo "Pod: $namespace/$name"
      echo "Status: $status"
      echo "Bitnami Images Found:"
      echo "$bitnami_images"
      echo "========================================="
      echo
    fi
  done
}

main() {
  if [[ $# -ne 0 ]]; then
    usage
  fi

  check_dependencies

  log "Starting search for Bitnami images in all pods across all namespaces"
  find_bitnami_images
  log "Search completed"
}

main "$@"

New: Force the removal of a node from the cluster.

To force the removal of a node from a Kubernetes cluster, you have several options depending on your situation:

To prevent new pods from being scheduled while you prepare:
```
kubectl cordon <node-name>
```
1. Graceful Node Removal (Recommended)

First, try the standard approach:
```
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

kubectl delete node <node-name>
```
2. Force Removal When Node is Unresponsive

If the node is unresponsive or the graceful removal fails:
```
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force --grace-period=0

kubectl delete node <node-name>
```
Immediate Forced Removal

For emergency situations where you need immediate removal:
```
kubectl delete node <node-name> --force --grace-period=0
```
Common Drain Options
- --ignore-daemonsets: Ignores DaemonSet pods (they'll be recreated anyway)
- --delete-emptydir-data: Deletes pods using emptyDir volumes
- --force: Forces deletion of pods not managed by controllers
- --grace-period=0: Immediately kills pods without waiting
- --timeout=300s: Sets timeout for the drain operation
New: Raise alert when value is empty.

Using vector(0)

One way to solve it is to use the vector(0) operator with the operation or on() vector(0)
```
(count_over_time({filename="/var/log/mail.log"} |= `Mail is sent` [24h]) or on() vector(0)) < 1
```
Using unless

If you're doing an aggregation over a label this approach won't work because it will add a new time series with value 0. In those cases use a broader search that includes other logs and the unless operator:
```
(sum by(hostname) (count_over_time({job="systemd-journal"} [1h]))
unless
sum by(hostname) (count_over_time({service_name="watchtower"} [1d]))) > 0
```
This will return a value > 0 for any hostname that has systemd-journal logs but no watchtower logs in the past day, which is perfect for alerting conditions.
New: Upgrade postgres.

Using dump and import

Dump your database

Dump your existing database with a command similar to docker compose exec postgresql pg_dump -U authentik -d authentik -cC > upgrade_backup_12.sql.

Before continuing, ensure the SQL dump file upgrade_backup_12.sql includes all your database content.

Stop your application stack

Stop all services with docker compose down.

Backup your existing database

Move the directory where your data is to a new one: mv /path/to/database /path/to/v12-backup

Modify your docker-compose.yml file

Update the PostgreSQL service image from docker.io/library/postgres:12-alpine to docker.io/library/postgres:17-alpine.

Add network_mode: none and comment out any network directive to prevent connections being established to the database during the upgrade.

Recreate the database container

Pull new images and re-create the PostgreSQL container: docker compose pull && docker compose up --force-recreate -d postgresql

Apply your backup to the new database: cat upgrade_backup_12.sql | docker compose exec -T postgresql psql -U authentik

Remove the network configuration setting network_mode: none that you added to the Compose file in the previous step.

Bring the service up

Start again the service with docker compose up and see that everything is working as expected.
New: Debug a failed backup.

To get the available backups you can use:
```
velero get backups
```
Check the ones that have Failed in the name and then see the logs with:
```
velero backup logs backup-1h-20251215113154 | grep -v 'level=info'
```

vector

New: Add the stats of the Enterprise Capacity seagate disk.

Specs	IronWolf	IronWolf Pro	Exos 7E8 8TB	Exos 7E10 8TB	Exos X18 16TB	Enterpri. Capacity
Bays	1-8	1-24	?	?	?	?
Capacity	1-12TB	2-20TB	8TB	8TB	16 TB	10 TB
RPM	5,400 RPM (3-6TB)	7200 RPM	7200 RPM	7200 RPM	7200 RPM	7200 RPM
RPM	5,900 RPM (1-3TB)	7200 RPM	7200 RPM	7200 RPM	7200 RPM	7200 RPM
RPM	7,200 RPM (8-12TB)	7200 RPM	7200 RPM	7200 RPM	7200 RPM	7200 RPM
Speed	180MB/s (1-12TB)	214-260MB/s (4-18TB)	249 MB/s	255 MB/s	258 MB/s	254 MB/s
Cache	64MB (1-4TB)	256 MB	256 MB	256 MB	256 MB	256 MB
Cache	256MB (3-12TB)	256 MB	256 MB	256 MB	256 MB	256 MB
Power Consumption	10.1 W	10.1 W	12.81 W	11.03 W	9.31 W	8 W
Power Consumption Rest	7.8 W	7.8 W	7.64 W	7.06 W	5.08 W	4.5 W
Workload	180TB/yr	300TB/yr	550TB/yr	550TB/yr	550TB/yr	< 550TB/yr
MTBF	1 million	1 million	2 millions	2 millions	2.5 millions	2.5 millions
Noise idle	?	?	?	?	?	3.0 bels max
Noise performance seek	?	?	?	?	?	3.4 bels max

diff --git a/mkdocs.yml b/mkdocs.yml index 7d0201801c..ade48234b0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -5,19 +5,18 @@ site_author: Lyz site_url: https://lyz-code.github.io/blue-book nav: - Introduction: index.md - - Projects: projects.md - Activism: - activism.md - Antifascism: - antifascism.md - Antifascist Actions: antifascist_actions.md - Hacktivism: - - Gatherings: - - hacktivist_gatherings.md - - Chaos Communication Congress: ccc.md - - Collectives: hacktivist_collectives.md - - Tools: - - Mobile Verification Toolkit: mobile_verification_toolkit.md + - Gatherings: + - hacktivist_gatherings.md + - Chaos Communication Congress: ccc.md + - Collectives: hacktivist_collectives.md + - Tools: + - Mobile Verification Toolkit: mobile_verification_toolkit.md - Anti-Colonialism: anticolonialism.md - Anti-Transphobia: antitransphobia.md - Anti-Racism: antiracism.md @@ -27,6 +26,7 @@ nav: - Feminism: - Privileges: feminism/privileges.md - Palestine: palestine.md + - Anarchism: anarchism.md - Memoria histórica: memoria_historica.md - Anti-Tourism: antitourism.md - Mentoring: mentoring.md @@ -36,153 +36,154 @@ nav: - Laboral: laboral.md - Collaborating tools: collaborating_tools.md - Conference organisation: conference_organisation.md - - Ludditest: luddites.md + - Ludditest: luddites.md - Life Management: - life_management.md - Time management: - - time_management.md - - Time management abstraction levels: time_management_abstraction_levels.md - - Action Management: action_management.md - - Roadmap Adjustment: - - roadmap_adjustment.md - - Strategy: strategy.md - - Systems Thinking: systems_thinking.md - - Roadmap Management Tools: - - roadmap_tools.md - - Org Mode: - - orgmode.md - - Org-rw: org_rw.md - - Orgzly: orgzly.md - - OpenProject: openproject.md - - Habit management: habit_management.md - - Interruption management: - - interruption_management.md - - Interruption Management Analysis: - - Work Interruption Analysis: work_interruption_analysis.md - - Personal Interruption Analysis: personal_interruption_analysis.md - - Week management: week_management.md - - Calendar management: - - calendar_management.md - - Calendar automation: - - vdirsyncer: vdirsyncer.md - - Calendar clients: - - Khal: khal.md - - Gancio: gancio.md - - Time management theories: - - Getting Things Done: gtd.md + - time_management.md + - Time management abstraction levels: time_management_abstraction_levels.md + - Action Management: action_management.md + - Roadmap Adjustment: + - roadmap_adjustment.md + - Strategy: strategy.md + - Systems Thinking: systems_thinking.md + - Roadmap Management Tools: + - roadmap_tools.md + - Org Mode: + - orgmode.md + - Org-rw: org_rw.md + - Orgzly: orgzly.md + - OpenProject: openproject.md + - Habit management: habit_management.md + - Interruption management: + - interruption_management.md + - Interruption Management Analysis: + - Work Interruption Analysis: work_interruption_analysis.md + - Personal Interruption Analysis: personal_interruption_analysis.md + - Week management: week_management.md + - Calendar management: + - calendar_management.md + - Calendar automation: + - vdirsyncer: vdirsyncer.md + - Calendar clients: + - Khal: khal.md + - Gancio: gancio.md + - Time management theories: + - Getting Things Done: gtd.md - Life chores management: - - Trip management: - - Route management: route_management.md - - Map management: map_management.md - - Food management: food_management.md - - Stock management: - - Grocy: grocy_management.md - - Money management: - - money_management.md - - beancount: - - beancount.md - - bean-sql: bean_sql.md - - Fava Dashboards: fava_dashboards.md - - Tools management: - - tool_management.md - - Email management: - - email_management.md - - Email automation: - - email_automation.md - - Email automation tools: - - mbsync: mbsync.md - - mirador: mirador.md - - afew: afew.md - - notmuch: notmuch.md - - Email automation libraries: - - mailbox: mailbox.md - - IMAP: - - IMAP library comparison: python_imap.md - - imap-tools: imap_tools.md - - Email clients: - - himalaya: himalaya.md - - alot: alot.md - - Email protocols: - - Maildir: maildir.md - - Instant Messages Management: - - instant_messages_management.md - - XMPP/Jabber: - - Dino: dino.md - - Gajim: gajim.md - - Profanity: profanity.md - - Matrix: - - matrix.md - - Matrix Highlight: matrix_highlight.md - - Rocketchat: rocketchat.md - - Computer configuration management: configuration_management.md + - Trip management: + - Route management: route_management.md + - Map management: map_management.md + - Food management: food_management.md + - Stock management: + - Grocy: grocy_management.md + - Money management: + - money_management.md + - beancount: + - beancount.md + - bean-sql: bean_sql.md + - Fava Dashboards: fava_dashboards.md + - Tools management: + - tool_management.md + - Email management: + - email_management.md + - Email automation: + - email_automation.md + - Email automation tools: + - mbsync: mbsync.md + - mirador: mirador.md + - afew: afew.md + - notmuch: notmuch.md + - Email automation libraries: + - mailbox: mailbox.md + - IMAP: + - IMAP library comparison: python_imap.md + - imap-tools: imap_tools.md + - Email clients: + - himalaya: himalaya.md + - alot: alot.md + - Email protocols: + - Maildir: maildir.md + - Instant Messages Management: + - instant_messages_management.md + - XMPP/Jabber: + - Dino: dino.md + - Gajim: gajim.md + - Profanity: profanity.md + - Matrix: + - matrix.md + - Matrix Highlight: matrix_highlight.md + - Signal: signal.md + - Rocketchat: rocketchat.md + - Computer configuration management: configuration_management.md - Content Management: - - Music Management: - - music_management.md - - MusicBrainz: musicbrainz.md - - Mopidy: mopidy.md - - Beets: beets.md - - Koel: koel.md - - yt-dlp: yt-dlp.md - - Book Management: - - book_management.md - - Bookwyrm: bookwyrm.md - - Movies Management: - - Jellyfin: jellyfin.md - - Ombi: ombi.md - - Mediatracker: mediatracker.md - - ffmpeg: ffmpeg.md - - transcoding: - - transcoding.md - - unmanic: unmanic.md - - Kodi: kodi.md - - News Management: - - news_management.md - - RSS: rss.md - - Wallabag: wallabag.md - - Photo management: - - photo_self_hosted.md - - Immich: immich.md - - Video management: video_management.md - - Videogames management: - - moonlight: moonlight.md - - retroarch: retroarch.md + - Music Management: + - music_management.md + - MusicBrainz: musicbrainz.md + - Mopidy: mopidy.md + - Beets: beets.md + - Koel: koel.md + - yt-dlp: yt-dlp.md + - Book Management: + - book_management.md + - Bookwyrm: bookwyrm.md + - Movies Management: + - Jellyfin: jellyfin.md + - Ombi: ombi.md + - Mediatracker: mediatracker.md + - ffmpeg: ffmpeg.md + - transcoding: + - transcoding.md + - unmanic: unmanic.md + - Kodi: kodi.md + - News Management: + - news_management.md + - RSS: rss.md + - Wallabag: wallabag.md + - Photo management: + - photo_self_hosted.md + - Immich: immich.md + - Video management: video_management.md + - Videogames management: + - moonlight: moonlight.md + - retroarch: retroarch.md - Knowledge Management: - - knowledge_management.md - - Spaced Repetition: - - spaced_repetition.md - - Anki: anki.md - - Mochi: mochi.md - - Analytical web reading: - - analytical_web_reading.md - - Hypothesis: linux/hypothesis.md - - Digital Gardens: - - digital_garden.md - - mkdocs: linux/mkdocs.md - - Build your own Digital Garden: writing/build_your_own_wiki.md - - Forking this garden: forking_this_wiki.md - - Aleph: aleph.md - - Wordpress: wordpress.md - - Relationship Management: - - relationship_management.md - - monica: linux/monica.md + - knowledge_management.md + - Spaced Repetition: + - spaced_repetition.md + - Anki: anki.md + - Mochi: mochi.md + - Analytical web reading: + - analytical_web_reading.md + - Hypothesis: linux/hypothesis.md + - Digital Gardens: + - digital_garden.md + - mkdocs: linux/mkdocs.md + - Build your own Digital Garden: writing/build_your_own_wiki.md + - Forking this garden: forking_this_wiki.md + - Aleph: aleph.md + - Wordpress: wordpress.md + - Relationship Management: + - relationship_management.md + - monica: linux/monica.md - Process Automation: - - process_automation.md - - AI: - - Open WebUI: openwebui.md - - NLP: - - spacy: spacy.md - - Virtual Assistant: virtual_assistant.md - - Monitor web changes: - - Changedetection.io: changedetection.md - - Monitorizar billetes de renfe: renfe.md + - process_automation.md + - AI: + - Open WebUI: openwebui.md + - NLP: + - spacy: spacy.md + - Virtual Assistant: virtual_assistant.md + - Monitor web changes: + - Changedetection.io: changedetection.md + - Monitorizar billetes de renfe: renfe.md - Torrent management: - - torrents.md - - qBittorrent: qbittorrent.md - - Rtorrent: rtorrent.md - - Unpackerr: unpackerr.md - - Life logging: - - life_logging.md - - ActivityWatch: activitywatch.md + - torrents.md + - qBittorrent: qbittorrent.md + - Rtorrent: rtorrent.md + - Unpackerr: unpackerr.md + - Life logging: + - life_logging.md + - ActivityWatch: activitywatch.md - Health: - Sleep: sleep.md - Teeth: @@ -213,7 +214,8 @@ nav: - Create the documentation repository: >- coding/python/python_project_template/python_docs.md - Load config from YAML: coding/python/python_config_yaml.md - - Configure SQLAlchemy to use the MariaDB/Mysql backend: >- + - Configure SQLAlchemy to use the MariaDB/Mysql backend: + >- coding/python/python_project_template/python_sqlalchemy_mariadb.md - Configure Docker to host the application: >- coding/python/python_project_template/python_docker.md @@ -269,9 +271,9 @@ nav: - Python Mysql: python_mysql.md - pythonping: pythonping.md - Python Prometheus: python-prometheus.md - - Python Telegram: - - python-telegram.md - - pytelegrambotapi: pytelegrambotapi.md + - Python Telegram: + - python-telegram.md + - pytelegrambotapi: pytelegrambotapi.md - Python VLC: python_vlc.md - Playwright: playwright.md - Plotly: coding/python/plotly.md @@ -299,13 +301,13 @@ nav: - Code Styling: coding/python/python_code_styling.md - Docstrings: coding/python/docstrings.md - Properties: python_properties.md - - Protocols: python_protocols.md + - Protocols: python_protocols.md - Package Management: - - python_package_management.md - - PDM: pdm.md - - pipx: pipx.md - - Pipenv: pipenv.md - - Poetry: python_poetry.md + - python_package_management.md + - PDM: pdm.md + - pipx: pipx.md + - Pipenv: pipenv.md + - Poetry: python_poetry.md - Lazy loading: lazy_loading.md - Plugin System: python_plugin_system.md - Profiling: python_profiling.md @@ -351,65 +353,66 @@ nav: - JWT: devops/jwt.md - React: coding/react/react.md - Coding tools: - - IDES: - - Vim: - - vim.md - - Vim configuration: - - vim_config.md - - Vim Keymaps: vim_keymaps.md - - Vim Package Manager: - - vim_plugin_managers.md - - LazyVim: lazyvim.md - - Packer: vim_packer.md - - UI management configuration: - - Vim foldings: vim_foldings.md - - Vim movement: vim_movement.md - - Tabs vs Buffers: vim_tabs.md - - File management configuration: - - NeoTree: neotree.md - - Telescope: telescope.md - - Editing specific configuration: - - vim_editor_plugins.md - - Vim formatters: vim_formatters.md - - Vim autocomplete: vim_completion.md - - Vim markdown: vim_markdown.md - - Vim spelling: vim_spelling.md - - Vim autosave: vim_autosave.md - - Coding specific configuration: - - vim_coding_plugins.md - - Treesitter: vim_treesitter.md - - LSP: vim_lsp.md - - Snippets: luasnip.md - - DAP: vim_dap.md - - Git management configuration: - - vim_git.md - - Diffview: diffview.md - - gitsigns: gitsigns.md - - Testing management configuration: vim_testing.md - - Email management: vim_email.md - - Other Vim Plugins: - - linux/vim/vim_plugins.md - - Vim Snippets: vim_snippets.md - - Vim Troubleshooting: vim_troubleshooting.md - - Neovim Plugin Development: vim_plugin_development.md - - Vi vs Vim vs Neovim: vim_vs_neovim.md - - Tridactyl: tridactyl.md - - VSCodium: vscodium.md - - Coding with AI: ai_coding.md - - Git: - - git.md - - Github cli: gh.md - - Forgejo: forgejo.md - - Gitea: gitea.md - - Data orchestrators: - - data_orchestrators.md - - Kestra: kestra.md - - memorious: memorious.md - - Scrapers: - - morph.io: morph_io.md - - ETL: - - Singer: singer.md - - Espanso: espanso.md + - IDES: + - Vim: + - vim.md + - Vim configuration: + - vim_config.md + - Vim Keymaps: vim_keymaps.md + - Vim Package Manager: + - vim_plugin_managers.md + - LazyVim: lazyvim.md + - Packer: vim_packer.md + - UI management configuration: + - Vim foldings: vim_foldings.md + - Vim movement: vim_movement.md + - Tabs vs Buffers: vim_tabs.md + - File management configuration: + - NeoTree: neotree.md + - Telescope: telescope.md + - Editing specific configuration: + - vim_editor_plugins.md + - Vim formatters: vim_formatters.md + - Vim autocomplete: vim_completion.md + - Vim markdown: vim_markdown.md + - Vim spelling: vim_spelling.md + - Vim autosave: vim_autosave.md + - Coding specific configuration: + - vim_coding_plugins.md + - Treesitter: vim_treesitter.md + - LSP: vim_lsp.md + - Snippets: luasnip.md + - DAP: vim_dap.md + - Git management configuration: + - vim_git.md + - Diffview: diffview.md + - gitsigns: gitsigns.md + - Testing management configuration: vim_testing.md + - Email management: vim_email.md + - Other Vim Plugins: + - linux/vim/vim_plugins.md + - Vim Snippets: vim_snippets.md + - Vim Troubleshooting: vim_troubleshooting.md + - Neovim Plugin Development: vim_plugin_development.md + - Vi vs Vim vs Neovim: vim_vs_neovim.md + - Tridactyl: tridactyl.md + - VSCodium: vscodium.md + - Coding with AI: ai_coding.md + - Git: + - git.md + - Github cli: gh.md + - Forgejo: forgejo.md + - Gitea: gitea.md + - Radicle: radicle.md + - Data orchestrators: + - data_orchestrators.md + - Kestra: kestra.md + - memorious: memorious.md + - Scrapers: + - morph.io: morph_io.md + - ETL: + - Singer: singer.md + - Espanso: espanso.md - Generic Coding Practices: - How to code: how_to_code.md - Program Versioning: @@ -435,8 +438,8 @@ nav: - Microservices: architecture/microservices.md - Restful APIS: architecture/restful_apis.md - OCR: - - Table parsing: - - Camelot: camelot.md + - Table parsing: + - Camelot: camelot.md - Frontend Development: frontend_development.md - Park programming: park_programming.md - Sponsor: sponsor.md @@ -453,8 +456,8 @@ nav: - Helmfile: devops/helmfile.md - Terraform: terraform.md - Ansible: - - Ansible Snippets: ansible_snippets.md - - Molecule: molecule.md + - Ansible Snippets: ansible_snippets.md + - Molecule: molecule.md - Nix: nix.md - Dotfiles: - dotfiles.md @@ -517,8 +520,8 @@ nav: - S3: devops/aws/s3.md - WAF: aws_waf.md - Databases: - - Redis: architecture/redis.md - - RabbitMQ: rabbitmq.md + - Redis: architecture/redis.md + - RabbitMQ: rabbitmq.md - Continuous Deployment: - ArgoCD: argocd.md - Continuous Integration: @@ -553,21 +556,21 @@ nav: - renovate: renovate.md - letsencrypt: letsencrypt.md - Threat modeling: - - Privacy threat modeling: privacy_threat_modeling.md + - Privacy threat modeling: privacy_threat_modeling.md - Storage: - - storage.md - - NAS: nas.md - - OpenZFS: - - linux/zfs.md - - OpenZFS storage planning: zfs_storage_planning.md - - Sanoid: sanoid.md - - ZFS Prometheus exporter: zfs_exporter.md - - Hard drive health: hard_drive_health.md - - Resilience: - - linux_resilience.md - - Memtest: memtest.md - - watchdog: watchdog.md - - Magic keys: magic_keys.md + - storage.md + - NAS: nas.md + - OpenZFS: + - linux/zfs.md + - OpenZFS storage planning: zfs_storage_planning.md + - Sanoid: sanoid.md + - ZFS Prometheus exporter: zfs_exporter.md + - Hard drive health: hard_drive_health.md + - Resilience: + - linux_resilience.md + - Memtest: memtest.md + - watchdog: watchdog.md + - Magic keys: magic_keys.md - Monitoring: - Monitoring Comparison: monitoring_comparison.md - Prometheus: @@ -587,21 +590,21 @@ nav: devops/prometheus/prometheus_troubleshooting.md - Grafana: grafana.md - Log analysis: - - Loki: - - loki.md - - Logcli: logcli.md - - Promtail: promtail.md - - Graylog: graylog.md - - Elastic Security: elastic_security.md + - Loki: + - loki.md + - Logcli: logcli.md + - Promtail: promtail.md + - Graylog: graylog.md + - Elastic Security: elastic_security.md - SIEM: siem.md - Databases: - - PostgreSQL: - - postgres.md - - Postgres operators: - - postgres_operators.md - - Zalando Postgres operator: zalando_postgres_operator.md - - elasticsearch: linux/elasticsearch.md - - Oracle Database: oracle_database.md + - PostgreSQL: + - postgres.md + - Postgres operators: + - postgres_operators.md + - Zalando Postgres operator: zalando_postgres_operator.md + - elasticsearch: linux/elasticsearch.md + - Oracle Database: oracle_database.md - Authentication: - Authentik: authentik.md - API Management: @@ -613,11 +616,11 @@ nav: - Refinement Template: refinement_template.md - Hardware: - CPU: cpu.md - - RAM: - - ram.md - - ECC RAM: - - ecc.md - - rasdaemon: rasdaemon.md + - RAM: + - ram.md + - ECC RAM: + - ecc.md + - rasdaemon: rasdaemon.md - Power Supply Unit: psu.md - GPU: gpu.md - Pedal PC: pedal_pc.md @@ -627,52 +630,54 @@ nav: - linux.md - Linux Snippets: linux_snippets.md - Distros: - - Libreelec: libreelec.md - - Tails: tails.md + - Libreelec: libreelec.md + - Tails: tails.md - Recovery tools: - - finnix: finnix.md + - finnix: finnix.md - Security tools: - - fail2ban: linux/fail2ban.md - - pass: pass.md - - Wireshark: wireshark.md + - fail2ban: linux/fail2ban.md + - pass: pass.md + - Wireshark: wireshark.md - Sysadmin tools: - - brew: linux/brew.md - - detox: detox.md - - Docker: docker.md - - Watchtower: watchtower.md - - Dynamic DNS: dynamicdns.md - - goaccess: goaccess.md - - Gotify: gotify.md - - HAProxy: linux/haproxy.md - - journald: journald.md - - LUKS: linux/luks/luks.md - - Outrun: outrun.md - - rm: linux/rm.md - - sed: sed.md - - Syncthing: linux/syncthing.md - - Tahoe-LAFS: tahoe.md - - Wake on Lan: wake_on_lan.md - - Wireguard: linux/wireguard.md - - yq: yq.md - - zip: linux/zip.md + - brew: linux/brew.md + - detox: detox.md + - Docker: docker.md + - Watchtower: watchtower.md + - Dynamic DNS: dynamicdns.md + - goaccess: goaccess.md + - Gotify: gotify.md + - HAProxy: linux/haproxy.md + - journald: journald.md + - LUKS: linux/luks/luks.md + - Outrun: outrun.md + - rm: linux/rm.md + - sed: sed.md + - Syncthing: linux/syncthing.md + - Tahoe-LAFS: tahoe.md + - Wake on Lan: wake_on_lan.md + - Wireguard: linux/wireguard.md + - yq: yq.md + - zip: linux/zip.md - Window manager tools: - - dunst: dunst.md - - ferdium: ferdium.md - - i3wm: i3wm.md - - rofi: rofi.md + - dunst: dunst.md + - ferdium: ferdium.md + - i3wm: i3wm.md + - rofi: rofi.md - User tools: - - Browsers: - - google chrome: linux/google_chrome.md - - Chromium: chromium.md - - Hushboard: husboard.md - - Peek: peek.md - - Terminals: - - terminal_comparison.md - - Alacritty: alacritty.md - - Wezterm: wezterm.md - - Kitty: kitty.md - - Instant messaging apps: - - Delta Chat: deltachat.md + - Browsers: + - google chrome: linux/google_chrome.md + - Chromium: chromium.md + - Hushboard: husboard.md + - Peek: peek.md + - Terminals: + - terminal_comparison.md + - Alacritty: alacritty.md + - Wezterm: wezterm.md + - Kitty: kitty.md + - Instant messaging apps: + - Delta Chat: deltachat.md + - Simplex Chat: simplexchat.md + - Android: - Android Tips: android_tips.md - OS: @@ -687,7 +692,6 @@ nav: - Orgzly: orgzly.md - OsmAnd: osmand.md - Seedvault: seedvault.md - - Signal: signal.md - Android SDK Platform tools: android_sdk.md - Arts: - Writing: @@ -703,8 +707,8 @@ nav: - Dancing: - Rave Dances: dancing/rave_dances.md - Swing: - - Shag: shag.md - - Lindy Hop: lindy.md + - Shag: shag.md + - Lindy Hop: lindy.md - Shuffle: - Basics: dancing/shuffle_basics.md - Kicks: dancing/shuffle_kicks.md @@ -729,7 +733,7 @@ nav: - board_games.md - Regicide: regicide.md - Music: - - Sister Rosetta Tharpe: sister_rosetta_tharpe.md + - Sister Rosetta Tharpe: sister_rosetta_tharpe.md - Sudokus: sudokus.md - Drawing: - drawing/drawing.md @@ -741,23 +745,24 @@ nav: - Emojis: emojis.md - Languages: - Castellano: castellano.md - - Galego: + - Galego: - galego.md - Diccionario galego-castelan: diccionario_galego.md + - Esperanto: esperanto.md - Science: - - Artificial Intelligence: - - ai.md - - Speech to text: - - Whisper: whisper.md - - Speech recognition: speech_recognition.md - - Coding by Voice: coding_by_voice.md - - Data Analysis: + - Artificial Intelligence: + - ai.md + - Speech to text: + - Whisper: whisper.md + - Speech recognition: speech_recognition.md + - Coding by Voice: coding_by_voice.md + - Data Analysis: - data_analysis.md - Recommender Systems: >- data_analysis/recommender_systems/recommender_systems.md - Parsers: parsers.md - CSV exploring: - - csvlens: csvlens.md + - csvlens: csvlens.md - Psychology: - psychology.md - The XY Problem: psychology/the_xy_problem.md @@ -765,6 +770,18 @@ nav: - Botany: - Trees: botany/trees.md - Math: math.md + - Reviews: + - Year reviews: year_reviews.md + - Content reviews: + - Books: books.md + # - Movies: movies.md + # - Music: music.md + # - TV Shows: tv_shows.md + - Podcasts: podcasts.md + # - Videogames: videogames.md + # - Boardgames: boardgames.md + # - Streaming channels: streaming_channels.md + - Projects: projects.md - Contact: contact.md

plugins: @@ -825,7 +842,7 @@ theme:

palette: # Light mode - - media: '(prefers-color-scheme: light)' + - media: "(prefers-color-scheme: light)" scheme: default primary: blue grey accent: light blue @@ -834,7 +851,7 @@ theme: name: Switch to dark mode

 # Dark mode

- media: '(prefers-color-scheme: dark)'
- media: "(prefers-color-scheme: dark)" scheme: slate primary: blue grey accent: light blue

New: Suggest to look at the slimbook.

I built a server pretty much the same as the slimbook.
New: Introduce smartctl.

Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T. or SMART) is a monitoring system included in computer hard disk drives (HDDs) and solid-state drives (SSDs). Its primary function is to detect and report various indicators of drive reliability, or how long a drive can function while anticipating imminent hardware failures.

When S.M.A.R.T. data indicates a possible imminent drive failure, software running on the host system may notify the user so action can be taken to prevent data loss, and the failing drive can be replaced and no data is lost.

General information

Accuracy

A field study at Google covering over 100,000 consumer-grade drives from December 2005 to August 2006 found correlations between certain S.M.A.R.T. information and annualized failure rates:
- In the 60 days following the first uncorrectable error on a drive (S.M.A.R.T. attribute 0xC6 or 198) detected as a result of an offline scan, the drive was, on average, 39 times more likely to fail than a similar drive for which no such error occurred.
- First errors in reallocations, offline reallocations (S.M.A.R.T. attributes 0xC4 and 0x05 or 196 and 5) and probational counts (S.M.A.R.T. attribute 0xC5 or 197) were also strongly correlated to higher probabilities of failure.
- Conversely, little correlation was found for increased temperature and no correlation for usage level. However, the research showed that a large proportion (56%) of the failed drives failed without recording any count in the "four strong S.M.A.R.T. warnings" identified as scan errors, reallocation count, offline reallocation, and probational count.
- Further, 36% of failed drives did so without recording any S.M.A.R.T. error at all, except the temperature, meaning that S.M.A.R.T. data alone was of limited usefulness in anticipating failures.
Installation

On Debian systems:
```
sudo apt-get install smartmontools
```
By default when you install it all your drives are checked periodically with the smartd daemon under the smartmontools systemd service.

Usage

Running the tests

Test types

S.M.A.R.T. drives may offer a number of self-tests:
- Short: Checks the electrical and mechanical performance as well as the read performance of the disk. Electrical tests might include a test of buffer RAM, a read/write circuitry test, or a test of the read/write head elements. Mechanical test includes seeking and servo on data tracks. Scans small parts of the drive's surface (area is vendor-specific and there is a time limit on the test). Checks the list of pending sectors that may have read errors, and it usually takes under two minutes.
- Long/extended: A longer and more thorough version of the short self-test, scanning the entire disk surface with no time limit. This test usually takes several hours, depending on the read/write speed of the drive and its size. It is possible for the long test to pass even if the short test fails.
- Conveyance: Intended as a quick test to identify damage incurred during transporting of the device from the drive manufacturer to the computer manufacturer. Only available on ATA drives, and it usually takes several minutes.
Drives remain operable during self-test, unless a "captive" option (ATA only) is requested.

Long test

Start with a long self test with smartctl. Assuming the disk to test is /dev/sdd:
```
smartctl -t long /dev/sdd
```
The command will respond with an estimate of how long it thinks the test will take to complete.

To check progress use:
```
martctl -A /dev/sdd | grep remaining
smartctl -c /dev/sdd | grep remaining
```
Don't check too often because it can abort the test with some drives. If you receive an empty output, examine the reported status with:

`bash smartctl -l selftest /dev/sdd

If errors are shown, check the dmesg as there are usually useful traces of the error.

New: Understanding the tests.

The output of a smartctl command is difficult to read:

smartctl 5.40 2010-03-16 r3077 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     SAMSUNG SpinPoint F2 EG series
Device Model:     SAMSUNG HD502HI
Serial Number:    S1VZJ9CS712490
Firmware Version: 1AG01118
User Capacity:    500,107,862,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 3b
Local Time is:    Wed Feb  9 15:30:42 2011 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:          (6312) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 106) minutes.
Conveyance self-test routine
recommended polling time:      (  12) minutes.
SCT capabilities:            (0x003f)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   099   099   051    Pre-fail  Always       -       2376
  3 Spin_Up_Time            0x0007   091   091   011    Pre-fail  Always       -       3620
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       405
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   253   253   051    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0025   100   100   015    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       717
 10 Spin_Retry_Count        0x0033   100   100   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0012   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       405
 13 Read_Soft_Error_Rate    0x000e   099   099   000    Old_age   Always       -       2375
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
84 End-to-End_Error        0x0033   100   100   000    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       2375
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   084   074   000    Old_age   Always       -       16 (Lifetime Min/Max 16/16)
194 Temperature_Celsius     0x0022   084   071   000    Old_age   Always       -       16 (Lifetime Min/Max 16/16)
195 Hardware_ECC_Recovered  0x001a   100   100   000    Old_age   Always       -       3558
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   098   098   000    Old_age   Always       -       81
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x000a   100   100   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   253   253   000    Old_age   Always       -       0

MART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Checking overall health

Somewhere in your report you'll see something like:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

If it doesn’t return PASSED, you should immediately backup all your data. Your hard drive is probably failing.

That message can also be shown with smartctl -H /dev/sda

Checking the SMART attributes

Each drive manufacturer defines a set of attributes, and sets threshold values beyond which attributes should not pass under normal operation. But they do not agree on precise attribute definitions and measurement units, the following list of attributes is a general guide only.

If one or more attribute have the "prefailure" flag, and the "current value" of such prefailure attribute is smaller than or equal to its "threshold value" (unless the "threshold value" is 0), that will be reported as a "drive failure". In addition, a utility software can send SMART RETURN STATUS command to the ATA drive, it may report three status: "drive OK", "drive warning" or "drive failure".

SMART attributes columns

Every of the SMART attributes has several columns as shown by “smartctl -a ”:

ID: The ID number of the attribute, good for comparing with other lists like Wikipedia: S.M.A.R.T.: Known ATA S.M.A.R.T. attributes because the attribute names sometimes differ. Name: The name of the SMART attribute.
Value: The current, normalized value of the attribute. Higher values are always better (except for temperature for hard disks of some manufacturers). The range is normally 0-100, for some attributes 0-255 (so that 100 resp. 255 is best, 0 is worst). There is no standard on how manufacturers convert their raw value to this normalized one: when the normalized value approaches threshold, it can do linearily, exponentially, logarithmically or any other way, meaning that a doubled normalized value does not necessarily mean “twice as good”.
Worst: The worst (normalized) value that this attribute had at any point of time where SMART was enabled. There seems to be no mechanism to reset current SMART attribute values, but this still makes sense as some SMART attributes, for some manufacturers, fluctuate over time so that keeping the worst one ever is meaningful.
Threshold: The threshold below which the normalized value will be considered “exceeding specifications”. If the attribute type is “Pre-fail”, this means that SMART thinks the hard disk is just before failure. This will “trigger” SMART: setting it from “SMART test passed” to “SMART impending failure” or similar status.
Type: The type of the attribute. Either “Pre-fail” for attributes that are said to indicate impending failure, or “Old_age” for attributes that just indicate wear and tear. Note that one and the same attribute can be classified as “Pre-fail” by one manufacturer or for one model and as “Old_age” by another or for another model. This is the case for example for attribute Seek_Error_Rate (ID 7), which is a widespread phenomenon on many disks and not considered critical by some manufacturers, but Seagate has it as “Pre-fail”.
Raw value: The current raw value that was converted to the normalized value above. smartctl shows all as decimal values, but some attribute values of some manufacturers cannot be reasonably interpreted that way

New: Reacting to SMART Values.

It is said that a drive that starts getting bad sectors (attribute ID 5) or “pending” bad sectors (attribute ID 197; they most likely are bad, too) will usually be trash in 6 months or less. The only exception would be if this does not happen: that is, bad sector count increases, but then stays stable for a long time, like a year or more. For that reason, one normally needs a diagramming / journaling tool for SMART. Many admins will exchange the hard drive if it gets reallocated sectors (ID 5) or sectors “under investigation” (ID 197)

Critical SMART attributes

Of all the attributes I'm going to analyse only the critical ones

Read Error Rate

ID: 01 (0x01) deal: Low +Correlation with probability of failure: not clear

(Vendor specific raw value.) Stores data related to the rate of hardware read errors that occurred when reading data from a disk surface. The raw value has different structure for different vendors and is often not meaningful as a decimal number. For some drives, this number may increase during normal operation without necessarily signifying errors.

Reallocated Sectors Count

ID: 05 (0x05) Ideal: Low Correlation with probability of failure: Strong

Count of reallocated sectors. The raw value represents a count of the bad sectors that have been found and remapped. Thus, the higher the attribute value, the more sectors the drive has had to reallocate. This value is primarily used as a metric of the life expectancy of the drive; a drive which has had any reallocations at all is significantly more likely to fail in the immediate months. If Raw value of 0x05 attribute is higher than its Threshold value, that will reported as "drive warning".

Spin Retry Count

ID: 10 (0x0A) Ideal: Low Correlation with probability of failure: Strong

Count of retry of spin start attempts. This attribute stores a total count of the spin start attempts to reach the fully operational speed (under the condition that the first attempt was unsuccessful). An increase of this attribute value is a sign of problems in the hard disk mechanical subsystem.

Current Pending Sector Count

ID: 197 (0xC5) Ideal: Low Correlation with probability of failure: Strong

Count of "unstable" sectors (waiting to be remapped, because of unrecoverable read errors). If an unstable sector is subsequently read successfully, the sector is remapped and this value is decreased. Read errors on a sector will not remap the sector immediately (since the correct value cannot be read and so the value to remap is not known, and also it might become readable later); instead, the drive firmware remembers that the sector needs to be remapped, and will remap it the next time it has been successfully read.[76]

However, some drives will not immediately remap such sectors when successfully read; instead the drive will first attempt to write to the problem sector, and if the write operation is successful the sector will then be marked as good (in this case, the "Reallocation Event Count" (0xC4) will not be increased). This is a serious shortcoming, for if such a drive contains marginal sectors that consistently fail only after some time has passed following a successful write operation, then the drive will never remap these problem sectors. If Raw value of 0xC5 attribute is higher than its Threshold value, that will reported as "drive warning"

(Offline) Uncorrectable Sector Count

ID: 198 (0xC6) Ideal: Low Correlation with probability of failure: Strong

The total count of uncorrectable errors when reading/writing a sector. A rise in the value of this attribute indicates defects of the disk surface and/or problems in the mechanical subsystem.

In the 60 days following the first uncorrectable error on a drive (S.M.A.R.T. attribute 0xC6 or 198) detected as a result of an offline scan, the drive was, on average, 39 times more likely to fail than a similar drive for which no such error occurred.

Non critical SMART attributes

The next attributes appear to change in the logs but that doesn't mean that there is anything going wrong

Hardware ECC Recovered

ID: 195 (0xC3) Ideal: Varies Correlation with probability of failure: Low

(Vendor-specific raw value.) The raw value has different structure for different vendors and is often not meaningful as a decimal number. For some drives, this number may increase during normal operation without necessarily signifying errors.

New: Monitorization.

To monitor your drive health you can use prometheus with alertmanager for alerts and grafana for dashboards.

Installing the exporter

The prometheus community has it's own smartctl exporter

Using the binary

You can download the latest binary from the repository releases and configure the systemd service

unp smartctl_exporter-0.13.0.linux-amd64.tar.gz
sudo mv smartctl_exporter-0.13.0.linux-amd64/smartctl_exporter /usr/bin

Add the service to /etc/systemd/system/smartctl-exporter.service

[Unit]
Description=smartctl exporter service
After=network-online.target

[Service]
Type=simple
PIDFile=/run/smartctl_exporter.pid
ExecStart=/usr/bin/smartctl_exporter
User=root
Group=root
SyslogIdentifier=smartctl_exporter
Restart=on-failure
RemainAfterExit=no
RestartSec=100ms
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

hen enable it:

sudo systemctl enable smartctl-exporter
sudo service smartctl-exporter start

Using docker

---
services:
  smartctl-exporter:
    container_name: smartctl-exporter
    image: prometheuscommunity/smartctl-exporter
    privileged: true
    user: root
    ports:
      - "9633:9633"

Configuring prometheus

Add the next scraping metrics:

- job_name: smartctl_exporter
  metrics_path: /metrics
  scrape_timeout: 60s
  static_configs:
    - targets: [smartctl-exporter:9633]
      labels:
        hostname: "your-hostname"

Configuring the alerts

Taking as a reference the awesome prometheus rules and this wired post I'm using the next rules:

---
groups:
  - name: smartctl exporter
    rules:
      - alert: SmartDeviceTemperatureWarning
        expr: smartctl_device_temperature > 60
        for: 2m
       labels:
          severity: warning
        annotations:
          summary: Smart device temperature warning (instance {{ $labels.hostname }})
          description: "Device temperature  warning (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

      - alert: SmartDeviceTemperatureCritical
        expr: smartctl_device_temperature > 80
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: Smart device temperature critical (instance {{ $labels.hostname }})
          description: "Device temperature critical  (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

      - alert: SmartCriticalWarning
        expr: smartctl_device_critical_warning > 0
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: Smart critical warning (instance {{ $labels.hostname }})
          description: "device has critical warning (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

      - alert: SmartNvmeWearoutIndicator
        expr: smartctl_device_available_spare{device=~"nvme.*"} < smartctl_device_available_spare_threshold{device=~"nvme.*"}
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: Smart NVME Wearout Indicator (instance {{ $labels.hostname }})
          description: "NVMe device is wearing out (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

      - alert: SmartNvmeMediaError
        expr: smartctl_device_media_errors > 0
        for: 15m
        labels:
          severity: warning
       annotations:
          summary: Smart NVME Media errors (instance {{ $labels.hostname }})
          description: "Contains the number of occurrences where the controller detected an unrecovered data integrity error. Errors such as uncorrectable ECC, CRC checksum failure, or LBA tag mismatch are included in this field (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

      - alert: SmartSmartStatusError
        expr: smartctl_device_smart_status < 1
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: Smart general status error (instance {{ $labels.hostname }})
          description: " (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

     - alert: DiskReallocatedSectorsIncreased
        expr: smartctl_device_attribute{attribute_id="5", attribute_value_type="raw"} > max_over_time(smartctl_device_attribute{attribute_id="5", attribute_value_type="raw"}[1h])
        labels:
          severity: warning
        annotations:
          summary: "SMART Attribute Reallocated Sectors Count Increased"
          description: "The SMART attribute 5 (Reallocated Sectors Count) has increased on {{ $labels.device }} (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

      - alert: DiskSpinRetryCountIncreased
        expr: smartctl_device_attribute{attribute_id="10", attribute_value_type="raw"} > max_over_time(smartctl_device_attribute{attribute_id="10", attribute_value_type="raw"}[1h])
        labels:
          severity: warning
        annotations:
          summary: "SMART Attribute Spin Retry Count Increased"
          description: "The SMART attribute 10 (Spin Retry Count) has increased on {{ $labels.device }} (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

      - alert: DiskCurrentPendingSectorCountIncreased
        expr: smartctl_device_attribute{attribute_id="197", attribute_value_type="raw"} > max_over_time(smartctl_device_attribute{attribute_id="197", attribute_value_type="raw"}[1h])
        labels:
          severity: warning
        annotations:
          summary: "SMART Attribute Current Pending Sector Count Increased"
          description: "The SMART attribute 197 (Current Pending Sector Count) has increased on {{ $labels.device }} (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

      - alert: DiskUncorrectableSectorCountIncreased
        expr: smartctl_device_attribute{attribute_id="198", attribute_value_type="raw"} > max_over_time(smartctl_device_attribute{attribute_id="198", attribute_value_type="raw"}[1h])
        labels:
          severity: warning
        annotations:
          summary: "SMART Attribute Uncorrectable Sector Count Increased"
          description: "The SMART attribute 198 (Uncorrectable Sector Count) has increased on {{ $labels.device }} (instance {{ $labels.hostname }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Configuring the grafana dashboards

Of the different grafana dashboards (1, 2, 3) I went for the first one.

Import it with the UI of grafana, make it work and then export the json to store it in your infra as code respository.

References

New: Thoughts on adding new disks to ZFS.

When it comes to expanding an existing ZFS storage system, careful consideration is crucial. In my case, I faced a decision point with my storage cluster: after two years of reliable service from my 8TB drives, I needed more capacity. This led me to investigate the best way to integrate newly acquired refurbished 12TB drives into the system. Here's my journey through this decision-making process and the insights gained along the way.

The Starting Point

My existing setup consisted of 8TB drives purchased new, which had been running smoothly for two years. The need for expansion led me to consider refurbished 12TB drives as a cost-effective solution. However, mixing new and refurbished drives, especially of different capacities, raised several important considerations that needed careful analysis.

Initial Drive Assessment

The first step was to evaluate the reliability of all drives. Using smartctl, I analyzed the SMART data across both the existing and new drives:
```
for disk in a b c d e f g h i; do
    echo "/dev/sd$disk: old $(smartctl -a /dev/sd$disk | grep Old | wc -l) pre-fail: $(smartctl -a /dev/sd$disk | grep Pre- | wc -l)"
done
```
The results showed similar values across all drives, with "Old_Age" attributes ranging from 14-17 and "Pre-fail" attributes between 3-6. While this indicated all drives were aging, they were still functioning with acceptable parameters. However, raw SMART data doesn't tell the whole story, especially when comparing new versus refurbished drives.

Drive Reliability Considerations

After careful evaluation, I found myself trusting the existing 8TB drives more than the newer refurbished 12TB ones. This conclusion was based on several factors:
- The 8TB drives had a proven track record in my specific environment
- Their smaller size meant faster resilver times, reducing the window of vulnerability during recovery
- One of the refurbished 12TB drives was already showing concerning symptoms (8 reallocated sectors, although a badblocks didn't increase that number), which reduced confidence in the entire batch
- The existing drives were purchased new, while the 12TB drives were refurbished, adding an extra layer of uncertainty
Layout Options Analysis

When expanding a ZFS system, there's always the temptation to simply add more vdevs to the existing pool. However, I investigated two main approaches:
1. Creating a new separate ZFS pool with the new disks
2. Add another vdev to the existent pool
Resilver time

Adding the 12TB drives to the pool and redistributing the data across all 8 drives will help reduce the resilver time. Here's a detailed breakdown:
1. Current Situation
2. 4x 8TB drives at 95% capacity means each drive is heavily packed
3. High data density means longer resilver times
4. Limited free space for data movement and reconstruction
5. After Adding 12TB Drives
6. Total pool capacity increases significantly
7. ZFS will automatically start rebalancing data across all 8 drives
8. This process (sometimes called "data shuffling" or "data redistribution") has several benefits:
9. Reduces data density per drive
10. Creates more free space
11. Improves overall pool performance
12. Potentially reduces future resilver times
13. Resilver Time Reduction Mechanism
14. With data spread across more drives, each individual drive has less data to resilver
15. Less data per drive = faster resilver process
16. The redistribution happens gradually and in the background
Understanding Failure Scenarios

The key differentiator between these approaches came down to failure scenarios:

Single Drive Failure

Both configurations handle single drive failures similarly, though the 12TB drives' longer resilver time creates a longer window of vulnerability in the two-vdev configuration if the data load is evenly shared between the disks. This is particularly concerning with refurbished drives, where the failure probability might be higher.

However if as soon as you add the other vdev to the pool you defragment the data inside zfs, the 8TB drives will be less filled, so until more data is added you may reduce the resilver time as they have less data.

Double Drive Failure

This is where the configurations differ significantly:
- In a two-vdev pool, losing two drives from the same vdev would cause complete pool failure
- With separate pools, a double drive failure would only affect one pool, allowing the other to continue operating. This way you can store the critical data on the pool you trust more.
- Given the mixed drive origins (new vs refurbished), isolating potential failures becomes more critical
Performance Considerations

While investigating performance implications, I found several interesting points about IOPS and throughput:
- ZFS stripes data across vdevs, meaning more vdevs generally means better IOPS
- In RAIDZ configurations, IOPS are limited by the slowest drive in the vdev
- Multiple mirrored vdevs provide the best combined IOPS performance
- Streaming speeds scale with the number of data disks in a RAIDZ vdev
- When mixing drive sizes, ZFS tends to favor larger vdevs, which could lead to uneven wear
Easiness of configuration

Cache and log

If you already have a zpool with a cache and logs in nvme, then if you were to use two pools, you'd need to reformat your nvme drives to create space for the new partitions needed for the new zpool.

This would allow you to specify different cache sizes for each pool. But it comes at the cost of a more complex operation.

New pool creation

Adding a vdev to an existing pool is quicker and easier than to create a zpool. You need to make sure that you initialise it with the correct configuration.

Storage management

Having two pools doubles the operation tasks. One of the pools are to be filled soon, so you may need to manually move files and directories around to rebalance it.

Final Decision

After weighing all factors, if you favour reliability over easiness of your life implement two separate ZFS pools. This statement is primarily driven by:
1. Enhanced Reliability: By separating the pools, we can maintain service availability even if one pool fails completely
2. Data Prioritization: This allows placing critical application data on the more reliable pool (8TB drives), while using the refurbished drives for less critical data like media files
3. Risk Isolation: Keeping the proven, new-purchased drives separate from the refurbished ones minimizes the impact of potential issues with the refurbished drives
4. Consistent Performance: Following the best practice of keeping same-sized drives together in pools
However I'm currently favouring easiness of life and trust my backup solution (I hope not to read this line in the future with regret :P), so I'll go with two vdevs.

Key Takeaways

Through this investigation, I learned several important lessons about ZFS storage design:
1. Raw parity drive count isn't the only reliability metric - configuration matters more than simple redundancy numbers
2. Pool layout significantly impacts both performance and failure scenarios
3. Sometimes simpler configurations (like separate pools) can provide better overall reliability than more complex ones
4. Consider the full lifecycle of the storage, including maintenance operations like resilver times
5. When expanding storage, don't underestimate the value of isolating different generations or sources of hardware
6. The history and source of drives (new vs refurbished) should influence your pool design decisions
This investigation reinforced that storage design isn't just about maximizing space or performance - it's about finding the right balance of reliability, performance, and manageability for your specific needs. When dealing with mixed drive sources and different capacities, this balance becomes even more critical.

References and further reading
diff --git a/mkdocs.yml b/mkdocs.yml index 74292dd717..596f1506e8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -103,6 +103,8 @@ nav: - Email clients: - himalaya: himalaya.md - alot: alot.md + - k9: k9.md + - Email protocols: - Maildir: maildir.md - Instant Messages Management: @@ -371,6 +373,7 @@ nav: - File management configuration: - NeoTree: neotree.md - Telescope: telescope.md + - fzf.nvim: fzf_nvim.md - Editing specific configuration: - vim_editor_plugins.md - Vim formatters: vim_formatters.md @@ -566,7 +569,10 @@ nav: - OpenZFS storage planning: zfs_storage_planning.md - Sanoid: sanoid.md - ZFS Prometheus exporter: zfs_exporter.md - - Hard drive health: hard_drive_health.md + - Hard drive health: + - hard_drive_health.md + - Smartctl: smartctl.md + - badblocks: badblocks.md - Resilience: - linux_resilience.md - Memtest: memtest.md @@ -768,7 +774,8 @@ nav: # - Streaming channels: streaming_channels.md - Music: - Sister Rosetta Tharpe: sister_rosetta_tharpe.md - - Video Gaming: + - Videogames: + - DragonSweeper: dragonsweeper.md - King Arthur Gold: kag.md - The Battle for Wesnoth: - The Battle for Wesnoth: wesnoth.md
New: Add link tot he backblaze disk reports.

If you have some time take a look at backblaze disk reports they do quarterly analysis on their infra (around 300k disks).

@@ -26,6 +26,7 @@ nav: - Feminism: - Privileges: feminism/privileges.md - Palestine: palestine.md + - Detección de estupas: estupas.md - Anarchism: anarchism.md - Memoria histórica: memoria_historica.md - Anti-Tourism: antitourism.md @@ -33,19 +34,22 @@ nav: - Free Knowledge: free_knowledge.md - Free Software: free_software.md - Environmentalism: environmentalism.md - - Laboral: - - laboral.md - - Trabajadoras del hogar: trabajadoras_del_hogar.md + - Laboral: + - laboral.md + - Trabajadoras del hogar: trabajadoras_del_hogar.md - Collaborating tools: collaborating_tools.md - Conference organisation: - conference_organisation.md - pretalx: pretalx.md - - Ludditest: luddites.md - - Life Management: + - Luddites: luddites.md + - Life navigation: - life_management.md - - Time management: + - Time navigation: - time_management.md - - Time management abstraction levels: time_management_abstraction_levels.md + - Time navigation abstraction levels: + - time_management_abstraction_levels.md + - Identities: identities.md + - Axis: axis.md - Action Management: action_management.md - Roadmap Adjustment: - roadmap_adjustment.md @@ -134,6 +138,7 @@ nav: - Book Management: - book_management.md - Bookwyrm: bookwyrm.md + - Book DRM: book_drm.md - Movies Management: - Jellyfin: jellyfin.md - Ombi: ombi.md @@ -196,6 +201,7 @@ nav: - Teeth: - teeth.md - Deep cleaning: teeth_deep_cleaning.md + - Silence: silence.md - Remote Working: remote_work.md - Fitness Tracker: - fitness_band.md @@ -649,6 +655,8 @@ nav: - fail2ban: linux/fail2ban.md - pass: pass.md - Wireshark: wireshark.md + - Canary tokens: canary_tokens.md + - Sysadmin tools: - brew: linux/brew.md - detox: detox.md @@ -753,6 +761,7 @@ nav: - Speech to text: - Whisper: whisper.md - Speech recognition: speech_recognition.md + - Text to speech: text_to_speech.md - Coding by Voice: coding_by_voice.md - Data Analysis: - data_analysis.md
New: Remove the public IP of an ec2 instance.
- Navigate to the network interfaces of the instance
- Click on the one that contains the public IP
- Actions/Manage IP addresses
- Click on the Interface to unfold the configuration
- Click on Auto-assign public IP

New: Introduce vector.

Vector is a lightweight, ultra-fast tool for building observability pipelines

Installation

On debian

First, add the Vector repo:

bash -c "$(curl -L https://setup.vector.dev)"

Then you can install the vector package:

sudo apt-get install vector

Tweak the configuration and then enable the service.

To be sure that vector is able to push to loki create the /usr/local/bin/wait-for-loki.sh file

while true; do
  response=$(curl -s http://localhost:3100/ready 2>/dev/null)
  if [ "$response" = "ready" ]; then
    break
  fi
  sleep 1
done

Make it executable chmod +x /usr/local/bin/wait-for-loki.sh

Then update your vector.service (/usr/lib/systemd/system/vector.service)

ExecStartPre=/usr/local/bin/wait-for-loki.sh
ExecStartPre=/usr/bin/vector validate

Run systemctl daemon-reload to reload the service configuration.

Configuration

The config lives at /etc/vector/vector.yaml.

Docker

First add vector to the docker group: usermod -a -G docker vector

sources:
  docker:
    type: docker_logs

transforms:
  docker_labels:
    type: remap
    inputs:
      - docker
    source: |
      .service_name = get(.label, ["com.docker.compose.project"]) ?? "unknown"
sinks:
  loki_docker:
    type: loki
    inputs:
      - docker_labels
    endpoint: http://localhost:3100/
    encoding:
      codec: json
    labels:
      source: docker
      host: "{{ host }}"
      container: "{{ container_name }}"
      service_name: "{{ service_name }}"

journald

To avoid the services that run docker to be indexed twice

sources:
  journald:
    type: journald

transforms:
  journald_filter:
    type: filter
    inputs:
      - journald
    condition: |
      # Exclude docker-compose systemd services
      !contains(string!(.SYSLOG_IDENTIFIER), "docker-compose") &&
      !contains(string!(.SYSLOG_IDENTIFIER), "docker")

  journald_labels:
    type: remap
    inputs:
      - journald_filter
    source: |
      .service_name = ._SYSTEMD_UNIT || "unknown"

sinks:
  loki_systemd:
    type: loki
    inputs:
      - journald_labels
    endpoint: http://localhost:3100/
    encoding:
      codec: json
    labels:
      source: journald
      host: "{{ host }}"
      service_name: "{{ service_name }}"

ZFS

Prepare the file to be readable by vector:

chown root:vector /proc/spl/kstat/zfs/dbgmsg
chmod 640 /proc/spl/kstat/zfs/dbgmsg

sources:
  zfs_log:
    type: file
    include:
      - /proc/spl/kstat/zfs/dbgmsg
  zfs_files:
    type: loki
    inputs:
      - zfs_log
    endpoint: http://localhost:3100/
    encoding:
      codec: json
    labels:
      source: file
      service_name: zfs
      host: "{{ host }}"
      filename: "{{ file }}"
sinks:

Troubleshooting

Unable to open checkpoint file. path="/var/lib/vector/journald/checkpoint.txt"

ERROR source{component_kind="source" component_id=journald component_type=journald}: vector::internal_events::journald: Unable to open checkpoint file. path="/var/lib/vector/journald/checkpoint.txt" error=Permission denied (os error 13) error_type="io_failed" stage="receiving"

sudo mkdir -p /var/lib/vector/journald
sudo chown -R vector:vector /var/lib/vector
sudo chmod 755 /var/lib/vector
sudo chmod 755 /var/lib/vector/journald

References

Home

New: Vector Permission Debugging with systemd tmpfiles.

Vector fails to read log files after reboots or log rotation with permission errors:
```
ERROR: Failed reading file for fingerprinting. error=Permission denied (os error 13)
```
Solution with tmpfiles

Create /etc/tmpfiles.d/vector-permissions.conf:
```
z /data/apps/myapp/logs/logfile.log 0644 vector vector -
z /path/to/another/logfile.log 0644 vector vector -
```
Apply immediately:
```
systemd-tmpfiles --create /etc/tmpfiles.d/vector-permissions.conf
```
What is tmpfiles

systemd tmpfiles.d is a mechanism for managing temporary files, directories, and their permissions at boot time and periodically during system operation.

What it does:
- Creates/removes files and directories
- Sets ownership and permissions
- Runs at boot via systemd-tmpfiles-setup.service
- Can be triggered manually or periodically
Configuration format:
```
Type Path Mode UID GID Age Argument
```
Common types:
- d - Create directory
- f - Create file
- z - Set ownership/permissions on existing path
- Z - Recursively set ownership/permissions
- x - Ignore/exclude path
Example:
```
d /var/run/myapp 0755 myuser mygroup -
f /var/log/myapp.log 0644 myuser mygroup -
z /existing/file 0644 myuser mygroup -
```
Files in /etc/tmpfiles.d/ with .conf extension are processed automatically at boot and can be manually applied with systemd-tmpfiles --create. Also, if the systemd-tmpfiles-clean.timer is enabled (which is by default) it will be check each day.

OpenZFS

New: Check the health of a disk with badblocks.

The badblocks command will write and read the disk with different patterns, thus overwriting the whole disk, so you will loose all the data in the disk.

This test is good for rotational disks as there is no disk degradation on massive writes, do not use it on SSD though.

WARNING: be sure that you specify the correct disk!!
```
badblocks -wsv -b 4096 /dev/sde | tee disk_analysis_log.txt
```
If errors are shown is that all of the spare sectors of the disk are used, so you must not use this disk anymore. Again, check dmesg for traces of disk errors.

New: Get the node architecture of the pods of a deployment.

Here are a few ways to check the node architecture of pods in a deployment:

Get the nodes where the pods are running:
```
kubectl get pods -l app=your-deployment-label -o wide
```
This will show which nodes are running your pods.

Then check the architecture of those nodes:

kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture

Or you can combine this into a single command:

kubectl get pods -l app=your-deployment-label -o json | jq -r '.items[].spec.nodeName' | xargs -I {} kubectl get node {} -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture

You can also check if your deployment is explicitly targeting specific architectures through node selectors or affinity rules:

kubectl get deployment your-deployment-name -o yaml | grep -A 5 nodeSelector

New: Removing a disk from the pool.
```
zpool remove tank0 sda
```
This will trigger the data evacuation from the disk. Check zpool status to see when it finishes.
New: Encrypting ZFS Drives with LUKS.

Warning: Proceed with Extreme Caution

IMPORTANT SAFETY NOTICE:
- These instructions will COMPLETELY WIPE the target drive
- Do NOT attempt on production servers
- Experiment only on drives with no valuable data
- Seek professional help if anything is unclear
Prerequisites
- A drive you want to encrypt (will be referred to as /dev/sdx)
- Root access
- Basic understanding of Linux command line
- Backup of all important data
Step 1: Create LUKS Encryption Layer

First, format the drive with LUKS encryption:
```
sudo cryptsetup luksFormat /dev/sdx
```
- You'll be prompted for a sudo password
- Create a strong encryption password (mix of uppercase, lowercase, numbers, symbols)
- Note the precise capitalization in commands
Step 2: Open the Encrypted Disk

Open the newly encrypted disk:
```
sudo cryptsetup luksOpen /dev/sdx sdx_crypt
```
This creates a mapped device at /dev/mapper/sdx_crypt

Step 3: Create ZFS Pool or the vdev

For example to create a ZFS pool on the encrypted device:
```
sudo zpool create -f -o ashift=12 \
   -O compression=lz4 \
+    zpool /dev/mapper/sdx_crypt
```
Check the create zpool section to know which configuration flags to use.

Step 4: Set Up Automatic Unlocking

Generate a Keyfile

Create a random binary keyfile:
```
sudo dd bs=1024 count=4 if=/dev/urandom of=/etc/zfs/keys/sdx.key
sudo chmod 0400 /etc/zfs/keys/sdx.key
```
Add Keyfile to LUKS

Add the keyfile to the LUKS disk:
```
sudo cryptsetup luksAddKey /dev/sdx /etc/zfs/keys/sdx.key
```
- You'll be asked to enter the original encryption password
- This adds the binary file to the LUKS disk header
- Now you can unlock the drive using either the password or the keyfile
Step 5: Configure Automatic Mounting

Find Drive UUID

Get the drive's UUID:
```
sudo blkid
```
Look for the line with TYPE="crypto_LUKS". Copy the UUID.

Update Crypttab

Edit the crypttab file:
```
sudo vim /etc/crypttab
```
Add an entry like:
```
sdx_crypt UUID=your-uuid-here /etc/zfs/keys/sdx.key luks,discard
```
Final Step: Reboot
- Reboot your system
- The drive will be automatically decrypted and imported
Best Practices

+- Keep your keyfile and encryption password secure - Store keyfiles with restricted permissions - Consider backing up the LUKS header

Troubleshooting
- Double-check UUIDs
- Verify keyfile permissions
- Ensure cryptsetup and ZFS are installed
Security Notes
- This method provides full-disk encryption at rest
- Data is inaccessible without the key or password
- Protects against physical drive theft
Disclaimer

While these instructions are comprehensive, they come with inherent risks. Always:
- Have backups
- Test in non-critical environments first
- Understand each step before executing
Further reading
New: Add a disk to an existing vdev.
```
zpool add tank /dev/sdx
```
New: Add a vdev to an existing pool.

``bash zpool add main raidz1-1 /dev/disk-1 /dev/disk-2 /dev/disk-3 /dev/disk-4 ```

You don't need to specify the ashift or the autoexpand as they are set on zpool creation.
New: Add zfs book.
- FreeBSD Mastery: ZFS by Michael W Lucas and Allan Jude
Correction: Replacing a disk in the pool.

If the pool is not DEGRADED

If you want to do operations on your pool and want to prevent it from being DEGRADED you need to attach a new disk to the server and use the replace command
```
zfs replace tank0 ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx /dev/sdX
```
Where /dev/sdX is your temporal disk. Once the original disk is removed from the vdev you can do the operations you need.
New: Removing a disk from the pool.
```
zpool remove tank0 sda
```
This will trigger the data evacuation from the disk. Check zpool status to see when it finishes.

Sometimes zfs won't allow you to remove a disk if it will put at risk the pool. In that case try to replace a disk in the pool as explained above.

New: List all datasets that have zfs native encryption.

```bash ROOT_FS="main" is_encryption_enabled() { zfs get -H -o value encryption $1 | grep -q 'aes' }

list_datasets_with_encryption() {

# Initialize an array to hold dataset names
datasets=()

# List and iterate over all datasets starting from the root filesystem
for dataset in $(zfs list -H -o name | grep -E '^'$ROOT_FS'/'); do
    if is_encryption_enabled "$dataset"; then
        datasets+=("$dataset")
    fi
done

# Output the results
echo "ZFS datasets with encryption enabled:"
printf '%s\n' "${datasets[@]}"

}

list_datasets_with_encryption

New: Troubleshoot cannot destroy dataset: dataset is busy.

If you're experiencing this error and can reproduce the next traces:

cannot destroy 'zroot/2013-10-15T065955229209': dataset is busy

cannot unmount 'zroot/2013-10-15T065955229209': not currently mounted

zroot/2013-10-15T065955229209                2.86G  25.0G  11.0G  /var/lib/heaver/instances/2013-10-15T065955229209

umount: /var/lib/heaver/instances/2013-10-15T065955229209: not mounted

You can grep zroot/2013-10-15T065955229209 /proc/*/mounts to see which process is still using the dataset.

Another possible culprit are snapshots, you can then run:

zfs holds $snapshotname

To see if it has any holds, and if so, zfs release to remove the hold.

New: Upgrading ZFS Storage Pools.

If you have ZFS storage pools from a previous zfs release you can upgrade your pools with the zpool upgrade command to take advantage of the pool features in the current release. In addition, the zpool status command has been modified to notify you when your pools are running older versions. For example:

zpool status

  pool: tank
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on older software versions.
 scrub: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c1t0d0  ONLINE       0     0     0
            c1t1d0  ONLINE       0     0     0
errors: No known data errors

You can use the following syntax to identify additional information about a particular version and supported releases:

zpool upgrade -v

This system is currently running ZFS pool version 22.

The following versions are supported:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
 14  passthrough-x aclinherit
 15  user/group space accounting
 16  stmf property support
 17  Triple-parity RAID-Z
 18  Snapshot user holds
 19  Log device removal
 20  Compression using zle (zero-length encoding)
 21  Reserved
 22  Received properties

For more information on a particular version, including supported releases,
see the ZFS Administration Guide.

Then, you can run the zpool upgrade command to upgrade all of your pools. For example:

zpool upgrade -a

New: Monitor the zfs ram usage.

      - alert: HostOutOfMemory
        # if we don't add the node_zfs_arc_size, the ARC is taken as used space triggering the alert as a false positive
        expr: (node_memory_MemAvailable_bytes + node_zfs_arc_size)/ node_memory_MemTotal_bytes * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Host out of memory (instance {{ $labels.instance }})
          message: "Node memory is filling up (< 10% left)\n  VALUE = {{ $value\
            \ }}"

feat(linux_snippets#Unattended upgrades): Unattended upgrades

unattended-upgrades runs daily at a random time

How to tell when unattended upgrades will run today:

The random time is set by a cron job (/etc/cron.daily/apt.compat), and you can read the random time for today by asking systemd:

$ systemctl list-timers apt-daily.timer
NEXT                         LEFT     LAST                         PASSED      UNIT            ACTIVATES
Tue 2017-07-11 01:53:29 CDT  13h left Mon 2017-07-10 11:22:40 CDT  1h 9min ago apt-daily.timer apt-daily.service

In this case, you can see that it ran 1 hour and 9 minutes ago.

How to tell if unattended upgrades are still running:

One easy way is to check the timestamp files for the various apt components:

$ ls -l /var/lib/apt/periodic/
total 0
-rw-r--r-- 1 root root 0 Jul 10 11:24 unattended-upgrades-stamp
-rw-r--r-- 1 root root 0 Jul 10 11:23 update-stamp
-rw-r--r-- 1 root root 0 Jul 10 11:24 update-success-stamp
-rw-r--r-- 1 root root 0 Jul 10 11:24 upgrade-stamp

Putting the data together, you can see that the timer started apt at 11:22. It ran an update which completed at 11:23, then an upgrade which completed at 11:24. Finally, you can see that apt considered the upgrade to be a success (no error or other failure).

Obviously, if you see a recent timer without a corresponding completion timestamp, then you might want to check ps to see if apt is still running.

How to tell which step apt is running right now

One easy way is to check the logfile.

$ less /var/log/unattended-upgrades/unattended-upgrades.log
2017-07-10 11:23:00,348 INFO Initial blacklisted packages:
2017-07-10 11:23:00,349 INFO Initial whitelisted packages:
2017-07-10 11:23:00,349 INFO Starting unattended upgrades script
2017-07-10 11:23:00,349 INFO Allowed origins are: ['o=Ubuntu,a=zesty-security', 'o=Ubuntu,a=zesty-updates']
2017-07-10 11:23:10,485 INFO Packages that will be upgraded: apport apport-gtk libpoppler-glib8 libpoppler-qt5-1 libpoppler64 poppler-utils python3-apport python3-problem-report
2017-07-10 11:23:10,485 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2017-07-10 11:24:20,419 INFO All upgrades installed

Here you can see the normal daily process, including the 'started' and 'completed' lines, and the list of packages that were about to be upgraded.

If the list of packages is not logged yet, then apt can be safely interrupted. Once the list of packages is logged, DO NOT interrupt apt.

**Check the number of packages that need an upgrade **

apt list --upgradeable

**Manually run the unattended upgrades **

unattended-upgrade -d

unison

New: Installation.

If you are on debian or ubuntu, the version of the repositories does not allow you to run the program with the file watcher, so you may need to build it yourself:

First install the dependencies:

sudo apt-get install ocaml-native-compilers

export UNISON_VERSION=2.53.8
echo "Install Unison." \
    && pushd /tmp \
    && wget https://github.com/bcpierce00/unison/archive/v$UNISON_VERSION.tar.gz \
    && tar -xzvf v$UNISON_VERSION.tar.gz \
    && rm v$UNISON_VERSION.tar.gz \
    && pushd unison-$UNISON_VERSION \
    && make \
    && cp -t /usr/local/bin ./src/unison ./src/unison-fsmonitor \
    && popd \
    && rm -rf unison-$UNISON_VERSION \
    && popd

Then remove the ocaml compilers as they take quite some space:

sudo apt-get remove ocaml-native-compilers

New: Run in the background watching changes.

Create the systemd service in: ~/.config/systemd/user/unison.service (assuming that your profile is orgfiles)
```
[Unit]
Description=unison

[Service]
ExecStart=/usr/local/bin/unison orgfiles
Restart=on-failure
RestartSec=3

[Install]
WantedBy=default.target
```

renovate

New: Add mention to sixos.

Think about using sixos instead, a nixos without systemd
New: Installation in gitea actions.
- Create Renovate Bot Account and generate a token for the Gitea Action secret
- Add the renovate bot account as collaborator with write permissions to the repository you want to update.
- Create a repository to store our Renovate bot configurations, assuming called renovate-config.
In renovate-config, create a file config.js to configure Renovate:
```
module.exports = {
    "endpoint": "https://gitea.com/api/v1", // replace it with your actual endpoint
    "gitAuthor": "Renovate Bot <renovate-bot@yourhost.com>",
    "platform": "gitea",
    "onboardingConfigFileName": "renovate.json",
    "autodiscover": true,
    "optimizeForDisabled": true,
};
```
If you're using mysql or you see errors like .../repository/pulls 500 internal error you may need to set unicodeEmoji: false.
New: Configure renovate.

automerge

By default, Renovate raises PRs but leaves them to someone or something else to merge them. By configuring this setting, you allow Renovate to automerge PRs or even branches. Using automerge reduces the amount of human intervention required.

Usually you won't want to automerge all PRs, for example most people would want to leave major dependency updates to a human to review first. You could configure Renovate to automerge all but major this way:
```
{
  "packageRules": [
    {
      "matchUpdateTypes": ["minor", "patch", "pin", "digest"],
      "automerge": true
    }
  ]
}
```
Also note that this option can be combined with other nested settings, such as dependency type. So for example you could choose to automerge all (passing) devDependencies only this way:
```
{
  "packageRules": [
    {
      "matchDepTypes": ["devDependencies"],
      "automerge": true
    }
  ]
}
```
Configure docker version extraction
Combine all updates to one branch/PR
New: Add upgrade notes.
- Upgrade notes

Debugging

New: PVC or PV is stuck deleting.

When PVs and PVCs get stuck during deletion, it's usually due to finalizers that prevent the cleanup process from completing. Here are several approaches to resolve this:

Check for Finalizers

First, examine what's preventing the deletion:
```
kubectl get pv <pv-name> -o yaml | grep finalizers -A 5
kubectl get pvc <pvc-name> -n <namespace> -o yaml | grep finalizers -A 5
```
Remove Finalizers (Most Common Solution)

If you see finalizers like kubernetes.io/pv-protection or kubernetes.io/pvc-protection, you can remove them:
```
kubectl patch pvc <pvc-name> -n <namespace> -p '{"metadata":{"finalizers":null}}'

kubectl patch pv <pv-name> -p '{"metadata":{"finalizers":null}}'
```

Node Exporter

New: Monitor host requires a reboot.

Node exporter does not support this metric, but you can monitor reboot requirements using Prometheus node exporter's textfile collector. Here's how to set it up:

Create the monitoring script

First, create a script that checks for the reboot-required file and outputs metrics in Prometheus format:

Make the script executable and place it in the right location:

sudo cp reboot-required-check.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/reboot-required-check.sh

TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
METRIC_FILE="$TEXTFILE_DIR/reboot_required.prom"

mkdir -p "$TEXTFILE_DIR"

if [ -f /var/run/reboot-required ]; then
    REBOOT_REQUIRED=1
else
    REBOOT_REQUIRED=0
fi

cat > "$METRIC_FILE" << EOF
node_reboot_required $REBOOT_REQUIRED
EOF

chmod 644 "$METRIC_FILE"

Ensure the textfile collector directory exists:

sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown node_exporter:node_exporter /var/lib/node_exporter/textfile_collector

Create a systemd service to run the script periodically:

[Unit]
Description=Check if system requires reboot
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/reboot-required-check.sh
User=node_exporter
Group=node_exporter

[Install]
WantedBy=multi-user.target

Create a systemd timer to run it regularly:

[Unit]
Description=Check if system requires reboot every 5 minutes
Requires=reboot-check.service

[Timer]
OnBootSec=1min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

Install and enable the systemd units:

sudo cp reboot-check.service /etc/systemd/system/
sudo cp reboot-check.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable reboot-check.timer
sudo systemctl start reboot-check.timer

Configure node exporter to use the textfile collector: Make sure your node exporter is started with the --collector.textfile.directory flag:

node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

Prometheus alerting rule

You can create an alerting rule in Prometheus to notify when a reboot is required:

groups:
  - name: system.rules
    rules:
      - alert: SystemRebootRequired
        expr: node_reboot_required == 1
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "System {{ $labels.instance }} requires reboot"
          description: "System {{ $labels.instance }} requires a reboot due to: {{ $labels.reason }}"

Testing

You can test the setup by:

Run the script manually:

sudo /usr/local/bin/reboot-required-check.sh
cat /var/lib/node_exporter/textfile_collector/reboot_required.prom

Check if the timer is working:

sudo systemctl status reboot-check.timer
sudo journalctl -u reboot-check.service

Verify metrics are being collected: Visit http://your-server:9100/metrics and search for node_reboot_required

The metrics will show node_reboot_required 1 when a reboot is required and node_reboot_required 0 when it's not. The node_reboot_required_packages_info metric includes information about which packages triggered the reboot requirement.

Operating Systems

dunst

Correction: Tweak installation steps.

sudo apt install libdbus-1-dev libx11-dev libxinerama-dev libxrandr-dev libxss-dev libglib2.0-dev \
    libpango1.0-dev libgtk-3-dev libxdg-basedir-dev libgdk-pixbuf-2.0-dev

make WAYLAND=0
sudo make WAYLAND=0 install

If it didn't create the systemd service you can create it yourself with this service file

[Unit]
Description=Dunst notification daemon
Documentation=man:dunst(1)
PartOf=graphical-session.target

[Service]
Type=dbus
BusName=org.freedesktop.Notifications
ExecStart=/usr/local/bin/dunst
Slice=session.slice
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

[Install]
WantedBy=default.target

You may need to add more paths to PATH.

To see the logs of the service use journalctl --user -u dunst.service -f --since "15 minutes ago"

New: Configuration.

Read and tweak the ~/.dunst/dunstrc file to your liking. You have the default one here

You'll also need to configure the actions in your window manager. In my case i3wm:
```
bindsym $mod+b exec dunstctl close-all
bindsym $mod+v exec dunstctl context
```
New: Configure each application notification.

You can look at rosoau config for inspiration

References * Some dunst configs * Smarttech101 tutorials (1, 2) * Archwiki page on dunst

Linux Snippets

New: Prevent the screen from turning off.

VESA Display Power Management Signaling (DPMS) enables power saving behaviour of monitors when the computer is not in use. The time of inactivity before the monitor enters into a given saving power level—standby, suspend or off—can be set as described in DPMSSetTimeouts(3).

It is possible to turn off your monitor with the xset command
```
xset s off -dpms
```
It will disable DPMS and prevent screen from blanking

To query the current settings:
```
xset q
```
If that doesn't work you can use the keep-presence program
```
pip install keep-presence
keep-presence -c
```
That will move the cursor one pixel in circles each 300s, if you need to move it more often use the -s flag.
New: Protect the edition of a pdf with a password.

Use pdftk. From its man page:

Encrypt a PDF using 128-Bit Strength (the Default) and Withhold All Permissions (the Default)
```
$ pdftk [mydoc].pdf output [mydoc.128].pdf owner_pw [foopass]
```
Same as Above, Except a Password is Required to Open the PDF
```
$ pdftk [mydoc].pdf output [mydoc.128].pdf owner_pw [foo] user_pw [baz]
```
Same as Above, Except Printing is Allowed (after the PDF is Open)
```
$ pdftk [mydoc].pdf output [mydoc.128].pdf owner_pw [foo] user_pw [baz] allow pri
```
To check if it has set the password correctly you can run:
```
pdftk "input.pdf" dump_data output /dev/null dont_ask
```
New: Reduce the size of an image.

The simplest way of reducing the size of the image is by degrading the quality of the image.
```
convert <INPUT_FILE> -quality 50% <OUTPUT_FILE>
```
The main difference between convert and mogrify command is that mogrify command applies the operations on the original image file, whereas convert does not.
```
mogrify -quality 50 *.jpg
```
New: Change the default shell of a user using the command line.
```
chsh -s /usr/bin/zsh lyz
```
New: Introduce simplex chat.

Simplex chat is the first messenger without user IDs

I went to a talk in the 38c3 (december 2024), and even though the project looked good there were some stuff that pushed me away:
- The cypher has not been tested
- It's not fully open sourced

New: Record the audio from your computer.

You can record audio being played in a browser using ffmpeg

Check your default audio source:

pactl list sources | grep -E 'Name|Description'

Record using ffmpeg:

ffmpeg -f pulse -i <your_monitor_source> output.wav

Example:

ffmpeg -f pulse -i alsa_output.pci-0000_00_1b.0.analog-stereo.monitor output.wav

Stop recording with Ctrl+C.

New: Download videos from rtve.es.

Use descargavideos.tv (source)

New: Check if a domain is in a list of known disposable email domains.

You can check in known lists

wget https://raw.githubusercontent.com/andreis/disposable-email-domains/master/domains.txt
grep -i homapin.com domains.txt

Or using web services that either use the IPs (obtained by whois/dig)

https://www.blocklist.de/en/search.html?ip=142.132.166.12&action=search&send=start+search 👍
https://www.blocklist.de/en/search.html?ip=188.166.111.252&action=search&send=start+search 👍
https://www.blocklist.de/en/search.html?ip=46.101.111.206&action=search&send=start+search 👍
https://www.blocklist.de/en/search.html?ip=116.202.9.167&action=search&send=start+search 👍
https://check.spamhaus.org/results/?query=homapin.com 👍
https://verifymail.io/domain/homapin.com 👎
https://www.ipqualityscore.com/domain-reputation/homapin.com 👎
https://quickemailverification.com/tools/disposable-email-address-detector for
  - homapin.com 👎

New: Mount a cdrom or dvd.

TL;DR: The syntax is as follows for the mount command:
```
mount -t iso9660 -o ro /dev/deviceName /path/to/mount/point
```
Use the following command to find out the name Of DVD / CD-ROM / Writer / Blu-ray device on a Linux based system:
```
lsblk
```
OR use the combination of the dmesg command and grep/egrep as follow to print your CD/DVD device name. For example:
```
dmesg | grep -E -i --color 'cdrom|dvd|cd/rw|writer'
```
Sample outputs indicating that the /dev/sr0 is my device name:
```
[    5.437164] sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
[    5.437307] cdrom: Uniform CD-ROM driver Revision: 3.20
```
Create a mount point, type mkdir command as follows:
```
mkdir -p /mnt/cdrom
```
Mount the /dev/cdrom or /dev/sr0 as follows:
```
mount -t iso9660 -o ro /dev/cdrom /mnt/cdrom
```
New: Use lftp.

Connect with:
```
lftp -p <port> user@host
```
Navigate with ls and cd. Get with mget for multiple things
New: Difference between apt-get upgrate and apt-get full-upgrade.

The difference between upgrade and full-upgrade is that the later will remove the installed packages if that is needed to upgrade the whole system. Be extra careful when using this command

I will more frequently use autoremove to remove old packages and then just use upgrade.

New: Upgrade debian.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get full-upgrade

sudo vi /etc/apt/sources.list /etc/apt/sources.list.d/*

sudo apt-get clean
sudo apt-get update

sudo apt-get upgrade
sudo apt-get full-upgrade

sudo apt-get autoremove

sudo shutdown -r now

New: Get a list of extensions by file type.

There are community made lists such as dyne's file extension list
Correction: Upgrade ubuntu.

Upgrade your system:
```
sudo apt update
sudo apt upgrade
reboot
```
You must install ubuntu-release-upgrader-core package:
```
sudo apt install ubuntu-release-upgrader-core
```
Ensure the Prompt line in /etc/update-manager/release-upgrades is set to ‘lts‘ using the “grep” or “cat”
```
grep 'lts' /etc/update-manager/release-upgrades
cat /etc/update-manager/release-upgrades
```
Opening up TCP port 1022

For those using ssh-based sessions, open an additional SSH port using the ufw command, starting at port 1022. This is the default port set by the upgrade procedure as a fallback if the default SSH port dies during upgrades.
```
sudo /sbin/iptables -I INPUT -p tcp --dport 1022 -j ACCEPT
```
Finally, start the upgrade from Ubuntu 22.04 to 24.04 LTS version. Type:
```
sudo do-release-upgrade -d
```
New: Check the file encoding.
```
file -i <path_to_file>
```
New: Simulate the environment of a cron job.

Add this to your crontab (temporarily):
```
* * * * * env > ~/cronenv
```
After it runs, do this:
```
env - `cat ~/cronenv` /bin/sh
```
This assumes that your cron runs /bin/sh, which is the default regardless of the user's default shell.

Footnote: if env contains more advanced config, eg PS1=$(__git_ps1 " (%s)")$, it will error cryptically env: ": No such file or directory.
New: Resize a partition of an EC2 instance.

If it's the first partition of the first disk.
```
growpart /dev/nvme0n1 1
resize2fs /dev/nvme0n1p1
```

New: Debugging Inotify Watch Exhaustion: "No space left on device".

If you get No space left on device errors in systemd logs, but disk space is fine:

systemd[755863]: my-service.service: Failed to add control inotify watch descriptor for control group: No space left on device

This is actually an inotify watches exhaustion issue, not disk space.

Diagnosis

Check Current Limits

cat /proc/sys/fs/inotify/max_user_watches

Count Current Watch Usage

find /proc/*/fdinfo/* -exec grep -c "^inotify" {} \; 2>/dev/null | awk '{sum+=$1} END {print sum}'

for pid in $(ps -eo pid --no-headers); do
  if [[ -d /proc/$pid/fd ]]; then
    total=0
    for fd in /proc/$pid/fd/*; do
      if [[ -L "$fd" ]] && readlink "$fd" 2>/dev/null | grep -q "anon_inotify"; then
        fdnum=$(basename "$fd")
        watches=$(grep -c "^inotify" /proc/$pid/fdinfo/$fdnum 2>/dev/null || echo 0)
        total=$((total + watches))
      fi
    done
    if [[ $total -gt 0 ]]; then
      cmd=$(ps -p $pid -o comm --no-headers 2>/dev/null || echo "unknown")
      echo "$total $pid $cmd"
    fi
  fi
done 2>/dev/null | sort -nr | head -10

Find High File Descriptor Processes

Often the culprits have many open files:

for pid in $(ps -eo pid --no-headers); do
  if [[ -d /proc/$pid/fd ]]; then
    fd_count=$(ls /proc/$pid/fd 2>/dev/null | wc -l)
    if [[ $fd_count -gt 100 ]]; then
      cmd=$(ps -p $pid -o comm --no-headers 2>/dev/null)
      echo "$fd_count FDs: $pid $cmd"
    fi
  fi
done

Common Culprits

Media servers: Jellyfin, Plex
Download managers: Sonarr, Radarr, Lidarr
File sync: Syncthing, Nextcloud
Development tools: IDEs, file watchers
Container platforms: Docker, containerd

Solutions

1. Increase Limits (Quick Fix)

echo 500000 > /proc/sys/fs/inotify/max_user_watches

echo 'fs.inotify.max_user_watches=500000' >> /etc/sysctl.conf
sysctl -p

Memory cost: ~540 bytes per watch (500k watches ≈ 270MB kernel memory)

2. Configure Applications

Better long-term solution:

Sonarr/Radarr/Lidarr:

Settings → Media Management → Disable "Scan for changes"
Use scheduled scans instead

Jellyfin:

Admin → Dashboard → Libraries → Disable real-time monitoring
Use periodic library scans

Syncthing:

Use polling instead of inotify for large directories
Add .stignore for unnecessary paths

New: Manage bluetooth.

List devices

Once you've paired your devices you can see them with:
```
bluetoothctl devices
```
To check the ones that are connected use:
```
bluetoothctl devices Connected
```
Connect device

From the list above you will see the device ID, then you can:
```
bluetoothctl connect device_ID
```
But it's better to use blueman-manager because it handles better the connections and disconnections

Multidevice connection

Sometimes the laptop is not able to send the audio streams back to the connected device. Restart the controlling device with:
```
systemctl --user restart wireplumber.service
```
New: Bluetooth Pairing Troubleshooting: When BLE Devices Won't Connect.

Bluetooth Low Energy (BLE) devices like wireless earbuds appear in device scans but fail to pair with "Device not available" errors, even though they're visible to other devices.

The root ’cause may be that the HCI controller corruption causing discovery operations to fail. The Bluetooth hardware gets stuck in a state where it rejects pairing attempts with error code -16 (EBUSY).

Symptoms
- hcitool lescan shows the device
- bluetoothctl scan shows device briefly or not at all
- bluetoothctl pair [MAC] returns "Device not available"
- dmesg shows HCI opcode failures like: Bluetooth: hci0: Opcode 0x0401 failed: -16
Solution

Complete Bluetooth stack reset:
```
sudo systemctl stop bluetooth

sudo rmmod btusb

sudo modprobe btusb

sudo systemctl start bluetooth
```
If that doesn't work. Try forcing bluetoothctl to see LE devices specifically:
```
$: bluetoothctl  # this will open the bluetooth cli
menu scan
clear
transport le
back
scan on
```
New: How to Increase Touchpad Sensitivity on Linux.

Adjust touchpad sensitivity settings for better responsiveness and control on Linux systems.

First, identify your touchpad device:
```
xinput list
```
Check current properties:
```
xinput list-props "Synaptics TM3381-002"
```
Increase pointer sensitivity (range: -1.0 to 1.0):
```
xinput set-prop "Synaptics TM3381-002" "libinput Accel Speed" 1
```
Make scrolling more sensitive:
```
xinput set-prop "Synaptics TM3381-002" "libinput Scrolling Pixel Distance" 10
```
Once you have the correct values make it permanent across reboots by adding them to your startup scripts.
New: How to Disable Trackpoint on Linux.

The trackpoint (that red nub in the middle of ThinkPad keyboards) can be accidentally triggered while typing, or if you replace the keyboard it might make the mouse slide randomly without any user intervention. Here's how to disable it on Linux systems.

with xinput

Get the property of your trackpoint with xinput list | grep -i track

Then disable the trackpoint with xinput set-prop "TPPS/2 Elan TrackPoint" "Device Enabled" 0

To make it persistent add that line to your desktop startup scripts. It can be run by a non privileged user.

With udev rules

The problem with this approach is that the event number may change across reboots

First, find your trackpoint in the system:
```
cat /proc/bus/input/devices | grep -A5 -B5 -i trackpoint
```
Look for entries like "TPPS/2 Elan TrackPoint" or similar. Note the event number (e.g., event14).

Create a udev rule to ignore the trackpoint device:
```
sudo sh -c 'echo "KERNEL==\"event[0-9]*\", SUBSYSTEM==\"input\", ATTRS{name}==\"*TrackPoint*\", ENV{LIBINPUT_IGNORE_DEVICE}=\"1\"" > /etc/udev/rules.d/90-disable-trackpoint.rules'
```
Replace event[0-9]* with your specific event number if needed.

Reload udev rules:
```
sudo udevadm control --reload-rules
sudo udevadm trigger
```
Test that the trackpoint no longer responds to input. The udev method typically works immediately and persists across reboots.

This may have an ugly side effect, that you won't any longer be able to use the touchpad mouse buttons. Sadly the trackpoint and those buttons are controlled by the same device.

The solution is to use the touchpad instead (see below).

Be able to click using the touchpad

Activate tap for clicking in touchpad by pasting following lines in /etc/X11/xorg.confg.d/30-touchpad.conf
```
Section "InputClass"
    Identifier "touchpad"
    Driver "libinput"
    MatchIsTouchpad "on"
    Option "Tapping" "on"
    Option "TappingButtonMap" "lrm"
EndSection
```
The lrm means that:
- 1 finger tap is a left click
- 2 finger tap is a right click
- 3 finger tap is a middle click
You'll need to logout and back in for the change to be applied.

New: Fix systemd-tmpfiles Detected unsafe path transition / → /dev during canonicalization of /dev.

with each system start

Here are some examples of the logs:

systemd-tmpfiles[248]: Detected unsafe path transition / → /dev during canonicalization of /dev.
systemd-tmpfiles[485]: Detected unsafe path transition / → /var during canonicalization of /var.
systemd-tmpfiles[485]: Detected unsafe path transition / → /var during canonicalization of /var/lib.

This suggests that your root dir, /dev and /var are owned by differing users

i3wm

New: Add i3wm python actions.

You can also use it with async

Create the connection object

from i3ipc import Connection, Event
i3 = Connection()

Focus on a window by it's class

tree = i3.get_tree()
ff = tree.find_classed('Firefox')[0]
ff.command('focus')

Watchtower

New: Introduce Canary tokens.

Canary tokens are like motion sensors for your networks, computers and clouds. You can put them in folders, on network devices and on your phones.

Place them where nobody should be poking around and get a clear alarm if they are accessed. They are designed to look juicy to attackers to increase the likelihood that they are opened (and they are completely free).

Our Canarytokens are easy to sprinkle all over and forget about, until you get the notification that matters. They are super lightweight and don’t require installing software or running more background processes that can slow down your PC.

References

Docs
Home feat(kubernetes#Move a pvc between AZ in aws): Move a pvc between AZ in aws

set -e

if [ -z "$1" ] || [ -z "$2" ]; then
  echo "Usage: $0 <pv-name> <new-az>"
  exit 1
fi

PV_NAME=$1
NEW_AZ=$2

 Get Volume ID from PV
VOLUME_ID=$(kubectl get pv $PV_NAME -o jsonpath='{.spec.csi.volumeHandle}')
if [ -z "$VOLUME_ID" ]; then
  echo "Failed to get volume ID for PV $PV_NAME"
  exit 1
fi

PVC_NAME=$(kubectl get pv $PV_NAME -o jsonpath="{.spec.claimRef.name}")
NAMESPACE=$(kubectl get pv  $PV_NAME -o jsonpath="{.spec.claimRef.namespace}")

echo "PVC Name: $PVC_NAME"
echo "Namespace: $NAMESPACE"

echo "Found volume: $VOLUME_ID"

SNAPSHOT_ID=$(aws ec2 create-snapshot --volume-id $VOLUME_ID --description "Migration for $PV_NAME" --query 'SnapshotId' --output text)
echo "Snapshot created: $SNAPSHOT_ID"

echo "Waiting for snapshot to be ready..."
aws ec2 wait snapshot-completed --snapshot-ids $SNAPSHOT_ID
echo "Snapshot $SNAPSHOT_ID is ready"

VOLUME_TYPE=$(aws ec2 describe-volumes --volume-ids $VOLUME_ID --query 'Volumes[0].VolumeType' --output text)
NEW_VOLUME_ID=$(aws ec2 create-volume --snapshot-id $SNAPSHOT_ID --availability-zone $NEW_AZ --volume-type $VOLUME_TYPE --query 'VolumeId' --output text)
echo "New volume created: $NEW_VOLUME_ID"

echo "Waiting for new volume to be available..."
aws ec2 wait volume-available --volume-ids $NEW_VOLUME_ID
echo "New volume $NEW_VOLUME_ID is ready"

NEW_PV_NAME=${PV_NAME}-migrated
cat <<EOF > new-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: $NEW_PV_NAME
spec:
  capacity:
    storage: $(kubectl get pv $PV_NAME -o jsonpath='{.spec.capacity.storage}')
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: $(kubectl get pv $PV_NAME -o jsonpath='{.spec.storageClassName}')
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: $NEW_VOLUME_ID
    fsType: ext4
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.ebs.csi.aws.com/zone
          operator: In
          values:
          - $NEW_AZ
EOF

echo "New PV manifest generated: new-pv.yaml"

kubectl apply -f new-pv.yaml
echo "New PV $NEW_PV_NAME created"
kubectl get pvc $PVC_NAME -n $NAMESPACE -o yaml > ${PVC_NAME}-backup.yaml

echo "If you haven't size to 0 the statefulset, it is the time to kill the pod to rebind the PVC"

kubectl delete pvc $PVC_NAME -n $NAMESPACE
echo "Old PVC deleted"
cat <<EOF > new-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: $PVC_NAME
  namespace: $NAMESPACE
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: $(kubectl get pv $NEW_PV_NAME -o jsonpath='{.spec.capacity.storage}')
  storageClassName: $(kubectl get pv $NEW_PV_NAME -o jsonpath='{.spec.storageClassName}')
  volumeName: $NEW_PV_NAME
EOF

echo "New PVC manifest generated: new-pvc.yaml"

kubectl apply -f new-pvc.yaml
echo "New PVC $PVC_NAME created and bound to new PV"

kubectl delete pv $PV_NAME || echo "Failed to delete PV $PV_NAME, probably was not retained"
echo "Old PV $PV_NAME deleted"

echo "Deleting old volume $VOLUME_ID"
aws ec2 delete-volume --volume-id $VOLUME_ID || echo "Failed to delete volume $VOLUME_ID, probably was not retained"
echo "Old volume deleted"

echo "Deleting snapshot $SNAPSHOT_ID"
echo aws ec2 delete-snapshot --snapshot-id $SNAPSHOT_ID
echo "Snapshot deleted"

echo -e "Migration complete.\nNew PV: $NEW_PV_NAME\nNew PVC: $PVC_NAME"

New: Error opening terminal: xterm-kitty.

The not so good solution but that solves the issue is
```
export TERM=xterm
```
Correction: Change the default docker image.

From containrrr/watchtower to nickfedor/watchtower. The official repo has not been updated in a while use this fork instead

Mobile Keyboards

New: Introduce mobile keyboards comparison.

Finding the right mobile keyboard that balances functionality, privacy, and usability can be challenging. This guide explores the best open-source and privacy-focused keyboard options available for Android devices.

Quick Recommendations
- For Gboard users transitioning: HeliBoard
- For advanced features and AI: FUTO Keyboard
- For unique input method: Thumb-Key
- For future consideration: FlorisBoard (when stable)
FUTO Keyboard ⭐ Recommended

FUTO represents the cutting edge of privacy-focused keyboard technology, incorporating AI features while maintaining offline functionality.

What Makes FUTO Special

FUTO stands out with transformer-based predictions using llama.cpp and integrated voice input powered by whisper.cpp. Unlike other keyboards that require proprietary libraries, FUTO includes swipe/glide typing by default.

The keyboard is currently in pre-alpha, so expect some bugs and missing features. However, the privacy-preserving approach and innovative AI integration make it worth trying.

Key Features

Smart Text Prediction
- Uses pre-trained transformer models for intelligent autocorrect
- Personal language model that learns from your typing (locally only)
- Currently optimized for English, with other languages in development
- Spanish support is still limited
Privacy-First Design
- All AI processing happens on-device
- Your data never leaves your phone
- FUTO doesn't view or store any typing data
- Internet access only for updates and crash reporting (planned to be removed)
Customization Options
- Multilingual typing support
- Custom keyboard layouts
- Swipe typing works well out of the box
Current Limitations
- Pre-alpha software with occasional bugs
- Limited language support beyond English
- Uses a custom "Source First" license (not traditional open source)
- Screen movement issues when using swipe typing
Licensing Concerns

FUTO uses a custom license rather than traditional open source licenses like GPL. While the source code is available, the licensing terms are more restrictive than typical open source projects. The team promises to adopt proper open source licensing eventually, but this transition hasn't happened yet.

Resources
Not there yet
- The futo voice has a weird bug at least in spanish that sometimes adds at the end of the transcription phrases like: Subscribe! or Chau or Thanks for watching my video!. This is kind of annoying and scary (¬º-°)¬
HeliBoard - The Reliable Choice

HeliBoard serves as an excellent middle ground, especially for users transitioning from Gboard.

Why Choose HeliBoard
- Active development: Fork of OpenBoard with regular updates
- No network access: Completely offline operation
- User-friendly: Much simpler than AnySoftKeyboard
- Gboard-like experience: Familiar interface for Google Keyboard users
Trade-offs

The main limitation is glide typing, which requires a closed-source library. This compromises the fully open source nature but provides the swipe functionality many users expect.

Resources
- GitHub Repository
Thumb-Key - The Innovative Alternative

For users willing to try something completely different, Thumb-Key offers a unique approach to mobile typing.

The Thumb-Key Concept

Instead of traditional QWERTY, Thumb-Key uses a 3x3 grid layout with swipe gestures for less common letters. This design prioritizes:
- Large, predictable key positions
- Muscle memory development
- Eyes staying on the text area
- Fast typing speeds once mastered
Best For
- Users open to learning new input methods
- Those who prefer larger touch targets
- Privacy enthusiasts who want to avoid predictive text entirely
- People who find traditional keyboards cramped
The keyboard is highly configurable and focuses on accuracy through key positioning rather than AI predictions.

Resources
- GitHub Repository
FlorisBoard - Future Potential

FlorisBoard shows promise but isn't ready for daily use yet.

Current Status
- Early beta development
- Planned integration with GrapheneOS
- Missing key features like suggestions and glide typing
- Limited documentation available
Worth Watching

While not currently recommended for primary use, FlorisBoard could become a strong contender once it reaches stability.

Resources
- GitHub Repository
- Official Website
Alternative Approaches

Unexpected Keyboard

A minimalist keyboard with a unique layout approach.

Resources
- GitHub Repository
Using Proprietary Keyboards with Restrictions

On privacy-focused ROMs like GrapheneOS and DivestOS, you can use proprietary keyboards while blocking internet access. However, this approach has limitations due to inter-process communication between apps.

Note: This method isn't foolproof, as apps can still potentially communicate through IPC mechanisms.

My Current Setup

After testing various options:
1. Primary choice: FUTO Keyboard with swipe enabled
2. Backup plan: Try FUTO voice input for longer texts when privacy features improve
3. Alternative: Thumb-Key if FUTO doesn't work out
The main issue encountered is screen movement during swipe typing, which may be device-specific.

References and Further Reading

Wireguard

New: Configure the kill switch.

You can configure a kill-switch in order to prevent the flow of unencrypted packets through the non-WireGuard interfaces, by adding the following two lines ‘PostUp‘ and ‘PreDown‘ lines to the ‘[Interface]‘ section:
```
PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT
PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT
```
The ‘PostUp’ and ‘PreDown’ fields have been added to specify an iptables command which, when used with interfaces that have a peer that specifies 0.0.0.0/0 as part of the ‘AllowedIPs’, works together with wg-quick’s fwmark usage in order to drop all packets that are either not coming out of the tunnel encrypted or not going through the tunnel itself. Note that this continues to allow most DHCP traffic through, since most DHCP clients make use of PF_PACKET sockets, which bypass Netfilter. When IPv6 is in use, additional similar lines could be added using ip6tables.

If you want to allow the traffic to your LAN while keeping your kill-switch you can use:
```
PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && iptables -I OUTPUT -p tcp -d 192.168.0.0/24 -j ACCEPT
PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && iptables -D OUTPUT -p tcp -d 192.168.0.0/24 -j ACCEPT
```
Here I'm assuming that your LAN is defined by 192.168.0.0/24.

One way to test if the kill switch works is by deleting the IP address from the wireguard interface
```
sudo ip a del [IP address] dev [interface]
```
Where the [IP address] can be seen using the ip a command.

To gracefully recover from this, you will likely have to use the wg-quick command to take the connection down, then bring it back up.
New: User management.

Wireguard's default user management is not very user friendly as it's difficult to know which key belongs to what user.

I've been looking for WireGuard admin interface UI that is actively maintained but also isn't cloud-based and between all solutions I found wg-easy the best candidate because:
- It has just the features I need:
- Clean User management: add, remove, disable
- Clean UI interface
- Actively maintained: last commit 7h ago
- Really popular: 17.7k stars, 1.7k forks
- It's installable either with docker or docker-compose
- It exports prometheus metrics
- It has good documentation.
- It has an ansible playbook
- It has a grafana dashboard
- It has a clean release process
- It's a stable project: 3 years and 10 months old
If wg-easy doesn't work, I'd look at the next projects:
- WGDashboard (Docs)
- easy-wg-quick
- wireguard-ui
- wg-tool and wg-client
New: Introduce Rosenpass.

Rosenpass is free and open-source software based on the latest research in the field of cryptography. It is intended to be used with WireGuard VPN, but can work with all software that uses pre-shared keys. It uses two cryptographic methods (Classic McEliece and Kyber) to secure systems against attacks with quantum computers.
- Source code
- Docs
New: Add awesome wireguard link.
- Awesome wireguard
New: Introduce wg-easy.

wg-easy is the easiest way to install & manage WireGuard on any Linux hostthe easiest way to install & manage WireGuard on any Linux host

Features:
- All-in-one: WireGuard + Web UI.
- Easy installation, simple to use.
- List, create, edit, delete, enable & disable clients.
- Show a client's QR code.
- Download a client's configuration file.
- Statistics for which clients are connected.
- Tx/Rx charts for each connected client.
- Gravatar support.
- Automatic Light / Dark Mode
- Multilanguage Support
- One Time Links
- Client Expiration
- Prometheus metrics support
- IPv6 support
- CIDR support
Installation

With ansible

References
- Source
- Docs
New: Troubleshoot Failed to resolve interface "tun": No such device.
```
sudo apt purge resolvconf
```
New: More wg-easy configurations.

configure with ansible

Configuration

Split tunneling

Keep in mind though that the WG_ALLOWED_IPS only sets the routes on the client, it does not limit the traffic at server level. For example, if you set 172.30.1.0/24 as the allowed ips, but the client changes it to 172.30.0.0/16 it will be able to access for example 172.30.2.1. The suggested way to prevent this behavior is to add the kill switch in the Pre and Post hooks (WG_POST_UP and WG_POST_DOWN)

Restrict Access to Networks with iptables

If you need to restrict many networks you can use this allowed ips calculator

Kill switch

Monitorization

If you want to use the prometheus metrics you need to use a version greater than 14, as 15 is not yet released (as of 2025-03-20) I'm using nightly.

You can enable them with the environment variable ENABLE_PROMETHEUS_METRICS=true

Scrape the metrics

Add to your scrape config the required information
```
  - job_name: vpn-admin
    metrics_path: /metrics
    static_configs:
      - targets:
          - {your vpn private ip}:{your vpn exporter port}
```
Create the monitor client

To make sure that the vpn is working we'll add a client that is always connected. To do so we'll use linuxserver's wireguard docker

# References

diff --git a/mkdocs.yml b/mkdocs.yml index 77e5ba92b2..ef6703047d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -86,6 +86,7 @@ nav: - Trip management: - Route management: route_management.md - Map management: map_management.md + - dawarich: dawarich.md - Food management: food_management.md - Stock management: - Grocy: grocy_management.md @@ -212,514 +213,522 @@ nav: - Fitness Tracker: - fitness_band.md - Amazfit Band 5: amazfit_band_5.md - - Coding: - - Learning to code: - - code_learning.md - - Frontend developer: frontend_learning.md - - Languages: - - Python: - - python.md - - Project Template: - - coding/python/python_project_template.md - - Command-line Project Template: - - coding/python/python_project_template/python_cli_template.md - - Configure SQLAlchemy for projects without flask: >- - coding/python/python_project_template/python_sqlalchemy_without_flask.md - - Flask Project Template: >- - coding/python/python_project_template/python_flask_template.md - - Microservices Project Template: >- - coding/python/python_project_template/python_microservices_template.md - - Common configurations: - - Create the documentation repository: >- - coding/python/python_project_template/python_docs.md - - Load config from YAML: coding/python/python_config_yaml.md - - Configure SQLAlchemy to use the MariaDB/Mysql backend: - >- - coding/python/python_project_template/python_sqlalchemy_mariadb.md - - Configure Docker to host the application: >- - coding/python/python_project_template/python_docker.md - - Libraries: - - Alembic: coding/python/alembic.md - - asyncio: asyncio.md - - aiocron: aiocron.md - - Apprise: apprise.md - - aiohttp: aiohttp.md - - BeautifulSoup: beautifulsoup.md - - Boto3: boto3.md - - Click: coding/python/click.md - - Dash: coding/python/dash.md - - Dash Leaflet: coding/python/dash_leaflet.md - - DeepDiff: coding/python/deepdiff.md - - FactoryBoy: coding/python/factoryboy.md - - Faker: coding/python/faker.md - - FastAPI: fastapi.md - - Flask: coding/python/flask.md - - Flask Restplus: coding/python/flask_restplus.md - - Folium: coding/python/folium.md - - Feedparser: coding/python/feedparser.md - - Gettext: gettext.md - - GitPython: coding/python/gitpython.md - - Goodconf: goodconf.md - - ICS: ics.md - - Inotify: python_inotify.md - - watchdog: watchdog_python.md - - Jinja2: python_jinja2.md - - Maison: maison.md - - mkdocstrings: coding/python/mkdocstrings.md - - NetworkX: networkx.md - - Org-rw: org_rw.md - - Pandas: coding/python/pandas.md - - Passpy: coding/python/passpy.md - - pexpect: pexpect.md - - Prompt Toolkit: - - coding/python/prompt_toolkit.md - - REPL: prompt_toolkit_repl.md - - Full screen applications: prompt_toolkit_fullscreen_applications.md - - Pydantic: - - coding/python/pydantic.md - - Pydantic Field Types: coding/python/pydantic_types.md - - Pydantic Validators: coding/python/pydantic_validators.md - - Pydantic Exporting Models: coding/python/pydantic_exporting.md - - Pydantic Validating Functions: coding/python/pydantic_functions.md - - Pydantic Factories: pydantic_factories.md - - Pydantic Mypy Plugin: coding/python/pydantic_mypy_plugin.md - - Pypika: coding/python/pypika.md - - psycopg2: psycopg2.md - - Elasticsearch: python_elasticsearch.md - - python-gnupg: python_gnupg.md - - Python Mysql: python_mysql.md - - pythonping: pythonping.md + - Technology: + - Coding: + - Learning to code: + - code_learning.md + - Frontend developer: frontend_learning.md + - Languages: + - Python: + - python.md + - Project Template: + - coding/python/python_project_template.md + - Command-line Project Template: + - coding/python/python_project_template/python_cli_template.md + - Configure SQLAlchemy for projects without flask: >- + coding/python/python_project_template/python_sqlalchemy_without_flask.md + - Flask Project Template: >- + coding/python/python_project_template/python_flask_template.md + - Microservices Project Template: >- + coding/python/python_project_template/python_microservices_template.md + - Common configurations: + - Create the documentation repository: >- + coding/python/python_project_template/python_docs.md + - Load config from YAML: coding/python/python_config_yaml.md + - Configure SQLAlchemy to use the MariaDB/Mysql backend: + >- + coding/python/python_project_template/python_sqlalchemy_mariadb.md + - Configure Docker to host the application: >- + coding/python/python_project_template/python_docker.md + - Libraries: + - Alembic: coding/python/alembic.md + - asyncio: asyncio.md + - aiocron: aiocron.md + - Apprise: apprise.md + - aiohttp: aiohttp.md + - BeautifulSoup: beautifulsoup.md + - Boto3: boto3.md + - Click: coding/python/click.md + - Dash: coding/python/dash.md + - Dash Leaflet: coding/python/dash_leaflet.md + - DeepDiff: coding/python/deepdiff.md + - FactoryBoy: coding/python/factoryboy.md + - Faker: coding/python/faker.md + - FastAPI: fastapi.md + - Flask: coding/python/flask.md + - Flask Restplus: coding/python/flask_restplus.md + - Folium: coding/python/folium.md + - Feedparser: coding/python/feedparser.md + - Gettext: gettext.md + - GitPython: coding/python/gitpython.md + - Goodconf: goodconf.md + - ICS: ics.md + - Inotify: python_inotify.md + - watchdog: watchdog_python.md + - Jinja2: python_jinja2.md + - Maison: maison.md + - mkdocstrings: coding/python/mkdocstrings.md + - NetworkX: networkx.md + - Org-rw: org_rw.md + - Pandas: coding/python/pandas.md + - Passpy: coding/python/passpy.md + - pexpect: pexpect.md + - Prompt Toolkit: + - coding/python/prompt_toolkit.md + - REPL: prompt_toolkit_repl.md + - Full screen applications: prompt_toolkit_fullscreen_applications.md + - Pydantic: + - coding/python/pydantic.md + - Pydantic Field Types: coding/python/pydantic_types.md + - Pydantic Validators: coding/python/pydantic_validators.md + - Pydantic Exporting Models: coding/python/pydantic_exporting.md + - Pydantic Validating Functions: coding/python/pydantic_functions.md + - Pydantic Factories: pydantic_factories.md + - Pydantic Mypy Plugin: coding/python/pydantic_mypy_plugin.md + - Pypika: coding/python/pypika.md + - psycopg2: psycopg2.md + - Elasticsearch: python_elasticsearch.md + - python-gnupg: python_gnupg.md + - Python Mysql: python_mysql.md + - pythonping: pythonping.md + - Python Prometheus: python-prometheus.md + - Python Telegram: + - python-telegram.md + - pytelegrambotapi: pytelegrambotapi.md + - Python VLC: python_vlc.md + - Playwright: playwright.md + - Plotly: coding/python/plotly.md + - questionary: questionary.md + - rich: rich.md + - Ruamel YAML: coding/python/ruamel_yaml.md + - Selenium: selenium.md + - Streamlit: streamlit.md + - SQLAlchemy: coding/python/sqlalchemy.md + - sqlite3: sqlite3.md + - Redis-py: coding/python/redis-py.md + - Requests: requests.md + - Requests-mock: coding/python/requests_mock.md + - Rq: coding/python/rq.md + - python_systemd: python_systemd.md + - sh: python_sh.md + - Talkey: talkey.md + - Tenacity: tenacity.md + - TinyDB: coding/python/tinydb.md + - Torch: torch.md + - Typer: typer.md + - Yoyo: coding/python/yoyo.md + - Type Hints: coding/python/type_hints.md + - Logging: python_logging.md + - Code Styling: coding/python/python_code_styling.md + - Docstrings: coding/python/docstrings.md + - Properties: python_properties.md + - Protocols: python_protocols.md + - Package Management: + - python_package_management.md + - PDM: pdm.md + - pipx: pipx.md + - Pipenv: pipenv.md + - Poetry: python_poetry.md + - Lazy loading: lazy_loading.md + - Plugin System: python_plugin_system.md + - Profiling: python_profiling.md + - Optimization: python_optimization.md + - Anti-Patterns: coding/python/python_anti_patterns.md + - Pytest: + - coding/python/pytest.md + - Parametrized testing: coding/python/pytest_parametrized_testing.md + - Pytest-cases: coding/python/pytest_cases.md + - Pytest-HttpServer: pytest_httpserver.md + - Pytest-xprocess: pytest-xprocess.md + - Internationalization: python_internationalization.md + - Python Snippets: coding/python/python_snippets.md + - Data Classes: coding/python/data_classes.md + - Vue.js: + - vuejs.md + - Vue snippets: vue_snippets.md + - Vuetify: vuetify.md + - Development tools: + - Cypress: cypress.md + - Vite: vite.md + - Vitest: vitest.md + - Bash: + - Bash snippets: bash_snippets.md + - Bash testing: bats.md + - lua: lua.md + - JSON: coding/json/json.md + - SQL: coding/sql/sql.md + - SQLite: sqlite.md + - YAML: coding/yaml/yaml.md + - Promql: coding/promql/promql.md + - Logql: logql.md + - HTML: html.md + - CSS: css.md + - Javascript: + - coding/javascript/javascript.md + - Javascript snippets: javascript_snippets.md + - MermaidJS: mermaidjs.md + - Latex: latex.md + - Graphql: graphql.md + - Qwik: qwik.md + - nodejs: linux/nodejs.md + - JWT: devops/jwt.md + - React: coding/react/react.md + - Coding tools: + - IDES: + - Vim: + - vim.md + - Vim configuration: + - vim_config.md + - Vim Keymaps: vim_keymaps.md + - Vim Package Manager: + - vim_plugin_managers.md + - LazyVim: lazyvim.md + - Packer: vim_packer.md + - UI management configuration: + - Vim foldings: vim_foldings.md + - Vim movement: vim_movement.md + - Tabs vs Buffers: vim_tabs.md + - File management configuration: + - NeoTree: neotree.md + - Telescope: telescope.md + - fzf.nvim: fzf_nvim.md + - Editing specific configuration: + - vim_editor_plugins.md + - Vim formatters: vim_formatters.md + - Vim autocomplete: vim_completion.md + - Vim markdown: vim_markdown.md + - Vim spelling: vim_spelling.md + - Vim autosave: vim_autosave.md + - Coding specific configuration: + - vim_coding_plugins.md + - Treesitter: vim_treesitter.md + - LSP: vim_lsp.md + - Snippets: luasnip.md + - DAP: vim_dap.md + - Git management configuration: + - vim_git.md + - Diffview: diffview.md + - gitsigns: gitsigns.md + - Testing management configuration: vim_testing.md + - Email management: vim_email.md + - Other Vim Plugins: + - linux/vim/vim_plugins.md + - Vim Snippets: vim_snippets.md + - Vim Troubleshooting: vim_troubleshooting.md + - Neovim Plugin Development: vim_plugin_development.md + - Vi vs Vim vs Neovim: vim_vs_neovim.md + - Tridactyl: tridactyl.md + - VSCodium: vscodium.md + - Coding with AI: ai_coding.md + - Git: + - git.md + - Github cli: gh.md + - Forgejo: forgejo.md + - Gitea: gitea.md + - Radicle: radicle.md + - Data orchestrators: + - data_orchestrators.md + - Kestra: kestra.md + - memorious: memorious.md + - Scrapers: + - morph.io: morph_io.md + - ETL: + - Singer: singer.md + - Espanso: espanso.md + - Generic Coding Practices: + - How to code: how_to_code.md + - Program Versioning: + - versioning.md + - Semantic Versioning: semantic_versioning.md + - Calendar Versioning: calendar_versioning.md + - Use warnings to evolve your code: use_warnings.md + - Keep a Changelog: changelog.md + - Writing good documentation: documentation.md + - Conventional comments: conventional_comments.md + - TDD: coding/tdd.md + - GitOps: gitops.md + - Abstract Syntax Trees: abstract_syntax_trees.md + - Software Architecture: + - SOLID: architecture/solid.md + - Domain Driven Design: + - architecture/domain_driven_design.md + - Repository Pattern: architecture/repository_pattern.md + - Service Layer Pattern: architecture/service_layer_pattern.md + - Architecture Decision Record: adr.md + - Database Architecture: architecture/database_architecture.md + - ORM, Query Builder or Raw SQL: architecture/orm_builder_query_or_raw_sql.md + - Microservices: architecture/microservices.md + - Restful APIS: architecture/restful_apis.md + - OCR: + - Table parsing: + - Camelot: camelot.md + - Frontend Development: frontend_development.md + - Park programming: park_programming.md + - Sponsor: sponsor.md + - Issues: issues.md + - DevSecOps: + - devops/devops.md + - Infrastructure as Code: + - Helm: + - devops/helm/helm.md + - Helm Installation: devops/helm/helm_installation.md + - Helm Commands: devops/helm/helm_commands.md + - Helm Secrets: devops/helm/helm_secrets.md + - Helm Git: helm_git.md + - Helmfile: devops/helmfile.md + - Terraform: terraform.md + - Ansible: + - Ansible Snippets: ansible_snippets.md + - Molecule: molecule.md + - Nix: nix.md + - Dotfiles: + - dotfiles.md + - Home Manager: home-manager.md + - Chezmoi: chezmoi.md + - Dotdrop: dotdrop.md + - Infrastructure Solutions: + - Kubernetes: + - devops/kubernetes/kubernetes.md + - Architecture: devops/kubernetes/kubernetes_architecture.md + - Resources: + - Namespaces: devops/kubernetes/kubernetes_namespaces.md + - Pods: devops/kubernetes/kubernetes_pods.md + - ReplicaSets: devops/kubernetes/kubernetes_replicasets.md + - Deployments: devops/kubernetes/kubernetes_deployments.md + - Horizontal Pod Autoscaling: >- + devops/kubernetes/kubernetes_hpa.md + - Volumes: devops/kubernetes/kubernetes_volumes.md + - Services: devops/kubernetes/kubernetes_services.md + - Labels: devops/kubernetes/kubernetes_labels.md + - Annotations: devops/kubernetes/kubernetes_annotations.md + - Ingress: devops/kubernetes/kubernetes_ingress.md + - Jobs: devops/kubernetes/kubernetes_jobs.md + - Kubectl: + - devops/kubectl/kubectl.md + - Kubectl Installation: devops/kubectl/kubectl_installation.md + - Kubectl Commands: devops/kubectl/kubectl_commands.md + - Additional Components: + - Metrics Server: devops/kubernetes/kubernetes_metric_server.md + - Ingress Controller: >- + devops/kubernetes/kubernetes_ingress_controller.md + - External DNS: devops/kubernetes/kubernetes_external_dns.md + - Cluster Autoscaler: >- + devops/kubernetes/kubernetes_cluster_autoscaler.md + - Dashboard: devops/kubernetes/kubernetes_dashboard.md + - Storage Driver: devops/kubernetes/kubernetes_storage_driver.md + - Vertical Pod Autoscaler: >- + devops/kubernetes/kubernetes_vertical_pod_autoscaler.md + - Networking: devops/kubernetes/kubernetes_networking.md + - Debugging: kubernetes_debugging.md + - Backups: + - Velero: velero.md + - Operators: devops/kubernetes/kubernetes_operators.md + - Tools: + - devops/kubernetes/kubernetes_tools.md + - Krew: krew.md + - Ksniff: ksniff.md + - Mizu: mizu.md + - AWS: + - devops/aws/aws.md + - AWS Snippets: aws_snippets.md + - AWS Savings plan: aws_savings_plan.md + - Security groups workflow: devops/aws/security_groups.md + - EKS: devops/aws/eks.md + - EFS: efs.md + - IAM: + - devops/aws/iam/iam.md + - IAM Commands: devops/aws/iam/iam_commands.md + - IAM Debugging: devops/aws/iam/iam_debug.md + - S3: devops/aws/s3.md + - WAF: aws_waf.md + - Databases: + - Redis: architecture/redis.md + - RabbitMQ: rabbitmq.md + - Continuous Deployment: + - ArgoCD: argocd.md + - Continuous Integration: + - devops/ci.md + - Drone: drone.md + - Linters: + - Alex: devops/alex.md + - Flakeheaven: flakeheaven.md + - Flake8: devops/flake8.md + - Markdownlint: devops/markdownlint.md + - Proselint: devops/proselint.md + - Shellcheck: shellcheck.md + - Yamllint: devops/yamllint.md + - Write Good: devops/write_good.md + - Formatters/Fixers: + - Black: devops/black.md + - Yamlfix: yamlfix.md + - Pyment: pyment.md + - mdformat: mdformat.md + - Type Checkers: + - Mypy: devops/mypy.md + - Security Checkers: + - pip-audit: pip_audit.md + - Bandit: devops/bandit.md + - Safety: devops/safety.md + - Dependency managers: + - Pip-tools: devops/pip_tools.md + - Automating Processes: + - copier: copier.md + - cookiecutter: linux/cookiecutter.md + - cruft: linux/cruft.md + - renovate: renovate.md + - letsencrypt: letsencrypt.md + - Threat modeling: + - Privacy threat modeling: privacy_threat_modeling.md + - Storage: + - storage.md + - NAS: nas.md + - OpenZFS: + - linux/zfs.md + - OpenZFS storage planning: zfs_storage_planning.md + - Sanoid: sanoid.md + - ZFS Prometheus exporter: zfs_exporter.md + - Hard drive health: + - hard_drive_health.md + - Smartctl: smartctl.md + - badblocks: badblocks.md + - Resilience: + - linux_resilience.md + - Memtest: memtest.md + - watchdog: watchdog.md + - Magic keys: magic_keys.md + - Monitoring: + - Monitoring Comparison: monitoring_comparison.md + - Prometheus: + - devops/prometheus/prometheus.md + - Architecture: devops/prometheus/prometheus_architecture.md + - Prometheus Operator: devops/prometheus/prometheus_operator.md + - Prometheus Install: devops/prometheus/prometheus_installation.md + - AlertManager: devops/prometheus/alertmanager.md + - Blackbox Exporter: devops/prometheus/blackbox_exporter.md + - cAdvisor: cadvisor.md + - Elasticsearch Exporter: elasticsearch_exporter.md + - Node Exporter: devops/prometheus/node_exporter.md + - Process Exporter: process_exporter.md - Python Prometheus: python-prometheus.md - - Python Telegram: - - python-telegram.md - - pytelegrambotapi: pytelegrambotapi.md - - Python VLC: python_vlc.md - - Playwright: playwright.md - - Plotly: coding/python/plotly.md - - questionary: questionary.md - - rich: rich.md - - Ruamel YAML: coding/python/ruamel_yaml.md - - Selenium: selenium.md - - Streamlit: streamlit.md - - SQLAlchemy: coding/python/sqlalchemy.md - - sqlite3: sqlite3.md - - Redis-py: coding/python/redis-py.md - - Requests: requests.md - - Requests-mock: coding/python/requests_mock.md - - Rq: coding/python/rq.md - - python_systemd: python_systemd.md - - sh: python_sh.md - - Talkey: talkey.md - - Tenacity: tenacity.md - - TinyDB: coding/python/tinydb.md - - Torch: torch.md - - Typer: typer.md - - Yoyo: coding/python/yoyo.md - - Type Hints: coding/python/type_hints.md - - Logging: python_logging.md - - Code Styling: coding/python/python_code_styling.md - - Docstrings: coding/python/docstrings.md - - Properties: python_properties.md - - Protocols: python_protocols.md - - Package Management: - - python_package_management.md - - PDM: pdm.md - - pipx: pipx.md - - Pipenv: pipenv.md - - Poetry: python_poetry.md - - Lazy loading: lazy_loading.md - - Plugin System: python_plugin_system.md - - Profiling: python_profiling.md - - Optimization: python_optimization.md - - Anti-Patterns: coding/python/python_anti_patterns.md - - Pytest: - - coding/python/pytest.md - - Parametrized testing: coding/python/pytest_parametrized_testing.md - - Pytest-cases: coding/python/pytest_cases.md - - Pytest-HttpServer: pytest_httpserver.md - - Pytest-xprocess: pytest-xprocess.md - - Internationalization: python_internationalization.md - - Python Snippets: coding/python/python_snippets.md - - Data Classes: coding/python/data_classes.md - - Vue.js: - - vuejs.md - - Vue snippets: vue_snippets.md - - Vuetify: vuetify.md - - Development tools: - - Cypress: cypress.md - - Vite: vite.md - - Vitest: vitest.md - - Bash: - - Bash snippets: bash_snippets.md - - Bash testing: bats.md - - lua: lua.md - - JSON: coding/json/json.md - - SQL: coding/sql/sql.md - - SQLite: sqlite.md - - YAML: coding/yaml/yaml.md - - Promql: coding/promql/promql.md - - Logql: logql.md - - HTML: html.md - - CSS: css.md - - Javascript: - - coding/javascript/javascript.md - - Javascript snippets: javascript_snippets.md - - MermaidJS: mermaidjs.md - - Latex: latex.md - - Graphql: graphql.md - - Qwik: qwik.md - - nodejs: linux/nodejs.md - - JWT: devops/jwt.md - - React: coding/react/react.md - - Coding tools: - - IDES: - - Vim: - - vim.md - - Vim configuration: - - vim_config.md - - Vim Keymaps: vim_keymaps.md - - Vim Package Manager: - - vim_plugin_managers.md - - LazyVim: lazyvim.md - - Packer: vim_packer.md - - UI management configuration: - - Vim foldings: vim_foldings.md - - Vim movement: vim_movement.md - - Tabs vs Buffers: vim_tabs.md - - File management configuration: - - NeoTree: neotree.md - - Telescope: telescope.md - - fzf.nvim: fzf_nvim.md - - Editing specific configuration: - - vim_editor_plugins.md - - Vim formatters: vim_formatters.md - - Vim autocomplete: vim_completion.md - - Vim markdown: vim_markdown.md - - Vim spelling: vim_spelling.md - - Vim autosave: vim_autosave.md - - Coding specific configuration: - - vim_coding_plugins.md - - Treesitter: vim_treesitter.md - - LSP: vim_lsp.md - - Snippets: luasnip.md - - DAP: vim_dap.md - - Git management configuration: - - vim_git.md - - Diffview: diffview.md - - gitsigns: gitsigns.md - - Testing management configuration: vim_testing.md - - Email management: vim_email.md - - Other Vim Plugins: - - linux/vim/vim_plugins.md - - Vim Snippets: vim_snippets.md - - Vim Troubleshooting: vim_troubleshooting.md - - Neovim Plugin Development: vim_plugin_development.md - - Vi vs Vim vs Neovim: vim_vs_neovim.md - - Tridactyl: tridactyl.md - - VSCodium: vscodium.md - - Coding with AI: ai_coding.md - - Git: - - git.md - - Github cli: gh.md - - Forgejo: forgejo.md - - Gitea: gitea.md - - Radicle: radicle.md - - Data orchestrators: - - data_orchestrators.md - - Kestra: kestra.md - - memorious: memorious.md - - Scrapers: - - morph.io: morph_io.md - - ETL: - - Singer: singer.md - - Espanso: espanso.md - - Generic Coding Practices: - - How to code: how_to_code.md - - Program Versioning: - - versioning.md - - Semantic Versioning: semantic_versioning.md - - Calendar Versioning: calendar_versioning.md - - Use warnings to evolve your code: use_warnings.md - - Keep a Changelog: changelog.md - - Writing good documentation: documentation.md - - Conventional comments: conventional_comments.md - - TDD: coding/tdd.md - - GitOps: gitops.md - - Abstract Syntax Trees: abstract_syntax_trees.md - - Software Architecture: - - SOLID: architecture/solid.md - - Domain Driven Design: - - architecture/domain_driven_design.md - - Repository Pattern: architecture/repository_pattern.md - - Service Layer Pattern: architecture/service_layer_pattern.md - - Architecture Decision Record: adr.md - - Database Architecture: architecture/database_architecture.md - - ORM, Query Builder or Raw SQL: architecture/orm_builder_query_or_raw_sql.md - - Microservices: architecture/microservices.md - - Restful APIS: architecture/restful_apis.md - - OCR: - - Table parsing: - - Camelot: camelot.md - - Frontend Development: frontend_development.md - - Park programming: park_programming.md - - Sponsor: sponsor.md - - Issues: issues.md - - DevSecOps: - - devops/devops.md - - Infrastructure as Code: - - Helm: - - devops/helm/helm.md - - Helm Installation: devops/helm/helm_installation.md - - Helm Commands: devops/helm/helm_commands.md - - Helm Secrets: devops/helm/helm_secrets.md - - Helm Git: helm_git.md - - Helmfile: devops/helmfile.md - - Terraform: terraform.md - - Ansible: - - Ansible Snippets: ansible_snippets.md - - Molecule: molecule.md - - Nix: nix.md - - Dotfiles: - - dotfiles.md - - Home Manager: home-manager.md - - Chezmoi: chezmoi.md - - Dotdrop: dotdrop.md - - Infrastructure Solutions: - - Kubernetes: - - devops/kubernetes/kubernetes.md - - Architecture: devops/kubernetes/kubernetes_architecture.md - - Resources: - - Namespaces: devops/kubernetes/kubernetes_namespaces.md - - Pods: devops/kubernetes/kubernetes_pods.md - - ReplicaSets: devops/kubernetes/kubernetes_replicasets.md - - Deployments: devops/kubernetes/kubernetes_deployments.md - - Horizontal Pod Autoscaling: >- - devops/kubernetes/kubernetes_hpa.md - - Volumes: devops/kubernetes/kubernetes_volumes.md - - Services: devops/kubernetes/kubernetes_services.md - - Labels: devops/kubernetes/kubernetes_labels.md - - Annotations: devops/kubernetes/kubernetes_annotations.md - - Ingress: devops/kubernetes/kubernetes_ingress.md - - Jobs: devops/kubernetes/kubernetes_jobs.md - - Kubectl: - - devops/kubectl/kubectl.md - - Kubectl Installation: devops/kubectl/kubectl_installation.md - - Kubectl Commands: devops/kubectl/kubectl_commands.md - - Additional Components: - - Metrics Server: devops/kubernetes/kubernetes_metric_server.md - - Ingress Controller: >- - devops/kubernetes/kubernetes_ingress_controller.md - - External DNS: devops/kubernetes/kubernetes_external_dns.md - - Cluster Autoscaler: >- - devops/kubernetes/kubernetes_cluster_autoscaler.md - - Dashboard: devops/kubernetes/kubernetes_dashboard.md - - Storage Driver: devops/kubernetes/kubernetes_storage_driver.md - - Vertical Pod Autoscaler: >- - devops/kubernetes/kubernetes_vertical_pod_autoscaler.md - - Networking: devops/kubernetes/kubernetes_networking.md - - Debugging: kubernetes_debugging.md - - Backups: - - Velero: velero.md - - Operators: devops/kubernetes/kubernetes_operators.md - - Tools: - - devops/kubernetes/kubernetes_tools.md - - Krew: krew.md - - Ksniff: ksniff.md - - Mizu: mizu.md - - AWS: - - devops/aws/aws.md - - AWS Snippets: aws_snippets.md - - AWS Savings plan: aws_savings_plan.md - - Security groups workflow: devops/aws/security_groups.md - - EKS: devops/aws/eks.md - - EFS: efs.md - - IAM: - - devops/aws/iam/iam.md - - IAM Commands: devops/aws/iam/iam_commands.md - - IAM Debugging: devops/aws/iam/iam_debug.md - - S3: devops/aws/s3.md - - WAF: aws_waf.md + - Instance sizing analysis: devops/prometheus/instance_sizing_analysis.md + - Prometheus Troubleshooting: >- + devops/prometheus/prometheus_troubleshooting.md + - Grafana: grafana.md + - Log analysis: + - Loki: + - loki.md + - Logcli: logcli.md + - Promtail: promtail.md + - Graylog: graylog.md + - Elastic Security: elastic_security.md + - SIEM: siem.md - Databases: - - Redis: architecture/redis.md - - RabbitMQ: rabbitmq.md - - Continuous Deployment: - - ArgoCD: argocd.md - - Continuous Integration: - - devops/ci.md - - Drone: drone.md - - Linters: - - Alex: devops/alex.md - - Flakeheaven: flakeheaven.md - - Flake8: devops/flake8.md - - Markdownlint: devops/markdownlint.md - - Proselint: devops/proselint.md - - Shellcheck: shellcheck.md - - Yamllint: devops/yamllint.md - - Write Good: devops/write_good.md - - Formatters/Fixers: - - Black: devops/black.md - - Yamlfix: yamlfix.md - - Pyment: pyment.md - - mdformat: mdformat.md - - Type Checkers: - - Mypy: devops/mypy.md - - Security Checkers: - - pip-audit: pip_audit.md - - Bandit: devops/bandit.md - - Safety: devops/safety.md - - Dependency managers: - - Pip-tools: devops/pip_tools.md - - Automating Processes: - - copier: copier.md - - cookiecutter: linux/cookiecutter.md - - cruft: linux/cruft.md - - renovate: renovate.md - - letsencrypt: letsencrypt.md - - Threat modeling: - - Privacy threat modeling: privacy_threat_modeling.md - - Storage: - - storage.md - - NAS: nas.md - - OpenZFS: - - linux/zfs.md - - OpenZFS storage planning: zfs_storage_planning.md - - Sanoid: sanoid.md - - ZFS Prometheus exporter: zfs_exporter.md - - Hard drive health: - - hard_drive_health.md - - Smartctl: smartctl.md - - badblocks: badblocks.md - - Resilience: - - linux_resilience.md - - Memtest: memtest.md - - watchdog: watchdog.md - - Magic keys: magic_keys.md - - Monitoring: - - Monitoring Comparison: monitoring_comparison.md - - Prometheus: - - devops/prometheus/prometheus.md - - Architecture: devops/prometheus/prometheus_architecture.md - - Prometheus Operator: devops/prometheus/prometheus_operator.md - - Prometheus Install: devops/prometheus/prometheus_installation.md - - AlertManager: devops/prometheus/alertmanager.md - - Blackbox Exporter: devops/prometheus/blackbox_exporter.md - - cAdvisor: cadvisor.md - - Elasticsearch Exporter: elasticsearch_exporter.md - - Node Exporter: devops/prometheus/node_exporter.md - - Process Exporter: process_exporter.md - - Python Prometheus: python-prometheus.md - - Instance sizing analysis: devops/prometheus/instance_sizing_analysis.md - - Prometheus Troubleshooting: >- - devops/prometheus/prometheus_troubleshooting.md - - Grafana: grafana.md - - Log analysis: - - Loki: - - loki.md - - Logcli: logcli.md - - Promtail: promtail.md - - Graylog: graylog.md - - Elastic Security: elastic_security.md - - SIEM: siem.md - - Databases: - - PostgreSQL: - - postgres.md - - Postgres operators: - - postgres_operators.md - - Zalando Postgres operator: zalando_postgres_operator.md - - elasticsearch: linux/elasticsearch.md - - Oracle Database: oracle_database.md - - Authentication: - - Authentik: authentik.md - - API Management: - - devops/api_management.md - - Kong: devops/kong/kong.md - - Scrum: - - scrum.md - - Templates: - - Refinement Template: refinement_template.md - - Hardware: - - CPU: cpu.md - - RAM: - - ram.md - - ECC RAM: - - ecc.md - - rasdaemon: rasdaemon.md - - Power Supply Unit: psu.md - - GPU: gpu.md - - Pedal PC: pedal_pc.md - - Pentesting: pentesting.md - - Operating Systems: - - Linux: - - linux.md - - Linux Snippets: linux_snippets.md - - Distros: - - Libreelec: libreelec.md - - Tails: tails.md - - Recovery tools: - - finnix: finnix.md - - Security tools: - - fail2ban: linux/fail2ban.md - - pass: pass.md - - Wireshark: wireshark.md - - Canary tokens: canary_tokens.md + - PostgreSQL: + - postgres.md + - Postgres operators: + - postgres_operators.md + - Zalando Postgres operator: zalando_postgres_operator.md + - elasticsearch: linux/elasticsearch.md + - Oracle Database: oracle_database.md + - Authentication: + - Authentik: authentik.md + - API Management: + - devops/api_management.md + - Kong: devops/kong/kong.md + - Scrum: + - scrum.md + - Templates: + - Refinement Template: refinement_template.md + - Hardware: + - CPU: cpu.md + - RAM: + - ram.md + - ECC RAM: + - ecc.md + - rasdaemon: rasdaemon.md + - Power Supply Unit: psu.md + - GPU: gpu.md + - Pedal PC: pedal_pc.md + - Pentesting: pentesting.md + - Operating Systems: + - Linux: + - linux.md + - Linux Snippets: linux_snippets.md + - Distros: + - Libreelec: libreelec.md + - Tails: tails.md + - Recovery tools: + - finnix: finnix.md + - Security tools: + - fail2ban: linux/fail2ban.md + - pass: pass.md + - Wireshark: wireshark.md + - Canary tokens: canary_tokens.md
- - Sysadmin tools:
- - brew: linux/brew.md
- - detox: detox.md
- - Docker: docker.md
- - Watchtower: watchtower.md
- - Dynamic DNS: dynamicdns.md
- - goaccess: goaccess.md
- - Gotify: gotify.md
- - HAProxy: linux/haproxy.md
- - journald: journald.md
- - LUKS: linux/luks/luks.md
- - Outrun: outrun.md
- - rm: linux/rm.md
- - sed: sed.md
- - Syncthing: linux/syncthing.md
- - Tahoe-LAFS: tahoe.md
- - Wake on Lan: wake_on_lan.md
- - Wireguard:
- - linux/wireguard.md
- - wg-easy: wg-easy.md
- - yq: yq.md
- - zip: linux/zip.md
- - Window manager tools:
- - dunst: dunst.md
- - ferdium: ferdium.md
- - i3wm: i3wm.md
- - rofi: rofi.md
- - User tools:
- - Browsers:
- - google chrome: linux/google_chrome.md
- - Chromium: chromium.md
- - Hushboard: husboard.md
- - Peek: peek.md
- - Terminals:
- - terminal_comparison.md
- - Alacritty: alacritty.md
- - Wezterm: wezterm.md
- - Kitty: kitty.md
- - Instant messaging apps:
- - Delta Chat: deltachat.md
- - Simplex Chat: simplexchat.md
- - Sysadmin tools:
- - brew: linux/brew.md
- - detox: detox.md
- - Docker: docker.md
- - Watchtower: watchtower.md
- - Dynamic DNS: dynamicdns.md
- - goaccess: goaccess.md
- - Gotify: gotify.md
- - HAProxy: linux/haproxy.md
- - journald: journald.md
- - LUKS: linux/luks/luks.md
- - Outrun: outrun.md
- - rm: linux/rm.md
- - sed: sed.md
- - Syncthing: linux/syncthing.md
- - Tahoe-LAFS: tahoe.md
- - Wake on Lan: wake_on_lan.md
- - Wireguard:
- - linux/wireguard.md
- - wg-easy: wg-easy.md
- - yq: yq.md
- - zip: linux/zip.md
- - Window manager tools:
- - dunst: dunst.md
- - ferdium: ferdium.md
- - i3wm: i3wm.md
- - rofi: rofi.md
- - User tools:
- - Browsers:
- - google chrome: linux/google_chrome.md
- - Chromium: chromium.md
- - Hushboard: husboard.md
- - Peek: peek.md
- - Terminals:
- - terminal_comparison.md
- - Alacritty: alacritty.md
- - Wezterm: wezterm.md
- - Kitty: kitty.md
- - Instant messaging apps:
- - Delta Chat: deltachat.md
- - Simplex Chat: simplexchat.md
- - Android:
- - Android Tips: android_tips.md
- - OS:
- - GrapheneOS: grapheneos.md
- - FuriOS: furios.md
- - Apps:
- - Cone: cone.md
- - GadgetBridge: gadgetbridge.md
- - LibreTube: libretube.md
- - HappyCow: happycow.md
- - ICSx5: icsx5.md
- - Orgzly: orgzly.md
- - OsmAnd: osmand.md
- - Seedvault: seedvault.md
- - Android SDK Platform tools: android_sdk.md
- - Android:
- - Android Tips: android_tips.md
- - OS:
- - GrapheneOS: grapheneos.md
- - FuriOS: furios.md
- - Apps:
- - Cone: cone.md
- - GadgetBridge: gadgetbridge.md
- - LibreTube: libretube.md
- - HappyCow: happycow.md
- - ICSx5: icsx5.md
- - Orgzly: orgzly.md
- - OsmAnd: osmand.md
- - Seedvault: seedvault.md
- - Android SDK Platform tools: android_sdk.md
- - Hardware:
- - Redox: redox.md
- - Vial: vial.md
- - Rock64: rock64.md
- - Filosofía:
- - filosofía.md
- - Amor: amor.md
- Arts:
  - Writing:
    - writing/writing.md @@ -746,9 +755,6 @@ nav:
  - Calistenia: calistenia.md
  - Aerial Silk: aerial_silk.md
  - Meditation: meditation.md
- - Maker:
- - Redox: redox.md
- - Vial: vial.md
  - Sudokus: sudokus.md
  - Drawing:
    - drawing/drawing.md @@ -811,6 +817,7 @@ nav:
  - Board Games:
    - board_games.md
    - Regicide: regicide.md
- - Monologues: monologues.md
- Projects: projects.md
- Contact: contact.md

Hardware

Kobo

New: Kobo Forma suddenly drains battery in days.

The solution was to a factory restore, disconnect the wifi and detect which ebook is being the problem.

Some of the user suggestions - My battery started doing this many 3 times now. Every time it was a certain book. The first time was something I downloaded from the kobo store, I removed my recent batch of downloads and it was fine (still not sure which is the offending book tbh). The next two were sideloaded borrowbox books. I think its something to do with a download not being downloaded correctly/having all the correct information. The kobos system tries to keep getting the information it wants but fails and keeps doing this in a cycle causing it to drain the battery
- I haven't bought any new books lately, but I've been reading a ton of library books. About when the battery drain started, I had returned a few books directly from the Kobo, instead of doing it on my phone in the Libby app. That's got to be the problem.
I disabled the wifi, turned off my Kobo, turned it back on, then restarted the wifi, BUT I did it in Settings, instead of with the quick toggle at the top of the Home screen (when you touch the wifi signal strength indicator). It took longer for the wifi to fully connect than it had all the times I've tried this over the last week. Then I did a Sync, which went through quickly.

It's been 8+ hours, and my Kobo is still at 89%, which is where it was when I started the process!
- i did find out that it wasn't going to sleep or powering off if I just closed the cover anymore. It will sleep/ power off if the cover is not closed. I haven't changed any settings so I'm not sure why this started happening. I've had to power it off manually before closing the cover to make sure there is no battery drain.
- Basically, the Forma will either go to sleep on its own or I'll put it to sleep and the battery is at some reasonable amount. Usually overnight, but once while it was sitting on the desk next to me for about an hour, the battery will drain down to nothing. It will shut all the way down and sometimes display the warning that it needs charging. Other times, I don't even get that warning.
References

Rock64

New: Install Debian in a rock64.

Installation
- Go to the rock64 wiki page to get the download directory for the debian version you want to install
- Download firmware.rock64-rk3328.img.gz and partition.img.gz
- Combine the 2 parts into 1 image file: zcat firmware.rock64-rk3328.img.gz partition.img.gz > debian-installer.img
- Write the created .img file to microSD card or eMMC Module using dd: dd if=debian-installer.img of=/dev/sda bs=4M. Replace /dev/sda with your target drive.
- Plug the microSD/eMMC card in the Rock64 (and connect a serial console, or keyboard and monitor) and boot up to start the Debian Installer
Notes:
- An Ethernet connection is required for the above installer
- Remember to leave some space before your first partition for u-boot! You can do this by creating a 32M size unused partition at the start of the device.
- Auto creating all partitions does not work. You can use the following manual partition scheme:
```
 #1 - 34MB  Unused/Free Space
 #2 - 512MB ext2 /boot           (Remember to set the bootable flag)
 #3 - xxGB  ext4 /               (This can be as large as you want. You can also create separate partitions for /home /var /tmp)
 #4 - 1GB   swap                 (May not be a good idea if using an SD card)
```

Software tools

autorandr

New: Introduce autorandr.

autorandr is a command line tool to automatically select a display configuration based on connected devices.

Installation
```
apt-get install autorandr
```
Usage

Save your current display configuration and setup with:
```
autorandr --save mobile
```
Connect an additional display, configure your setup and save it:
```
autorandr --save docked
```
Now autorandr can detect which hardware setup is active:
```
$ autorandr
  mobile
  docked (detected)
```
To automatically reload your setup:
```
$ autorandr --change
```
To manually load a profile:
```
$ autorandr --load <profile>
```
or simply:
```
$ autorandr <profile>
```

Liberaforms

New: Usage of liberaforms.

Marked true

If you are a forms admin you can mark the answers which can be used to for example state that those have been checked and are not trolls, the user can then edit the answers through the magic link but it's not very probable that it becomes a trollo

Send edit email

If you want the users to receive an email with the magic link so that they can edit their answers you need to add a "Short text" field of type "Email" and then in the Options you need to enable the setting to send the users the magic link

Extract the results through API

Each form at the bottom of the Options tab has a section of API endpoints, once enabled you can extract them with curl:

BASE_URL=https://forms.komun.org
form_id=478
curl -sqH "Authorization: Bearer ${JWT_TOKEN}" "${BASE_URL}/api/form/${form_id}/answers"

That will give you an answer similar to:

{
  "answers": [
    {
      "created": "2025-04-25T15:05:02.384121",
      "data": {
        "radio-group-1712945984567": "Hitzaldia--Charla--Xerrada",
        "radio-group-1713092876455": "55",
        "radio-group-1713373271313": "Castellano_1",
        "radio-group-1713382758036": "Si_1",
        "radio-group-1744968040362-0": "d8b35c755d9d41e2a844a344ae2494d6",
        "text-1712945594310": "Historia de la criptograf\u00eda",
        "text-1712945631444": "",
        "text-1712945663611": "user",
        "text-1712947404812": "user@sindominio.net",
        "text-1744967213162-0": "Divulgativa",
        "text-1744967571620-0": "Ninguno",
        "textarea-1712945646944": "Aproximaci\u00f3n hist\u00f3rica a la criptograf\u00eda, desde la Antig\u00fcedad a d\u00eda de hoy",
        "textarea-1712945755946": "",
        "textarea-1712945806547": "HDMI",
        "textarea-1712945865865": "Privacidad, criptograf\u00eda, matem\u00e1ticas, historia",
       "textarea-1713380502724": ""
      },
      "form": 478,
      "id": 36148,
      "marked": false
    }
  ],
  "meta": {}
}
``

As you can see the fields have weird names, to get the details of each field you can do the same request but to `${BASE_URL}/api/form/${form_id}` instead of `${BASE_URL}/api/form/${form_id}/answers`

```json
{
  "form": {
    "created": "2025-04-25T11:45:43.633038",
    "introduction_md": "# Call4Nodes Hackmeeting 2025",
    "slug": "call4nodes-hackmeeting-2025-cas",
    "structure": [
     {
        "className": "form-control",
        "label": "T\u00edtulo",
        "name": "text-1712945594310",
        "required": true,
        "subtype": "text",
        "type": "text"
      },
      {
        "className": "form-control",
        "label": "Descripci\u00f3n",
        "name": "textarea-1712945646944",
        "required": true,
        "type": "textarea"
      },
      ...

Filosofía

New: Pensamientos sueltos.
- En el episodio de Ver al otro: narrativa y democracia de punzadas sonoras hacen un análisis muy interesante sobre cómo la literatura puede ser un mecanismo muy potente de transformación social.
New: Pensamientos sueltos.

Cómo transformar
- Punzadas Sonoras: Ver al otro: narrativa y democracia: cómo la literatura puede ser un mecanismo muy potente de transformación social.
- Punzadas Sonoras: Límite: lugar de enunciación: El límite como elemento transformador
Deseo
- Carne Cruda - PROGRAMAS: El deseo en tiempos de Tinder
Otros
- Punzadas Sonoras: Artesano y artista: desnaturalizar la distinción: Capítulo super interesante para analizar las dinámicas de poder en el mundo laboral, el concepto de mingei, pensar sobre "el arte de programar", ...
Recomiendan dos libros interesantes:
- La belleza del objeto cotidiano - Soetsu Yanagi
- Costumbres en común - E. P. Thompson
- Punzadas Sonoras: Matar al Autor: el destino del texto: No infantilizar al receptor
New: Nuevos capítulos relevantes sobre el tiempo y otras cosas.
- Punzadas Sonoras: Mirar atrás: un gesto íntimo: Ruptura de la concepción lineal del tiempo, aplicado entre otras cosas a las relaciones. Hablan también sobre el artículo de Leonor Cervantes, Ya no te gusto como antes que da una perspectiva muy chula sobre las relaciones.
- Punzadas Sonoras: Artesano y artista: desnaturalizar la distinción: Capítulo super interesante para analizar las dinámicas de poder en el mundo laboral, el concepto de mingei, pensar sobre "el arte de programar", ...
- Punzadas Sonoras: El ritmo del habitar con Blanca Lacasa: Filosofando sobre la casa, en especial sobre la cocina como jaula y reino. También sobre la relación entre madres e hijas

Amor

New: Amor no correspondido.

En el episodio Amor no correspondido: ¿Por qué? de punzadas sonoras hacen un análisis muy interesante. Puedes escucharlo directamente desde aquí.

Arts

Cooking

New: Instrumentos de cocina.
- cómo usar las sartenes de acero inoxidable

Cooking Basics

New: Todos los cortes para una cebolla.

Picada
- Dividir la cebolla por la raiz
- apoyar cada mitad en la tabla con la raiz en perpendicular a ti
- cortes grandes perpendicular a la raiz
- poner la raiz en tu dirección
- cortes grandes paralelos a la tabla
- cortes grandes perpendiculares a la tabla
Juliana
- Dividir la cebolla por la raiz
- apoyar cada mitad en la tabla con la raiz en paralelo a ti
- cortes al gusto de grosor perpendiculares a la tabla
Media luna
- Dividir la cebolla por la raiz
- apoyar cada mitad en la tabla con la raiz en perpendicular a ti
- cortes al gusto de grosor perpendiculares a la tabla
Brunoise
- Dividir la cebolla por la raiz
- apoyar cada mitad en la tabla con la raiz en paralelo a ti
- cortes al gusto de grosor perpendiculares a la tabla sin llegar hasta el final
- poner la raiz en perpendicular
- cortes al gusto de grosor paralelos a la tabla sin llegar hasta el final
- cortes al gusto de grosor perpendiculares a la tabla
Discos
- Sin dividir la cebolla, poner la raiz paralela a la tabla y en perpendicular a ti
- cortes al gusto de grosor perpendiculares a la tabla
Aros
- Corte de discos
- desmontar los discos

Calistenia

New: Introduce calistenia.

Técnica básica

Dominadas
- Video tutorial de jéssica martín
Referencias

Vídeos

+- Jéssica Martín Moreno
New: Sentadilla búlgara.

Hay bastante diversidad de opiniones: 1, 2, 3
New: Añadir vídeos de Cris heria.
- Cris Heria
New: Shrimp squat vs pistol squat.

https://gmb.io/shrimp-squats-vs-pistol-squats/ https://www.reddit.com/r/bodyweightfitness/comments/5loylk/pistol_squats_vs_shrimp_squats/
New: Nordic curl.

Languages

Galego

New: Descubrimiento de os arquivos da meiga.
- Os arquivos da meiga: Foro de contenido en galego

Esperanto

New: Introduce esperanto.

Personal notes on the 38C3 Esperanto gathering
- The language has 130 years
- It's difficult to reform: There is no defined process to change the language and they don't like newbies to propose changes although there is an academy of esperanto
- There are no different past, conditional tenses
- There is no conjugation, they always use the pronouns
Anki decks

There is this nice deck also available from the ankiweb site

References
- 38c3 workshop slides

Science

Artificial Intelligence

New: Add 38C3 talk on AI and ecologism.

38C3 talk: Resource Consumption of AI - Degrow or Die

OCR

New: Add ocr references.
- What is the best LLM based OCR open source available now?
New: Add deekseek ocr tool.
- DeekSeek-OCR

Text to speech

New: Add link to Exploring the World of Open-Source Text-to-Speech Models article.
- Exploring the World of Open-Source Text-to-Speech Models diff --git a/docs/vim_plugin_development.md b/docs/vim_plugin_development.md index 8b70311a61..1fa882233a 100644 --- a/docs/vim_plugin_development.md +++ b/docs/vim_plugin_development.md feat(vim_plugin_development): Explain how to load a plugin in a local directory with lazy
You can manually edit those files to develop new feature or fix issues on the plugins. Or if you're developing them in a directory you can specify the dir directive in the lazy loading.

Birding

New: Introduce android apps for birding.
- whoBIRD
- Merlin Bird ID: I've seen it working and it's amazing, I'm however trying first whoBIRD as it's in F-droid

Brain food

Galas

Galas 2025

New: Añadir las galas de 2025.

Los premios al mejor contenido de 2025.
Mejores libros de 2025
** Transformadores**

Qué hacen un crack dentro de ti que te transforma

# Treatise on efficacy de François Jullien
- disfrute: ★★★★★
- año: 2004
- género: filosofía
- longitud: 211 páginas
Mi libro favorito del año sin duda. Lo he leído 2 veces y media. Mucho subrayado, muchas notas al margen. Es uno de esos libros que te dan unas nuevas gafas para ver el mundo de otra forma.

Ha transformado completamente mi concepción del tiempo, la eficacia, eficiencia, acción, transformación, oportunidad, ...

Eso si, es lectura densa, pero muy muy recomendable.

Me encanta el puente que nos regala Jullien del pensamiento chino vivido y entendido desde la experiencia de ojos europeos.

** El deseo según Gilles Deleuze de Maite Larrauri**
- disfrute: ★★★★★
- año: 2000
- género: filosofía
- longitud: 94 páginas
Lo \"leí\" cómo nunca antes había leído un libro, simplemente maravilloso. Después lo he leído otras 2 veces.

Maite hace accesible a profanos conceptos clave de Deleuze sobre el deseo.

Capitalismo libidinal
- disfrute: ★★★★★
- año: 2024
- género: filosofía
- longitud: 224 páginas
Libro maravilloso que me abrió la mente a otra concepción del tiempo y del deseo.

La vida se ha hecho mercado. Como si fuese nuestra segunda naturaleza, nos movemos en Uber, viajamos con Airbnb, ligamos en Tinder, compramos en Glovo, nos entretenemos en Netflix, hablamos de nosotros mismos en el lenguaje del capital humano.

Esta segunda naturaleza, que Amador Fernández-Savater llama capitalismo libidinal, nos promete la felicidad, pero lo que produce realmente es sufrimiento y malestar, en forma de precariedad, endeudamiento y dolor psíquico. Paradójicamente, la derecha parece hoy más eficaz que nadie para canalizar esa desesperación y su fuerza de rechazo (Trump, Bolsonaro, Milei), mientras que las estrategias de comunicación y las políticas de contención de la izquierda se muestran insuficientes.

¿Es posible reapropiarnos de nuestro malestar como energía de transformación social? Será necesario aprender a escuchar y hablar el lenguaje del cuerpo, imaginar y activar políticas del deseo.

Utopía no es una isla de Layla Martínez
- disfrute: ★★★★★
- año: 2020
- género: ensayo
- longitud: 212 páginas
Una guía de vida, lo he vuelto a leer después de 3 o 4 años. Este libro fue el que me dio las energías para volver a militar tras un largo periodo de hartazgo y desilusión. También me abrió en su momento el camino a buscar la utopía.

El libro te atrapa y es un viaje maravilloso.

The Life-Changing Magic of Tidying Up de Marie Kondo
- disfrute: ★★★★☆
- año: 2014
- género: autoayuda
- longitud: 231 páginas
Me pilló en la mudanza y de nuevo quitando toda la mierda capitalista y las grilladas japas, tiene ideas de trasfondo muy interesantes.

Japanese cleaning consultant Marie Kondo takes tidying to a whole new level, promising that if you properly simplify and organize your house once, you'll never have to do it again. Most methods advocate a room-by-room approach, which doom you to pick away at your piles of stuff forever. The KonMari Method, with its revolutionary category-by-category system, leads to lasting results. In fact, none of Kondo's clients have lapsed (and she still has a three-month wait list).

With detailed guidance for determining which items in your house \"spark joy\" (and which don\'t), this international best-seller featuring Tokyo\'s newest lifestyle phenomenon will help you clear your clutter and enjoy the unique magic of a tidy home - and the calm, motivated mindset it can inspire.

The Bullet Journal method de Ryder Carroll
- disfrute: ★★★★☆
- año: 2018
- género: autoayuda
- longitud: 320 páginas
Un libro que me ha dado muchísimas ideas, pero para disfrutarlo tienes que ser capaz de obviar toda la mierda que lo rodea: su speech sobre su vida típico de libro de autoayuda, su enfoque belicista épico, horrenda visión individualista de que somos culpables de nuestras situaciones y que sólo nosotros podemos arreglarlas con una salida individual.

For years, Ryder Carroll tried countless organizing systems, online and off, but none of them fit the way his mind worked. Out of sheer necessity, he developed a method called the Bullet Journal that helped him become consistently focused and effective. When he started sharing his system with friends who faced similar challenges, it went viral. Just a few years later, to his astonishment, Bullet Journaling is a global movement. The Bullet Journal Method is about much more than organizing your notes and to-do lists. It\'s about what Carroll calls \"intentional living\": weeding out distractions and focusing your time and energy in pursuit of what\'s truly meaningful, in both your work and your personal life. It\'s about spending more time with what you care about, by working on fewer things. Carroll wrote this book for frustrated list-makers, overwhelmed multitaskers, and creatives who need some structure. Whether you\'ve used a Bullet Journal for years or have never seen one before, The Bullet Journal Method will help you go from passenger to pilot of your own life.

Unf*ck Your Habitat de Rachel Hoffman
- disfrute: ★★★★☆
- año: 2017
- género: autoayuda
- longitud: 256 páginas
Impresionantes

Looking Backward, 2000-1887 de Edward Bellamy
- disfrute: ★★★★★
- año: 1888!!
- género: novela, sci-fy
- longitud: 276 páginas
Ha despertado en mi sentimientos e ideas como no lo ha hecho un libro en mucho tiempo. Sobretodo la sorpresa de que mucho de la actualidad es realmente muy antigua.

Empecé leyéndolo en inglés pero no me enteraba de nada. El estilo tampoco me estaba enganchando. Hasta que descubrí que se escribió en el 1888!!! También ayuda a entender aquellas cosas que rechinan, como hablar sólo \"del hombre\".

Es impresionante y triste que ya imagina ideas socialistas aún inalcanzables que me siguen emocionando aún en 2025...

Flipo con la perspectiva feminista de los cuidados, la visión del trabajo, el antipunitivismo (imaginaba ya un mundo sin cárceles) ya en 1888. Me sorprende que habla de desahucios en masa, fraudes millonarios, especulaciones con productos de primera necesidad. Luego tiene ciertos campos en los que es un poco mas meh: la sanidad pública está un poco retrasada, la parte romántica es horrenda, es bastante clasista y tecnócrata. Y es muy curioso a la par que gracioso, con todo los avances que ha sido capaz de imaginar, que no pudiese ver el final de la religión.

El flipe se me relajó cuando vi que el manifiesto comunista se había escrito 40 años antes. Y luego vino el efecto rebote. En vez de ser una lectura inspiradora, que también, me está entrando un poco de desesperación por la mierda de mundo que me está tocando vivir, lo lejos que estamos de un mundo bello de vivir, y a la velocidad a la que nos estamos alejando

Es una bonita crónica del despertar comunista cuando se te cae la venda de los ojos. Apaga la tele, enciende la mente. En el paseo por el barrio pobre hace una bonita descripción del quitar la venda de la deshumanización del pobre como vía de acabar con dicha opresión.

Me ha gustado mucho el final y el tierno epílogo, se mascaba la revolución rusa ya en el 1888 Qué pena que el capitalismo saliese victorioso... Dónde estaríamos ahora si no...

A man being put into a hypnotic sleep, is awakened 113 years later to an entirely new social structure.

Entretenidos

Un pasatiempos agradable. Para días tontos.

Ñu de Pau luque
- disfrute: ★★★★☆
- año: 2024
- género: filosofía
Pau tiene una curiosa manera de filosofar, más cercana a la gente.Se lee muy bien, te ríes bastante a menudo. Eso si, no tiene un sólo capítulo, y eso hace que sea muy jodido dejar de leerlo.

Me hace mucha gracia Curiel, pienso que me gustaría conocerla pero luego pienso, que menudo vértigo!

«Lo más sospechoso de las soluciones es que se las encuentra siempre que se quiere.» Esta frase de Rafael Sánchez Ferlosio abre un libro excepcional, tan brillante como inclasificable. Entre el relato, la autobiografía y el ensayo filosófico, Pau Luque convoca una galería de personajes extravagantes y tiernos para pensar con ellos la incertidumbre que caracteriza a toda existencia humana: bellos italianos de oficio desconocido, boxeadores frustrados, adolescentes con dentaduras caóticas, poetas clandestinas, émulos de san Ignacio de Loyola, filósofos abrumados por las cuestiones prácticas más triviales o swingers confundidos vagan y divagan por las calles de Barcelona, Génova, Ciudad de México o Vilafranca del Penedès. Son criaturas mugrientas y deslumbrantes que se enfrentan a problemas cotidianos pero también trascendentales.

Frente a las soluciones simples (el ñu que se suele utilizar en los crucigramas en español para rellenar huecos --casi un chiste entre los aficionados--) y a las recetas de manual, Pau Luque hace una loa a los secretos, los equívocos, los errores e incluso las contradicciones. Ñu es un iluminador libro de antiayuda.

We have always lived in the castle de Shirley Jackson
- disfrute: ★★★★☆
- año: 1962
- género: novela, drama
- longitud: 187 páginas
Recomendado por Layla Martinez, autora de utopía no es una isla y carcoma, el estilo ne recuerda mucho a Aury en the slow regard of silent things.

La casa tiene mucha presencia como en carcoma. Al principio hubo un momento que casi lo dejo, pero me alegro de haberlo acabado.

El estilo es curioso y agradable de leer, me cuadra que le guste a layla.

Shirley Jackson\'s beloved gothic tale of a peculiar girl named Merricat and her family\'s dark secret

Taking readers deep into a labyrinth of dark neurosis, We Have Always Lived in the Castle is a deliciously unsettling novel about a perverse, isolated, and possibly murderous family and the struggle that ensues when a cousin arrives at their estate.

Una belleza terrible de Edurne Portela
- disfrute: ★★★★☆
- año: 2025
- género: novela histórica
- longitud: 335 páginas
Una mirada novelada a la historia de mujeres troskistas. Al final engancha pero me costaron un poco ciertas partes.

Algunos pensamientos que fui apuntando mientras lo leía:
- Ay qué ilusión me dio ver que había sacado un nuevo libro. Al leer la descripción no me pude contener y empecé a dar palmadas y emitir ruidos nerviosos.
- Ya lo tengo entre mis manos, huele bien, como todos los de gutenberg.
- El primer capítulo me encanta, no solo leo su obra sino que encima la acompaño en esta aventura. Me recuerda un poco a su reflexión con maddie y las fronteras pero está vez está acompañada y la reflexión es más madura. Debe ser que Maddie le dio fuerzas. Al decir que compartía casa con jose ovejero me puse a buscar como loco si eran pareja, y luego pensé, qué más da!
- De nuevo, como en sus otros libros, el estilo de la escritura te atrapa, ay qué bien tener estas 300 páginas por delante.
- Me gusta cómo se centra en las historias de ellas incluso cuando están rodeadas de peña tan importante
- Me gusta mucho su filosofía de escritura, el imaginar sobre el inventar incluso a costa de potencial novelístico. Creo que tiene mucho más fuerza y es mucho más respetuosa por las historias verdaderas que intentan representar.
- Sus finales que siempre son maravillosos Ganas de comentarla con henar, caps y rosa
A Deadly Education de Naomi Novik
- disfrute: ★★★★☆
- año: 2020
- género: novela fantasía mágica
- longitud: 320 páginas
Crea un mundo muy interesante, es como un harry potter más oscuro.

Mejores series de 2025

Este 2025 mis series favoritas han sido:
Impactantes

Contenido original, que deja su huella.

Say nothing
- disfrute: ★★★★★
- año: 2024
- género: drama
- longitud: 9 episodios
Ambientada a lo largo de toda la historia del IRA, es interesante que no cuentan la versión de los vencedores, sin dejar de ser crítica con los vencidos. Me sorprendieron mucho los paralelismos con ETA.

Cada capítulo es una joya. Probablemente la mejor serie que he visto en 2025.

Adolescencia
- disfrute: ★★★★★
- año: 2024
- género: drama, crimen
- longitud: 4 episodios largos e intensos
Serie que engancha muy fuertemente, que nos hace reflexionar sobre la relación adulto - joven, el papel de la tecnología y a donde tiende el mundo.

Creo que es una serie que deberían de ver todos los padres y madres con sus hijes, también en los institutos. Creo que se podrían generar unos debates muy ricos.

El formato también es muy especial. La grabación contínua hace que la acción no te deje un respiro. Yo pensaba que el formato de grabación contínua lo conseguían haciendo cortes y uniéndolo con efectos gráficos, pero no. Cada capítulo está grabado de un tirón, y si se equivocan, a empezar de nuevo. Flipo.

Cómo actua todo el reparto es impresionante.

El único pero que le pongo es que sólo muestra una visión muy pesimista de la juventud, en la que es fácil caer en la desesperación de que no tienen remedio y nos vamos al carajo. Que aunque es cierto, también hay otra gran cantidad de jóvenes que vienen pisando con mucha fuerza sobre los que podríamos aprender mucho.

El mundo de una familia se pone patas arriba cuando Jamie Miller, de 13 años, es arrestado y acusado de asesinar a una compañera de clase. Los cargos contra su hijo les obliga a enfrentarse a la peor pesadilla de cualquier padre.

Tiernas

Contenido de domingo noche, aquel que te abraza, que es calentito, en el que los personajes se convierten parte de tu familia.

Doctor en Alaska
- disfrute: ★★★★★
- año: 1990
- género: comedia, drama, fantasía
- longitud: 110 episodios
Zona de comfort absoluta que ha envejecido genial. Después de verla hace unos años, he revisitado capítulos sueltos con la familia. Si no la has visto es un acompañamiento bueno para el invierno.

Joel Fleishman es un médico recién licenciado. Debido a una cláusula de la letra pequeña del contrato de su beca acaba en la remota y descaradamente extraña ciudad de Cicely (Alaska). Cordialmente bienvenido por el fundador de Cicely, Maurice Minnifield, antiguo astronauta de la NASA, y por el resto de la peña de inadaptados y excéntricos que forma el vecindario, Joel descubre que es cada vez más difícil abandonar la ciudad a la que inconscientemente ha llegado. Todo se complica por la presencia de Maggie O'Connell, alcaldesa de Cicely y piloto de la localidad, una mujer hermosa pero completamente independiente; ambientado todo ello por el musical y filosófico programa de radio presentado por Chris en la KBHR.

Graciosas

Contenido que arranca carcajadas

Animal S01
- disfrute: ★★★★★
- año: 2025
- género: comedia
- longitud: 9 episodios
La serie más graciosa que he visto en el año. Carcajada tras carcajada, la retranca gallega en todo su explendor. Mais unha pena que non fose en galego :(

Antón, un veterinario sin un duro, acepta trabajar en una tienda de mascotas de lujo y pasa de cuidar animales en el campo a vender cucaditas y caprichos para perros mimados.

División Palermo S02
- disfrute: ★★★★☆
- año: 2025
- género: comedia
- longitud: 6 episodios
Quizá es por la pérdida de novedad, pero no me reí tanto como con la primera. Aun así tiene muchos puntos muy muy graciosos.

Una Guardia Urbana inclusiva, ideada como operación de marketing para mejorar la imagen de las fuerzas de seguridad, descubrirá algo que no debía y se enfrentará con unos extraños narcos.

The Bear S03
- disfrute: ★★★★☆
- año: 2024
- género: comedia, drama
- longitud: 6 episodios
Su majestad S01
- disfrute: ★★★★☆
- año: 2025
- género: comedia
- longitud: 7 episodios
Me la vi casi del tirón en un momento de bajona anímica. Es muy divertido cómo se mete con los borbones, pero duele que consigue que al final empatices con ella. Maldito cine.

Cuando se ve salpicado por un escándalo financiero, el rey Alfonso XIV decide apartarse de la primera línea pública durante unos meses. Pero alguien debe quedarse al frente de la institución. No hay más alternativa que su única hija, Pilar. Ahora la princesa tendrá que demostrarle al país que no es la irresponsable, vaga e inútil que todos creen. Lo que pasa es que igual tienen razón.

Entretenidas

Un pasatiempos agradable. Para días tontos.

Detective Touré
- disfrute: ★★★★☆
- año: 2024
- género: comedia, misterio
- longitud: 6 episodios
Es bastante divertida y está genial cómo meten el contenido político

Touré, un inmigrante guineano asentado en Bilbao, se gana la vida como improvisado detective de poca monta. Su pericia, intuición y particular sentido del humor le hacen ganarse la confianza de la Ertzaintza y le sumergen en una compleja investigación en la que se enfrentará a múltiples peligros: conocerá de cerca la corrupción inmobiliaria, los tejemanejes del conflictivo barrio de San Francisco e incluso se topará con la mafia nigeriana.

Andor S02
- disfrute: ★★★★☆
- año: 2025
- género: comedia, crímen
- longitud: 12 episodios
No le llega a la suela del zapato de la primera temporada (que si no has visto, merece mucho la pena). Pero sigue siendo entretenida, empieza flojilla y va ganando con los episodios.

Son bonitos los guiños a Palestina

Eric
- disfrute: ★★★★☆
- año: 2025
- género: drama, misterio
- longitud: 6 episodios
Un padre desesperado y un tenaz policía luchan contra sus propios demonios en la Nueva York de los años 80 mientras buscan a su hijo de nueve años, que ha desaparecido.

La residencia S01
- disfrute: ★★★★☆
- año: 2025
- género: comedia, crímen
- longitud: 8 episodios
Empieza un poco flojilla pero con el tiempo va mejorando.

Una excéntrica y brillante inspectora investiga un crimen en la Casa Blanca, donde todos los empleados e invitados a la cena de Estado esconden un secreto y podrían ser el asesino.

Mejores películas de 2025

Este año mis películas favoritas han sido:
Es curioso cómo ha pesado el cine antíguo frente al moderno.

Transformadoras

Barrio
- disfrute: ★★★★★
- año: 1998
- género: drama
- longitud: 1h34m
En uno de esos barrios situados al sur de las grandes ciudades, a los que no llega ni el metro ni el dinero, Javi, Manu y Rai son compañeros de instituto, pero, sobre todo, amigos. Tienen esa edad en la que ni se es hombre ni se es niño, en la que se habla mucho de chicas y muy poco con ellas. Comparten también la vida en el barrio, el calor del verano y un montón de problemas. El primero es el propio barrio, un lugar de grandes bloques de viviendas sociales, de ladrillo oscuro y arquitectura deprimente y depresiva. Allí hay pocas cosas que hacer, y en agosto aún menos. El centro de la ciudad queda lejos y las comunicaciones son malas, así que los tres amigos pasan la mayor parte del tiempo por las calles del barrio.[

No Other Land
- disfrute: ★★★★★
- año: 2024
- género: documental
- longitud: 1h25m
Podría haber caído tanto en impactante como en transformadora. Una producción muy necesaria tras más de un año de genocidio en Palestina...

Un joven activista palestino documenta durante cinco años los violentos desalojos de palestinos en Cisjordania. Con la colaboración de cineastas israelíes y palestinos, narra la compleja relación que surge entre él y un periodista israelí dispuesto a unirse a su lucha, a pesar de sus diferentes circunstancias. Una obra conmovedora que explora resistencia, empatía y esperanza en un contexto de conflicto.

Cómo tener sexo
- disfrute: ★★★★☆
- año: 2023
- género: drama
- longitud: 1h31m
Película que sería interesante ver con los jóvenes ya sea en casa o en los institutos. Aborda muy bien el tema del consentimiento.

Tres adolescentes británicas se van de vacaciones para celebrar sus ritos de iniciación: beber, salir de fiesta y ligar, en lo que debería ser el mejor verano de sus vidas.

Graciosas

Contenido que arranca carcajadas.

Arsénico por compasión
- disfrute: ★★★★★
- año: 1944
- género: comedia, crimen
- longitud: 1h58m
La vi dos veces del tirón en una semana, me parece una obra maestra. Siento que ya no se hacen comedias como esta. El guión es maravilloso, y la traducción también. Aunque soy un aférrimo defensor de ver las películas en versión original, esta recomendaría verla en castellano.

Un crítico teatral que acaba de casarse decide visitar a sus ancianas tías antes de marcharse de luna de miel. Durante la visita descubrirá que las encantadoras viejecitas tienen una manera muy peculiar de practicar la caridad.

El gran dictador
- disfrute: ★★★★★
- año: 1940
- género: comedia, bélica
- longitud: 2h05m
El discurso final es memorable.

Un humilde barbero judío tiene un parecido asombroso con el dictador de la nación Tomania, que promete sacar adelante y que culpa a los judíos de la situación del país. El dictador ataca al país fronterizo, pero es confundido con el barbero por sus propios guardias, siendo ingresado en un campo de concentración. Simultáneamente, el pobre barbero es confundido con el dictador...

Tiempos modernos
- disfrute: ★★★★★
- año: 1936
- género: comedia, drama, romance
- longitud: 1h27m
Es muy triste que sea tan actual...

Extenuado por el frenético ritmo de la cadena de montaje, un obrero metalúrgico acaba perdiendo la razón. Después de recuperarse en un hospital, sale y es encarcelado por participar en una manifestación en la que se encontraba por casualidad. En la cárcel, también sin pretenderlo, ayuda a controlar un motín, gracias a lo cual queda en libertad. Una vez fuera, reemprende la lucha por la supervivencia en compañía de una joven huérfana a la que conoce en la calle.

El apartamento
- disfrute: ★★★★★
- año: 1960
- género: comedia, drama, romance
- longitud: 2h05m
Otra comedia impresionante, con crítica social.

C.C. Baxter es un modesto pero ambicioso empleado de una compañía de seguros de Manhattan. Está soltero y vive solo en un discreto apartamento que presta ocasionalmente a sus superiores para sus citas amorosas. Tiene la esperanza de que estos favores le sirvan para mejorar su posición en la empresa. Pero la situación cambia cuando se enamora de una ascensorista que resulta ser la amante de uno de los jefes que usan su apartamento.

La vida segun Philomena Cunk
- disfrute: ★★★★☆
- año: 2024
- género: comedia
- longitud: 1h10m
Conmovedoras

Coco
- disfrute: ★★★★★
- año: 2017
- género: animación, aventura
- longitud: 1h49m
Segunda vez que la veo y no pierde. Al principio me dio mucho miedo de lo que podía hacer disney con méxico, pero la verdad es que es una obra de arte.

Un joven aspirante a músico llamado Miguel se embarca en un viaje extraordinario a la mágica tierra de sus ancestros. Allí, el encantador embaucador Héctor se convierte en su inesperado amigo y le ayuda a descubrir los misterios detrás de las historias y tradiciones de su familia.

Entretenidas

Un pasatiempos agradable. Para días tontos.

La princesa prometida
- disfrute: ★★★★★
- año: 1987
- género: Aventura, comedia, fantasía, romance
- longitud: 1h45m
Me resistí mucho tiempo a verla porque tenía miedo que fuese una ñoñería de amor romantico. Que tristemente lo es xD, pero todo lo demás hace que sea una maravilla. Los diálogos, las coreografías, los personajes... Eso si, prepárate para exasperarte ante el papel de princesa objeto inútil.

Me volvió a despertar el sentimiento de \"ya no se hacen películas como estas\". Tiene un algo que lo despierta (además de la retrogradez machista)

Después de buscar fortuna durante cinco años, Westley retorna a su tierra para casarse con su amada, la bella Buttercup, a la que había jurado amor eterno. Sin embargo, para recuperarla habrá de enfrentarse a Vizzini y sus esbirros. Una vez derrotados éstos, tendrá que superar el peor de los obstáculos: el príncipe Humperdinck pretende desposar a la desdichada Buttercup, pese a que ella no lo ama, ya que sigue enamorada de Westley.

Kneecap
- disfrute: ★★★★★
- año: 2024
- género: Comedia, drama
- longitud: 1h45m
Me encanta que sean ellos mismos los que actuan, la historia está muy chula y el grupo mola todo.

En Irlanda hay 80.000 hablantes de irlandés, 6.000 viven en el norte y tres de ellos lo van a poner todo patas arriba cuando formen un trío de rap llamado Kneecap. Anárquicos, salvajes y dispuestos a todo para salvar su lengua materna.

Inside Out 2
- disfrute: ★★★★★
- año: 2024
- género: animación, aventura, comedia
- longitud: 1h36m
Riley entra en la adolescencia y el Cuartel General de su cabeza sufre una repentina reforma para hacerle hueco a algo totalmente inesperado propio de la pubertad: ¡nuevas emociones! Alegría, Tristeza, Ira, Miedo y Asco, con años de impecable gestión a sus espaldas (según ellos...) no saben muy bien qué sentir cuando aparece con enorme ímpetu Ansiedad. Y no viene sola: le acompañan envidia, vergüenza y aburrimiento.

El tercer hombre
- disfrute: ★★★★★
- año: 1949
- género: misterio, suspense
- longitud: 1h44m
Orson Welles es Harry Lime y Joseph Cotten actúa como su amigo de la infancia, Holly Martins, en este clásico thriller de Graham Greene, dirigido por Carol Reed. Martins busca a Lime en la caótica postguerra de Viena, y se encuentra así mismo metido en un entorno de amores, decepción y asesinatos.

Los Rose
- disfrute: ★★★★☆
- año: 2025
- género: comedia, drama, romance
- longitud: 1h45m
La vida parece fácil para la pareja perfecta que forman Ivy y Theo: carreras de éxito, un matrimonio feliz y unos hijos estupendos. Pero detrás de la fachada de su supuesta vida ideal, se avecina una tormenta: la carrera de Theo se desploma mientras que las ambiciones de Ivy despegan, lo que desencadena una caja de Pandora de competitividad y resentimiento ocultos.

Una historia verdadera
- disfrute: ★★★★☆
- año: 2025
- género: comedia, drama, romance
- longitud: 1h45m
Alvin Straight (Richard Farnsworth) es un achacoso anciano que vive en Iowa con una hija discapacitada (Sissy Spacek). Además de sufrir un enfisema y pérdida de visión, tiene graves problemas de cadera que casi le impiden permanecer de pie. Cuando recibe la noticia de que su hermano Lyle (Stanton), con el que está enemistado desde hace diez años, ha sufrido un infarto, a pesar de su precario estado de salud, decide ir a verlo a Wisconsin. Para ello tendrá que recorrer unos 500 kilometros, y lo hace en el único medio de transporte del que dispone: una máquina cortacésped.

El mago de Oz
- disfrute: ★★★★☆
- año: 1939
- género: aventura, fantasía
- longitud: 1h43m
La gran evasión
- disfrute: ★★★★☆
- año: 1963
- género: aventura, bélica, drama
- longitud: 2h43m
La quimera del oro
- disfrute: ★★★★☆
- año: 1925
- género: aventura, comedia, drama
- longitud: 1h35m
Aventuras de un solitario buscador de oro en Alaska, donde se topa con varios personajes rudos, y se enamora de la hermosa Georgia, a la que trata de conquistar.

Mujeres al borde de un ataque de nervios
- disfrute: ★★★★☆
- año: 1988
- género: comedia, drama
- longitud: 1h28m
Pepa e Iván son actores de doblaje. Él es un mujeriego empedernido y, después de una larga relación, rompe con Pepa: le deja un mensaje en el contestador pidiéndole que le prepare una maleta con sus cosas. Al quedarse sola, Pepa no soporta vivir en una casa llena de recuerdos y decide alquilarla. Mientras espera que Iván vaya a recoger la maleta, la casa se le va llenando de gente extravagante de la que aprenderá muchas cosas sobre la soledad y la locura.

Mejores podcasts de 2025
1. Punzadas sonoras
2. Quieto todo el mundo
3. (de eso no se habla) temporada 2: Se llamaba como yo
4. No es el fin del mundo
Punzadas sonoras

Paula Ducay e Inés García me han recordado la importancia de la Filosofía como nadie. Gracias a ellas he descubierto nuevas maneras de ver, me han regalado palabras que describen a la percepción pensamientos y sensaciones que tengo y gracias a ellas he conseguido esclarecer un poco más mi entendimiento del mundo y de mi mismo. En especial han sido la base para desenterrar pilares imprescindibles de mi vida como son el deambular creativo y la solitud.

Siento que mi camino con ellas ha sido perfecto, me enamoré de ellas en los derroteros de Carne cruda en 2023. Eran episodios de 20 minutos muy accesibles, y una vez que ya me enganché a ellas, los programas de una hora se hacen cortos.

Escucharlas es un absoluto placer.

Quieto todo el mundo

Facu Díaz y Miguel Maldonado han conseguido alegrarme las horas de cocina de los lunes con la manera más amable de acercarme desde la comedia a la actualidad política. Hacen un combo maravilloso.

Con un tono divertido y relajado, en el programa van comentando desde la ignorancia, la improvisación y, en ocasiones, la desidia las noticias más importantes de la semana.

Esta misa roja ha creado una verdadera religión a lo largo de los programas, no sólo por el lore sino por el vacío existencial que dejan esa semana que no publican programa. Especialmente cuando es porque los nazis les ocupan los espacios, asco de pais...

(de eso no se habla) temporada 3: Se llamaba como yo

En «De eso no se habla» hablamos de los silencios que crea esa frase, tanto en nosotras como en la sociedad: de las historias que se esconden detrás de ellos, y de qué pasa cuando los rompemos.

Esta tercera temporada trata sobre la memoria de la niña Begoña Urroz, sobre las cinco décadas de silencio de una familia… Y sobre el ruido que lo rompió. Muestra cómo la guerra sucia del estado contra ETA

No es el fin del mundo

Una mirada muy interesante a la geopolítica mundial

El podcast semanal de El Orden Mundial (EOM) para entender qué pasa en el mundo. Análisis, contexto y matices sobre la realidad internacional. Porque estar al día de qué pasa más allá de nuestras fronteras no debería ser ni complicado ni aburrido.

Mejores videojuegos de 2025

Mi juego favorito de este año ha sido [Thronefall]{.spurious-link target="Thronefall"}

Thronefall
- disfrute: ★★★★★
- platforms: linux
- year: 2024
- genre: Strategy, tactical, puzzle, arcade
Al principio era un poco escéptico porque tienes unas localizaciones fijas para las torres, pero sigue siendo un juego difícil y entretenido. Además que los gráficos son preciosos.

A minimalist game about building and defending a little kingdom. Thronefall is a classic strategy game without unnecessary complexity, just some healthy hack-and-slay. Build up your base during the day, and defend it 'til your last breath at night.

Dragonsweeper
- disfrute: ★★★★★
- year: 2025
- genre: puzzle
- platform: desktop, mobile
- price: free
Simple de jugar pero muy adictivo.

A minesweeper game that requires observation by mixing roguelike elements with the classic gameplay.

You Must Build A Boat
- disfrute: ★★★★★
- platforms: android, linux
- year: 2015
- genre: puzzle
You Must Build A Boat is the sequel to the award winning \"10000000\". Travel the world, run procedurally generated dungeons finding artifacts, capturing monsters and recruiting crew for your...

10000000 {#section-1}
- disfrute: ★★★★★
- platforms: android, linux
- year: 2012
- genre: puzzle
10000000 is an award winning hybrid RPG/Action/Puzzle game. Matching tiles controls your character enabling you to explore, fight and loot. When you are not facing monsters you will be back in your prison, constructing buildings and getting stronger for your next run.

Books

New: Mejores libros del 2024.

2024 ha sido un año muy potente para mi en cuanto a lectura se refiere,

La mayor parte de los 23 libros que me he terminado han tenido buena puntuación.

5: 8 4: 7 3: 6 2: 1 1: 1

6 se han quedado por el camino y sólamente recuerdo uno que se me haya hecho pesado de leer y me he forzado a terminar de leerlo.

Política

The global police state by William I. Robinson

William ha puesto palabras bonitas y claras a mis pensamientos como no lo hacia un libro desde el manifiesto comunista hace muchos años. Un análisis impoluto sobre la crisis del capitalismo y el mundo al que nos estamos dirigiendo. A la vez que dando una dirección a la que apuntar para combatirlo. Ambiciosa y difícil, pero la que más me cuadra. Me encantaría debatirlo con la gente y lo regalaré allá donde vaya. Da un poco de yuyu porque hasta el final final final no da atisbo de luz al final del túnel tan necesaria en estos tiempos oscuros, pero aguantad, que merece la pena (✿◠‿◠).

A muchos nos aterra el nuevo auge del fascismo. Solo en Europa, la extrema derecha integra cinco gobiernos y tiene representación parlamentaria destacada en veintisiete países. Pero esto es apenas la punta del iceberg de un proceso bastante más complejo: el auge del Estado policial global como respuesta a la profunda crisis del sistema capitalista actual. A medida que el neoliberalismo dispara las desigualdades hasta límites insospechados (los veintiséis millonarios más importantes del mundo poseen hoy más de la mitad de la riqueza mundial mientras dos mil millones de personas viven en situación de pobreza), los individuos se vuelven «desechables». Una población excedente que supone una amenaza de rebelión para la clase capitalista. Para refrenarla, se hacen ubicuos todo tipo de sistemas de control, rastreos biométricos, encarcelamientos generalizados, barcos‐prisión, violencia policial, persecución de migrantes, represión contra activistas medioambientales, eliminación de prestaciones sociales, desahucios, precarización de las clases medias, guerras estratégicas sustentadas por capital privado... Así, el Estado policial global no remite ya a un mecanismo policial y militar, sino a la propia economía global como totalidad represiva, cuya lógica es tan mercantil como política y cultural. Y, mientras la codicia infinita de la clase dominante hunde al capitalismo en una crisis sin precedentes (llevando la degradación ecológica y el deterioro social a su límite absoluto), el neofascismo afianza su posición en ese Estado policial global cuyo objetivo es la exclusión coercitiva de la humanidad excedente. Basándose en datos estremecedores y argumentos incontrovertibles, William I. Robinson demuestra hasta qué punto el capitalismo del siglo XXI se ha convertido en un sistema absoluto de represión como único método para mantenerse en pie más allá de sus contradicciones terminales, y defiende la urgencia de crear un movimiento que trascienda los meros llamados a la justicia social y ataque a la yugular.

Joyful militancy by carla bergman and Nick Montgomery

Un libro muy interesante desde todas las perspectivas. Por cómo debió de ser el proceso creativo, por el cuidado y respeto a todos los distintos movimientos que representa, por las personas entrevistadas y las ideas que transmiten... Para mi ha sido un libro clave para una de las transformaciones más importantes de concepto de vida que he dado este año, entrar más en contacto con mi deseo y dejar que este fluya sobre las rigideces autoimpuestas entre otras cosas por el concepto del deber. Esto aplicado a mi vida en general y a mi militancia en particular.

Es cierto que la mayor parte de los conceptos transgresores son heredados del feminismo, pero el libro los refleja muy bien y puede ser un buen punto de entrada para los que no nos hemos zambullido aún muy profundamente en leer teoría feminista.

Why do radical movements and spaces sometimes feel laden with fear, anxiety, suspicion, self-righteousness and competition? The authors call this phenomenon rigid radicalism: congealed and toxic ways of relating that have seeped into radical movements, posing as the ‘correct’ way of being radical. In conversation with organizers and intellectuals from a wide variety of currents, the authors explore how rigid radicalism smuggles itself into radical spaces, and how it is being undone. Rather than proposing ready-made solutions, they amplify the questions that are already being asked among movements. Fusing together movement-based perspectives and contemporary affect theory, they trace emergent forms of trust, care and responsibility in a wide variety of radical currents today, including indigenous resurgence, anarchism, transformative justice, and youth liberation. Joyful Militancy foregrounds forms of life in the cracks of Empire, revealing the ways that fierceness, tenderness, curiosity, and commitment can be intertwined.

Interviewees include Silvia Federici, adrienne maree brown, Marina Sitrin, Gustavo Esteva, Tasnim Nathoo, Kian Cham, Leanne Betasamosake Simpson, Sebastian Touza, Walidah Imarisha, Margaret Killjoy, Glen Coulthard, Richard Day, Melanie Matining, Zainab Amadahy and Mik Turje.

Verano sin vacaciones. Las hijas de la Costa del Sol por Ana geranios

Libro que dolorosamente me quitó la venda de los ojos en cuanto al turismo y la restauración. Tiene un formato perfecto, la primera parte (Verano sin vacaciones) te llega a la patata haciéndote vivir en las entrañas lo podrido que está el sector y luego en la segunda (Las hijas de la Costa del Sol) le da forma de ensayo y te llega al coco.

Lo leímos en un club de lectura muy chulo organizado por la Escuela de las Periferias, que junto a Estuve aquí y me acordé de nosotros de Anna Pacheco, nos ayudó a tener unas discusiones super interesantes que terminaron de definir mi nuevo concepto sobre el turismo. Además tuve la suerte de asistir a una mesa redonda impresionante con Ana, Valeria del Sindicato de Inquilinas y dos compas de la PAH que le dieron distintos matices a la problemática de la vivienda que tenemos que sufrir. Y para colmo luego estuvimos rajando en un parque con Ana y luego dimos un paseo por el barrio. Un final maravilloso para un libro fantástico.

¿Cómo sería un mundo sin hostelería? ¿Es posible pensar en una sociedad en la que ninguna persona tuviera que servir ni ser servida, donde las bandejas no tuvieran ninguna utilidad?

Este libro no va de eso. Es justo lo contrario: el análisis de un sector económico que se enriquece gracias al trabajo de quienes se dedican a servir a un público que puede permitírselo.

Verano sin vacaciones es el diario de una trabajadora del sector hostelero de la costa malagueña; un relato al que se suma Las hijas de la Costa del Sol, un ensayo situado que nos interpela como turistas, pero también nos hace comprender qué hay detrás de una industria que descansa sobre la explotación laboral, el servilismo político y la voracidad ecológica.

El leitmotiv es hacernos preguntas, dialogar, pensar, compartir; imaginarnos, ahora sí, cómo sería un mundo sin hostelería.

Thinking in systems by Donella H. Meadows

Me ha encantado. Al principio dudaba de la autora por no saber de que pie cojeaba pero al final es un libro que abre mentes. Muchas ganas de resumirlo. Al principio del año descubrí el concepto de systems thinking a way of making sense of the complexity of the world by looking at it in terms of wholes and relationships rather than by splitting it down into its parts. It has been used as a way of exploring and developing effective action in complex contexts, enabling systems change. Systems thinking draws on and contributes to systems theory and the system sciences.

Me maravilló la idea de tener un sistema nuevo y sistemático de analizar el mundo, me encantan los modelos y creo que systems thinking puede llegar a ser muy potente. De los cuatro libros que me leí del tema este es sin duda el mejor.

This is a primer that brings you to a tangible world where anyone can understand systems and engage with them in meaningful ways. The problems we face – war, hunger, poverty, climate change, racism, gender-based violence cannot be solved by quick fixes in isolation. We need to see the whole system and reach deeper to the structures and mindsets that are at play. Written with a hopeful and visionary tone, Thinking in Systems helps readers overcome confusion and helplessness, which is a first step in the work of change.

Novela

Mejor la ausencia de Edurne Portela

Lo empecé el 6 de marzo y esa noche me leí 209 páginas. A las 5:35 dije que ya era suficiente, aunque me fuese rabia no terminarmelo en un día. Hoy ha caído en la madrugada del 7 al 8 de marzo. Muy icónico todo, ya no me da rabia haberlo acabado hoy. Me ha maravillado, cómo escribe Edurne, es una pasada. Te coje con la primera frase y es tu cuerpo el que suplica que dejes de leer. No hay piedad. El cambio de lenguaje a medida que va avanzando la vida de Amaia es alucinante . Cómo le da un repaso a todo el conflicto de Euskadi visto desde alguien que sin estar dentro está salpicada. No se si tuvo miedo al publicarlo, es bastante crítica con toda la movida. Me sorprende porque ella si que está politizada. He pensado que me gustaría preguntarle su opinión respecto a lo que pasó y como ella lo vivió. He pensado varias veces a lo largo de la novela si es autobiográfica. Es impresionante que sea su primera novela.

Ha sido un año del (no estoy seguro de si bien llamado) conflicto vasco, ya que también leí Las fieras de Clara Usón que también me gustó mucho.

Crecer siempre implica alguna forma de violencia, contra uno mismo o contra aquellos que quieren imponer su autoridad. Cuando además la vida trascurre en un pueblo de la margen izquierda del Nervión durante los años 80 y 90, y todo es heroína, paro, detritus medioambiental, cuando en las calles silban cada semana las pelotas de goma y los gases lacrimógenos y las paredes están llenas de consignas asesinas, la violencia no es sólo un problema personal. Mejor la ausencia nos presenta una familia destruida, atravesada por la violencia de su entorno. Amaia, la pequeña de cuatro hermanos, narra ese entorno brutal desde su mirada de niña y adolescente. Compartimos con ella su miedo, su perplejidad, su rabia, ante un padre que hiere, una madre que se esconde, tres hermanos que, como ella, sólo buscan salir adelante.

Amaia es la joven que se enfrenta, hasta alcanzar sus propios límites, a este mundo hostil. Amaia es también la mujer que años después vuelve a su pueblo para encontrarse con un pasado irresuelto. En ese camino de ida y vuelta, en sus huidas y regresos, descubrirá, a su pesar, que nadie escapa del entorno en el que se cría, de la familia que le toca en suerte. Y que reconocerlo es la única manera de sobrevivir.

To kill a mockingbird de Harper Lee

Inglés muy dificil. La historia muy bien contada, es como que todo el pueblo es de la familia. Engancha bastante.

One of the best-loved stories of all time, To Kill a Mockingbird has been translated into more than 40 languages, sold more than 30 million copies worldwide, served as the basis for an enormously popular motion picture, and voted one of the best novels of the 20^th century by librarians across the United States. A gripping, heart-wrenching, and wholly remarkable tale of coming-of-age in a South poisoned by virulent prejudice, it views a world of great beauty and savage inequities through the eyes of a young girl, as her father -- a crusading local lawyer -- risks everything to defend a black man unjustly accused of a terrible crime.

Lawyer Atticus Finch defends Tom Robinson -- a black man charged with the rape of a white girl. Writing through the young eyes of Finch's children Scout and Jem, Harper Lee explores with rich humor and unswerving honesty the irrationality of adult attitudes toward race and class in small-town Alabama during the mid-1930s Depression years. The conscience of a town steeped in prejudice, violence, and hypocrisy is pricked by the stamina and quiet heroism of one man's struggle for justice. But the weight of history will only tolerate so much.

Gestión del tiempo

Four thousand weeks by Oliver Burkeman

Me ha encantado. Me flipa encontrar un libro de auto ayuda y time management con una perspectiva bastante anticapitalista. Probablemente es el mejor libro de gestión de tiempo que conozco, ha generado en mi esos momentos preciosos en los que surgen ideas fuera de los límites mentales que tenía antes. Ha sido bastante liberador y ha influido mucho en crear mi nueva manera de entender el tiempo y cómo navegarlo. Lo he utilizado mucho este año para rediseñar todos mis roadmap adjustments, en especial el trimestral y el anual. Muy muy recomendable.

The average human lifespan is absurdly, outrageously, insultingly brief: if you live to 80, you have about four thousand weeks on earth. How should we use them best?

Of course, nobody needs telling that there isn't enough time. We're obsessed by our lengthening to-do lists, our overfilled inboxes, the struggle against distraction, and the sense that our attention spans are shrivelling. Yet we rarely make the conscious connection that these problems only trouble us in the first place thanks to the ultimate time-management problem: the challenge of how best to use our four thousand weeks.

Four Thousand Weeks is an uplifting, engrossing and deeply realistic exploration of this problem. Rejecting the futile modern obsession with 'getting everything done,' it introduces readers to tools for constructing a meaningful life, showing how the unhelpful ways we've come to think about time aren't inescapable, unchanging truths, but choices we've made, as individuals and as a society - and its many revelations will transform the reader's worldview.

Drawing on the insights of both ancient and contemporary philosophers, psychologists, and spiritual teachers, Oliver Burkeman sets out to realign our relationship with time - and in doing so, to liberate us from its grasp.

Essentialism: The disciplined pursuit of less

Me apesta su tono emprendedor de sueño americano y su falta de perspectiva de clase. Eso sumado a su prepotencia mandaloriana de "this is the way"" con el essentialist path y non essentialist path hace que la lectura sea bastante horrenda. Dicho esto, si consigues abstraerte de toda esa mierda, el author aporta conceptos interesantes que me han ayudado mucho a diseñar las revisiones trimestrales y anuales del roadmap adjustments. Es especialmente interesante para aquellas personas que no sabemos decir que no y acabamos enfrascados en mil movidas.

Essentialism isn't about getting more done in less time. It's about getting only the right things done. Have you ever found yourself stretched too thin? Do you simultaneously feel overworked and underutilized? Are you often busy but not productive? Do you feel like your time is constantly being hijacked by other people's agendas? If you answered yes to any of these, the way out is the Way of the Essentialist. Essentialism is more than a time-management strategy or a productivity technique. It is a systematic discipline for discerning what is absolutely essential, then eliminating everything that is not, so we can make the highest possible contribution toward the things that really matter. By forcing us to apply more selective criteria for what is Essential, the disciplined pursuit of less empowers us to reclaim control of our own choices about where to spend our precious time and energy -- instead of giving others the implicit permission to choose for us. Essentialism is not one more thing. It's a whole new way of doing everything. It's about doing less, but better, in every area of our lives.

Ensayo

La novia grulla de CJ Hauser

Me ha encantado cómo trata temas tan complicados, su filosofía de vida y lo bien que está escrito. Una ventana a todo tipo de "amor" desde la "nueva" perspectiva feminista. Me entraron muchas ganas de leerlo ya que no paraban de mencionarlo en Punzadas sonoras, me parece que Paula e Inés me van a dar muy buen material de lectura, aunque no pueda seguirlas el ritmo ni de lejos xD.

Una mirada franca y divertida sobre el amor, la intimidad y la identidad en el siglo XXI.

Diez días después de cancelar su boda, CJ Hauser se embarcó en una expedición a Texas para estudiar a la grulla trompetera. Tras una semana chapoteando en las marismas del golfo comprendió que había estado a punto de firmar un contrato para vivir la vida de otra persona.

¿Qué pasaría si decidiéramos liberarnos de la idea tradicional de felicidad y nos abriéramos a lo inesperado? Hauser se sirve de su propia experiencia para explorar las relaciones sentimentales, los fracasos amorosos, la intimidad y la identidad en el siglo XXI. Disecciona la personalidad de los protagonistas de Expediente X mientras intenta entender qué es el amor, rememora sus peores citas de Tinder, chatea con desconocidos que conversan como robots y analiza a Katharine Hepburn en Historias de Filadelfia para aprender a no perderse en una relación.

Divertido, inclasificable y brutalmente sincero, este libro trata de cómo modelamos nuestra vida sentimental y nuestra comprensión de los demás a través de los relatos; una lectura para aquellos que aprenden a encontrar la alegría en el no saber e intentan, aunque a veces fracasen, construir particulares formas de vida, familia y hogar.

Las dos amigas de Tony Morrison

Un libro breve, inquietante y curioso que te hace cuestionarte a ti mismo. Me gustaría dar una reflexión más larga pero es imposible no meter spoilers. Cuando te lo leas lo hablamos :P

Podcasts

New: Mejores podcasts de 2024.

Estoy muy contento porque este año he descubierto podcast muy buenos, hasta el punto de considerar unidireccionalmente a varias de ellas ya parte de mi familia.

Quieto todo el mundo

Facu Díaz y Miguel Maldonado han conseguido alegrarme las horas de cocina de los lunes con la manera más amable de acercarme desde la comedia a la actualidad política.

Con un tono divertido y relajado, en el programa van comentando desde la ignorancia, la improvisación y, en ocasiones, la desidia las noticias más importantes de la semana.

Esta misa roja ha creado una verdadera religión a lo largo de los programas, no sólo por el lore sino por el vacío existencial que dejan esa semana que no publican programa. Especialmente cuando es porque los nazis les ocupan los espacios, asco de pais...

Punzadas sonoras

Paula Ducay e Inés García me han recordado la importancia de la Filosofía como nadie. Gracias a ellas he descubierto nuevas maneras de ver, me han regado palabras que describen a la percepción pensamientos y sensaciones que tengo y gracias con ellas he conseguido esclarecer un poco más mi entendimiento del mundo y de mi mismo. En especial han sido la base para desenterrar pilares imprescindibles de mi vida como son el deambular creativo y la solitud.

Siento que mi camino con ellas ha sido perfecto, me enamoré de ellas en los derroteros de Carne cruda de la temporada pasada (una pena enorme que no las hayan renovado). Son episodios de 20 minutos muy accesibles, y una vez que ya me enganché a ellas, los programas de una hora se hacen cortos.

Escucharlas es un absoluto placer.

No hay negros en el Tibet

Asaari Bibang, Lamine Thior y Frank T nos regalan un par de veces al mes una ventana a la realidad de ser una persona afrodescendiente en España. Es un podcast muy amable de escuchar, te partes con las puyas que se echan, además de ser super interesante ver el salto generacional que hay entre Frank y (Asaari y Lamine). Eso si, tienes que aprender a vivir con que se interrumpan continuamente. Al principio esto me rayó bastante ya que es especialmente irritante cuando se lo hacen a Asaari. Luego ves que no tienen filtro y no sólo se cortan entre ellas, también lo hacen con las personas invitadas.

Una manera muy fácil de introducir el antiracismo en tu vida ya que con sus relatos eres capaz de verlo, identificártelo, aprender y corregir elementos racistas que tenemos inculcados por este sistema asqueroso. Y creo que como material divulgativo es perfecto ya que lo abordan desde un punto de vista antipunitivista nada agresivo, sin que por ello se dejen títere sin cabeza.

Carne cruda

Violeta Muñoz y Javier Gallego han sido la voz seria de la actualidad para mi este año.

Si es cierto que esta temporada estoy un poco más desencantado con el equipo ya que:
- Ya no cuentan con Punzadas Sonoras para los derroteros y a Santiago Alba Rico nunca le he tragado.
- Hace tiempo que no llaman a Pablo Elorduy del Salto, me encantaba oírle y le echo de menos :(.
- Han cambiado el Nido de rojos de la temporada pasada por "A diestra y siniestra" en el que incluyen a peña de derechas en los debates para darle otro punto de vista. Ya se van regulando pero los primeros programas tuve que dejar de escucharlo con cabreo porque tenía que escuchar cosas como que el genocidio de palestina estaba justificado y lindezas similares. Entiendo que puede darle color al debate, pero para eso ya me voy a los medios tradicionales la verdad...
Aun así sigue siendo el referente para mi para saber qué está pasando tanto en el mundo en general como en el mundo activista.

Muriendo porque saquen más

Este año hay unos cuantos podcast que no han continuado y que me encantarían que volviesen como:
- (de eso no se habla)
- Gabinete de curiosidades

Videogames

DragonSweeper

New: Introduce dragonsweeper.

DragonSweeper is an addictive simple RPG-tinged take on the Minesweeper formula. You can play it for free in your browser.

If you're lost at the beginning start reading the ArsTechnica blog post.

Tips
- Use Shift to mark numbers you already know.
References

Age of Empires

New: New teutons vs portuguese video.
- Teutons vs Portuguese | 1v1 Arabia | Hera vs MbL

Board Games

New: Add cooperative games references.

Favourite coop games

Monologues

New: Add Sammy Obeid.
- Sammy Obeid: Antizionist comedian

2024

Lyz — 2025-01-01 06:45:54+00:00

Activism

New: Introduction to activism.

Activism consists of efforts to promote, impede, direct or intervene in social, political, economic or environmental reform with the desire to make changes in society toward a perceived greater good.
New: Recomendar el episodio de podcast el diario de Jadiya.

Diario de Jadiya (link al archivo): es algo que todo xenófobo debería de escuchar, es un diario de una chavala saharaui que formó parte del programa de veranos en familias españolas.
New: Add Rafeef Ziadah pro-palestine poem.

Rafeef Ziadah - "Nosotros enseñamos vida, señor"
New: Añadir el vídeo del racismo no se sostiene.

El racismo no se sostiene
New: Nuevo artículo contra el turismo.
- Abolir el turismo - Escuela de las periferias: Lleguemos a donde lleguemos, no puede ser que sea más fácil imaginar el fin del capitalismo que el fin del turismo.

Antifascism

New: Animaros a colaborar con el Crowdfunding para los 6 de Zaragoza.

Crowdfunding para la libertad de los 6 de Zaragoza

Hacktivism

Chaos Communication Congress

New: Introduce the CCC.

Chaos Communication Congress is the best gathering of hacktivism in europe.

Prepare yourself for the congress

You can follow MacLemon's checklist

Install the useful apps

The schedule app

You can use either the Fahrplan app or giggity, I've been using the second for a while, so is the one I use

The navigation app

c3nav is an application to get around the congress. The official F-droid is outdated, so add their repository to get the latest version.

Reference - [Home](https://events.ccc.de/en/ - Engelsystem - Angel FAQ
New: Introduce the Angel's system.

Angels are participants who volunteer to make the event happen. They are neither getting paid for their work nor do they get free admission.

Expectation

Helping at our events also comes with some simple, but important expectations of you:
- Be on time for your shift or give Heaven early notice.
- Be well rested, sober and not hungry.
- Be open-minded and friendly in attitude.
- Live our moral values:
- Be excellent to each other.
- All creatures are welcome.
Quickstart
- Create yourself an Engelsystem account
- Arrive at the venue
- Find the Heaven and go there.
- Talk to a welcome angel or a shift coordinator to get your angel badge and get marked as arrived.
- If you have any questions, you can always ask the shift coordinators behind the counter.
- Attend an angel meeting
- Announced in the Engelsystem news
- Click yourself an interesting shift
- Read shift descriptions first
- Participate in your shift
- Use the navigation to find the right place.
- Arrive a little bit early at the meeting point
- Rest for at least one hour
- Repeat from step 5
And always, have a lot of fun.

To get more insights read this article

The engelsystem

The Engelsystem is the central place to distribute the work to all the helping angels. It can be a bit overwhelming at the beginning but you will get used to it and find your way around.

As you might have seen there are many different shifts and roles for angels — some sounding more appealing than others. There are shifts where you need to have some knowledge before you can take them. This knowledge is given in introduction meetings or by taking an unrestricted shift in the team and getting trained on the job. These introduction meetings are announced in the Engelsystem under the tab "Meetings". Heaven and the teams try to make sure that there are only restrictions for shifts in place where they are absolutely needed.

Most restrictions really only need a meeting or some unrestricted shifts at the team to get lifted. Harder restrictions are in place where volunteers need to have special certification, get access to certain systems with a huge amount of data (e.g. mail-queues with emails from participants) or handling big piles of money. Usually the requirements for joining an angeltype are included in the description of the angeltype.

Especially the restricted shifts are tempting because after all we want to get the event running, aren’t we? From our personal experience what gets the event running are the most common things: Guarding a door, collecting bottle/trash, washing dishes in the angel kitchen, being on standby to hop in when spontaneous help is needed or check the wrist band at the entrance.

If there are any further questions about angeltypes, the description of the angeltype usually includes contact data such as a DECT number or an e-mail address that can be used. Alternatively, you can also ask one of the persons of the respective angeltype mentioned under "Supporter".

Teams

Congress is organized from different teams, each with its own area of expertise.

All teams are self-organized and provide their own set of services to the event.

Teams spawn into existence by a need not fulfilled. They are seldom created by an authority.

Check out the different teams to see which one suits you best.

Some poeple suggest not to try to fit into special roles at your first event. The roles will find you – not the other way around. Our community is not about personal growing but about contributing to each other and growing by doing this.

Perks

Being an angel also comes with some perks. While we hope that participation is reward enough, here is a list of things that are exclusive to angels:
- Community acknowledgement
- Hanging out in Heaven and the angel hack center with its chill out area
- Free coffee and (sparkling) water
- Warm drinks or similar to make the cold night shifts more bearable
Rewards

If you have contributed a certain amount of time, you may receive access to:
- Fantastic hot vegan and vegetarian meals
- The famous limited™ angel T-shirt in Congress design
- Maybe some other perks

Feminism

Privileges

New: Add nice video on male privileges.

La intuición femenina, gracias al lenguaje

Free Knowledge

Correction: Update the way of seeding ill knowledge torrents.

A good way to contribute is by seeding the ill torrents. You can generate a list of torrents that need seeding up to a limit in TB. If you follow this path, take care of IP leaking, they're

Free Software

New: Recomendar el artículo El software libre también necesita jardineros.
- El software libre también necesita jardineros

Conference organisation

New: Software to manage the conference.

There are some open source software that can make your life easier when hosting a conference:
- Frab
- Pretalx
- Wafer
In addition to the management of talks from the call for papers till the event itself it can help the users visualise the talks schedule with EventFahrplan which is what's used in the ChaosComputerClub congress.

If you also want to coordinate helpers and shifts take a look to Engelsystem

Ludditest

New: Nice comic about the luddites.

Comic about luddites

Life Management

Time management

New: Anticapitalist approach to time management.

Time management is being used to perpetrate the now hegemonic capitalist values. Its a pity because the underlying concepts are pretty useful and interesting but they are oriented towards improving productivity and being able to deal with an increasing amount of work. Basically they're always telling you to be a better cog. It doesn't matter how good you are, there is always room for improvement. I've fallen on this trap for a long time (I'm still getting my head out of the hole) and I'm trying to amend things by applying the concepts on an anticapitalist mindset. The turning point was to read Four thousand weeks: Time management for mortals by Oliver Burkeman, the article will have book extracts mixed with my way of thinking.

Some (or most) of what's written in this article may not apply if you're not a male, white, young, cis, hetero, European, university graduated, able-bodied, """wealthy""" person. You need to be at a certain level of the social ladder to even start thinking in these terms. And depending on the number of oppressions you're suffering you'll have more or less room to maneuver. That margin is completely outside our control so by no means we should feel guilty of not being able to manage time. What follows are just guidelines to deal with this time anxiety imposed by the capitalist system with whatever air we have to breath.

Changing the language

The easiest way to change the underlying meaning is to change the language. Some substitutions are:
- work -> focus: Nowadays we use work everywhere even if it's not in the laboral environment. For example I work very hard to achieve my goals. Working is the action of selling your time and energies in order to get the resources you need to live. It has an intrinsic meaning of sacrifice, doing something we don't want to do to get another thing in return. That's a tainted way of thinking about your personal time. I find focus is a great substitute as it doesn't have all those connotations. There are similar substitutions based on the same argument, such as: workspace -> space of focus, workflow -> action flow or just flow.
- task -> action: Similar to work a task is something you kind of feel obliged to do. It uses a negative mindset to set the perfect scenario to feel guilty when you fail to do them. But you're on your personal time, it should be fine not to do an action for whatever reason. Action on the other side fosters a positive way of thinking, it suggests change, movement in a way that helps you move forward. There are also other derived words such as task manager -> action manager.
- productivity -> efficiency: Productivy is the measurement of how fast or good you create products. And products are something that is made to be sold. Again this introduces a monetary mindset on all aspects of our life. Efficiency on the other side is the quality of achieving the largest amount of change using as little time, energy or effort as possible (Cambridge doesn't agree with me though :P. It may be because universities are also another important vector of spreading the capitalist values :(). So using efficiency we're focusing more on improving the process itself, so it can be applied for example on how to optimize your enjoyment of doing nothing. Which is completely antagonistic to the concept of productivity.
Changing the mindset

There is a widespread feeling that we're always short on time. We're obsessed with our overfilled inboxes and lengthening todo lists, haunted by the guilty feeling that we ought to be getting more done, or different things done, or both. At the same time we're deluged with advice on living the fully optimized life to squeeze the most from your time. And it get's worse as you age because time seems to speed up as you get older, steadily accelerating until months begging to flash by in what feels like minutes.

The real problem isn't our limited time. It's that we've unwittingly inherited, and feel pressured to live by a troublesome set of ideas about how to use our limited time, all of which are pretty much guaranteed to make things worse. What follows are a list of mindset changes from the traditional time management bibliography that can set the bases of a healthier Anticapitalist one.

Time is not a resource to spend

Before timetables life rhythms emerged organically from the tasks they needed to do. You milked the cows when they needed milking and harvested the crops when it was harvest time. Anyone who would tried imposing an external schedule on any of that, for example, doing a month's milking in a single day to get it out of the way would rightly have been considered a lunatic.

There was no need to think of time as something abstract and separate from life. In those days before clocks, when you did need to explain how long something might take, your only option was to compare it with some other concrete activity. They were untroubled by any notion of time "ticking away" thus living a heightened awareness of the vividness of things, the feeling of timelesness. Also known as living in deep time, or being in the flow, when the boundary separating the self from the rest of reality grows blurry and time stands still.

There's one huge drawback in giving so little thought to the abstract idea of time, though, which is that it severely limits what you can accomplish. As soon as you want to coordinate the actions of more than a handful of people, you need a reliable, agreed-upon method of measuring time. This is why the first mechanical clocks came to be invented.

Making time standardized and visible in this fashion inevitably encourages people to think of it as an abstract thing with an independent existence, distinct from the specific activities on which one might spend it. "time" is what ticks away as the hands move around the clock face.

The next step was to start treating time as a resource, something to be bought and sold and used as efficiently as possible. This mindset shift serves as the precondition for all the uniquely modern ways in which we struggle with time today. Once time is a resource to be used, you start to feel pressure, whether from external forces or from yourself, to use it well, ant to berate yourself when you feel you've wasted it. When you're faced with too many demands, it's easy to assume that the only answer must be to make better use of time, by becoming more efficient, driving yourself harder, or working longer instead of asking whether the demands themselves might be unreasonable.

Soon your sense of self-worth gets completely bound up with how you're using time: it stops being merely the water in which you swim and turns into something you fell you need to dominate or control if you're to avoid feeling guilty, panicked or overwhelmed.

The fundamental problem is that this attitude towards time sets up a rigged game in which it's impossible ever to feel as though you're doing well enough. Instead of simply living our lives as they unfold in time it becomes difficult not to value each moment primarily according to its usefulness for some future goal, or for some future oasis of relaxation you hope to reach once your tasks are finally "out of the way".

Ultimately it backfires. It wrenches us out of the present, leading to a life spent leaning into the future, worrying about whether things will work out, experiencing everything in terms of some later, hoped-for benefit, so that peace of mind never quite arrives. And it makes it all but impossible to experience the flow, that sense of timeless time which depends on forgetting the abstract yardstick and plunging back into the vividness of reality instead.

If you don't disavow capitalism an increase in efficiency will only make things worse

All this context makes us eager to believe the promises of time management frameworks (like GTD) that if you improve your efficiency you'll get more time to enjoy your life. If you follow the right time management system, build the right habits, and apply sufficient self-discipline, you will win the struggle with time.

Reality then kicks in you never win the struggle and only feel more stressed and unhappy. You realize that all the time you've saved is automatically filled up by more things to do in a never ending feedback loop. It's true that you get more done, and yet, paradoxically, you only feel busier, more anxious and somehow emptier as a result. Time feels like an unstoppable conveyor belt, bringing us new actions as fast as we can dispatch the old ones; and becoming more efficient just seems to cause the belt to speed up. Or else, eventually, to break down.

It also has another side-effect. As life accelerates, everyone grows more impatient. It's somehow vastly more aggravating to wait two minutes for the microwave than two hours for the oven, or ten seconds for a slow loading web page versus three days to receive the same information by mail.

Denying reality never works though. It may provide some immediate relief, because it allows you to go on thinking that at some point in the future you might, at last, feel totally in control. But it can't ever bring the sense that you're doing enough (that you are enough) because it defines enough as a kind of limitless control that no human can attain. Instead, the endless struggle leads to more anxiety and less fulfilling life. For example, the more you believe yo might succeed in "fitting everything in", the more commitments you naturally take on, and the less you feel the need to ask whether each new commitment is truly worth a portion of your time, and so your days inevitably fill with more activities you don't especially value. The more you hurry, the more frustrating it is to encounter tasks that won't be hurried, the more compulsively you plan for the future, the more anxious you feel about any remaining uncertainties, of which there will always be plenty.

Time management used this way serves as a distraction to numb our minds:
- It may hide the sense of precariousness inherent to the capitalist world we live in. If you could meet every boss's demand, while launching various side projects on your own, maybe one day You'd finally feel secure in your career and your finances.
- Divert your energies from fully experiencing the reality in which you find yourself, holding at bay certain scary questions about what you're doing with your life, and whether major changes might not be needed. As long as you're always just on the cusp of mastering time, you can avoid the thought that what life is really demanding from you might involve surrendering the craving for mastery and diving into the unknown instead.
Embrace the finitude of time

We recoil from the notion that this is it. That this life, with all its flaws and inescapable vulnerabilities, its extreme brevity, and our limited influence over how it unfolds, is the only one we'll get a shot at. Instead, we mentally fight against the way things are, so that we don't have to consciously participate in what it's like to feel claustrophobic, imprisoned, powerless, and constrained by reality.

Our troubled relationship with time arises largely from this same effort to avoid the painful constrains of reality. And most of our strategies for becoming more efficient make things worse, because they're really just ways of furthering the avoidance. After all, it's painful to confront how limited your time is, because it means that tough choices are inevitable and that you won't have time for all you once dreamed you might do. It's also painful to accept the limited control over the time you do get: maybe you simply lack the stamina or talent or other resources to perform well in all the roles you feel you should. And so, rather than face our limitations, we engage in avoidance strategies, in an effort to carry on feeling limitless. We push ourselves harder, chasing fantasies of the perfect work-life balance, or we implement time management systems that promise to make time for everything, so that tough choices won't be required. Or we procrastinate, which is another means of maintaining the feeling of omnipotent control over life, because you needn't risk the upsetting experience of failing at an intimidating project if you never even start it. We fill our minds with busyness and distraction to numb ourselves emotionally. Or we plan compulsively, because the alternative is to confront how little control over the future we really have.

Heal yourself from FOMO

In practical terms, a limit-embracing attitude to time means organizing your days with the understanding that you definitely won't have time for everything you want to do, or that other people want you to do, and so, at the very least, you can stop beating yourself up for failing. Since hard choices are unavoidable, what matters is learning to make them consciously, deciding what to focus on and what to neglect, rather than letting them get made by default, or deceiving yourself that, with enough hard work and the right time management tricks, you might not have to make them at all. It also means resisting the temptation to "keep your options open" in favour of deliberately making big, daunting, irreversible commitments, which you can't know in advance will turn out for the best, but which reliably prove more fulfilling in the end. And it means standing firm in the face of FOMO (fear of missing out) because you come to realize that missing out on something (indeed on almost everything) is basically guaranteed. Which isn't actually a problem anyway, it turns to, because "missing out" is what makes your choices meaningful in the first place. Every decision to use a portion of time on anything represents the sacrifice of all the other ways in which you could have spent that time, but didn't, and to willingly make that sacrifice is to take a stand, without reservation, on what matters most to you.

Embrace your control limits

The more you try to manage your time with the goal of achieving a feeling of total control and freedom from the inevitable constrains of being human, the more stressful, empty, and frustrating life gets. But the more you confront the facts of finitude instead, and work with them, rather than against them, the more efficient, meaningful and joyful life becomes. Anxiety won't ever completely go away, we're even limited, apparently, in our capacity to embrace our limitations. But I'm aware of no other time management technique that's half as effective as just facing the way things truly are.

Time pressure comes largely from forces outside our control: from a cutthroat economy; from the loss of the social safety networks that used to help ease the burdens of work and childcare; and from the sexist expectation that women must excel in their careers while assuming most of the responsibilities at home. None of that will be solved with time management. Fully facing the reality of it can only help though. So long as you continue to respond to impossible demands on your time by trying to persuade yourself that you might one day find some way to do the impossible, you're implicitly collaboration with those demands. Whereas once you deeply grasp that they are impossible, you'll stop believing the delusion that any of that is ever going to bring satisfaction and will be newly empowered to resist them, letting you focus instead on building the most meaningful life you can, in whatever situation you're in.

Seeing and accepting our limited powers over our time can prompt us to question the very idea that time is something you use in the first place. There is an alternative: the notion of letting time use you, approaching life not as an opportunity to implement your predetermined plans for success but as a matter of responding to the needs of your place and your moment in history.

Embrace the community constrains

Moreover, most of us seek a specifically individualistic kind of mastery over time. Our culture's ideal is that you alone should control your schedule, doing whatever you prefer, whenever you want, because it's scary to confront the truth that almost everything worth doing depends on cooperating with others, and therefore on exposing yourself to the emotional uncertainties of relationships. In the end the more individual sovereignty you achieve over your time, the lonelier you get. The truth then is that freedom sometimes is to be found not in achieving greater sovereignty over your own schedule but in allowing yourself to be constrained by the rhythms of community. Participating in forms of social life where you don't get to decide exactly what you do or when you doi it. And it leads to the insight that meaningful efficiency often comes not from hurrying things up but from letting them take the time they take.

Live for today not tomorrow

It doesn't matter what you do, we all sense that there are always more important and fulfilling ways we could be spending our time, even if we can't say exactly what they are, yet we systematically spend our days doing other things instead. This feeling can take many forms: the desire to devote yourself to some larger cause, continuously demanding more from yourself, desiring to spend more time with your loved ones.

Our attempts to become more efficient may have the effect of pushing the genuinely important stuff even further over the horizon. Our days are spent trying to "get through" tasks, in order to get them "out of the way", with the result that we live mentally in the future, waiting for when we'll finally get around to what really matters, and worrying in the meantime, that we don't measure up, that we might lack the drive or stamina to keep pace with the speed at which life now seems to move. We live in a constant spirit of joyless urgency.
New: Time is not a resource to be tamed.

You'll see everywhere the concept of time management. I feel it's daring to suggest that you have the power to actually manage time. No you can't as much as you can't tame the sea. Time is not a resource to be spent or be managed, the best we can do is to try to understand its flows and navigate it the best we can.
New: Keep on summing up Oliver Burkeman book.

Efficiency doesn't necessarily give you more time

We're eager to believe the promises of time management frameworks (like GTD) that if you improve your efficiency you'll get more time to enjoy your life. If you follow the right time management system, build the right habits, and apply sufficient self-discipline, you will win the struggle with time.

Reality then kicks in you never win the struggle and only feel more stressed and unhappy. You realize that all the time you've saved is automatically filled up by more things to do in a never ending feedback loop. Time feels like an unstoppable conveyor belt, bringing us new actions as fast as we can dispatch the old ones; and becoming more efficient just seems to cause the belt to speed up. Or else, eventually, to break down. It's true that you get more done, and yet, paradoxically, you only feel busier, more anxious and somehow emptier as a result.

It get's even worse because importance is relative and you may fall into efficiency traps.

Heal yourself from FOMO

Another problem that FOMO brings us is that it leads us to lives where you "truly lived" only if you've lived all the experiences you could live. This leads to a frustrating life as the world has infinite of them, so getting a handful of them under your belt brings you no closer to a sense of having feasted on life's possibilities. You lead yourself in another efficiency trap where the more you experience the more additional wonderful experiences you sarta to feel you could have on top of all those you've already had, with the result that the feeling of existential overwhelm gets worse. To fight this existential overwhelm you can resist the urge to consume more and more experiences and embrace the idea that you're going to miss most of them. You'll then be able to focus on fully enjoying the tiny slice of experiences you actually do have time for.

This FOMO fever is normal given the facts that we're more conscious of the limits of our time (after deterring the after life), the increase of choices that the world has brought us, and the internet amplifier.

You do what you can do

It's usual to feel as though you absolutely must do more than you can do. We live overwhelmed in a constant anxiety of fearing, or knowing for certain, that the actions we want to carry out won't fit on our available time. It looks like this feeling arises on every step of the economic ladder (shown in the works of Daniel Markovits).

The thing is that the idea in itself doesn't make any sense. You can't do more than you can do even if you must. If you truly don't have time for everything you want to do, or feel you ought to do, or that others are badgering you to do, then, well, you don't have time, no matter how grave the consequences of failing to do it all might prove to be. So technically it's irrational to feel troubled by an overwhelming to-do list. You'll do what you can, you won't do what you can't, and the tyrannical inner voice insisting that you must do everything is simply mistaken. We rarely stop to consider things so rationally, though, because that would mean confronting the painful truth of our limitations. We would be forced to acknowledge that there are hard choices to be made: which balls to let drop, which people to disappoint, which ambitions to abandon, which roles to fail at... Instead, in an attempt to avoid these unpleasant truths, we deploy the strategy that dominates most conventional advice on how to deal with busyness: we tell ourselves we'll just have to find a way to do more. So to address our busyness we're making ourselves busier still.

Importance is relative

The problem here is that you'll never be able to make time for everything that feels important. A similar mindset of the section Efficiency doesn't give you more time can be applied. The reason isn't that you haven't yet discovered the right time management tricks, or applied sufficient effort, or that you're generally useless. It's that the underlying assumption is unwarranted: there's no reason to believe you'll make time for everything that matters simply by getting more done. For a start, what "matters" is subjective, so you've no grounds for assuming that there will be time for everything that you, or anyone else deems important. But the other exasperating issue is that if you succeed in fitting more in, you'll find the goalposts start to shift: more things will begin to seem important, meaningful or obligatory. Acquire a reputation for doing your work at amazing speed, and you'll be given more of it. An example of this is gathered in Ruth Schwartz's book More work for mother, where it shows that when washing machines and vacuum cleaners appeared no time was saved at all, because society's standards of cleanliness rose to offset the benefits. What needs doing expands so as to fill the time available for its completion.

Be mindful of the efficiency trap

Sometimes improving your efficiency may lead you to a worse scenario ("efficiency trap") where you won't generally result in the feeling of having "enough time", because, all else being equal, the demands will increase to offset any benefits. Far from getting things done, you'll be creating new things to do. A clear example of this is email management. Every time you reply to an email, there's a good chance of provoking a reply to that email, which itself may require another reply, and so on and so on. At the same time, you'll become known as someone who responds promptly to email, so more people will consider it worth their while to message you to begin with. So it's not simply that you never get though your email; it's that the process of "getting through your email" actually generates more email.

For most of us, most of the time, it isn't feasible to avoid the efficiency trap altogether, but you can stop believing you'll ever solve the challenge of busyness by cramming more in, because that just makes matters worse. And once you stop investing in the idea that you might one day achieve peace of mind that way, it becomes easier to find peace of mind in the present, in the midst of overwhelming demands, because you're no longer making your peace of mind dependent on dealing with all the demands. Once you stop believing that it might somehow be possible to avoid hard choices about time, it gets easier to make better ones.

If you also have the knowledge of the existence of the efficiency traps you may detect them and try to get the benefits without the penalties.

Do the important stuff

The worst aspect of the trap is that it's also a matter of quality. The harder you struggle to fit everything in, the more of your time you'll find yourself spending on the least meaningful things. This is because the more firmly you believe it ought to be possible to find time for everything, the less pressure you'll feel to ask whether any given activity sis the best use of a portion of your time. Each time something new shows up, you'll be strongly biased in favor of accepting it, because you'll assume you needn't sacrifice any other tasks or opportunities in order to make space for it. Soon your life will be automatically filled with not just more things but with more trivial or tedious things.

The important stuff gets postponed because such tasks need your full focus, which means to wait until you have a good chunk of free time and fewer small-but-urgent tasks tugging at your attention. So you spend your energy into clearing the decks, cranking through the smaller stuff to get it out of the way, only to discover that doing so takes the whole day, that the decks are filled up again overnight and that the moment for doing the important stuff never arrives. One can waste years this way, systematically postponing precisely the things one cares the most.

What's needed in these situations is to resist the urges of being on top of everything and learn to live with the anxiety of feeling overwhelmed without automatically responding by trying to fit more in. Instead of clearing the decks, decline to do so, focusing instead on what's truly of greatest consequence while tolerating the discomfort of knowing that, as you do so, the decks will be filling up further, with emails and errands and other to-dos, many of which you may never get around to at all.

You'll sometimes still decide to drive yourself hard in an effort to squeeze more in, when circumstances absolutely require it. But that won't be your default mode, because you'll no longer be operating under the illusion of one day making time for everything.

Evaluate what you miss when you increase your efficiency

Part of the benefits of efficiency is that you free yourself from tedious experiences, the side effect is that some times we're not conscious of being removing experiences that we valued. So even if everything runs more smoothly, smoothness is a dubious virtue, since it's often the unsmoothed textures of life that makes them livable, helping nurture the relationships that are crucial for mental and physical health, and for the resilience of our communities. For example if you buy online the groceries you miss the chance to regularly meet with your neighbours at your local grocery store.

Convenience makes things easy, but without regard to whether easiness is truly what's most valuable in any given context. When you render the process more convenient you drain it of its meaning. The effect of convenience isn't just that the given activity starts to feel less valuable, but that we stop engaging in certain valuable activities altogether, in favour of more convenient ones. Because you can stay home, order food online, and watch sitcoms on a streaming service, you find yourself doing so although you might be perfectly aware that you'd have had a better time if you had met with your friends.

Meanwhile, those aspects of life that resist being made to run more smoothly start to seem repellent. When you can skip the line and buy concert tickets on your phone, waiting in line to vote in an election is irritating. As convenience colonizes everyday life, activities gradually sort themselves into two types: the kind that are now far more convenient, but that feel empty or out of sync with our true preferences; and the kind that now seem intensely annoying because of how inconvenient they remain. Resisting all this is difficult because the Capital is winning this discourse and you'll have more pressure from your environment to stay convenient.

vdirsyncer

New: Troubleshoot Database is locked.

First try to kill all stray vdirsyncer processes, if that doesn't work check for more solutions in this issue
New: Sync to a read-only ics.

```ini [pair calendar_name] a = "calendar_name_local" b = "calendar_name_remote" collections = null conflict_resolution = ["command", "vimdiff"] metadata = ["displayname", "color"]

[storage calendar_name_local] type = "filesystem" path = "~/.calendars/calendar_name" fileext = ".ics"

[storage calendar_name_remote] type = "http" url = "https://example.org/calendar.ics"
New: Automatically sync calendars.

You can use the script shown in the automatically sync emails

Org Mode

New: Start working on a task dates.

SCHEDULED defines when you are plan to start working on that task.

The headline is listed under the given date. In addition, a reminder that the scheduled date has passed is present in the compilation for today, until the entry is marked as done or disabled.

Although is not a good idea (as it promotes the can pushing through the street), if you want to delay the display of this task in the agenda, use SCHEDULED: <2004-12-25 Sat -2d> the task is still scheduled on the 25^th but will appear two days later. In case the task contains a repeater, the delay is considered to affect all occurrences; if you want the delay to only affect the first scheduled occurrence of the task, use --2d instead.

Scheduling an item in Org mode should not be understood in the same way that we understand scheduling a meeting. Setting a date for a meeting is just a simple appointment, you should mark this entry with a simple plain timestamp, to get this item shown on the date where it applies. This is a frequent misunderstanding by Org users. In Org mode, scheduling means setting a date when you want to start working on an action item.

You can set it with <leader>s (Default: <leader>ois)
New: Deadlines.

DEADLINE are like appointments in the sense that it defines when the task is supposed to be finished on. On the deadline date, the task is listed in the agenda. The difference with appointments is that you also see the task in your agenda if it is overdue and you can set a warning about the approaching deadline, starting org_deadline_warning_days before the due date (14 by default). It's useful then to set DEADLINE for those tasks that you don't want to miss that the deadline is over.

An example:
```
* TODO Do this
DEADLINE: <2023-02-24 Fri>
```
You can set it with <leader>d (Default: <leader>oid).

If you need a different warning period for a special task, you can specify it. For example setting a warning period of 5 days DEADLINE: <2004-02-29 Sun -5d>.

If you're as me, you may want to remove the warning feature of DEADLINES to be able to keep your agenda clean. Most of times you are able to finish the task in the day, and for those that you can't specify a SCHEDULED date. To do so set the default number of days to 0.
```
require('orgmode').setup({
  org_deadline_warning_days = 0,
})
```
Using too many tasks with a DEADLINE will clutter your agenda. Use it only for the actions that you need to have a reminder, instead try to using appointment dates instead. The problem of using appointments is that once the date is over you don't get a reminder in the agenda that it's overdue, if you need this, use DEADLINE instead.
New: How to deal with overdue SCHEDULED and DEADLINE tasks.

New: Introduce org-rw.

org-rw is a Python library to process your orgmode files.

Installation:

pip install org-rw

Load an orgmode file:

from org_rw import load

with open('your_file.org', 'r') as f:
    doc = load(f)

New: Install using lazyvim.

return {
  'nvim-orgmode/orgmode',
  ```lua
    {
        'nvim-orgmode/orgmode',
        dependencies = {
      { 'nvim-treesitter/nvim-treesitter', lazy = true },
    },
        event = 'VeryLazy',
        config = function()
    -- Load treesitter grammar for org
    require('orgmode').setup_ts_grammar()

    -- Setup treesitter
    require('nvim-treesitter.configs').setup({
        highlight = {
          enable = true,
          additional_vim_regex_highlighting = { 'org' },
        },
        ensure_installed = { 'org' },
      })

    -- Setup orgmode
    require('orgmode').setup({
        org_agenda_files = '~/orgfiles/**/*',
        org_default_notes_file = '~/orgfiles/refile.org',
      })
  end,
  }
  ```
  dependencies = {
    { 'nvim-treesitter/nvim-treesitter', lazy = true },
  },
  event = 'VeryLazy',
  config = function()
    -- Load treesitter grammar for org
    require('orgmode').setup_ts_grammar()

    -- Setup treesitter
    require('nvim-treesitter.configs').setup({
      highlight = {
        enable = true,
        additional_vim_regex_highlighting = { 'org' },
      },
      ensure_installed = { 'org' },
    })

    -- Setup orgmode
    require('orgmode').setup({
      org_agenda_files = '~/orgfiles/**/*',
      org_default_notes_file = '~/orgfiles/refile.org',
    })
  end,
}

New: Troubleshoot orgmode with dap.

Use the next config and follow the steps of Create an issue in the orgmode repository.

vim.cmd([[set runtimepath=$VIMRUNTIME]])
vim.cmd([[set packpath=/tmp/nvim/site]])

local package_root = '/tmp/nvim/site/pack'
local install_path = package_root .. '/packer/start/packer.nvim'

local function load_plugins()
  require('packer').startup({
    {
      'wbthomason/packer.nvim',
      { 'nvim-treesitter/nvim-treesitter' },
      { 'nvim-lua/plenary.nvim'},
      { 'nvim-orgmode/orgmode'},
      { 'nvim-telescope/telescope.nvim'},
      { 'lyz-code/telescope-orgmode.nvim' },
      { 'jbyuki/one-small-step-for-vimkind' },
      { 'mfussenegger/nvim-dap' },
      { 'kristijanhusak/orgmode.nvim', branch = 'master' },
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. '/plugin/packer_compiled.lua',
    },
  })
end

_G.load_config = function()
  require('orgmode').setup_ts_grammar()
  require('nvim-treesitter.configs').setup({
    highlight = {
      enable = true,
      additional_vim_regex_highlighting = { 'org' },
    },
  })

  vim.cmd([[packadd nvim-treesitter]])
  vim.cmd([[runtime plugin/nvim-treesitter.lua]])
  vim.cmd([[TSUpdateSync org]])

  -- Close packer after install
  if vim.bo.filetype == 'packer' then
    vim.api.nvim_win_close(0, true)
  end

  require('orgmode').setup({
    org_agenda_files = {
      './*'
    }
  }
  )

  -- Reload current file if it's org file to reload tree-sitter
  if vim.bo.filetype == 'org' then
    vim.cmd([[edit!]])
  end
end
if vim.fn.isdirectory(install_path) == 0 then
  vim.fn.system({ 'git', 'clone', 'https://github.com/wbthomason/packer.nvim', install_path })
  load_plugins()
  require('packer').sync()
  vim.cmd([[autocmd User PackerCompileDone ++once lua load_config()]])
else
  load_plugins()
  load_config()
end

require('telescope').setup{
  defaults = {
    preview = {
     enable = true,
      treesitter = false,
    },
    vimgrep_arguments = {
      "ag",
      "--nocolor",
      "--noheading",
      "--numbers",
      "--column",
      "--smart-case",
      "--silent",
      "--follow",
      "--vimgrep",
    },
    file_ignore_patterns = {
      "%.svg",
      "%.spl",
      "%.sug",
      "%.bmp",
      "%.gpg",
      "%.pub",
      "%.kbx",
      "%.db",
      "%.jpg",
      "%.jpeg",
      "%.gif",
      "%.png",
      "%.org_archive",
      "%.flf",
      ".cache",
      ".git/",
      ".thunderbird",
      ".nas",
    },
    mappings = {
      i = {
        -- Required so that folding works when opening a file in telescope
        -- https://github.com/nvim-telescope/telescope.nvim/issues/559
        ["<CR>"] = function()
            vim.cmd [[:stopinsert]]
            vim.cmd [[call feedkeys("\<CR>")]]
        end,
        ['<C-j>'] = 'move_selection_next',
        ['<C-k>'] = 'move_selection_previous',
      }
    }
  },
  pickers = {
    find_files = {
      find_command = { "rg", "--files", "--hidden", "--glob", "!**/.git/*" },
      hidden = true,
      follow = true,
    }
  },
  extensions = {
    fzf = {
      fuzzy = true,                    -- false will only do exact matching
      override_generic_sorter = true,  -- override the generic sorter
      override_file_sorter = true,     -- override the file sorter
      case_mode = "smart_case",        -- or "ignore_case" or "respect_case"
                                       -- the default case_mode is "smart_case"
    },
    heading = {
      treesitter = true,
    },
  }
}

require('telescope').load_extension('orgmode')

local key = vim.keymap
vim.g.mapleader = ' '

local builtin = require('telescope.builtin')
key.set('n', '<leader>f', builtin.find_files, {})
key.set('n', '<leader>F', ':Telescope file_browser<cr>')

vim.api.nvim_create_autocmd('FileType', {
  pattern = 'org',
  group = vim.api.nvim_create_augroup('orgmode_telescope_nvim', { clear = true }),
  callback = function()
    vim.keymap.set('n', '<leader>r', require('telescope').extensions.orgmode.refile_heading)
    vim.keymap.set('n', '<leader>g', require('telescope').extensions.orgmode.search_headings)
  end,
})

require('orgmode').setup_ts_grammar()
local org = require('orgmode').setup({
  org_agenda_files = {
    "./*"
  },
  org_todo_keywords = { 'TODO(t)', 'CHECK(c)', 'DOING(d)', 'RDEACTIVATED(r)', 'WAITING(w)', '|','DONE(e)', 'REJECTED(j)', 'DUPLICATE(u)' },
  org_hide_leading_stars = true,
  org_deadline_warning_days = 0,
  win_split_mode = "horizontal",
  org_priority_highest = 'A',
  org_priority_default = 'C',
  org_priority_lowest = 'D',
  mappings = {
    global = {
      org_agenda = 'ga',
      org_capture = ';c',
    },
    org = {
      -- Enter new items
      org_meta_return = '<c-cr>',
      org_insert_heading_respect_content = ';<cr>',
      org_insert_todo_heading = "<c-t>",
      org_insert_todo_heading_respect_content = ";t",

      -- Heading promoting and demoting
      org_toggle_heading = '<leader>h',
      org_do_promote = '<h',
      org_do_demote = '>h',
      org_promote_subtree = '<H',
      org_demote_subtree = '>H',

      -- Heading moving
      org_move_subtree_up = "<leader>k",
      org_move_subtree_down = "<leader>j",

      -- Heading navigation
      org_next_visible_heading = '<c-j>',
      org_previous_visible_heading = '<c-k>',
      org_forward_heading_same_level = '<c-n>',
      org_backward_heading_same_level = '<c-p>',
      outline_up_heading = 'gp',
      org_open_at_point = 'gx',

      -- State transition
      org_todo = 't',

      -- Priority change
      org_priority_up = '-',
      org_priority_down = '=',

      -- Date management
      org_deadline = '<leader>d',
      org_schedule = '<leader>s',
      org_time_stamp = ';d',
      org_change_date = '<c-e>',

      -- Tag management
      org_set_tags_command = 'T',

      -- Archive management and refiling
      org_archive_subtree = ';a',
      org_toggle_archive_tag = ';A',
      -- org_refile = '<leader>r',  The refile is being handled below
    },
    agenda = {
      org_agenda_later = '<c-n>',
      org_agenda_earlier = '<c-p>',
      org_agenda_switch_to = '<tab>',
      org_agenda_goto = '<cr>',
      org_agenda_priority_up = '=',
      org_agenda_set_tags = 'T',
      org_agenda_deadline = '<leader>d',
      org_agenda_schedule = '<leader>s',
      org_agenda_archive = 'a',
    },
    capture = {
      org_capture_finalize = ';w',
      org_capture_refile = ';r',
      org_capture_kill = 'qqq',
    },
  }
})
local dap = require"dap"
dap.configurations.lua = {
  {
    type = 'nlua',
    request = 'attach',
    name = "Attach to running Neovim instance",
  }
}

dap.adapters.nlua = function(callback, config)
  callback({ type = 'server', host = config.host or "127.0.0.1", port = config.port or 8086 })
end

vim.api.nvim_set_keymap('n', '<leader>b', [[:lua require"dap".toggle_breakpoint()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>c', [[:lua require"dap".continue()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>n', [[:lua require"dap".step_over()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>m', [[:lua require"dap".repl.open()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>N', [[:lua require"dap".step_into()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<F12>', [[:lua require"dap.ui.widgets".hover()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<F5>', [[:lua require"osv".launch({port = 8086})<CR>]], { noremap = true })

New: Hide the state changes in the folds.

The folding of the recurring tasks iterations is also kind of broken. For the next example

** TODO Recurring task
   DEADLINE: <2024-02-08 Thu .+14d -0d>
   :PROPERTIES:
   :LAST_REPEAT: [2024-01-25 Thu 11:53]
   :END:
   - State "DONE" from "TODO" [2024-01-25 Thu 11:53]
   - State "DONE" from "TODO" [2024-01-10 Wed 23:24]
   - State "DONE" from "TODO" [2024-01-03 Wed 19:39]
   - State "DONE" from "TODO" [2023-12-11 Mon 21:30]
   - State "DONE" from "TODO" [2023-11-24 Fri 13:10]

   - [ ] Do X

When folded the State changes is not added to the Properties fold. It's shown something like:

** TODO Recurring task
   DEADLINE: <2024-02-08 Thu .+14d -0d>
   :PROPERTIES:...
   - State "DONE" from "TODO" [2024-01-25 Thu 11:53]
   - State "DONE" from "TODO" [2024-01-10 Wed 23:24]
   - State "DONE" from "TODO" [2024-01-03 Wed 19:39]
   - State "DONE" from "TODO" [2023-12-11 Mon 21:30]
   - State "DONE" from "TODO" [2023-11-24 Fri 13:10]

   - [ ] Do X

I don't know if this is a bug or a feature, but when you have many iterations it's difficult to see the task description. So it would be awesome if they could be included into the properties fold or have their own fold.

I've found though that if you set the org_log_into_drawer = "LOGBOOK" in the config this is fixed.

New: Things that are still broken or not developed.
Correction: Rename Task to Action.

To remove the productivity capitalist load from the concept
New: How to deal with recurring tasks that are not yet ready to be acted upon.

By default when you mark a recurrent task as DONE it will transition the date (either appointment, SCHEDULED or DEADLINE) to the next date and change the state to TODO. I found it confusing because for me TODO actions are the ones that can be acted upon right now. That's why I'm using the next states instead:
- INACTIVE: Recurrent task which date is not yet close so you should not take care of it.
- READY: Recurrent task which date is overdue, we acknowledge the fact and mark the date as inactive (so that it doesn't clobber the agenda).
The idea is that once an INACTIVE task reaches your agenda, either because the warning days of the DEADLINE make it show up, or because it's the SCHEDULED date you need to decide whether to change it to TODO if it's to be acted upon immediately or to READY and deactivate the date.

INACTIVE then should be the default state transition for the recurring tasks once you mark it as DONE. To do this, set in your config:
```
org_todo_repeat_to_state = "INACTIVE",
```
If a project gathers a list of recurrent subprojects or subactions it can have the next states:
- READY: If there is at least one subelement in state READY and the rest are INACTIVE
- TODO: If there is at least one subelement in state TODO and the rest may have READY or INACTIVE
- INACTIVE: The project is not planned to be acted upon soon.
- WAITING: The project is planned to be acted upon but all its subelements are in INACTIVE state.
New: Debug doesn't go up in the jump list.

It's because is a synonym of , and org_cycle is mapped by default as If you're used to use zc then you can disable the org_cycle by setting the mapping org_cycle = "<nop>".
New: Python libraries.

org-rw

org-rw is a library designed to handle Org-mode files, offering the ability to modify data and save it back to the disk.
- Pros:
- Allows modification of data and saving it back to the disk
- Includes tests to ensure functionality
- Cons:
- Documentation is lacking, making it harder to understand and use
- The code structure is complex and difficult to read
- Uses unittest instead of pytest, which some developers may prefer
- Tests are not easy to read
- Last commit was made five months ago, indicating potential inactivity
- Not very popular, with only one contributor, three stars, and no forks
orgparse

orgparse is a more popular library for parsing Org-mode files, with better community support and more contributors. However, it has significant limitations in terms of editing and saving changes.
- Pros:
- More popular with 13 contributors, 43 forks, and 366 stars
- Includes tests to ensure functionality
- Provides some documentation, available here
- Cons:
- Documentation is not very comprehensive
- Cannot write back to Org-mode files, limiting its usefulness for editing content
  - The author suggests using inorganic to convert Org-mode entities to text, with examples available in doctests and the orger library.
  - inorganic is not popular, with one contributor, four forks, 24 stars, and no updates in five years
  - The library is only 200 lines of code
  - The ast is geared towards single-pass document reading. While it is possible to modify the document object tree, writing back changes is more complicated and not a common use case for the author.
Tree-sitter

Tree-sitter is a powerful parser generator tool and incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited.
- Pros:
- General enough to parse any programming language
- Fast enough to parse on every keystroke in a text editor
- Robust enough to provide useful results even in the presence of syntax errors
- Dependency-free, with a runtime library written in pure C
- Supports multiple languages through community-maintained parsers
- Used by Neovim, indicating its reliability and effectiveness
- Provides good documentation, available here
- Python library, py-tree-sitter, simplifies the installation process
- Cons:
- Requires installation of Tree-sitter and the Org-mode language parser separately
- The Python library does not handle the Org-mode language parser directly
To get a better grasp of Tree-sitter you can check their talks:
lazyblorg orgparser.py

lazyblorg orgparser.py is another tool for working with Org-mode files. However, I didn't look at it.
Correction: Tweak area concept.

Model a group of projects that follow the same interest, roles or accountabilities. These are not things to finish but rather to use as criteria for analyzing, defining a specific aspect of your life and to prioritize its projects to reach a higher outcome. We'll use areas to maintain balance and sustainability on our responsibilities as we operate in the world. Areas' titles don't contain verbs as they don't model actions. An example of areas can be health, travels or economy.

To filter the projects by area I set an area tag that propagates downstream. To find the area documents easily I add a section in the index.org of the documentation repository. For example:

New: Change the default org-todo-keywords.

orig = '''* NEW_TODO_STATE First entry

* NEW_DONE_STATE Second entry'''
doc = loads(orig, environment={
  'org-todo-keywords': "NEW_TODO_STATE | NEW_DONE_STATE"
})

Orgzly

New: Migrate from Orgzly to Orgzly Revived.

Gancio

Correction: Change the concept of Task for Action.

To remove the capitalist productive mindset from the concept
Correction: Action cleaning.

Marking steps as done make help you get an idea of the evolution of the action. It can also be useful if you want to do some kind of reporting. On the other hand, having a long list of done steps (specially if you have many levels of step indentation may make the finding of the next actionable step difficult. It's a good idea then to often clean up all done items.
- For non recurring actions use the LOGBOOK to move the done steps. for example:
```
** DOING Do X
   :LOGBOOK:
   - [x] Done step 1
   - [-] Doing step 2
     - [x] Done substep 1
   :END:
   - [-] Doing step 2
     - [ ] substep 2
```
This way the LOGBOOK will be automatically folded so you won't see the progress but it's at hand in case you need it.
- For recurring actions:
- Mark the steps as done
- Archive the todo element.
- Undo the archive.
- Clean up the done items.
This way you have a snapshot of the state of the action in your archive.
New: Project cleaning.

Similar to action cleaning we want to keep the state clean. If there are not that many actions under the project we can leave the done elements as DONE, once they start to get clobbered up we can create a Closed section.

For recurring projects:
- Mark the actions as done
- Archive the project element.
- Undo the archive.
- Clean up the done items.
New: Trimester review.

The objectives of the trimester review are:
- Identify the areas to focus on for the trimester
- Identify the tactics you want to use on those areas.
- Review the previous trimester tactics
The objectives are not:
- To review what you've done or why you didn't get there.
When to do the trimester reviews

As with personal integrity review, it's interesting to do analysis at representative moments. It gives it an emotional weight. You can for example use the solstices or my personal version of the solstices:
- Spring analysis (1^st of March): For me the spring is the real start of the year, it's when life explodes after the stillness of the winter. The sun starts to set later enough so that you have light in the afternoons, the climate gets warmer thus inviting you to be more outside, the nature is blooming new leaves and flowers. It is then a moment to build new projects and set the current year on track.
- Summer analysis (1^st of June): I hate heat, so summer is a moment of retreat. Everyone temporarily stop their lives, we go on holidays and all social projects slow their pace. Even the news have even less interesting things to report. It's so hot outside that some of us seek the cold refuge of home or remote holiday places. Days are long and people love to hang out till late, so usually you wake up later, thus having less time to actually do stuff. Even in the moments when you are alone the heat drains your energy to be productive. It is then a moment to relax and gather forces for the next trimester. It's also perfect to develop easy and chill personal projects that have been forgotten in a drawer. Lower your expectations and just flow with what your body asks you.
- Autumn analysis (1^st of September): September it's another key moment for many people. We have it hardcoded in our life since we were children as it was the start of school. People feel energized after the summer holidays and are eager to get back to their lives and stopped projects. You're already 6 months into the year, so it's a good moment to review your year plan and decide how you want to invest your energy reserves.
- Winter analysis (1^st of December): December is the cue that the year is coming to an end. The days grow shorter and colder, they basically invite you to enjoy a cup of tea under a blanket. It is then a good time to get into your cave and do an introspection analysis on the whole year and prepare the ground for the coming year. Some of the goals of this season are:
- Think everything you need to guarantee a good, solid and powerful spring start.
- Do the year review to adjust your principles.
The year is then divided in two sets of an expansion trimester and a retreat one. We can use this information to adjust our life plan accordingly. In the expansion trimester we could invest more energies in the planning, and in the retreat ones we can do more throughout reviews.

Listen to your desires

The trimester review requires an analysis that doesn't fill in a day session. It requires slow thinking over some time. So I'm creating a task 10 days before the actual review to start thinking about the next trimester. Whether it's ideas, plans, desires, objectives, values, or principles.

Is useful for that document to be available wherever you go, so that in any spare time you can pop it up and continue with the train of thought.

Doing the reflection without seeing your life path prevents you from being tainted by it, thus representing the real you of right now.

On the day to actually do the review, follow the steps of the Month review prepare adjusting them to the trimester case.

Answer some meaningful guided questions

To be done, until then you can read chapters 13, 14 and the epilogue of the book Four thousand weeks by Oliver Burkman.

Refactor your gathered thoughts

If you've followed the prepare steps, you've already been making up your mind on what do you want the next trimester to look like. Now it's the time to refine those thoughts.

In your roadmap document add a new section for the incoming trimester similar to:
```
* Roadmap
** 2024
*** Summer 2024
**** Essential intent
**** Trimester analysis
**** Trimester objectives
***** TODO Objective 1
****** TODO SubObjective 1
```
Go one by one (don't peek!) of your gathered items and translate them in the next sections:
- Trimester analysis: A text with as many paragraphs as you need to order your thoughts
- Trimester objectives: These can be concrete emotional projects you want to carry through.
- Essential intent: This is the main headline of your trimester, probably you won't be able to define it until the last parts of the review process. It should be concrete and emotional too, it's going to be the idea that gives you strength on your weak moments and your guide to decide which projects to do and which not to.
Don't be too concerned on the format of the content of the objectives, this is the first draft, and we'll refine it through the planning.
New: Wordpress plugin.

This plugin allows you to embed a list of events or a single event from your Gancio website using a shortcode. It also allows you to connects a Gancio instance to a your wordpress website to automatically push events published on WordPress: for this to work an event manager plugin is required, Event Organiser and The Events Calendar are supported. Adding another plugin it’s an easy task and you have a guide available in the repo that shows you how to do it.

The source code of the plugin is in the wp-plugin directory of the official repo

Habit management

New: Introduce habit management.

A habit is a routine of behavior that is repeated regularly and tends to occur subconsciously.

A 2002 daily experience study found that approximately 43% of daily behaviors are performed out of habit. New behaviours can become automatic through the process of habit formation. Old habits are hard to break and new habits are hard to form because the behavioural patterns that humans repeat become imprinted in neural pathways, but it is possible to form new habits through repetition.

When behaviors are repeated in a consistent context, there is an incremental increase in the link between the context and the action. This increases the automaticity of the behavior in that context. Features of an automatic behavior are all or some of: efficiency, lack of awareness, unintentionality, and uncontrollability.

Mastering habit formation can be a powerful tool to change yourself. Usually with small changes you get massive outcomes in the long run. The downside is that it's not for the impatient people as it often appears to make no difference until you cross a critical threshold that unlocks a new level of performance.
New: Why are habits interesting.

Whenever you face a problem repeatedly, your brain begins to automate the process of solving it. Habits are a series of automatic resolutions that solve the problems and stresses you face regularly.

As habits are created, the level of activity in the brain decreases. You learn to lock in on the cues that predict success and tune out everything else. When a similar situation arises in the future, you know exactly what you look for. There is no longer a need to analyze every angle of a situation. Your brain skips the process of trial and error and creates a mental rule: if this, then that.

Habit formation is incredibly useful because the conscious mind is the bottleneck of the brain. It can only pay attention to one problem at a time. Habits reduce the cognitive load and free up mental capacity, so they can be carried on with your nonconscious mind and you can allocate your attention to other tasks.
New: Identity focused changes.

Changing our habits is challenging because we try to change the wrong thing in the wrong way.

There are three levels at which change can occur:
- Outcomes: Changing your results. Goals fall under this category: publishing a book, run daily
- Process: Changing your habits and systems: decluttering your desk for a better workflow, developing a meditation practice.
- Identity: Changing your beliefs, assumptions and biases: your world view, your self-image, your judgments.
Many people begin the process of changing their habits by focusing on what they want to achieve. This leads to outcome-based habits. The alternative is to build identity-based habits. With this approach, we start by focusing on who we wish to become.

The first path of change is doomed because maintaining behaviours that are incongruent with the self is expensive and will not last. Even if they make rational sense. Thus it's hard to change your habits if you never change the underlying beliefs that led to your past behaviour. On the other hand it's easy to find motivation once a habit has changed your identity as you may be proud of it and will be willing to maintain all the habits and systems associated with it. For example: The goal is not to read a book, but to become a reader.

Focusing on outcomes may also bring the next problems:
- Focusing on the results may lead you to temporal solutions. If you focus on the source of the issue at hand you may solve it with less effort and get you to a more stable one.
- Goals create an "either-or" conflict: either you achieve your goal and are successful or you fail and you are disappointed. Thus you only get a positive reward if you fulfill a goal. If you instead focus on the process rather than the result, you will be satisfied anytime your system is running.
- When your hard work is focused on a goal you may feel depleted once you meet it and that could make you loose the condition that made you meet the goal in the first place.
Research has shown that once a person believes in a particular aspect of their identity, they are more likely to act in alignment with that belief. This of course is a double-edged sword. Identity change can be a powerful force for self-improvement. When working against you, identity change can be a curse.
New: Changing your identity.

Whatever your identity is right now, you only believe it because you have proof of it. The more evidence you have for a belief, the more strongly you will believe it.

Your habits and systems are how you embody your identity. When you make your bed each day, you embody the identity of an organized person. The more you repeat a behaviour, the more you reinforce the identity associated with that behaviour. To the point that your self-image begins to change. The effect of one-off experiences tends to fade away while the effect of habits gets reinforced with time, which means your habits contribute most of the evidence that shapes your identity.

Every action you take is a vote for the type of person you wish to become. This is one reason why meaningful change does not require radical change. Small habits can make a meaningful difference by providing evidence of a new identity.

Once you start the ball rolling things become easier as building habits is a feedback loop. Your habits shape your identity, and your identity shapes your habits.

The most practical way to change the identity is to:
- Decide the type of person you want to be
- Prove it to yourself with small wins
Another advantage of focusing in what type of person you want to be is that maybe the outcome you wanted to focus on is not the wisest smallest step to achieve your identity change. Thinking on the identity you want to embrace can make you think outside the box.
New: Decide the type of person you want to be.

One way to decide the person you want to be is to answer big questions like: what do you want to stand for? What are your principles and values? Who do you wish to become?

As we're more result oriented, another way is to work backwards from them to the person you want to be. Ask yourself: Who is the type of person that could get the outcome I want?
New: How to change a habit.

The process of building a habit from a behaviour can be divided into four stages:
- Reward is the end goal.
- Cue is the trigger in your brain that initiate a behaviour. It's contains the information that predicts a reward.
- Cravings are the motivational force fueled by the desire of the reward. Without motivation we have no reason to act.
- Response is the thought or action you perform to obtain the reward. The response depends on the amount of motivation you have, how much friction is associated with the behaviour and your ability to actually do it.
If a behaviour is insufficient in any of the four stages, it will not become a habit. Eliminate the cue and your habit will never start. Reduce the craving and you won't have enough motivation to act. Make the behaviour difficult and you won't be able to do it. And if the reward fails to satisfy your desire, then you'll have no reason to do it again in the future.

We chase rewards because they:
- Deliver contentment.
- Satisfy your craving.
- Teach us which actions are worth remembering in the future.
If a reward is met then it becomes associated with the cue, thus closing the habit feedback loop.

If we keep these stages in mind then:
- To build good habits we need to:
  - Cue: Make it obvious
  - Craving: Make it attractive
  - Response: Make it easy
  - Reward: Make it satisfying
- To break bad habits we need to:
  - Cue: Make it invisible
  - Craving: Make it unattractive
  - Response: Make it difficult
  - Reward: Make it unsatisfying
New: Select which habits you want to work with.

Our responses to the cues are so deeply encoded that it may feel like the urge to act comes from nowhere. For this reason, we must begin the process of behavior change with awareness. Before we can effectively build new habits, we need to get a handle on our current ones. The author suggests to do a list of your daily habits and rate them positively, negatively or neutral under the judgement of whether it brings you closer to the desired person you want to be.

I find this approach expensive time-wise if you already have a huge list of habits to work with. As it's my case I'll skip this part. You can read it in more detail in the chapter "4: The Man Who Didn't Look Right".
New: Working with the habit cues.

The first place to start the habit design is to understand and tweak the triggers that produce them. We'll do it by:
New: Clearly formulate the habit you want to change.

The cues that can trigger an habit can come in a wide range of forms but the two most common are time and location. Being specific about what you want and how you will achieve it helps you say no to things that derail progress, distract your attention and pull you off course. And with enough repetition, you will get the urge to do the right thing at the right time, even if you can't say why. That's why it's interesting to formulate your habits as "I will [behaviour] at [time] in [location]".

You want the cue to be highly specific and immediately actionable. If there is room for doubt the implementation will suffer. Continuously refine the habit definitions as you catch the exceptions that drift you off.

If you aren't sure of when to start your habit, try the first day of the week, month or year. People are more likely to take action at those times because hope is usually higher as you get the feeling of a fresh start.
New: Habit stacking.

Many behaviours are linked together where the action of the first is the cue that triggers the next one. You can use this connection to build new habits based on your established ones. This may be called habit stacking. The formulation in this case is "After [current habit], I will [new habit]".

The key is to tie your desired behaviour into something you already do each day. Once you have mastered this basic structure, you can begin to create larger stacks by chaining small habits together. The catch is that the new habit should have the same frequency as the established one.

One way to find the right trigger for your habit stack is by brainstorming over:
- The list of your current habits.
- A new list of things that always happen to you with that frequency.
With these two lists, you can begin searching for the best triggers for the stack.
New: Use the environment to tweak your cues.

The cues that trigger a habit can start out very specific, but over time your habits become associated not with a single trigger but with the entire context surrounding the behaviour. This stacks over itself and your habits change depending on the room you are in and the cues in front of you. The context or the environment is then the invisible hand that shapes behaviours. They are not defined by the objects in the environment but by our relationship to them.

A new environment is a good foundation to make new habits, as you are free from the subtle triggers that nudge you toward your current habits. When you can't manage to get an entirely new environment, you can redefine or rearrange your current one.

When building good habits you can rearrange the environment to create obvious visual cues that draw your attention towards the desired habit. By sprinkling triggers throughout your surroundings, you increase the odds that you'll think about your habit throughout the day.

Once a habit has been encoded, the urge to act follows whenever the environmental cues reappear. This is why bad habits reinforce themselves. As you carry through the behaviour you spiral into a situation where the craving keeps growing and points you to keep on going with the same response. For example watching TV makes you feel sluggish, so you watch more television because you don't have the energy to do anything else.

Even if you manage to break a habit, you are unlikely to forget it's cues even if you don't do it for a while. That means that simply resisting temptation is an ineffective strategy. In the short run it may work. In the long run, as self-control is an exhausting task that consumes willpower, we become a product of the environment we live in. Trying to change a habit with self-control is doomed to fail as you may be able to resist temptation once or twice, but it's unlikely you can muster the willpower to override your desires every time. It's also very hard and frustrating to try to achieve change when you're under the mood influences of a bad habit.

A more reliable approach is to cut bad habits off at the source. Tweak the environment to make the cue virtually impossible to happen. That way you won't even have the chance to fall for the craving.
New: Temptation bundling.

Dopamine is a neurotransmitter that can be used as the scientific measurement of craving. For years we assumed that it was all about pleasure, but now we know it plays a central role in many neurological processes, including motivation, learning and memory, punishment and aversion and voluntary movement.

Habits are a dopamine-driven feed back loop. It is released not only when you receive a reward but also when you anticipate it. This anticipation, and not the fulfillment of it, is what gets us to take action.

If we make a habit more attractive it will release more dopamine which will gives us more motivation to carry it through.

Temptation bundling works by pairing an action you want to do with an action you need to do. You're more likely to find a behaviour attractive if you get to do one of your favourite things at the same time. In the end you may even look forward to do the habit you need as it's related to the habit you want.
New: Align your personal identity change with an existent shared identity.

We pick up habits from the people around us. As a general rule, the closer we are to someone, the more likely we are to imitate some of their habits. One of the most effective things you can do to build better habits is to join a culture where your desired behaviour is the normal one. This transforms your personal identity transformation into the building of a shared one. Shared identities have great benefits over single ones:
- They foster belonging. A powerful feeling that creates motivation.
- They are more resilient: When one falters others will take their place so all together you'll guarantee the maintenance of the identity.
- They create friendship and community
- They expose you to an environment where more habits tied to that identity thrive.
Likewise, if you're trying to run from a bad habit cut your ties to communities that embrace that habit.
New: Track your habit management.

You can have a habits.org file where you prioritize, analyze, track them.

I'm using the next headings:
- Habits being implemented: It's subdivided in two:
- Habits that need attention
- Habits that don't need attention
- Unclassified habits: Useful when refiling habits from your inbox. This list will be analyzedwhen you do habit analysis.
- Backlog of habits: Unrefined and unordered list of habits
- Implemented habits:
- Rejected habits:
Each habit is a TODO item with the usual states: TODO, DOING, DONE, REJECTED. In it's body I keep a log of the evolution and the analysis of the habit.
New: Habit management workflow.

Each month I'm trying to go through the list of habits to:
- Update the state of the habits: Some will be done, rejected or to register ideas about them.
- Decide which ones need attention.
- Do habit analysis on the ones that need attention.
For each of the habits that need analysis, apply the learnings of the next sections:

Calendar management

New: Add calendar event notification system tool.

Set up a system that notifies you when the next calendar event is about to start to avoid spending mental load on it and to reduce the possibilities of missing the event.

I've created a small tool that:
- Tells me the number of pomodoros that I have until the next event.
- Once a pomodoro finishes it makes me focus on the amount left so that I can prepare for the event
- Catches my attention when the event is starting.

Life chores management

himalaya

New: Doing the inventory review.

I haven't found a way to make the grocy inventory match the reality because for me it's hard to register when I consume a product. Even more if other people also use them. Therefore I use grocy only to know what to buy without thinking about it. For that use case the inventory needs to meet reality only before doing the groceries. I usually do a big shopping of non-perishable goods at the supermarket once each two or three months, and a weekly shopping of the rest.

Tracking the goods that are bought each week makes no sense as those are things that are clearly seen and are very variable depending on the season. Once I've automated the ingestion and consumption of products it will, but so far it would mean investing more time than the benefit it gives.

This doesn't apply to the big shopping, as this one is done infrequently, so we need a better planning.

To do the inventory review I use a tablet and the android app.
- Open the stock overview and iterate through the locations to:
- Make sure that the number of products match the reality
  - Iterate over the list of products checking the quantity
  - Look at the location to see if there are missing products in the inventory
- Adjust the product properties (default location, minimum amount)
- Check the resulting shopping list and adjust the minimum values.
- Check the list of missing products to adjust the minimum values. I have a notepad in the fridge where I write the things I miss.
New: Introduce route management.

To analyze which hiking routes are available in a zone I'm following the next process
- Check in my trips orgmode directory to see if the zone has already been indexed.
- Do a first search of routes
- Identify which books or magazines describe the zone
- For each of the described routes in each of these books:
  - Create the Routes section with tag :route: if it doesn't exist
  - Fill up the route form in a TODO heading. Something similar to:
```
Reference: Book Page
Source: Where does it start
Distance: X km
Slope: X m
Type: [Lineal/Circular/Semi-lineal]
Difficulty:
Track: URL (only if you don't have to search for it)
```
  - Add tags of the people I'd like to do it with
  - Put a postit on the book/magazine if it's likely I'm going to do it
  - Open a web maps tab with the source of the route to calculate the time from the different lodgins
- If there are not enough, repeat the process above for each of your online route reference blogs
- Choose the routes to do
- Show the gathered routes to the people you want to go with
- Select which ones you'll be more likely to do
- For each of the chosen routes
- Search the track in wikiloc if it's missing
- Import the track in OsmAnd+
New: Add API and python library docs.

There is no active python library, although it existed pygrocy
- API Docs

New: Introduce himalaya.

himalaya is a Rust CLI to manage emails.

Features:

Multi-accounting
Interactive configuration via wizard (requires wizard feature)
Mailbox, envelope, message and flag management
Message composition based on $EDITOR
IMAP backend (requires imap feature)
Maildir backend (requires maildir feature)
Notmuch backend (requires notmuch feature)
SMTP backend (requires smtp feature)
Sendmail backend (requires sendmail feature)
Global system keyring for managing secrets (requires keyring feature)
OAuth 2.0 authorization (requires oauth2 feature)
JSON output via --output json
PGP encryption:
via shell commands (requires pgp-commands feature)
via GPG bindings (requires pgp-gpg feature)
via native implementation (requires pgp-native feature)

Cons:

Documentation is inexistent, you have to dive into the --help to understand stuff.

Installation

The v1.0.0 is currently being tested on the master branch, and is the prefered version to use. Previous versions (including GitHub beta releases and repositories published versions) are not recommended.

Himalaya CLI v1.0.0 can be installed with a pre-built binary. Find the latest pre-release GitHub workflow and look for the Artifacts section. You should find a pre-built binary matching your OS.

Himalaya CLI v1.0.0 can also be installed with cargo:

$ cargo install --git https://github.com/pimalaya/himalaya.git --force himalaya

Configuration

Just run himalaya, the wizard will help you to configure your default account.

You can also manually edit your own configuration, from scratch:

Copy the content of the documented ./config.sample.toml
Paste it in a new file ~/.config/himalaya/config.toml
Edit, then comment or uncomment the options you want

If using mbrsync

My generic configuration for an mbrsync account is:

[accounts.account_name]

email = "lyz@example.org"
display-name = "lyz"
envelope.list.table.unseen-char = "u"
envelope.list.table.replied-char = "r"
backend.type = "maildir"
backend.root-dir = "/home/lyz/.local/share/mail/lyz-example"
backend.maildirpp = false
message.send.backend.type = "smtp"
message.send.backend.host = "example.org"
message.send.backend.port = 587
message.send.backend.encryption = "start-tls"
message.send.backend.login = "lyz"
message.send.backend.auth.type = "password"
message.send.backend.auth.command = "pass show mail/lyz.example"

Once you've set it then you need to fix the INBOX directory.

Then you can check if it works by running himalaya envelopes list -a lyz-example

Vim plugin installation

Using lazy:

return {
  {
    "pimalaya/himalaya-vim",
  },
}

You can then run :Himalaya account_name and it will open himalaya in your editor.

Configure the account bindings

To avoid typing :Himalaya account_name each time you want to check the email you can set some bindings:

return {
  {
    "pimalaya/himalaya-vim",
    keys = {
      { "<leader>ma", "<cmd>Himalaya account_name<cr>", desc = "Open account_name@example.org" },
      { "<leader>ml", "<cmd>Himalaya lyz<cr>", desc = "Open lyz@example.org" },
    },
  },
}

Setting the description is useful to see the configured accounts with which-key by typing <leader>m and waiting.

Configure extra bindings

The default plugin doesn't yet have all the bindings I'd like so I've added the next ones:

In the list of emails view:
dd in normal mode or d in visual: Delete emails
q: exit the program
In the email view:
d: Delete email
q: Return to the list of emails view

If you want them too set the next config:

return {
  {
    "pimalaya/himalaya-vim",
    config = function()
      vim.api.nvim_create_augroup("HimalayaCustomBindings", { clear = true })
      vim.api.nvim_create_autocmd("FileType", {
        group = "HimalayaCustomBindings",
        pattern = "himalaya-email-listing",
        callback = function()
          -- Bindings to delete emails
          vim.api.nvim_buf_set_keymap(0, "n", "dd", "<plug>(himalaya-email-delete)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "x", "d", "<plug>(himalaya-email-delete)", { noremap = true, silent = true })
          -- Bind `q` to close the window
          vim.api.nvim_buf_set_keymap(0, "n", "q", ":bd<CR>", { noremap = true, silent = true })
        end,
      })

      vim.api.nvim_create_augroup("HimalayaEmailCustomBindings", { clear = true })
      vim.api.nvim_create_autocmd("FileType", {
        group = "HimalayaEmailCustomBindings",
        pattern = "mail",
        callback = function()
          -- Bind `q` to close the window
          vim.api.nvim_buf_set_keymap(0, "n", "q", ":q<CR>", { noremap = true, silent = true })
          -- Bind `d` to delete the email and close the window
          vim.api.nvim_buf_set_keymap(
            0,
            "n",
            "d",
            "<plug>(himalaya-email-delete):q<CR>",
            { noremap = true, silent = true }
          )
        end,
      })
    end,
  },
}

Configure email fetching from within vim

Fetching emails from within vim is not yet supported, so I'm manually refreshing by account:

return {
  {
    "pimalaya/himalaya-vim",
    keys = {
      -- Email refreshing bindings
      { "<leader>rj", ':lua FetchEmails("lyz")<CR>', desc = "Fetch lyz@example.org" },
    },
    config = function()
      function FetchEmails(account)
        vim.notify("Fetching emails for " .. account .. ", please wait...", vim.log.levels.INFO)
        vim.cmd("redraw")
        vim.fn.jobstart("mbsync " .. account, {
          on_exit = function(_, exit_code, _)
            if exit_code == 0 then
              vim.notify("Emails for " .. account .. " fetched successfully!", vim.log.levels.INFO)
            else
              vim.notify("Failed to fetch emails for " .. account .. ". Check the logs.", vim.log.levels.ERROR)
            end
          end,
        })
      end
    end,
  },
}

You still need to open again :Himalaya account_name as the plugin does not reload if there are new emails.

Show notifications when emails arrive

You can set up mirador to get those notifications.

Not there yet

Troubleshooting

Cannot find maildir matching name INBOX

mbrsync uses Inbox instead of the default INBOX so it doesn't find it. In theory you can use folder.alias.inbox = "Inbox" but it didn't work with me, so I finally ended up doing a symbolic link from INBOX to Inbox.

Cannot find maildir matching name Trash

That's because the Trash directory does not follow the Maildir structure. I had to create the cur tmp and new directories.

References - Source - Vim plugin source

New: Introduce mailbox.

mailbox is a python library to work with MailDir and mbox local mailboxes.

It's part of the core python libraries, so you don't need to install anything.

Usage

The docs are not very pleasant to read, so I got most of the usage knowledge from these sources:
One thing to keep in mind is that an account can have many mailboxes (INBOX, Sent, ...), there is no "root mailbox" that contains all of the other

initialise a mailbox
```
mbox = mailbox.Maildir('path/to/your/mailbox')
```
Where the path/to/your/mailbox is the directory that contains the cur, new, and tmp directories.

Working with mailboxes

It's not very clear how to work with them, the Maildir mailbox contains the emails in iterators [m for m in mbox], it acts kind of a dictionary, you can get the keys of the emails with [k for k in mbox.iterkeys], and then you can mbox[key] to get an email, you cannot modify those emails (flags, subdir, ...) directly in the mbox object (for example mbox[key].set_flags('P') doesn't work). You need to mail = mbox.pop(key), do the changes in the mail object and then mbox.add(mail) it again, with the downside that after you added it again, the key has changed! But it's the return value of the add method.

If the program gets interrupted between the pop and the add then you'll loose the email. The best way to work with it would be then:
- mail = mbox.get(key) the email
- Do all the process you need to do with the email
- mbox.pop(key) and key = mbox.add(mail)
In theory mbox has an update method that does this, but I don't understand it and it doesn't work as expected :S.

Moving emails around

You can't just move the files between directories like you'd do with python as each directory contains it's own identifiers.

Moving a message between the maildir directories

The Message has a set_subdir

Creating folders

Even though you can create folders with mailbox it creates them in a way that mbsync doesn't understand it. It's easier to manually create the cur, tmp, and new directories. I'm using the next function:
```
if not (mailbox_dir / "cur").exists():
    for dir in ["cur", "tmp", "new"]:
        (mailbox_dir / dir).mkdir(parents=True)
    log.info(f"Initialized mailbox: {mailbox}")
else:
    log.debug(f"{mailbox} already exists")
```
References
- Reference Docs
- Non official useful docs
New: Introduce maildir.

The Maildir e-mail format is a common way of storing email messages on a file system, rather than in a database. Each message is assigned a file with a unique name, and each mail folder is a file system directory containing these files.

A Maildir directory (often named Maildir) usually has three subdirectories named tmp, new, and cur.
- The tmp subdirectory temporarily stores e-mail messages that are in the process of being delivered. This subdirectory may also store other kinds of temporary files.
- The new subdirectory stores messages that have been delivered, but have not yet been seen by any mail application.
- The cur subdirectory stores messages that have already been seen by mail applications.
References
- Wikipedia
New: My emails are not being deleted on the source IMAP server.

That's the default behavior of mbsync, if you want it to actually delete the emails on the source you need to add:
```
Expunge Both
```
Under your channel (close to Sync All, Create Both)
New: Mbsync error: UID is beyond highest assigned UID.

If during the sync you receive the following errors:
```
mbsync error: UID is 3 beyond highest assigned UID 1
```
Go to the place where mbsync is storing the emails and find the file that is giving the error, you need to find the files that contain U=3, imagine that it's something like 1568901502.26338_1.hostname,U=3:2,S. You can strip off everything from the ,U= from that filename and resync and it should be fine, e.g.
```
mv '1568901502.26338_1.hostname,U=3:2,S' '1568901502.26338_1.hostname'
```
feat(mirador): introduce mirador

DEPRECATED: as of 2024-11-15 the tool has many errors (1, 2), few stars (4) and few commits (8). use watchdog instead and build your own solution.

mirador is a CLI to watch mailbox changes made by the maintaner of himalaya.

Features:
- Watches and executes actions on mailbox changes
- Interactive configuration via wizard (requires wizard feature)
- Supported events: on message added.
- Supported actions: send system notification, execute shell command.
- Supports IMAP mailboxes (requires imap feature)
- Supports Maildir folders (requires maildir feature)
- Supports global system keyring to manage secrets (requires keyring feature)
- Supports OAuth 2.0 (requires oauth2 feature)
Mirador CLI is written in Rust, and relies on cargo features to enable or disable functionalities. Default features can be found in the features section of the Cargo.toml.

Installation

The v1.0.0 is currently being tested on the master branch, and is the preferred version to use. Previous versions (including GitHub beta releases and repositories published versions) are not recommended.*

Cargo (git)

Mirador CLI v1.0.0 can also be installed with cargo:
```
$ cargo install --frozen --force --git https://github.com/pimalaya/mirador.git
```
Pre-built binary

Mirador CLI v1.0.0 can be installed with a pre-built binary. Find the latest pre-release GitHub workflow and look for the Artifacts section. You should find a pre-built binary matching your OS.

Configuration

Just run mirador, the wizard will help you to configure your default account.

You can also manually edit your own configuration, from scratch:
- Copy the content of the documented ./config.sample.toml
- Paste it in a new file ~/.config/mirador/config.toml
- Edit, then comment or uncomment the options you want
- Source

New: Configure navigation bindings.

The default bindings conflict with my git bindings, and to make them similar to orgmode agenda I'm changing the next and previous page:

return {
  {
    "pimalaya/himalaya-vim",
    keys = {
      { "b", "<plug>(himalaya-folder-select-previous-page)", desc = "Go to the previous email page" },
      { "f", "<plug>(himalaya-folder-select-next-page)", desc = "Go to the next email page" },
    },
  },
}

Correction: Configure the account bindings.
Correction: Tweak the bindings.

Move forward and backwards in the history of emails:
```
    vim.api.nvim_create_autocmd("FileType", {
      group = "HimalayaCustomBindings",
      pattern = "himalaya-email-listing",
      callback = function()
        vim.api.nvim_buf_set_keymap(0, "n", "b", "<plug>(himalaya-folder-select-previous-page)", { noremap = true, silent = true })
        vim.api.nvim_buf_set_keymap(0, "n", "f", "<plug>(himalaya-folder-select-next-page)", { noremap = true, silent = true })
      end,
    })
```
Better bindings for the email list view:
```
          -- Refresh emails
          vim.api.nvim_buf_set_keymap(0, "n", "r", ":lua FetchEmails()<CR>", { noremap = true, silent = true })
          -- Email list view bindings
          vim.api.nvim_buf_set_keymap(0, "n", "b", "<plug>(himalaya-folder-select-previous-page)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "f", "<plug>(himalaya-folder-select-next-page)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "R", "<plug>(himalaya-email-reply-all)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "F", "<plug>(himalaya-email-forward)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "m", "<plug>(himalaya-folder-select)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "M", "<plug>(himalaya-email-move)", { noremap = true, silent = true })
```
feat(himalaya#Searching emails): Searching emails

You can use the g/ binding from within nvim to search for emails. The query syntax supports filtering and sorting query.

I've tried changing it to / without success :'(

Filters

A filter query is composed of operators and conditions. There is 3 operators and 8 conditions:
- not <condition>: filter envelopes that do not match the condition
- <condition> and <condition>: filter envelopes that match both conditions
- <condition> or <condition>: filter envelopes that match one of the conditions
- date <yyyy-mm-dd>: filter envelopes that match the given date
- before <yyyy-mm-dd>: filter envelopes with date strictly before the given one
- after <yyyy-mm-dd>: filter envelopes with date stricly after the given one
- from <pattern>: filter envelopes with senders matching the given pattern
- to <pattern>: filter envelopes with recipients matching the given pattern
- subject <pattern>: filter envelopes with subject matching the given pattern
- body <pattern>: filter envelopes with text bodies matching the given pattern
- flag <flag>: filter envelopes matching the given flag
Sorting A sort query starts by "order by", and is composed of kinds and orders. There is 4 kinds and 2 orders:
- date [order]: sort envelopes by date
- from [order]: sort envelopes by sender
- to [order]: sort envelopes by recipient
- subject [order]: sort envelopes by subject
- <kind> asc: sort envelopes by the given kind in ascending order
- <kind> desc: sort envelopes by the given kind in descending order
Examples

subject foo and body bar: filter envelopes containing "foo" in their subject and "bar" in their text bodies order by date desc subject: sort envelopes by descending date (most recent first), then by ascending subject subject foo and body bar order by date desc subject: combination of the 2 previous examples
New: List more detected issues.
- Replying an email doesn't mark it as replied
New: Troubleshoot cannot install the program.

Sometimes the installation steps fail as it's still not in stable. A workaround is to download the binary created by the pre-release CI. You can do it by:
- Click on the latest job
- Click on jobs
- Click on the job of your architecture
- Click on "Upload release"
- Search for "Artifact download URL" and download the file
- Unpack it and add it somewhere in your $PATH

alot

New: Comments.

Any text on a line after the character ; is ignored, text like this:

; I paid and left the taxi, forgot to take change, it was cold.
2015-01-01 * "Taxi home from concert in Brooklyn"
  Assets:Cash      -20 USD  ; inline comment
  Expenses:Taxi

New: Introduce matrix_highlight.

Matrix Highlight is a decentralized and federated way of annotating the web based on Matrix.

Think of it as an open source alternative to hypothesis.

It's similar to Populus but for the web.

I want to try it and investigate further specially if you can:
- Easily extract the annotations
- Activate it by default everywhere
New: Get the schema of a table.

Postgres:
```
\d+ table_name
```

New: Get the last row of a table.

SELECT * FROM Table ORDER BY ID DESC LIMIT 1

New: Introduce beanSQL.

bean-sql is a language to query beancount data.

References: - Docs - Examples

New: Get the quarter of a date.

Use the quarter(date) selector in the SELECT . For example:

SELECT quarter(date) as quarter, sum(position) AS value
WHERE
    account ~ '^Expenses:' OR
    account ~ '^Income:'
GROUP BY quarter

It will return the quarter in the format YYYY-QX.

New: Building your own dashboards.

I was wondering whether to create fava dashboards or to create them directly in grafana.

Pros of fava dashboards: - They are integrated in fava so it would be easy to browse other beancount data. Although this could be done as well in another window if I used grafana. - There is no need to create the beancount grafana data source logic. - It's already a working project, I would need just to tweak an existent example.

Cons: - I may need to learn echarts and write JavaScript to tweak some of the dashboards. - I wouldn't have all my dashboards in the same place. - It only solves part of the problem, I'd still need to write the bean-sql queries. But using beanql is probably the best way to extract data from beancount anyway. - It involves more magic than using grafana. - grafana dashboards are prettier. - I wouldn't use the grafana knowledge. - I'd learn a new tool only to use it here instead of taking the chance to improve my grafana skillset.

I'm going to try with fava dashboards and see how it goes
Correction: Deprecate in favour of himalaya.

DEPRECATED: Use himalaya instead.

New: Automatically sync emails.

I have many emails, and I want to fetch them with different frequencies, in the background and be notified if anything goes wrong.

For that purpose I've created a python script, a systemd service and some loki rules to monitor it.

Script to sync emails and calendars with different frequencies

The script iterates over the configured accounts in accounts_config and runs mbsync for email accounts and vdirsyncer for email accounts based on some cron expressions. It logs the output in logfmt format so that it's easily handled by loki

To run it you'll first need to create a virtualenv, I use mkvirtualenv account_syncer which creates a virtualenv in ~/.local/share/virtualenv/account_syncer.

Then install the dependencies:

pip install aiocron

Then place this script somewhere, for example (~/.local/bin/account_syncer.py)

import asyncio
import logging
from datetime import datetime
import asyncio.subprocess
import aiocron

accounts_config = {
    "emails": [
      {
          "account_name": "lyz",
          "cron_expressions": ["*/15 9-23 * * *"],
      },
      {
          "account_name": "work",
          "cron_expressions": ["*/60 8-17 * * 1-5"],  # Monday-Friday
      },
      {
          "account_name": "monitorization",
          "cron_expressions": ["*/5 * * * *"],
      },
    ],
    "calendars": [
      {
          "account_name": "lyz",
          "cron_expressions": ["*/15 9-23 * * *"],
      },
      {
          "account_name": "work",
          "cron_expressions": ["*/60 8-17 * * 1-5"],  # Monday-Friday
      },
    ],
}

class LogfmtFormatter(logging.Formatter):
    """Custom formatter to output logs in logfmt style."""

    def format(self, record: logging.LogRecord) -> str:
        log_message = (
            f"level={record.levelname.lower()} "
            f"logger={record.name} "
            f'msg="{record.getMessage()}"'
        )
        return log_message

def setup_logging(logging_name: str) -> logging.Logger:
    """Configure logging to use logfmt format.
    Args:
        logging_name (str): The logger's name and identifier in the systemd journal.
    Returns:
        Logger: The configured logger.
    """
    console_handler = logging.StreamHandler()
    logfmt_formatter = LogfmtFormatter()
    console_handler.setFormatter(logfmt_formatter)
    logger = logging.getLogger(logging_name)
    logger.setLevel(logging.INFO)
    logger.addHandler(console_handler)
    return logger

log = setup_logging("account_syncer")

async def run_mbsync(account_name: str) -> None:
    """Run mbsync command asynchronously for email accounts.

    Args:
        account_name (str): The name of the email account to sync.
    """
    command = f"mbsync {account_name}"
    log.info(f"Syncing emails for {account_name}...")
    process = await asyncio.create_subprocess_shell(
        command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
    )
    stdout, stderr = await process.communicate()
    if stdout:
        log.info(f"Output for {account_name}: {stdout.decode()}")
    if stderr:
        log.error(f"Error for {account_name}: {stderr.decode()}")

async def run_vdirsyncer(account_name: str) -> None:
    """Run vdirsyncer command asynchronously for calendar accounts.

    Args:
        account_name (str): The name of the calendar account to sync.
    """
    command = f"vdirsyncer sync {account_name}"
    log.info(f"Syncing calendar for {account_name}...")
    process = await asyncio.create_subprocess_shell(
        command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
    )
    _, stderr = await process.communicate()
    if stderr:
        command_log = stderr.decode().strip()
        if "error" in command_log or "critical" in command_log:
            log.error(f"Output for {account_name}: {command_log}")
        elif len(command_log.splitlines()) > 1:
            log.info(f"Output for {account_name}: {command_log}")

def should_i_sync_today(cron_expr: str) -> bool:
    """Check if the current time matches the cron expression day and hour constraints."""
    _, hour, _, _, day_of_week = cron_expr.split()
    now = datetime.now()
    if "*" in hour:
        return True
    elif not (int(hour.split("-")[0]) <= now.hour <= int(hour.split("-")[1])):
        return False
    if day_of_week != "*" and str(now.weekday()) not in day_of_week.split(","):
        return False
    return True

async def main():
    log.info("Starting account syncer for emails and calendars")
    accounts_to_sync = {"emails": [], "calendars": []}

    # Schedule email accounts
    for account in accounts_config["emails"]:
        account_name = account["account_name"]
        for cron_expression in account["cron_expressions"]:
            if (
                should_i_sync_today(cron_expression)
                and account_name not in accounts_to_sync["emails"]
            ):
                accounts_to_sync["emails"].append(account_name)
            aiocron.crontab(cron_expression, func=run_mbsync, args=[account_name])
            log.info(
                f"Scheduled mbsync for {account_name} with cron expression: {cron_expression}"
            )

    # Schedule calendar accounts
    for account in accounts_config["calendars"]:
        account_name = account["account_name"]
        for cron_expression in account["cron_expressions"]:
            if (
                should_i_sync_today(cron_expression)
                and account_name not in accounts_to_sync["calendars"]
            ):
                accounts_to_sync["calendars"].append(account_name)
            aiocron.crontab(cron_expression, func=run_vdirsyncer, args=[account_name])
            log.info(
                f"Scheduled vdirsyncer for {account_name} with cron expression: {cron_expression}"
            )

    log.info("Running an initial fetch on today's accounts")
    for account_name in accounts_to_sync["emails"]:
        await run_mbsync(account_name)
    for account_name in accounts_to_sync["calendars"]:
        await run_vdirsyncer(account_name)

    log.info("Finished loading accounts")
    while True:
        await asyncio.sleep(60)

if __name__ == "__main__":
    asyncio.run(main())

Where:

accounts_config: Holds your account configuration. Each account must contain an account_name which should be the name of the mbsync or vdirsyncer profile, and cron_expressions must be a list of cron valid expressions you want the email to be synced.

Create the systemd service

We're using a non-root systemd service. You can follow these instructions to configure this service:

[Unit]
Description=Account Sync Service for emails and calendars
After=graphical-session.target

[Service]
Type=simple
ExecStart=/home/lyz/.local/share/virtualenvs/account_syncer/bin/python /home/lyz/.local/bin/
WorkingDirectory=/home/lyz/.local/bin
Restart=on-failure
StandardOutput=journal
StandardError=journal
SyslogIdentifier=account_syncer
Environment="PATH=/home/lyz/.local/share/virtualenvs/account_syncer/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="DISPLAY=:0"
Environment="DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"

[Install]
WantedBy=graphical-session.target

Remember to tweak the service to match your current case and paths.

As we'll probably need to enter our pass password we need the service to start once we've logged into the graphical interface.

Monitor the automation

It's always nice to know if the system is working as expected without adding mental load. To do that I'm creating the next loki rules:

t">

groups: - name: account_sync rules: - alert: AccountSyncIsNotRunningWarning expr: | (sum by(hostname) (count_over_time({job="systemd-journal", syslog_identifier="account_syncer"}[15m])) or sum by(hostname) (count_over_time({hostname="my_computer"} [15m])) * 0 ) == 0 for: 0m labels: severity: warning annotations: summary: "The account sync script is not running {{ $labels.hostname}}" - alert: AccountSyncIsNotRunningError expr: | (sum by(hostname) (count_over_time({job="systemd-journal", syslog_identifier="account_syncer"}[3h])) or sum by(hostname) (count_over_time({hostname="my_computer"} [3h])) * 0 ) == 0 for: 0m labels: severity: error annotations: summary: "The account sync script has been down for at least 3 hours {{ $labels.hostname}}" - alert: AccountSyncError expr: | count(rate({job="systemd-journal", syslog_identifier="account_syncer"} |= `` | logfmt | level_extracted=`error` [5m])) > 0 for: 0m labels: severity: warning annotations: summary: "There are errors in the account sync log at {{ $labels.hostname}}" - alert: EmailAccountIsOutOfSyncLyz expr: | (sum by(hostname) (count_over_time({job="systemd-journal", syslog_identifier="account_syncer"} | logfmt | msg=`Syncing emails for lyz...`[1h])) or sum by(hostname) (count_over_time({hostname="my_computer"} [1h])) * 0 ) == 0 for: 0m labels: severity: error annotations: summary: "The email account lyz has been out of sync for 1h {{ $labels.hostname}}" - alert: CalendarAccountIsOutOfSyncLyz expr: | (sum by(hostname) (count_over_time({job="systemd-journal", syslog_identifier="account_syncer"} | logfmt | msg=`Syncing calendar for lyz...`[3h])) or sum by(hostname) (count_over_time({hostname="my_computer"} [3h])) * 0 ) == 0 for: 0m labels: severity: error annotations: summary: "The calendar account lyz has been out of sync for 3h {{ $labels.hostname}}" Where: - You need to change my_computer for the hostname of the device running the service - Tweak the OutOfSync alerts to match your account (change the lyz part).
 These rules will raise: - A warning if the sync has not shown any activity in the last 15 minutes. - An error if the sync has not shown any activity in the last 3 hours. - An error if there is an error in the logs of the automation.

 Fava Dashboards
   New: Disable automatic OMEMO key acceptance.
 Dino automatically accepts new OMEMO keys from your own other devices and your chat partners by default. This default behaviour leads to the fact that the admin of the XMPP server could inject own public OMEMO keys without user verification, which enables the owner of the associated private OMEMO keys to decrypt your OMEMO secured conversation without being easily noticed.
 To prevent this, two actions are required, the second consists of several steps and must be taken for each new chat partner.
  First, the automatic acceptance of new keys from your own other devices must be deactivated. Configure this in the account settings of your own accounts.
 Second, the automatic acceptance of new keys from your chat partners must be deactivated. Configure this in the contact details of every chat partner. Be aware that in the case of group chats, the entire communication can be decrypted unnoticed if even one partner does not actively deactivate automatic acceptance of new OMEMO keys.
 
 Always confirm new keys from your chat partner before accepting them manually
 
  New: Dino does not use encryption by default.
 You have to initially enable encryption in the conversation window by clicking the lock-symbol and choose OMEMO. Future messages and file transfers to this contact will be encrypted with OMEMO automatically.
  Every chat partner has to enable encryption separately.
 If only one of two chat partner has activated OMEMO, only this part of the communication will be encrypted. The same applies with file transfers.
 If you get a message "This contact does not support OMEMO" make sure that your chatpartner has accepted the request to add him to your contact list and you accepted vice versa
 
 
  New: Install in Tails.
 If you have more detailed follow this article at the same time as you read this one. That one is more outdated but more detailed.
  Boot a clean Tails
 Create and configure the Persistent Storage
  Restart Tails and open the Persistent Storage
 
  Configure the persistence of the directory: 
echo -e '/home/amnesia/.local/share/dino source=dino' | sudo tee -a /live/persistence/TailsData_unlocked/persistence.conf > /dev/null
 
  Restart Tails
 
  Install the application: 
sudo apt-get update
sudo apt-get install dino-im
 
  Configure the dino-im alias to use torsocks
 sudo echo 'alias dino="torsocks dino-im &> /dev/null &"' >> /live/persistence/TailsData_unlocked/dotfiles/.bashrc
echo 'alias dino="torsocks dino-im &> /dev/null &"' >> ~/.bashrc
 
 
 
  New: Introduce Fava Dashboards.
 Installation
 pip install git+https://github.com/andreasgerstmayr/fava-dashboards.git
 Enable this plugin in Fava by adding the following lines to your ledger:
 2010-01-01 custom "fava-extension" "fava_dashboards"
 Then you'll need to create a dashboards.yaml file where your ledger lives.
 Configuration
 The plugin looks by default for a dashboards.yaml file in the directory of the Beancount ledger (e.g. if you run fava personal.beancount, the dashboards.yaml file should be in the same directory as personal.beancount).
 The configuration file can contain multiple dashboards, and a dashboard contains one or more panels. A panel has a relative width (e.g. 50% for 2 columns, or 33.3% for 3 column layouts) and a absolute height.
 The queries field contains one or multiple queries. The Beancount query must be stored in the bql field of the respective query. It can contain Jinja template syntax to access the panel and ledger variables described below (example: use {{ledger.ccy}} to access the first configured operating currency). The query results can be accessed via panel.queries[i].result, where i is the index of the query in the queries field.
 Note: Additionally to the Beancount query, Fava's filter bar further filters the available entries of the ledger.
 Common code for utility functions can be defined in the dashboards configuration file, either inline in utils.inline or in an external file defined in utils.path.
 Start your configuration
 It's best to tweak the example than to start from scratch. Get the example by: 
cd $(mktemp -d)
git clone https://github.com/andreasgerstmayr/fava-dashboards
cd fava-dashboards/example
fava example.beancount
 Configuration reference
 HTML, echarts and d3-sankey panels: The script field must contain valid JavaScript code. It must return a valid configuration depending on the panel type. The following variables and functions are available: * ext: the Fava ExtensionContext * ext.api.get("query", {bql: "SELECT ..."}: executes the specified BQL query * panel: the current (augmented) panel definition. The results of the BQL queries can be accessed with panel.queries[i].result. * ledger.dateFirst: first date in the current date filter * ledger.dateLast: last date in the current date filter * ledger.operatingCurrencies: configured operating currencies of the ledger * ledger.ccy: shortcut for the first configured operating currency of the ledger * ledger.accounts: declared accounts of the ledger * ledger.commodities: declared commodities of the ledger * helpers.urlFor(url): add current Fava filter parameters to url * utils: the return value of the utils code of the dashboard configuration
 Jinja2 panels: The template field must contain valid Jinja2 template code. The following variables are available: * panel: see above * ledger: see above * favaledger: a reference to the FavaLedger object
 Common Panel Properties * title: title of the panel. Default: unset * width: width of the panel. Default: 100% * height: height of the panel. Default: 400px * link: optional link target of the panel header. * queries: a list of dicts with a bql attribute. * type: panel type. Must be one of html, echarts, d3_sankey or jinja2.
 HTML panel The script code of HTML panels must return valid HTML. The HTML code will be rendered in the panel.
 ECharts panel The script code of Apache ECharts panels must return valid Apache ECharts chart options. Please take a look at the ECharts examples to get familiar with the available chart types and options.
 d3-sankey panel The script code of d3-sankey panels must return valid d3-sankey chart options. Please take a look at the example dashboard configuration dashboards.yaml.
 Jinja2 panel The template field of Jinja2 panels must contain valid Jinja2 template code. The rendered template will be shown in the panel.
 Debugging
 Add console.log strings in the javascript code to debug it.
 References
  Code
 Article
 
 Examples
  Fava Portfolio returns
 Fava investor
 
 
  New: Dashboard prototypes.
 Vertical bars with one serie using year
   - title: Net Year Profit 💰
    width: 50%
    link: /beancount/income_statement/
    queries:
    - bql: |
        SELECT year, sum(position) AS value
        WHERE
            account ~ '^Expenses:' OR
            account ~ '^Income:'
        GROUP BY year

      link: /beancount/balance_sheet/?time={time}
    type: echarts
    script: |
      const currencyFormatter = utils.currencyFormatter(ledger.ccy);
      const years = utils.iterateYears(ledger.dateFirst, ledger.dateLast)
      const amounts = {};

      // the beancount query only returns periods where there was at least one matching transaction, therefore we group by period
      for (let row of panel.queries[0].result) {
        amounts[`${row.year}`] = -row.value[ledger.ccy];
      }

      return {
        tooltip: {
          trigger: "axis",
          valueFormatter: currencyFormatter,
        },
        xAxis: {
          data: years,
        },
        yAxis: {
          axisLabel: {
            formatter: currencyFormatter,
          },
        },
        series: [
          {
            type: "bar",
            data: years.map((year) => amounts[year]),
            color: utils.green,
          },
        ],
      };
 Vertical bars using one serie using quarters
   - title: Net Quarter Profit 💰
    width: 50%
    link: /beancount/income_statement/
    queries:
    - bql: |
        SELECT quarter(date) as quarter, sum(position) AS value
        WHERE
            account ~ '^Expenses:' OR
            account ~ '^Income:'
        GROUP BY quarter

      link: /beancount/balance_sheet/?time={time}
    type: echarts
    script: |
      const currencyFormatter = utils.currencyFormatter(ledger.ccy);
      const quarters = utils.iterateQuarters(ledger.dateFirst, ledger.dateLast).map((q) => `${q.year}-${q.quarter}`);
      const amounts = {};

      // the beancount query only returns periods where there was at least one matching transaction, therefore we group by period
      for (let row of panel.queries[0].result) {
        amounts[`${row.quarter}`] = -row.value[ledger.ccy];
      }

      return {
        tooltip: {
          trigger: "axis",
          valueFormatter: currencyFormatter,
        },
        xAxis: {
          data: quarters,
        },
        yAxis: {
          axisLabel: {
            formatter: currencyFormatter,
          },
        },
        series: [
          {
            type: "bar",
            data: quarters.map((quarter) => amounts[quarter]),
          },
        ],
      };
 Vertical bars showing the evolution of one query over the months
   - title: Net Year Profit Distribution 💰
    width: 50%
    link: /beancount/income_statement/
    queries:
    - bql: |
        SELECT year, month, sum(position) AS value
        WHERE
            account ~ '^Expenses:' OR
            account ~ '^Income:'
        GROUP BY year, month
      link: /beancount/balance_sheet/?time={time}
    type: echarts
    script: |
      const currencyFormatter = utils.currencyFormatter(ledger.ccy);
      const years = utils.iterateYears(ledger.dateFirst, ledger.dateLast);
      const amounts = {};

      for (let row of panel.queries[0].result) {
        if (!amounts[row.year]) {
          amounts[row.year] = {};
        }
        amounts[row.year][row.month] = -row.value[ledger.ccy];
      }

      return {
        tooltip: {
          valueFormatter: currencyFormatter,
        },
        legend: {
          top: "bottom",
        },
        xAxis: {
          data: ['0','1','2','3','4','5','6','7','8','9','10','11','12'],
        },
        yAxis: {
          axisLabel: {
            formatter: currencyFormatter,
          },
        },
        series: years.map((year) => ({
          type: "bar",
          name: year,
          data: Object.values(amounts[year]),
          label: {
            show: false,
            formatter: (params) => currencyFormatter(params.value),
          },
        })),
      };
 
 
 Rocketchat
   New: How to use Rocketchat's API.
 The API docs are a bit weird, you need to go to endpoints and find the one you need. Your best bet though is to open the browser network console and see which requests they are doing and then to find them in the docs.
 
  New: Add end of life link.
 Warning they only support 6 months of versions! and they advice you with 12 days that you'll loose service if you don't update.
  End of life for the versions
 
 
 
 Content Management
 Jellyfin
   New: Introduce moonlight.
 Moonlight is an open source client implementation of NVIDIA GameStream that allows you to to stream your collection of games and apps from your GameStream-compatible PC to another device on your network or the Internet. You can play your favorite games on your PC, phone, tablet, or TV with Moonlight..
 References:
  Home
 Docs
 
 
  New: Python library.
 This is the API client from Jellyfin Kodi extracted as a python package so that other users may use the API without maintaining a fork of the API client. Please note that this API client is not complete. You may have to add API calls to perform certain tasks.
 It doesn't (yet) support async
 
  New: Troubleshoot pSystem.InvalidOperationException: There is an error in XML document (0, 0).
 This may happen if you run out of disk and some xml file in the jellyfin data directory becomes empty. The solution is to restore that file from backup.
 
  New: Enable hardware transcoding.
 Enable NVIDIA hardware transcoding
 Remove the artificial limit of concurrent NVENC transcodings
 Consumer targeted Geforce and some entry-level Quadro cards have an artificial limit on the number of concurrent NVENC encoding sessions (max of 8 on most modern ones). This restriction can be circumvented by applying an unofficial patch to the NVIDIA Linux and Windows driver.
 To apply the patch:
 First check that your current version is supported nvidia-smi, if it's not try to upgrade the drivers to a supported one, or think if you need more than 8 transcodings. 
wget https://raw.githubusercontent.com/keylase/nvidia-patch/refs/heads/master/patch.sh
chmod +x patch.sh
./patch.sh
 If you need to rollback the changes run ./patch.sh -r.
 You can also patch it within the docker itself
 services:
  jellyfin:
    image: jellyfin/jellyfin
    user: 1000:1000
    network_mode: 'host'
    volumes:
      - /path/to/config:/config
      - /path/to/cache:/cache
      - /path/to/media:/media
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
 Restart the docker and then check that you can access the graphics card with:
 docker exec -it jellyfin nvidia-smi
 Enable NVENC in Jellyfin and uncheck the unsupported codecs.
 Tweak the docker-compose
 The official Docker image doesn't include any NVIDIA proprietary driver.
 You have to install the NVIDIA driver and NVIDIA Container Toolkit on the host system to allow Docker access to your GPU.
 
 
 Immich
   New: Set default quality of request per user.
 Sometimes one specific user continuously asks for a better quality of the content. If you go into the user configuration (as admin) you can set the default quality profiles for that user.
 
  New: Introduce immich.
 Self-hosted photo and video backup solution directly from your mobile phone.
 References:
  Home
 Api
 Docs
 Source.
 Blog
 Demo
 
 
  New: Installation.
  Create a directory of your choice (e.g. ./immich-app) to hold the docker-compose.yml and .env files.
 
 mkdir ./immich-app
cd ./immich-app
  Download docker-compose.yml, example.env and optionally the hwaccel.yml files:
 
 
wget -O docker-compose.yaml https://github.com/immich-app/immich/releases/latest/download/dockr-compose.yml
wget -O .env https://github.com/immich-app/immich/releases/latest/download/example.env
wget https://github.com/immich-app/immich/releases/latest/download/hwaccel.yml
 - Tweak those files with these thoughts in mind: - immich won't respect your upload media directory structure, so until you trust the softwar copy your media to the uploads directory. - immich is not stable so you need to disable the upgrade from watchtower. The easiest way is to pin the latest stable version in the .env file. - Populate custom database information if necessary. - Populate UPLOAD_LOCATION with your preferred location for storing backup assets. - Consider changing DB_PASSWORD to something randomly generated
  From the directory you created in Step 1, (which should now contain your customized docker-compose.yml and .env files) run:
 
 docker compose up -d
 
  New: Configure smart search for other language.
 You can change to a multilingual model listed here by going to Administration > Machine Learning Settings > Smart Search and replacing the name of the model.
 Choose the one that has more downloads. For example, if you'd want the +immich-app/XLM-Roberta-Large-Vit-B-16Plus model, you should only enter XLM-Roberta-Large-Vit-B-16Plus in the program configuration. Be careful not to add trailing whitespaces.
 Be sure to re-run Smart Search on all assets after this change. You can then search in over 100 languages.
 
  New: External storage.
 If you have an already existing library somewhere immich is installed you can use an external library. Immich will respect the files on that directory.
 It won't create albums from the directory structure. If you want to do that check this or this solutions.
 
  New: My personal workflow.
 I've tailored a personal workflow given the next thoughts:
  I don't want to expose Immich to the world, at least until it's a stable product.
 I already have in place a sync mechanism with syncthing for all the mobile stuff
 I do want to still be able to share some albums with my friends and family.
 I want some mobile directories to be cleaned after importing the data (for example the camera/DCIM), but others should leave the files as they are after the import (OsmAnd+ notes).
 
 Ingesting the files:
 As all the files I want to ingest are sent to the server through syncthing, I've created a cron script that copies or moves the required files. Something like:
 date
echo 'Updating the OsmAnd+ data'
rsync -auhvEX --progress /data/apps/syncthing/data/Osmand/avnotes /data/media/pictures/unclassified

echo 'Updating the Camera data'
mv /data/apps/syncthing/data/camera/Camera/* /data/media/pictures/unclassified/

echo 'Cleaning laptop home'
mv /data/media/downloads/*jpeg /data/media/downloads/*jpg /data/media/downloads/*png /data/media/pictures/unclassified/
echo
 Where :
  /data/media/pictures/unclassified is a subpath of my external library.
 The last echo makes sure that the program exits with a return code of 0. The script is improbable as it only takes into account the happy path, and I'll silently miss errors on it's execution. But as a first iteration it will do the job.
 
 Then run the script in a cron and log the output to journald:
 0 0 * * * /bin/bash /usr/local/bin/archive-photos.sh | /usr/bin/logger -t archive_fotos
 Make sure to configure the update library cron job to run after this script has ended.
 
  New: Not there yet.
 There are some features that are still lacking:
  Image rotation
 Smart albums
 Image rating
 Tags
 Nested albums
 Duplication management
 Search guide
 
 
  New: Edit an image metadata.
 You can't do it directly through the interface yet, use exiftool instead.
 This is interesting to remove the geolocation of the images that are not yours
 
  New: Keyboard shortcuts.
 You can press ? to see the shortcuts. Some of the most useful are:
  f: Toggle favourite
 Shift+a: Archive element
 
 
 
 Mediatracker
   New: How to use the mediatracker API.
 I haven't found a way to see the api docs from my own instance. Luckily you can browse it at the official instance.
 You can create an application token on your user configuration. Then you can use it with something similar to:
 curl -H 'Content-Type: application/json' https://mediatracker.your-domain.org/api/logs\?token\=your-token | jq
 
  New: Introduce python library.
 There is a python library although it's doesn't (yet) have any documentation and the functionality so far is only to get information, not to push changes.
 
  New: Get list of tv shows.
 With /api/items?mediaType=tv you can get a list of all tv shows with the next interesting fields:
  id: mediatracker id
 tmdbId:
 tvdbId:
 imdbId:
 title:
 lastTimeUpdated: epoch time
 lastSeenAt: epoch time
 seen: bool
 onWatchlist: bool
 firstUnwatchedEpisode:
 id: mediatracker episode id
 episodeNumber:
 seasonNumber
 tvShowId:
 seasonId:
 lastAiredEpisode: same schema as before
 
 Then you can use the api/details/{mediaItemId} endpoint to get all the information of all the episodes of each tv show.
 
  New: Add missing books.
  Register an account in openlibrary.com
 Add the book
 Then add it to mediatracker
 
 
 
 ffmpeg
   New: Reduce the video size.
 If you don't mind using H.265 replace the libx264 codec with libx265, and push the compression lever further by increasing the CRF value — add, say, 4 or 6, since a reasonable range for H.265 may be 24 to 30. Note that lower CRF values correspond to higher bitrates, and hence produce higher quality videos.
 ffmpeg -i input.mp4 -vcodec libx265 -crf 28 output.mp4
 If you want to stick to H.264 reduce the bitrate. You can check the current one with ffprobe input.mkv. Once you've chosen the new rate change it with:
 ffmpeg -i input.mp4 -b 3000k output.mp4
 Additional options that might be worth considering is setting the Constant Rate Factor, which lowers the average bit rate, but retains better quality. Vary the CRF between around 18 and 24 — the lower, the higher the bitrate.
 ffmpeg -i input.mp4 -vcodec libx264 -crf 20 output.mp4
 
 
 Photo management
   New: Do comparison of selfhosted photo software.
 There are many alternatives to self host a photo management software, here goes my personal comparison. You should complement this article with meichthys one.
  TL;DR: I'd first go with Immich, then LibrePhotos and then LycheeOrg
 
    Software  Home-Gallery  Immich  LibrePhotos  
 
   UI  Fine  Good  Fine  
  Popular (stars)  614  25k  6k  
  Active (PR/Issues)(1)  ?  251/231  27/16  
  Easy deployment  ?  True  Complicated  
  Good docs  True  True  True  
  Stable  True  False  True  
  Smart search  ?  True  True  
  Language  Javascript  Typescript  Python  
  Batch edit  True  True  ?  
  Multi-user  False  True  ?  
  Mobile app  ?  True  ?  
  Oauth support  ?  True  ?  
  Facial recognition  ?  True  ?  
  Scales well  False  True  ?  
  Favourites  ?  True  ?  
  Archive  ?  True  ?  
  Has API  True  True  ?  
  Map support  True  True  ?  
  Video Support  True  True  ?  
  Discover similar  True  True  ?  
  Static site  True  False  ?  
 
 
  (1): It refers to the repository stats of the last month
 
 Immich:
 References:
  Home
 Demo
 Source.
 
 Pros: - Smart search is awesome Oo - create shared albums that people can use to upload and download - map with leaflet - explore by people and places - docker compose - optional hardware acceleration - very popular 25k stars, 1.1k forks - has a CLI - can load data from a directory - It has an android app on fdroid to automatically upload media - sharing libraries with other users and with the public - favorites and archive - public sharing - oauth2, specially with authentik <3 - extensive api: https://immich.app/docs/api/introduction - It has an UI similar to google photos, so it would be easy for non technical users to use. - Batch edit - Discover similar through the smart search
 Cons:
  If you want to get results outside the smart search you are going to have a bad time. There is still no way to filter the smart search results or even sort them. You're sold to the AI.
 dev suggests not to use watchtower as the project is in unstable alpha
 Doesn't work well in firefox
 It doesn't work with tags which you don't need because the smart search is so powerful.
 Scans pictures on the file system
 
 LibrePhotos:
 References:
  Source
 Docs
 Demo
 Outdated comparison
 
 Pros:
  docker compose, although you need to build the dockers yourself
 android app
 6k stars, 267 forks
 object, scene ai extraction
 
 Cons:
  Not as good as Immich.
 
 Home-Gallery:
 You can see the demo here.
 Nice features:
  Simple UI
 Discover similar images
 Static site generator
 Shift click to do batch editing
 
 Cons:
  All users see all media
 The whole database is loaded into the browser and requires recent (mobile) devices and internet connection
 Current tested limits are about 400,000 images/videos
 
 Lycheeorg:
 References:
  Home
 Docs
 Source
 
 Pros:
  Sharing like it should be. One click and every photo and album is ready for the public. You can also protect albums with passwords if you want. It's under your control.
 Manual tags
 apparently safe upgrades
 docker compose
 2.9k stars
 
 Cons: - demo doesn't show many features - no ai
 Photoview:
  Home
 Source
 Docs
 
 Pros:
  Syncs with file system
 Albums and individual photos or videos can easily be shared by generating a public or password protected link.
 users support
 maps support
 4.4k stars
 Face recognition
 
 Cons:
  Demo difficult to understand as it's not in english
 mobile app only for ios
 last commit 6 months ago
 
 Pigallery2:
 References:
  Home
 
 Pros:
  map
 The gallery also supports *.gpx file to show your tracked path on the map too
 App supports full boolean logic with negation and exact or wildcard search. It also provides handy suggestions with autocomplete.
 face recognitiom: PiGallery2 can read face reagions from photo metadata. Current limitation: No ML-based, automatic face detection.
 rating and grouping by rating
 easy query builder
 video transcoding
 blog support. Markdown based blogging support
 
 You can write some note in the *.md files for every directory
  You can create logical albums (a.k.a.: Saved search) from any search query. Current limitation: It is not possible to create albums from a manually picked photos.
 PiGallery2 has a rich settings page where you can easily set up the gallery.
 
 Cons: - no ml face recognition
 Piwigo:
 References:
  Home
 Source
 
 Piwigo is open source photo management software. Manage, organize and share your photo easily on the web. Designed for organisations, teams and individuals
 Pros:
  Thousands of organizations and millions of individuals love using Piwigo
 shines when it comes to classifying thousands or even hundreds of thousands of photos.
 Born in 2002, Piwigo has been supporting its users for more than 21 years. Always evolving!
 You can add photos with the web form, any FTP client ora desktop application like digiKam, Shotwell, Lightroom ormobile applications.
 Filter photos from your collection, make a selection and apply actions in batch: change the author, add some tags, associate to a new album, set geolocation...
 Make your photos private and decide who can see each of them. You can set permissions on albums and photos, for groups or individual users.
 Piwigo can read GPS latitude and longitude from embedded metadata. Then, with plugin for Google Maps or OpenStreetMap, Piwigo can display your photos on an interactive map.
 Change appearance with themes. Add features with plugins. Extensions require just a few clicks to get installed. 350 extensions available, and growing!
 With the Fotorama plugin, or specific themes such as Bootstrap Darkroom, you can experience the full screen slideshow.
 Your visitors can post comments, give ratings, mark photos as favorite, perform searches and get notified of news by email.
 Piwigo web API makes it easy for developers to perform actions from other applications
 GNU General Public License, or GPL
 2.9 k stars, 400 forks
 still active
 nice release documents: https://piwigo.org/release-14.0.0
 
 Cons:
  Official docs don't mention docker
 no demo: https://piwigo.org/demo
 Unpleasant docs: https://piwigo.org/doc/doku.php
 Awful plugin search: https://piwigo.org/ext/
 
 Damselfly:
 Fast server-based photo management system for large collections of images. Includes face detection, face & object recognition, powerful search, and EXIF Keyword tagging. Runs on Linux, MacOS and Windows.
 Very ugly UI
 Saigal:
 Too simple
 Spis:
 Low number of maintainers Too simple
 
 
 Kodi
  New: Start working on a migration script to mediatracker.
  New: Extract kodi data from the database.
 At ~/.kodi/userdata/Database/MyVideos116.db you can extract the data from the next tables:
  In the movie_view table there is:
 idMovie: kodi id for the movie
 c00: Movie title
 userrating
 uniqueid_value: The id of the external web service
 uniqueid_type: The web it extracts the id from
 lastPlayed: The reproduction date
 In the tvshow_view table there is:
 idShow: kodi id of a show
 c00: title
 userrating
 lastPlayed: The reproduction date
 uniqueid_value: The id of the external web service
 uniqueid_type: The web it extracts the id from
 In the season_view there is no interesting data as the userrating is null on all rows.
 In the episode_view table there is:
 idEpisodie: kodi id for the episode
 idShow: kodi id of a show
 `idSeason: kodi id of a season
 c00: title
 userrating
 lastPlayed: The reproduction date
 uniqueid_value: The id of the external web service
 uniqueid_type: The web it extracts the id from. I've seen mainly tvdb and sonarr
 Don't use the rating table as it only stores the ratings from external webs such as themoviedb:
 
 
 
 Knowledge Management
   New: Use ebops to create anki cards.
  Ask the AI to generate Anki cards based on the content.
 Save those anki cards in an orgmode (anki.org) document
 Use ebops add-anki-notes to automatically add them to Anki
 
 
 
 Anki
   New: What to do when you need to edit a card but don't have the time.
 You can mark it with a red flag so that you remember to edit it the next time you see it.
 
  New: Center images.
 In your config enable the attr_list extension:
 markdown_extensions:
  - attr_list
 On your extra.css file add the center class
 .center {
    display: block;
    margin: 0 auto;
}
 Now you can center elements by appending the attribute:
 ![image](../_imatges/ebc_form_01.jpg){: .center}
 
  New: Install the official sync server.
 
 
 Analytical web reading
   New: Introduce Analytical web reading.
 One part of the web 3.0 is to be able to annotate and share comments on the web. This article is my best try to find a nice open source privacy friendly tool. Spoiler: there aren't any :P
 The alternative I'm using so far is to process the data at the same time as I underline it.
  At the mobile/tablet you can split your screen and have Orgzly on one tab and the browser in the other. So that underlining, copy and paste doesn't break too much the workflow.
 At the eBook I underline it and post process it after.
 
 The idea of using an underlining tool makes sense in the case to post process the content in a more efficient environment such as a laptop.
 The use of Orgzly is kind of a preprocessing. If the underlining software can easily export the highlighted content along with the link to the source then it would be much quicker
 The advantage of using Orgzly is also that it works today both online and offline and it is more privacy friendly.
 On the post I review some of the existent solutions
 
 
 Digital Gardens
   New: Add the not by AI badge.
 Not by AI is an initiative to mark content as created by humans instead of AI.
 To automatically add the badge to all your content you can use the next script:
 
echo "Checking the Not by AI badge"
find docs -iname '*md' -print0 | while read -r -d $'\0' file; do
    if ! grep -q not-by-ai.svg "$file"; then
        echo "Adding the Not by AI badge to $file"
        echo "[![](../img/not-by-ai.svg){: .center}](https://notbyai.fyi)" >>"$file"
    fi
done
 You can see how it's used in this blog by looking at the Makefile and the gh-pages.yaml workflow.
 
 
 Aleph
   New: Add note on aleph and prometheus.
 Aleph now exposes prometheus metrics on the port 9100
 
  New: Debug ingestion errors.
 Assuming that you've set up Loki to ingest your logs I've so far encountered the next ingest issues:
  Cannot open image data using Pillow: broken data stream when reading image files: The log trace that has this message also contains a field trace_id which identifies the ingestion process. With that trace_id you can get the first log trace with the field logger = "ingestors.manager" which will contain the file path in the message field. Something similar to Ingestor [<E('9972oiwobhwefoiwefjsldkfwefa45cf5cb585dc4f1471','path_to_the_file_to_ingest.pdf')>]
 A traceback with the next string Failed to process: Could not extract PDF file: FileDataError('cannot open broken document'): This log trace has the file path in the message field. Something similar to [<E('9972oiwobhwefoiwefjsldkfwefa45cf5cb585dc4f1471','path_to_the_file_to_ingest.pdf')>] Failed to process: Could not extract PDF file: FileDataError('cannot open broken document')
 
 I thought of making a python script to automate the files that triggered an error, but in the end I extracted the file names manually as they weren't many.
 Once you have the files that triggered the errors, the best way to handle them is to delete them from your investigation and ingest them again.
 
  New: Add support channel.
 Support chat
 
  New: API Usage.
 The Aleph web interface is powered by a Flask HTTP API. Aleph supports an extensive API for searching documents and entities. It can also be used to retrieve raw metadata, source documents and other useful details. Aleph's API tries to follow a pragmatic approach based on the following principles:
  All API calls are prefixed with an API version; this version is /api/2/.
 Responses and requests are both encoded as JSON. Requests should have the Content-Type and Accept headers set to application/json.
 The application uses Representational State Transfer (REST) principles where convenient, but also has some procedural API calls.
 The API allows API Authorization via an API key or JSON Web Tokens.
 
 Authentication and authorization
 By default, any Aleph search will return only public documents in responses to API requests.
 If you want to access documents which are not marked public, you will need to sign into the tool. This can be done through the use on an API key. The API key for any account can be found by clicking on the "Profile" menu item in the navigation menu.
 The API key must be sent on all queries using the Authorization HTTP header:
 Authorization: ApiKey 363af1e2b03b41c6b3adc604956e2f66
 Alternatively, the API key can also be sent as a query parameter under the api_key key.
 Similarly, a JWT can be sent in the Authorization header, after it has been returned by the login and/or OAuth processes. Aleph does not use session cookies or any other type of stateful API.
 
  New: Crossreferencing mentions with entities.
 Mentions are names of people or companies that Aleph automatically extracts from files you upload. Aleph includes mentions when cross-referencing a collection, but only in one direction.
 Consider the following example:
  "Collection A" contains a file. The file mentions "John Doe".
 "Collection B" contains a Person entity named "John Doe".
 
 If you cross-reference “Collection A”, Aleph includes the mention of “John Doe” in the cross-referencing and will find a match for it in “Collection B”.
 However, if you cross-reference “Collection B”, Aleph doesn't consider mentions when trying to find a match for the Person entity.
 As long as you only want to compare the mentions in one specific collection against entities (but not mentions) in another collection, Aleph’s cross-ref should be able to do that. If you want to compare entities in a specific collection against other entities and mentions in other collections, you will have to do that yourself.
 If you have a limited number of collection, one option might be to fetch all mentions and automatically create entities for each mention using the API.
 To fetch a list of mentions for a collection you can use the /api/2/entities?filter:collection_id=137&filter:schemata=Mention API request.
 
  New: Alephclient cli tool.
 alephclient is a command-line client for Aleph. It can be used to bulk import structured data and files and more via the API, without direct access to the server.
 Installation
 You can now install alephclient using pip although I recommend to use pipx instead:
 pipx install alephclient
 alephclient needs to know the URL of the Aleph instance to connect to. For privileged operations (e.g. accessing private datasets or writing data), it also needs your API key. You can find your API key in your user profile in the Aleph UI.
 Both settings can be provided by setting the environment variables ALEPHCLIENT_HOST and ALEPHCLIENT_API_KEY, respectively, or by passing them in with --host and --api-key options.
 export ALEPHCLIENT_HOST=https://aleph.occrp.org/
export ALEPHCLIENT_API_KEY=YOUR_SECRET_API_KEY
 You can now start using alephclient for example to upload an entire directory to Aleph.
 Upload an entire directory to Aleph While you can upload multiple files and even entire directories at once via the Aleph UI, using the alephclient CLI allows you to upload files in bulk much quicker and more reliable.
 Run the following alephclient command to upload an entire directory to Aleph:
 alephclient crawldir --foreign-id wikileaks-cable /Users/sunu/data/cable
 This will upload all files in the directory /Users/sunu/data/cable (including its subdirectories) into an investigation with the foreign ID wikileaks-cable. If no investigation with this foreign ID exists, a new investigation is created (in theory, but it didn't work for me, so manually create the investigation and then copy it's foreign ID).
 If you’d like to import data into an existing investigation and do not know its foreign ID, you can find the foreign ID in the Aleph UI. Navigate to the investigation homepage. The foreign ID is listed in the sidebar on the right.
 
  New: Other tools for the ecosystem.
 
  New: Available datasets.
 OpenSanctions helps investigators find leads, allows companies to manage risk and enables technologists to build data-driven products.
 You can check their datasets.
 
  New: Offshore-graph.
 offshore-graph contains scripts that will merge the OpenSanctions Due Diligence dataset with the ICIJ OffshoreLeaks database in order create a combined graph for analysis.
 The result is a Cypher script to load the full graph into the Neo4J database and then browse it using the Linkurious investigation platform.
 Based on name-based entity matching between the datasets, an analyst can use this graph to find offshore holdings linked to politically exposed and sanctioned individuals.
 As a general alternative, you can easily export and convert entities from an Aleph instance to visualize them in Neo4j or Gephi using the ftm CLI: https://docs.aleph.occrp.org/developers/how-to/data/export-network-graphs/
 
 
 Torrent management
 qBittorrent
   New: Troubleshoot Trackers stuck on Updating.
 Sometimes the issue comes from an improvable configuration. In advanced:
  Ensure that there are enough Max concurrent http announces: I changed from 50 to 500
 Select the correct interface and Optional IP address to bind to. In my case I selected tun0 as I'm using a vpn and All IPv4 addresses as I don't use IPv6.
 
 
 
 Unpackerr
   New: Completed item still waiting no extractable files found at.
 This trace in the logs (which is super noisy) is not to worry.
 Unpackerr is just telling you something is stuck in your sonar queue. It's not an error, and it's not trying to extract it (because it has no compressed files). The fix is to figure out why it's stuck in the queue.
 
 
 Health
 Teeth
   New: Suggestion on how to choose the toothpaste to buy.
 When choosing a toothpaste choose the one that has a higher percent of fluoride.
 
 
 Coding
 Languages
 Bash snippets
   New: Show the progresion of a long running task with dots.
 echo -n "Process X is running."

sleep 1
echo -n "."
sleep 1
echo -n "."

echo ""
 
  New: Self delete shell script.
 Add at the end of the script
 rm -- "$0"
 $0 is a magic variable for the full path of the executed script.
 
  New: Add a user to the sudoers through command line.
 Add the user to the sudo group:
 sudo  usermod -a -G sudo <username>
 The change will take effect the next time the user logs in.
 This works because /etc/sudoers is pre-configured to grant permissions to all members of this group (You should not have to make any changes to this):
 %sudo   ALL=(ALL:ALL) ALL
 
  New: Error management done well in bash.
 If you wish to capture error management in bash you can use the next format
 if ( ! echo "$EMAIL" >> "$USER_TOTP_FILE" ) then
    echo "** Error: could not associate email for user $USERNAME"
    exit 1
fi
 
  New: Compare two semantic versions.
 This article gives a lot of ways to do it. For my case the simplest is to use dpkg to compare two strings in dot-separated version format in bash.
 Usage: dpkg --compare-versions <condition>
 If the condition is true, the status code returned by dpkg will be zero (indicating success). So, we can use this command in an if statement to compare two version numbers: 
$ if $(dpkg --compare-versions "2.11" "lt" "3"); then echo true; else echo false; fi
true
 
  New: Exclude list of extensions from find command.
 find . -not \( -name '*.sh' -o -name '*.log' \)
 
  New: Do relative import of a bash library.
 If you want to import a file lib.sh that lives in the same directory as the file that is importing it you can use the next snippet:
 source "$(dirname "$(realpath "$0")")/lib.sh"
 If you use source ./lib.sh you will get an import error if you run the script on any other place that is not the directory where lib.sh lives.
 
  New: Check the battery status.
 This article gives many ways to check the status of a battery, for my purposes the next one is enough
 
cat /sys/class/power_supply/BAT0/capacity
 feat(bash_snippets#Check if file is being sourced): Check if file is being sourced
 Assuming that you are running bash, put the following code near the start of the script that you want to be sourced but not executed:
 if [ "${BASH_SOURCE[0]}" -ef "$0" ]
then
    echo "Hey, you should source this script, not execute it!"
    exit 1
fi
 Under bash, ${BASH_SOURCE[0]} will contain the name of the current file that the shell is reading regardless of whether it is being sourced or executed.
 By contrast, $0 is the name of the current file being executed.
 -ef tests if these two files are the same file. If they are, we alert the user and exit.
 Neither -ef nor BASH_SOURCE are POSIX. While -ef is supported by ksh, yash, zsh and Dash, BASH_SOURCE requires bash. In zsh, however, ${BASH_SOURCE[0]} could be replaced by ${(%):-%N}.
 
  New: Parsing bash arguments.
 Long story short, it's nasty, think of using a python script with typer instead.
 There are some possibilities to do this:
  The old getops
 argbash library
 Build your own parser
 
 
  New: Fix docker error: KeyError ContainerConfig.
 You need to run docker-compose down and then up again.
 
  New: Set static ip with nmcli.
 nmcli con mod "your-ssid" ipv4.addresses
  ipv4.method "manual" \
  ipv4.addresses "your_desired_ip" \
  ipv4.gateway "your_desired_gateway" \
  ipv4.dns "1.1.1.1,2.2.2.2" \
  ipv4.routes "192.168.32.0 0.0.0.0" \
 The last one is to be able to connect to your LAN, change the value accordingly.
 
  New: Fix unbound variable error.
 You can check if the variable is set and non-empty with: 
[ -n "${myvariable-}" ]
 
  New: Compare two semantic versions with sort.
 If you want to make it work in non-Debian based systems you can use sort -V -C
 printf "2.0.0\n2.1.0\n" | sort -V -C  # Return code 0
printf "2.2.0\n2.1.0\n" | sort -V -C  # Return code 1
 
 
 Bash testing
   New: Introduce bats.
 Bash Automated Testing System is a TAP-compliant testing framework for Bash 3.2 or above. It provides a simple way to verify that the UNIX programs you write behave as expected.
 A Bats test file is a Bash script with special syntax for defining test cases. Under the hood, each test case is just a function with a description.
 @test "addition using bc" {
  result="$(echo 2+2 | bc)"
  [ "$result" -eq 4 ]
}

@test "addition using dc" {
  result="$(echo 2 2+p | dc)"
  [ "$result" -eq 4 ]
}
 Bats is most useful when testing software written in Bash, but you can use it to test any UNIX program.
 References: - Source - Docs
 
 
 aiocron
   New: Introduce aiocron.
 aiocron is a python library to run cron jobs in python asyncronously.
 Usage
 You can run it using a decorator
 >>> import aiocron
>>> import asyncio
>>>
>>> @aiocron.crontab('*/30 * * * *')
... async def attime():
...     print('run')
...
>>> asyncio.get_event_loop().run_forever()
 Or by calling the function yourself
 >>> cron = crontab('0 * * * *', func=yourcoroutine, start=False)
 Here's a simple example on how to run it in a script:
 import asyncio
from datetime import datetime
import aiocron

async def foo(param):
    print(datetime.now().time(), param)

async def main():
    cron_min = aiocron.crontab('*/1 * * * *', func=foo, args=("At every minute",), start=True)
    cron_hour = aiocron.crontab('0 */1 * * *', func=foo, args=("At minute 0 past every hour.",), start=True)
    cron_day = aiocron.crontab('0 9 */1 * *', func=foo, args=("At 09:00 on every day-of-month",), start=True)
    cron_week = aiocron.crontab('0 9 * * Mon', func=foo, args=("At 09:00 on every Monday",), start=True)

    while True:
        await asyncio.sleep(1)

asyncio.run(main())
 You have more complex examples in the repo
 Installation
 pip install aiocron
 References - Source
 
 
 Configure Docker to host the application
   New: Inspect contents of Lua table in Neovim.
 When using Lua inside of Neovim you may need to view the contents of Lua tables, which are a first class data structure in Lua world. Tables in Lua can represent ordinary arrays, lists, symbol tables, sets, records, graphs, trees, etc.
 If you try to just print a table directly, you will get the reference address for that table instead of the content, which is not very useful for most debugging purposes:
 :lua print(vim.api.nvim_get_mode())
" table: 0x7f5b93e5ff88
 To solve this, Neovim provides the vim.inspect function as part of its API. It serializes the content of any Lua object into a human readable string.
 For example you can get information about the current mode like so:
 :lua print(vim.inspect(vim.api.nvim_get_mode()))
" {  blocking = false,  mode = "n"}
 
  New: Send logs to journald.
 The journald logging driver sends container logs to the systemd journal. Log entries can be retrieved using the journalctl command, through use of the journal API, or using the docker logs command.
 In addition to the text of the log message itself, the journald log driver stores the following metadata in the journal with each message: | Field | Description | | --- | ---- | | CONTAINER_ID | The container ID truncated to 12 characters. | | CONTAINER_ID_FULL | The full 64-character container ID. | | CONTAINER_NAME | The container name at the time it was started. If you use docker rename to rename a container, the new name isn't reflected in the journal entries. | | CONTAINER_TAG, | SYSLOG_IDENTIFIER The container tag ( log tag option documentation). | | CONTAINER_PARTIAL_MESSAGE | A field that flags log integrity. Improve logging of long log lines. |
 To use the journald driver as the default logging driver, set the log-driver and log-opts keys to appropriate values in the daemon.json file, which is located in /etc/docker/.
 {
  "log-driver": "journald"
}
 Restart Docker for the changes to take effect.
 
  New: Send the logs to loki.
 There are many ways to send logs to loki
  Using the json driver and sending them to loki with promtail with the docker driver
 Using the docker plugin: Grafana Loki officially supports a Docker plugin that will read logs from Docker containers and ship them to Loki.
 
 I would not recommend to use this path because there is a known issue that deadlocks the docker daemon :S. The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of max_retries has been exceeded. To avoid the dropping of log entries, setting max_retries to zero allows unlimited retries; the driver will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck.
 The wait time can be lowered by setting loki-retries=2, loki-max-backoff_800ms, loki-timeout=1s and keep-file=true. This way the daemon will be locked only for a short time and the logs will be persisted locally when the Loki client is unable to re-connect.
 To avoid this issue, use the Promtail Docker service discovery. - Using the journald driver and sending them to loki with promtail with the journald driver. This has worked for me but the labels extracted are not that great.
 
  New: Solve syslog getting filled up with docker network recreation.
 If you find yourself with your syslog getting filled up by lines similar to:
  Jan 15 13:19:19 home kernel: [174716.097109] eth2: renamed from veth0adb07e
 Jan 15 13:19:20 home kernel: [174716.145281] IPv6: ADDRCONF(NETDEV_CHANGE): vethcd477bc: link becomes ready
 Jan 15 13:19:20 home kernel: [174716.145337] br-1ccd0f48be7c: port 5(vethcd477bc) entered blocking state
 Jan 15 13:19:20 home kernel: [174716.145338] br-1ccd0f48be7c: port 5(vethcd477bc) entered forwarding state
 Jan 15 13:19:20 home kernel: [174717.081132] br-fbe765bc7d0a: port 2(veth31cdd6f) entered disabled state
 Jan 15 13:19:20 home kernel: [174717.081176] vethc4da041: renamed from eth0
 Jan 15 13:19:21 home kernel: [174717.214911] br-fbe765bc7d0a: port 2(veth31cdd6f) entered disabled state
 Jan 15 13:19:21 home kernel: [174717.215917] device veth31cdd6f left promiscuous mode
 Jan 15 13:19:21 home kernel: [174717.215919] br-fbe765bc7d0a: port 2(veth31cdd6f) entered disabled state
 It probably means that some docker is getting recreated continuously. Those traces are normal logs of docker creating the networks, but as they do each time the docker starts, if it's restarting continuously then you have a problem.
 
  New: Minify the images.
 dive and slim are two cli tools you can use to optimise the size of your dockers.
 
 
 Logging
   New: Configure the logging module to log directly to systemd's journal.
 To use systemd.journal in Python, you need to install the systemd-python package. This package provides bindings for systemd functionality.
 Install it using pip:
 
pip install systemd-python
 Below is an example Python script that configures logging to send messages to the systemd journal:
 import logging
from systemd.journal import JournalHandler

logger = logging.getLogger('my_app')
logger.setLevel(logging.DEBUG)  # Set the logging level

journal_handler = JournalHandler()
journal_handler.setLevel(logging.DEBUG)  # Adjust logging level if needed
journal_handler.addFilter(
    lambda record: setattr(record, "SYSLOG_IDENTIFIER", "mbsync_syncer") or True
)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
journal_handler.setFormatter(formatter)

logger.addHandler(journal_handler)

logger.info("This is an info message.")
logger.error("This is an error message.")
logger.debug("Debugging information.")
 When you run the script, the log messages will be sent to the systemd journal. You can view them using the journalctl command:
 sudo journalctl -f
 This command will show the latest log entries in real time. You can filter by your application name using:
 sudo journalctl -f -t my_app
 Replace my_app with the logger name you used (e.g., 'my_app').
 Additional Tips - Tagging: You can add a custom identifier for your logs by setting logging.getLogger('your_tag'). This will allow you to filter logs using journalctl -t your_tag. - Log Levels: You can control the verbosity of the logs by setting different levels (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL).
 Example Output in the Systemd Journal
 You should see entries similar to the following in the systemd journal:
 Nov 15 12:45:30 my_hostname my_app[12345]: 2024-11-15 12:45:30,123 - my_app - INFO - This is an info message.
Nov 15 12:45:30 my_hostname my_app[12345]: 2024-11-15 12:45:30,124 - my_app - ERROR - This is an error message.
Nov 15 12:45:30 my_hostname my_app[12345]: 2024-11-15 12:45:30,125 - my_app - DEBUG - Debugging information.
 This approach ensures that your logs are accessible through standard systemd tools and are consistent with other system logs. Let me know if you have any additional requirements or questions!
 
 
 SQLite
   New: Get running instances.
 import boto3

ec2 = boto3.client('ec2')

running_instances = [
    instance
    for page in ec2.get_paginator('describe_instances').paginate()
    for reservation in page['Reservations']
    for instance in reservation['Instances']]
    if instance['State']['Name'] == 'running'
]
 
  New: Order by a column descending.
 SELECT
   select_list
FROM
   table
ORDER BY
    column_1 ASC,
    column_2 DESC;
 
 
 Protocols
   New: Introduce Python Protocols.
 The Python type system supports two ways of deciding whether two objects are compatible as types: nominal subtyping and structural subtyping.
 Nominal subtyping is strictly based on the class hierarchy. If class Dog inherits class Animal, it’s a subtype of Animal. Instances of Dog can be used when Animal instances are expected. This form of subtyping subtyping is what Python’s type system predominantly uses: it’s easy to understand and produces clear and concise error messages, and matches how the native isinstance check works – based on class hierarchy.
 Structural subtyping is based on the operations that can be performed with an object. Class Dog is a structural subtype of class Animal if the former has all attributes and methods of the latter, and with compatible types.
 Structural subtyping can be seen as a static equivalent of duck typing, which is well known to Python programmers. See PEP 544 for the detailed specification of protocols and structural subtyping in Python.
 Usage
 You can define your own protocol class by inheriting the special Protocol class:
 from typing import Iterable
from typing_extensions import Protocol

class SupportsClose(Protocol):
    # Empty method body (explicit '...')
    def close(self) -> None: ...

class Resource:  # No SupportsClose base class!

    def close(self) -> None:
       self.resource.release()

    # ... other methods ...

def close_all(items: Iterable[SupportsClose]) -> None:
    for item in items:
        item.close()

close_all([Resource(), open('some/file')])  # OK
 Resource is a subtype of the SupportsClose protocol since it defines a compatible close method. Regular file objects returned by open() are similarly compatible with the protocol, as they support close().
 If you want to define a docstring on the method use the next syntax:
     def load(self, filename: Optional[str] = None) -> None:
        """Load a configuration file."""
        ...
 Make protocols work with isinstance To check an instance against the protocol using isinstance, we need to decorate our protocol with @runtime_checkable
 Make a protocol property variable
 Make protocol of functions
 References - Mypy article on protocols - Predefined protocols reference
 
 
 Logql
   New: Compare the values of a metric with the past.
 The offset modifier allows changing the time offset for individual range vectors in a query.
 For example, the following expression counts all the logs within the last ten minutes to five minutes rather than last five minutes for the MySQL job. Note that the offset modifier always needs to follow the range vector selector immediately.
 count_over_time({job="mysql"}[5m] offset 5m) // GOOD
count_over_time({job="mysql"}[5m]) offset 5m // INVALID
 
 
 FastAPI
   New: Launch the server from within python.
 import uvicorn
if __name__ == "__main__":
  uvicorn.run("main:app", host="0.0.0.0", port=8000, reload=True)
 
  New: Add the request time to the logs.
 For more information on changing the logging read 1
 To set the datetime of the requests use this configuration
 @asynccontextmanager
async def lifespan(api: FastAPI):
    logger = logging.getLogger("uvicorn.access")
    console_formatter = uvicorn.logging.ColourizedFormatter(
        "{asctime} {levelprefix} : {message}", style="{", use_colors=True
    )
    logger.handlers[0].setFormatter(console_formatter)
    yield

api = FastAPI(lifespan=lifespan)
 
 
 Graphql
   New: Introduce GraphQL.
 GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.
 To use it with python you can use Ariadne (source)
 
 
 Pytest
  New: Changing the directory when running tests but switching it back after the test ends.
 
 nodejs
   New: Install using nvm.
 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

nvm install 22

node -v # should print `v22.12.0`

npm -v # should print `10.9.0`
 
 
 Python Snippets
   New: Get unique items between two lists.
 If you want all items from the second list that do not appear in the first list you can write:
 x = [1,2,3,4]
f = [1,11,22,33,44,3,4]

result = set(f) - set(x)
 
  New: Pad number with zeros.
 number = 1
print(f"{number:02d}")
 
  New: Parse a datetime from an epoch.
 >>> import datetime
>>> datetime.datetime.fromtimestamp(1347517370).strftime('%c')
  '2012-09-13 02:22:50'
 
  New: Fix variable is unbound pyright error.
 You may receive these warnings if you set variables inside if or try/except blocks such as the next one:
   def x():
    y = True
    if y:
        a = 1
    print(a)  # "a" is possibly unbound
 The easy fix is to set a = None outside those blocks
   def x():
    a = None
    y = True
    if y:
        a = 1
    print(a)  # "a" is possibly unbound
 
  New: Investigate a class attributes.
 Investigate a class attributes with inspect
 
  New: Expire the cache of the lru_cache.
 The lru_cache decorator caches forever, a way to prevent it is by adding one more parameter to your expensive function: ttl_hash=None. This new parameter is so-called "time sensitive hash", its the only purpose is to affect lru_cache. For example:
 from functools import lru_cache
import time

@lru_cache()
def my_expensive_function(a, b, ttl_hash=None):
    del ttl_hash  # to emphasize we don't use it and to shut pylint up
    return a + b  # horrible CPU load...

def get_ttl_hash(seconds=3600):
    """Return the same value withing `seconds` time period"""
    return round(time.time() / seconds)

res = my_expensive_function(2, 2, ttl_hash=get_ttl_hash())
 
  New: Kill a process by it's PID.
 import os
import signal

os.kill(pid, signal.SIGTERM) #or signal.SIGKILL
 
  New: Convert the parameter of an API get request to a valid field.
 For example if the argument has /:
 from urllib.parse import quote

quote("value/with/slashes")
 Will return value%2Fwith%2Fslashes
 
  New: Get the type hints of an object.
 ```python from typing import get_type_hints
 Student(NamedTuple): name: Annotated[str, 'some marker']
 get_type_hints(Student) == {'name': str} get_type_hints(Student, include_extras=False) == {'name': str} get_type_hints(Student, include_extras=True) == { 'name': Annotated[str, 'some marker'] } ````
 
  New: Type hints of a python module.
 from types import ModuleType
import os

assert isinstance(os, ModuleType)
 
  New: Get all the classes of a python module.
 def _load_classes_from_directory(self, directory):
    classes = []
    for file_name in os.listdir(directory):
        if file_name.endswith(".py") and file_name != "__init__.py":
            module_name = os.path.splitext(file_name)[0]
            module_path = os.path.join(directory, file_name)

            # Import the module dynamically
            spec = spec_from_file_location(module_name, module_path)
            if spec is None or spec.loader is None:
                raise ValueError(
                    f"Error loading the spec of {module_name} from {module_path}"
                )
            module = module_from_spec(spec)
            spec.loader.exec_module(module)

            # Retrieve all classes from the module
            module_classes = inspect.getmembers(module, inspect.isclass)
            classes.extend(module_classes)
 
  New: Import files from other directories.
 Add the directory where you have your function to sys.path
 import sys

sys.path.append("**Put here the directory where you have the file with your function**")

from file import function
 
  New: Use Path of pathlib write_text in append mode.
 It's not supported you need to open it:
 with my_path.open("a") as f:
    f.write("...")
 
  New: Suppress ANN401 for dynamically typed *args and **kwargs.
 Use object instead:
 def function(*args: object, **kwargs: object) -> None:
 
  New: One liner conditional.
 To write an if-then-else statement in Python so that it fits on one line you can use:
 fruit = 'Apple'
isApple = True if fruit == 'Apple' else False
 
  New: Get package data relative path.
 If you want to reference files from the foo/package1/resources folder you would want to use the file variable of the module. Inside foo/package1/init.py:
 from os import path
resources_dir = path.join(path.dirname(__file__), 'resources')
 
  New: Compare file and directories.
 The filecmp module defines functions to compare files and directories, with various optional time/correctness trade-offs. For comparing files, see also the difflib module.
 from filecmp import dircmp

def print_diff_files(dcmp):
    for name in dcmp.diff_files:
        print("diff_file %s found in %s and %s" % (name, dcmp.left, dcmp.right))
    for sub_dcmp in dcmp.subdirs.values():
        print_diff_files(sub_dcmp)
dcmp = dircmp('dir1', 'dir2')
print_diff_files(dcmp)
 
  New: Send a linux desktop notification.
 To show a Linux desktop notification from a Python script, you can use the notify2 library (although it's last commit was done on 2017. This library provides an easy way to send desktop notifications on Linux.
 Alternatively, you can use the subprocess module to call the notify-send command-line utility directly. This is a more straightforward method but requires notify-send to be installed.
 import subprocess

def send_notification(title: str, message: str = "", urgency: str = "normal") -> None:
    """Send a desktop notification using notify-send.

    Args:
        title (str): The title of the notification.
        message (str): The message body of the notification. Defaults to an empty string.
        urgency (str): The urgency level of the notification. Can be 'low', 'normal', or 'critical'. Defaults to 'normal'.
    """
    subprocess.run(["notify-send", "-u", urgency, title, message])
 
  New: Get the error string.
 import traceback

def cause_error():
    return 1 / 0  # This will raise a ZeroDivisionError

try:
    cause_error()
except Exception as error:
    # Capture the exception traceback as a string
    error_message = "".join(traceback.format_exception(None, error, error.__traceback__))
    print("An error occurred:\n", error_message)
 
 
 Goodconf
   New: Initialize the config with a default value if the file doesn't exist.
 
    def load(self, filename: Optional[str] = None) -> None:
        self._config_file = filename
        if not self.store_dir.is_dir():
            log.warning("The store directory doesn't exist. Creating it")
            os.makedirs(str(self.store_dir))
        if not Path(self.config_file).is_file():
            log.warning("The yaml store file doesn't exist. Creating it")
            self.save()
        super().load(filename)
 feat(goodconf#Config saving)
 So far goodconf doesn't support saving the config. Until it's ready you can use the next snippet:
 
class YamlStorage(GoodConf):
    """Adapter to store and load information from a yaml file."""

    @property
    def config_file(self) -> str:
        """Return the path to the config file."""
        return str(self._config_file)

    @property
    def store_dir(self) -> Path:
        """Return the path to the store directory."""
        return Path(self.config_file).parent

    def reload(self) -> None:
        """Reload the contents of the authentication store."""
        self.load(self.config_file)

    def load(self, filename: Optional[str] = None) -> None:
        """Load a configuration file."""
        if not filename:
            filename = f"{self.store_dir}/data.yaml"
        super().load(self.config_file)

    def save(self) -> None:
        """Save the contents of the authentication store."""
        with open(self.config_file, "w+", encoding="utf-8") as file_cursor:
            yaml = YAML()
            yaml.default_flow_style = False
            yaml.dump(self.dict(), file_cursor)
 feat(google_chrome#Open a specific profile): Open a specific profile
 google-chrome --profile-directory="Profile Name"
 Where Profile Name is one of the profiles listed under ls ~/.config/chromium | grep -i profile.
 
 
 Inotify
   New: Introduce python_inotify.
 inotify is a python library that acts as a bridge to the inotify linux kernel which allows you to register one or more directories for watching, and to simply block and wait for notification events. This is obviously far more efficient than polling one or more directories to determine if anything has changed.
 Installation:
 pip install inotify
 Basic example using a loop:
 import inotify.adapters

def _main():
    i = inotify.adapters.Inotify()

    i.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    for event in i.event_gen(yield_nones=False):
        (_, type_names, path, filename) = event

        print("PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format(
              path, filename, type_names))

if __name__ == '__main__':
    _main()
 Output:
 PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_MODIFY']
PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_OPEN']
PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_CLOSE_WRITE']
 Basic example without a loop:
 import inotify.adapters

def _main():
    i = inotify.adapters.Inotify()

    i.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    events = i.event_gen(yield_nones=False, timeout_s=1)
    events = list(events)

    print(events)

if __name__ == '__main__':
    _main()
 The wait will be done in the list(events) line
 
  Correction: Deprecate inotify.
 DEPRECATED: As of 2024-11-15 it's been 4 years since the last commit. watchdog has 6.6k stars and last commit was done 2 days ago.
 
 
 watchdog
   New: Introduce watchdog.
 watchdog is a Python library and shell utilities to monitor filesystem events.
 Cons:
  The docs suck.
 
 Installation
 pip install watchdog
 Usage
 A simple program that uses watchdog to monitor directories specified as command-line arguments and logs events generated:
 import time

from watchdog.events import FileSystemEvent, FileSystemEventHandler
from watchdog.observers import Observer

class MyEventHandler(FileSystemEventHandler):
    def on_any_event(self, event: FileSystemEvent) -> None:
        print(event)

event_handler = MyEventHandler()
observer = Observer()
observer.schedule(event_handler, ".", recursive=True)
observer.start()
try:
    while True:
        time.sleep(1)
finally:
    observer.stop()
    observer.join()
 References - Source - Docs
 
 
 Pydantic
   New: Nicely show validation errors.
 A nice way of showing it is to capture the error and print it yourself:
 try:
    model = Model(
        state=state,
    )
except ValidationError as error:
    log.error(f'Error building model with state {state}')
    raise error
 
  New: Load a pydantic model from json.
 You can use the model_validate_json method that will validate and return an object with the loaded data.
 from datetime import date

from pydantic import BaseModel, ConfigDict, ValidationError

class Event(BaseModel):
    model_config = ConfigDict(strict=True)

    when: date
    where: tuple[int, int]

json_data = '{"when": "1987-01-28", "where": [51, -1]}'
print(Event.model_validate_json(json_data))

try:
    Event.model_validate({'when': '1987-01-28', 'where': [51, -1]})

except ValidationError as e:
    print(e)
    """
    2 validation errors for Event
    when
      Input should be a valid date [type=date_type, input_value='1987-01-28', input_type=str]
    where
      Input should be a valid tuple [type=tuple_type, input_value=[51, -1], input_type=list]
    """
 
  New: Create part of the attributes in the initialization stage.
 class Sqlite(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)

    path: Path
    db: sqlite3.Cursor

    def __init__(self, **kwargs):
        conn = sqlite3.connect(kwargs['path'])
        kwargs['db'] = conn.cursor()
        super().__init__(**kwargs)
 
 
 psycopg2
   New: Introduce psycopg2.
 Installation
 Install the dependencies:
 sudo apt install libpq-dev python3-dev
 Then install the package
 pip install psycopg2
 
 
 questionary
   New: Unit testing questionary code.
 Testing questionary code can be challenging because it involves interactive prompts that expect user input. However, there are ways to automate the testing process. You can use libraries like pexpect, pytest, and pytest-mock to simulate user input and test the behavior of your code.
 Here’s how you can approach testing questionary code using pytest-mock to mock questionary functions
 You can mock questionary functions like questionary.select().ask() to simulate user choices without actual user interaction.
 Testing a single questionary.text prompt
 Let's assume you have a function that asks the user for their name:
 import questionary

def ask_name() -> str:
    name = questionary.text("What's your name?").ask()
    return name
 You can test this function by mocking the questionary.text prompt to simulate the user's input.
 import pytest
from your_module import ask_name

def test_ask_name(mocker):
    # Mock the text function to simulate user input
    mock_text = mocker.patch('questionary.text')

    # Define the response for the prompt
    mock_text.return_value.ask.return_value = "Alice"

    result = ask_name()

    assert result == "Alice"
 Test a function that has many questions
 Here’s an example of how to test a function that contains two questionary.text prompts using pytest-mock.
 Let's assume you have a function that asks for the first and last names of a user:
 import questionary

def ask_full_name() -> dict:
    first_name = questionary.text("What's your first name?").ask()
    last_name = questionary.text("What's your last name?").ask()
    return {"first_name": first_name, "last_name": last_name}
 You can mock both questionary.text calls to simulate user input for both the first and last names:
 import pytest
from your_module import ask_full_name

def test_ask_full_name(mocker):
    # Mock the text function for the first name prompt
    mock_text_first = mocker.patch('questionary.text')
    # Define the response for the first name prompt
    mock_text_first.side_effect = ["Alice", "Smith"]

    result = ask_full_name()

    assert result == {"first_name": "Alice", "last_name": "Smith"}
 
 
 rich
   New: Adding a footer to a table.
 Adding a footer is not easy task. This answer doesn't work anymore as table doesn't have the add_footer. You need to create the footer in the add_column so you need to have the data that needs to go to the footer before building the rows.
 You would do something like:
 table = Table(title="Star Wars Movies", show_footer=True)
table.add_column("Title", style="magenta", footer='2342')
 
 
 Coding tools
 Singer
   New: Introduce neotree.
 General keymaps:
  <cr>: Open the file in the current buffer
 <s>: Open in a vertical split
 <S>: Open in an horizontal split
 <bs>: Navigate one directory up (even if it's the root of the cwd)
 
 File and directory management:
  a: Create a new file or directory. Add a / to the end of the name to make a directory.
 d: Delete the selected file or directory
 r: Rename the selected file or directory
 y: Mark file to be copied (supports visual selection)
 x: Mark file to be cut (supports visual selection)
 m: Move the selected file or directory
 c: Copy the selected file or directory
 
 References:
  Docs
 Wiki
 Wiki Recipes
 
 
  New: Show hidden files.
 return {
  "nvim-neo-tree/neo-tree.nvim",
  opts = {
    filesystem = {
      filtered_items = {
        visible = true,
        show_hidden_count = true,
        hide_dotfiles = false,
        hide_gitignored = true,
        hide_by_name = {
          '.git',
        },
        never_show = {},
      },
    }
  }
}
 
  New: Autoclose on open file.
 This example uses the file_open event to close the Neo-tree window when a file is opened. This applies to all windows and all sources at once.
 require("neo-tree").setup({
  event_handlers = {

    {
      event = "file_opened",
      handler = function(file_path)
        -- auto close
        -- vimc.cmd("Neotree close")
        -- OR
        require("neo-tree.command").execute({ action = "close" })
      end
    },

  }
})
 
  New: Configuring vim folds.
 Copy the code under implementation in your config file.
 
  New: Can't copy file/directory to itself.
 If you want to copy a directory you need to assume that the prompt is done from within the directory. So if you want to copy it to a new name at the same level you need to use ../new-name instead of new-name.
 
  New: Introduce the vim foldings workflow.
 ne way to easily work with folds is by using the fold-cycle plugin to be able to press <tab> or <enter> to toggle a fold.
 If you're using lazyvim you can use the next configuration:
 return {
  {
    "jghauser/fold-cycle.nvim",
    config = function()
      require("fold-cycle").setup()
    end,
    keys = {
      {
        "<tab>",
        function()
          return require("fold-cycle").open()
        end,
        desc = "Fold-cycle: open folds",
        silent = true,
      },
      {
        "<cr>",
        function()
          return require("fold-cycle").open()
        end,
        desc = "Fold-cycle: open folds",
        silent = true,
      },
      {
        "<s-tab>",
        function()
          return require("fold-cycle").close()
        end,
        desc = "Fold-cycle: close folds",
        silent = true,
      },
      {
        "zC",
        function()
          return require("fold-cycle").close_all()
        end,
        remap = true,
        silent = true,
        desc = "Fold-cycle: close all folds",
      },
    },
  },
}
 
  New: Introduce singer.
 Singer is an open-source standard for writing scripts that move data.
 It describes how data extraction scripts—called “taps” —and data loading scripts—called “targets”— should communicate, allowing them to be used in any combination to move data from any source to any destination. Send data between databases, web APIs, files, queues, and just about anything else you can think of.
 It has many "taps" and "targets" that can help you interact with third party tools without needing to write the code.
 References - Home
 
  New:  is not well mapped.
 It's because <c-i> is a synonym of <TAB>.
 
 
 Coding with AI
   New: Introduce LazyVim.
  Source
 Docs
 Home
 
 
  New: Adding plugins configuration.
 Configuring LazyVim plugins is exactly the same as using lazy.nvim to build a config from scratch.
 For the full plugin spec documentation please check the lazy.nvim readme.
 LazyVim comes with a list of preconfigured plugins, check them here before diving on your own.
 
  New: Adding a plugin.
 Adding a plugin is as simple as adding the plugin spec to one of the files under `lua/plugins/*.lua``. You can create as many files there as you want.
 You can structure your lua/plugins`` folder with a file per plugin, or a separate file containing all the plugin specs for some functionality. For example:lua/plugins/lsp.lua`
 return {
  -- add symbols-outline
  {
    "simrat39/symbols-outline.nvim",
    cmd = "SymbolsOutline",
    keys = { { "<leader>cs", "<cmd>SymbolsOutline<cr>", desc = "Symbols Outline" } },
    opts = {
      -- add your options that should be passed to the setup() function here
      position = "right",
    },
  },
}
 Customizing plugin specs. Defaults merging rules:
  cmd: the list of commands will be extended with your custom commands
 event: the list of events will be extended with your custom events
 ft: the list of filetypes will be extended with your custom filetypes
 keys: the list of keymaps will be extended with your custom keymaps
 opts: your custom opts will be merged with the default opts
 dependencies: the list of dependencies will be extended with your custom dependencies
 any other property will override the defaults
 
 For ft, event, keys, cmd and opts you can instead also specify a values function that can make changes to the default values, or return new values to be used instead.
 -- change trouble config
{
  "folke/trouble.nvim",
  -- opts will be merged with the parent spec
  opts = { use_diagnostic_signs = true },
}

-- add cmp-emoji
{
  "hrsh7th/nvim-cmp",
  dependencies = { "hrsh7th/cmp-emoji" },
  ---@param opts cmp.ConfigSchema
  opts = function(_, opts)
    table.insert(opts.sources, { name = "emoji" })
  end,
}
 Defining the plugin keymaps:
 Adding keys= follows the rules as explained above. You don't have to specify a mode for normal mode keymaps.
 You can also disable a default keymap by setting it to false. To override a keymap, simply add one with the same lhs and a new rhs. For example lua/plugins/telescope.lua
 return {
  "nvim-telescope/telescope.nvim",
  keys = {
    -- disable the keymap to grep files
    {"<leader>/", false},
    -- change a keymap
    { "<leader>ff", "<cmd>Telescope find_files<cr>", desc = "Find Files" },
    -- add a keymap to browse plugin files
    {
      "<leader>fp",
      function() require("telescope.builtin").find_files({ cwd = require("lazy.core.config").options.root }) end,
      desc = "Find Plugin File",
    },
  },
},
 Make sure to use the exact same mode as the keymap you want to disable.
 
return {
  "folke/flash.nvim",
  keys = {
    -- disable the default flash keymap
    { "s", mode = { "n", "x", "o" }, false },
  },
}
 You can also return a whole new set of keymaps to be used instead. Or return {} to disable all keymaps for a plugin.
 return {
  "nvim-telescope/telescope.nvim",
  -- replace all Telescope keymaps with only one mapping
  keys = function()
    return {
      { "<leader>ff", "<cmd>Telescope find_files<cr>", desc = "Find Files" },
    }
  end,
},
 
  New: Auto update plugins.
 Add this to ~/.config/nvim/lua/config/autocomds.lua
 local function augroup(name)
  return vim.api.nvim_create_augroup("lazyvim_" .. name, { clear = true })
end

vim.api.nvim_create_autocmd("VimEnter", {
  group = augroup("autoupdate"),
  callback = function()
    if require("lazy.status").has_updates then
      require("lazy").update({ show = false })
    end
  end,
})
 
  New: Introduce vim keymaps.
 LazyVim comes with some sane default keybindings, you can see them here. You don't need to remember them all, it also comes with which-key to help you remember your keymaps. Just press any key like  and you'll see a popup with all possible keymaps starting with .
  default <leader> is <space>
 default <localleader> is \
 
 General editor bindings:
  Save file: <C-s>
 Quit all: <leader>qq
 Open a floating terminal: <C-/>
 
 Movement keybindings:
  Split the windows:
 Vertically: <C-|
 Horizontally: <C--
 Delete window: <leader>wd
 To move around the windows you can use: , , , .
 To resize the windows use: , , , 
 To move between buffers:
 Next and previous with , 
 Switch to the previously opened buffer: <leader>bb
 
 Coding keybindings:
 Diagnostics:
  <leader>cd>: Shows you the diagnostics message of the current line in a floating window
 ]d and [d: iterates over all diagnostics
 ]e and [e: iterates over all error diagnostics
 ]w and [w: iterates over all warning diagnostics
 
 
  New: Setting keymaps in lua.
 If you need to set keymaps in lua you can use vim.keymap.set. For example:
 vim.keymap.set('n', '<space>w', '<cmd>write<cr>', {desc = 'Save'})
 After executing this, the sequence Space + w will call the write command. Basically, we can save changes made to a file with Space + w.
 Let's dive into what does the vim.keymap.set parameters mean.
 vim.keymap.set({mode}, {lhs}, {rhs}, {opts})
  {mode}: mode where the keybinding should execute. It can be a list of modes. We need to speify the mode's short name. Here are some of the most common.
 n: Normal mode.
 i: Insert mode.
 x: Visual mode.
 s: Selection mode.
 v: Visual + Selection.
 t: Terminal mode.
 o: Operator-pending.
  '': Yes, an empty string. Is the equivalent of n + v + o.
 
  {lhs}: is the key we want to bind.
 
 {rhs} is the action we want to execute. It can be a string with a command or an expression. You can also provide a lua function.
  {opts} this must be a lua table. If you don't know what is a "lua table" just think is a way of storing several values in one place. Anyway, it can have these properties.
 
  desc: A string that describes what the keybinding does. You can write anything you want.
 
 remap: A boolean that determines if our keybinding can be recursive. The default value is false. Recursive keybindings can cause some conflicts if used incorrectly. Don't enable it unless you know what you're doing.
 buffer: It can be a boolean or a number. If we assign the boolean true it means the keybinding will only be effective in the current file. If we assign a number, it needs to be the "id" of an open buffer.
 silent: A boolean. Determines whether or not the keybindings can show a message. The default value is false.
 expr: A boolean. If enabled it gives the chance to use vimscript or lua to calculate the value of {rhs}. The default value is false.
 
 
  New: The leader key.
 When creating keybindings we can use the special sequence <leader> in the {lhs} parameter, it'll take the value of the global variable mapleader.
 So mapleader is a global variable in vimscript that can be string. For example.
 vim.g.mapleader = ' '
 After defining it we can use it as a prefix in our keybindings.
 vim.keymap.set('n', '<leader>w', '<cmd>write<cr>')
 This will make <space key> + w save the current file.
 There are different opinions on what key to use as the <leader> key. The <space> is the most comfortable as it's always close to your thumbs, and it works well with both hands. Nevertheless, you can only use it in normal mode, because in insert <space><whatever> will be triggered as you write. An alternative is to use ; which is also comfortable (if you use the english key distribution) and you can use it in insert mode.
 If you want to define more than one leader key you can either:
  Change the mapleader many times in your file: As the value of mapleader is used at the moment the mapping is defined, you can indeed change that while plugins are loading. For that, you have to explicitly :runtime the plugins in your ~/.vimrc (and count on the canonical include guard to prevent redefinition later):
 
 
let mapleader = ','
runtime! plugin/NERD_commenter.vim
runtime! ...
let mapleader = '\'
runime! plugin/mark.vim
...
 * Use the keys directly instead of using <leader>
 " editing mappings
nnoremap ,a <something>
nnoremap ,k <something else>
nnoremap ,d <and something else>

" window management mappings
nnoremap gw <something>
nnoremap gb <something else>
 Defining mapleader and/or using <leader> may be useful if you change your mind often on what key to use a leader but it won't be of any use if your mappings are stable.
 
  New: Configure vim from scratch.
 Neovim configuration is a complex thing to do, both to start and to maintain. The configurations are endless, the plugins are too. Be ready to spend a lot of energy on it and to get lost reading a lot.
 If I'm scaring you, you are right to be scared! xD. Once you manage to get it configured to your liking you'll think that in the end it doesn't even matter spending all that time. However if you're searching for something that is plug and play try vscodium.
 To make things worse, the configuration is done in lua, so you may need a small refreshment to understand what are you doing.
 
  New: Vim distributions.
 One way to make vim's configuration more bearable is to use vim distributions. These are projects that maintain configurations with sane defaults and that work with the whole ecosystem of plugins.
 Using them is the best way to:
  Have something usable fast
 Minimize the maintenance efforts as others are doing it for you (plugin changes, breaking changes, ..)
 Keep updated with the neovim ecosystem, as you can see what is the community adding to the default config.
 
 However, there are so many good Neovim configuration distributions that it becomes difficult for a Neovim user to decide which distribution to use and how to tailor it for their use case.
 By far, the top 5 Neovim configuration distributions are AstroNvim, kickstart, LazyVim, LunarVim, and NvChad. That is not to say these are the “best” configuration distributions, simply that they are the most popular.
 Each of these configuration distributions has value. They all provide excellent starting points for crafting your own custom configuration, they are all extensible and fairly easy to learn, and they all provide an out-of-the-box setup that can be used effectively without modification.
 Distinguishing features of the top Neovim configuration distributions are:
   AstroNvim:
  An excellent community repository
 Fully featured out-of-the-box
 Good documentation
 
 
  kickstart
  Minimal out-of-the-box setup
 Easy to extend and widely used as a starting point
 A good choice if your goal is hand-crafting your own config
 
 
  LazyVim
  Very well maintained by the author of lazy.nvim
 Nice architecture, it’s a plugin with which you can import preconfigured plugins
 Good documentation
 
 
  LunarVim
  Well maintained and mature
 Custom installation processs installs LunarVim in an isolated location
 Been around a while, large community, widespread presence on the web
 
 
  NvChad
  Really great base46 plugin enables easy theme/colorscheme management
 Includes an impressive mappings cheatsheet
 ui plugin and nvim-colorizer
 
 
 
 Personally I tried LunarVim and finally ended up with LazyVim because:
  It's more popular
 I like the file structure
 It's being maintained by folke one of the best developers of neovim plugins.
 
 
  New: Starting your configuration with LazyVim.
 Installing the requirements:
 LazyVim needs the next tools to be able to work:
  Neovim >= 0.9.0 (needs to be built with LuaJIT). Follow these instructions
 Git >= 2.19.0 (for partial clones support). sudo apt-get install git.
 a Nerd Font(v3.0 or greater) (optional, but strongly suggested as they rae needed to display some icons). Follow these instructions if you're using kitty.
 lazygit (optional and I didn't like it)
 a C compiler for nvim-treesitter. apt-get install gcc
 for telescope.nvim (optional)
 live grep: ripgrep
 find files: fd
 a terminal that support true color and undercurl:
 kitty (Linux & Macos)
 wezterm (Linux, Macos & Windows)
 alacritty (Linux, Macos & Windows)
 iterm2 (Macos)
 
 Install the starter:
  Make a backup of your current Neovim files: # required
mv ~/.config/nvim{,.old}

# optional but recommended
mv ~/.local/share/nvim{,.old}
mv ~/.local/state/nvim{,.old}
mv ~/.cache/nvim{,.old}
  Clone the starter
 git clone https://github.com/LazyVim/starter ~/.config/nvim
 
  Remove the .git folder, so you can add it to your own repo later
 rm -rf ~/.config/nvim/.git
 
  Start Neovim!
 
nvim
 - It is recommended to run :LazyHealth after installation. This will load all plugins and check if everything is working correctly.
 
 
 Understanding the file structure:
 The files under config will be automatically loaded at the appropriate time, so you don't need to require those files manually.
 You can add your custom plugin specs under lua/plugins/. All files there will be automatically loaded by lazy.nvim.
 
~/.config/nvim
├── lua
│   ├── config
│   │   ├── autocmds.lua
│   │   ├── keymaps.lua
│   │   ├── lazy.lua
│   │   └── options.lua
│   └── plugins
│       ├── spec1.lua
│       ├── **
│       └── spec2.lua
└── init.toml
 The files autocmds.lua, keymaps.lua, lazy.lua and options.lua under lua/config will be automatically loaded at the appropriate time, so you don't need to require those files manually. LazyVim comes with a set of default config files that will be loaded before your own.
 You can continue your config by adding plugins.
 
  New: Introduce the vim movement workflow.
 Moving around vim can be done in many ways, which an lead to being lost on how to do it well.
 LazyVim has a very nice way to deal with buffers - Use H and L if the buffer you want to go to is visually close to where you are. - Otherwise, if the buffer is open, use <leader>, - For other files, use <leader><space> - Close buffers you no longer need with <leader>bd - <leader>ss to quickly jump to a function in the buffer you're on - Using the jump list with <c-o>, <c-i> and gd to navigate the code - You can pin buffers with <leader>bp and delete all non pinned buffers with <leader>bP
 
  New: Using the jump list.
 Vim has a feature called the “Jump List”, which saves all the locations you’ve recently visited, including their line number, column number, and what else not in the .viminfo file, to help you get exactly the position you were last in. Not only does it save the locations in your current buffer, but also previous buffers you may have edited in other Vim sessions. Which means, if you’re currently working on a file, and there aren’t many last-location saves in this one, you’ll be redirected to the previous file you had edited. But how do you do that? Simply press Ctrl + O, and it’ll get you back to the previous location you were in, or more specifically, your cursor was in.
 If you want to go back to the newer positions, after you’re done with what you wanted to do, you can then press Ctrl + i to go back to the newer position. This is exceptionally useful when you’re working with a lot of project files at a time, and you need to go back and forth between multiple blocks in different files. This could instantly give you a boost, as you won’t need to have separate buffers opened up or windows to be setted up, you can simply jump between the files and edit them.
 Ctrl + O is probably not meant for a single task, as far as Vim’s philosophy is concerned. The jumping mentioned in the previous section only works when you’re in Normal Mode, and not when you’re in Insert Mode. When you press Ctrl + O in Insert Mode, what happens instead is that you’ll enter Normal Mode, and be able to execute a single command, after which Vim will automatically switch back to Insert Mode.
 
  New: Install using Lazyvim.
 return {
  {
    "sindrets/diffview.nvim",
    dependencies = {
      { "nvim-tree/nvim-web-devicons", lazy = true },
    },

    keys = {
      {
        "dv",
        function()
          if next(require("diffview.lib").views) == nil then
            vim.cmd("DiffviewOpen")
          else
            vim.cmd("DiffviewClose")
          end
        end,
        desc = "Toggle Diffview window",
      },
    },
  },
}
 Which sets the next bindings: - dv: Toggle the opening and closing of the diffview windows
 
  New: Use diffview as merge tool.
 Add to your ~/.gitconfig:
 [alias]
  mergetool = "!nvim -c DiffviewOpen"
 
  New: Resolve merge conflicts.
 If you call :DiffviewOpen during a merge or a rebase, the view will list the conflicted files in their own section. When opening a conflicted file, it will open in a 3-way diff allowing you to resolve the conflict with the context of the target branch's version (OURS, left), and the version from the branch which is being merged (THEIRS, right).
 The conflicted file's entry in the file panel will show the remaining number of conflict markers (the number following the file name). If what follows the file name is instead an exclamation mark (!), this indicates that the file has not yet been opened, and the number of conflicts is unknown. If the sign is a check mark, this indicates that there are no more conflicts in the file.
 You can interact with the merge tool with the next bindings:
  ]x and [x: Jump between conflict markers. This works from the file panel as well.
 dp: Put the contents on the other buffer
 do: Get the contents from the other buffer
 2do: to obtain the hunk from the OURS side of the diff
 3do to obtain the hunk from the THEIRS side of the diff
 1do to obtain the hunk from the BASE in a 4-way diff
 
 Additionally there are mappings for operating directly on the conflict markers:
  <leader>co: Choose the OURS version of the conflict.
 <leader>ct: Choose the THEIRS version of the conflict.
 <leader>cb: Choose the BASE version of the conflict.
 <leader>ca: Choose all versions of the conflict (effectively just deletes the markers, leaving all the content).
 dx: Choose none of the versions of the conflict (delete the conflict region).
 
 
  New: Introduce ai coding prompts.
 These are some useful AI prompts to help you while you code:
  create a function with type hints and docstring using google style called { } that { }
 create the tests for the function { } adding type hints and following the AAA style where the Act section is represented contains a returns = (thing to test) line or if the function to test doesn't return any value append an # act comment at the end of the line. Use paragraphs to separate the AAA blocks and don't add comments inside the tests for the sections
 
 If you use espanso you can simplify the filling up of these prompts on the AI chats. For example:
 ---
matches:
  - trigger: :function
    form: |
      Create a function with type hints and docstring using google style called [[name]] that:
      [[text]]
    form_fields:
      text:
        multiline: true
  - trigger: :tweak
    form: |
      Tweak the next code:
      [[code]]

      So that:

      [[text]]
    form_fields:
      text:
        multiline: true
      code:
        multiline: true
  - trigger: :test
    form: |
      create the tests for the function:
      [[text]]

      Following the next guidelines:

      - Add type hints
      - Follow the AAA style
      - In the Act section if the function to test returns a value always name that variable returns. If the function to test doesn't return any value append an # act comment at the end of the line.
      - Use paragraphs to separate the AAA blocks and don't add comments like # Arrange or # Act or # Act/Assert or # Assert

    form_fields:
      text:
        multiline: true
  - trigger: :refactor
    form: |
     Refactor the next code
     [[code]]
     with the next conditions
     [[conditions]]
    form_fields:
      code:
        multiline: true
      conditions:
        multiline: true
 
  New: Introduce Kestra.
 Kestra is an open-source orchestrator designed to bring Infrastructure as Code (IaC) best practices to all workflows — from those orchestrating mission-critical operations, business processes, and data pipelines to simple Zapier-style automation. Built with an API-first philosophy, Kestra enables users to define and manage data pipelines through a simple YAML configuration file. This approach frees you from being tied to a specific client implementation, allowing for greater flexibility and easier integration with various tools and services.
 Look at this 4 minute video for a visual introduction
 References - Docs - Home - 4 minute introduction video
 
  New: Add new prompts for developers.
   - trigger: :polish
    form: |
     Polish the next code
     [[code]]
     with the next conditions:
     - Use type hints on all functions and methods
     - Add or update the docstring using google style on all functions and methods
    form_fields:
      code:
        multiline: true
  - trigger: :commit
    form: |
     Act as an expert developer. Create a message commit with the next conditions:
     - follow semantic versioning
     - create a semantic version comment per change
     - include all comments in a raw code block so that it's easy to copy

     for the following diff
     [[text]]
    form_fields:
      text:
        multiline: true
 
  Correction: Update the ai prompts.
 ```yaml matches: - trigger: :function form: | Create a function with: - type hints - docstrings for all classes, functions and methods - docstring using google style with line length less than 89 characters - adding logging traces using the log variable log = logging.getLogger(name) - Use fstrings instead of %s - If you need to open or write a file always set the encoding to utf8 - If possible add an example in the docstring - Just give the code, don't explain anything
   Called [[name]] that:
  [[text]]
form_fields:
  text:
    multiline: true
   trigger: :class form: | Create a class with:
  type hints
 docstring using google style with line length less than 89 characters
 use docstrings on the class and each methods
 adding logging traces using the log variable log = logging.getLogger(name)
 Use fstrings instead of %s
 If you need to open or write a file always set the encoding to utf8
 If possible add an example in the docstring
 Just give the code, don't explain anything
 
 Called [[name]] that: [[text]] form_fields: text: - trigger: :class form: | ... - Use paragraphs to separate the AAA blocks and don't add comments like # Arrange or # Act or # Act/Assert or # Assert. So the test will only have black lines between sections - In the Act section if the function to test returns a value always name that variable result. If the function to test doesn't return any value append an # act comment at the end of the line. - If the test uses a pytest.raises there is no need to add the # act comment - Don't use mocks - Use fstrings instead of %s - Gather all tests over the same function on a common class - If you need to open or write a file always set the encoding to utf8 - Just give the code, don't explain anything
 form_fields: text: - trigger: :polish form: | ... - Add or update the docstring using google style on all classes, functions and methods - Wrap the docstring lines so they are smaller than 89 characters - All docstrings must start in the same line as the """ - Add logging traces using the log variable log = logging.getLogger(name) - Use f-strings instead of %s - Just give the code, don't explain anything form_fields: code: multiline: true - trigger: :text form: | Polish the next text by:
  Summarising each section without losing relevant data
 Tweak the markdown format
 Improve the wording
 
 [[text]] form_fields: text: multiline: true
 
  trigger: :readme form: | Create the README.md taking into account:
  Use GPLv3 for the license
 Add Lyz as the author
 Add an installation section
 Add an usage section
 
 of: [[text]]
 form_fields: text: multiline: true ``` feat(aleph#Get all documents of a collection): Get all documents of a collection
 
 
 list_aleph_collection_documents.py is a Python script designed to interact with an API to retrieve and analyze documents from specified collections. It offers a command-line interface (CLI) to list and check documents within a specified collection.
 Features
  Retrieve documents from a specified collection.
 Analyze document processing statuses and warn if any are not marked as successful.
 Return a list of filenames from the retrieved documents.
 Supports verbose output for detailed logging.
 Environment variable support for API key management.
 
 Installation
 To install the required dependencies, use pip:
 pip install typer requests
 Ensure you have Python 3.6 or higher installed.
 Create the file list_aleph_collection_documents.py with the next contents:
 import logging
import requests
from typing import List, Dict, Any, Optional
import logging
import typer
from typing import List, Dict, Any

log = logging.getLogger(__name__)
app = typer.Typer()

@app.command()
def get_documents(
    collection_name: str = typer.Argument(...),
    api_key: Optional[str] = typer.Option(None, envvar="API_KEY"),
    base_url: str = typer.Option("https://your.aleph.org"),
    verbose: bool = typer.Option(
        False, "--verbose", "-v", help="Enable verbose output"
    ),
):
    """CLI command to retrieve documents from a specified collection."""
    if verbose:
        logging.basicConfig(level=logging.DEBUG)
        log.debug("Verbose mode enabled.")
    else:
        logging.basicConfig(level=logging.INFO)
    if api_key is None:
        log.error(
            "Please specify your api key either through the --api-key argument "
            "or through the API_KEY environment variable"
        )
        raise typer.Exit(code=1)
    try:
        documents = list_collection_documents(api_key, base_url, collection_name)
        filenames = check_documents(documents)
        if filenames:
            print("\n".join(filenames))
        else:
            log.warning("No documents found.")
    except Exception as e:
        log.error(f"Failed to retrieve documents: {e}")
        raise typer.Exit(code=1)

def list_collection_documents(
    api_key: str, base_url: str, collection_name: str
) -> List[Dict[str, Any]]:
    """
    Retrieve documents from a specified collection using pagination.

    Args:
        api_key (str): The API key for authentication.
        base_url (str): The base URL of the API.
        collection_name (str): The name of the collection to retrieve documents from.

    Returns:
        List[Dict[str, Any]]: A list of documents from the specified collection.

    Example:
        >>> docs = list_collection_documents("your_api_key", "https://api.example.com", "my_collection")
        >>> print(len(docs))
        1000
    """
    headers = {
        "Authorization": f"ApiKey {api_key}",
        "Accept": "application/json",
        "Content-Type": "application/json",
    }

    collections_url = f"{base_url}/api/2/collections"
    documents_url = f"{base_url}/api/2/entities"
    log.debug(f"Requesting collections list from {collections_url}")
    collections = []
    params = {"limit": 300}

    while True:
        response = requests.get(collections_url, headers=headers, params=params)
        response.raise_for_status()
        data = response.json()
        collections.extend(data["results"])
        log.debug(
            f"Fetched {len(data['results'])} collections, "
            f"page {data['page']} of {data['pages']}"
        )
        if not data["next"]:
            break
        params["offset"] = params.get("offset", 0) + data["limit"]

    collection_id = next(
        (c["id"] for c in collections if c["label"] == collection_name), None
    )
    if not collection_id:
        log.error(f"Collection {collection_name} not found.")
        return []

    log.info(f"Found collection '{collection_name}' with ID {collection_id}")

    documents = []
    params = {
        "q": "",
        "filter:collection_id": collection_id,
        "filter:schemata": "Document",
        "limit": 300,
    }

    while True:
        log.debug(f"Requesting documents from collection {collection_id}")
        response = requests.get(documents_url, headers=headers, params=params)
        response.raise_for_status()
        data = response.json()
        documents.extend(data["results"])
        log.info(
            f"Fetched {len(data['results'])} documents, "
            f"page {data['page']} of {data['pages']}"
        )
        if not data["next"]:
            break
        params["offset"] = params.get("offset", 0) + data["limit"]

    log.info(f"Retrieved {len(documents)} documents from collection {collection_name}")

    return documents

def check_documents(documents: List[Dict[str, Any]]) -> List[str]:
    """Analyze the processing status of documents and return a list of filenames.

    Args:
        documents (List[Dict[str, Any]]): A list of documents in JSON format.

    Returns:
        List[str]: A list of filenames from documents with a successful processing status.

    Raises:
        None, but logs warnings if a document's processing status is not 'success'.

    Example:
        >>> docs = [{"properties": {"processingStatus": ["success"], "fileName": ["file1.txt"]}},
        >>>         {"properties": {"processingStatus": ["failed"], "fileName": ["file2.txt"]}}]
        >>> filenames = check_documents(docs)
        >>> print(filenames)
        ['file1.txt']
    """
    filenames = []

    for doc in documents:
        status = doc.get("properties", {}).get("processingStatus")[0]
        filename = doc.get("properties", {}).get("fileName")[0]

        if status != "success":
            log.warning(
                f"Document with filename {filename} has processing status: {status}"
            )

        if filename:
            filenames.append(filename)

    log.debug(f"Collected filenames: {filenames}")
    return filenames

if __name__ == "__main__":
    app()
 Get your API key
 By default, any Aleph search will return only public documents in responses to API requests.
 If you want to access documents which are not marked public, you will need to sign into the tool. This can be done through the use on an API key. The API key for any account can be found by clicking on the "Settings" menu item in the navigation menu.
 Usage
 You can run the script directly from the command line. Below are examples of usage:
 Retrieve and list documents from a collection:
 python list_aleph_collection_documents.py --api-key "your-api-key" 'Name of your collection'
 Using an Environment Variable for the API Key
 This is better from a security perspective. 
export API_KEY=your_api_key
python list_aleph_collection_documents.py 'Name of your collection'
 Enabling Verbose Logging
 To enable detailed debug logs, use the --verbose or -v flag:
 python list_aleph_collection_documents.py -v 'Name of your collection'
 Getting help
 python list_aleph_collection_documents.py --help
 
 
 memorious
   New: Switch to the previous opened buffer.
 Often the buffer that you want to edit is the buffer that you have just left. Vim provides a couple of convenient commands to switch back to the previous buffer. These are <C-^> (or <C-6>) and :b#. All of them are inconvenient so I use the next mapping:
 nnoremap <Leader><Tab> :b#<CR>
 
  New: Troubleshoot Undefined global vim warning.
 Added to my lua/plugins directory:
   {
    "neovim/nvim-lspconfig",
    opts = {
      servers = {
        lua_ls = {
          settings = {
            Lua = {
              diagnostics = {
                globals = { "vim" },
              },
            },
          },
        },
      },
    },
  },
 
  New: Get the version of the packages installed by Packer.
 Go into the plugin directory cd ~/.local/share/nvim/site/pack/packer/start/your-plugin and check it with git
 
  New: Introduce memorious.
 Memorious is a light-weight web scraping toolkit. It supports scrapers that collect structured or un-structured data. This includes the following use cases:
  Make crawlers modular and simple tasks re-usable
 Provide utility functions to do common tasks such as data storage, HTTP session management
 Integrate crawlers with the Aleph and FollowTheMoney ecosystem
 
 References
  Memorious
 
 
 
 Data orchestrators
   Correction: Update disable regular login with oauth.
 The last signin_inner.tmpl failed with the latest version. I've uploaded the new working one.
 
  New: Configure vim to work with markdown.
 Markdown specific plugins:
  mkdnflow looks awesome.
 
 
  New: Enable folds.
 If you have set the foldmethod to indent by default you won't be able to use folds in markdown.
 To fix this you can create the next autocommand (in lua/config/autocmds.lua if you're using lazyvim).
 vim.api.nvim_create_autocmd("FileType", {
  pattern = "markdown",
  callback = function()
    vim.wo.foldmethod = "expr"
    vim.wo.foldexpr = "v:lua.vim.treesitter.foldexpr()"
  end,
})
 
  New: Aligning tables in markdown.
 In the past I used Tabular but it doesn't work with the latest neovim and the project didn't have any update in the last 5 years.
 A good way to achieve this without installing any plugin is to:
  select the table, including the header and footer lines (with shift V, for example).
 Prettify the table with :!column -t -s '|' -o '|'
 
 If you don't want to remember that command you can bind it to a key:
 vim.keymap.set("v", "<leader>tf", "!column -t -s '|' -o '|'<cr>", { desc = "Format table" })
 How the hell this works?
  shift V switches to Visual mode linewise. This is to select all the lines of the table.
 : switches to Command line mode, to type commands.
 ! specifies a filter command. This means we will send data to a command to modify it (or to filter) and replace the original lines. In this case we are in Visual mode, we defined the input text (the selected lines) and we will use an external command to modify the data.
 column is the filter command we are using, from the util-linux package. column’s purpose is to “columnate”. The -t flag tells column to use the Table mode. The -s flag specifies the delimiters in the input data (the default is whitespace). And the -o flag is to specify the output delimiter to use (we need that because the default is two whitespaces).
 
 
  New: Fix Server does not allow request for unadvertised object error.
 Fetching the whole history with fetch-depth: 0 worked for us:
 - name: Checkout the codebase
  uses: actions/checkout@v3
  with:
    fetch-depth: 0
 
  New: Introduce data orchestrators.
 Data orchestration is the process of moving siloed data from multiple storage locations into a centralized repository where it can then be combined, cleaned, and enriched for activation.
 Data orchestrators are web applications that make this process easy. The most popular right now are:
  Apache Airflow
 Kestra
 Prefect
 
 There are several comparison pages:
  Geek Culture comparison
 Kestra's comparison to Airflow
 Kestra's comparison to Prefect
 
 When looking at the return on investment when choosing an orchestration tool, there are several points to consider:
  Time of installation/maintenance
 Time to write pipeline
 Time to execute (performance)
 
 Kestra
 Pros:
  Easier to write pipelines
 Nice looking web UI
 It has a terraform provider
 Prometheus and grafana integration
 
 Cons:
  Built in Java, so extending it might be difficult
 Plugins are made in Java
 
 Kestra offers a higher ROI globally compared to Airflow:
  Installing Kestra is easier than Airflow; it doesn’t require Python dependencies, and it comes with a ready-to-use docker-compose file using few services and without the need to understand what’s an executor to run task in parallel.
 Creating pipelines with Kestra is simple, thanks to its syntax. You don’t need knowledge of a specific programming language because Kestra is designed to be agnostic. The declarative YAML design makes Kestra flows more readable compared to Airflow’s DAG equivalent, allowing developers to significantly reduce development time.
 In this benchmark, Kestra demonstrates better execution time than Airflow under any configuration setup.
 
 
 
 Scrapers
   New: Introduce Debug Adapter Protocol.
 nvim-dap](https://github.com/mfussenegger/nvim-dap) implements a client for the Debug Adapter Protocol. This allows a client to control a debugger over a documented API. That allows us to control the debugger from inside neovim, being able to set breakpoints, evaluate runtime values of variables, and much more.
 nvim-dap is not configured for any language by default. You will need to set up a configuration for each language. For the configurations you will need adapters to run.
 I would suggest starting with 2 actions. Setting breakpoints and “running” the debugger. The debugger allows us to stop execution and look at the current state of the program. Setting breakpoints will allow us to stop execution and see what the current state is.
 vim.api.nvim_set_keymap('n', '<leader>b', [[:lua require"dap".toggle_breakpoint()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>c', [[:lua require"dap".continue()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>n', [[:lua require"dap".step_over()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>N', [[:lua require"dap".step_into()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<F5>', [[:lua require"osv".launch({port = 8086})<CR>]], { noremap = true })
 Go to a line where a conditional or value is set and toggle a breakpoint. Then, we’ll start the debugger. If done correctly, you’ll see an arrow next to your line of code you set a breakpoint at.
 There is no UI with dap by default. You have a few options for UI nvim-dap-ui
 In the dap repl you can use the next operations:
  .exit: Closes the REPL
 .c or .continue: Same as |dap.continue|
 .n or .next: Same as |dap.step_over|
 .into: Same as |dap.step_into|
 .into_target: Same as |dap.step_into{askForTargets=true}|
 .out: Same as |dap.step_out|
 .up: Same as |dap.up|
 .down: Same as |dap.down|
 .goto: Same as |dap.goto_|
 .scopes: Prints the variables in the current scopes
 .threads: Prints all threads
 .frames: Print the stack frames
 .capabilities: Print the capabilities of the debug adapter
 .b or .back: Same as |dap.step_back|
 .rc or .reverse-continue: Same as |dap.reverse_continue|
 
 
  New: Introduce nvim-dap-ui.
 Install with packer:
 use { "rcarriga/nvim-dap-ui", requires = {"mfussenegger/nvim-dap"} }
 It is highly recommended to use neodev.nvim to enable type checking for nvim-dap-ui to get type checking, documentation and autocompletion for all API functions.
 
require("neodev").setup({
  library = { plugins = { "nvim-dap-ui" }, types = true },
  ...
})
 nvim-dap-ui is built on the idea of "elements". These elements are windows which provide different features.
 Elements are grouped into layouts which can be placed on any side of the screen. There can be any number of layouts, containing whichever elements desired.
 Elements can also be displayed temporarily in a floating window.
 Each element has a set of mappings for element-specific possible actions, detailed below for each element. The total set of actions/mappings and their default shortcuts are:
  edit: e
 expand: <CR> or left click
 open: o
 remove: d
 repl: r
 toggle: t
 
 See :h dapui.setup() for configuration options and defaults.
 To get started simply call the setup method on startup, optionally providing custom settings.
 require("dapui").setup()
 You can open, close and toggle the windows with corresponding functions:
 require("dapui").open()
require("dapui").close()
require("dapui").toggle()
 
  New: Debug neovim plugins with DAP.
 one-small-step-for-vimkind is an adapter for the Neovim lua language. It allows you to debug any lua code running in a Neovim instance.
 Install it with Packer:
 
use 'mfussenegger/nvim-dap'
 After installing one-small-step-for-vimkind, you will also need a DAP plugin which will allow you to interact with the adapter. Check the install instructions here.
 Then add these lines to your config:
 local dap = require"dap"
dap.configurations.lua = {
  {
    type = 'nlua',
    request = 'attach',
    name = "Attach to running Neovim instance",
  }
}

dap.adapters.nlua = function(callback, config)
  callback({ type = 'server', host = config.host or "127.0.0.1", port = config.port or 8086 })
end
 
  New: Debugging with DAP.
 You can debug Lua code running in a separate Neovim instance with jbyuki/one-small-step-for-vimkind.
 The plugin uses the Debug Adapter Protocol. Connecting to a debug adapter requires a DAP client like mfussenegger/nvim-dap. Check how to configure here
 Once you have all set up and assuming you're using the next keybindings for nvim-dap:
 vim.api.nvim_set_keymap('n', '<leader>b', [[:lua require"dap".toggle_breakpoint()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>c', [[:lua require"dap".continue()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>n', [[:lua require"dap".step_over()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>N', [[:lua require"dap".step_into()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<F5>', [[:lua require"osv".launch({port = 8086})<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>B', [[:lua require"dapui".toggle()<CR>]], { noremap = true })
 You will debug the plugin by:
  Launch the server in the debuggee using F5.
 Open another Neovim instance with the source file (the debugger).
 Place breakpoint with <leader>b.
 On the debugger connect to the DAP client with <leader>c.
 Optionally open the nvim-dap-ui with <leader>B in the debugger.
 Run your script/plugin in the debuggee
 Interact in the debugger using <leader>n to step to the next step, and <leader>N to step into. Then use the dap console to inspect and change the values of the state.
 
 
  New: Introduce morph.io.
 morph.io is a web service that runs your scrapers for you.
 Write your scraper in the language you know and love, push your code to GitHub, and they take care of the boring bits. Things like running your scraper regularly, alerting you if there's a problem, storing your data, and making your data available for download or through a super-simple API.
 To sign in you'll need a GitHub account. This is where your scraper code is stored.
 The data is stored in an sqlite
 Usage limits
 Right now there are very few limits. They are trusting you that you won't abuse this.
 However, they do impose a couple of hard limits on running scrapers so they don't take up too many resources
  max 512 MB memory
 max 24 hours run time for a single run
 
 If a scraper runs out of memory or runs too long it will get killed automatically.
 There's also a soft limit:
  max 10,000 lines of log output
 
 If a scraper generates more than 10,000 lines of log output the scraper will continue running uninterrupted. You just won't see any more output than that. To avoid this happening simply print less stuff to the screen.
 Note that they are keeping track of the amount of cpu time (and a whole bunch of other metrics) that you and your scrapers are using. So, if they do find that you are using too much they reserve the right to kick you out. In reality first they'll ask you nicely to stop.
 References
  Docs
 Home
 
 
 
 Vim autosave
   Correction: Search for alternatives to git-sweep.
 The tool is no longer maintained but there is still no good alternative. I've found some but are either not popular and/or not maintained:
  gitsweeper
 git-removed-brances
 git-sweep rewrite in go
 
 
  New: Update all git submodules.
 If it's the first time you check-out a repo you need to use --init first:
 git submodule update --init --recursive
 To update to latest tips of remote branches use:
 git submodule update --recursive --remote
 
  New: Manually toggle the autosave function.
 Besides running auto-save at startup (if you have enabled = true in your config), you may as well:
  ASToggle: toggle auto-save
 
 
 
 Espanso
   New: Introduce espanso.
 Espanso is a cross-platform Text Expander written in Rust.
 A text expander is a program that detects when you type a specific keyword and replaces it with something else. This is useful in many ways:
  Save a lot of typing, expanding common sentences or fixing common typos.
 Create system-wide code snippets.
 Execute custom scripts
 Use emojis like a pro.
 
 Installation Espanso ships with a .deb package, making the installation convenient on Debian-based systems.
 Start by downloading the package by running the following command inside a terminal:
 wget https://github.com/federico-terzi/espanso/releases/download/v2.2.1/espanso-debian-x11-amd64.deb
 You can now install the package using:
 sudo apt install ./espanso-debian-x11-amd64.deb
 From now on, you should have the espanso command available in the terminal (you can verify by running espanso --version).
 At this point, you are ready to use espanso by registering it first as a Systemd service and then starting it with:
 espanso service register
 Start espanso
 espanso start
 Espanso ships with very few built-in matches to give you the maximum flexibility, but you can expand its capabilities in two ways: creating your own custom matches or installing packages.
 Configuration
 Your configuration lives at ~/.config/espanso. A quick way to find the path of your configuration folder is by using the following command espanso path.
  The files contained in the match directory define what Espanso should do. In other words, this is where you should specify all the custom snippets and actions (aka Matches). The match/base.yml file is where you might want to start adding your matches.
 The files contained in the config directory define how Espanso should perform its expansions. In other words, this is were you should specify all Espanso's parameters and options. The config/default.yml file defines the options that will be applied to all applications by default, unless an app-specific configuration is present for the current app.
 
 Using packages
 Custom matches are great, but sometimes it can be tedious to define them for every common operation, especially when you want to share them with other people.
 Espanso offers an easy way to share and reuse matches with other people, packages. In fact, they are so important that Espanso includes a built-in package manager and a store, the Espanso Hub.
 Installing a package
 Get the id of the package from the Espanso Hub and then run espanso install <<package_name>>.
 Of all the packages, I've found the next ones the most useful:
  typofixer-en
 typofixer-es
 misspell-en-uk
 
 Overwriting the snippets of a package
 For example the typofixer-en replaces si to is, although si is a valid spanish word. To override the fix you can create your own file on ~/.config/espanso/match/typofix_overwrite.yml with the next content:
 matches:
  # Simple text replacement
  - trigger: "si"
    replace: "si"
 Creating a package
 Auto-restart on config changes
 Set auto_restart: true on ~/.config/espanso/config/default.yml.
 Changing the search bar shortcut
 If the default search bar shortcut conflicts with your i3 configuration set it with:
 search_shortcut: CTRL+SHIFT+e
 Hiding the notifications
 You can hide the notifications by adding the following option to your $CONFIG/config/default.yml config:
 show_notifications: false
 Usage
 Just type and you'll see the text expanded.
 You can use the search bar if you don't remember your snippets.
 References - Code - Docs
 
  New: Desktop application to add words easily.
 Going into the espanso config files to add words is cumbersome, to make things easier you can use the espansadd Python script.
 I'm going to assume that you have the following prerequisites:
  A Linux distribution with i3 window manager installed.
 Python 3 installed.
 Espanso installed and configured.
 ruyaml and tkinter Python libraries installed.
 notify-send installed.
 Basic knowledge of editing configuration files in i3.
 
 Installation
 Create a new Python script named espansadd.py with the following content:
 import tkinter as tk
from tkinter import simpledialog
import traceback
import subprocess
import os
import sys

from ruyaml import YAML
from ruyaml.scanner import ScannerError

file_path = os.path.expanduser("~/.config/espanso/match/typofixer_overwrite.yml")

def append_to_yaml(file_path: str, trigger: str, replace: str) -> None:
    """Appends a new entry to the YAML file.

    Args:ath
        file_path (str): The file to append the new entry.
        trigger (str): The trigger string to be added.
        replace (str): The replacement string to be added.
    """

    # Define the new snippet
    new_entry = {
        "trigger": trigger,
        "replace": replace,
        "propagate_case": True,
        "word": True,
    }

    # Load existing data or initialize an empty list
    try:
        with open(os.path.expanduser(file_path), "r") as f:
            try:
                data = YAML().load(f)
            except ScannerError as e:
                send_notification(
                    f"Error parsing yaml of configuration file {file_path}",
                    f"{e.problem_mark}: {e.problem}",
                    "critical",
                )
                sys.exit(1)
    except FileNotFoundError:
        send_notification(
            f"Error opening the espanso file {file_path}", urgency="critical"
        )
        sys.exit(1)

    data["matches"].append(new_entry)

    # Write the updated data back to the file
    with open(os.path.expanduser(file_path), "w+") as f:
        yaml = YAML()
        yaml.default_flow_style = False
        yaml.dump(data, f)

def send_notification(title: str, message: str = "", urgency: str = "normal") -> None:
    """Send a desktop notification using notify-send.

    Args:
        title (str): The title of the notification.
        message (str): The message body of the notification. Defaults to an empty string.
        urgency (str): The urgency level of the notification. Can be 'low', 'normal', or 'critical'. Defaults to 'normal'.
    """
    subprocess.run(["notify-send", "-u", urgency, title, message])

def main() -> None:
    """Main function to prompt user for input and append to the YAML file."""
    # Create the main Tkinter window (it won't be shown)
    window = tk.Tk()
    window.withdraw()  # Hide the main window

    # Prompt the user for input
    trigger = simpledialog.askstring("Espanso add input", "Enter trigger:")
    replace = simpledialog.askstring("Espanso add input", "Enter replace:")

    # Check if both inputs were provided
    try:
        if trigger and replace:
            append_to_yaml(file_path, trigger, replace)
            send_notification("Espanso snippet added successfully")
        else:
            send_notification(
                "Both trigger and replace are required", urgency="critical"
            )
    except Exception as error:
        error_message = "".join(
            traceback.format_exception(None, error, error.__traceback__)
        )
        send_notification(
            "There was an unknown error adding the espanso entry",
            error_message,
            urgency="critical",
        )

if __name__ == "__main__":
    main()
 Ensure the script has executable permissions. Run the following command:
 chmod +x espansadd.py
 To make the espansadd script easily accessible, we can configure a key binding in i3 to run the script. Open your i3 configuration file, typically located at ~/.config/i3/config or ~/.i3/config, and add the following lines:
 bindsym $mod+Shift+e exec --no-startup-id /path/to/your/espansadd.py
 Replace /path/to/your/espansadd.py with the actual path to your script.
 If you also want the popup windows to be in floating mode add
 for_window [title="Espanso add input"] floating enable
 After editing the configuration file, reload i3 to apply the changes. You can do this by pressing Mod + Shift + R (where Mod is typically the Super or Windows key) or by running the following command:
 i3-msg reload
 Usage
 Now that everything is set up, you can use the espansadd script by pressing Mod + Shift + E. This will open a dialog where you can enter the trigger and replacement text for the new Espanso snippet. After entering the information and pressing Enter, a notification will appear confirming the snippet has been added, or showing an error message if something went wrong.
 
 
 Generic Coding Practices
 Writing good documentation
   New: Add diátaxis as documentation writing guideline.
 Diátaxis: A systematic approach to technical documentation authoring
 
 
 Conventional comments
   New: Introduce conventional comments.
 Conventional comments is the practice to use a specific format in the review comments to express your intent and tone more clearly. It's strongly inspired by semantic versioning.
 Let's take the next comment:
 This is not worded correctly.
 Adding labels you can tell the difference on your intent:
 
**suggestion:** This is not worded correctly.
 Or **issue (non-blocking):** This is not worded correctly.
 Labels also prompt the reviewer to give more actionable comments.
 **suggestion:** This is not worded correctly.

Can we change this to match the wording of the marketing page?
 Labeling comments encourages collaboration and saves hours of undercommunication and misunderstandings. They are also parseable by machines!
 Format
 Adhering to a consistent format improves reader's expectations and machine readability. Here's the format we propose: 
<label> [decorations]: <subject>

[discussion]
 - label - This is a single label that signifies what kind of comment is being left. - subject - This is the main message of the comment. - decorations (optional) - These are extra decorating labels for the comment. They are surrounded by parentheses and comma-separated. - discussion (optional) - This contains supporting statements, context, reasoning, and anything else to help communicate the "why" and "next steps" for resolving the comment. For example: **question (non-blocking):** At this point, does it matter which thread has won?

Maybe to prevent a race condition we should keep looping until they've all won?
 Can be automatically parsed into:
 
{
  "label": "question",
  "subject": "At this point, does it matter which thread has won?",
  "decorations": ["non-blocking"],
  "discussion": "Maybe to prevent a race condition we should keep looping until they've all won?"
}
 Labels
 We strongly suggest using the following labels: | | | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | praise: | Praises highlight something positive. Try to leave at least one of these comments per review. Do not leave false praise (which can actually be damaging). Do look for something to sincerely praise. | | quibble: | Quibbles are trivial preference-based requests. These should be non-blocking by nature. Similar to polish but clearly preference-based.| | suggestion: | Suggestions propose improvements to the current subject. It's important to be explicit and clear on what is being suggested and why it is an improvement. These are non-blocking proposals. If it's blocking use todo instead.| | todo: | TODO's are necessary changes. Distinguishing todo comments from issues or suggestions helps direct the reader's attention to comments requiring more involvement. | | issue: | Issues highlight specific problems with the subject under review. These problems can be user-facing or behind the scenes. It is strongly recommended to pair this comment with a suggestion. If you are not sure if a problem exists or not, consider leaving a question. | | question: | Questions are appropriate if you have a potential concern but are not quite sure if it's relevant or not. Asking the author for clarification or investigation can lead to a quick resolution. | | thought: | Thoughts represent an idea that popped up from reviewing. These comments are non-blocking by nature, but they are extremely valuable and can lead to more focused initiatives and mentoring opportunities. | | chore: | Chores are simple tasks that must be done before the subject can be "officially" accepted. Usually, these comments reference some common process. Try to leave a link to the process description so that the reader knows how to resolve the chore. | | note: | Notes are always non-blocking and simply highlight something the reader should take note of. |
 If you like to be a bit more expressive with your labels, you may also consider:
      
 
   typo:  Typo comments are like todo:, where the main issue is a misspelling.  
  polish:  Polish comments are like a suggestion, where there is nothing necessarily wrong with the relevant content, there's just some ways to immediately improve the quality. Similar but not exactly the same as quibble.  
 
 
 Decorations
 Decorations give additional context for a comment. They help further classify comments which have the same label (for example, a security suggestion as opposed to a test suggestion). 
**suggestion (security):** I'm a bit concerned that we are implementing our own DOM purifying function here...
Could we consider using the framework instead?
 **suggestion (test,if-minor):** It looks like we're missing some unit test coverage that the cat disappears completely.
 Decorations may be specific to each organization. If needed, we recommend establishing a minimal set of decorations (leaving room for discretion) with no ambiguity. Possible decorations include: | | | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | (non-blocking) | A comment with this decoration should not prevent the subject under review from being accepted. This is helpful for organizations that consider comments blocking by default. | | (blocking) | A comment with this decoration should prevent the subject under review from being accepted, until it is resolved. This is helpful for organizations that consider comments non-blocking by default. | | (if-minor) | This decoration gives some freedom to the author that they should resolve the comment only if the changes ends up being minor or trivial. |
 Adding a decoration to a comment should improve understandability and maintain readability. Having a list of many decorations in one comment would conflict with this goal.
 More examples 
**quibble:** `little star` => `little bat`

Can we update the other references as well?
 **chore:** Let's run the `jabber-walk` CI job to make sure this doesn't break any known references.
Here are [the docs](https://en.wikipedia.org/wiki/Jabberwocky) for running this job. Feel free to reach out if you need any help!
 **praise:** Beautiful test!
 Best Practices
 Some best practices for writing helpful review feedback:
  Mentoring pays off exponentially
 Leave actionable comments
 Combine similar comments
 Replace "you" with "we"
 Replace "should" with "could"
 
 References - Home
 
 
 OCR
 Camelot
   New: Introduce Camelot.
 Camelot is a Python library that can help you extract tables from PDFs
 import camelot

tables = camelot.read_pdf('foo.pdf')

tables
<TableList n=1>

tables.export('foo.csv', f='csv', compress=True) # json, excel, html, markdown, sqlite

tables[0]
<Table shape=(7, 7)>

tables[0].parsing_report
{
    'accuracy': 99.02,
    'whitespace': 12.24,
    'order': 1,
    'page': 1
}

tables[0].to_csv('foo.csv') # to_json, to_excel, to_html, to_markdown, to_sqlite

tables[0].df # get a pandas DataFrame!
 Installation
 To install Camelot from PyPI using pip, please include the extra cv requirement as shown:
 $ pip install "camelot-py[base]"
 It requires Ghostscript to be able to use the lattice mode. Which is better than using tabular-py that requires java to be installed.
 Usage
 Process background lines
 To detect line segments, Lattice needs the lines that make the table to be in the foreground. To process background lines, you can pass process_background=True.
 tables = camelot.read_pdf('background_lines.pdf', process_background=True)
 tables[1].df
 References - Docs
 
 
 DevSecOps
 Infrastructure as Code
 Ansible Snippets
   New: Fix the ERROR! 'become' is not a valid attribute for a IncludeRole error.
 If you're trying to do something like: 
tasks:
  - name: "Install nfs"
    become: true
    ansible.builtin.include_role:
      name: nfs
 You need to use this other syntax:
 tasks:
  - name: "Install nfs"
    ansible.builtin.include_role:
      name: nfs
      apply:
        become: true
 
  New: Set host variables using a dynamic inventory.
 As with a normal inventory you can use the host_vars files with the proper name.
 
  New: Avoid arbitrary disk mount.
 Instead of using /dev/sda use /dev/disk/by-id/whatever
 
  New: Get the user running ansible in the host.
 If you gather_facts use the ansible_user_id variable.
 
  New: Filter json data.
 To select a single element or a data subset from a complex data structure in JSON format (for example, Ansible facts), use the community.general.json_query filter. The community.general.json_query filter lets you query a complex JSON structure and iterate over it using a loop structure.
 This filter is built upon jmespath, and you can use the same syntax. For examples, see jmespath examples.
 A complex example would be:
 "{{ ec2_facts | json_query('instances[0].block_device_mappings[?device_name!=`/dev/sda1` && device_name!=`/dev/xvda`].{device_name: device_name, id: ebs.volume_id}') }}"
 This snippet:
  Gets all dictionaries under the block_device_mappings list which device_name is not equal to /dev/sda1 or /dev/xvda
 From those results it extracts and flattens only the desired values. In this case device_name and the id which is at the key ebs.volume_id of each of the items of the block_device_mappings list.
 
 
  New: Do asserts.
 - name: After version 2.7 both 'msg' and 'fail_msg' can customize failing assertion message
  ansible.builtin.assert:
    that:
      - my_param <= 100
      - my_param >= 0
    fail_msg: "'my_param' must be between 0 and 100"
    success_msg: "'my_param' is between 0 and 100"
 
  New: Split a variable in ansible.
 {{ item | split ('@') | last }}
 
  New: Get a list of EC2 volumes mounted on an instance an their mount points.
 Assuming that each volume has a tag mount_point you could:
 - name: Gather EC2 instance metadata facts
  amazon.aws.ec2_metadata_facts:

- name: Gather info on the mounted disks
  delegate_to: localhost
  block:
    - name: Gather information about the instance
      amazon.aws.ec2_instance_info:
        instance_ids:
          - "{{ ansible_ec2_instance_id }}"
      register: ec2_facts

    - name: Gather volume tags
      amazon.aws.ec2_vol_info:
        filters:
          volume-id: "{{ item.id }}"
      # We exclude the root disk as they are already mounted and formatted
      loop: "{{ ec2_facts | json_query('instances[0].block_device_mappings[?device_name!=`/dev/sda1` && device_name!=`/dev/xvda`].{device_name: device_name, id: ebs.volume_id}') }}"
      register: volume_tags_data

    - name: Save the required volume data
      set_fact:
        volumes: "{{ volume_tags_data | json_query('results[0].volumes[].{id: id, mount_point: tags.mount_point}') }}"

    - name: Display volumes data
      debug:
        msg: "{{ volumes }}"

    - name: Make sure that all volumes have a mount point
      assert:
        that:
          - item.mount_point is defined
          - item.mount_point|length > 0
        fail_msg: "Configure the 'mount_point' tag on the volume {{ item.id }} on the instance {{ ansible_ec2_instance_id }}"
        success_msg: "The volume {{ item.id }} has the mount_point tag well set"
      loop: "{{ volumes }}"
 
  New: Create a list of dictionaries using ansible.
 - name: Create and Add items to dictionary
  set_fact:
      userdata: "{{ userdata | default({}) | combine ({ item.key : item.value }) }}"
  with_items:
    - { 'key': 'Name' , 'value': 'SaravAK'}
    - { 'key': 'Email' , 'value': 'sarav@gritfy.com'}
    - { 'key': 'Location' , 'value': 'Coimbatore'}
    - { 'key': 'Nationality' , 'value': 'Indian'}
 
  New: Merge two dictionaries on a key.
 If you have these two lists:
 
"list1": [
  { "a": "b", "c": "d" },
  { "a": "e", "c": "f" }
]

"list2": [
  { "a": "e", "g": "h" },
  { "a": "b", "g": "i" }
]
 And want to merge them using the value of key "a":
 "list3": [
  { "a": "b", "c": "d", "g": "i" },
  { "a": "e", "c": "f", "g": "h" }
]
 If you can install the collection community.general use the filter lists_mergeby. The expression below gives the same result
 list3: "{{ list1|community.general.lists_mergeby(list2, 'a') }}"
 
  New: Loop over dict fails when only one element detected.
 If you see the If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup." error in an Ansible log it means that the content of the variable is not the type you expect it to be. This can happen for example for lists that have only one or zero elements, which gets translated into a string thus breaking the loop structure.
 So instead of: 
- name: Create filesystem on device
  community.general.filesystem:
    fstype: ext4
    dev: "/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol{{ item.id | split('-') | last }}"
  loop: "{{ volumes }}"
 You can use:
 - name: Create filesystem on device
  community.general.filesystem:
    fstype: ext4
    dev: "/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol{{ item.id | split('-') | last }}"
  loop: "{{ lookup('list', volumes, wantlist=True) }}"
 If that gives you issues you can use this other construction instead:
 - name: Save the required volume data
  set_fact:
    volumes: "{{ volume_tags_data | json_query('results[0].volumes[].{id: id, mount_point: tags.mount_point}') }}"

- name: Get result type for the volumes
  set_fact:
    volumes_type: "{{ volumes | type_debug }}"

- name: Display volumes type
  debug:
    msg: "{{ volumes_type }}"

- name: Force list of volumes if it's a string
  set_fact:
    volumes: "{{ [] }}"
  when:
    - volumes_type == 'str'

- name: Force list of volumes if it's a dictionary
  set_fact:
    volumes: "{{ [volumes] }}"
  when:
    - volumes_type == 'dict'

- name: Create filesystem on device
  community.general.filesystem:
    fstype: ext4
    dev: "/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol{{ item.id | split('-') | last }}"
  loop: "{{ volumes }}"
 
  New: Set the ssh connection port using dynamic inventories.
 To specify a custom SSH port, you can use a host_vars or group_vars file. For example, create a group_vars directory and a file named all.yaml inside it:
 ansible_port: 2222
 
 
 Infrastructure Solutions
 Kubernetes
   New: Introduce IceKube.
 IceKube tool for finding complex attack paths in Kubernetes clusters. It's like Bloodhound for Kubernetes. It uses Neo4j to store & analyze Kubernetes resource relationships → identify attack paths & security misconfigs
 
  New: Add reloader to tools to test.
 stakater/reloader: A Kubernetes controller to watch changes in ConfigMap and Secrets and do rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet and DeploymentConfig. Useful for not that clever applications that need a reboot when a configmap changes.
 
 
 AWS
  New: Get the role used by the instance.
 
 Kubectl Commands
   New: Understanding how reserved instances are applied.
 A Reserved Instance that is purchased for a Region is called a regional Reserved Instance, and provides Availability Zone and instance size flexibility.
  The Reserved Instance discount applies to instance usage in any Availability Zone in that Region.
 The Reserved Instance discount applies to instance usage within the instance family, regardless of size—this is known as instance size flexibility.
 
 With instance size flexibility, the Reserved Instance discount applies to instance usage for instances that have the same family, generation, and attribute. The Reserved Instance is applied from the smallest to the largest instance size within the instance family based on the normalization factor.
 The discount applies either fully or partially to running instances of the same instance family, depending on the instance size of the reservation, in any Availability Zone in the Region. The only attributes that must be matched are the instance family, tenancy, and platform.
 The following table lists the different sizes within an instance family, and the corresponding normalization factor. This scale is used to apply the discounted rate of Reserved Instances to the normalized usage of the instance family.
    Instance size  Normalization factor  
 
   nano  0.25  
  micro  0.5  
  small  1  
  medium  2  
  large  4  
  xlarge  8  
  2xlarge  16  
  3xlarge  24  
  4xlarge  32  
  6xlarge  48  
  8xlarge  64  
  9xlarge  72  
  10xlarge  80  
  12xlarge  96  
  16xlarge  128  
  18xlarge  144  
  24xlarge  192  
  32xlarge  256  
  48xlarge  384  
  56xlarge  448  
  112xlarge  896  
 
 
 For example, a t2.medium instance has a normalization factor of 2. If you purchase a t2.medium default tenancy Amazon Linux/Unix Reserved Instance in the US East (N. Virginia) and you have two running t2.small instances in your account in that Region, the billing benefit is applied in full to both instances.
 Or, if you have one t2.large instance running in your account in the US East (N. Virginia) Region, the billing benefit is applied to 50% of the usage of the instance.
 Limitations:
  Supported: Instance size flexibility is only supported for Regional Reserved Instances.
 Not supported: Instance size flexibility is not supported for the following Reserved Instances: Reserved Instances that are purchased for a specific Availability Zone (zonal Reserved Instances)
 Reserved Instances for G4ad, G4dn, G5, G5g, and Inf1 instances
 Reserved Instances for Windows Server, Windows Server with SQL Standard, Windows Server with SQL Server Enterprise, Windows Server with SQL Server Web, RHEL, and SUSE Linux Enterprise Server
 Reserved Instances with dedicated tenancy
 
 
 
 
  New: EC2 Instance savings plan versus reserved instances.
 I've been comparing the EC2 Reserved Instances and of the EC2 instance family savings plans and decided to go with the second because:
  They both have almost the same rates. Reserved instances round the price at the 3^rd decimal and the savings plan at the fourth, but this difference is neglegible.
 Savings plan are easier to calculate, as you just need to multiply the number of instances you want times the current rate and add them all up.
 Easier to understand: To reserve instances you need to take into account the instance flexibility and the normalization factors which makes it difficult both to make the plans and also to audit how well you're using it.
 Easier to audit: In addition to the above point, you have nice dashboards to see the coverage and utilization over time of your ec2 instance savings plans, which are at the same place as the other savings plans.
 
 
  New: Important notes when doing a savings plan.
  Always use the reservation rates instead of the on-demand rates!
 Analyze your coverage reports. You don't want to have many points of 100% coverage as it means that you're using less resources than you've reserved. On the other hand it's fine to sometimes use less resources than the reserved if that will mean a greater overall savings. It's a tight balance.
 The Savings plan reservation is taken into account at hour level, not at month or year level. That means that if you reserve 1 $/hour of an instance type and you use for example 2$ /hour half the day and 0 $/hour half the day, you'll have a 100% coverage of your plan the first hour and another 1$ /hour of on-demand infrastructure cost for the first part of the day. On the second part of the day you'll have a 0% coverage. This means that you should only reserve the amount of resources you plan to be using 100% of the time throughout your savings plan. Again you may want to overcommit a little bit, reducing the utilization percentage of a plan but getting better savings in the end.
 
 
  New: Upload a file to a pod.
 kubectl cp {{ path_to_local_file }} {{ container_id }}:{{ path_to_file }}
 
  New: Delete pods that are stuck in terminating state for a while.
 kubectl delete pod <pod-name> --grace-period=0 --force
 
 
 Volumes
   New: Troubleshoot don't have enough permissions to start restore from backup.
 That may be because the default EFS backup policy doesn't let you do that (stupid, I know).
 To fix it go into the backup policy and remove the next line from the Deny policy:
 backup:StartRestoreJob
 
  New: Specify a path of a configmap.
 If you have a configmap with a key ssh-known-hosts and you want to mount it's content in a file, in the deployment volumeMounts section you can use the subPath field:
       - mountPath: /home/argocd/.ssh/known_hosts
        name: ssh-known-hosts
        subPath: ssh_known_hosts
        readOnly: true
 
  New: List the size of the recovery points.
 BACKUP_VAULT_NAME="your-vault-name"

RECOVERY_POINTS=$(aws backup list-recovery-points-by-backup-vault --backup-vault-name $BACKUP_VAULT_NAME --query 'RecoveryPoints[*].[RecoveryPointArn,BackupSizeInBytes,CreationDate]' --output text)
echo -e "Creation Date\t\tRecovery Point ARN\t\t\t\t\t\t\t\t\tSize (TB)"
echo "---------------------------------------------------------------------------------------------------------------------"

while read -r RECOVERY_POINT_ARN BACKUP_SIZE_BYTES CREATION_DATE; do
    # Remove the decimal part from the epoch time
    EPOCH_TIME=$(echo $CREATION_DATE | cut -d'.' -f1)
    # Convert the creation date from epoch time to YYYY-MM-DD format
    FORMATTED_DATE=$(date -d @$EPOCH_TIME +"%Y-%m-%d")
    SIZE_TB=$(echo "scale=6; $BACKUP_SIZE_BYTES / (1024^4)" | bc)
    # echo -e "$FORMATTED_DATE\t$RECOVERY_POINT_ARN\t$SIZE_TB"
    printf "%-16s %-80s %10.6f\n" "$FORMATTED_DATE" "$RECOVERY_POINT_ARN" "$SIZE_TB"
done <<< "$RECOVERY_POINTS"
 
  New: List the size of the jobs.
 To list AWS Backup jobs and display their completion dates and sizes in a human-readable format, you can use the following AWS CLI command combined with jq for parsing and formatting the output. This command handles cases where the backup size might be null and rounds the size to the nearest whole number in gigabytes.
 
aws backup list-backup-jobs --output json | jq -r '
  .BackupJobs[] |
  [
    (.CompletionDate | strftime("%Y-%m-%d %H:%M:%S")),
    (if .BackupSizeInBytes == null then "0GB" else ((.BackupSizeInBytes / 1024 / 1024 / 1024) | floor | tostring + " GB") end)
  ] |
  @tsv' | column -t -s$'\t'
 Explanation:
  aws backup list-backup-jobs --output json: Lists all AWS Backup jobs in JSON format.
 .BackupJobs[]: Iterates over each backup job.
 (.CompletionDate | strftime("%Y-%m-%d %H:%M:%S")): Converts the Unix timestamp in CompletionDate to a human-readable date format (YYYY-MM-DD HH:MM:SS).
 (if .BackupSizeInBytes == null then "0GB" else ((.BackupSizeInBytes / 1024 / 1024 / 1024) | floor | tostring + " GB") end): Checks if BackupSizeInBytes is null. If it is, outputs "0GB". Otherwise, converts the size from bytes to gigabytes, rounds it down to the nearest whole number, and appends " GB".
 | @tsv: Formats the output as tab-separated values.
 column -t -s$'\t': Formats the TSV output into a table with columns aligned.
 
 
 
 Continuous Deployment
 ArgoCD
   New: Not there yet.
  Support git webhook on Applicationsets for gitea/forgejo: although you could use an ugly fix adding spec.generators[i].requeueAfterSeconds to change the interval that ArgoCD uses to refresh the repositories, which is 3 minutes by default.
 
 
 
 Continuous Integration
 Bandit
   New: Solving warning B603: subprocess_without_shell_equals_true.
 The B603: subprocess_without_shell_equals_true issue in Bandit is raised when the subprocess module is used without setting shell=True. Bandit flags this because using shell=True can be a security risk if the command includes user-supplied input, as it opens the door to shell injection attacks.
 To fix it:
  Avoid shell=True if possible: Instead, pass the command and its arguments as a list to subprocess.Popen (or subprocess.run, subprocess.call, etc.). This way, the command is executed directly without invoking the shell, reducing the risk of injection attacks.
 
 Here's an example:
 import subprocess

# Instead of this:
# subprocess.Popen("ls -l", shell=True)

# Do this:
subprocess.Popen(["ls", "-l"])
  When you must use shell=True: - If you absolutely need to use shell=True (e.g., because you are running a complex shell command or using shell features like wildcards), ensure that the command is either hardcoded or sanitized to avoid security risks.
 
 Example with shell=True:
 import subprocess

# Command is hardcoded and safe
command = "ls -l | grep py"
subprocess.Popen(command, shell=True)
 If the command includes user input, sanitize the input carefully:
 import subprocess

user_input = "some_directory"
command = f"ls -l {subprocess.list2cmdline([user_input])}"
subprocess.Popen(command, shell=True)
 Note: Even with precautions, using shell=True is risky with user input, so avoid it if possible.
  Explicitly tell bandit you have considered the risk: If you have reviewed the code and are confident that the code is safe in your particular case, you can mark the line with a # nosec comment to tell Bandit to ignore the issue:
 
 import subprocess

command = "ls -l | grep py"
subprocess.Popen(command, shell=True)  # nosec
 
 
 Safety
   New: Add deprecation warning.
 Since 2024-05-27 it requires an account to work, use pip-audit instead.
 
 
 Security Checkers
   New: Introduce pip-audit.
 pip-audit is the official pypa tool for scanning Python environments for packages with known vulnerabilities. It uses the Python Packaging Advisory Database (https://github.com/pypa/advisory-database) via the PyPI JSON API as a source of vulnerability reports.
 Installation
 pip install pip-audit
 Usage
 
pip-audit
 On completion, pip-audit will exit with a code indicating its status.
 The current codes are:
  0: No known vulnerabilities were detected.
 1: One or more known vulnerabilities were found.
 
 pip-audit's exit code cannot be suppressed. See Suppressing exit codes from pip-audit for supported alternatives.
 References
  Code
 
 
 
 Yamllint
   New: Ignore certain files.
 It is possible to exclude specific files or directories, so that the linter doesn’t process them. They can be provided either as a list of paths, or as a bulk string.
 You can either totally ignore files (they won’t be looked at):
 extends: default

ignore: |
  /this/specific/file.yaml
  all/this/directory/
  *.template.yaml

ignore:
  - /this/specific/file.yaml
  - all/this/directory/
  - '*.template.yaml'
 
 
 Automating Processes
 letsencrypt
   New: Manually renew a certificate.
 Linuxserver swag container renews the certificates at night. If you don't have your server up at those hours your certificate won't be renewed automatically and you need to react to the prometheus alert manually. To do so get into the container and run certbot renew.
 
 
 Storage
 NAS
   Correction: Add suggestions when buying a motherboard.
 When choosing a motherboard make sure that:
  If you want ECC that it truly supports ECC.
 It is IPMI compliant, if you want to have hardware watchdog support
 
 
 
 OpenZFS
   New: Solve the pool or dataset is busy error.
 If you get an error of pool or dataset is busy run the next command to see which process is still running on the pool:
 lsof 2>/dev/null | grep dataset-name
 
  New: Monitor dbgmsg with loki.
 If you use loki remember to monitor the /proc/spl/kstat/zfs/dbgmsg file:
 - job_name: zfs
    static_configs:
      - targets:
          - localhost
        labels:
          job: zfs
          __path__: /proc/spl/kstat/zfs/dbgmsg
 
  Correction: Add loki alerts on the kernel panic error.
 You can monitor this issue with loki using the next alerts:
 groups:
  - name: zfs
    rules:
      - alert: SlowSpaSyncZFSError
       expr: |
          count_over_time({job="zfs"} |~ `spa_deadman.*slow spa_sync` [5m])
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Slow sync traces found in the ZFS debug logs at {{ $labels.hostname}}"
          message: "This usually happens before the ZFS becomes unresponsible"
 
  New: Monitorization.
 You can monitor this issue with loki using the next alerts:
 groups:
  - name: zfs
    rules:
      - alert: ErrorInSanoidLogs
        expr: |
          count_over_time({job="systemd-journal", syslog_identifier="sanoid"} |= `ERROR` [5m])
        for: 1m
        labels:
            severity: critical
        annotations:
            summary: "Errors found on sanoid log at {{ $labels.hostname}}"
 
  New: Set zfs module parameters or options.
 Most of the ZFS kernel module parameters are accessible in the SysFS /sys/module/zfs/parameters directory. Current values can be observed by
 cat /sys/module/zfs/parameters/PARAMETER
 Many of these can be changed by writing new values. These are denoted by Change|Dynamic in the PARAMETER details below.
 echo NEWVALUE >> /sys/module/zfs/parameters/PARAMETER
 If the parameter is not dynamically adjustable, an error can occur and the value will not be set. It can be helpful to check the permissions for the PARAMETER file in SysFS.
 In some cases, the parameter must be set prior to loading the kernel modules or it is desired to have the parameters set automatically at boot time. For many distros, this can be accomplished by creating a file named /etc/modprobe.d/zfs.conf containing a text line for each module parameter using the format:
 options zfs PARAMETER=VALUE
 Some parameters related to ZFS operations are located in module parameters other than in the zfs kernel module. These are documented in the individual parameter description. Unless otherwise noted, the tunable applies to the zfs kernel module. For example, the icp kernel module parameters are visible in the /sys/module/icp/parameters directory and can be set by default at boot time by changing the /etc/modprobe.d/icp.conf file.
 
  New: Configure the deadman failsafe measure.
 ZFS has a safety measure called the zfs_deadman_failmode. When a pool sync operation takes longer than zfs_deadman_synctime_ms, or when an individual I/O operation takes longer than zfs_deadman_ziotime_ms, then the operation is considered to be "hung". If zfs_deadman_enabled is set, then the deadman behavior is invoked as described by zfs_deadman_failmode. By default, the deadman is enabled and set to wait which results in "hung" I/O operations only being logged. The deadman is automatically disabled when a pool gets suspended.
 zfs_deadman_failmode configuration can have the next values:
  wait: Wait for a "hung" operation to complete. For each "hung" operation a "deadman" event will be posted describing that operation.
 continue: Attempt to recover from a "hung" operation by re-dispatching it to the I/O pipeline if possible.
 panic: Panic the system. This can be used to facilitate automatic fail-over to a properly configured fail-over partner.
 
 Follow the guides under Set zfs module parameters or options to change this value.
 
  New: Monitor the ZFS events.
 You can see the ZFS events using zpool events -v. If you want to be alerted on these events you can use this service to ingest them into Loki and raise alerts.
 
  New: Manually create a backup.
 To create a snapshot of tank/home/ahrens that is named friday run:
 zfs snapshot tank/home/ahrens@friday
 
  New: Tweak loki alerts.
       - alert: SyncoidCorruptedSnapshotSendError
        expr: |
          count_over_time({syslog_identifier="syncoid_send_backups"} |= `cannot receive incremental stream: invalid backup stream` [15m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Error trying to send a corrupted snapshot at {{ $labels.hostname}}"
          message: "Look at the context on loki to identify the snapshot in question. Delete it and then run the sync again"
      - alert: SanoidNotRunningError
        expr: |
          sum by (hostname) (count_over_time({job="systemd-journal", syslog_identifier="sanoid"}[1h])) or sum by (hostname) (count_over_time({job="systemd-journal"}[1h]) * 0)
        for: 0m
        labels:
            severity: critical
        annotations:
            summary: "Sanoid has not shown signs to be alive for the last hour at least in arva and helm"
      - alert: SlowSpaSyncZFSError
        expr: |
          count_over_time({job="zfs"} |~ `spa_deadman.*slow spa_sync` [10m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Slow sync traces found in the ZFS debug logs at {{ $labels.hostname}}"
          message: "This usually happens before the ZFS becomes unresponsible"
 The SanoidNotRunningError alert uses a broader search that ensures that all hosts are included and multiplies it to 0 to raise the alert if none is shown for the sanoid service.
 
  New: Sync an already created cold backup.
 Mount the existent pool
 Imagine your pool is at /dev/sdf2:
  Connect your device
 Check for available ZFS pools: First, check if the system detects any ZFS pools that can be imported:
 
 sudo zpool import
 This command will list all pools that are available for import, including the one stored in /dev/sdf2. Look for the pool name you want to import.
  Import the pool: If you see the pool listed and you know its name (let's say the pool name is mypool), you can import it with:
 
 sudo zpool import mypool
  Import the pool from a specific device: If the pool isn't showing up or you want to specify the device directly, you can use:
 
 sudo zpool import -d /dev/sdf2
 This tells ZFS to look specifically at /dev/sdf2 for any pools. If you don't know the name of the pool this is also the command to run.
 This should list any pools found on the device. If it shows a pool, import it using:
 sudo zpool import -d /dev/sdf2 <pool_name>
  Mount the pool: Once the pool is imported, ZFS should automatically mount any datasets associated with the pool. You can check the status of the pool with:
 
 sudo zpool status
 Additional options:
  If the pool was exported cleanly, you can use zpool import without additional flags.
 If the pool wasn’t properly exported or was interrupted, you might need to use -f (force) to import it:
 
 
sudo zpool import -f mypool
 feat(linux_snippets#Create a systemd service for a non-root user): Create a systemd service for a non-root user
 To set up a systemd service as a non-root user, you can create a user-specific service file under your home directory. User services are defined in ~/.config/systemd/user/ and can be managed without root privileges.
  Create the service file:
 
 Open a terminal and create a new service file in ~/.config/systemd/user/. For example, if you want to create a service for a script named my_script.py, follow these steps:
 mkdir -p ~/.config/systemd/user
nano ~/.config/systemd/user/my_script.service
  Edit the service file:
 
 In the my_script.service file, add the following configuration:
 [Unit]
Description=My Python Script Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /path/to/your/script/my_script.py
WorkingDirectory=/path/to/your/script/
SyslogIdentifier=my_script
Restart=on-failure
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target
  Description: A short description of what the service does.
  ExecStart: The command to run your script. Replace /path/to/your/script/my_script.py with the full path to your Python script. If you want to run the script within a virtualenv you can use /path/to/virtualenv/bin/python instead of /usr/bin/python3.
 You'll need to add the virtualenv path to Path 
# Add virtualenv's bin directory to PATH
Environment="PATH=/path/to/virtualenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
 - WorkingDirectory: Set the working directory to where your script is located (optional). - Restart: Restart the service if it fails. - StandardOutput and StandardError: This ensures that the output is captured in the systemd journal. - WantedBy: Specifies the target to which this service belongs. default.target is commonly used for user services.
 
  Reload systemd to recognize the new service:
 
 
 Run the following command to reload systemd's user service files:
 systemctl --user daemon-reload
  Enable and start the service:
 
 To start the service immediately and enable it to run on boot (for your user session), use the following commands:
 systemctl --user start my_script.service
systemctl --user enable my_script.service
   Check the status and logs:
 
  To check if the service is running:
 systemctl --user status my_script.service
 
  To view logs specific to your service:
 journalctl --user -u my_script.service -f
 
 
 If you need to use the graphical interface
 If your script requires user interaction (like entering a GPG passphrase), it’s crucial to ensure that the service is tied to your graphical user session, which ensures that prompts can be displayed and interacted with.
 To handle this situation, you should make a few adjustments to your systemd service:
 Ensure service is bound to graphical session
 Change the WantedBy target to graphical-session.target instead of default.target. This makes sure the service waits for the full graphical environment to be available.
 Use Type=forking instead of Type=simple (optional)
 If you need the service to wait until the user is logged in and has a desktop session ready, you might need to tweak the service type. Usually, Type=simple is fine, but you can also experiment with Type=forking if you notice any issues with user prompts.
 Here’s how you should modify your mbsync_syncer.service file:
 [Unit]
Description=My Python Script Service
After=graphical-session.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /path/to/your/script/my_script.py
WorkingDirectory=/path/to/your/script/
Restart=on-failure
StandardOutput=journal
StandardError=journal
SyslogIdentifier=my_script
Environment="DISPLAY=:0"
Environment="DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"

[Install]
WantedBy=graphical-session.target
 After modifying the service, reload and restart it:
 systemctl --user daemon-reload
systemctl --user restart my_script.service
 
 
 ZFS Prometheus exporter
   Correction: Tweak the zfs_exporter target not available error.
 Remember to set the scrape_timeout to at least of 60s as the exporter is sometimes slow to answer, specially on low hardware resources.
  - job_name: zfs_exporter
   metrics_path: /metrics
   scrape_timeout: 60s
   static_configs:
   - targets: [192.168.3.236:9134]
   metric_relabel_configs:
   ...
 
  Correction: Tweak the ZfsPoolUnhealthy alert.
    - alert: ZfsPoolUnhealthy
     expr: last_over_time(zfs_pool_health[1h]) > 0
     for: 5m
     labels:
       severity: critical
 
 
 Resilience
   New: Introduce linux resilience.
 Increasing the resilience of the servers is critical when hosting services for others. This is the roadmap I'm following for my servers.
 Autostart services if the system reboots Using init system services to manage your services
 **Get basic metrics traceability and alerts ** Set up Prometheus with:
  The blackbox exporter to track if the services are available to your users and to monitor SSL certificates health.
 The node exporter to keep track on the resource usage of your machines and set alerts to get notified when concerning events happen (disks are getting filled, CPU usage is too high)
 
 **Get basic logs traceability and alerts **
 Set up Loki and clear up your system log errors.
 Improve the resilience of your data If you're still using ext4 for your filesystems instead of zfs you're missing a big improvement. To set it up:
  Plan your zfs storage architecture
 Install ZFS
 Create ZFS local and remote backups
 [Monitor your ZFS ]
 
 Automatically react on system failures - Kernel panics - watchdog
 Future undeveloped improvements - Handle the system reboots after kernel upgrades
 
 
 Memtest
   New: Introduce memtest.
 memtest86 is a testing software for RAM.
 Installation 
apt-get install memtest86+
 After the installation you'll get Memtest entries in grub which you can spawn.
 For some unknown reason the memtest of the boot menu didn't work for me. So I downloaded the latest free version of memtest (It's at the bottom of the screen), burnt it in a usb and booted from there.
 Usage It will run by itself. For 64GB of ECC RAM it took aproximately 100 minutes to run all the tests.
 Check ECC errors MemTest86 directly polls ECC errors logged in the chipset/memory controller registers and displays it to the user on-screen. In addition, ECC errors are written to the log and report file.
 
 
 watchdog
   New: Introduce the watchdog.
 A watchdog timer (WDT, or simply a watchdog), sometimes called a computer operating properly timer (COP timer), is an electronic or software timer that is used to detect and recover from computer malfunctions. Watchdog timers are widely used in computers to facilitate automatic correction of temporary hardware faults, and to prevent errant or malevolent software from disrupting system operation.
 During normal operation, the computer regularly restarts the watchdog timer to prevent it from elapsing, or "timing out". If, due to a hardware fault or program error, the computer fails to restart the watchdog, the timer will elapse and generate a timeout signal. The timeout signal is used to initiate corrective actions. The corrective actions typically include placing the computer and associated hardware in a safe state and invoking a computer reboot.
 Microcontrollers often include an integrated, on-chip watchdog. In other computers the watchdog may reside in a nearby chip that connects directly to the CPU, or it may be located on an external expansion card in the computer's chassis.
 Hardware watchdog
 Before you start using the hardware watchdog you need to check if your hardware actually supports it.
 If you see Watchdog hardware is disabled error on boot things are not looking good.
 Check if the hardware watchdog is enabled You can see if hardware watchdog is loaded by running wdctl. For example for a machine that has it enabled you'll see:
 Device:        /dev/watchdog0
Identity:      iTCO_wdt [version 0]
Timeout:       30 seconds
Pre-timeout:    0 seconds
Timeleft:      30 seconds
FLAG           DESCRIPTION               STATUS BOOT-STATUS
KEEPALIVEPING  Keep alive ping reply          1           0
MAGICCLOSE     Supports magic close char      0           0
SETTIMEOUT     Set timeout (in seconds)       0           0
 On a machine that doesn't you'll see:
 wdctl: No default device is available.: No such file or directory
 Another option is to run dmesg | grep wd or dmesg | grep watc -i. For example for a machine that has enabled the hardware watchdog you'll see something like:
 [   20.708839] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   20.708894] iTCO_wdt: Found a Intel PCH TCO device (Version=4, TCOBASE=0x0400)
[   20.709009] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
 For one that is not you'll see:
 [    1.934999] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver
[    1.935057] sp5100-tco sp5100-tco: Using 0xfed80b00 for watchdog MMIO address
[    1.935062] sp5100-tco sp5100-tco: Watchdog hardware is disabled
 If you're out of luck and your hardware doesn't support it you can delegate the task to the software watchdog or get some usb watchdog
 Systemd watchdog
 Starting with version 183 systemd provides full support for hardware watchdogs (as exposed in /dev/watchdog to userspace), as well as supervisor (software) watchdog support for invidual system services. The basic idea is the following: if enabled, systemd will regularly ping the watchdog hardware. If systemd or the kernel hang this ping will not happen anymore and the hardware will automatically reset the system. This way systemd and the kernel are protected from boundless hangs -- by the hardware. To make the chain complete, systemd then exposes a software watchdog interface for individual services so that they can also be restarted (or some other action taken) if they begin to hang. This software watchdog logic can be configured individually for each service in the ping frequency and the action to take. Putting both parts together (i.e. hardware watchdogs supervising systemd and the kernel, as well as systemd supervising all other services) we have a reliable way to watchdog every single component of the system.
 Configuring the watchdog To make use of the hardware watchdog it is sufficient to set the RuntimeWatchdogSec= option in /etc/systemd/system.conf. It defaults to 0 (i.e. no hardware watchdog use). Set it to a value like 20s and the watchdog is enabled. After 20s of no keep-alive pings the hardware will reset itself. Note that systemd will send a ping to the hardware at half the specified interval, i.e. every 10s.
 Note that the hardware watchdog device (/dev/watchdog) is single-user only. That means that you can either enable this functionality in systemd, or use a separate external watchdog daemon, such as the aptly named watchdog. Although the built-in hardware watchdog support of systemd does not conflict with other watchdog software by default. systemd does not make use of /dev/watchdog by default, and you are welcome to use external watchdog daemons in conjunction with systemd, if this better suits your needs.
 ShutdownWatchdogSec=`` is another option that can be configured in/etc/systemd/system.conf`. It controls the watchdog interval to use during reboots. It defaults to 10min, and adds extra reliability to the system reboot logic: if a clean reboot is not possible and shutdown hangs, we rely on the watchdog hardware to reset the system abruptly, as extra safety net.
 Now, let's have a look how to add watchdog logic to individual services.
 First of all, to make software watchdog-supervisable it needs to be patched to send out "I am alive" signals in regular intervals in its event loop. Patching this is relatively easy. First, a daemon needs to read the WATCHDOG_USEC= environment variable. If it is set, it will contain the watchdog interval in usec formatted as ASCII text string, as it is configured for the service. The daemon should then issue sd_notify("WATCHDOG=1") calls every half of that interval. A daemon patched this way should transparently support watchdog functionality by checking whether the environment variable is set and honouring the value it is set to.
 To enable the software watchdog logic for a service (which has been patched to support the logic pointed out above) it is sufficient to set the WatchdogSec= to the desired failure latency. See systemd.service(5) for details on this setting. This causes WATCHDOG_USEC= to be set for the service's processes and will cause the service to enter a failure state as soon as no keep-alive ping is received within the configured interval.
 The next step is to configure whether the service shall be restarted and how often, and what to do if it then still fails. To enable automatic service restarts on failure set Restart=on-failure for the service. To configure how many times a service shall be attempted to be restarted use the combination of StartLimitBurst= and StartLimitInterval= which allow you to configure how often a service may restart within a time interval. If that limit is reached, a special action can be taken. This action is configured with StartLimitAction=. The default is a none, i.e. that no further action is taken and the service simply remains in the failure state without any further attempted restarts. The other three possible values are reboot, reboot-force and reboot-immediate.
  reboot attempts a clean reboot, going through the usual, clean shutdown logic.
 reboot-force is more abrupt: it will not actually try to cleanly shutdown any services, but immediately kills all remaining services and unmounts all file systems and then forcibly reboots (this way all file systems will be clean but reboot will still be very fast).
 reboot-immediate does not attempt to kill any process or unmount any file systems. Instead it just hard reboots the machine without delay. reboot-immediate hence comes closest to a reboot triggered by a hardware watchdog. All these settings are documented in systemd.service(5).
 
 Putting this all together we now have pretty flexible options to watchdog-supervise a specific service and configure automatic restarts of the service if it hangs, plus take ultimate action if that doesn't help.
 Here's an example unit file:
 [Unit]
Description=My Little Daemon
Documentation=man:mylittled(8)

[Service]
ExecStart=/usr/bin/mylittled
WatchdogSec=30s
Restart=on-failure
StartLimitInterval=5min
StartLimitBurst=4
StartLimitAction=reboot-force
````

This service will automatically be restarted if it hasn't pinged the system manager for longer than 30s or if it fails otherwise. If it is restarted this way more often than 4 times in 5min action is taken and the system quickly rebooted, with all file systems being clean when it comes up again.

To write the code of the watchdog service you can follow one of these guides:

- [Python based watchdog](https://sleeplessbeastie.eu/2022/08/15/how-to-create-watchdog-for-systemd-service/)
- [Bash based watchdog](https://www.medo64.com/2019/01/systemd-watchdog-for-any-service/)

**[Testing a watchdog](https://serverfault.com/questions/375220/how-to-check-what-if-hardware-watchdogs-are-available-in-linux)**
One simple way to test a watchdog is to trigger a kernel panic. This can be done as root with:

```bash
echo c > /proc/sysrq-trigger
 The kernel will stop responding to the watchdog pings, so the watchdog will trigger.
 SysRq is a 'magical' key combo you can hit which the kernel will respond to regardless of whatever else it is doing, unless it is completely locked up. It can also be used by echoing letters to /proc/sysrq-trigger, like we're doing here.
 In this case, the letter c means perform a system crash and take a crashdump if configured.
 Troubleshooting
 Watchdog hardware is disabled error on boot
 According to the discussion at the kernel mailing list it means that the system contains hardware watchdog but it has been disabled (probably by BIOS) and Linux cannot enable the hardware.
 If your BIOS doesn't have a switch to enable it, consider the watchdog hardware broken for your system.
 Some people are blacklisting the module so that it's not loaded and therefore it doesn't return the error (1, 2
 References
  0pointer post on systemd watchdogs
 Heckel post on how to reboot using watchdogs
 
 
 
 Magic keys
   New: Introduce the Magic Keys.
 The magic SysRq key is a key combination understood by the Linux kernel, which allows the user to perform various low-level commands regardless of the system's state. It is often used to recover from freezes, or to reboot a computer without corrupting the filesystem.[1] Its effect is similar to the computer's hardware reset button (or power switch) but with many more options and much more control.
 This key combination provides access to powerful features for software development and disaster recovery. In this sense, it can be considered a form of escape sequence. Principal among the offered commands are means to forcibly unmount file systems, kill processes, recover keyboard state, and write unwritten data to disk. With respect to these tasks, this feature serves as a tool of last resort.
 The magic SysRq key cannot work under certain conditions, such as a kernel panic[2] or a hardware failure preventing the kernel from running properly.
 The key combination consists of Alt+Sys Req and another key, which controls the command issued.
 On some devices, notably laptops, the Fn key may need to be pressed to use the magic SysRq key.
 Reboot the machine
 A common use of the magic SysRq key is to perform a safe reboot of a Linux computer which has otherwise locked up (abbr. REISUB). This can prevent a fsck being required on reboot and gives some programs a chance to save emergency backups of unsaved work. The QWERTY (or AZERTY) mnemonics: "Raising Elephants Is So Utterly Boring", "Reboot Even If System Utterly Broken" or simply the word "BUSIER" read backwards, are often used to remember the following SysRq-keys sequence:
  unRaw (take control of keyboard back from X),
 tErminate (send SIGTERM to all processes, allowing them to terminate gracefully),
 kIll (send SIGKILL to all processes, forcing them to terminate immediately),
 Sync (flush data to disk),
 Unmount (remount all filesystems read-only),
 reBoot.
 
 When magic SysRq keys are used to kill a frozen graphical program, the program has no chance to restore text mode. This can make everything unreadable. The commands textmode (part of SVGAlib) and the reset command can restore text mode and make the console readable again.
 On distributions that do not include a textmode executable, the key command Ctrl+Alt+F1 may sometimes be able to force a return to a text console. (Use F1, F2, F3,..., F(n), where n is the highest number of text consoles set up by the distribution. Ctrl+Alt+F(n+1) would normally be used to reenter GUI mode on a system on which the X server has not crashed.)
 Interact with the sysrq through the commandline It can also be used by echoing letters to /proc/sysrq-trigger, for example to trigger a system crash and take a crashdump you can:
 echo c > /proc/sysrq-trigger
 
 
 Monitoring
 Loki
   New: Prevent the too many outstanding requests error.
 Add to your loki config the next options 
limits_config:
  split_queries_by_interval: 24h
  max_query_parallelism: 100

query_scheduler:
  max_outstanding_requests_per_tenant: 4096

frontend:
  max_outstanding_per_tenant: 4096
 
  Correction: Use fake when using one loki instance in docker.
 If you only have one Loki instance you need to save the rule yaml files in the /etc/loki/rules/fake/ otherwise Loki will silently ignore them (it took me a lot of time to figure this out -.-).
 
  New: Add alerts.
 Surprisingly I haven't found any compilation of Loki alerts. I'll gather here the ones I create.
 There are two kinds of rules: alerting rules and recording rules.
  ECC error alerts
 ZFS errors
 Sanoid errors
 
 
  New: Alert when query returns no data.
 Sometimes the queries you want to alert happen when the return value is NaN or No Data. For example if you want to monitory the happy path by setting an alert if a string is not found in some logs in a period of time.
 count_over_time({filename="/var/log/mail.log"} |= `Mail is sent` [24h]) < 1
 This won't trigger the alert because the count_over_time doesn't return a 0 but a NaN. One way to solve it is to use the vector(0) operator with the operation or on() vector(0) 
(count_over_time({filename="/var/log/mail.log"} |= `Mail is sent` [24h]) or on() vector(0)) < 1
 
  New: Monitor loki metrics.
 Since Loki reuses the Prometheus code for recording rules and WALs, it also gains all of Prometheus’ observability.
 To scrape loki metrics with prometheus add the next snippet to the prometheus configuration:
   - job_name: loki
    metrics_path: /metrics
    static_configs:
    - targets:
      - loki:3100
 This assumes that loki is a docker in the same network as prometheus.
 There are some rules in the awesome prometheus alerts repo
 ---
groups:
- name: Awesome Prometheus loki alert rules
  # https://samber.github.io/awesome-prometheus-alerts/rules#loki
  rules:
  - alert: LokiProcessTooManyRestarts
    expr: changes(process_start_time_seconds{job=~".*loki.*"}[15m]) > 2
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: Loki process too many restarts (instance {{ $labels.instance }})
      description: "A loki process had too many restarts (target {{ $labels.instance }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - alert: LokiRequestErrors
    expr: 100 * sum(rate(loki_request_duration_seconds_count{status_code=~"5.."}[1m])) by (namespace, job, route) / sum(rate(loki_request_duration_seconds_count[1m])) by (namespace, job, route) > 10
    for: 15m
    labels:
      severity: critical
    annotations:
      summary: Loki request errors (instance {{ $labels.instance }})
      description: "The {{ $labels.job }} and {{ $labels.route }} are experiencing errors\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - alert: LokiRequestPanic
    expr: sum(increase(loki_panic_total[10m])) by (namespace, job) > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: Loki request panic (instance {{ $labels.instance }})
      description: "The {{ $labels.job }} is experiencing {{ printf \"%.2f\" $value }}% increase of panics\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - alert: LokiRequestLatency
    expr: (histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket{route!~"(?i).*tail.*"}[5m])) by (le)))  > 1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: Loki request latency (instance {{ $labels.instance }})
      description: "The {{ $labels.job }} {{ $labels.route }} is experiencing {{ printf \"%.2f\" $value }}s 99th percentile latency\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
 And there are some guidelines on the rest of the metrics in the grafana documentation
 Monitor the ruler
 Prometheus exposes a number of metrics for its WAL implementation, and these have all been prefixed with loki_ruler_wal_.
 For example: prometheus_remote_storage_bytes_total → loki_ruler_wal_prometheus_remote_storage_bytes_total
 Additional metrics are exposed, also with the prefix loki_ruler_wal_. All per-tenant metrics contain a tenant label, so be aware that cardinality could begin to be a concern if the number of tenants grows sufficiently large.
 Some key metrics to note are:
  loki_ruler_wal_appender_ready: whether a WAL appender is ready to accept samples (1) or not (0)
 loki_ruler_wal_prometheus_remote_storage_samples_total: number of samples sent per tenant to remote storage
 loki_ruler_wal_prometheus_remote_storage_samples_pending_total: samples buffered in memory, waiting to be sent to remote storage
 loki_ruler_wal_prometheus_remote_storage_samples_failed_total: samples that failed when sent to remote storage
 loki_ruler_wal_prometheus_remote_storage_samples_dropped_total: samples dropped by relabel configurations
 loki_ruler_wal_prometheus_remote_storage_samples_retried_total: samples re-resent to remote storage
 loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds: highest timestamp of sample appended to WAL
 loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds: highest timestamp of sample sent to remote storage.
 
 
  New: Get a useful Source link in the alertmanager.
 This still doesn't work. Currently for the ruler external_url if you use the URL of your Grafana installation: e.g. external_url: "https://grafana.example.com" it creates a Source link in alertmanager similar to https://grafana.example.com/graph?g0.expr=%28sum+by%28thing%29%28count_over_time%28%7Bnamespace%3D%22foo%22%7D+%7C+json+%7C+bar%3D%22maxRetries%22%5B5m%5D%29%29+%3E+0%29&g0.tab=1, which isn't valid.
 This url templating (via /graph?g0.expr=%s&g0.tab=1) appears to be coming from prometheus. There is not a workaround yet
 
  New: Maximum of series reached for a single query.
 Go to the loki-local-config.yaml, then find the limits_config configuration. Then modify this to the limits_config:
 limits_config:
   max_query_series: 100000
 But probably you're doing something wrong. feat(orgzly#Both local and remote notebook have been modified): Both local and remote notebook have been modified
 You can force load or force save a single note with a long tap.
 
  Correction: Don't use vector(0) on aggregation over labels.
 If you're doing an aggregation over a label this approach won't work because it will add a new time series with value 0. In those cases use a broader search that includes other logs from the label you're trying to aggregate and multiply it by 0. For example:
 (
sum by (hostname) (
  count_over_time({job="systemd-journal", syslog_identifier="sanoid"}[1h])
)
or
sum by (hostname) (
  count_over_time({job="systemd-journal"}[1h]) * 0
)
) < 1
 The first part of the query returns all log lines of the service sanoid for each hostname. If one hostname were not to return any line that query alone won't show anything for that host. The second part of the query counts all the log lines of each hostname, so if it's up it will probably be sending at least one line per hour. As we're not interested in those number of lines we multiply it by 0, so that the target is shown.
 
  New: Interact with loki through python.
 There is no client library for python (1, 2) they suggest to interact with the API with requests. Although I'd rather use logcli with the sh library.
 
  New: Download the logs.
 The web UI only allows you to download the logs that are loaded in the view, if you want to download big amounts of logs you need to either use logcli or interact with the API.
 One user did a query on loop:
 set -x

JOB_ID=9079dc54-2f5c-4d74-a9aa-1d9eb39dd3c2

for I in `seq 0 655`; do
    FILE=logs_$I.txt
    ID="$JOB_ID:$I"
    QUERY="{aws_job_id=\"$ID\",job=\"varlogs\"}"
    docker run grafana/logcli:main-1b6d0bf-amd64 --addr=http://localhost:3100/ -o raw -q query $QUERY --limit 100000 --batch 100 --forward --from "2022-09-25T10:00:00Z" > $FILE
done
 
 
 Logcli
   New: Introduce Promtail.
 Promtail is an agent which ships the contents of local logs to a Loki instance.
 It is usually deployed to every machine that runs applications which need to be monitored.
 It primarily:
  Discovers targets
 Attaches labels to log streams
 Pushes them to the Loki instance.
 
 
  New: Scrape journald logs.
 On systems with systemd, Promtail also supports reading from the journal. Unlike file scraping which is defined in the static_configs stanza, journal scraping is defined in a journal stanza:
 scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 12h
      path: /var/log/journal
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: unit
      - source_labels: ['__journal__hostname']
        target_label: hostname
      - source_labels: ['__journal_syslog_identifier']
        target_label: syslog_identifier
      - source_labels: ['__journal_transport']
        target_label: transport
      - source_labels: ['__journal_priority_keyword']
        target_label: keyword
 All fields defined in the journal section are optional, and are just provided here for reference.
  max_age ensures that no older entry than the time specified will be sent to Loki; this circumvents entry too old errors.
 path tells Promtail where to read journal entries from.
 labels map defines a constant list of labels to add to every journal entry that Promtail reads.
 matches field adds journal filters. If multiple filters are specified matching different fields, the log entries are filtered by both, if two filters apply to the same field, then they are automatically matched as alternatives.
 When the json field is set to true, messages from the journal will be passed through the pipeline as JSON, keeping all of the original fields from the journal entry. This is useful when you don’t want to index some fields but you still want to know what values they contained.
 When Promtail reads from the journal, it brings in all fields prefixed with __journal_ as internal labels. Like in the example above, the _SYSTEMD_UNIT field from the journal was transformed into a label called unit through relabel_configs. Keep in mind that labels prefixed with __ will be dropped, so relabeling is required to keep these labels. Look at the systemd man pages for a list of fields exposed by the journal.
 
 By default, Promtail reads from the journal by looking in the /var/log/journal and /run/log/journal paths. If running Promtail inside of a Docker container, the path appropriate to your distribution should be bind mounted inside of Promtail along with binding /etc/machine-id. Bind mounting /etc/machine-id to the path of the same name is required for the journal reader to know which specific journal to read from.
 docker run \
  -v /var/log/journal/:/var/log/journal/ \
  -v /run/log/journal/:/run/log/journal/ \
  -v /etc/machine-id:/etc/machine-id \
  grafana/promtail:latest \
  -config.file=/path/to/config/file.yaml
 
  New: Scrape docker logs.
 Docker service discovery allows retrieving targets from a Docker daemon. It will only watch containers of the Docker daemon referenced with the host parameter. Docker service discovery should run on each node in a distributed setup. The containers must run with either the json-file or journald logging driver.
 Note that the discovery will not pick up finished containers. That means Promtail will not scrape the remaining logs from finished containers after a restart.
 
scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_id']
        target_label: docker_id
      - source_labels: ['__meta_docker_container_name']
        target_label: docker_name
 The available meta labels are:
  __meta_docker_container_id: the ID of the container
 __meta_docker_container_name: the name of the container
 __meta_docker_container_network_mode: the network mode of the container
 __meta_docker_container_label_<labelname>: each label of the container
 __meta_docker_container_log_stream: the log stream type stdout or stderr
 __meta_docker_network_id: the ID of the network
 __meta_docker_network_name: the name of the network
 __meta_docker_network_ingress: whether the network is ingress
 __meta_docker_network_internal: whether the network is internal
 __meta_docker_network_label_<labelname>: each label of the network
 __meta_docker_network_scope: the scope of the network
 __meta_docker_network_ip: the IP of the container in this network
 __meta_docker_port_private: the port on the container
 __meta_docker_port_public: the external port if a port-mapping exists
 __meta_docker_port_public_ip: the public IP if a port-mapping exists
 
 These labels can be used during relabeling. For instance, the following configuration scrapes the container named flog and removes the leading slash (/) from the container name. yaml
 scrape_configs:
  - job_name: flog_scrape
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: name
            values: [flog]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
 
  New: Pipeline building.
 In this issue there are nice examples on different pipelines.
 
  New: Drop logs.
 If you don't want the logs that have the keyword systemd-journal and value docker-compose you can add the next pipeline stage:
 pipeline_stages:
  - drop:
      source: syslog_identifier
      value: docker-compose
 
  New: Install with ansible.
 Use patrickjahns ansible role. Some interesting variables are:
 loki_url: localhost
promtail_system_user: root

promtail_config_clients:
  - url: "http://{{ loki_url }}:3100/loki/api/v1/push"
    external_labels:
      hostname: "{{ ansible_hostname }}"
 
  New: Troubleshooting promtail.
 Find where is the positions.yaml file and see if it evolves.
 Sometimes if you are not seeing the logs in loki it's because the query you're running is not correct.
 
  New: Set the hostname label on all logs.
 There are many ways to do it:
  Setting the label in the promtail launch command bash sudo ./promtail-linux-amd64 --client.url=http://xxxx:3100/loki/api/v1/push --client.external-labels=hostname=$(hostname) --config.file=./config.yaml
 
 This won't work if you're using promtail within a docker-compose because you can't use bash expansion in the docker-compose.yaml file - Allowing env expansion and setting it in the promtail conf. You can launch the promtail command with -config.expand-env and then set in each scrape jobs: 
labels:
    host: ${HOSTNAME}
 This won't work either if you're using promtail within a docker as it will give you the ID of the docker - Set it in the promtail_config_clients field as external_labels of each promtail config: promtail_config_clients:
  - url: "http://{{ loki_url }}:3100/loki/api/v1/push"
    external_labels:
      hostname: "{{ ansible_hostname }}"
 - Hardcode it for each promtail config scraping config as static labels. If you're using ansible or any deployment method that supports jinja expansion set it that way labels:
    host: {{ ansible_hostname }}
 
  New: Introduce logcli.
 logcli is the command-line interface to Grafana Loki. It facilitates running LogQL queries against a Loki instance.
 Installation Download the logcli binary from the Loki releases page and install it somewhere in your $PATH.
 Usage logcli points to the local instance http://localhost:3100 directly, if you want another one export the LOKI_ADDR environment variable.
 Run a query:
 logcli query '{job="loki-ops/consul"}'
 You can also set the time range and output format
 logcli query \
     --timezone=UTC  \
     --from="2024-06-10T07:23:36Z" \
     --to="2024-06-12T16:23:58Z" \
     --output=jsonl \
     '{job="docker", container="aleph_ingest-file_1"} | json | __error__=`` | severity =~ `WARNING|ERROR` | message !~ `Queueing failed task for retry.*` | logger!=`ingestors.manager`'
 References
  Docs
 
 
 
 Grafana
   New: Copy panels between dashboards.
 On each panel on the top right you can select copy, then on the menu to add a new panel you can click on Paste panel from clipboard.
 So far you can't do this for rows.
 
 
 AlertManager
   Correction: Add another source on how to silence alerts.
 If previous guidelines don't work for you, you can use the sleep peacefully guidelines to tackle it at query level.
 
  New: Using time intervals.
 The values of time_intervals can be:
 - times:
  [ - <time_range> ...]
  weekdays:
  [ - <weekday_range> ...]
  days_of_month:
  [ - <days_of_month_range> ...]
  months:
  [ - <month_range> ...]
  years:
  [ - <year_range> ...]
  location: <string>
 All fields are lists. Within each non-empty list, at least one element must be satisfied to match the field. If a field is left unspecified, any value will match the field. For an instant of time to match a complete time interval, all fields must match. Some fields support ranges and negative indices, and are detailed below. If a time zone is not specified, then the times are taken to be in UTC.
   time_range: Ranges inclusive of the starting time and exclusive of the end time to make it easy to represent times that start/end on hour boundaries. For example, start_time: '17:00' and end_time: '24:00' will begin at 17:00 and finish immediately before 24:00. They are specified like so: 
times:
- start_time: HH:MM
  end_time: HH:MM
 
  weekday_range: A list of days of the week, where the week begins on Sunday and ends on Saturday. Days should be specified by name (e.g. 'Sunday'). For convenience, ranges are also accepted of the form <start_day>:<end_day> and are inclusive on both ends. For example: ['monday:wednesday','saturday', 'sunday']
 
  days_of_month_range: A list of numerical days in the month. Days begin at 1. Negative values are also accepted which begin at the end of the month, e.g. -1 during January would represent January 31. For example: ['1:5', '-3:-1']. Extending past the start or end of the month will cause it to be clamped. E.g. specifying ['1:31'] during February will clamp the actual end date to 28 or 29 depending on leap years. Inclusive on both ends.
 
  month_range: A list of calendar months identified by a case-insensitive name (e.g. 'January') or by number, where January = 1. Ranges are also accepted. For example, ['1:3', 'may:august', 'december']. Inclusive on both ends.
 
  year_range: A numerical list of years. Ranges are accepted. For example, ['2020:2022', '2030']. Inclusive on both ends.
 
  location: A string that matches a location in the IANA time zone database. For example, 'Australia/Sydney'. The location provides the time zone for the time interval. For example, a time interval with a location of 'Australia/Sydney' that contained something like:
 
 
 times:
- start_time: 09:00
  end_time: 17:00
weekdays: ['monday:friday']
 Would include any time that fell between the hours of 9:00AM and 5:00PM, between Monday and Friday, using the local time in Sydney, Australia. You may also use 'Local' as a location to use the local time of the machine where Alertmanager is running, or 'UTC' for UTC time. If no timezone is provided, the time interval is taken to be in UTC.
 
 
 Process Exporter
   New: Introduce the process exporter.
 process_exporter is a rometheus exporter that mines /proc to report on selected processes.
 References - Source - Grafana dashboard
 
 
 Databases
 PostgreSQL
   New: Store expensive calculation values in a postgresql database.
 First you need to think if you actually need to store the calculations or you can do them on the fly with views. If views are too slow you can either use materialized views or triggers over calculation tables.
 Materialized views are simpler to maintain but have some disadvantages such as outdated data or unneeded processing of data. If you need totally current information or if you don't want to periodically do the calculations on all the rows then triggers are probably the better solution.
 
  New: Drop all tables of a database.
 drop schema public cascade;
create schema public;
 
  New: Views.
 A view is a named query stored in the PostgreSQL database server. A view is defined based on one or more tables which are known as base tables, and the query that defines the view is referred to as a defining query.
 After creating a view, you can query data from it as you would from a regular table. Behind the scenes, PostgreSQL will rewrite the query against the view and its defining query, executing it to retrieve data from the base tables.
 Views do not store data except the materialized views. In PostgreSQL, you can create special views called materialized views that store data physically and periodically refresh it from the base tables.
 Simple views can be updatable.
 Advantages of views - Simplifying complex queries: Views help simplify complex queries. Instead of dealing with joins, aggregations, or filtering conditions, you can query from views as if they were regular tables.
 Typically, first, you create views based on complex queries and store them in the database. Then, you can use simple queries based on views instead of using complex queries.
   Logical data independence: If your applications use views, you can freely modify the structure of the base tables. In other words, views enable you to create a layer of abstraction over the underlying tables.
 
  Security and access control: Views enable fine-grained control over data access. You can create views that expose subsets of data in the base tables, hiding sensitive information.
 
 
 This is particularly useful when you have applications that require access to distinct portions of the data.
 Creating a view In PostgreSQL, a view is a named query stored in the database server. To create a new view, you can use the CREATE VIEW statement.
 CREATE VIEW view_name
AS
  query;
 In this syntax:
  Specify the name of the view after the CREATE VIEW keywords.
 Specify a SELECT statement (query) that defines the view. The query is often referred to as the defining query of the view.
 
 Creating a view examples
 We’ll use the customer table from the sample database:
 Basic CREATE VIEW statement example
 The following example uses the CREATE VIEW statement to create a view based on the customer table:
 CREATE VIEW contact AS
SELECT
  first_name,
  last_name,
  email
FROM
  customer;
 Output:
 CREATE VIEW
 The following query data from the contact view:
 SELECT * FROM contact;
 Output:
  first_name  |  last_name   |                  email
-------------+--------------+------------------------------------------
 Jared       | Ely          | jared.ely@sakilacustomer.org
 Mary        | Smith        | mary.smith@sakilacustomer.org
 Patricia    | Johnson      | patricia.johnson@sakilacustomer.org
...
 Using the CREATE VIEW statement to create a view based on a complex query
 The following example creates a view based on the tables customer, address, city, and country:
 CREATE VIEW customer_info AS
SELECT
  first_name,
  last_name,
  email,
  phone,
  city,
  postal_code,
  country
FROM
  customer
  INNER JOIN address USING (address_id)
  INNER JOIN city USING (city_id)
  INNER JOIN country USING (country_id);
 The following query retrieves data from the customer_info view:
 SELECT * FROM customer_info;
 Output:
  first_name  |  last_name   |                  email                   |    phone     |            city            | postal_code |                country
-------------+--------------+------------------------------------------+--------------+----------------------------+-------------+---------------------------------------
 Jared       | Ely          | jared.ely@sakilacustomer.org             | 35533115997  | Purwakarta                 | 25972       | Indonesia
 Mary        | Smith        | mary.smith@sakilacustomer.org            | 28303384290  | Sasebo                     | 35200       | Japan
 Patricia    | Johnson      | patricia.johnson@sakilacustomer.org      | 838635286649 | San Bernardino             | 17886       | United States
...
 Creating a view based on another view
 The following statement creates a view called customer_usa based on the customer_info view. The customer_usa returns the customers who are in the United States:
 CREATE VIEW customer_usa
AS
SELECT
  *
FROM
  customer_info
WHERE
  country = 'United States';
 Here’s the query that retrieves data from the customer_usa view:
 SELECT * FROM customer_usa;
 Output:
  first_name | last_name  |                email                 |    phone     |          city           | postal_code |    country
------------+------------+--------------------------------------+--------------+-------------------------+-------------+---------------
 Zachary    | Hite       | zachary.hite@sakilacustomer.org      | 191958435142 | Akron                   | 88749       | United States
 Richard    | Mccrary    | richard.mccrary@sakilacustomer.org   | 262088367001 | Arlington               | 42141       | United States
 Diana      | Alexander  | diana.alexander@sakilacustomer.org   | 6171054059   | Augusta-Richmond County | 30695       | United States
...
 Replacing a view Note: for simple changes check alter views
 To change the defining query of a view, you use the CREATE OR REPLACE VIEW statement:
 CREATE OR REPLACE VIEW view_name
AS
  query;
 In this syntax, you add the OR REPLACE between the CREATE and VIEW keywords. If the view already exists, the statement replaces the existing view; otherwise, it creates a new view.
 For example, the following statement changes the defining query of the contact view to include the phone information from the address table:
 CREATE OR REPLACE VIEW contact AS
SELECT
  first_name,
  last_name,
  email,
  phone
FROM
  customer
INNER JOIN address USING (address_id);
 Display a view on psql
 To display a view on psql, you follow these steps:
 First, open the Command Prompt on Windows or Terminal on Unix-like systems and connect to the PostgreSQL server:
 psql -U postgres
 Second, change the current database to dvdrental:
 \c dvdrental
 Third, display the view information using the \d+ view_name command. For example, the following shows the contact view:
 \d+ contact
 Output:
                                     View "public.contact"
   Column   |         Type          | Collation | Nullable | Default | Storage  | Description
------------+-----------------------+-----------+----------+---------+----------+-------------
 first_name | character varying(45) |           |          |         | extended |
 last_name  | character varying(45) |           |          |         | extended |
 email      | character varying(50) |           |          |         | extended |
 phone      | character varying(20) |           |          |         | extended |
View definition:
 SELECT customer.first_name,
    customer.last_name,
    customer.email,
    address.phone
   FROM customer
     JOIN address USING (address_id);
 Updatable views Recursive views Alter views
 
  New: Materialized Views.
 PostgreSQL extends the view concept to the next level which allows views to store data physically. These views are called materialized views.
 Materialized views cache the result set of an expensive query and allow you to refresh data periodically.
 The materialized views can be useful in many cases that require fast data access. Therefore, you often find them in data warehouses and business intelligence applications.
 Benefits of materialized views
  Improve query efficiency: If a query takes a long time to run, it could be because there are a lot of transformations being done to the data: subqueries, functions, and joins, for example.
 
 A materialized view can combine all of that into a single result set that’s stored like a table.
 This means that any user or application that needs to get this data can just query the materialized view itself, as though all of the data is in the one table, rather than running the expensive query that uses joins, functions, or subqueries.
 Calculations can also be added to materialized views for any fields you may need, which can save time, and are often not stored in the database.
  Simplify a query: Like a regular view, a materialized view can also be used to simplify a query. If a query is using a lot of logic such as joins and functions, using a materialized view can help remove some of that logic and place it into the materialized view.
 
 Disadvantages of a Materialized View - Updates to data need to be set up: The main disadvantage to using materialized views is that the data needs to be refreshed.
 The data that’s used to populate the materialized view is stored in the database tables. These tables can have their data updated, inserted, or deleted. When that happens, the data in the materialized view needs to be updated.
 This can be done manually, but it should be done automatically.
  Incremental updates are not supported: So the whole view is generated on each refresh.
 Data may be inconsistent: Because the data is stored separately in the materialized view, the data in the materialized view may be inconsistent with the data in the underlying tables.
 
 This may be an issue if you are expecting or relying on data to be consistent.
 However, for scenarios where it doesn’t matter (e.g. monthly reporting on months in the past), then it may be OK.
  Storage Requirements: Materialized Views can consume significant storage space, depending on the size of your dataset. This consideration is crucial, especially in resource-limited environments.
 
 Creating materialized views
 To create a materialized view, you use the CREATE MATERIALIZED VIEW statement as follows:
 CREATE MATERIALIZED VIEW [IF NOT EXISTS] view_name
AS
query
WITH [NO] DATA;
 How it works.
  First, specify the view_name after the CREATE MATERIALIZED VIEW clause
 Second, add the query that retrieves data from the underlying tables after the AS keyword.
 Third, if you want to load data into the materialized view at the creation time, use the WITH DATA option; otherwise, you use WITH NO DATA option. If you use the WITH NO DATA option, the view is flagged as unreadable. It means that you cannot query data from the view until you load data into it.
 Finally, use the IF NOT EXISTS option to conditionally create a view only if it does not exist.
 
 Refreshing data for materialized views
 Postgresql will never refresh the data by it's own, you need to define the processes that will update it.
 To load or update the data into a materialized view, you use the REFRESH MATERIALIZED VIEW statement:
 REFRESH MATERIALIZED VIEW view_name;
 When you refresh data for a materialized view, PostgreSQL locks the underlying tables. Consequently, you will not be able to retrieve data from underlying tables while data is loading into the view.
 To avoid this, you can use the CONCURRENTLY option.
 REFRESH MATERIALIZED VIEW CONCURRENTLY view_name;
 With the CONCURRENTLY option, PostgreSQL creates a temporary updated version of the materialized view, compares two versions, and performs INSERT and UPDATE only the differences.
 PostgreSQL allows you to retrieve data from a materialized view while it is being updated. One requirement for using CONCURRENTLY option is that the materialized view must have a UNIQUE index.
 Automatic update of materialized views
 Removing materialized views
 To remove a materialized view, you use the DROP MATERIALIZED VIEW statement:
 DROP MATERIALIZED VIEW view_name;
 In this syntax, you specify the name of the materialized view that you want to drop after the DROP MATERIALIZED VIEW keywords.
 Materialized view example We’ll use the tables in the sample database for creating a materialized view.
 First, create a materialized view named rental_by_category using the CREATE MATERIALIZED VIEW statement:
 CREATE MATERIALIZED VIEW rental_by_category
AS
 SELECT c.name AS category,
    sum(p.amount) AS total_sales
   FROM (((((payment p
     JOIN rental r ON ((p.rental_id = r.rental_id)))
     JOIN inventory i ON ((r.inventory_id = i.inventory_id)))
     JOIN film f ON ((i.film_id = f.film_id)))
     JOIN film_category fc ON ((f.film_id = fc.film_id)))
     JOIN category c ON ((fc.category_id = c.category_id)))
  GROUP BY c.name
  ORDER BY sum(p.amount) DESC
WITH NO DATA;
 Because of the WITH NO DATA option, you cannot query data from the view. If you attempt to do so, you’ll get the following error message:
 SELECT * FROM rental_by_category;
 Output:
 [Err] ERROR: materialized view "rental_by_category" has not been populated
HINT: Use the REFRESH MATERIALIZED VIEW command.
 PostgreSQL is helpful to give you a hint to ask for loading data into the view.
 Second, load data into the materialized view using the REFRESH MATERIALIZED VIEW statement:
 REFRESH MATERIALIZED VIEW rental_by_category;
 Third, retrieve data from the materialized view:
 SELECT * FROM rental_by_category;
 Output:
  category   | total_sales
-------------+-------------
 Sports      |     4892.19
 Sci-Fi      |     4336.01
 Animation   |     4245.31
 Drama       |     4118.46
 Comedy      |     4002.48
 New         |     3966.38
 Action      |     3951.84
 Foreign     |     3934.47
 Games       |     3922.18
 Family      |     3830.15
 Documentary |     3749.65
 Horror      |     3401.27
 Classics    |     3353.38
 Children    |     3309.39
 Travel      |     3227.36
 Music       |     3071.52
(16 rows)
 From now on, you can refresh the data in the rental_by_category view using the REFRESH MATERIALIZED VIEW statement.
 However, to refresh it with CONCURRENTLY option, you need to create a UNIQUE index for the view first.
 CREATE UNIQUE INDEX rental_category
ON rental_by_category (category);
 Let’s refresh data concurrently for the rental_by_category view.
 REFRESH MATERIALIZED VIEW CONCURRENTLY rental_by_category;
 
 
 Hardware
 ECC RAM
   New: Introduce ECC RAM.
 Error Correction Code (ECC) is a mechanism used to detect and correct errors in memory data due to environmental interference and physical defects. ECC memory is used in high-reliability applications that cannot tolerate failure due to corrupted data.
 Installation: Due to additional circuitry required for ECC protection, specialized ECC hardware support is required by the CPU chipset, motherboard and DRAM module. This includes the following:
  Server-grade CPU chipset with ECC support (Intel Xeon, AMD Ryzen)
 Motherboard supporting ECC operation
 ECC RAM
 
 Consult the motherboard and/or CPU documentation for the specific model to verify whether the hardware supports ECC. Use vendor-supplied list of certified ECC RAM, if provided.
 Most ECC-supported motherboards allow you to configure ECC settings from the BIOS setup. They are usually on the Advanced tab. The specific option depends on the motherboard vendor or model such as the following:
  DRAM ECC Enable (American Megatrends, ASUS, ASRock, MSI)
 ECC Mode (ASUS)
 
 Monitorization
 The mechanism for how ECC errors are logged and reported to the end-user depends on the BIOS and operating system. In most cases, corrected ECC errors are written to system/event logs. Uncorrected ECC errors may result in kernel panic or blue screen.
 The Linux kernel supports reporting ECC errors for ECC memory via the EDAC (Error Detection And Correction) driver subsystem. Depending on the Linux distribution, ECC errors may be reported by the following:
  rasdaemon: monitor ECC memory and report both correctable and uncorrectable memory errors on recent Linux kernels.
 mcelog (Deprecated): collects and decodes MCA error events on x86.
 edac-utils (Deprecated): fills DIMM labels data and summarizes memory errors.
 
 To configure rasdaemon follow this article
 Confusion on boards supporting ECC
 I've read that even if some motherboards say that they "Support ECC" some of them don't do anything with it.
 On this post and the kernel docs show that you should see references to ACPI/WHEA in the specs manual. Ideally ACPI5 support.
 From the ) EINJ provides a hardware error injection mechanism. It is very useful for debugging and testing APEI and RAS features in general.
 You need to check whether your BIOS supports EINJ first. For that, look for early boot messages similar to this one:
 ACPI: EINJ 0x000000007370A000 000150 (v01 INTEL           00000001 INTL 00000001)
 Which shows that the BIOS is exposing an EINJ table - it is the mechanism through which the injection is done.
 Alternatively, look in /sys/firmware/acpi/tables for an "EINJ" file, which is a different representation of the same thing.
 It doesn't necessarily mean that EINJ is not supported if those above don't exist: before you give up, go into BIOS setup to see if the BIOS has an option to enable error injection. Look for something called WHEA or similar. Often, you need to enable an ACPI5 support option prior, in order to see the APEI,EINJ,... functionality supported and exposed by the BIOS menu.
 To use EINJ, make sure the following are options enabled in your kernel configuration:
 CONFIG_DEBUG_FS
CONFIG_ACPI_APEI
CONFIG_ACPI_APEI_EINJ
 One way to test it can be to run memtest as it sometimes shows ECC errors such as ** Warning** ECC injection may be disabled for AMD Ryzen (70h-7fh).
 Other people (1, 2 say that there are a lot of motherboards that NEVER report any corrected errors to the OS. In order to see corrected errors, PFEH (Platform First Error Handling) has to be disabled. On some motherboards and FW versions this setting is hidden from the user and always enabled, thus resulting in zero correctable errors getting reported.
 They also suggest to disable "Quick Boot". In order to initialize ECC, memory has to be written before it can be used. Usually this is done by BIOS, but with some motherboards this step is skipped if "Quick Boot" is enabled.
 The people behind memtest have a paid tool to test ECC
 
  New: Introduce rasdaemon the ECC monitor.
 rasdaemon is a RAS (Reliability, Availability and Serviceability) logging tool. It records memory errors, using the EDAC tracing events. EDAC is a Linux kernel subsystem with handles detection of ECC errors from memory controllers for most chipsets on i386 and x86_64 architectures. EDAC drivers for other architectures like arm also exists.
 Installation 
apt-get install rasdaemon
 The output will be available via syslog but you can show it to the foreground (-f) or to an sqlite3 database (-r)
 To post-process and decode received MCA errors on AMD SMCA systems, run:
 rasdaemon -p --status <STATUS_reg> --ipid <IPID_reg> --smca --family <CPU Family> --model <CPU Model> --bank <BANK_NUM>
 Status and IPID Register values (in hex) are mandatory. The smca flag with family and model are required if not decoding locally. Bank parameter is optional.
 You may also start it via systemd:
 systemctl start rasdaemon
 The rasdaemon will then output the messages to journald.
 Usage
 At this point rasdaemon should already be running on your system. You can now use the ras-mc-ctl tool to query the errors that have been detected. If everything is well configured you'll see something like:
 $: ras-mc-ctl --error-count
Label                 CE      UE
mc#0csrow#2channel#0  0   0
mc#0csrow#2channel#1  0   0
mc#0csrow#3channel#1  0   0
mc#0csrow#3channel#0  0   0
 If it's not you'll see:
 ras-mc-ctl: Error: No DIMMs found in /sys or new sysfs EDAC interface not found.
 The CE column represents the number of corrected errors for a given DIMM, UE represents uncorrectable errors that were detected. The label on the left shows the EDAC path under /sys/devices/system/edac/mc/ of every DIMM. This is not very readable, if you wish to improve the labeling read this article
 More ways to check is to run:
 $: ras-mc-ctl --status
ras-mc-ctl: drivers are loaded.
 You can also see a summary of the state with:
 $: ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1183.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1184.
 Monitorization
 You can use loki to monitor ECC errors shown in the logs with the next alerts:
 
groups:
  - name: ecc
    rules:
      - alert: ECCError
        expr: |
          count_over_time({job="systemd-journal", unit="rasdaemon.service", level="error"} [5m])  > 0
        for: 1m
        labels:
            severity: critical
        annotations:
            summary: "Possible ECC error detected in {{ $labels.hostname}}"

      - alert: ECCWarning
        expr: |
          count_over_time({job="systemd-journal", unit="rasdaemon.service", level="warning"} [5m])  > 0
        for: 1m
        labels:
            severity: warning
        annotations:
            summary: "Possible ECC warning detected in {{ $labels.hostname}}"
      - alert: ECCAlert
        expr: |
          count_over_time({job="systemd-journal", unit="rasdaemon.service", level!~"info|error|warning"} [5m]) > 0
        for: 1m
        labels:
            severity: info
        annotations:
            summary: "ECC log trace with unknown severity level detected in {{ $labels.hostname}}"
 References
  Source
 
 
  New: Check if system is actually using ECC.
 Another way is to run dmidecode. For ECC support you'll see: 
$: dmidecode -t memory | grep ECC
  Error Correction Type: Single-bit ECC
  # or
  Error Correction Type: Multi-bit ECC
 No ECC:
 $: dmidecode -t memory | grep ECC
  Error Correction Type: None
 You can also test it with rasdaemon
 
 
 GPU
   New: Introduce GPU.
 GPU or Graphic Processing Unit is a specialized electronic circuit initially designed to accelerate computer graphics and image processing (either on a video card or embedded on motherboards, mobile phones, personal computers, workstations, and game consoles).
 For years I've wanted to buy a graphic card but I've been stuck in the problem that I don't have a desktop. I have a X280 lenovo laptop used to work and personal use with an integrated card that has let me so far to play old games such as King Arthur Gold or Age of Empires II, but has hard times playing "newer" games such as It takes two. Last year I also bought a NAS with awesome hardware. So it makes no sense to buy a desktop just for playing.
 Now that I host Jellyfin on the NAS and that machine learning is on the hype with a lot of interesting solutions that can be self-hosted (whisper, chatgpt similar solutions...), it starts to make sense to add a GPU to the server. What made me give the step is that you can also self-host a gaming server to stream to any device! It makes so much sense to have all the big guns inside the NAS and stream the content to the less powerful devices.
 That way if you host services, you make the most use of the hardware.
 
  New: Install cuda.
 CUDA is a parallel computing platform and programming model invented by NVIDIA®. It enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU). If you're not using Debian 11 follow these instructions
 Base Installer
 wget https://developer.download.nvidia.com/compute/cuda/12.5.1/local_installers/cuda-repo-debian11-12-5-local_12.5.1-555.42.06-1_amd64.deb
sudo dpkg -i cuda-repo-debian11-12-5-local_12.5.1-555.42.06-1_amd64.deb
sudo cp /var/cuda-repo-debian11-12-5-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo add-apt-repository contrib
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-5
 Additional installation options are detailed here.
 Driver Installer
 To install the open kernel module flavor:
 sudo apt-get install -y nvidia-kernel-open-dkms
sudo apt-get install -y cuda-drivers
 Install cuda:
 apt-get install cuda
reboot
 Install nvidia card
 Check if your card is supported in the releases supported by your OS - If it's supported - If it's not supported
 Ensure the GPUs are Installed
 Install pciutils:
 Ensure that the lspci command is installed (which lists the PCI devices connected to the server):
 sudo apt-get -y install pciutils
 Check Installed Nvidia Cards: Perform a quick check to determine what Nvidia cards have been installed:
 lspci | grep VGA
 The output of the lspci command above should be something similar to:
 00:02.0 VGA compatible controller: Intel Corporation 4th Gen ...
01:00.0 VGA compatible controller: Nvidia Corporation ...
 If you do not see a line that includes Nvidia, then the GPU is not properly installed. Otherwise, you should see the make and model of the GPU devices that are installed.
 Disable Nouveau
 Blacklist Nouveau in Modprobe: The nouveau driver is an alternative to the Nvidia drivers generally installed on the server. It does not work with CUDA and must be disabled. The first step is to edit the file at /etc/modprobe.d/blacklist-nouveau.conf.
 Create the file with the following content:
 cat <<EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off
EOF
 Then, run the following commands:
 echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
sudo update-initramfs -u
 Update Grub to Blacklist Nouveau:
 Backup your grub config template:
 sudo cp /etc/default/grub /etc/default/grub.bak
 Then, update your grub config template at /etc/default/grub. Add rd.driver.blacklist=nouveau and rcutree.rcu_idle_gp_delay=1 to the GRUB_CMDLINE_LINUX variable. For example, change:
 GRUB_CMDLINE_LINUX="quiet"
 to:
 GRUB_CMDLINE_LINUX="quiet rd.driver.blacklist=nouveau rcutree.rcu_idle_gp_delay=1"
 Then, rebuild your grub config:
 sudo grub2-mkconfig -o /boot/grub/grub.cfg
 Install prerequisites
 The following prerequisites should be installed before installing the Nvidia drivers:
 sudo apt-get -y install linux-headers-$(uname -r) make gcc-4.8
sudo apt-get -y install acpid dkms
 Close X Server:
 Before running the install, you should exit out of any X environment, such as Gnome, KDE, or XFCE. To exit the X session, switch to a TTY console using Ctrl-Alt-F1 and then determine whether you are running lightdm or gdm by running:
 sudo ps aux | grep "lightdm|gdm|kdm"
 Depending on which is running, stop the service, running the following commands (substitute gdm or kdm for lightdm as appropriate):
 sudo service lightdm stop
sudo init 3
 Install Drivers Only:
 To accommodate GL-accelerated rendering, OpenGL and GL Vendor Neutral Dispatch (GLVND) are now required and should be installed with the Nvidia drivers. OpenGL is an installation option in the *.run type of drivers. In other types of the drivers, OpenGL is enabled by default in most modern versions (dated 2016 and later). GLVND can be installed using the installer menus or via the --glvnd-glx-client command line flag.
 This section deals with installing the drivers via the *.run executables provided by Nvidia.
 To download only the drivers, navigate to http://www.nvidia.com/object/unix.html and click the Latest Long Lived Branch version under the appropriate CPU architecture. On the ensuing page, click Download and then click Agree and Download on the page that follows.
 The Unix drivers found in the link above are also compatible with all Nvidia Tesla models.
 If you'd prefer to download the full driver repository, Nvidia provides a tool to recommend the most recent available driver for your graphics card at http://www.Nvidia.com/Download/index.aspx?lang=en-us.
 If you are unsure which Nvidia devices are installed, the lspci command should give you that information:
 lspci | grep -i "nvidia"
 Download the recommended driver executable. Change the file permissions to allow execution:
 chmod +x ./NVIDIA-Linux-$(uname -m)-*.run
 Run the install.
 To check that the GPU is well installed and functioning properly, you can use the nvidia-smi command. This command provides detailed information about the installed Nvidia GPUs, including their status, utilization, and driver version.
 First, ensure the Nvidia drivers are installed. Then, run:
 nvidia-smi
 If the GPU is properly installed, you should see an output that includes information about the GPU, such as its model, memory usage, and driver version. The output will look something like this:
 +-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66       Driver Version: 450.66       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   38C    P8    29W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
 If you encounter any errors or the GPU is not listed, there may be an issue with the installation or configuration of the GPU drivers.
 Measure usage
 For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU.
 Load test the gpu
 First make sure you have CUDA installed, then install the gpu_burn tool 
git clone https://github.com/wilicc/gpu-burn
cd gpu-burn
make
 To run a test for 60 seconds run: 
./gpu_burn 60
 Monitor it with Prometheus
 NVIDIA DCGM is a set of tools for managing and monitoring NVIDIA GPUs in large-scale, Linux-based cluster environments. It’s a low overhead tool that can perform a variety of functions including active health monitoring, diagnostics, system validation, policies, power and clock management, group configuration, and accounting. For more information, see the DCGM User Guide.
 You can use DCGM to expose GPU metrics to Prometheus using dcgm-exporter.
  Install NVIDIA Container Kit: The NVIDIA Container Toolkit allows users to build and run GPU accelerated containers. The toolkit includes a container runtime library and utilities to automatically configure containers to leverage NVIDIA GPUs.
 
 sudo apt-get install -y nvidia-container-toolkit
  Configure the container runtime by using the nvidia-ctk command:
 
 
sudo nvidia-ctk runtime configure --runtime=docker
 - Restart the Docker daemon:
 sudo systemctl restart docker
  Install NVIDIA DCGM: Follow the Getting Started Guide.
 
 Determine the distribution name:
 distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
 Download the meta-package to set up the CUDA network repository:
 wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.1-1_all.deb
 Install the repository meta-data and the CUDA GPG key:
 sudo dpkg -i cuda-keyring_1.1-1_all.deb
 Update the Apt repository cache:
 sudo apt-get update
 Now, install DCGM:
 sudo apt-get install -y datacenter-gpu-manager
 Enable the DCGM systemd service (on reboot) and start it now:
 sudo systemctl --now enable nvidia-dcgm
 You should see output similar to this:
 ● dcgm.service - DCGM service
  Loaded: loaded (/usr/lib/systemd/system/dcgm.service; disabled; vendor preset: enabled)
  Active: active (running) since Mon 2020-10-12 12:18:57 PDT; 14s ago
Main PID: 32847 (nv-hostengine)
    Tasks: 7 (limit: 39321)
  CGroup: /system.slice/dcgm.service
          └─32847 /usr/bin/nv-hostengine -n

Oct 12 12:18:57 ubuntu1804 systemd[1]: Started DCGM service.
Oct 12 12:18:58 ubuntu1804 nv-hostengine[32847]: DCGM initialized
Oct 12 12:18:58 ubuntu1804 nv-hostengine[32847]: Host Engine Listener Started
 To verify installation, use dcgmi to query the system. You should see a listing of all supported GPUs (and any NVSwitches) found in the system:
 dcgmi discovery -l
 Output:
 8 GPUs found.
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information                                                   |
+--------+----------------------------------------------------------------------+
| 0      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:07:00.0                                         |
|        | Device UUID: GPU-1d82f4df-3cf9-150d-088b-52f18f8654e1                |
+--------+----------------------------------------------------------------------+
| 1      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:0F:00.0                                         |
|        | Device UUID: GPU-94168100-c5d5-1c05-9005-26953dd598e7                |
+--------+----------------------------------------------------------------------+
| 2      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:47:00.0                                         |
|        | Device UUID: GPU-9387e4b3-3640-0064-6b80-5ace1ee535f6                |
+--------+----------------------------------------------------------------------+
| 3      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:4E:00.0                                         |
|        | Device UUID: GPU-cefd0e59-c486-c12f-418c-84ccd7a12bb2                |
+--------+----------------------------------------------------------------------+
| 4      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:87:00.0                                         |
|        | Device UUID: GPU-1501b26d-f3e4-8501-421d-5a444b17eda8                |
+--------+----------------------------------------------------------------------+
| 5      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:90:00.0                                         |
|        | Device UUID: GPU-f4180a63-1978-6c56-9903-ca5aac8af020                |
+--------+----------------------------------------------------------------------+
| 6      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:B7:00.0                                         |
|        | Device UUID: GPU-8b354e3e-0145-6cfc-aec6-db2c28dae134                |
+--------+----------------------------------------------------------------------+
| 7      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:BD:00.0                                         |
|        | Device UUID: GPU-a16e3b98-8be2-6a0c-7fac-9cb024dbc2df                |
+--------+----------------------------------------------------------------------+
6 NvSwitches found.
+-----------+
| Switch ID |
+-----------+
| 11        |
| 10        |
| 13        |
| 9         |
| 12        |
| 8         |
+-----------+
 Install the dcgm-exporter
 As it doesn't need any persistence I've added it to the prometheus docker compose:
   dcgm-exporter:
    # latest didn't work
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.6-3.4.2-ubuntu22.04
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    restart: unless-stopped
    container_name: dcgm-exporter
 And added the next scraping config in prometheus.yml
   - job_name: dcgm-exporter
    metrics_path: /metrics
    static_configs:
    - targets:
      - dcgm-exporter:9400
 Adding alerts
 Tweak the next alerts for your use case.
 ---
groups:
- name: dcgm-alerts
  rules:
  - alert: GPUHighTemperature
    expr: DCGM_FI_DEV_GPU_TEMP > 80
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "GPU High Temperature (instance {{ $labels.instance }})"
      description: "The GPU temperature is above 80°C for more than 5 minutes.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: GPUMemoryUtilizationHigh
    expr: DCGM_FI_DEV_MEM_COPY_UTIL > 90
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "GPU Memory Utilization High (instance {{ $labels.instance }})"
      description: "The GPU memory utilization is above 90% for more than 10 minutes.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: GPUComputeUtilizationHigh
    expr: DCGM_FI_DEV_GPU_UTIL > 90
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "GPU Compute Utilization High (instance {{ $labels.instance }})"
      description: "The GPU compute utilization is above 90% for more than 10 minutes.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: GPUPowerUsageHigh
    expr: DCGM_FI_DEV_POWER_USAGE > 160
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "GPU Power Usage High (instance {{ $labels.instance }})"
      description: "The GPU power usage is above 160W for more than 5 minutes.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: GPUUnavailable
    expr: up{job="dcgm-exporter"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "GPU Unavailable (instance {{ $labels.instance }})"
      description: "The DCGM Exporter instance is down or unreachable for more than 5 minutes.\n  LABELS: {{ $labels }}"
 Adding a dashboard
 I've tweaked this dashboard to simplify it. Check the article for the full json
 
 
 Operating Systems
 Linux
   New: Linux loki alerts.
       - alert: TooManyLogs
        expr: |
          sum by(hostname) (count_over_time({job="systemd-journal"} [1d])) / sum by(hostname) (count_over_time({job="systemd-journal"} [1d] offset 1d)) > 1.5
        for: 0m
        labels:
            severity: warning
        annotations:
            summary: "The server {{ $labels.hostname}} is generating too many logs"

      - alert: TooFewLogs
        expr: |
          sum by(hostname) (count_over_time({job="systemd-journal"} [1d])) / sum by(hostname) (count_over_time({job="systemd-journal"} [1d] offset 1d)) < 0.5
        for: 0m
        labels:
            severity: warning
        annotations:
            summary: "The server {{ $labels.hostname}} is generating too few logs"
 
 
 Linux Snippets
   New: Makefile use bash instead of sh.
 The program used as the shell is taken from the variable SHELL. If this variable is not set in your makefile, the program /bin/sh is used as the shell.
 So put SHELL := /bin/bash at the top of your makefile, and you should be good to go.
 
  New: Recover the message of a commit if the command failed.
 git commit can fail for reasons such as gpg.commitsign = true && gpg fails, or when running a pre-commit. Retrying the command opens a blank editor and the message seems to be lost.
 The message is saved though in .git/COMMIT_EDITMSG, so you can:
 git commit -m "$(cat .git/COMMIT_EDITMSG)"
 Or in general (suitable for an alias for example):
 git commit -m "$(cat "$(git rev-parse --git-dir)/COMMIT_EDITMSG)")"
 
  New: Configure nginx to restrict methods.
 server {
    listen 80;
    server_name yourdomain.com;

    location / {
        if ($request_method !~ ^(GET|POST)$ ) {
            return 405;
        }

        try_files $uri $uri/ =404;
    }
}
 
  New: Configure nginx location regexp to accept dashes.
 location ~* /share/[\w-]+ {
  root /home/project_root;
}
 
  New: Configure nginx location to accept many paths.
 location ~ ^/(static|media)/ {
  root /home/project_root;
}
 
  New: Remove image metadata.
 exiftool -all:all= /path/to/file
 
  New: Get the size increment of a directory between two dates.
 To see how much has a directory grown between two dates you can use:
 find /path/to/directory -type f -newerat 2022-12-31 ! -newerat 2024-01-01 -printf "%s\\n" | awk '{s+=$1} END {print s}'
 It finds all the files in that directory that were created in the 2023, it only prints their size in bytes and then it adds them all up.
 
  New: Introduce detox.
 detox cleans up filenames from the command line.
 Installation:
 apt-get install detox
 Usage:
 detox *
 
  New: Rotate image with the command line.
 If you want to overwrite in-place, mogrify from the ImageMagick suite seems to be the easiest way to achieve this:
 mogrify -rotate -90 *.jpg

mogrify -rotate 90 *.jpg
 
  New: Configure desktop icons in gnome.
 Latest versions of gnome dont have desktop icons read this article to fix this
 
  New: Send multiline messages with notify-send.
 The title can't have new lines, but the body can.
 
notify-send "Title" "This is the first line.\nAnd this is the second.")
 feat(linux_snippets#Find BIOS version): Find BIOS version
 dmidecode | less
 
  New: Reboot server on kernel panic.
 The proc/sys/kernel/panic file gives read/write access to the kernel variable panic_timeout. If this is zero, the kernel will loop on a panic; if nonzero it indicates that the kernel should autoreboot after this number of seconds. When you use the software watchdog device driver, the recommended setting is 60.
 To set the value add the next contents to the /etc/sysctl.d/99-panic.conf
 kernel.panic = 60
 Or with an ansible task:
 - name: Configure reboot on kernel panic
  become: true
  lineinfile:
    path: /etc/sysctl.d/99-panic.conf
    line: kernel.panic = 60
    create: true
    state: present
 
  New: Share a calculated value between github actions steps.
 You need to set a step's output parameter. Note that the step will need an id to be defined to later retrieve the output value.
 echo "{name}={value}" >> "$GITHUB_OUTPUT"
 For example:
 - name: Set color
  id: color-selector
  run: echo "SELECTED_COLOR=green" >> "$GITHUB_OUTPUT"
- name: Get color
  env:
    SELECTED_COLOR: ${{ steps.color-selector.outputs.SELECTED_COLOR }}
  run: echo "The selected color is $SELECTED_COLOR"
 
  New: Split a zip into sizes with restricted size.
 Something like:
 zip -9 myfile.zip *
zipsplit -n 250000000 myfile.zip
 Would produce myfile1.zip, myfile2.zip, etc., all independent of each other, and none larger than 250MB (in powers of ten). zipsplit will even try to organize the contents so that each resulting archive is as close as possible to the maximum size.
 
  New: Find files that were modified between dates.
 The best option is the -newerXY. The m and t flags can be used.
  m The modification time of the file reference
 t reference is interpreted directly as a time
 
 So the solution is
 find . -type f -newermt 20111222 \! -newermt 20111225
 The lower bound in inclusive, and upper bound is exclusive, so I added 1 day to it. And it is recursive.
 
  New: Add rofi launcher.
 pass is a command line password store
 Configure rofi launcher
  Save this script somewhere in your $PATH
 Configure your window manager to launch it whenever you need a password.
 
 
  New: Get info of a mkv file.
 ffprobe file.mkv
 
  New: Clean the logs of a unit of the journal.
 journalctl --vacuum-time=1s --unit=your.service
 If you wish to clear all logs use journalctl --vacuum-time=1s
 
  New: Introduce Alacritty.
 Alacritty is a modern terminal emulator that comes with sensible defaults, but allows for extensive configuration. By integrating with other applications, rather than reimplementing their functionality, it manages to provide a flexible set of features with high performance.
 Installation
  Clone the repo git clone https://github.com/alacritty/alacritty.git
cd alacritty
 Install rustup curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
 To make sure you have the right Rust compiler installed, run rustup override set stable
rustup update stable
 Install the dependencies apt install cmake pkg-config libfreetype6-dev libfontconfig1-dev libxcb-xfixes0-dev libxkbcommon-dev python3
 Build the release cargo build --release
 If all goes well, this should place a binary at target/release/alacritty
 Move the binary to somewhere in your PATH
 
 
mv target/release/alacritty ~/.local/bin
 - Check the terminfo: To make sure Alacritty works correctly, either the alacritty or alacritty-direct terminfo must be used. The alacritty terminfo will be picked up automatically if it is installed. If the following command returns without any errors, the alacritty terminfo is already installed:
 infocmp alacritty
 If it is not present already, you can install it globally with the following command:
 sudo tic -xe alacritty,alacritty-direct extra/alacritty.info
 Configuration
 Alacritty's configuration file uses the TOML format. It doesn't create the config file for you, but it looks for one in ~/.config/alacrity/alacritty.toml
 Not there yet - Support for ligatures
 References - Homepage - Source - Docs
 
  New: Set the vim filetype syntax in a comment.
 Add somewhere in your file:
 
 
  New: Export environment variables in a crontab.
 If you need to expand the PATH in theory you can do it like this:
 PATH=$PATH:/usr/local/bin

* * * * * /path/to/my/script
 I've found however that sometimes this doesn't work and you need to specify it in the crontab line:
 * * * * * PATH=$PATH:/usr/local/bin /path/to/my/script
 
  Correction: Docker prune without removing the manual networks.
 If you run the command docker system prune in conjunction with watchtower and manually defined networks you may run into the issue that the docker system prune acts just when the dockers are stopped and thus removing the networks, which will prevent the dockers to start. In those cases you can either make sure that docker system prune is never run when watchtower is doing the updates or you can split the command into the next script:
 date
echo "Pruning the containers"
docker container prune -f --filter "label!=prune=false"
echo "Pruning the images"
docker image prune -f --filter "label!=prune=false"
echo "Pruning the volumes"
docker volume prune -f
 
  New: Debugging high IOwait.
 High I/O wait (iowait) on the CPU, especially at 50%, typically indicates that your system is spending a large portion of its time waiting for I/O operations (such as disk access) to complete. This can be caused by a variety of factors, including disk bottlenecks, overloaded storage systems, or inefficient applications making disk-intensive operations.
 Here’s a structured approach to debug and analyze high I/O wait on your server:
 ** Monitor disk I/O**
 First, verify if disk I/O is indeed the cause. Tools like iostat, iotop, and dstat can give you an overview of disk activity:
  iostat: This tool reports CPU and I/O statistics. You can install it with apt-get install sysstat. Run the following command to check disk I/O stats:
 
 
iostat -x 1
 The -x flag provides extended statistics, and 1 means it will report every second. Look for high values in the %util and await columns, which represent: - %util: Percentage of time the disk is busy (ideally should be below 90% for most systems). - await: Average time for I/O requests to complete.
 If either of these values is unusually high, it indicates that the disk subsystem is likely overloaded.
  iotop: If you want a more granular look at which processes are consuming disk I/O, use iotop:
 
 sudo iotop -o
 This will show you the processes that are actively performing I/O operations.
  dstat: Another useful tool for monitoring disk I/O in real-time:
 
 dstat -cdl 1
 This shows CPU, disk, and load stats, refreshing every second. Pay attention to the dsk/await value.
 Check disk health Disk issues such as bad sectors or failing drives can also lead to high I/O wait times. To check the health of your disks:
  Use smartctl: This tool can give you a health check of your disks if they support S.M.A.R.T.
 
 sudo smartctl -a /dev/sda
 Check for any errors or warnings in the output. Particularly look for things like reallocated sectors or increasing "pending sectors."
  dmesg logs: Look at the system logs for disk errors or warnings:
 
 dmesg | grep -i "error"
 If there are frequent disk errors, it may be time to replace the disk or investigate hardware issues.
 Look for disk saturation If the disk is saturated, no matter how fast the CPU is, it will be stuck waiting for data to come back from the disk. To further investigate disk saturation:
  df -h: Check if your disk partitions are full or close to full.
 
 df -h
  lsblk: Check how your disks are partitioned and how much data is written to each partition:
 
 lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
  blktrace: For advanced debugging, you can use blktrace, which traces block layer events on your system.
 
 sudo blktrace -d /dev/sda -o - | blkparse -i -
 This will give you very detailed insights into how the system is interacting with the block device.
 Check for heavy disk-intensive processes Identify processes that might be using excessive disk I/O. You can use tools like iotop (as mentioned earlier) or pidstat to look for processes with high disk usage:
  pidstat: Track per-process disk activity:
 
 pidstat -d 1
 This command will give you I/O statistics per process every second. Look for processes with high I/O values (r/s and w/s).
  top or htop: While top or htop can show CPU usage, they can also show process-level disk activity. Focus on processes consuming high CPU or memory, as they might also be performing heavy I/O operations.
 
 check file system issues Sometimes the file system itself can be the source of I/O bottlenecks. Check for any file system issues that might be causing high I/O wait.
  Check file system consistency: If you suspect the file system is causing issues (e.g., due to corruption), run a file system check. For ext4:
 
 sudo fsck /dev/sda1
 Ensure you unmount the disk first or do this in single-user mode.
  Check disk scheduling: Some disk schedulers (like cfq or deadline) might perform poorly depending on your workload. You can check the scheduler used by your disk with:
 
 cat /sys/block/sda/queue/scheduler
 You can change the scheduler with:
 echo deadline > /sys/block/sda/queue/scheduler
 This might improve disk performance, especially for certain workloads.
 Examine system logs The system logs (/var/log/syslog or /var/log/messages) may contain additional information about hardware issues, I/O bottlenecks, or kernel-related warnings:
 sudo tail -f /var/log/syslog
 or
 sudo tail -f /var/log/messages
 Look for I/O or disk-related warnings or errors.
 Consider hardware upgrades or tuning
  SSD vs HDD: If you're using HDDs, consider upgrading to SSDs. HDDs can be much slower in terms of I/O, especially if you have a high number of random read/write operations.
 RAID Configuration: If you are using RAID, check the RAID configuration and ensure it's properly tuned for performance (e.g., using RAID-10 for a good balance of speed and redundancy).
 Memory and CPU Tuning: If the server is swapping due to insufficient RAM, it can result in increased I/O wait. You might need to add more RAM or optimize the system to avoid excessive swapping.
 
 Check for swapping issues Excessive swapping can contribute to high I/O wait times. If your system is swapping (which happens when physical RAM is exhausted), I/O wait spikes as the system reads from and writes to swap space on disk.
  Check swap usage:
 
 free -h
 If swap usage is high, you may need to add more physical RAM or optimize applications to reduce memory pressure.
 
  New: Create a file with random data.
 Of 3.5 GB
 dd if=/dev/urandom of=random_file.bin bs=1M count=3584
 
  New: Convert an html to a pdf.
 Using weasyprint
 Install it with pip install weasyprint PyMuPDF
 weasyprint input.html output.pdf
 It gave me better result than wkhtmltopdf
 Using wkhtmltopdf To convert the given HTML into a PDF with proper styling and formatting using a simple method on Linux, you can use wkhtmltopdf with some custom options.
 First, ensure that you have wkhtmltopdf installed on your system. If not, install it using your package manager (e.g., Debian: sudo apt-get install wkhtmltopdf).
 Then, convert the HTML to PDF using wkhtmltopdf with the following command:
 wkhtmltopdf --page-size A4 --margin-top 15mm --margin-bottom 15mm --encoding utf8 input.html output.pdf
 In this command: - --page-size A4: Sets the paper size to A4. - --margin-top 15mm and --margin-bottom 15mm: Adds top and bottom margins of 15 mm to the PDF.
 After running the command, you should have a nicely formatted output.pdf file in your current directory. This method preserves most of the original HTML styling while providing a simple way to export it as a PDF on Linux.
 If you need to zoom in, you can use the --zoom 1.2 flag. For this to work you need your css to be using the em sizes.
 
  New: Format a drive to use a FAT32 system.
 sudo mkfs.vfat -F 32 /dev/sdX
 Replace /dev/sdX with your actual drive identifier
 
  New: Get the newest file of a directory with nested directories and files.
 find . -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" "
 
  Correction: How to debug a CPU Throttling high alert.
 If the docker is using less resources than the limits but they are still small (for example 0.1 CPUs) the issue may be that the CPU spikes are being throttle before they are shown in the CPU usage, the solution is then to increase the CPU limits
 # Create a systemd service for a non-root user
 To set up a systemd service as a non-root user, you can create a user-specific service file under your home directory. User services are defined in ~/.config/systemd/user/ and can be managed without root privileges.
 
  New: Check the docker images sorted by size.
 docker images --format "{{.Repository}}:{{.Tag}}\t{{.Size}}" | sort -k2 -h
 You can also use the builtin docker system df -v to get a better understanding of the usage of disk space.
 
 
 Wezterm
   New: Introduce wezterm.
 WezTerm is a powerful cross-platform terminal emulator and multiplexer implemented in Rust.
 Installation
 You can configure your system to use that APT repo by following these steps:
 curl -fsSL https://apt.fury.io/wez/gpg.key | sudo gpg --yes --dearmor -o /usr/share/keyrings/wezterm-fury.gpg
echo 'deb [signed-by=/usr/share/keyrings/wezterm-fury.gpg] https://apt.fury.io/wez/ * *' | sudo tee /etc/apt/sources.list.d/wezterm.list
 Update your dependencies:
 sudo apt update
 Now you can install wezterm:
 sudo apt install wezterm
 or to install a nightly build:
 
sudo apt install wezterm-nightly
 Troubleshooting
 Install in Debian 12 error
 Install from nightly.
 References
  Code
 Docs
 Home
 
 
 
 Terminals
   New: Suggest to use nerd fonts.
 If you're thinking of adding a new font, you should take a look at the nerd fonts as they patch developer targeted fonts with a high number of glyphs (icons). Specifically to add a high number of extra glyphs from popular ‘iconic fonts’ such as Font Awesome, Devicons, Octicons, and others.
 For example you can get the FiraCode Nerd Font
 
  New: Introduce Rofi.
 Rofi is a window switcher, application launcher and dmenu replacement.
 Installation
 sudo apt-get install rofi
 Usage To launch rofi directly in a certain mode, specify a mode with rofi -show <mode>. To show the run dialog:
 rofi -show run
 Or get the options from a script:
 ~/my_script.sh | rofi -dmenu
 Specify an ordered, comma-separated list of modes to enable. Enabled modes can be changed at runtime. Default key is Ctrl+Tab. If no modes are specified, all configured modes will be enabled. To only show the run and ssh launcher:
 rofi -modes "run,ssh" -show run
 The modes to combine in combi mode. For syntax to -combi-modes , see -modes. To get one merge view, of window,run, and ssh:
 rofi -show combi -combi-modes "window,run,ssh" -modes combi
 Configuration
 The configuration lives at ~/.config/rofi/config.rasi to create this file with the default conf run:
 rofi -dump-config > ~/.config/rofi/config.rasi
 Use fzf to do the matching
 To run once:
 rofi -show run -sorting-method fzf -matching fuzzy
 To persist them change those same values in the configuration.
 Theme changing To change the theme: - Choose the one you like most looking here - Run rofi-theme-selector to select it - Accept it with Alt + a
 Keybindings change
 Plugins You can write your custom plugins. If you're on python using python-rofi seems to be the best option although it looks unmaintained.
 Some interesting examples are:
  Python based plugin
 Creation of nice menus
 Nice collection of possibilities
 Date picker
 Orgmode capture
 
 Other interesting references are:
  List of key bindings
 Theme guide
 
 References - Source - Docs - Plugins
 
  New: Do terminal comparison.
 Alacritty
 Pros:
 Cons:
  The installation is difficult if you're not used to Rust.
 Doesn't support for ligatures so Fira Code with ligatures looks weird.
 Awful docs
 Difficult to keep updated
 
 Kitty
 Pros:
  Built in python
 Supports ligatures
 Nice docs
 Easy installation
 
 Cons:
  Vim bindings to move around and copy the buffer don't work well
 When you sudo su on a server you need to copy the ~/.terminfo otherwise the shell is broken
 You need to replace ssh so that they copy that file.
 
 Wezterm
 Pros:
  Nice docs
 
 Cons:
  Dev is enough narcissistic enough to not only add it in the name of the terminal but also to say his name as the main developer.
 
 
 
 Gotify
   New: Complete installation.
  Create the data directories: mkdir -p /data/config/gotify/ /data/gotify
 Assuming you're using an external proxy create the next docker compose in /data/config/gotify.
 
 ---
version: "3"

services:
  gotify:
    image: gotify/server
    container_name: gotify
    networks:
      - swag
    env_file:
      - .env
    volumes:
      - gotify-data:/app/data

networks:
  swag:
    external:
      name: swag

volumes:
  gotify-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/gotify
 With the next .env file:
 
  GOTIFY_SERVER_SSL_ENABLED=false

  GOTIFY_DATABASE_DIALECT=sqlite3
  GOTIFY_DATABASE_CONNECTION=data/gotify.db

  GOTIFY_DEFAULTUSER_NAME=admin
  GOTIFY_DEFAULTUSER_PASS=changeme

  GOTIFY_PASSSTRENGTH=10
  GOTIFY_UPLOADEDIMAGESDIR=data/images
  GOTIFY_PLUGINSDIR=data/plugins
  GOTIFY_REGISTRATION=false
  ```

* Create the service by adding a file `gotify.service` into `/etc/systemd/system/`
 [Unit] Description=gotify Requires=docker.service After=docker.service
 [Service] Restart=always User=root Group=docker WorkingDirectory=/data/config/gotify TimeoutStartSec=100 RestartSec=2s ExecStart=/usr/bin/docker-compose -f docker-compose.yaml up ExecStop=/usr/bin/docker-compose -f docker-compose.yaml down
 [Install] WantedBy=multi-user.target 
* Copy the nginx configuration in your `site-confs`

  ```

  server {
      listen 443 ssl;
      listen [::]:443 ssl;

      server_name gotify.*;

      include /config/nginx/ssl.conf;

      client_max_body_size 0;

      # enable for ldap auth (requires ldap-location.conf in the location block)
      #include /config/nginx/ldap-server.conf;

      # enable for Authelia (requires authelia-location.conf in the location block)
      #include /config/nginx/authelia-server.conf;

      location / {
          # enable the next two lines for http auth
          #auth_basic "Restricted";
          #auth_basic_user_file /config/nginx/.htpasswd;

          # enable for ldap auth (requires ldap-server.conf in the server block)
          #include /config/nginx/ldap-location.conf;

          # enable for Authelia (requires authelia-server.conf in the server block)
          #include /config/nginx/authelia-location.conf;

          include /config/nginx/proxy.conf;
          include /config/nginx/resolver.conf;
          set $upstream_app gotify;
          set $upstream_port 80;
          set $upstream_proto http;
          proxy_pass $upstream_proto://$upstream_app:$upstream_port;
      }
  }
  ```
* Start the service `systemctl start gotify`
* Restart the nginx service `systemctl restart swag`
* Enable the service `systemctl enable gotify`.
* Login with the `admin` user
* Create a new user with admin permissions
* Delete the `admin` user

**Configuration**

- [Android client](https://github.com/gotify/android)
- Linux clients
  - [command line client](#command-line-client)
  - [Dunst client](https://github.com/ztpnk/gotify-dunst)
  - [gotify-desktop](https://github.com/desbma/gotify-desktop)
  - [rofi client](https://github.com/diddypod/rotify)

**Connect it with Alertmanager**

It's not trivial to connect it to Alertmanager([1](https://github.com/prometheus/alertmanager/issues/2120), [2](https://github.com/gotify/contrib/issues/21), [3](https://github.com/prometheus/alertmanager/issues/3729), [4](https://github.com/prometheus/alertmanager/issues/2120). The most popular way is to use [`alertmanager_gotify_bridge`](https://github.com/DRuggeri/alertmanager_gotify_bridge?tab=readme-ov-file).

We need to tweak the docker-compose to add the bridge:

```yaml
 Connect it with Authentik
 Here are some guides to connect it to authentik. The problem is that the clients you want to use must support it
  https://github.com/gotify/server/issues/203
 https://github.com/gotify/server/issues/553
 
 References
  Docs
 
 
 
 HAProxy
   New: Automatically ban offending traffic.
 Check these two posts:
  https://serverfault.com/questions/853806/blocking-ips-in-haproxy
 https://www.loadbalancer.org/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2/
 
 
  New: Configure haproxy logs to be sent to loki.
 In the fronted config add the next line:
   # For more options look at https://www.chrisk.de/blog/2023/06/haproxy-syslog-promtail-loki-grafana-logfmt/
  log-format 'client_ip=%ci client_port=%cp frontend_name=%f backend_name=%b server_name=%s performance_metrics=%TR/%Tw/%Tc/%Tr/%Ta status_code=%ST bytes_read=%B termination_state=%tsc haproxy_metrics=%ac/%fc/%bc/%sc/%rc srv_queue=%sq  backend_queue=%bq user_agent=%{+Q}[capture.req.hdr(0)] http_hostname=%{+Q}[capture.req.hdr(1)] http_version=%HV http_method=%HM http_request_uri="%HU"'
 At the bottom of chrisk post is a table with all the available fields.
 Programming VIP also has an interesting post.
 
  New: Reload haproxy.
  Check the config is alright service haproxy configtest
# Or
/usr/sbin/haproxy -c -V -f /etc/haproxy/haproxy.cfg
 Reload the service service haproxy reload
 
 If you want to do a better reload you can drop the SYN before a restart, so that clients will resend this SYN until it reaches the new process.
 iptables -I INPUT -p tcp --dport 80,443 --syn -j DROP
sleep 1
service haproxy reload
iptables -D INPUT -p tcp --dport 80,443 --syn -j DROP
service haproxy reload
 
 
 journald
   New: Introduce journald.
 journald is a system service that collects and stores logging data. It creates and maintains structured, indexed journals based on logging information that is received from a variety of sources:
  Kernel log messages, via kmsg
 Simple system log messages, via the libc syslog call
 Structured system log messages via the native Journal API.
 Standard output and standard error of service units.
 Audit records, originating from the kernel audit subsystem.
 
 The daemon will implicitly collect numerous metadata fields for each log messages in a secure and unfakeable way.
 Journald provides a good out-of-the-box logging experience for systemd. The trade-off is, journald is a bit of a monolith, having everything from log storage and rotation, to log transport and search. Some would argue that syslog is more UNIX-y: more lenient, easier to integrate with other tools. Which was its main criticism to begin with. When the change was made not everyone agreed with the migration from syslog or the general approach systemd took with journald. But by now, systemd is adopted by most Linux distributions, and it includes journald as well. journald happily coexists with syslog daemons, as:
  Some syslog daemons can both read from and write to the journal
 journald exposes the syslog API
 
 It provides lots of features, most importantly:
  Indexing. journald uses a binary storage for logs, where data is indexed. Lookups are much faster than with plain text files.
 Structured logging. Though it’s possible with syslog, too, it’s enforced here. Combined with indexing, it means you can easily filter specific logs (e.g. with a set priority, in a set timeframe).
 Access control. By default, storage files are split by user, with different permissions to each. As a regular user, you won’t see everything root sees, but you’ll see your own logs.
 Automatic log rotation. You can configure journald to keep logs only up to a space limit, or based on free space.
 
 
 
 Syncthing
   New: Change the path of a folder.
  Shutdown Syncthing
 Edit the config file (~/.config/syncthing/config.xml)
 Search and replace the path
 Start again syncthing
 
 
 
 Wireguard
   New: Improve logging.
 WireGuard doesn’t do any logging by default. If you use the WireGuard Linux kernel module (on kernel versions 5.6 or newer), you can turn on WireGuard’s dyndbg logging, which sends log messages to the kernel message buffer, kmsg. You can then use the standard dmesg utility to read these messages. Also, many Linux systems have a logging daemon like rsyslogd or journald that automatically captures and stores these messages.
 First, enable WireGuard dyndbg logging with the following commands:
 modprobe wireguard
echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control
 Once you do that, you’ll be able to see WireGuard log messages in the kernel message facility, if your system is set up with rsyslogd, journald, or a similar logging daemon. With rsyslogd, check the /var/log/kern.log or /var/log/messages file. With journald, run journalctl -ek.
 
  New: Monitor wireguard.
  https://www.procustodibus.com/blog/2021/01/how-to-monitor-wireguard-activity/
 https://techoverflow.net/2021/12/31/how-to-check-if-wireguard-client-peer-is-connected/
 
 
  New: Check the status of the tunnel.
 One method is to do ping between VPN IP addresses or run command wg show`` from the server or from the client. Below you can seewg show`` command output where VPN is not up.
 $: wg show
interface: wg0
  public key: qZ7+xNeXCjKdRNM33Diohj2Y/KSOXwvFfgTS1LRx+EE=
  private key: (hidden)
  listening port: 45703

peer: mhLzGkqD1JujPjEfZ6gkbusf3sfFzy+1KXBwVNBRBHs=
  endpoint: 3.133.147.235:51820
  allowed ips: 10.100.100.1/32
  transfer: 0 B received, 592 B sent
  persistent keepalive: every 21 seconds
 The below output from the wg show command indicates the VPN link is up. See the line with last handshake time
 $: wg show
interface: wg0
  public key: qZ7+xNeXCjKdRNM33Diohj2Y/KSOXwvFfgTS1LRx+EE=
  private key: (hidden)
  listening port: 49785

peer: 6lf4SymMbY+WboI4jEsM+P9DhogzebSULrkFowDTt0M=
  endpoint: 3.133.147.235:51820
  allowed ips: 10.100.100.1/32
  latest handshake: 14 seconds ago
  transfer: 732 B received, 820 B sent
  persistent keepalive: every 21 seconds
 
 
 Android
 GrapheneOS
   New: Disable Bluetooth and Wifi once it's disconnected.
 If you don't want to go spreading your SSIDs you can configure graphene to disable wifi and bluetooth X minutes after loosing connection.
 For Wifi:
  Go to Settings > network & internet > internet > network preferences
 Select Turn off Wi-Fi automatically, for example after 2 minutes
 
 For Bluetooth:
  Go to Settings > connected devices > connection preferences> bluetooth
 Select Turn Bluetooth off automatically, for example after 2 minutes
 
 
 
 GadgetBridge
   Correction: Installation on GrapheneOS.
 On GrapheneOS you may need to enable the restricted permissions
 
 
 Android SDK Platform tools
   New: Introduce android_sdk.
 Android SDK Platform tools is a component for the Android SDK. It includes tools that interface with the Android platform, primarily adb and fastboot.
 Installation
 While many Linux distributions already package Android Platform Tools (for example android-platform-tools-base on Debian), it is preferable to install the most recent version from the official website. Packaged versions might be outdated and incompatible with most recent Android handsets.
  Download the latest toolset
 Extract it somewhere in your filesystem
 Create links to the programs you want to use in your $PATH
 
 Next you will need to enable debugging on the Android device you are testing. Please follow the official instructions on how to do so.
 Usage
 Connecting over USB
 To use adb with a device connected over USB, you must enable USB debugging in the device system settings, under Developer options. On Android 4.2 (API level 17) and higher, the Developer options screen is hidden by default.
 Enable the Developer options
 To make it visible, enable Developer options. On Android 4.1 and lower, the Developer options screen is available by default. On Android 4.2 and higher, you must enable this screen.
  On your device, find the Build number option (Settings > About phone > Build number)
 Tap the Build Number option seven times until you see the message You are now a developer! This enables developer options on your device.
 Return to the previous screen to find Developer options at the bottom.
 
 Enable USB debugging
 Before you can use the debugger and other tools, you need to enable USB debugging, which allows Android Studio and other SDK tools to recognize your device when connected via USB.
 Enable USB debugging in the device system settings under Developer options. You can find this option in one of the following locations, depending on your Android version:
  Android 9 (API level 28) and higher: Settings > System > Advanced > Developer Options > USB debugging
 Android 8.0.0 (API level 26) and Android 8.1.0 (API level 27): Settings > System > Developer Options > USB debugging
 Android 7.1 (API level 25) and lower: Settings > Developer Options > USB debugging
 
 Test it works
 If everything is configured appropriately you should see your device when launching the command adb devices.
 Create udev rules if it fails
 If you see the next error:
 failed to open device: Access denied (insufficient permissions)

* failed to start daemon
adb: failed to check server version: cannot connect to daemon
 It indicates an issue with permissions when adb tries to communicate with the device via USB. Here are some steps you can take to resolve this issue:
  Check USB permissions
 Ensure that you have the necessary permissions to access the USB device. If you're running on Linux, check if the device has appropriate udev rules.
  You can try adding your user to the plugdev group:
 sudo usermod -aG plugdev $USER
 
  Make sure you have a udev rule for Android devices in /etc/udev/rules.d/. If not, you can create one by adding a file like 51-android.rules:
 sudo touch /etc/udev/rules.d/51-android.rules
 
  Add this line to the file to grant access to Android devices:
 SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0666", GROUP="plugdev"
 
  Reload the udev rules:
 sudo udevadm control --reload-rules
sudo service udev restart
 
  Unplug and reconnect the USB device. References
 
 Home
 
 
 
 ICSx5
   New: Introduce ICSx5.
 ICSx5 is an android app to sync calendars.
 References
  Source
 F-droid
 
 
 
 Signal
   New: Use the Molly FOSS android client.
 Molly is an independent Signal fork for Android. The advantages are:
  Contains no proprietary blobs, unlike Signal.
 Protects database with passphrase encryption.
 Locks down the app automatically when you are gone for a set period of time.
 Securely shreds sensitive data from RAM.
 Automatic backups on a daily or weekly basis.
 Supports SOCKS proxy and Tor via Orbot.
 
 Migrate from Signal
 Note, the migration should be done when the available Molly version is equal to or later than the currently installed Signal app version.
  Verify your Signal backup passphrase. In the Signal app: Settings > Chats > Chat backups > Verify backup passphrase.
 Optionally, put your phone offline (enable airplane mode or disable data services) until after Signal is uninstalled in step 5. This will prevent the possibility of losing any Signal messages that are received during or after the backup is created.
 Create a Signal backup. In the Signal app, go to Settings > Chats > Chat backups > Create backup.
 Uninstall the Signal app. Now you can put your phone back online (disable airplane mode or re-enable data services).
 Install the Molly or Molly-FOSS app.
 Open the Molly app. Enable database encryption if desired. As soon as the option is given, tap Transfer or restore account. Answer any permissions questions.
 Choose to Restore from backup and tap Choose backup. Navigate to your Signal backup location (Signal/Backups/, by default) and choose the backup that was created in step 3.
 Check the backup details and then tap Restore backup to confirm. Enter the backup passphrase when requested.
 If asked, choose a new folder for backup storage. Or choose Not Now and do it later.
 
 Consider also:
  Any previously linked devices will need to be re-linked. Go to Settings > Linked devices in the Molly app. If Signal Desktop is not detecting that it is no longer linked, try restarting it.
 Verify your Molly backup settings and passphrase at Settings > Chats > Chat backups (to change the backup folder, disable and then enable backups). Tap Create backup to create your first Molly backup.
 When you are satisfied that Molly is working, you may want to delete the old Signal backups (in Signal/Backups, by default).
 
 
 
 Arts
 Cooking
   New: Comer naranja por la noche.
 "Por la mañana oro, al mediodía plata y por la noche mata". Así de tajante se muestra el refranero español con respecto a la naranja y el melón. La naranja es una fruta ácida y por esta razón es recomendable evitar su consumo por la noche. Y es que este alto nivel de acidez puede provocar una digestión lenta y sensación de acidez y ardor en el estómago. Directamente relacionado con estos síntomas puede sobrevenir dolor de estómago y esto conducirnos a un descanso poco placentero durante la noche. Así, lo mejor es evitar su consumo a última hora del día. Si encima le añades unos trocitos de chocolate negro dificultarás el tránsito intestinal (por las grasas) y harás que te cueste más conciliar el sueño ya que es estimulante. Pero qué rico sabe de postre de cena xD.
 Estas consecuencias varían en función de las personas y de su sistema digestivo. Así, este alimento sí puede resultar saludable para ciertas personas sin problemas gástricos y un alto nivel de tolerancia a este alimento.
 Lo más aconsejable es consumirla por la mañana. Tomar una naranja a primera hora del día te aportará una buena dosis de vitamina C y fibra.
 Siempre es recomendable tomar la fruta directamente y evitar los zumos, aunque sean naturales. De esta forma nos sentiremos más saciados y añadiremos más fibra y menos azúcares a nuestra dieta.
 Además, la acidez de las naranjas que podría perjudicarnos por la noche es perfectamente tolerable por la mañana incluso si la tomamos en ayunas. De esta forma, podremos ir asimilando sus propiedades durante todo el día.
 Estas frutas tienen un gran efecto antioxidante. Además, su gran contenido en vitamina C es un refuerzo inigualable para el sistema inmunológico y ayuda a combatir la anemia. También es un remedio ideal para tratar los resfriados y las gripes.
 
  New: Por qué no vale con quitar la parte del moho de los alimentos.
 Artículo
 
 
 Cleaning
 Cleaning tips
   New: Cleaning car headlights.
 If you need to clean the car headlights you can use a mixture of one squeezed lemon and two spoonfuls of baking soda
 
 
 Parkour
   New: Warming up.
 Never do static stretches if you're cold, it's better to do dynamic stretches.
 Take the joints through rotations
  Head:
 Nod 10 times
 Say no 10 times
 Ear shoulder 10 times
  Circles 10 times each direction
 
  Shoulders
 
 Circles back 10 times
  Circles forward 10 times
 
  Elbows
 
  Circles 10 each direction
 
  Wrists:
 
  Circle 10 each direction
 
  Chest:
 
 Chest out/in 10 times
 Chest one side to the other 10 times
  Chest in circles
 
  Hips:
 
 Circles 10 each direction
  Figure eight 10 times each direction
 
  Knees:
 
 Circular rotations 10 each direction feet and knees together
 10 ups and downs with knees together
  Circular rotations 10 each direction feet waist width
 
  Ankles:
 
 Circular 10 rotations each direction
 
 Light exercises
  10 steps forward of walking on your toes, 10 back
 10 steps forward of walking on your toes feet rotated outwards, 10 back
 10 steps forward of walking on your toes feet rotated inwards, 10 back
 10 steps forward of walking on your heels feet rotated outwards, 10 back
  10 steps forward of walking on your heels feet rotated inwards, 10 back
 
  2 x 10 x Side step, carry the leg up (from out to in) while you turn 180, keep on moving on that direction
 
 2 x 10 x Front step carrying the leg up (from in to out)while you turn 45, then side step, keep on moving on that direction
 10 light skips on one leg: while walking forward lift your knee and arms and do a slight jump
 10 steps with high knees
 10 steps with heel to butt
  10 side shuffles (like basketball defense)
 
  5 lunges forward, 5 backwards
 
  10 rollups and downs from standing position
 
 5 push-ups
 10 rotations from the pushup position on each direction with straigth arms
 5 push-ups
 10 rotations from the pushup position on each direction with shoulders at ankle level
  3 downward monkeys: from piramid do a low pushup and go to cobra, then a pushup
 
  10 steps forward walking on all fours
 
 
 Strengthen your knees
 Follow there steps
 Transit to the parkour place
 Go by bike, skate, jogging to the parkour place
 
 
 Music
 Sister Rosetta Tharpe
   New: Introduce Sister Rosetta Tharpe.
 Sister Rosetta Tharpe was a visionary, born in 1915 she started shredding the guitar in ways that did not exist in that time. Yes, she founded Rock and Roll. It's lovely to see a gospel singer with an electrical guitar.
 In this video you'll be able to understand how awesome she ws.
 Videos:
  Up Above my head
 Didn't it rain? 1964
 Nice Best of compilation
 
 
 
 Drawing
  New: Drawing book recommendation.
 
 Languages
 Castellano
   New: El agua o la agua?.
 El sustantivo agua es de género femenino, pero tiene la particularidad de comenzar por /a/ tónica (la vocal tónica de una palabra es aquella en la que recae el acento de intensidad: [água]). Por razones de fonética histórica, este tipo de palabras seleccionan en singular la forma el del artículo, en lugar de la forma femenina normal la. Esta regla solo opera cuando el artículo antecede inmediatamente al sustantivo, de ahí que digamos el agua, el área, el hacha; pero, si entre el artículo y el sustantivo se interpone otra palabra, la regla queda sin efecto, de ahí que digamos la misma agua, la extensa área, la afilada hacha. Puesto que estas palabras son femeninas, los adjetivos deben concordar siempre en femenino: el agua clara, el área extensa, el hacha afilada (y no el agua claro, el área extenso, el hacha afilado).
 Por su parte, el indefinido una toma generalmente la forma un cuando antecede inmediatamente a sustantivos femeninos que comienzan por /a/ tónica: un área, un hacha, un águila (si bien no es incorrecto, aunque sí poco frecuente, utilizar la forma plena una: una área, una hacha, una águila). Asimismo, los indefinidos alguna y ninguna pueden adoptar en estos casos las formas apocopadas (algún alma, ningún alma) o mantener las formas plenas (alguna alma, ninguna alma).
 Al tratarse de sustantivos femeninos, con los demostrativos este, ese, aquel o con cualquier otro adjetivo determinativo, como todo, mucho, poco, otro, etc., deben usarse las formas femeninas correspondientes: esta hacha, aquella misma arma, toda el agua, mucha hambre, etc. (y no este hacha, aquel mismo arma, todo el agua, mucho hambre, etc.)
 
 
 Galego
  New: Add some galego vocabulary.
  New: Introduce galego.
 O galego é unha lingua indoeuropea que pertence á póla de linguas románicas. É a lingua propia de Galiza, onde é falada por uns 2.4 millóns de galegas. Á parte de en Galiza, a lingua falase tamén en territórios limítrofes con esta comunidade, ainda que sen estatuto de oficialidade, asi como pola diáspora galega que emigrou a outras partes do estado español, América latina, os Estados Unidos, Suíza e outros países do Europa.
 
  New: Te e che. Trucos para saber diferencialos.
 En galego temos dúas formas para o pronome átono da segunda persoa do singular: te e che.
 O pronome te ten a función de complemento directo (CD) e o pronome che de complemento indirecto (CI).
 Cando se utiliza o pronome te?
 O pronome te utilízase cando ten a función de CD, propio dos verbos transitivos, xa que alude ao ser ou ao obxecto sobre o que recae a acción verbal.
 Se convertemos a oración en pasiva, o CD pasa a ser o suxeito. Por exemplo:
 Vinte na cafetería / Ti fuches visto por min na cafetería.
 Cando se utiliza o pronome che?
 O pronome che utilízase cando ten a función de CI, xa que indica o destinatario da acción expresada polo verbo. Por exemplo:
 Díxenche a verdade.
 Compreiche unhas lambonadas.
 Truco para saber diferencialos
 Un truco moi rápido para diferenciarmos os pronomes te e che é substituír eses pronomes de segunda persoa polos de terceira.
 Se podemos cambiar ese pronome por o/lo/no ou a/la/na, quere dicir que o pronome vai ser de CD. Polo tanto, temos que poñer te.
 Saudeite onte pola rúa / Saudeino onte pola rúa.
 Chameite por teléfono / Chameina por teléfono.
 Se podemos substituílo por un lle, significa que é un pronome de CI e que debemos utilizar o che.
 Lévoche mañá os apuntamentos / Lévolle mañá os apuntamentos.
 Collinche as entradas do concerto / Collinlle as entradas do concerto.
 
  New: Uso de asemade.
 Asemade pode utilizarse como adverbio cando ten o significado de ‘ao mesmo tempo’ ou ‘simultaneamente’. Ainda que normalmente úsase no registro culto, non utilizalo na fala.
  Non se pode comer e falar asemade.
 Non podes facer os deberes e ver a televisión asemade, pois non te concentras.
 
 Tamén se pode utilizar como conxunción co significado de ‘tan pronto como’.
  Foi o primeiro que vimos asemade entramos.
 Recoñecino asemade o vin.
 
 É incorrecto empregar asemade como sinónimo de tamén, ademais ou igualmente.
 
  New: Referencias e libros de gramática.
 Referencias:
  Dicionario
 Traductor
 Juego Pensatermos
 Conxugador de verbos
 Celga-1 materiais
  Recursos para aprender o galego
 
  Recompilación de grupos de música en galego
 
 Conversas do fenómeno das persoas neofalantes e o futuro do idioma
 
 Libros gramática:
  Gramática da Lingua Galega de Xosé Feixó Cid
 Como falar e escribir en galego con corrección e fluidez de Carlos Callón
 Manual de conxugación verbal da lingua galega de Avelino Hermida
 Dicionario Galaxia de usos e dificultades da lingua galega de Benigno Fernández Salgado
 
 
 
 Science
 Artificial Intelligence
   New: Add aider tool.
  Aider lets you pair program with LLMs, to edit code in your local git repository. Start a new project or work with an existing git repo. Aider works best with GPT-4o & Claude 3.5 Sonnet and can connect to almost any LLM.
 
 
 
 Whisper
   New: Introduce whisper.
 Installation
 Web interfaces
  Whishper
 Whisper-WebUI
 
 Command line tools
  faster-whisper
 whisper-ctranslate2
 whisperX
 whisper-diarization
 wscribe
 
 References
  Pypi
 
 
 
 Coding by Voice
   New: Introduce Coding by voice.
 Coding by voice command requires two kinds of software: a speech-recognition engine and a platform for voice coding. Dragon from Nuance, a speech-recognition software developer in Burlington, Massachusetts, is an advanced engine and is widely used for programming by voice. On the platform side, VoiceCode by Ben Meyer and Talon by Ryan Hileman (both are for Mac OS only) are popular.
 Coding by voice platforms:
 Two platforms for voice programming are Caster and Aenea, the latter of which runs on Linux. Both are free and open source, and enable voice-programming functionality in Dragonfly, which is an open-source Python framework that links actions with voice commands detected by a speech-recognition engine. Saphra tried Dragonfly, but found that setup required more use of her hands than she could tolerate.
 All of these platforms for voice command work independently of coding language and text editor, and so can also be used for tasks outside programming. Pimentel, for instance, uses voice recognition to write e-mails, which he finds easier, faster and more natural than typing.
 To the untrained ear, coding by voice command sounds like staccato bursts of a secret language. Rudd’s video is full of terms like ‘slap’ (hit return), ‘sup’ (search up) and ‘mara’ (mark paragraph).
 Unlike virtual personal assistants such as Apple’s Siri or Google’s Alexa, VoiceCode and Talon don’t do natural-language processing, so spoken instructions have to precisely match the commands that the system already knows. But both platforms use continuous command recognition, so users needn’t pause between commands, as Siri and Alexa require.
 VoiceCode commands typically use words not in the English language, because if you use an English word as a command, such as ‘return’, it means you can never type out that word. By contrast, Talon, Aenea and Caster feature dynamic grammar, a tool that constantly updates which words the software can recognize on the basis of which applications are open. This means users can give English words as commands without causing confusion.
 In addition to voice recognition, Talon can also replace a computer mouse with eye tracking, which requires a Tobii 4c eye tracker (US$150). Other eye-mousing systems generally require both the eye tracker and head-tracking hardware, such as the TrackIR from NaturalPoint. “I want to make fully hands-free use of every part of a desktop computer a thing,” says Hileman. Other mouse replacements also exist; Pimentel uses one called SmartNav.
 Voice command requires at least a decent headset or microphone. Many users choose a unidirectional microphone so that others can talk to them while they are dictating code. One such mic, a cardioid mic, requires special equipment to supply power, and hardware costs can reach $400, says Pimentel.
 The software can cost several hundred dollars too. The speech-recognition engine Dragon Professional costs $300, as does VoiceCode. Caster and Aenea are free and open source. Talon is available for free, but requires a separate speech-recognition engine. A beta version of Talon that includes a built-in speech-recognition engine is currently available to Hileman’s Patreon supporters for $15 per month. 
 Whether or not users have RSI, it can be difficult and frustrating to start programming by voice. It took a month and a half for Pimentel to get up to speed, he says, and there were days when he was ready to throw in the towel. He printed out 40 pages of commands and forced himself to look at them until he learnt them. Saphra needed two months of coding, a little every day, before she felt that it was a “perfectly enjoyable experience and I could see myself doing this for a living”.
 After the initial learning curve, users often create custom prompts for commonly used commands as the need arises. 
 
 
 Data Analysis
 Parsers
   New: Learning about parsers.
 Parsers are a whole world. I kind of feel a bit lost right now and I'm searching for good books on the topic. So far I've found:
  Crafting Interpreters.
 
 Pros: - Pleasant to read - Doesn't use external tools, you implement it from scratch. - Multiple format: EPUB, PDF, web - You can read it for free - Cute drawings <3
 Cons: - Code snippets are on Java and C - Doesn't use external tools, you implement it from scratch - It's long
  Compilers: Principles, Techniques, and Tools by Aho, Alfred V. & Monica S. Lam & Ravi Sethi & Jeffrey D. Ullman
 
 Pros: - EPUB
 Cons: - Code snippets are on C++
  Parsing Techniques: A Practical Guide by Dick Grune and Ceriel J.H Jacobs
 
 Pros: - Gives an overview of many grammars and parsers
 Cons: - Only in PDF - It's long - Too focused on the theory, despite the name xD
 
 
 Other
   New: Inhibit rules between times.
 To prevent some alerts to be sent between some hours you can use the time_intervals alertmanager configuration.
 This can be useful for example if your backup system triggers some alerts that you don't need to act on.
 route:
  receiver: 'email'
  group_by: [job, alertname, severity]
  group_wait: 5m
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - receiver: 'email'
      matchers:
        - alertname =~ "HostCpuHighIowait|HostContextSwitching|HostUnusualDiskWriteRate"
        - hostname = backup_server
      mute_time_intervals:
        - night
time_intervals:
  - name: night
    time_intervals:
      - times:
          - start_time: 02:00
            end_time: 07:00
 
  New: Thoughts on the reviews themselves.
   Keep It Simple: It's important for the process to be light enough that you want to actually do it, so you see it as a help instead of a burden. It's always better to do a small and quick review rather than nothing at all. At the start of the review analyze yourself to assess how much energy do you have and decide which steps of the review you want to do.
 
  Review approaches: In the past I used the life logging tools to analyze the past in order to understand what I achieved and take it as a base to learn from my mistakes. It was useful when I needed the endorphines boost of seeing all the progress done. Once I assumed that progress speed and understood that we always do the best we can given how we are, I started to feel that the review process was too cumbersome and that it was holding me into the past.
 
 
 Nowadays I try not to look back but forward, analyze the present: how I feel, how's the environment around me, and how can I tweak both to fulfill my life goals. This approach leads to less reviewing of achievements and logs and more introspection, thinking and imagining. Which although may be slower to correct mistakes of the past, will surely make you live closer to the utopy.
 The reviews below then follow that second approach.
  Personal alive reviews: Reviews have to reflect ourselves, and we change continuously, so take for granted that your review is going to change.
 
 I've gone for full blown reviews of locking myself up for a week to not doing reviews for months.
 This article represent the guidelines I follow to do my life review. It may seem a lot to you or may be very simple. Please take it as a base or maybe to get some ideas and then create your own that fits your needs.
 
  New: Update the Month review process.
 
  New: When to do the trimester reviews.
 As with moth reviews, it's interesting to do analysis at representative moments. It gives it an emotional weight. You can for example use the solstices or my personal version of the solstices:
   Spring analysis (1^st of March): For me the spring is the real start of the year, it's when life explodes after the stillness of the winter. The sun starts to set later enough so that you have light in the afternoons, the climate gets warmer thus inviting you to be more outside, the nature is blooming new leaves and flowers. It is then a moment to build new projects and set the current year on track.
 
  Summer analysis (1^st of June): I hate heat, so summer is a moment of retreat. Everyone temporarily stop their lives, we go on holidays and all social projects slow their pace. Even the news have even less interesting things to report. It's so hot outside that some of us seek the cold refuge of home or remote holiday places. Days are long and people love to hang out till late, so usually you wake up later, thus having less time to actually do stuff. Even in the moments when you are alone the heat drains your energy to be productive. It is then a moment to relax and gather forces for the next trimester. It's also perfect to develop easy and chill personal projects that have been forgotten in a drawer. Lower your expectations and just flow with what your body asks you.
 
  Autumn analysis (1^st of September): September it's another key moment for many people. We have it hardcoded in our life since we were children as it was the start of school. People feel energized after the summer holidays and are eager to get back to their lives and stopped projects. You're already 6 months into the year, so it's a good moment to review your year plan and decide how you want to invest your energy reserves.
 
  Winter analysis (1^st of December): December is the cue that the year is coming to an end. The days grow shorter and colder, they basically invite you to enjoy a cup of tea under a blanket. It is then a good time to get into your cave and do an introspection analysis on the whole year and prepare the ground for the coming year. Some of the goals of this season are:
 
 Think everything you need to guarantee a good, solid and powerful spring start.
 Do the year review to adjust your principles.
 
 The year is then divided in two sets of an expansion trimester and a retreat one. We can use this information to adjust our life plan accordingly. In the expansion trimester we could invest more energies in the planning, and in the retreat ones we can do more throughout reviews.
 
  New: The principle documents.
 Principle documents for me are orgmode documents where I think about the principle itself. It acts both as a way of understanding it and evolving my idea around it, and to build the roadmap to materialize the principle's path.
 Without ever having created one I feel that it makes sense to make the reflection part public in the blue book, while I keep for myself the private one. This may also change between principles.
 
  New: The life path document.
 The life path document is an orgmode document where I think about what I want to do with my life and how. It's the highest level of abstraction of the life management system.
 The structure so far is as follows:
 * Life path
** {year}
*** Principles of {season} {year}
    {Notes on the season}
    - Principle 1
    - Principle 2
    ...

**** Objectives of {month} {year}
     - [-] Objective 1
       - [X] SubObjective 1
       - [ ] SubObjective 2
     - [ ] Objective 2
     - [ ] ...
 Where the principles are usually links to principle documents and the objectives links to tasks.
 
  New: Trimester prepare.
 The trimester review requires an analysis that doesn't fill in a day session. It requires slow thinking over some time. So I'm creating a task 10 days before the actual review to start thinking about the next trimester. Whether it's ideas, plans, desires, objectives, or principles.
 Is useful for that document to be available wherever you go, so that in any spare time you can pop it up and continue with the train of thought.
 Doing the reflection without seeing your life path prevents you from being tainted by it, thus representing the real you of right now.
 On the day to actually do the review, follow the steps of the Month review prepare adjusting them to the trimester case.
 
  New: Update the month planning.
 Decide the month objectives:
 Create the month objectives in your roadmap file after addressing each element of:
  Your last month review document.
 The trimester objectives of your roadmap.
 
 Once they are ready, copy them to the description of your todo.org file. That way you'll see it every day.
 For each of your month objectives:
  Decide whether it makes sense to address it this month. If not, archive it
 Create a clear plan of action for this month on that objective
 Define the todo of each device (mobile, tablet, laptop)
 Tweak your things to think about list.
 Tweak your reading list.
 Tweak your week distribution (what to do in each day)
 If you selected maintenance week days tweak the priorities on your maintenance list.
 If you selected improvement week days tweak the priorities on your improvements list.
 Tweak your habit manager system.
 
 
  New: Use deadlines.
 Identify hard deadlines: Add a warning days before the deadline to make sure you're reminded until it's done.
 
  New: Add 2024 Hidden Cup 5 awesome match.
 Semifinal Viper vs Lierey
 
  New: Prometheus metrics.
 Use prometheus-fastapi-instrumentator
 
  Correction: Tweak the month planning.
 Add the next steps:
  Clean your agenda and get an feeling of the busyness of the month:
 Open the orgmode month view agenda and clean it
 Read the rest of your calendars
 
 Then reorder the objectives in order of priority. Try to have at least one objective that improves your life.
  For each of your month and trimester objectives:
 Decide whether it makes sense to address it this month. If not, mark it as inactive
  Create a clear plan of action for this month on that objective.
  Reorder the projects as needed
 Mark as INACTIVE the ones that you don't feel need to be focused on this month.
 
 
  Refine the roadmap of each of the selected areas (change this to the trimestral planning)
 
 Select at least one coding project in case you enter in programming mode
 Clean your mobile browser tabs
 
 
  Correction: Deprecate not-by-ai.
 As they have introduced pricing, which makes no sense, and we had a discussion that using that badge it's a nice way to tell the AI which content to use and which not to
 
  Reorganization: Into roadmap_adjustment.
 
 
      2025-01-01

Software	Home-Gallery	Immich	LibrePhotos
UI	Fine	Good	Fine
Popular (stars)	614	25k	6k
Active (PR/Issues)(1)	?	251/231	27/16
Easy deployment	?	True	Complicated
Good docs	True	True	True
Stable	True	False	True
Smart search	?	True	True
Language	Javascript	Typescript	Python
Batch edit	True	True	?
Multi-user	False	True	?
Mobile app	?	True	?
Oauth support	?	True	?
Facial recognition	?	True	?
Scales well	False	True	?
Favourites	?	True	?
Archive	?	True	?
Has API	True	True	?
Map support	True	True	?
Video Support	True	True	?
Discover similar	True	True	?
Static site	True	False	?


typo:	Typo comments are like todo:, where the main issue is a misspelling.
polish:	Polish comments are like a suggestion, where there is nothing necessarily wrong with the relevant content, there's just some ways to immediately improve the quality. Similar but not exactly the same as `quibble`.

Instance size	Normalization factor
nano	0.25
micro	0.5
small	1
medium	2
large	4
xlarge	8
2xlarge	16
3xlarge	24
4xlarge	32
6xlarge	48
8xlarge	64
9xlarge	72
10xlarge	80
12xlarge	96
16xlarge	128
18xlarge	144
24xlarge	192
32xlarge	256
48xlarge	384
56xlarge	448
112xlarge	896



2023
Lyz — 2024-11-27 15:36:58+00:00
  Activism
   New: Add song for Gaza.
 We Will Not Go Down (Song for Gaza Palestine) - Michael Heart
 
 
 Collaborating tools
   New: Introduce collaborating tools.
 Collaborating document creation:
  https://pad.riseup.net
 https://rustpad.io . Can be self hosted
 
 Collaborating through terminals:
  sshx looks promising although I think it uses their servers to do the connection, which is troublesome.
 
 
 
 Life Management
 Life Management
 Life planning
   New: Introduce the month planning process.
 The objectives of the month plan are:
  Define the month objectives according to the trimester plan and the insights gathered in the past month review.
 Make your backlog and todo list match the month objectives
 Define the philosophical topics to address
 Define the topics to learn
 Define the are of habits to incorporate?
 Define the checks you want to do at the end of the month.
 Plan when is it going to be the next review.
 
 It's interesting to do the plannings on meaningful days such as the first one of the month. Usually we don't have enough flexibility in our life to do it exactly that day, so schedule it the closest you can to that date. It's a good idea to do both the review and the planning on the same day.
 We'll divide the planning process in these phases:
  Prepare
 Clarify your state
 Decide the month objectives
 
 Prepare:
 It's important that you prepare your environment for the planning. You need to be present and fully focused on the process itself. To do so you can:
  Make sure you don't get interrupted: Check your task manager tools to make sure that you don't have anything urgent to address in the next hour.
 Disable all notifications
 
 
 Set your analysis environment: Put on the music that helps you get in the zone.
 Get all the things you may need for the review: The checklist that defines the process of your planning (this document in my case).
 Somewhere to write down the insights.
 Your task manager system
 Your habit manager system
 Your Objective list.
 Your Thinking list.
 Your Reading list.
 
 
 Remove from your environment everything else that may distract you
 
 
 
 Clarify your state:
 To be able to make a good decision on your month's path you need to sort out which is your current state. To do so:
  Clean your inbox: Refile each item until it's empty
 Clean your todo: Review each todo element by deciding if they should still be in the todo. If they do and they belong to a month objective, add it. If they don't need to be in the todo, refile it.
 Clean your someday: Review each relevant someday element (not the ones that are archive at greater levels than month) and decide if they should be refiled elsewhere and if they are part of a month objective that should be dealt with this month.
 Adress each of the trimester objectives by creating month objectives that get you closer to the desired objective.
 
 Decide the next steps:
 For each of your month objectives:
  Decide wheter it makes sense to address it this month. If not, archive it
 Create a clear plan of action for this month on that objective
 Tweak your things to think about list.
 Tweak your reading list.
 Tweak your habit manager system.
 
 
 
 Task Management
 Org Mode
   New: Introduce Nvim Org Mode.
 nvim-orgmode is a Orgmode clone written in Lua for Neovim. Org-mode is a flexible note-taking system that was originally created for Emacs. It has gained wide-spread acclaim and was eventually ported to Neovim.
 The article includes:
  Installation
 Key bindings
 Usage
 Org File
 Headings Toogle line to headline
 Change heading level
 Moving headings
 Folding headings
 Navigate through headings
 
 
 TODO items TODO state customization
 
 
 Priority
 Dates Appointments
 Recurring tasks
 Scheduled
 Deadline
 Date management
 
 
 Tags
 Lists
 Checkboxes
 Links
 Archiving
 Refiling
 Agenda Move around the agenda
 Act on the agenda elements
 Agenda views
 
 
 
 
  New: Capture all your stuff.
 The focus of this process is to capture everything that has your attention, otherwise some part of you will still not totally trust that you're working with the whole picture. While you're doing it, create a list of all the sources of inputs in your world.
 What you're going to do is methodically go through each piece of your life and search for anything that doesn’t permanently belong where it is, the way it is, and put it into your in-tray. You’ll be gathering things that are incomplete or things that have some decision about potential action tied to them. They all go into your “inbox”, so they’ll be available for later processing.
 Be patient, this process may take between 1 and 6 hours, and at the end you'll have a huge pile of stuff in your inbox. You might be scared and get the thought of "what am I doing with my life?", but don't worry you'll get everything in order soon :).
 The steps described in the section so far are:
  Define what is going to be your in-tray
 Physical gathering
 
 
  New: Digital general reference.
 It is very helpful to have a visual map sorted in ways that make sense, either by indexes or data groups organized effectively, usually in an alphabetic order.
 The biggest issue for digitally oriented people is that the ease of capturing and storing has generated a write-only syndrome: all they’re doing is capturing information—not actually accessing and using it intelligently. Some consciousness needs to be applied to keep one’s potentially huge digital library functional, versus a black hole of data easily dumped in there with a couple of keystrokes.
 You need to consistently check how much room to give yourself so that the content remains meaningfully and easily accessible, without creating a black hole of an inordinate amount of information amorphously organized.
 
  New: Physical general reference.
 One idea is to have one system/place where you order the content alphabetically, not multiple ones. People have a tendency to want to use their files as a personal management system, and therefore they attempt to organize them in groupings by projects or areas of focus. This magnifies geometrically the number of places something isn’t when you forget where you filed it.
 
  New: Use telescope plugin for refiling.
 Refiling lets you easily move around elements of your org file, such as headings or TODOs. You can refile with <leader>r with the next snippet:
  When you press the refile key binding you are supposed to press <tab> to see the available options, once you select the correct file, if you will be shown a autocomplete with the possible items to refile it to. Luckily there is a Telescope plugin.
 Install it by adding to your plugin config:
 use 'joaomsa/telescope-orgmode.nvim'
 Then install it with :PackerInstall.
 You can setup the extension by doing:
 require('telescope').load_extension('orgmode')
 To replace the default refile prompt:
 vim.api.nvim_create_autocmd('FileType', {
  pattern = 'org',
  group = vim.api.nvim_create_augroup('orgmode_telescope_nvim', { clear = true })
  callback = function()
    vim.keymap.set('n', '<leader>r', require('telescope').extensions.orgmode.refile_heading)
    vim.keymap.set('n', '<leader>g', require('telescope').extensions.orgmode.search_headings)
  end,
})
 If the auto command doesn't override the default orgmode one, bind it to another keys and never use it.
 The plugin also allows you to use telescope to search through the headings of the different files with search_headings, with the configuration above you'd use <leader>g.
 
  New: Define the Todo list.
 This list contains all the next actions and projects you are going to actively work on. Projects are any desired result that can be accomplished within a year that requires more than one action step. This means that some rather small things you might not normally call projects are going to be on your Projects list, as well as some big ones. If one step won’t complete something, some kind of goalpost needs to be set up to remind you that there’s something still left to do. If you don’t have a placeholder to remind you about it, it will slip back into your head. The reason for the one-year time frame is that anything you are committed to finish within that scope needs to be reviewed weekly to feel comfortable about its status. Another way to think of this is as a list of open loops, no matter what the size. This is going to be one of the lists that you'll review more often, and it needs to be manageable, if the items start to grow you may want to track the elements you want to do in the semester, or trimester.
 Projects do not initially need to be listed in any particular order, by size, or by priority. They just need to be on a master list so you can review them regularly enough to ensure that appropriate next actions have been defined for each of them. That being said, I like to order them a little bit so that I don't need to read the whole list to choose what to do.
 There may be reasons to sort your projects into different subcategories, based upon different areas of your focus, but initially creating a single list of all of them will make it easier to customize your system appropriately as you get more comfortable with its usage. To sort them use tags instead of hierarchical structures, they are more flexible. For example you could use tags for:
  Context: Where can you do the element: home, computer, mobile, ...
 Area: Broad categories where the element falls in: activism, caring, self-caring, home, digital services, ...
 Type: I like to separate the tasks that are meant to survive (maintenance) from the ones that are meant to improve things (improvement)
 Mood, energy level, time: It's useful to have a quick way to see the tasks you can work on when you don't have that much time (small), you don't have that much mental energy (brainless), when you're sad, ...
 
 For many of your projects, you will accumulate relevant information that you will want to organize by theme or topic or project name. Your Projects list will be merely an index. All of the details, plans, and supporting information that you may need as you work on your various projects should be contained in your References system.
 
  New: Define the calendar.
 The calendar holds reminders of actions you need to take fall into two categories: those about things that have to happen on a specific day or time, and those about things that just need to get done as soon as possible. Your calendar handles the first type of reminder.
 These things go on your calendar:
  Time-Specific actions or appointments.
 Day-Specific actions: These are things that you need to do sometime on a certain day, but not necessarily at a specific time.
 Day-Specific information: Information that may be useful on a certain date. This might include directions for appointments, activities that other people will be involved in then, or events of interest. It’s helpful to put short-term tickler information here, too, such as a reminder to call someone after he or she returns from vacation. This is also where you would want to park important reminders about when something might be due, or when something needs to be started, given a determined lead time.
 
 Daily to-do lists don't belong to the calendar because:
   Constant new input and shifting tactical priorities reconfigure daily work so consistently that it’s virtually impossible to nail down to-do items ahead of time. Having a working game plan as a reference point is always useful, but it must be able to be renegotiated at any moment. Trying to keep a list on the calendar, which must then be reentered on another day if items don’t get done, is demoralizing and a waste of time. The Next Actions lists will hold all of those action reminders, even the most time-sensitive ones. And they won’t have to be rewritten daily.
 
  If there’s something on a daily to-do list that doesn’t absolutely have to get done that day, it will dilute the emphasis on the things that truly do. The calendar should be sacred territory. If you write something there, it must get done that day or not at all.
 
 
 That said, there’s absolutely nothing wrong with creating a quick, informal, short list of “if I have time, I’d really like to . . .” kinds of things, picked from your Next Actions inventory. It just should not be confused with your “have-tos,” and it should be treated lightly enough to discard or change quickly as the inevitable surprises of the day unfold.
 
  New: Give an overview of how I'm using gtd.
 Before you start moving stuff around it's a good idea to get the first design of your whole system, in my case I'm going to heavily rely on org-mode to track most of the stuff with a repository with the next structure:
 .
├── calendar
│   ├── personal.org
│   │   ├── One time events
│   │   ├── Recurring events
│   │   ├── Birthdays
│   │   └── Deathdays
│   ├── day.org
│   ├── tickler.org
│   ├── work_1.org
│   ├── work_2.org
│   ├── partner.org
│   ├── family.org
├── inbox
│   ├── computer.org
│   ├── mobile.org
│   └── tablet.org
├── reference
│   ├── blue
│   ├── reference.org
│   └── red
└── todo
    ├── personal.org
    ├── work_1.org
    ├── work_2.org
    └── someday.org
 Where:
  The subtrees behind the .org files are the heading trees.
 All org files go with their respective org_archive ones, they're not shown in the above diagram to keep it simple.
 calendar/personal.org is my personal calendar.
 calendar/day.org is my day planner.
 
 
  New: Define how to use the Personal calendar.
 The calendar/personal.org file holds:
  Appointments: Meant to be used for elements of the org file that have a defined date to occur. You whether do it that date or not do it at all. Avoid using dates to organize your tasks and if you don't do it that day, reschedule it to another date, it's a waste of time, use next actions in your todo instead. If you need to act on it use a TODO element, otherwise a headline is enough An example would be.
 
 * TODO Meet with Marie
<2023-02-24 Fri>

* Internet's birthday
<2023-03-13 Mon>
  Recurring events: Events that not only happen on the given date, but again and again after a certain interval of N hours (h), days (d), weeks (w), months (m), or years (y). The following shows up in the agenda every Wednesday:
 
 * TODO Go to pilates
  <2007-05-16 Wed 12:30 +1w>
  Tickler events: Check the Tickler management section for more details.
 
 Each section has it's own tag: :recurring:, :day:, :birthday:, :deathday:, and the whole file has the :event: tag for easy filtering.
 In rare cases you may want to use the DEADLINE property if you want to be warned in the agenda some days before the date arrives or the SCHEDULED one in case you want to see in the agenda when you start working on the task. Again, don't waste time postponing these dates, if you do, you're using the system wrong.
 
  New: Define how to use the Day planner.
 Some of my day events are semi-regular, meaning that the recurrence options are not powerful enough. For example, I usually go to pilates on Tuesdays, but some weeks I go at 18:00 and others at 19:00. In the past I used a script that interacts with ikhal to create the elements of the day based on some questionary. The idea is to migrate the tool to create appointments in the day under the diary.org file using a datetree structure:
 * 2010
** 2010-12 December
*** 2010-12-20 Tuesday
**** TODO Go to pilates
    <2010-12-20 Tue 19:00-20:00>
 I also use this file to add any diary annotations for my life log. Once this issue is solved it will be really easy to add diary thoughts through the capture feature
 
  New: Define how to use the todo files.
 The todo files are where you track the todo list, which holds your projects and their next steps to work on. The todo/personal.org, todo/work_1.org and todo/work_2.org files of the above schema will be divided into these level 1 headings:
  * Necessary: These projects need to be dealt with immediately and finished as soon as possible
 * Desirable: Here is where most of your elements will be, these are the ones that you think it's important to work on but there is no hard pressure.
 * Optional: These are the projects that it would be nice to work on, but if you don't it's fine.
 
 Projects are any the second level headings with TODO keywords. To see the list of your projects just fold all the items in the todo.org file.
 Inside each section the elements are more less ordered by what I want to work on first. But all projects are actionable, so if I'm not in the mood to do the first ones, I tackle the rest. As such, I try not to spend too much time ordering them.
 I find useful to split the tasks between my life silos, so that I don't even have a chance to think of anything of work_1 when I'm doing my personal stuff or work_2 stuff.
 
  New: Define how to work with projects.
 Given the broad definition of what we consider a project and how they are usually cooked, the system that represents it must be equally flexible, quick to interact with and easy to evolve.
 Every project starts with the title:
 * TODO Improve task management system
 Optionally you can add a description
 * TODO Improve task management system
  Using Openprojects is uncomfortable, I need to find a better system.
 You may have noticed that the description doesn't follow the rules we defined for next actions, that's fine as you don't act on projects, but on their underlying actions. Nevertheless I like to start them with a verb. It may even make sense not to use TODO items but simple headings to define your projects. On one side you don't act on projects so it would make sense to use headings, on the other, it's also interesting to know the project state, which can be easily tracked with the TODO keywords. If you could tell apart headings from TODO items in the agenda views it would make sense to use them. Right now nvim-orgmode let's you select in the agenda views only TODO items or TODO and headings, but you can't select only headings, so at the moment I don't see any good reason not to use TODO items for the projects.
 To define the next actions of a project you can use checklists
 * TODO Improve task management system
  - [-] Read David Allen's GTD book
    - [x] Read chapters 6 and 7
    - [ ] Read chapters 8 and 9
  - [ ] Sum up the book in the blue book
 As your checklists grow they may start to be uncomfortable, for example if it has:
  More than two levels of indentation: It may be hard to follow the logic of the task structure.
 A lot of elements: You can't archive parts of checklists, so as you complete elements, they will still be shown diverting your attention from the things you can actually act upon or making you loose time scrolling to find where they are.
 
 In these cases it makes sense to promote the first level of headings to subprojects:
 * TODO Improve task management system
  * DOING Read David Allen's GTD book
    - [x] Read chapters 6 and 7
    - [ ] Read chapters 8 and 9
  * TODO up the book in the blue book
 That way when Read David Allen's GTD book is done, you can archive it and forget about it.
 If the project starts having many subprojects, it may help to have a section "Outcomes" to define what do you want to achieve with the project. It can be accompanied with a "Next Steps" section to add any subproject or action that doesn't match the defined outcomes, and once you finish the project, you can refile them into new projects.
 
  New: The NEXT state.
 It's useful to have a NEXT state to track the first next action you need to deal with for each project. That way when you open the file, you can go to the top of it and search for NEXT and it will lead you directly to where you need to work on.
 
  New: Define how to manage tags.
 As explained in the todo list section, you can use tags to filter your tasks. I'm using the next ones:
  Area: Broad categories where the element falls in: activism, caring, self-caring, home, digital services, ...
 Type: I like to separate the tasks that are meant to survive (maintenance) from the ones that are meant to improve things (improvement). I use these only in the big projects.
 :long_break:: I'm using this tag to track the small projects that can be done in the long pomodoro breaks. Depending on the kind of long break that I need I then filter for the next tags:
 brainless: If I want to keep on thinking on what I was doing, an example could be emptying the dishwasher, watering the plants, ...
 call: If I want to completely change context and want some social interaction. For example call mom.
 
 
  New: Define how to manage waiting tasks.
 Waiting actions are elements that are blocked for any reason. I use the WAITING TODO keyword to track this state. Under each element you should add that reason and optionally the process you want to follow to unblock it.
 If you need to actively track the evolution of the WAITING status, leave it on the top of your todo. Otherwise set the date you want to check its status and move it to the ticker.org file.
 
  New: Define how to use the tickler.
 The tickler is a system where you postpone actions to a specific date, but not with a calendar mindset where the action needs to be done at that date. With the tickler you schedule the action to enter your inbox that day to decide what are you going to do with it.
 To implement this in orgmode you can add the :tickler: tag to any element that is tracked in the agenda files and once a day you can look at the day's agenda and decide what to do with the action. It's important though that whatever you do with it, you have to remove it from the agenda view in order to only keep the elements that you need to do in the day. You can follow this workflow by:
  Opening the agenda view gaa
 Go to the view of the day vd
 Go to today .
 Search by tickler /tickler
 
 It can also help to review in the weeklies all the ticklers of the week to avoid surprises.
 If you want to make the project go away from your todo or someday until the tickler date, move it to the tickler.org file.
 
  Correction: Keep on defining steps to capture all your stuff.
 As you engage in the capturing step, you may run into one or more of the following problems:
  An item is too big to go in the in-tray: create a post it that represents it or add it as an entry in your digital inbox. If you can, add the date too
 The pile is too big to fit the in-tray: Create visually distinct stacks around the in-tray, even on the floor.
 Doubts whether to trash something: When in doubt keep it, you'll decide about it later when you process the in-tray. What you need to avoid is to get caught up in deciding what to do with the element. That's going to be the next step in the process, let's go one at a time.
 Getting caught up in cleaning and organizing: If it doesn't take that long it's fine but remember the purpose of this process and the fact that we want to finish it as soon as possible. If you discover things you want to change, add them to the in-tray.
 If you encounter stuff that is already on lists and organizers, treat them as everything else in the "in".
 
 Now that the process it's clear let's start.
 Start with the space where you actually do stuff, scan the place centimeter by centimeter with the mindset defined above, check your desk, drawers, floors, walls, shelves, equipment, furniture, fixtures...Then repeat the process with each room of your home.
 
  New: Explain how to capture all your mental stuff.
 Once you already have a nice pile of stuff, think of what has your attention that isn’t represented by something already in your in-tray and record each thought, each idea, each project or thing that occurs you and add it to the inbox.
 To assist in clearing your head, you may want to review the following the next trigger list, item by item, to see if you’ve forgotten anything.
 
  New: Define priorities from A to D.
 I feel more comfortable with these priorities:
  A: Critical
 B: High
 C: Normal
 D: Low
 
 This gives you room to usually work on priorities B-D and if something shows up that is really really important, you can use A. You can set this setting with the next snippet:
 require('orgmode').setup({
  org_priority_highest = 'A',
  org_priority_default = 'C',
  org_priority_lowest = 'D',
})
 
  New: Warn against using DEADLINE.
 Using too many tasks with a DEADLINE will clutter your agenda. Use it only for the actions that you need to have a reminder, instead try to using appointment dates instead.
 If you need a different warning period for a special task, you can specify it. For example setting a warning period of 5 days DEADLINE: <2004-02-29 Sun -5d>. To configure the default number of days add:
 require('orgmode').setup({
  org_deadline_warning_days = 10,
})
 
  Correction: Improve how to use tags.
 When you press the tag key binding you can type:
  tag1: It will add :tag1:.
 tag1:tag2: It will add :tag1:tag2:.
 Press ESC: It will remove all tags from the item.
 
 Tags make use of the hierarchical structure of outline trees. If a heading has a certain tag, all subheadings inherit the tag as well. For example, in the list
 * Meeting with the French group      :work:
** Summary by Frank                  :boss:notes:
*** TODO Prepare slides for him      :action:
 The final heading has the tags work, boss, notes, and action even though the final heading is not explicitly marked with those tags. You can also set tags that all entries in a file should inherit just as if these tags were defined in a hypothetical level zero that surrounds the entire file. Using a line like the next one:
  #+FILETAGS: :Peter:Boss:Secret:
 
  New: How to use links.
 One final aspect of the org file syntax are links. Links are of the form [[link][description]], where link can be an:
  Internal reference
 External reference
 
 A link that does not look like a URL refers to the current document. You can follow it with gx when point is on the link (Default <leader>oo) if you use the next configuration.
 org = {
  org_open_at_point = 'gx',
}
 
  New: Internal document links.
 Org provides several refinements to internal navigation within a document. Most notably:
  [[*Some section]]: points to a headline with the name Some section.
 [[#my-custom-id]]: targets the entry with the CUSTOM_ID property set to my-custom-id.
 
 When the link does not belong to any of the cases above, Org looks for a dedicated target: the same string in double angular brackets, like <<My Target>>.
 If no dedicated target exists, the link tries to match the exact name of an element within the buffer. Naming is done, unsurprisingly, with the NAME keyword, which has to be put in the line before the element it refers to, as in the following example
  #+NAME: My Target
| a  | table      |
|----+------------|
| of | four cells |
 Ultimately, if none of the above succeeds, Org searches for a headline that is exactly the link text but may also include a TODO keyword and tags, or initiates a plain text search.
 Note that you must make sure custom IDs, dedicated targets, and names are unique throughout the document. Org provides a linter to assist you in the process, if needed, but I have not searched yet one for nvim.
 
  New: How to use properties.
 Properties are key-value pairs associated with an entry. They live in a special drawer with the name PROPERTIES. Each property is specified on a single line, with the key (surrounded by colons) first, and the value after it:
 * CD collection
** Classic
*** Goldberg Variations
    :PROPERTIES:
    :Title:     Goldberg Variations
    :Composer:  J.S. Bach
    :Publisher: Deutsche Grammophon
    :NDisks:    1
    :END:
 You may define the allowed values for a particular property Xyz by setting a property Xyz_ALL. This special property is inherited, so if you set it in a level 1 entry, it applies to the entire tree. When allowed values are defined, setting the corresponding property becomes easier and is less prone to typing errors. For the example with the CD collection, we can pre-define publishers and the number of disks in a box like this:
 * CD collection
  :PROPERTIES:
  :NDisks_ALL:  1 2 3 4
  :Publisher_ALL: "Deutsche Grammophon" Philips EMI
  :END:
 If you want to set properties that can be inherited by any entry in a file, use a line like:
  #+PROPERTY: NDisks_ALL 1 2 3 4
 This can be interesting for example if you want to track when was a header created:
 *** Title of header
   :PROPERTIES:
   :CREATED: <2023-03-03 Fri 12:11>
   :END:
 
  New: How to do Agenda searches.
 When using the search agenda view you can:
  Search by TODO states with /WAITING
  Search by tags +home. The syntax for such searches follows a simple boolean logic:
 
  |: or
 
 &: and
 +: include matches
 -: exclude matches
 
 Here are a few examples:
  +computer&+urgent: Returns all items tagged both computer and urgent.
 +computer|+urgent: Returns all items tagged either computer or urgent.
 +computer&-urgent: Returns all items tagged computer and not urgent.
 
 As you may have noticed, the syntax above can be a little verbose, so org-mode offers convenient ways of shortening it. First, - and + imply and if no boolean operator is stated, so example three above could be rewritten simply as:
 +computer-urgent
 Second, inclusion of matches is implied if no + or - is present, so example three could be further shortened to:
 computer-urgent
 Example number two, meanwhile, could be shortened to:
 computer|urgent
 There is no way (as yet) to express search grouping with parentheses. The and operators (&, +, and -) always bind terms together more strongly than or (|). For instance, the following search
 computer|work+email
 Results in all headlines tagged either with computer or both work and email. An expression such as (computer|work)&email is not supported at the moment. You can construct a regular expression though:
 +{computer\|work}+email
  Search by properties: You can search by properties with the PROPERTY="value" syntax. Properties with numeric values can be queried with inequalities PAGES>100. To search by partial searches use a regular expression, for example if the entry had :BIB_TITLE: Mysteries of the Amazon you could use BIB_TITLE={Amazon}
 
 
  New: How to use Capture in orgmode.
 Capture lets you quickly store notes with little interruption of your work flow. It works the next way:
  Open the interface with ;c (Default <leader>oc) that asks you what kind of element you want to capture.
 Select the template you want to use. By default you only have the Task template, that introduces a task into the same file where you're at, select it by pressing t.
 Fill up the template.
 Choose what to do with the captured content:
 Save it to the configured file by pressing ;w (Default <control>c)
 Refile it to a file by pressing ;r (Default <leader>or).
 Abort the capture ;q (Default <leader>ok).
 
 mappings = {
  global = {
    org_capture = ';c',
    },
  capture = {
    org_capture_finalize = ';w',
    org_capture_refile = ';r',
    org_capture_kill = ';q',
  },
}
 
  New: Configure the capture templates.
 Capture lets you define different templates for the different inputs. Each template has the next elements:
  Keybinding: Keys to press to activate the template
 Description: What to show in the capture menu to describe the template
 Template: The actual template of the capture, look below to see how to create them.
 Target: The place where the captured element will be inserted to. For example ~/org/todo.org. If you don't define it it will go to the file configured in org_default_notes_file.
 Headline: An optional headline of the Target file to insert the element.
 
 For example:
 org_capture_templates = {
  t = { description = 'Task', template = '* TODO %?\n  %u' }
}
 For the template you can use the next variables:
  %?:Default cursor position when template is opened
 %t: Prints current date (Example: <2021-06-10 Thu>)
 %T: Prints current date and time (Example: <2021-06-10 Thu 12:30>)
 %u: Prints current date in inactive format (Example: [2021-06-10 Thu])
 %U: Prints current date and time in inactive format (Example: [2021-06-10 Thu 12:30])
 %<FORMAT>: Insert current date/time formatted according to lua date format (Example: %<%Y-%m-%d %A> produces 2021-07-02 Friday)
 %x: Insert content of the clipboard via the "+" register (see :help clipboard)
 %^{PROMPT|DEFAULT|COMPLETION...}: Prompt for input, if completion is provided an :h inputlist will be used
 %(EXP): Runs the given lua code and inserts the result. NOTE: this will internally pass the content to the lua load() function. So the body inside %() should be the body of a function that returns a string.
 %f: Prints the file of the buffer capture was called from.
 %F: Like %f but inserts the full path.
 %n: Inserts the current $USER
 %a: File and line number from where capture was initiated (Example: [[file:/home/user/projects/myfile.txt +2]])
 
 For example:
 {
  T = {
    description = 'Todo',
    template = '* TODO %?\n %u',
    target = '~/org/todo.org'
  },
 j = {
    description = 'Journal',
    template = '\n*** %<%Y-%m-%d %A>\n**** %U\n\n%?',
    target = '~/sync/org/journal.org'
  },
  -- Nested key example:
  e =  'Event',
  er = {
    description = 'recurring',
    template = '** %?\n %T',
    target = '~/org/calendar.org',
    headline = 'recurring'
  },
  eo = {
    description = 'one-time',
    template = '** %?\n %T',
    target = '~/org/calendar.org',
    headline = 'one-time'
  },
  -- Example using a lua function
  r = {
    description = "Repo URL",
    template = "* [[%x][%(return string.match('%x', '([^/]+)$'))]]%?",
    target = "~/org/repos.org",
  }
}
 
  New: Synchronize with external calendars.
 You may want to synchronize your calendar entries with external ones shared with other people, such as nextcloud calendar or google.
 The orgmode docs have a tutorial to sync with google and suggests some orgmode packages that do that, sadly it won't work with nvim-orgmode. We'll need to go the "ugly way" by:
  Downloading external calendar events to ics with vdirsyncer.
 Importing the ics to orgmode
 Editing the events in orgmode
 Exporting from orgmode to ics
 Uploading then changes to the external calendar events with vdirsyncer.
 
 
  New: Comparison with Markdown.
 What I like of Org mode over Markdown:
  The whole interface to interact with the elements of the document through key bindings:
 Move elements around.
 Create elements
 The TODO system is awesome
 The Agenda system
 How it handles checkboxes <3
 Easy navigation between references in the document
 Archiving feature
 Refiling feature
 # is used for comments.
 Create internal document links is easier, you can just copy and paste the heading similar to [[*This is the heading]] on markdown you need to edit it to [](#this-is-the-heading).
 
 What I like of markdown over Org mode:
  The syntax of the headings ## Title better than ** Title. Although it makes sense to have # for comments.
 The syntax of the links: [reference](link) is prettier to read and write than [[link][reference]], although this can be improved if only the reference is shown by your editor (nvim-orgmode doesn't do his yet)
 
 
  New: Note the importance of isolated containers.
 It's critical that all of these containers be kept distinct from one another. They each represent a discrete type of agreement we make with ourselves, to be reminded of at a specific time and in a specific way, and if they lose their edges and begin to blend, much of the value of organizing will be lost. That's why capturing and clarifying what your relationship to them is primary to getting organized.
 
  New: Inbox management.
 Inbox is the container where you capture your stuff. I've found myself capturing stuff in each of my devices: computer, mobile phone and tablet. Each of them has their own org file under the inbox directory. Each of these files has the #+FILETAGS: :inbox: heading so that all elements share the tag.
 Part of the daily planning is to check the computer and mobile inboxes to see if there is anything that needs to be processed on the day. I don't check the tablet inbox as there's usually no urgent stuff there. The rest of the elements will be processed on the weekly review leaving all the inbox files empty.
 
  New: Computer inbox management.
 nvim-orgmode has an awesome feature called capture which lets you capture thoughts with a keystroke. This is awesome as no matter what are you doing inside neovim you can quickly record your thought, action or idea and keep on doing whatever you were doing. It's a very efficient way to record your stuff at the same time as you keep your focus.
 You can use the next capture template:
   org_capture_templates = {
    i = {
      description = "Inbox",
      template = "* TODO %?\n %U",
      target = "~/org/inbox/computer.org",
    },
  }
 
  New: Mobile and tablet inbox management.
 To capture the content on the go you can use the orgzly and then sync them with your computer through syncthing.
 
  New: Calendar management.
 You need to trust your calendar as sacred territory, reflecting the exact hard edges of your day's commitments, which should be noticeable at a glance while you're on the run.
 So for each element you encounter in the calendar ask yourself, does this element need to be done on this hard date? If the answer is no, then the calendar is not the correct place for the element to be.
 Using dates to order your tasks it's a waste of time, because there will always be a thousand of reasons why you can't do all the things you allocate to that day. As these not done issues start piling up, you'll start to get stressed with a lot of things that you were not able to do on the dates you deceived yourself you were supposed to do at and then you need to spend time defining a new date. Use next actions in your todo instead.
 
  New: Priority management.
 You shouldn’t bother to create some external structuring of the priorities on your lists that you’ll then have to rearrange or rewrite as things change. Attempting to impose such scaffolding has been a big source of frustration in many people’s organizing. You’ll be prioritizing more intuitively as you see the whole list against quite a number of shifting variables. The list is just a way for you to keep track of the total inventory of active things to which you have made a commitment, and to have that inventory available for review.
 Therefore I'm going to try not to use orgmode's priorities for the tasks.
 
  New: Soft recurrent tasks.
 There are some tasks that have a soft recurrence meaning that once you do them you don't want them to show up in your list of actions until a specific time has passed. You could use a recurrent DEADLINE or SCHEDULED but as we've seen earlier that will clutter your calendar pretty soon. Try following the next workflow with these tasks:
  Add the :soft_recurrence: tag to keep them tracked.
 Add them to the tickler file with a recurrent appointment date <2023-03-13 Mon ++1w> and the :tickler: tag so that it doesn't show up in the agenda view even if you move it to another file.
 When the appointed day comes you'll review the tickler elements as part of your day's routine. If you think it's time to do it, refile it to the todo.org file, if not, adjust the recurrence period and set the next date. Even though this workflow is reproducing the "kick the can forward" that we want to avoid, the idea is that once you get the period right you'll never have to do it again. If you see that after some iterations the period keeps on changing, maybe this workflow is not working for that kind of task and you may need to think of a better system ¯\(°_o)/¯.
 Once you complete the item, the new one will be spawned, once it has refile it to the tickler file again.
 
 We use appointments instead of DEADLINE or SCHEDULED so that they don't clutter the tickler view if you don't do them on the appointment date.
 Another option is not to archive the DONE tasks and in the weekly reset them to TODO the ones that you want to do the next week.
 
  New: Create an issue in the orgmode repository.
 
  New: Refile from the capture window.
 If you refile from the capture window, until this issue is solved, your task will be refiled but the capture window won't be closed.
 Be careful that it only refiles the first task there is, so you need to close the capture before refiling the next
 
  New: Code blocks syntax.
 Org offers two ways to structure source code in Org documents: in a source code block, and directly inline. Both specifications are shown below.
 A source code block conforms to this structure:
   <body>
 You need to use snippets for this to be usable.
 An inline code block conforms to this structure:
 src_<language>{<body>}
 or
 src_<language>[<header arguments>]{<body>}
 Where:
  #+NAME: <name>: (Optional) Names the source block so it can be called, like a function, from other source blocks or inline code to evaluate or to capture the results. Code from other blocks, other files.
 #+BEGIN_SRC’ … ‘#+END_SRC: (Mandatory) They mark the start and end of a block that Org requires.
 <language>: (Mandatory) It is the identifier of the source code language in the block. See Languages for identifiers of supported languages.
 <switches>: (Optional) Switches provide finer control of the code execution, export, and format.
 <header arguments>: (Optional) Heading arguments control many aspects of evaluation, export and tangling of code blocks. Using Org’s properties feature, header arguments can be selectively applied to the entire buffer or specific subtrees of the Org document.
 <body>: Source code in the dialect of the specified language identifier.
 
 
  New: Narrow/Widen to subtree.
 It's not yet supported to focus or zoom on one task.
 
  New: Interesting things to investigate.
  org-bullets.nvim: Show org mode bullets as UTF-8 characters.
 headlines.nvim: Add few highlight options for code blocks and headlines.
 Sniprun: A neovim plugin to run lines/blocs of code (independently of the rest of the file), supporting multiples languages.
 Python library: Org-rw
 List of plugins
 
 
  New: The orgmode repository file organization.
 How to structure the different orgmode files is something that has always confused me, each one does it's own way, and there are no good posts on why one structure is better than other, people just state what they do.
 I've started with a typical gtd structure with a directory for the todo another for the calendar then another for the references. In the todo I had a file for personal stuff, another for each of my work clients, and the someday.org. Soon making the internal links was cumbersome so I decided to merge the personal todo.org and the someday.org into the same file and use folds to hide uninteresting parts of the file. The reality is that I feel that orgmode is less responsive and that I often feel lost in the file.
 I'm now more into the idea of having files per project in a flat structure and use an index.org file to give it some sense in the same way I do with the mkdocs repositories. Then I'd use internal links in the todo.org file to organize the priorities of what to do next.
 Benefits:
  As we're using a flat structure at file level, the links between the files are less cumbersome file:./project.org::*heading. We only need to have unique easy to remember names for the files, instead of having to think on which directory was the file I want to make the link to. The all in one file structure makes links even easier, just *heading, but the disadvantages make it not worth it.
 You have the liberty to have a generic link like Work on project or if you want to fine grain it, link the specific task of the project
 The todo file will get smaller.
 It has been the natural evolution of other knowledge repositories such as blue
 
 Cons:
  Filenames must be unique. It hasn't been a problem in blue.
 Blue won't be flattened into Vida as it's it's own knowledge repository
 
 
  New: Syncronize orgmode repositories.
 I use orgmode both at the laptop and the mobile, I want to syncronize some files between both with the next requisites:
  The files should be available on the devices when I'm not at home
 The synchronization will be done only on the local network
 The synchronization mechanism will only be able to see the files that need to be synched.
 Different files can be synced to different devices. If I have three devices (laptop, mobile, tablet) I want to sync all mobile files to the laptop but just some to the tablet).
 
 Right now I'm already using syncthing to sync files between the mobile and my server, so it's tempting to use it also to solve this issue. So the first approach is to spawn a syncthing docker at the laptop that connects with the server to sync the files whenever I'm at home.
 I've investigated the next options:
  Mount the whole orgmode repository with syncthing
 Mount a specific directory to sync
 Use the org-orgzly script
 
 
 
 OpenProject
   Correction: Suggest not to use openproject.
 I've decided to use orgmode instead.
 
 
 Knowledge Management
 Spaced Repetition
   New: Introduce spaced repetition references.
  https://collegeinfogeek.com/spaced-repetition-memory-technique/
 The leitner box
 
 
 
 Anki
   New: How to install the latest version.
 Install the dependencies:
 sudo apt-get install zstd
 Download the latest release package.
 Open a terminal and run the following commands, replacing the filename as appropriate:
 tar xaf Downloads/anki-2.1.XX-linux-qt6.tar.zst
cd anki-2.1.XX-linux-qt6
sudo ./install.sh
 
  New: How long to do study sessions.
 I have two study modes:
  When I'm up to date with my cards, I study them until I finish, but usually less than 15 minutes.
 If I have been lazy and haven't checked them in a while (like now) I assume I'm not going to see them all and define a limited amount of time to review them, say 10 to 20 minutes depending on the time/energy I have at the moment.
 
 The relief thought you can have is that as long as you keep a steady pace of 10/20 mins each day, inevitably you'll eventually finish your pending cards as you're more effective reviewing cards than entering new ones
 
  New: What to do with "hard" cards.
 If you're afraid to be stuck in a loop of reviewing "hard" cards, don't be. In reality after you've seen that "hard" card three times in a row you won't mark it as hard again, because you will remember. If you don't maybe there are two reasons:
  The card has too much information that should be subdivided in smaller cards.
 You're not doing a good process of memorizing the contents once they show up.
 
 
  New: What to do with unneeded cards.
 You have three options:
  Suspend: It stops it from showing up permanently until you reactivate it through the browser.
 Bury: Just delays it until the next day.
 Delete: It deletes it forever.
 
 Unless you're certain that you are not longer going to need it, suspend it.
 
  New: Configure self hosted synchronization.
 Explain how to install anki-sync-server and how to configure Ankidroid and Anki. In the end I dropped this path and used Ankidroid alone with syncthing as I didn't need to interact with the decks from the computer. Also the ecosystem of synchronization in Anki at 2023-11-10 is confusing as there are many servers available, not all are compatible with the clients and Anki itself has released it's own so some of the community ones will eventually die.
 
 
 Computer configuration management
   New: Introduce configuration management.
 Configuring your devices is boring, disgusting and complex. Specially when your device dies and you need to reinstall. You usually don't have the time or energy to deal with it, you just want it to work.
 To have a system that allows you to recover from a disaster it's expensive in both time and knowledge, and many people have different solutions.
 This article shows the latest step of how I'm doing it.
 
 
 Game Theory
   New: Add the evolution of trust game theory game.
 Evolution of trust
 Game theory shows us the three things we need for the evolution of trust:
  Repeat interactions: Trust keeps a relationship going, but you need the knowledge of possible future repeat interactions before trust can evolve
 Possible win-wins: You must be playing a non-zero-sum game, a game where it's at least possible that both players can be better off -- a win-win.
 Low miscommunication: If the level of miscommunication is too high, trust breaks down. But when there's a little bit of miscommunication, it pays to be more forgiving
 
 
 
 Coding
 Languages
 Bash snippets
   New: Move a file.
 Use one of the following
 import os
import shutil

os.rename("path/to/current/file.foo", "path/to/new/destination/for/file.foo")
os.replace("path/to/current/file.foo", "path/to/new/destination/for/file.foo")
shutil.move("path/to/current/file.foo", "path/to/new/destination/for/file.foo")
 
  New: Get the root path of a git repository.
 git rev-parse --show-toplevel
 
  New: Get epoch gmt time.
 date -u '+%s'
 
  New: Check the length of an array with jq.
 echo '[{"username":"user1"},{"username":"user2"}]' | jq '. | length'
 
  New: Exit the script if there is an error.
 set -eu
 
  New: Prompt the user for data.
 read -p "Ask whatever" choice
 
  New: How to deal with HostContextSwitching alertmanager alert.
 A context switch is described as the kernel suspending execution of one process on the CPU and resuming execution of some other process that had previously been suspended. A context switch is required for every interrupt and every task that the scheduler picks.
 Context switching can be due to multitasking, Interrupt handling , user & kernel mode switching. The interrupt rate will naturally go high, if there is higher network traffic, or higher disk traffic. Also it is dependent on the application which every now and then invoking system calls.
 If the cores/CPU's are not sufficient to handle load of threads created by application will also result in context switching.
 It is not a cause of concern until performance breaks down. This is expected that CPU will do context switching. One shouldn't verify these data at first place since there are many statistical data which should be analyzed prior to looking into kernel activities. Verify the CPU, memory and network usage during this time.
 You can see which process is causing issue with the next command:
 10:15:24 AM     UID     PID     cswch/s         nvcswch/s       Command
10:15:27 AM     0       1       162656.7        16656.7         systemd
10:15:27 AM     0       9       165451.04       15451.04        ksoftirqd/0
10:15:27 AM     0       10      158628.87       15828.87        rcu_sched
10:15:27 AM     0       11      156147.47       15647.47        migration/0
10:15:27 AM     0       17      150135.71       15035.71        ksoftirqd/1
10:15:27 AM     0       23      129769.61       12979.61        ksoftirqd/2
10:15:27 AM     0       29      2238.38         238.38          ksoftirqd/3
10:15:27 AM     0       43      1753            753             khugepaged
10:15:27 AM     0       443     1659            165             usb-storage
10:15:27 AM     0       456     1956.12         156.12          i915/signal:0
10:15:27 AM     0       465     29550           29550           kworker/3:1H-xfs-log/dm-3
10:15:27 AM     0       490     164700          14700           kworker/0:1H-kblockd
10:15:27 AM     0       506     163741.24       16741.24        kworker/1:1H-xfs-log/dm-3
10:15:27 AM     0       594     154742          154742          dmcrypt_write/2
10:15:27 AM     0       629     162021.65       16021.65        kworker/2:1H-kblockd
10:15:27 AM     0       715     147852.48       14852.48        xfsaild/dm-1
10:15:27 AM     0       886     150706.86       15706.86        irq/131-iwlwifi
10:15:27 AM     0       966     135597.92       13597.92        xfsaild/dm-3
10:15:27 AM     81      1037    2325.25         225.25          dbus-daemon
10:15:27 AM     998     1052    118755.1        11755.1         polkitd
10:15:27 AM     70      1056    158248.51       15848.51        avahi-daemon
10:15:27 AM     0       1061    133512.12       455.12          rngd
10:15:27 AM     0       1110    156230          16230           cupsd
10:15:27 AM     0       1192    152298.02       1598.02         sssd_nss
10:15:27 AM     0       1247    166132.99       16632.99        systemd-logind
10:15:27 AM     0       1265    165311.34       16511.34        cups-browsed
10:15:27 AM     0       1408    10556.57        1556.57         wpa_supplicant
10:15:27 AM     0       1687    3835            3835            splunkd
10:15:27 AM     42      1773    3728            3728            Xorg
10:15:27 AM     42      1996    3266.67         266.67          gsd-color
10:15:27 AM     0       3166    32036.36        3036.36         sssd_kcm
10:15:27 AM     119349  3194    151763.64       11763.64        dbus-daemon
10:15:27 AM     119349  3199    158306          18306           Xorg
10:15:27 AM     119349  3242    15.28           5.8             gnome-shell

pidstat -wt 3 10  > /tmp/pidstat-t.out

Linux 4.18.0-80.11.2.el8_0.x86_64 (hostname)    09/08/2020  _x86_64_    (4 CPU)

10:15:15 AM   UID      TGID       TID   cswch/s   nvcswch/s  Command
10:15:19 AM     0         1         -   152656.7   16656.7   systemd
10:15:19 AM     0         -         1   152656.7   16656.7   |__systemd
10:15:19 AM     0         9         -   165451.04  15451.04  ksoftirqd/0
10:15:19 AM     0         -         9   165451.04  15451.04  |__ksoftirqd/0
10:15:19 AM     0        10         -   158628.87  15828.87  rcu_sched
10:15:19 AM     0         -        10   158628.87  15828.87  |__rcu_sched
10:15:19 AM     0        23         -   129769.61  12979.61  ksoftirqd/2
10:15:19 AM     0         -        23   129769.61  12979.33  |__ksoftirqd/2
10:15:19 AM     0        29         -   32424.5    2445      ksoftirqd/3
10:15:19 AM     0         -        29   32424.5    2445      |__ksoftirqd/3
10:15:19 AM     0        43         -   334        34        khugepaged
10:15:19 AM     0         -        43   334        34        |__khugepaged
10:15:19 AM     0       443         -   11465      566       usb-storage
10:15:19 AM     0         -       443   6433       93        |__usb-storage
10:15:19 AM     0       456         -   15.41      0.00      i915/signal:0
10:15:19 AM     0         -       456   15.41      0.00      |__i915/signal:0
10:15:19 AM     0       715         -   19.34      0.00      xfsaild/dm-1
10:15:19 AM     0         -       715   19.34      0.00      |__xfsaild/dm-1
10:15:19 AM     0       886         -   23.28      0.00      irq/131-iwlwifi
10:15:19 AM     0         -       886   23.28      0.00      |__irq/131-iwlwifi
10:15:19 AM     0       966         -   19.67      0.00      xfsaild/dm-3
10:15:19 AM     0         -       966   19.67      0.00      |__xfsaild/dm-3
10:15:19 AM    81      1037         -   6.89       0.33      dbus-daemon
10:15:19 AM    81         -      1037   6.89       0.33      |__dbus-daemon
10:15:19 AM     0      1038         -   11567.31   4436      NetworkManager
10:15:19 AM     0         -      1038   1.31       0.00      |__NetworkManager
10:15:19 AM     0         -      1088   0.33       0.00      |__gmain
10:15:19 AM     0         -      1094   1340.66    0.00      |__gdbus
10:15:19 AM   998      1052         -   118755.1   11755.1   polkitd
10:15:19 AM   998         -      1052   32420.66   25545     |__polkitd
10:15:19 AM   998         -      1132   0.66       0.00      |__gdbus
 Then with help of PID which is causing issue, one can get all system calls details: Raw
 
 Let this command run for a few minutes while the load/context switch rates are high. It is safe to run this on a production system so you could run it on a good system as well to provide a comparative baseline. Through strace, one can debug & troubleshoot the issue, by looking at system calls the process has made.
 
  New: Redirect stderr of all subsequent commands of a script to a file.
 {
    somecommand
    somecommand2
    somecommand3
} 2>&1 | tee -a $DEBUGLOG
 
  New: Loop through a list of files found by find.
 For simple loops use the find -exec syntax:
 find . -name '*.txt' -exec process {} \;
 For more complex loops use a while read construct:
 find . -name "*.txt" -print0 | while read -r -d $'\0' file
do
    …code using "$file"
done
 The loop will execute while the find command is executing. Plus, this command will work even if a file name is returned with whitespace in it. And, you won't overflow your command line buffer.
 The -print0 will use the NULL as a file separator instead of a newline and the -d $'\0' will use NULL as the separator while reading.
 How not to do it:
 If you try to run the next snippet:
 for file in $(find . -name "*.txt")
do
    …code using "$file"
done
 You'll get the next shellcheck warning:
 SC2044: For loops over find output are fragile. Use find -exec or a while read loop.
 You should not do this because:
 Three reasons:
  For the for loop to even start, the find must run to completion.
 If a file name has any whitespace (including space, tab or newline) in it, it will be treated as two separate names.
 Although now unlikely, you can overrun your command line buffer. Imagine if your command line buffer holds 32KB, and your for loop returns 40KB of text. That last 8KB will be dropped right off your for loop and you'll never know it.
 
 
 
 pytelegrambotapi
   New: Custom file generation.
 During the build, you may want to generate other files or download resources from the internet. You can achieve this by the setup-script build configuration:
 ``toml [tool.pdm.build] setup-script = "build.py" 
In the `build.py` script, pdm-pep517 looks for a build function and calls it with two arguments:

* `src`: the path to the source directory
* `dst`: the path to the distribution directory

Example:

```python
def build(src, dst):
    target_file = os.path.join(dst, "mypackage/myfile.txt")
    os.makedirs(os.path.dirname(target_file), exist_ok=True)
    download_file_to(dst)
 The generated file will be copied to the resulted wheel with the same hierarchy, you need to create the parent directories if necessary.
 
  Correction: Custom file generation.
 Warning: this method only works if you install the package with pdm if you use pip or any other package manager the build.py script won't be called. Thus a more generic approach is to run the initialization steps in a your_command init step or run the checks on each command.
 
  New: Basic concepts of concurrency.
 Concurrency is best explained by an example stolen from Miguel Grinberg.
 Chess master Judit Polgár hosts a chess exhibition in which she plays multiple amateur players. She has two ways of conducting the exhibition: synchronously and asynchronously.
 Assumptions:
  24 opponents
 Judit makes each chess move in 5 seconds
 Opponents each take 55 seconds to make a move
 Games average 30 pair-moves (60 moves total)
 
 Synchronous version: Judit plays one game at a time, never two at the same time, until the game is complete. Each game takes (55 + 5) * 30 == 1800 seconds, or 30 minutes. The entire exhibition takes 24 * 30 == 720 minutes, or 12 hours.
 Asynchronous version: Judit moves from table to table, making one move at each table. She leaves the table and lets the opponent make their next move during the wait time. One move on all 24 games takes Judit 24 * 5 == 120 seconds, or 2 minutes. The entire exhibition is now cut down to 120 * 30 == 3600 seconds, or just 1 hour.
 Async IO takes long waiting periods in which functions would otherwise be blocking and allows other functions to run during that downtime. (A function that blocks effectively forbids others from running from the time that it starts until the time that it returns.)
 
  New: Basic concepts.
  AsyncIO is not easy
 The async/await Syntax and Native Coroutines
 The Rules of Async IO
 Async IO Design Patterns
 Chaining Coroutines
 Using a Queue
 async for and list comprehensions
 Creating and gathering tasks
 When and Why Is Async IO the Right Choice?
 Async IO It Is, but Which One?
 Testing
 
 
  New: Introduce pytelegrambotapi.
 pyTelegramBotAPI is an synchronous and asynchronous implementation of the Telegram Bot API.
 Installation:
 pip install pyTelegramBotAPI
 
  New: Create your bot.
 Use the /newbot command to create a new bot. @BotFather will ask you for a name and username, then generate an authentication token for your new bot.
  The name of your bot is displayed in contact details and elsewhere.
 The username is a short name, used in search, mentions and t.me links. Usernames are 5-32 characters long and not case sensitive – but may only include Latin characters, numbers, and underscores. Your bot's username must end in 'bot’, like tetris_bot or TetrisBot.
 The token is a string, like 110201543:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsaw, which is required to authorize the bot and send requests to the Bot API. Keep your token secure and store it safely, it can be used by anyone to control your bot.
 
 To edit your bot, you have the next available commands:
  /setname: change your bot's name.
 /setdescription: change the bot's description (short text up to 512 characters). Users will see this text at the beginning of the conversation with the bot, titled 'What can this bot do?'.
 /setabouttext: change the bot's about info, a shorter text up to 120 characters. Users will see this text on the bot's profile page. When they share your bot with someone, this text is sent together with the link.
 /setuserpic: change the bot's profile picture.
 /setcommands: change the list of commands supported by your bot. Users will see these commands as suggestions when they type / in the chat with your bot. See commands for more info.
 /setdomain: link a website domain to your bot. See the login widget section.
 /deletebot: delete your bot and free its username. Cannot be undone.
 
 
  New: Synchronous TeleBot.
 import telebot

API_TOKEN = '<api_token>'

bot = telebot.TeleBot(API_TOKEN)

@bot.message_handler(commands=['help', 'start'])
def send_welcome(message):
    bot.reply_to(message, """\
Hi there, I am EchoBot.
I am here to echo your kind words back to you. Just say anything nice and I'll say the exact same thing to you!\
""")

@bot.message_handler(func=lambda message: True)
def echo_message(message):
    bot.reply_to(message, message.text)

bot.infinity_polling()
 
  New: Asynchronous TeleBot.
 from telebot.async_telebot import AsyncTeleBot
bot = AsyncTeleBot('TOKEN')

@bot.message_handler(commands=['help', 'start'])
async def send_welcome(message):
    await bot.reply_to(message, """\
Hi there, I am EchoBot.
I am here to echo your kind words back to you. Just say anything nice and I'll say the exact same thing to you!\
""")

@bot.message_handler(func=lambda message: True)
async def echo_message(message):
    await bot.reply_to(message, message.text)

import asyncio
asyncio.run(bot.polling())
 
 
 Libraries
   New: How to encrypt a file.
 gpg.encrypt_file('path/to/file', recipients)
 Where recipients is a List[str] of gpg Key IDs.
 
  New: List the recipients that can decrypt a file.
 
def list_recipients(self, path: Path) -> List['GPGKey']:
    """List the keys that can decrypt a file.

    Args:
       path: Path to the file to check.
    """
    keys = []
    for short_key in self.gpg.get_recipients_file(str(path)):
        keys.append(self.gpg.list_keys(keys=[short_key])[0]['fingerprint'])

    return keys
 feat(requests#Use a proxy): Use a proxy
 http_proxy  = "http://10.10.1.10:3128"
https_proxy = "https://10.10.1.11:1080"
ftp_proxy   = "ftp://10.10.1.10:3128"

proxies = {
  "http"  : http_proxy,
  "https" : https_proxy,
  "ftp"   : ftp_proxy
}

r = requests.get(url, headers=headers, proxies=proxies)
 
  New: Introduce aiohttp.
 aiohttp is an asynchronous HTTP Client/Server for asyncio and Python. Think of it as the requests for asyncio.
  Installation
 How to make a request
 Passing parameters in urls
 Setting custom headers
 Set custom cookies
 Proxy support
 How to use the ClientSession
 An aiohttp adapter
 
 
  New: Receive keys from a keyserver.
 import_result = gpg.recv_keys('server-name', 'keyid1', 'keyid2', ...)
 
 
 Configure Docker to host the application
   New: Troubleshoot Docker python not showning prints.
 Use CMD ["python","-u","main.py"] instead of CMD ["python","main.py"].
 
  New: [Get the difference of two lists.](../coding/python/python_project_template/python_docker.md#prevent-pip-install--r-requirements.txt-to-run-on-each-docker-build:-prevent-pip-install--r-requirements.txt-to-run-on-each-docker-build
 
 
 i'm-assuming-that-at-some-point-in-your-build-process,-you're-copying-your-entire-application-into-the-docker-image-with-copy-or-add:
 copy-.-/opt/app
workdir-/opt/app
run-pip-install--r-requirements.txt
 the-problem-is-that-you're-invalidating-the-docker-build-cache-every-time-you're-copying-the-entire-application-into-the-image.-this-will-also-invalidate-the-cache-for-all-subsequent-build-steps.
 to-prevent-this,-i'd-suggest-copying-only-the-requirements.txt-file-in-a-separate-build-step-before-adding-the-entire-application-into-the-image:
 copy-requirements.txt-/opt/app/requirements.txt
workdir-/opt/app
run-pip-install--r-requirements.txt
copy-.-/opt/app
 feat(python_snippets)
 If we want to substract the elements of one list from the other you can use:

```python
for x in b:
  if x in a:
    a.remove(x)
```
   New: Override entrypoint.
 sudo docker run -it --entrypoint /bin/bash [docker_image]
 
  New: Disable ipv6.
 sysctl net.ipv6.conf.all.disable_ipv6=1
sysctl net.ipv6.conf.default.disable_ipv6=1
 
  New: Remove the apt cache after installing a package.
 RUN apt-get update && apt-get install -y \
  python3 \
  python3-pip \
  && rm -rf /var/lib/apt/lists/*
 
  New: Add the contents of a directory to the docker.
 ADD ./path/to/directory /path/to/destination
 
  New: Add healthcheck to your dockers.
 Health checks allow a container to expose its workload’s availability. This stands apart from whether the container is running. If your database goes down, your API server won’t be able to handle requests, even though its Docker container is still running.
 This makes for unhelpful experiences during troubleshooting. A simple docker ps would report the container as available. Adding a health check extends the docker ps output to include the container’s true state.
 You configure container health checks in your Dockerfile. This accepts a command which the Docker daemon will execute every 30 seconds. Docker uses the command’s exit code to determine your container’s healthiness:
  0: The container is healthy and working normally.
 1: The container is unhealthy; the workload may not be functioning.
 
 Healthiness isn’t checked straightaway when containers are created. The status will show as starting before the first check runs. This gives the container time to execute any startup tasks. A container with a passing health check will show as healthy; an unhealthy container displays unhealthy.
 In docker-compose you can write the healthchecks like the next snippet:
 ---
version: '3.4'

services:
  jellyfin:
    image: linuxserver/jellyfin:latest
    container_name: jellyfin
    restart: unless-stopped
    healthcheck:
      test: curl http://localhost:8096/health || exit 1
      interval: 10s
      retries: 5
      start_period: 5s
      timeout: 10s
 
  New: List the dockers of a registry.
 List all repositories (effectively images):
 $: curl -X GET https://myregistry:5000/v2/_catalog
> {"repositories":["redis","ubuntu"]}
 List all tags for a repository:
 $: curl -X GET https://myregistry:5000/v2/ubuntu/tags/list
> {"name":"ubuntu","tags":["14.04"]}
 If the registry needs authentication you have to specify username and password in the curl command
 curl -X GET -u <user>:<pass> https://myregistry:5000/v2/_catalog
curl -X GET -u <user>:<pass> https://myregistry:5000/v2/ubuntu/tags/list
 
  New: Searching by attribute and value.
 soup = BeautifulSoup(html)
results = soup.findAll("td", {"valign" : "top"})
 
  New: Install a specific version of Docker.
 Follow these instructions
 If that doesn't install the version of docker-compose that you want use the next snippet:
 VERSION=$(curl --silent https://api.github.com/repos/docker/compose/releases/latest | grep -Po '"tag_name": "\K.*\d')
DESTINATION=/usr/local/bin/docker-compose
sudo curl -L https://github.com/docker/compose/releases/download/${VERSION}/docker-compose-$(uname -s)-$(uname -m) -o $DESTINATION
sudo chmod 755 $DESTINATION
 If you don't want the latest version set the VERSION variable.
 
  New: Dockerize a PDM application.
 It is possible to use PDM in a multi-stage Dockerfile to first install the project and dependencies into __pypackages__ and then copy this folder into the final stage, adding it to PYTHONPATH.
 FROM python:3.11-slim-bookworm AS builder

RUN pip install pdm

COPY pyproject.toml pdm.lock README.md /project/
COPY src/ /project/src

WORKDIR /project
RUN mkdir __pypackages__ && pdm sync --prod --no-editable

FROM python:3.11-slim-bookworm

ENV PYTHONPATH=/project/pkgs
COPY --from=builder /project/__pypackages__/3.11/lib /project/pkgs

COPY --from=builder /project/__pypackages__/3.11/bin/* /bin/

CMD ["python", "-m", "project"]
 
 
 Click
   New: Split stdout from stderr in tests.
 By default the runner is configured to mix stdout and stderr, if you wish to tell apart both sources use:
 def test(runner: CliRunner):
  ...
  runner.mix_stderr = False
 
  New: File system isolation.
 You may need to isolate the environment variables if your application read the configuration from them. To do that override the runner fixture:
 @pytest.fixture(name="runner")
def fixture_runner() -> CliRunner:
    """Configure the Click cli test runner."""
    return CliRunner(
        env={
            'PASSWORD_STORE_DIR': '',
            'GNUPGHOME': '',
            'PASSWORD_AUTH_DIR': '',
        },
        mix_stderr=False
    )
 If you define the fixture in conftest.py you may need to use another name than runner otherwise it may be skipped, for example cli_runner.
 
 
 SQLite
   New: Import a table from another database.
 If you have an SQLite databases named database1 with a table t1 and database2 with a table t2 and want to import table t2 from database2 into database1. You need to open database1 with litecli:
 litecli database1
 Attach the other database with the command:
 ATTACH 'database2file' AS db2;
 Then create the table t2, and copy the data over with:
 INSERT INTO t2 SELECT * FROM db2.t2;
 
 
 Promql
   New: Add basic operations.
 Selecting series:
  Select latest sample for series with a given metric name:
 
 node_cpu_seconds_total
  Select 5-minute range of samples for series with a given metric name:
 
 node_cpu_seconds_total[5m]
  Only series with given label values:
 
 node_cpu_seconds_total{cpu="0",mode="idle"}
  Complex label matchers (=: equality, !=: non-equality, =~: regex match, !~: negative regex match):
 
 node_cpu_seconds_total{cpu!="0",mode=~"user|system"}
  Select data from one day ago and shift it to the current time:
 
 process_resident_memory_bytes offset 1d
 Rates of increase for counters:
  Per-second rate of increase, averaged over last 5 minutes:
 
 rate(demo_api_request_duration_seconds_count[5m])
  Per-second rate of increase, calculated over last two samples in a 1-minute time window:
 
 irate(demo_api_request_duration_seconds_count[1m])
  Absolute increase over last hour:
 
 increase(demo_api_request_duration_seconds_count[1h])
 Aggregating over multiple series:
  Sum over all series:
 
 sum(node_filesystem_size_bytes)
  Preserve the instance and job label dimensions:
 
 sum by(job, instance) (node_filesystem_size_bytes)
  Aggregate away the instance and job label dimensions:
 
 sum without(instance, job) (node_filesystem_size_bytes)
 Available aggregation operators: sum(), min(), max(), avg(), stddev(), stdvar(), count(), count_values(), group(), bottomk(), topk(), quantile().
 Time:
  Get the Unix time in seconds at each resolution step:
 
 time()
  Get the age of the last successful batch job run:
 
 time() - demo_batch_last_success_timestamp_seconds
  Find batch jobs which haven't succeeded in an hour:
 
 time() - demo_batch_last_success_timestamp_seconds > 3600
 
  New: Run operation only on the elements that match a condition.
 Imagine we want to run the zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes operation only on the elements that match zfs_dataset_used_by_dataset_bytes > 200e3. You can do this with and:
 zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes and zfs_dataset_used_by_dataset_bytes > 200e3
 
  New: Substracting two metrics.
 To run binary operators between vectors you need them to match. Basically it means that it will only do the operation on the elements that have the same labels. Sometimes you want to do operations on metrics that don't have the same labels. In those cases you can use the on operator. Imagine that we want to substract the next vectors:
 zfs_dataset_used_bytes{type='filesystem'}
 And
 sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'})
 That only have in common the labels hostname and filesystem`.
 You can use the next expression then:
 zfs_dataset_used_bytes{type='filesystem'} - on (hostname, filesystem) sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'})
 To learn more on Vector matching read this article
 
  New: Ranges only allowed for vector selectors.
 You may need to specify a subquery range such as [1w:1d].
 
 
 Logql
   New: Introduce LogQL.
 LogQL is Grafana Loki’s PromQL-inspired query language. Queries act as if they are a distributed grep to aggregate log sources. LogQL uses labels and operators for filtering.
 There are two types of LogQL queries:
  Log queries: Return the contents of log lines.
 Metric queries: Extend log queries to calculate values based on query results.
 
 
  New: Apply a pattern to the value of a label.
 Some logs are sent in json and then one of their fields can contain other structured data. You may want to use that structured data to further filter the logs.
 {app="ingress-nginx"} | json | line_format `{{.log}}` | pattern `<_> - - <_> "<method> <_> <_>" <status> <_> <_> "<_>" <_>` | method != `GET`
  {app="ingress-nginx"}: Show only the logs of the ingress-nginx.
 | json: Interpret the line as a json.
 ``| line_format}| pattern<> - - <> " <> <>"  <> <> "<>" <>```: interpret thelog` json field of the trace with the selected pattern
 ``| method !=GET````: Filter the line using a key extracted by the pattern.
 
 
  New: Count the unique values of a label.
 Sometimes you want to alert on the values of a log. For example if you want to make sure that you're receiving the logs from more than 20 hosts (otherwise something is wrong). Assuming that your logs attach a host label you can run
 sum(count by(host) (rate({host=~".+"} [24h])) > bool 0)
 This query will: - {host=~".+"}: Fetch all log lines that contain the label host - count by(host) (rate({host=~".+"} [24h]): Calculates the number of entries in the last 24h. - count by(host) (rate({host=~".+"} [24h])) > bool 0: Converts to 1 all the vector elements that have more than 1 message. - sum(count by(host) (rate({host=~".+"} [24h])) > bool 0): Sums all the vector elements to get the number of hosts that have more than one message.
 journald promtail parser is known to fail between upgrades, it's useful too to make an alert to make sure that all your hosts are sending the traces. You can do it with: sum(count by(host) (rate({job="systemd-journal"} [24h])) > bool 0)
 
 
 Pytest
   New: Stop pytest right at the start if condition not met.
 Use the pytest_configure initialization hook.
 In your global conftest.py:
 import requests
import pytest

def pytest_configure(config):
    try:
        requests.get(f'http://localhost:9200')
    except requests.exceptions.ConnectionError:
        msg = 'FATAL. Connection refused: ES does not appear to be installed as a service (localhost port 9200)'
        pytest.exit(msg)
  Note that the single argument of pytest_configure has to be named config.
 Using pytest.exit makes it look nicer.
 
 
  New: Introduce pytest-xprocess.
 pytest-xprocess is a pytest plugin for managing external processes across test runs.
 Installation:
 pip install pytest-xprocess
 Usage:
 Define your process fixture in conftest.py:
 import pytest
from xprocess import ProcessStarter

@pytest.fixture
def myserver(xprocess):
    class Starter(ProcessStarter):
        # startup pattern
        pattern = "[Ss]erver has started!"

        # command to start process
        args = ['command', 'arg1', 'arg2']

    # ensure process is running and return its logfile
    logfile = xprocess.ensure("myserver", Starter)

    conn = # create a connection or url/port info to the server
    yield conn

    # clean up whole process tree afterwards
    xprocess.getinfo("myserver").terminate()
 Now you can use this fixture in any test functions where myserver needs to be up and xprocess will take care of it for you.
 Matching process output with pattern:
 In order to detect that your process is ready to answer queries, pytest-xprocess allows the user to provide a string pattern by setting the class variable pattern in the Starter class. pattern will be waited for in the process logfile for a maximum time defined by timeout before timing out in case the provided pattern is not matched.
 It’s important to note that pattern is a regular expression and will be matched using python re.search.
 Controlling Startup Wait Time with timeout:
 Some processes naturally take longer to start than others. By default, pytest-xprocess will wait for a maximum of 120 seconds for a given process to start before raising a TimeoutError. Changing this value may be useful, for example, when the user knows that a given process would never take longer than a known amount of time to start under normal circunstancies, so if it does go over this known upper boundary, that means something is wrong and the waiting process must be interrupted. The maximum wait time can be controlled through the class variable timeout.
    @pytest.fixture
   def myserver(xprocess):
       class Starter(ProcessStarter):
           # will wait for 10 seconds before timing out
           timeout = 10
 Passing command line arguments to your process with args:
 In order to start a process, pytest-xprocess must be given a command to be passed into the subprocess.Popen constructor. Any arguments passed to the process command can also be passed using args. As an example, if I usually use the following command to start a given process:
 $> myproc -name "bacon" -cores 4 <destdir>
 That would look like:
 args = ['myproc', '-name', '"bacon"', '-cores', 4, '<destdir>']
 when using args in pytest-xprocess to start the same process.
 @pytest.fixture
def myserver(xprocess):
    class Starter(ProcessStarter):
        # will pass "$> myproc -name "bacon" -cores 4 <destdir>"  to the
        # subprocess.Popen constructor so the process can be started with
        # the given arguments
        args = ['myproc', '-name', '"bacon"', '-cores', 4, '<destdir>']

        # ...
 
 
 Python Snippets
   New: Substract two paths.
 It can also framed to how to get the relative path between two absolute paths:
 >>> from pathlib import Path
>>> p = Path('/home/lyz/')
>>> h = Path('/home/')
>>> p.relative_to(h)
PosixPath('lyz')
 
  New: Copy files from a python package.
 pkgdir = sys.modules['<mypkg>'].__path__[0]
fullpath = os.path.join(pkgdir, <myfile>)
shutil.copy(fullpath, os.getcwd())
 
  New: Sort the returned paths of glob.
 glob order is arbitrary, but you can sort them yourself.
 If you want sorted by name:
 sorted(glob.glob('*.png'))
 sorted by modification time:
 import os
sorted(glob.glob('*.png'), key=os.path.getmtime)
 sorted by size:
 import os
sorted(glob.glob('*.png'), key=os.path.getsize)
 
  New: Read file with Pathlib.
 file_ = Path('/to/some/file')
file_.read_text()
 
  New: Get changed time of a file.
 import os

os.path.getmtime(path)
 
  New: Configure the logging of a program to look nice.
 def load_logger(verbose: bool = False) -> None:  # pragma no cover
    """Configure the Logging logger.

    Args:
        verbose: Set the logging level to Debug.
    """
    logging.addLevelName(logging.INFO, "\033[36mINFO\033[0m")
    logging.addLevelName(logging.ERROR, "\033[31mERROR\033[0m")
    logging.addLevelName(logging.DEBUG, "\033[32mDEBUG\033[0m")
    logging.addLevelName(logging.WARNING, "\033[33mWARNING\033[0m")

    if verbose:
        logging.basicConfig(
            format="%(asctime)s %(levelname)s %(name)s: %(message)s",
            stream=sys.stderr,
            level=logging.DEBUG,
            datefmt="%Y-%m-%d %H:%M:%S",
        )
        telebot.logger.setLevel(logging.DEBUG)  # Outputs debug messages to console.
    else:
        logging.basicConfig(
            stream=sys.stderr, level=logging.INFO, format="%(levelname)s: %(message)s"
        )
 
  New: Get the modified time of a file with Pathlib.
 file_ = Path('/to/some/file')
file_.stat().st_mtime
 You can also access:
  Created time: with st_ctime
 Accessed time: with st_atime
 
 They are timestamps, so if you want to compare it with a datetime object use the timestamp method:
 assert datetime.now().timestamp - file_.stat().st_mtime < 60
 
  New: Show the date in the logging module traces.
 To display the date and time of an event, you would place %(asctime)s in your format string:
 import logging
logging.basicConfig(format='%(asctime)s %(message)s')
logging.warning('is when this event was logged.')
 
  New: Remove html url characters.
 To transform an URL string into normal string, for example replacing %20 with space use:
 >>> from urllib.parse import unquote
>>> print(unquote("%CE%B1%CE%BB%20"))
αλ
 
 
 Jinja2
   New: Escape jinja expansion on a jinja template.
 {% raw %}

Anything in this block is treated as raw text,
including {{ curly braces }} and
{% other block-like syntax %}

{% endraw %}
 
 
 mkdocstrings
   Correction: Correct the watch directive.
 watch is a list of directories to watch while serving the documentation. So if any file is changed in those directories, the documentation is rebuilt.
 
 
 Pydantic
   New: Investigate libraries.
 Integration of pydantic with pandas
 
  Correction: Initialize attributes at object creation.
 pydantic recommends using root validators, but it's difficult to undestand how to do it and to debug the errors. You also don't have easy access to the default values of the model. I'd rather use the overwriting the __init__ method.
 class fish(BaseModel):
    name: str
    color: str

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        print("Fish initialization successful!")
        self.color=complex_function()
 
 
 Python Telegram
   New: Analyze the different python libraries to interact with telegram.
 There are two ways to interact with Telegram through python:
  Client libraries
 Bot libraries
 
 Client libraries:
 Client libraries use your account to interact with Telegram itself through a developer API token.
 The most popular to use is Telethon.
 Bot libraries:
 Telegram lists many libraries to interact with the bot API, the most interesting are:
  python-telegram-bot
 pyTelegramBotAPI
 aiogram
 
 If there comes a moment when we have to create the messages ourselves, telegram-text may be an interesting library to check.
 python-telegram-bot:
 Pros:
  Popular: 23k stars, 4.9k forks
 Maintained: last commit 3 days ago
 They have a developers community to get help in this telegram group
 I like how they try to minimize third party dependencies, and how you can install the complements if you need them
 Built on top of asyncio
 Nice docs
 Fully supports the Telegram bot API
 Has many examples
 
 Cons:
  Interface is a little verbose and complicated at a first look
 Only to be run in a single thread (not a problem)
 
 References:
  Package documentation is the technical reference for python-telegram-bot. It contains descriptions of all available classes, modules, methods and arguments as well as the changelog.
 Wiki is home to number of more elaborate introductions of the different features of python-telegram-bot and other useful resources that go beyond the technical documentation.
 Examples section contains several examples that showcase the different features of both the Bot API and python-telegram-bot
 Source
 
 pyTelegramBotAPI:
 Pros:
  Popular: 7.1k stars, 1.8k forks
 Maintained: last commit 3 weeks ago
 Both sync and async
 Nicer interface with decorators and simpler setup
 They have an example on how to split long messages
 Nice docs on how to test
 They have a developers community to get help in this telegram group
 Fully supports the Telegram bot API
 Has examples
 
 Cons:
  Uses lambdas inside the decorators, I don't know why it does it.
 The docs are not as throughout as python-telegram-bot one.
 
 References:
  Documentation
 Source
 Async Examples
 
 aiogram:
 Pros:
  Popular: 3.8k stars, 717k forks
 Maintained: last commit 4 days ago
 Async support
 They have a developers community to get help in this telegram group
 Has type hints
 Cleaner interface than python-telegram-bot
 Fully supports the Telegram bot API
 Has examples
 
 Cons:
  Less popular than python-telegram-bot
 Docs are written at a developer level, difficult initial barrier to understand how to use it.
 
 References:
  Documentation
 Source
 Examples
 
 Conclusion:
 Even if python-telegram-bot is the most popular and with the best docs, I prefer one of the others due to the easier interface. aiograms documentation is kind of crap, and as it's the first time I make a bot I'd rather have somewhere good to look at.
 So I'd say to go first with pyTelegramBotAPI and if it doesn't go well, fall back to python-telegram-bot.
 
 
 questionary
   New: Autocomplete answers.
 If you want autocomplete with fuzzy finding use:
 import questionary
from prompt_toolkit.completion import FuzzyWordCompleter

questionary.autocomplete(
    "Save to (q to cancel): ",
    choices=destination_directories,
    completer=FuzzyWordCompleter(destination_directories),
).ask()
 
 
 rich
   New: Tree console view.
 Rich has a Tree class which can generate a tree view in the terminal. A tree view is a great way of presenting the contents of a filesystem or any other hierarchical data. Each branch of the tree can have a label which may be text or any other Rich renderable.
 The following code creates and prints a tree with a simple text label:
 from rich.tree import Tree
from rich import print

tree = Tree("Rich Tree")
print(tree)
 With only a single Tree instance this will output nothing more than the text “Rich Tree”. Things get more interesting when we call add() to add more branches to the Tree. The following code adds two more branches:
 tree.add("foo")
tree.add("bar")
print(tree)
 The tree will now have two branches connected to the original tree with guide lines.
 When you call add() a new Tree instance is returned. You can use this instance to add more branches to, and build up a more complex tree. Let’s add a few more levels to the tree:
 baz_tree = tree.add("baz")
baz_tree.add("[red]Red").add("[green]Green").add("[blue]Blue")
print(tree)
 
 
 Selenium
   New: Solve element isn't clickable in headless mode.
 There are many things you can try to fix this issue. Being the first to configure the driver to use the full screen. Assuming you're using the undetectedchromedriver:
 import undetected_chromedriver.v2 as uc

options = uc.ChromeOptions()

options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")
options.add_argument("--headless")
options.add_argument("--start-maximized")
options.add_argument("--window-size=1920,1080")
driver = uc.Chrome(options=options)
 If that doesn't solve the issue use the next function:
 def click(driver: uc.Chrome, xpath: str, mode: Optional[str] = None) -> None:
    """Click the element marked by the XPATH.

    Args:
        driver: Object to interact with selenium.
        xpath: Identifier of the element to click.
        mode: Type of click. It needs to be one of [None, position, wait]

    The different ways to click are:

    * None: The normal click of the driver.
    * wait: Wait until the element is clickable and then click it.
    * position: Deduce the position of the element and then click it with a javascript script.
    """
    if mode is None:
       driver.find_element(By.XPATH, xpath).click()
    elif mode == 'wait':
        # https://stackoverflow.com/questions/59808158/element-isnt-clickable-in-headless-mode
        WebDriverWait(driver, 20).until(
            EC.element_to_be_clickable((By.XPATH, xpath))
        ).click()
    elif mode == 'position':
        # https://stackoverflow.com/questions/16807258/selenium-click-at-certain-position
        element = driver.find_element(By.XPATH, xpath)
        driver.execute_script("arguments[0].click();", element)
 
 
 sh
   New: Passing environmental variables to commands.
 The _env special kwarg allows you to pass a dictionary of environment variables and their corresponding values:
 import sh
sh.google_chrome(_env={"SOCKS_SERVER": "localhost:1234"})
 _env replaces your process’s environment completely. Only the key-value pairs in _env will be used for its environment. If you want to add new environment variables for a process in addition to your existing environment, try something like this:
 import os
import sh

new_env = os.environ.copy()
new_env["SOCKS_SERVER"] = "localhost:1234"

sh.google_chrome(_env=new_env)
 
  New: Use commands that return a SyntaxError.
 pass is a reserved python word so sh fails when calling the password store command pass.
 pass_command = sh.Command('pass')
pass_command('show', 'new_file')
 
 
 Typer
   New: Print to stderr.
 You can print to "standard error" with a Rich Console(stderr=True)
 from rich.console import Console

err_console = Console(stderr=True)
err_console.print("error message")
 
 
 Generic Coding Practices
 How to code
   New: Personal evolution on how I code.
 Over the years I've tried different ways of developing my code:
  Mindless coding: write code as you need to make it work, with no tests, documentation or any quality measure.
 TDD.
 Try to abstract everything to minimize the duplication of code between projects.
 
 Each has it's advantages and disadvantages. After trying them all and given that right now I only have short spikes of energy and time to invest in coding my plan is to:
  Make the minimum effort to design the minimum program able to solve the problem at hand. This design will be represented in an orgmode task.
 Write the minimum code to make it work without thinking of tests or generalization, but with the domain driven design concepts so the code remains flexible and maintainable.
 Once it's working see if I have time to improve it:
 Create the tests to cover the critical functionality (no more 100% coverage).
 If I need to make a package or the program evolves into something complex I'd use this scaffold template.
 
 Once the spike is over I'll wait for a new spike to come either because I have time or because something breaks and I need to fix it.
 
 
 Git
   New: Set the upstream remote by default.
 git config --global --add push.default current
git config --global --add push.autoSetupRemote true
 
  New: Remove tags.
 To delete a tag you can run:
 git tag -d {{tag_name}}
 To remove them remotely do
 git push --delete origin {{ tag_name }}
 
 
 DevOps
 Infrastructure as Code
 Ansible Snippets
   New: Introduce Forgejo.
 Forgejo is a self-hosted lightweight software forge. Easy to install and low maintenance, it just does the job. The awful name comes from forĝejo, the Esperanto word for forge. I kind of like though the concept of forge for the repositories.
 Brought to you by an inclusive community under the umbrella of Codeberg e.V., a democratic non-profit organization, Forgejo can be trusted to be exclusively Free Software. It is a "soft" fork of Gitea with a focus on scaling, federation and privacy.
 In October 2022 the domains and trademark of Gitea were transferred to a for-profit company without knowledge or approval of the community. Despite writing an open letter, the takeover was later confirmed. The goal of Forgejo is to continue developing the code with a healthy democratic governance.
 On the 15^th of December of 2022 the project was born with these major objectives:
  The community is in control, and ensures we develop to address community needs.
 We will help liberate software development from the shackles of proprietary tools.
 
 One of the approaches to achieve the last point is through pushing for the Forgejo federation a much needed feature in the git web application ecosystem.
 On the 29^th of December of 2022 they released the first stable release and they have released several security releases between then and now.
 Despite what you choose, the good thing is that as long as it's a soft fork migrating between these software should be straight forward.
 Forgejo outshines Gitea in:
  Being built up by the people for the people. The project may die but it's not likely it will follow Gitea's path.
 They are transparent regarding the gobernance of the project which is created through open community discussions.
 It's a political project that fights for the people's rights, for example through federation and freely incorporating the new additions of Gitea
 They'll eventually have a better license
 They get all the features and fixes of Gitea plus the contributions of the developers of the community that run out of Gitea.
 
 Gitea on the other hand has the next advantages:
  It's a more stable project, it's been alive for much more time and now has the back up of a company trying to make profit out of it. Forgejo's community and structure is still evolving to a stable state though, although it looks promising!
 Quicker releases. As Forgejo needs to review and incorporate Gitea's contributions, it takes longer to do a release.
 
 Being a soft-fork has it's disadvantages too, for example deciding where to open the issues and pull requests, they haven't yet decided which is their policy around this topic.
 
  New: Authorize an SSH key.
 - name: Authorize the sender ssh key
  authorized_key:
    user: syncoid
    state: present
    key: "{{ syncoid_receive_ssh_key }}"
 
  New: Create a user.
 The following snippet creates a user with password login disabled.
 - name: Create the syncoid user
  ansible.builtin.user:
    name: syncoid
    state: present
    password: !
    shell: /usr/sbin/nologin
 If you don't set a password any user can do su your_user to set a random password use the next snippet:
 - name: Create the syncoid user
  ansible.builtin.user:
    name: syncoid
    state: present
    password: "{{ lookup('password', '/dev/null', length=50, encrypt='sha512_crypt') }}"
    shell: /bin/bash
 This won't pass the idempotence tests as it doesn't save the password anywhere (/dev/null) in the controler machine.
 
  New: Create an ssh key.
 - name: Create .ssh directory
  become: true
  file:
    path: /root/.ssh
    state: directory
    mode: 700

- name: Create the SSH key to directory
  become: true
  openssh_keypair:
    path: /root/.ssh/id_ed25519
    type: ed25519
  register: ssh

- name: Show public key
  debug:
    var: ssh.public_key
 
  New: Skip ansible-lint for some tasks.
 - name: Modify permissions
  command: >
    chmod -R g-w /home/user
  tags:
    - skip_ansible_lint
  sudo: yes
 
  New: Start and enable a systemd service.
 - name: Start the service
  become: true
  systemd:
    name: zfs_exporter
    enabled: true
    daemon_reload: true
    state: started
 
  New: Download an decompress a tar.gz.
 - name: Unarchive a file that needs to be downloaded (added in 2.0)
  ansible.builtin.unarchive:
    src: https://example.com/example.zip
    dest: /usr/local/bin
    remote_src: yes
 If you want to only extract a file you can use the includes arg
 - name: Download the zfs exporter
  become: true
  ansible.builtin.unarchive:
    src: https://github.com/pdf/zfs_exporter/releases/download/v{{ zfs_exporter_version }}/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64.tar.gz
    dest: /usr/local/bin
    include: zfs_exporter
    remote_src: yes
    mode: 0755
 But that snippet sometimes fail, you can alternatively download it locally and copy it:
 - name: Test if zfs_exporter binary exists
  stat:
    path: /usr/local/bin/zfs_exporter
  register: zfs_exporter_binary

- name: Install the zfs exporter
  block:
    - name: Download the zfs exporter
      delegate_to: localhost
      ansible.builtin.unarchive:
        src: https://github.com/pdf/zfs_exporter/releases/download/v{{ zfs_exporter_version }}/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64.tar.gz
        dest: /tmp/
        remote_src: yes

    - name: Upload the zfs exporter to the server
      become: true
      copy:
        src: /tmp/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64/zfs_exporter
        dest: /usr/local/bin
        mode: 0755
  when: not zfs_exporter_binary.stat.exists
 
  New: Run command on a working directory.
 - name: Change the working directory to somedir/ and run the command as db_owner
  ansible.builtin.command: /usr/bin/make_database.sh db_user db_name
  become: yes
  become_user: db_owner
  args:
    chdir: somedir/
    creates: /path/to/database
 
  New: Run handlers in the middle of the tasks file.
 If you need handlers to run before the end of the play, add a task to flush them using the meta module, which executes Ansible actions:
 tasks:
  - name: Some tasks go here
    ansible.builtin.shell: ...

  - name: Flush handlers
    meta: flush_handlers

  - name: Some other tasks
    ansible.builtin.shell: ...
 The meta: flush_handlers task triggers any handlers that have been notified at that point in the play.
 Once handlers are executed, either automatically after each mentioned section or manually by the flush_handlers meta task, they can be notified and run again in later sections of the play.
 
  New: Run command idempotently.
 - name: Register the runner in gitea
  become: true
  command: act_runner register --config config.yaml --no-interactive --instance {{ gitea_url }} --token {{ gitea_docker_runner_token }}
  args:
    creates: /var/lib/gitea_docker_runner/.runner
 
  New: Get the correct architecture string.
 If you have an amd64 host you'll get x86_64, but sometimes you need the amd64 string. On those cases you can use the next snippet:
 ---
deb_architecture:
  aarch64: arm64
  x86_64: amd64

---
- name: Download the act runner binary
  become: True
  ansible.builtin.get_url:
    url: https://dl.gitea.com/act_runner/act_runner-linux-{{ deb_architecture[ansible_architecture] }}
    dest: /usr/bin/act_runner
    mode: '0755'
 
  New: Check the instances that are going to be affected by playbook run.
 Useful to list the instances of a dynamic inventory
 ansible-inventory -i aws_ec2.yaml --list
 
  New: Check if variable is defined or empty.
 In Ansible playbooks, it is often a good practice to test if a variable exists and what is its value.
 Particularity this helps to avoid different “VARIABLE IS NOT DEFINED” errors in Ansible playbooks.
 In this context there are several useful tests that you can apply using Jinja2 filters in Ansible.
 
  New: Check if Ansible variable is defined (exists).
 tasks:

- shell: echo "The variable 'foo' is defined: '{{ foo }}'"
  when: foo is defined

- fail: msg="The variable 'bar' is not defined"
  when: bar is undefined
 
  New: Check if Ansible variable is empty.
 tasks:

- fail: msg="The variable 'bar' is empty"
  when: bar|length == 0

- shell: echo "The variable 'foo' is not empty: '{{ foo }}'"
  when: foo|length > 0
 
  New: Check if Ansible variable is defined and not empty.
 tasks:

- shell: echo "The variable 'foo' is defined and not empty"
  when: (foo is defined) and (foo|length > 0)

- fail: msg="The variable 'bar' is not defined or empty"
  when: (bar is not defined) or (bar|length == 0)
 
  New: Download a file.
 - name: Download foo.conf
  ansible.builtin.get_url:
    url: http://example.com/path/file.conf
    dest: /etc/foo.conf
    mode: '0440'
 
  New: Ansible condition that uses a regexp.
 - name: Check if an instance name or hostname matches a regex pattern
  when: inventory_hostname is not match('molecule-.*')
  fail:
    msg: "not a molecule instance"
 
  New: Ansible-lint doesn't find requirements.
 It may be because you're using requirements.yaml instead of requirements.yml. Create a temporal link from one file to the other, run the command and then remove the link.
 It will work from then on even if you remove the link. ¯\(°_o)/¯
 
  New: Run task only once.
 Add run_once: true on the task definition:
 - name: Do a thing on the first host in a group.
  debug:
    msg: "Yay only prints once"
  run_once: true
 
  New: Ansible add a sleep.
 - name: Pause for 5 minutes to build app cache
  ansible.builtin.pause:
    minutes: 5
 
  New: Ansible lint skip some rules.
 Add a .ansible-lint-ignore file with a line per rule to ignore with the syntax path/to/file rule_to_ignore.
 
 
 Gitea
   New: Introduce chezmoi.
 Chezmoi stores the desired state of your dotfiles in the directory ~/.local/share/chezmoi. When you run chezmoi apply, chezmoi calculates the desired contents for each of your dotfiles and then makes the minimum changes required to make your dotfiles match your desired state.
 What I like:
  Supports pass to retrieve credentials.
 Popular
 Can remove directories on apply
 It has a diff
 It can include dotfiles from an URL
 Encrypt files with gpg
 There's a vim plugin
 Actively maintained
 Good documentation
 
 What I don't like:
  Go templates, although it supports autotemplating and it's well explained
 Written in Go
 
 In the article you can also find:
  How to install it
 How to use it
 How to install a binary from an external url
 
 
  Correction: Update the project url of helm-secrets.
 From https://github.com/futuresimple/helm-secrets to https://github.com/jkroepke/helm-secrets
 
  New: Disable the regular login, use only Oauth.
 You need to add a file inside your custom directory. The file is too big to add in this digest, please access the article to get it.
 
  New: Configure it with terraform.
 Gitea can be configured through terraform too. There is an official provider that doesn't work, there's a fork that does though. Sadly it doesn't yet support configuring Oauth Authentication sources. Be careful gitea_oauth2_app looks to be the right resource to do that, but instead it configures Gitea to be the Oauth provider, not a consumer.
 In the article you can find how to configure and use it to:
  Create an organization
 
 
  New: Create an admin user through the command line.
 gitea --config /etc/gitea/app.ini admin user create --admin --username user_name --password password --email email
 Or you can change the admin's password:
 
gitea --config /etc/gitea/app.ini admin user change-password -u username -p password
 feat(gtd): Introduce Getting things done
 First summary of David Allen's book Getting things done. It includes:
  Theory principles
 Managing commitments
 Managing stuff
 Managing actions
  Workflow steps
  Capture what has our attention.
 Clarify what each item means and what to do about it.
 Organize the results.
 Reflect on the options.
 Engage the chosen options.
 
 
  Setting up the system
 
 Setting up the space
 Setting up your filing system
 
 
  New: Gitea actions overview.
 We've been using Drone as CI runner for some years now as Gitea didn't have their native runner. On Mar 20, 2023 however Gitea released the version 1.19.0 which promoted to stable the Gitea Actions which is a built-in CI system like GitHub Actions. With Gitea Actions, you can reuse your familiar workflows and Github Actions in your self-hosted Gitea instance. While it is not currently fully compatible with GitHub Actions, they intend to become as compatible as possible in future versions. The typical procedure is as follows:
  Register a runner (at the moment, act runners are the only option). This can be done on the following scopes:
 site-wide (by site admins)
 organization-wide (by organization owners)
 repository-wide (by repository owners)
 Create workflow files under .gitea/workflows/<workflow name>.yaml or .github/workflows/<workflow name>.yaml. The syntax is the same as the GitHub workflow syntax where supported.
 
 Gitea Actions advantages are:
  Uses the same pipeline syntax as Github Actions, so it's easier to use for new developers
 You can reuse existent Github actions.
 Migration from Github repositories to Gitea is easier.
 You see the results of the workflows in the same gitea webpage, which is much cleaner than needing to go to drone
 Define the secrets in the repository configuration.
 
 Drone advantages are:
  They have the promote event. Not critical as we can use other git events such as creating a tag.
 They can be run as a service by default. The gitea runners will need some work to run on instance restart.
 Has support for running kubernetes pipelines. Gitea actions doesn't yet support this
 
 
  New: Setup Gitea actions.
 You need a Gitea instance with a version of 1.19.0 or higher. Actions are disabled by default (as they are still an feature-in-progress), so you need to add the following to the configuration file to enable it:
 [actions]
ENABLED=true
 Even if you enable at configuration level you need to manually enable the actions on each repository until this issue is solved.
 So far there is only one possible runner which is based on docker and act. Currently, the only way to install act runner is by compiling it yourself, or by using one of the pre-built binaries. There is no Docker image or other type of package management yet. At the moment, act runner should be run from the command line. Of course, you can also wrap this binary in something like a system service, supervisord, or Docker container.
 Before running a runner, you should first register it to your Gitea instance using the following command:
 ./act_runner register --no-interactive --instance <instance> --token <token>
 There are two arguments required, instance and token.
 instance refers to the address of your Gitea instance, like http://192.168.8.8:3000. The runner and job containers (which are started by the runner to execute jobs) will connect to this address. This means that it could be different from the ROOT_URL of your Gitea instance, which is configured for web access. It is always a bad idea to use a loopback address such as 127.0.0.1 or localhost, as we will discuss later. If you are unsure which address to use, the LAN address is usually the right choice.
 token is used for authentication and identification, such as P2U1U0oB4XaRCi8azcngmPCLbRpUGapalhmddh23. It is one-time use only and cannot be used to register multiple runners. You can obtain tokens from your_gitea.com/admin/runners.
 After registering, a new file named .runner will appear in the current directory. This file stores the registration information. Please do not edit it manually. If this file is missing or corrupted, you can simply remove it and register again.
 Finally, it’s time to start the runner.
 ./act_runner daemon
 
  New: Use the gitea actions.
 Even if Actions is enabled for the Gitea instance, repositories still disable Actions by default. Enable it on the settings page of your repository.
 You will need to study the workflow syntax for Actions and write the workflow files you want.
 However, we can just start from a simple demo:
 name: Gitea Actions Demo
run-name: ${{ gitea.actor }} is testing out Gitea Actions
on: [push]
jobs:
  Explore-Gitea-Actions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "The job was automatically triggered by a ${{ gitea.event_name }} event."
      - run: echo "This job is now running on a ${{ runner.os }} server hosted by Gitea!"
      - run: echo "The name of your branch is ${{ gitea.ref }} and your repository is ${{ gitea.repository }}."
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: echo "The ${{ gitea.repository }} repository has been cloned to the runner."
      - run: echo "The workflow is now ready to test your code on the runner."
      - name: List files in the repository
        run: |
          ls ${{ gitea.workspace }}
      - run: echo "This job's status is ${{ gitea.status }}."
 You can upload it as a file with the extension .yaml in the directory .gitea/workflows/ or .github/workflows of the repository, for example .gitea/workflows/demo.yaml.
 You may be aware that there are tens of thousands of marketplace actions in GitHub. However, when you write uses: actions/checkout@v3, it actually downloads the scripts from gitea.com/actions/checkout by default (not GitHub). This is a mirror of github.com/actions/checkout, but it’s impossible to mirror all of them. That’s why you may encounter failures when trying to use some actions that haven’t been mirrored.
 The good news is that you can specify the URL prefix to use actions from anywhere. This is an extra syntax in Gitea Actions. For example:
  uses: https://github.com/xxx/xxx@xxx
 uses: https://gitea.com/xxx/xxx@xxx
 uses: http://your_gitea_instance.com/xxx@xxx
 
 Be careful, the https:// or http:// prefix is necessary!
 
  New: Import organisations into terraform.
 To import organisations and teams you need to use their ID. You can see the ID of the organisations in the Administration panel. To get the Teams ID you need to use the API. Go to https://your.gitea.com/api/swagger#/organization/orgListTeams and enter the organisation name.
 
  Correction: Give some tip to deal with big diffs.
 Sometimes the diff is too big and you need to work with it chuck by chunk. For each change you can either:
  chezmoi add <target> if you want to keep the changes you've manually made to the files that match the <target>.
 chezmoi apply <target> if you want to apply the changes that chezmoi proposes for the <target>.
 
 Here <target> is any directory or file listed in the diff.
 
  New: Add systemd service for the actions runner.
 [Unit]
Description=Gitea Actions Runner
After=network.target

[Service]
WorkingDirectory=/var/gitea/gitea/act_runner/main
ExecStart=/var/gitea/gitea/act_runner/main/act_runner-main-linux-amd64 daemon

[Install]
WantedBy=default.target
 
  New: Tweak the runner image.
 The gitea runner uses the node:16-bullseye image by default, in that image the setup-python action doesn't work. You can tweak the docker image that the runner runs by editing the .runner file that is in the directory where you registered the runner (probably close to the act_runner executable).
 If you open that up, you’ll see that there is a section called labels, and it (most likely) looks like this:
 "labels": [
  "ubuntu-latest:docker://node:16-bullseye",
  "ubuntu-22.04:docker://node:16-bullseye",
  "ubuntu-20.04:docker://node:16-bullseye",
  "ubuntu-18.04:docker://node:16-buster"
]
 You can specify any other docker image. Adding new labels doesn't work yet.
 
  New: Introduce molecule.
 Molecule is a testing tool for ansible roles.
  Source
 Docs
 
 
  New: CI configuration.
 Since gitea supports github actions you can use the setup-molecule and setup-lint actions. For example:
 ---
name: Molecule

"on":
  pull_request:

env:
  PY_COLORS: "1"
  ANSIBLE_FORCE_COLOR: "1"
jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the codebase
        uses: actions/checkout@v3

      - name: Setup Lint
        uses: bec-galaxy/setup-lint@{Version}

      - name: Run Lint tests
        run: ansible-lint

  molecule:
    name: Molecule
    runs-on: ubuntu-latest
    needs: lint
    steps:
      - name: Checkout the codebase
        uses: actions/checkout@v3

      - name: Setup Molecule
        uses: bec-galaxy/setup-molecule@{Version}

      - name: Run Molecule tests
        run: molecule test
 That action installs the latest version of the packages, if you need to check a specific version of the packages you may want to create your own step or your own action.
 
  New: Upgrade to v5.0.0.
 They've removed the lint command, the reason behind is that there are two different testing methods which are expected to be run in very different ways. Linting should be run per entire repository. Molecule executions are per scenario and one project can have even >100 scenarios. Running lint on each of them would not only slowdown but also increase the maintenance burden on linter configuration and the way is called.
 They recommend users to run ansible-lint using pre-commit with or without `tox. That gives much better control over how/when it is updated.
 You can see an example on how to do this in the CI configuration section.
 
  Correction: Configure the gitea actions.
 So far there is only one possible runner which is based on docker and act. Currently, the only way to install act runner is by compiling it yourself, or by using one of the pre-built binaries. There is no Docker image or other type of package management yet. At the moment, act runner should be run from the command line. Of course, you can also wrap this binary in something like a system service, supervisord, or Docker container.
 You can create the default configuration of the runner with:
 ./act_runner generate-config > config.yaml
 You can tweak there for example the capacity so you are able to run more than one workflow in parallel.
 Before running a runner, you should first register it to your Gitea instance using the following command:
 ./act_runner register --config config.yaml --no-interactive --instance <instance> --token <token>
 Finally, it’s time to start the runner.
 ./act_runner --config config.yaml daemon
 If you want to create your own act docker, you can start with this dockerfile:
 FROM node:16-bullseye

LABEL prune=false

RUN mkdir /root/.aws
COPY files/config /root/.aws/config
COPY files/credentials /root/.aws/credentials

RUN apt-get update && apt-get install -y \
  python3 \
  python3-pip \
  python3-venv \
  screen \
  vim \
  && python3 -m pip install --upgrade pip \
  && rm -rf /var/lib/apt/lists/*

RUN pip install \
  molecule==5.0.1 \
  ansible==8.0.0 \
  ansible-lint \
  yamllint \
  molecule-plugins[ec2,docker,vagrant] \
  boto3 \
  botocore \
  testinfra \
  pytest

RUN wget https://download.docker.com/linux/static/stable/x86_64/docker-24.0.2.tgz \
  && tar xvzf docker-24.0.2.tgz \
  && cp docker/* /usr/bin \
  && rm -r docker docker-*
 It's prepared for:
  Working within an AWS environment
 Run Ansible and molecule
 Build dockers
 
 
  New: Build a docker within a gitea action.
 Assuming you're using the custom gitea_runner docker proposed above you can build and upload a docker to a registry with this action:
 ---
name: Publish Docker image

"on": [push]

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: https://github.com/actions/checkout@v3

      - name: Login to Docker Registry
        uses: https://github.com/docker/login-action@v2
        with:
          registry: my_registry.org
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}

      - name: Set up QEMU
        uses: https://github.com/docker/setup-qemu-action@v2

      - name: Set up Docker Buildx
        uses: https://github.com/docker/setup-buildx-action@v2

      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: https://github.com/docker/metadata-action@v4
        with:
          images: my_registry.org/the_name_of_the_docker_to_build

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          cache-from: type=registry,ref=my_registry.org/the_name_of_the_docker_to_build:buildcache
          cache-to: type=registry,ref=my_registry.org/the_name_of_the_docker_to_build:buildcache,mode=max
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
 It uses a pair of nice features:
  Multi-arch builds
 Cache to speed up the builds
 
 As it reacts to all events it will build and push:
  A tag with the branch name on each push to that branch
 A tag with the tag on tag push
 
 
  New: Bump the version of a repository on commits on master.
  Create a SSH key for the CI to send commits to protected branches.
 Upload the private key to a repo or organization secret called DEPLOY_SSH_KEY.
 Upload the public key to the repo configuration deploy keys
  Create the bump.yaml file with the next contents:
 ---
name: Bump version

"on":
  push:
    branches:
      - main

jobs:
  bump_version:
    if: "!startsWith(github.event.head_commit.message, 'bump:')"
    runs-on: ubuntu-latest
    name: "Bump version and create changelog"
    steps:
      - name: Check out
        uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Fetch all history

      - name: Configure SSH
        run: |
            echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
            chmod 600 ~/.ssh/deploy_key
            dos2unix ~/.ssh/deploy_key
            ssh-agent -a $SSH_AUTH_SOCK > /dev/null
            ssh-add ~/.ssh/deploy_key

      - name: Bump the version
        run: cz bump --changelog --no-verify

      - name: Push changes
        run: |
          git remote add ssh git@gitea-production.cloud.icij.org:templates/ansible-role.git
          git pull ssh main
          git push ssh main
          git push ssh --tags
 It assumes that you have cz (commitizen) and dos2unix installed in your runner.
 
 
 
  New: Skip gitea actions job on changes of some files.
 There are some expensive CI pipelines that don't need to be run for example if you changed a line in the README.md, to skip a pipeline on changes of certain files you can use the paths-ignore directive:
 ---
name: Ansible Testing

"on":
  push:
    paths-ignore:
      - 'meta/**'
      - Makefile
      - README.md
      - renovate.json
      - CHANGELOG.md
      - .cz.toml
      - '.gitea/workflows/**'

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
        ...
 The only downside is that if you set this pipeline as required in the branch protection, the merge button will look yellow instead of green when the pipeline is skipped.
 
  New: Molecule doesn't find the molecule.yaml file.
 This is expected default behavior since Molecule searches for scenarios using the molecule/*/molecule.yml glob. But if you would like to change the suffix to yaml, you can do that if you set the MOLECULE_GLOB environment variable like this:
 export MOLECULE_GLOB='molecule/*/molecule.yaml'
 
  Correction: Using paths-filter custom action to skip job actions.
 jobs:
  test:
    if: "!startsWith(github.event.head_commit.message, 'bump:')"
    name: Test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the codebase
        uses: https://github.com/actions/checkout@v3

      - name: Check if we need to run the molecule tests
        uses: https://github.com/dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            molecule:
              - 'defaults/**'
              - 'tasks/**'
              - 'handlers/**'
              - 'tasks/**'
              - 'templates/**'
              - 'molecule/**'
              - 'requirements.yaml'
              - '.github/workflows/tests.yaml'

      - name: Run Molecule tests
        if: steps.filter.outputs.molecule == 'true'
        run: make molecule
 You can find more examples on how to use paths-filter here.
 
  New: Get variables from the environment.
 You can configure your molecule.yaml file to read variables from the environment with:
 provisioner:
  name: ansible
  inventory:
    group_vars:
      all:
        my_secret: ${MY_SECRET}
 It's useful to have a task that checks if this secret exists:
 - name: Verify that the secret is set
  fail:
    msg: 'Please export my_secret: export MY_SECRET=$(pass show my_secret)'
  run_once: true
  when: my_secret == None
 In the CI you can set it as a secret in the repository.
 
  New: Run jobs if other jobs failed.
 This is useful to send notifications if any of the jobs failed.
 Right now you can't run a job if other jobs fail, all you can do is add a last step on each workflow to do the notification on failure:
 - name: Send mail
    if: failure()
    uses: https://github.com/dawidd6/action-send-mail@v3
    with:
        to: ${{ secrets.MAIL_TO }}
        from: Gitea <gitea@hostname>
        subject: ${{ gitea.repository }} ${{gitea.workflow}} ${{ job.status }}
        priority: high
        convert_markdown: true
        html_body: |
            ### Job ${{ job.status }}

            ${{ github.repository }}: [${{ github.ref }}@${{ github.sha }}](${{ github.server_url }}/${{ github.repository }}/actions)
 
 
 Helmfile
   New: Troubleshoot Yaml templates in go templates.
 If you are using a values.yaml.gotmpl file you won't be able to use {{ whatever }}. The solution is to extract that part to a yaml file and include it in the go template. For example:
  values.yaml.gotmpl:
 
 metrics:
serviceMonitor:
  enabled: true
  annotations:
  additionalLabels:
    release: prometheus-operator

{{ readFile "prometheus_rules.yaml" }}
  prometheus_rules.yaml
 
 prometheusRule:
  enabled: true
  additionalLabels:
    release: prometheus-operator
  spec:
    - alert: VeleroBackupPartialFailures
      annotations:
        message: Velero backup {{ $labels.schedule }} has {{ $value | humanizePercentage }} partialy failed backups.
      expr: increase(velero_backup_partial_failure_total{schedule!=""}[1h]) > 0
      for: 15m
      labels:
        severity: warning
 
  New: Introduce dotdrop.
 The main idea of Dotdropis to have the ability to store each dotfile only once and deploy them with a different content on different hosts/setups. To achieve this, it uses a templating engine that allows to specify, during the dotfile installation with dotdrop, based on a selected profile, how (with what content) each dotfile will be installed.
 What I like:
  Popular
 Actively maintained
 Written in Python
 Uses jinja2
 Has a nice to read config file
 
 What I don't like:
  Updating dotfiles doesn't look as smooth as with chezmoi
 Uses {{@@ @@}} instead of {{ }} :S
 Doesn't support pass.
 Not easy way to edit the files.
 
 
 
 Terraform
   New: How to install it.
 Go to the releases page, download the latest release, decompress it and add it to your $PATH.
 
  New: How to store sensitive information in terraform.
 One of the most common questions we get about using Terraform to manage infrastructure as code is how to handle secrets such as passwords, API keys, and other sensitive data.
 In the article you'll find how to store your sensitive data in:
  The Terraform state: Using the state backend encryption
 The Terraform source code: Usingsops and gpg.
 
 
  New: Create a list of resources based on a list of strings.
 variable "subnet_ids" {
  type = list(string)
}

resource "aws_instance" "server" {
  # Create one instance for each subnet
  count = length(var.subnet_ids)

  ami           = "ami-a1b2c3d4"
  instance_type = "t2.micro"
  subnet_id     = var.subnet_ids[count.index]

  tags = {
    Name = "Server ${count.index}"
  }
}
 If you want to use this generated list on another resource extracting for example the id you can use
 aws_instance.server.*.id
 
 
 Dotfiles
   New: Introduce dotfiles.
 User-specific application configuration is traditionally stored in so called dotfiles (files whose filename starts with a dot). It is common practice to track dotfiles with a version control system such as Git to keep track of changes and synchronize dotfiles across various hosts. There are various approaches to managing your dotfiles (e.g. directly tracking dotfiles in the home directory v.s. storing them in a subdirectory and symlinking/copying/generating files with a shell script or a dedicated tool).
 Note: this is not meant to configure files that are outside your home directory, use Ansible for that use case.
 You can find different ways to track your dotfiles:
  Tracking dotfiles directly with Git
 Using Ansible to manage the dotfiles
 Using dotfiles specific tools, and an analysis of the state of the art
 
 
 
 Infrastructure Solutions
 Velero
   New: Introduce velero.
 Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.
 In the article you'll also find how to:
  Install the client
 Configure the server
 Monitor it with Prometheus
 Restore backups
 
 And an overview of the whole application infrastructure.
 
  New: Create a backup.
 If you already have schedules select the one you want to use:
 velero schedules get
 Then create the backup with:
 velero backup create --from-schedule selected-schedule
 You can see the other options to create backups in velero backup create --help
 
 
 AWS Snippets
   Correction: Recommend to use distro repos when installing.
 It's available now in debian
 
  New: Stop an EC2 instance.
 aws ec2 stop-instances --instance-ids i-xxxxxxxx
 
  New: Get EC2 metadata from within the instance.
 The quickest way to fetch or retrieve EC2 instance metadata from within a running EC2 instance is to log in and run the command:
 Fetch metadata from IPv4:
 curl -s http://169.254.169.254/latest/dynamic/instance-identity/document
 You can also download the ec2-metadata tool to get the info:
 wget http://s3.amazonaws.com/ec2metadata/ec2-metadata

chmod +x ec2-metadata

./ec2-metadata --all
 
  New: [Remove the lock screen in ubuntu.](../aws_snippets.md#invalidate-a-cloudfront-distribution
 
 
 aws-cloudfront-create-invalidation---paths-"/pages/about"---distribution-id-my-distribution-id
 feat(bash_snippets)
 Create the `/usr/share/glib-2.0/schemas/90_ubuntu-settings.gschema.override` file with the next content:

```ini
[org.gnome.desktop.screensaver]
lock-enabled = false
[org.gnome.settings-daemon.plugins.power]
idle-dim = false
```

Then reload the schemas with:

```bash
sudo glib-compile-schemas /usr/share/glib-2.0/schemas/
```
 Kubectl Commands
   New: Show the remaining space of a persistent volume claim.
 Either look it in Prometheus or run in the pod that has the PVC mounted:
 kubectl -n <namespace> exec <pod-name> -- df -ah
 You may need to use kubectl get pod <pod-name> -o yaml to know what volume is mounted where.
 
  New: Run a pod in a defined node.
 Get the node hostnames with kubectl get nodes, then override the node with:
 kubectl run mypod --image ubuntu:18.04 --overrides='{"apiVersion": "v1", "spec": {"nodeSelector": { "kubernetes.io/hostname": "my-node.internal" }}}' --command -- sleep 100000000000000
 
 
 Tools
   Correction: Recommend rke2 over k3s.
 A friend told me that it works better.
 
 
 Continuous Integration
 Mypy
   New: Module "typing" has no attribute "Annotated".
 This one happens only because annotated is not available in python < 3.9.
 try:
    # mypy is complaining that it can't import it, but it's solved below
    from typing import Annotated # type: ignore
except ImportError:
    from typing_extensions import Annotated
 
 
 Drone
   New: Create the administrators.
 When you configure the Drone server you can create the initial administrative account by passing the below environment variable, which defines the account username (e.g. github handle) and admin flag set to true.
 DRONE_USER_CREATE=username:octocat,admin:true
 If you need to grant the primary administrative role to an existing user, you can provide an existing username. Drone will update the account and grant administrator role on server restart.
 You can create administrator accounts using the command line tools. Please see the command line tools documentation for installation instructions.
 Create a new administrator account:
 $ drone user add octocat --admin
 Or grant the administrator role to existing accounts:
 $ drone user update octocat --admin
 
  New: Linter: untrusted repositories cannot mount host volumes.
 Thats because the repository is not trusted.
 You have to set the trust as an admin of drone through the GUI or through the CLI with
 drone repo update --trusted <your/repo>
 If you're not an admin the above command returns a success but you'll see that the trust has not changed if you run
 drone repo info <your/repo>
 
 
 ArgoCD
   New: Introduce ArgoCD.
 Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
 Argo CD follows the GitOps pattern of using Git repositories as the source of truth for defining the desired application state. Kubernetes manifests can be specified in several ways:
  kustomize applications
 helm charts
 jsonnet files
 Plain directory of YAML/json manifests
 Any custom config management tool configured as a config management plugin, for example with helmfile
 
 Argo CD automates the deployment of the desired application states in the specified target environments. Application deployments can track updates to branches, tags, or pinned to a specific version of manifests at a Git commit. See tracking strategies for additional details about the different tracking strategies available.
 
 
 Shellcheck
   New: Introduce shellcheck.
 Shellcheck is a linting tool to finds bugs in your shell scripts.
 Installation
 apt-get install shellcheck
 Errors
 SC2143: Use grep -q instead of comparing output with [ -n .. ].
 Problematic code:
 if [ "$(find . | grep 'IMG[0-9]')" ]
then
  echo "Images found"
fi
 Correct code:
 if find . | grep -q 'IMG[0-9]'
then
  echo "Images found"
fi
 Rationale:
 The problematic code has to iterate the entire directory and read all matching lines into memory before making a decision.
 The correct code is cleaner and stops at the first matching line, avoiding both iterating the rest of the directory and reading data into memory.
 
 
 Automating Processes
 copier
   New: Introduce copier.
 Copier is a library and CLI app for rendering project templates.
  Works with local paths and Git URLs.
 Your project can include any file and Copier can dynamically replace values in any kind of text file.
 It generates a beautiful output and takes care of not overwriting existing files unless instructed to do so.
 
 This long article covers:
  Installation
 Basic concepts
 Usage
 
 
 
 cruft
   Correction: Suggest to use copier instead.
 copier looks a more maintained solution nowadays.
 
 
 letsencrypt
   New: Introduce letsencrypt.
 Letsencrypt is a free, automated, and open certificate authority brought to you by the nonprofit Internet Security Research Group (ISRG). Basically it gives away SSL certificates, which are required to configure webservers to use HTTPS instead of HTTP for example.
 In the article you can also find:
  How to configure a wildcard dns when the provider is not supported
 
 
 
 Storage
 OpenZFS storage planning
  New: Introduce ZFS storage planning.
  New: Analyze the Exos X18 of 16TB disk.
    Specs  IronWolf  IronWolf Pro  Exos 7E8 8TB  Exos 7E10 8TB  Exos X18 16TB  
 
   Technology  CMR  CMR  CMR  SMR  CMR  
  Bays  1-8  1-24  ?  ?  ?  
  Capacity  1-12TB  2-20TB  8TB  8TB  16 TB  
  RPM  5,400 RPM (3-6TB)  7200 RPM  7200 RPM  7200 RPM  7200 RPM  
  RPM  5,900 RPM (1-3TB)  7200 RPM  7200 RPM  7200 RPM  7200 RPM  
  RPM  7,200 RPM (8-12TB)  7200 RPM  7200 RPM  7200 RPM  7200 RPM  
  Speed  180MB/s (1-12TB)  214-260MB/s (4-18TB)  249 MB/s  255 MB/s  258 MB/s  
  Cache  64MB (1-4TB)  256 MB  256 MB  256 MB  256 MB  
  Cache  256MB (3-12TB)  256 MB  256 MB  256 MB  256 MB  
  Power Consumption  10.1 W  10.1 W  12.81 W  11.03 W  9.31 W  
  Power Consumption Rest  7.8 W  7.8 W  7.64 W  7.06 W  5.08 W  
  Workload  180TB/yr  300TB/yr  550TB/yr  550TB/yr  550TB/yr  
  MTBF  1 million  1 million  2 millions  2 millions  2.5 millions  
  Warranty  3 years  5 years  5 years  5 years  5 years  
  Price  From $60 (2022)  From $83 (2022)  249$ (2022)  249$ (2022)  249$ (2023)  
 
 
 
 
 OpenZFS
  New: How to create a pool and datasets.
  New: Configure NFS.
 With ZFS you can share a specific dataset via NFS. If for whatever reason the dataset does not mount, then the export will not be available to the application, and the NFS client will be blocked.
 You still must install the necessary daemon software to make the share available. For example, if you wish to share a dataset via NFS, then you need to install the NFS server software, and it must be running. Then, all you need to do is flip the sharing NFS switch on the dataset, and it will be immediately available.
 
  New: Backup.
 Please remember that RAID is not a backup, it guards against one kind of hardware failure. There's lots of failure modes that it doesn't guard against though:
  File corruption
 Human error (deleting files by mistake)
 Catastrophic damage (someone dumps water onto the server)
 Viruses and other malware
 Software bugs that wipe out data
 Hardware problems that wipe out data or cause hardware damage (controller malfunctions, firmware bugs, voltage spikes, ...)
 
 That's why you still need to make backups.
 ZFS has the builtin feature to make snapshots of the pool. A snapshot is a first class read-only filesystem. It is a mirrored copy of the state of the filesystem at the time you took the snapshot. They are persistent across reboots, and they don't require any additional backing store; they use the same storage pool as the rest of your data.
 If you remember ZFS's awesome nature of copy-on-write filesystems, you will remember the discussion about Merkle trees. A ZFS snapshot is a copy of the Merkle tree in that state, except we make sure that the snapshot of that Merkle tree is never modified.
 Creating snapshots is near instantaneous, and they are cheap. However, once the data begins to change, the snapshot will begin storing data. If you have multiple snapshots, then multiple deltas will be tracked across all the snapshots. However, depending on your needs, snapshots can still be exceptionally cheap.
 The article also includes:
  ZFS snapshot lifecycle management
 Restore a backup
 
 
  New: Introduce Sanoid.
 Sanoid is the most popular tool right now, with it you can create, automatically thin, and monitor snapshots and pool health from a single eminently human-readable TOML config file.
 The article includes:
  Installation
 Configuration
 Pros and cons
 Usage
 Troubleshooting
 
 
  New: Get compress ratio of a filesystem.
 zfs get compressratio {{ filesystem }}
 
  Correction: Use the recursive flag.
 recursive is not set by default, so the dataset's children won't be backed up unless you set this option.
  [main/backup]
   use_template = daily
   recursive = yes
 
  New: See how much space do your snapshots consume.
 When a snapshot is created, its space is initially shared between the snapshot and the file system, and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot, and thus is counted in the snapshot’s used property.
 Additionally, deleting snapshots can increase the amount of space that is unique for use by other snapshots.
 Note: The value for a snapshot’s space referenced property is the same as that for the file system when the snapshot was created.
 You can display the amount of space that is consumed by snapshots and descendant file systems by using the zfs list -o space command.
 NAME                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool                            10.2G  5.16G         0   4.52M              0      5.15G
rpool/ROOT                       10.2G  3.06G         0     31K              0      3.06G
rpool/ROOT/solaris               10.2G  3.06G     55.0M   2.78G              0       224M
rpool/ROOT/solaris@install           -  55.0M         -       -              -          -
rpool/ROOT/solaris/var           10.2G   224M     2.51M    221M              0          0
rpool/ROOT/solaris/var@install       -  2.51M         -       -              -          -
 From this output, you can see the amount of space that is:
  AVAIL: The amount of space available to the dataset and all its children, assuming that there is no other activity in the pool.
  USED: The amount of space consumed by this dataset and all its descendants. This is the value that is checked against this dataset's quota and reservation. The space used does not include this dataset's reservation, but does take into account the reservations of any descendants datasets.
 The used space of a snapshot is the space referenced exclusively by this snapshot. If this snapshot is destroyed, the amount of used space will be freed. Space that is shared by multiple snapshots isn't accounted for in this metric. * USEDSNAP: Space being consumed by snapshots of each data set * USEDDS: Space being used by the dataset itself * USEDREFRESERV: Space being used by a refreservation set on the dataset that would be freed if it was removed. * USEDCHILD: Space being used by the children of this dataset.
 
 
 Other space properties are:
  LUSED: The amount of space that is "logically" consumed by this dataset and all its descendents. It ignores the effect of compression and copies properties, giving a quantity closer to the amount of data that aplication ssee. However it does include space consumed by metadata.
 REFER: The amount of data that is accessible by this dataset, which may or may not be shared with other dataserts in the pool. When a snapshot or clone is created, it initially references the same amount of space as the filesystem or snapshot it was created from, since its contents are identical.
 
 
  New: Rename or move a dataset.
 NOTE: if you want to rename the topmost dataset look at rename the topmost dataset instead. File systems can be renamed by using the zfs rename command. You can perform the following operations:
  Change the name of a file system.
 Relocate the file system within the ZFS hierarchy.
 Change the name of a file system and relocate it within the ZFS hierarchy.
 
 The following example uses the rename subcommand to rename of a file system from kustarz to kustarz_old:
 zfs rename tank/home/kustarz tank/home/kustarz_old
 The following example shows how to use zfs rename to relocate a file system:
 zfs rename tank/home/maybee tank/ws/maybee
 In this example, the maybee file system is relocated from tank/home to tank/ws. When you relocate a file system through rename, the new location must be within the same pool and it must have enough disk space to hold this new file system. If the new location does not have enough disk space, possibly because it has reached its quota, rename operation fails.
 The rename operation attempts an unmount/remount sequence for the file system and any descendent file systems. The rename command fails if the operation is unable to unmount an active file system. If this problem occurs, you must forcibly unmount the file system.
 You'll loose the snapshots though, as explained below.
 
  New: Rename the topmost dataset.
 If you want to rename the topmost dataset you need to rename the pool too as these two are tied.
 $: zpool status -v

  pool: tets
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tets        ONLINE       0     0     0
          c0d1      ONLINE       0     0     0
          c1d0      ONLINE       0     0     0
          c1d1      ONLINE       0     0     0

errors: No known data errors
 To fix this, first export the pool:
 $ zpool export tets
 And then imported it with the correct name:
 $ zpool import tets test
 After the import completed, the pool contains the correct name:
 $ zpool status -v

  pool: test
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        test        ONLINE       0     0     0
          c0d1      ONLINE       0     0     0
          c1d0      ONLINE       0     0     0
          c1d1      ONLINE       0     0     0

errors: No known data errors
 Now you may need to fix the ZFS mountpoints for each dataset
 zfs set mountpoint="/opt/zones/[Newmountpoint]" [ZFSPOOL/[ROOTor other filesystem]
 
  New: Rename or move snapshots.
 If the dataset has snapshots you need to rename them too. They must be renamed within the same pool and dataset from which they were created though. For example:
 zfs rename tank/home/cindys@083006 tank/home/cindys@today
 In addition, the following shortcut syntax is equivalent to the preceding syntax:
 zfs rename tank/home/cindys@083006 today
 The following snapshot rename operation is not supported because the target pool and file system name are different from the pool and file system where the snapshot was created:
 $: zfs rename tank/home/cindys@today pool/home/cindys@saturday
cannot rename to 'pool/home/cindys@today': snapshots must be part of same
dataset
 You can recursively rename snapshots by using the zfs rename -r command. For example:
 $: zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
users                        270K  16.5G    22K  /users
users/home                    76K  16.5G    22K  /users/home
users/home@yesterday            0      -    22K  -
users/home/markm              18K  16.5G    18K  /users/home/markm
users/home/markm@yesterday      0      -    18K  -
users/home/marks              18K  16.5G    18K  /users/home/marks
users/home/marks@yesterday      0      -    18K  -
users/home/neil               18K  16.5G    18K  /users/home/neil
users/home/neil@yesterday       0      -    18K  -
$: zfs rename -r users/home@yesterday @2daysago
$: zfs list -r users/home
NAME                        USED  AVAIL  REFER  MOUNTPOINT
users/home                   76K  16.5G    22K  /users/home
users/home@2daysago            0      -    22K  -
users/home/markm             18K  16.5G    18K  /users/home/markm
users/home/markm@2daysago      0      -    18K  -
users/home/marks             18K  16.5G    18K  /users/home/marks
users/home/marks@2daysago      0      -    18K  -
users/home/neil              18K  16.5G    18K  /users/home/neil
users/home/neil@2daysago       0      -    18K  -
 
  New: See the differences between two backups.
 To identify the differences between two snapshots, use syntax similar to the following:
 $ zfs diff tank/home/tim@snap1 tank/home/tim@snap2
M       /tank/home/tim/
+       /tank/home/tim/fileB
 The following table summarizes the file or directory changes that are identified by the zfs diff command.
    File or Directory Change  Identifier  
 
   File or directory has been modified or file or directory link has changed  M  
  File or directory is present in the older snapshot but not in the more recent snapshot  —  
  File or directory is present in the more recent snapshot but not in the older snapshot  +  
  File or directory has been renamed  R  
 
 
 
  New: Create a cold backup of a series of datasets.
 If you've used the -o keyformat=raw -o keylocation=file:///etc/zfs/keys/home.key arguments to encrypt your datasets you can't use a keyformat=passphase encryption on the cold storage device. You need to copy those keys on the disk. One way of doing it is to:
  Create a 100M LUKS partition protected with a passphrase where you store the keys.
 The rest of the space is left for a partition for the zpool.
 
 
  New: Clear a permanent ZFS error in a healthy pool.
 Sometimes when you do a zpool status you may see that the pool is healthy but that there are "Permanent errors" that may point to files themselves or directly to memory locations.
 You can read this long discussion on what does these permanent errors mean, but what solved the issue for me was to run a new scrub
 zpool scrub my_pool
 It takes a long time to run, so be patient.
 
  New: ZFS pool is in suspended mode.
 Probably because you've unplugged a device without unmounting it.
 If you want to remount the device you can follow these steps to symlink the new devfs entries to where zfs thinks the vdev is. That way you can regain access to the pool without a reboot.
 So if zpool status says the vdev is /dev/disk2s1, but the reattached drive is at disk4, then do the following:
 cd /dev
sudo rm -f disk2s1
sudo ln -s disk4s1 disk2s1
sudo zpool clear -F WD_1TB
sudo zpool export WD_1TB
sudo rm disk2s1
sudo zpool import WD_1TB
 If you don't care about the zpool anymore, sadly your only solution is to reboot the server. Real ugly, so be careful when you umount zpools.
 
  New: Prune snapshots.
 If you want to manually prune the snapshots after you tweaked sanoid.conf you can run:
 sanoid --prune-snapshots
 
  New: Send encrypted backups to a encrypted dataset.
 syncoid's default behaviour is to create the destination dataset without encryption so the snapshots are transferred and can be read without encryption. You can check this with the zfs get encryption,keylocation,keyformat command both on source and destination.
 To prevent this from happening you have to [pass the --sendoptions='w'](https://github.com/jimsalterjrs/sanoid/issues/548) tosyncoidso that it tells zfs to send a raw stream. If you do so, you also need to [transfer the key file](https://github.com/jimsalterjrs/sanoid/issues/648) to the destination server so that it can do azfs loadkey` and then mount the dataset. For example:
 server-host:$ sudo zfs list -t filesystem
NAME                    USED  AVAIL     REFER  MOUNTPOINT
server_data             232M  38.1G      230M  /var/server_data
server_data/log         111K  38.1G      111K  /var/server_data/log
server_data/mail        111K  38.1G      111K  /var/server_data/mail
server_data/nextcloud   111K  38.1G      111K  /var/server_data/nextcloud
server_data/postgres    111K  38.1G      111K  /var/server_data/postgres

server-host:$ sudo zfs get keylocation server_data/nextcloud
NAME                   PROPERTY     VALUE                                    SOURCE
server_data/nextcloud  keylocation  file:///root/zfs_dataset_nextcloud_pass  local

server-host:$ sudo syncoid --recursive --skip-parent --sendoptions=w server_data root@192.168.122.94:backup_pool
INFO: Sending oldest full snapshot server_data/log@autosnap_2021-06-18_18:33:42_yearly (~ 49 KB) to new target filesystem:
17.0KiB 0:00:00 [1.79MiB/s] [=================================================>                                                                                                  ] 34%
INFO: Updating new target filesystem with incremental server_data/log@autosnap_2021-06-18_18:33:42_yearly ... syncoid_caedrium.com_2021-06-22:10:12:55 (~ 15 KB):
41.2KiB 0:00:00 [78.4KiB/s] [===================================================================================================================================================] 270%
INFO: Sending oldest full snapshot server_data/mail@autosnap_2021-06-18_18:33:42_yearly (~ 49 KB) to new target filesystem:
17.0KiB 0:00:00 [ 921KiB/s] [=================================================>                                                                                                  ] 34%
INFO: Updating new target filesystem with incremental server_data/mail@autosnap_2021-06-18_18:33:42_yearly ... syncoid_caedrium.com_2021-06-22:10:13:14 (~ 15 KB):
41.2KiB 0:00:00 [49.4KiB/s] [===================================================================================================================================================] 270%
INFO: Sending oldest full snapshot server_data/nextcloud@autosnap_2021-06-18_18:33:42_yearly (~ 49 KB) to new target filesystem:
17.0KiB 0:00:00 [ 870KiB/s] [=================================================>                                                                                                  ] 34%
INFO: Updating new target filesystem with incremental server_data/nextcloud@autosnap_2021-06-18_18:33:42_yearly ... syncoid_caedrium.com_2021-06-22:10:13:42 (~ 15 KB):
41.2KiB 0:00:00 [50.4KiB/s] [===================================================================================================================================================] 270%
INFO: Sending oldest full snapshot server_data/postgres@autosnap_2021-06-18_18:33:42_yearly (~ 50 KB) to new target filesystem:
17.0KiB 0:00:00 [1.36MiB/s] [===============================================>                                                                                                    ] 33%
INFO: Updating new target filesystem with incremental server_data/postgres@autosnap_2021-06-18_18:33:42_yearly ... syncoid_caedrium.com_2021-06-22:10:14:11 (~ 15 KB):
41.2KiB 0:00:00 [48.9KiB/s] [===================================================================================================================================================] 270%

server-host:$ sudo scp /root/zfs_dataset_nextcloud_pass 192.168.122.94:
 backup-host:$ sudo zfs set keylocation=file:///root/zfs_dataset_nextcloud_pass  backup_pool/nextcloud
backup-host:$ sudo zfs load-key backup_pool/nextcloud
backup-host:$ sudo zfs mount backup_pool/nextcloud
 If you also want to keep the encryptionroot you need to let zfs take care of the recursion instead of syncoid. In this case you can't use syncoid's stuff like --exclude from the manpage of zfs:
 -R, --replicate
   Generate a replication stream package, which will replicate the specified file system, and all descendent file systems, up to the named snapshot.  When received, all properties, snap‐
   shots, descendent file systems, and clones are preserved.

   If the -i or -I flags are used in conjunction with the -R flag, an incremental replication stream is generated.  The current values of properties, and current snapshot and file system
   names are set when the stream is received.  If the -F flag is specified when this stream is received, snapshots and file systems that do not exist on the sending side are destroyed.
   If the -R flag is used to send encrypted datasets, then -w must also be specified.
 In this case this should work:
 /sbin/syncoid --recursive --force-delete --sendoptions="Rw" zpool/backups zfs-recv@10.29.3.27:zpool/backups
 
  New: Repair a DEGRADED pool.
 First let’s offline the device we are going to replace:
 zpool offline tank0 ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx
 Now let us have a look at the pool status.
 zpool status

NAME                                            STATE     READ WRITE CKSUM
tank0                                           DEGRADED     0     0     0
  raidz2-1                                      DEGRADED     0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-WDC_WD80EFZX-68UW8N0_xxxxxxxx           ONLINE       0     0     0
    ata-TOSHIBA_HDWG180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-TOSHIBA_HDWG180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx  OFFLINE      0     0     0
    ata-ST4000VX007-2DT166_xxxxxxxx             ONLINE       0     0     0
 Sweet, the device is offline (last time it didn't show as offline for me, but the offline command returned a status code of 0).
 Time to shut the server down and physically replace the disk.
 shutdown -h now
 When you start again the server, it’s time to instruct ZFS to replace the removed device with the disk we just installed.
 zpool replace tank0 \
    ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx \
    /dev/disk/by-id/ata-TOSHIBA_HDWG180_xxxxxxxxxxxx
 zpool status tank0

pool: main
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Sep 22 12:40:28 2023
        4.00T scanned at 6.85G/s, 222G issued at 380M/s, 24.3T total
        54.7G resilvered, 0.89% done, 18:28:03 to go
NAME                                              STATE     READ WRITE CKSUM
tank0                                             DEGRADED     0     0     0
  raidz2-1                                        DEGRADED     0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx              ONLINE       0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx              ONLINE       0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx              ONLINE       0     0     0
    ata-WDC_WD80EFZX-68UW8N0_xxxxxxxx             ONLINE       0     0     0
    ata-TOSHIBA_HDWG180_xxxxxxxxxxxx              ONLINE       0     0     0
    ata-TOSHIBA_HDWG180_xxxxxxxxxxxx              ONLINE       0     0     0
    replacing-6                                   DEGRADED     0     0     0
      ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx  OFFLINE      0     0     0
      ata-TOSHIBA_HDWG180_xxxxxxxxxxxx            ONLINE       0     0     0  (resilvering)
    ata-ST4000VX007-2DT166_xxxxxxxx               ONLINE       0     0     0
 The disk is replaced and getting resilvered (which may take a long time to run (18 hours in a 8TB disk in my case).
 Once the resilvering is done; this is what the pool looks like.
 zpool list

NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank0    43.5T  33.0T  10.5T     14.5T     7%    75%  1.00x  ONLINE  -
 If you want to read other blogs that have covered the same topic check out 1.
 
  New: Stop a ZFS scrub.
 zpool scrub -s my_pool
 
  New: Mount a dataset that is encrypted.
 If your dataset is encrypted using a key file you need to:
  Mount the device that has your keys
 Import the pool without loading the key because you want to override the keylocation attribute with zfs load-key. Without the -l option, any encrypted datasets won't be mounted, which is what you want.
 Load the key(s) for the dataset(s)
 Mount the dataset(s).
 
 zpool import rpool    # without the `-l` option!
zfs load-key -L file:///path/to/keyfile rpool
zfs mount rpool
 
  New: Umount a pool.
 zpool export pool-name
 
  Correction: Improve the Repair a DEGRADED pool instructions.
 First you need to make sure that it is in fact a problem of the disk. Check the dmesg to see if there are any traces of reading errors, or SATA cable errors.
 A friend suggested to mark the disk as healthy and do a resilver on the same disk. If the error is reproduced in the next days, then replace the disk. A safer approach is to resilver on a new disk, analyze the disk when it's not connected to the pool, and if you feel it's safe then save it as a cold spare.
 
  New: Remove all snapshots of a dataset.
 zfs list -t snapshot -o name path/to/dataset | tail -n+2 | tac | xargs -n 1 zfs destroy -r
 
 
 ZFS Prometheus exporter
   New: Introduce the ZFS exporter.
 You can use a zfs exporter to create alerts on your ZFS pools, filesystems, snapshots and volumes.
 It's not easy to match the exporter metrics with the output of zfs list -o space. Here is a correlation table:
  USED: zfs_dataset_used_bytes{type="filesystem"}
 AVAIL: zfs_dataset_available_bytes{type="filesystem"}
 LUSED: zfs_dataset_logical_used_bytes{type="filesystem"}
 USEDDS: zfs_dataset_used_by_dataset_bytes="filesystem"}
 USEDSNAP: Currently there is no published metric to get this data. You can either use zfs_dataset_used_bytes - zfs_dataset_used_by_dataset_bytes which will show wrong data if the dataset has children or try to do sum by (hostname,filesystem) (zfs_dataset_used_bytes{type='snapshot'}) which returns smaller sizes than expected.
 
 It also covers the installation as well as some nice alerts.
 
  Correction: Improve alerts.
 
 Correction: Update the alerts to the more curated version.
  New: Useful inhibits.
 Some you may want to inhibit some of these rules for some of your datasets. These subsections should be added to the alertmanager.yml file under the inhibit_rules field.
 Ignore snapshots on some datasets: Sometimes you don't want to do snapshots on a dataset
 - target_matchers:
    - alertname = ZfsDatasetWithNoSnapshotsError
    - hostname = my_server_1
    - filesystem = tmp
 Ignore snapshots growth: Sometimes you don't mind if the size of the data saved in the filesystems doesn't change too much between snapshots doesn't change much specially in the most frequent backups because you prefer to keep the backup cadence. It's interesting to have the alert though so that you can get notified of the datasets that don't change that much so you can tweak your backup policy (even if zfs snapshots are almost free).
   - target_matchers:
    - alertname =~ "ZfsSnapshotType(Frequently|Hourly)SizeError"
    - filesystem =~ "(media/(docs|music))"
 
 
 Monitoring
 Loki
   New: Introduce loki.
 Loki is a set of components that can be composed into a fully featured logging stack.
 Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even locally on the filesystem.
 A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.
 
  New: How to install loki.
 There are many ways to install Loki, we're going to do it using docker-compose taking their example as a starting point and complementing our already existent grafana docker-compose.
 It makes use of the environment variables to configure Loki, that's why we have the -config.expand-env=true flag in the command line launch.
 In the grafana datasources directory add loki.yaml:
 ---
apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    orgId: 1
    url: http://loki:3100
    basicAuth: false
    isDefault: true
    version: 1
    editable: false
 Storage configuration:
 Unlike other logging systems, Grafana Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.
 Loki 2.0 brings an index mechanism named ‘boltdb-shipper’ and is what we now call Single Store. This type only requires one store, the object store, for both the index and chunks.
 Loki 2.8 adds TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki as it improves query performance, reduces TCO and has the same feature parity as “boltdb-shipper”.
 
 
 Grafana
   New: Introduce grafana.
 Grafana is a web application to create dashboards.
 Installation: We're going to install it with docker-compose and connect it to Authentik.
 Create the Authentik connection:
 Assuming that you have the terraform authentik provider configured, use the next terraform code:
 variable "grafana_name" {
  type        = string
  description = "The name shown in the Grafana application."
  default     = "Grafana"
}

variable "grafana_redirect_uri" {
  type        = string
  description = "The redirect url configured on Grafana."
}

variable "grafana_icon" {
  type        = string
  description = "The icon shown in the Grafana application"
  default     = "/application-icons/grafana.svg"
}

resource "authentik_application" "grafana" {
  name              = var.grafana_name
  slug              = "grafana"
  protocol_provider = authentik_provider_oauth2.grafana.id
  meta_icon         = var.grafana_icon
  lifecycle {
    ignore_changes = [
      # The terraform provider is continuously changing the attribute even though it's set
      meta_icon,
    ]
  }
}

resource "authentik_provider_oauth2" "grafana" {
  name               = var.grafana_name
  client_id          = "grafana"
  authorization_flow = data.authentik_flow.default-authorization-flow.id
  property_mappings = [
    data.authentik_scope_mapping.email.id,
    data.authentik_scope_mapping.openid.id,
    data.authentik_scope_mapping.profile.id,
  ]
  redirect_uris = [
    var.grafana_redirect_uri,
  ]
  signing_key = data.authentik_certificate_key_pair.default.id
  access_token_validity = "minutes=120"
}

data "authentik_certificate_key_pair" "default" {
  name = "authentik Self-signed Certificate"
}

data "authentik_flow" "default-authorization-flow" {
  slug = "default-provider-authorization-implicit-consent"
}

output "grafana_oauth_id" {
  value = authentik_provider_oauth2.grafana.client_id
}

output "grafana_oauth_secret" {
  value = authentik_provider_oauth2.grafana.client_secret
}
 
  Correction: Install grafana.
 ---
version: "3.8"
services:
  grafana:
    image: grafana/grafana-oss:${GRAFANA_VERSION:-latest}
    container_name: grafana
    restart: unless-stopped
    volumes:
      - data:/var/lib/grafana
    networks:
      - grafana
      - monitorization
      - swag
    env_file:
      - .env
    depends_on:
      - db
  db:
    image: postgres:${DATABASE_VERSION:-15}
    restart: unless-stopped
    container_name: grafana-db
    environment:
      - POSTGRES_DB=${GF_DATABASE_NAME:-grafana}
      - POSTGRES_USER=${GF_DATABASE_USER:-grafana}
      - POSTGRES_PASSWORD=${GF_DATABASE_PASSWORD:?database password required}
    networks:
      - grafana
    volumes:
      - db-data:/var/lib/postgresql/data
    env_file:
      - .env

networks:
  grafana:
    external:
      name: grafana
  monitorization:
    external:
      name: monitorization
  swag:
    external:
      name: swag

volumes:
  data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/grafana/app
  db-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/grafana/database
 Where the monitorization network is where prometheus and the rest of the stack listens, and swag the network to the gateway proxy.
 It uses the .env file to store the required configuration, to connect grafana with authentik you need to add the next variables:
 GF_AUTH_GENERIC_OAUTH_ENABLED="true"
GF_AUTH_GENERIC_OAUTH_NAME="authentik"
GF_AUTH_GENERIC_OAUTH_CLIENT_ID="<Client ID from above>"
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET="<Client Secret from above>"
GF_AUTH_GENERIC_OAUTH_SCOPES="openid profile email"
GF_AUTH_GENERIC_OAUTH_AUTH_URL="https://authentik.company/application/o/authorize/"
GF_AUTH_GENERIC_OAUTH_TOKEN_URL="https://authentik.company/application/o/token/"
GF_AUTH_GENERIC_OAUTH_API_URL="https://authentik.company/application/o/userinfo/"
GF_AUTH_SIGNOUT_REDIRECT_URL="https://authentik.company/application/o/<Slug of the application from above>/end-session/"
GF_AUTH_OAUTH_AUTO_LOGIN="true"
GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH="contains(groups[*], 'Grafana Admins') && 'Admin' || contains(groups[*], 'Grafana Editors') && 'Editor' || 'Viewer'"
 In the configuration above you can see an example of a role mapping. Upon login, this configuration looks at the groups of which the current user is a member. If any of the specified group names are found, the user will be granted the resulting role in Grafana.
 In the example shown above, one of the specified group names is "Grafana Admins". If the user is a member of this group, they will be granted the "Admin" role in Grafana. If the user is not a member of the "Grafana Admins" group, it moves on to see if the user is a member of the "Grafana Editors" group. If they are, they are granted the "Editor" role. Finally, if the user is not found to be a member of either of these groups, it fails back to granting the "Viewer" role.
 Also make sure in your configuration that root_url is set correctly, otherwise your redirect url might get processed incorrectly. For example, if your grafana instance is running on the default configuration and is accessible behind a reverse proxy at https://grafana.company, your redirect url will end up looking like this, https://grafana.company/. If you get user does not belong to org error when trying to log into grafana for the first time via OAuth, check if you have an organization with the ID of 1, if not, then you have to add the following to your grafana config:
 [users]
auto_assign_org = true
auto_assign_org_id = <id-of-your-default-organization>
 Once you've made sure that the oauth works, go to /admin/users and remove the admin user.
 
  New: Configure grafana.
 Grafana has default and custom configuration files. You can customize your Grafana instance by modifying the custom configuration file or by using environment variables. To see the list of settings for a Grafana instance, refer to View server settings.
 To override an option use GF_<SectionName>_<KeyName>. Where the section name is the text within the brackets. Everything should be uppercase, . and - should be replaced by _. For example, if you have these configuration settings:
 instance_name = ${HOSTNAME}

[security]
admin_user = admin

[auth.google]
client_secret = 0ldS3cretKey

[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true

[feature_toggles]
enable = newNavigation
 You can override variables on Linux machines with:
 export GF_DEFAULT_INSTANCE_NAME=my-instance
export GF_SECURITY_ADMIN_USER=owner
export GF_AUTH_GOOGLE_CLIENT_SECRET=newS3cretKey
export GF_PLUGIN_GRAFANA_IMAGE_RENDERER_RENDERING_IGNORE_HTTPS_ERRORS=true
export GF_FEATURE_TOGGLES_ENABLE=newNavigation
 And in the docker compose you can edit the .env file. Mine looks similar to:
 GRAFANA_VERSION=latest
GF_DEFAULT_INSTANCE_NAME="production"
GF_SERVER_ROOT_URL="https://your.domain.org"

GF_DATABASE_TYPE=postgres
DATABASE_VERSION=15
GF_DATABASE_HOST=grafana-db:5432
GF_DATABASE_NAME=grafana
GF_DATABASE_USER=grafana
GF_DATABASE_PASSWORD="change-for-a-long-password"
GF_DATABASE_SSL_MODE=disable

GF_AUTH_GENERIC_OAUTH_ENABLED="true"
GF_AUTH_GENERIC_OAUTH_NAME="authentik"
GF_AUTH_GENERIC_OAUTH_CLIENT_ID="<Client ID from above>"
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET="<Client Secret from above>"
GF_AUTH_GENERIC_OAUTH_SCOPES="openid profile email"
GF_AUTH_GENERIC_OAUTH_AUTH_URL="https://authentik.company/application/o/authorize/"
GF_AUTH_GENERIC_OAUTH_TOKEN_URL="https://authentik.company/application/o/token/"
GF_AUTH_GENERIC_OAUTH_API_URL="https://authentik.company/application/o/userinfo/"
GF_AUTH_SIGNOUT_REDIRECT_URL="https://authentik.company/application/o/<Slug of the application from above>/end-session/"
GF_AUTH_OAUTH_AUTO_LOGIN="true"
GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH="contains(groups[*], 'Grafana Admins') && 'Admin' || contains(groups[*], 'Grafana Editors') && 'Editor' || 'Viewer'"
 
  New: Configure datasources.
 You can manage data sources in Grafana by adding YAML configuration files in the provisioning/datasources directory. Each config file can contain a list of datasources to add or update during startup. If the data source already exists, Grafana reconfigures it to match the provisioned configuration file.
 The configuration file can also list data sources to automatically delete, called deleteDatasources. Grafana deletes the data sources listed in deleteDatasources before adding or updating those in the datasources list.
 For example to configure a Prometheus datasource use:
 apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    # Access mode - proxy (server in the UI) or direct (browser in the UI).
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      manageAlerts: true
      prometheusType: Prometheus
      prometheusVersion: 2.44.0
      cacheLevel: 'High'
      disableRecordingRules: false
      incrementalQueryOverlapWindow: 10m
      exemplarTraceIdDestinations: []
 
  New: Configure dashboards.
 You can manage dashboards in Grafana by adding one or more YAML config files in the provisioning/dashboards directory. Each config file can contain a list of dashboards providers that load dashboards into Grafana from the local filesystem.
 Create one file called dashboards.yaml with the next contents:
 ---
apiVersion: 1
providers:
  - name: default # A uniquely identifiable name for the provider
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards/definitions
 Then inside the config directory of your docker compose create the directory provisioning/dashboards/definitions and add the json of the dashboards themselves. You can download them from the dashboard pages. For example:
  Node Exporter
 Blackbox Exporter
 Alertmanager
 
 
  New: Configure the plugins.
 To install plugins in the Docker container, complete the following steps:
  Pass the plugins you want to be installed to Docker with the GF_INSTALL_PLUGINS environment variable as a comma-separated list.
 This sends each plugin name to grafana-cli plugins install ${plugin} and installs them when Grafana starts.
 
 For example:
 docker run -d -p 3000:3000 --name=grafana \
  -e "GF_INSTALL_PLUGINS=grafana-clock-panel, grafana-simple-json-datasource" \
  grafana/grafana-oss
 To specify the version of a plugin, add the version number to the GF_INSTALL_PLUGINS environment variable. For example: GF_INSTALL_PLUGINS=grafana-clock-panel 1.0.1.
 To install a plugin from a custom URL, use the following convention to specify the URL: <url to plugin zip>;<plugin install folder name>. For example: GF_INSTALL_PLUGINS=https://github.com/VolkovLabs/custom-plugin.zip;custom-plugin.
 
  Correction: Improve installation method.
 Add more configuration values such as:
 GF_SERVER_ENABLE_GZIP="true"

GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN="true"

GF_LOG_MODE="console file"
GF_LOG_LEVEL="info"
 
  Correction: Warning when configuring datasources.
 Be careful to set the timeInterval variable to the value of how often you scrape the data from the node exporter to avoid this issue.
 
 
 SIEM
   New: Alertmanager routes.
 A route block defines a node in a routing tree and its children. Its optional configuration parameters are inherited from its parent node if not set.
 Every alert enters the routing tree at the configured top-level route, which must match all alerts (i.e. not have any configured matchers). It then traverses the child nodes. If continue is set to false, it stops after the first matching child. If continue is true on a matching node, the alert will continue matching against subsequent siblings. If an alert does not match any children of a node (no matching child nodes, or none exist), the alert is handled based on the configuration parameters of the current node.
 A basic configuration would be:
 route:
  group_by: [job, alertname, severity]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'email'
  routes:
    - match:
        alertname: Watchdog
      receiver: 'null'
 
  New: Add Wazuh SIEM.
 Wazuh
 
 
 Blackbox Exporter
   New: Check TCP with TLS.
 If you want to test for example if an LDAP is serving the correct certificate on the port 636 you can use:
 tcp_ssl_connect:
  prober: tcp
  timeout: 10s
  tls: true
 - name: Ldap
  url: my-ldap-server:636
  module: tcp_ssl_connect
 
 
 Node Exporter
  Correction: Improve how to install it.
 
 Authentication
 Authentik
   New: Introduce Authentik.
 Authentik is an open-source Identity Provider focused on flexibility and versatility.
 What I like:
  Is maintained and popular
 It has a clean interface
 They have their own terraform provider Oo!
 
 What I don't like:
  It's heavy focused on GUI interaction, but you can export the configuration to YAML files to be applied without the GUI interaction.
 The documentation is oriented to developers and not users. It's a little difficult to get a grasp on how to do things in the platform without following blog posts.
 
 In the article you can also find:
  How to install it
 How to configure and use the terraform provider
 How to configure some applications, such as gitea
 How to configure the user registration by invitation link
 How to use blueprints
 
 
  Correction: Configure the invitation flow with terraform.
 
  New: Hide and application from a user.
 Application access can be configured using (Policy) Bindings. Click on an application in the applications list, and select the Policy / Group / User Bindings tab. There you can bind users/groups/policies to grant them access. When nothing is bound, everyone has access. You can use this to grant access to one or multiple users/groups, or dynamically give access using policies.
 With terraform you can use authentik_policy_binding, for example:
 resource "authentik_policy_binding" "admin" {
  target = authentik_application.gitea.uuid
  group  = authentik_group.admins.id
  order  = 0
}
 
  New: Configure password recovery.
 Password recovery is not set by default, in the article you can find the terraform resources needed for it to work.
 
  New: Protect applications that don't have authentication.
 Some applications don't have authentication, for example prometheus. You can use Authentik in front of such applications to add the authentication and authorization layer.
 Authentik can be used as a (very) simple reverse proxy by using its Provider feature with the regular "Proxy" setting. This let's you wrap authentication around a sub-domain / app where it normally wouldn't have authentication (or not the type of auth that you would specifically want) and then have Authentik handle the proxy forwarding and Auth.
 In this mode, there is no domain level nor 'integrated' authentication into your desired app; Authentik becomes both your reverse proxy and auth for this one particular app or (sub) domain. This mode does not forward authentication nor let you log in into any app. It's just acts like an authentication wrapper.
 It's best to use a normal reverse proxy out front of Authentik. This adds a second layer of routing to deal with but Authentik is not NGINX or a reverse proxy system, so it does not have that many configuration options.
 We'll use the following fake domains in this example:
  Authentik domain: auth.yourdomain.com
 App domain: app.yourdomain.com
 Nginx: nginx.yourdomain.com
 Authentik's docker conter name: auth_server
 
 The steps are:
  Configure the proxy provider:
 
 # ---------------
# -- Variables --
# ---------------

variable "prometheus_url" {
  type        = string
  description = "The url to access the service."
}

# ----------
# -- Data --
# ----------

data "authentik_flow" "default-authorization-flow" {
  slug = "default-provider-authorization-implicit-consent"
}

# --------------------
# --    Provider    --
# --------------------

resource "authentik_provider_proxy" "prometheus" {
  name               = "Prometheus"
  internal_host      = "http://prometheus:9090"
  external_host      = var.prometheus_url
  authorization_flow = data.authentik_flow.default-authorization-flow.id
  internal_host_ssl_validation = false
}
 
  Correction: Finish the installation of prometheus.
 
  New: Disregard monitorization.
 I've skimmed through the prometheus metrics exposed at :9300/metrics in the core and they aren't that useful :(
 
  New: Troubleshoot I can't log in to authentik.
 In case you can't login anymore, perhaps due to an incorrectly configured stage or a failed flow import, you can create a recovery key.
 To create the key, run the following command:
 docker run --it authentik bash
ak create_recovery_key 1 akadmin
 This will output a link, that can be used to instantly gain access to authentik as the user specified above. The link is valid for amount of years specified above, in this case, 1 year.
 
 
 Operating Systems
 Linux
   New: Interesting learning pages.
  https://explainshell.com/
 https://linuxcommandlibrary.com/
 
 
 
 Linux Snippets
   New: Use a pass password in a Makefile.
 TOKEN ?= $(shell bash -c '/usr/bin/pass show path/to/token')

diff:
    @AUTHENTIK_TOKEN=$(TOKEN) terraform plan
 
  New: Install a new font.
 Install a font manually by downloading the appropriate .ttf or otf files and placing them into /usr/local/share/fonts (system-wide), ~/.local/share/fonts (user-specific) or ~/.fonts (user-specific). These files should have the permission 644 (-rw-r--r--), otherwise they may not be usable.
 
  New: Get VPN password from pass.
 To be able to retrieve the user and password from pass you need to run the openvpn command with the next flags:
 sudo bash -c "openvpn --config config.ovpn  --auth-user-pass <(echo -e 'user_name\n$(pass show vpn)')"
 Assuming that vpn is an entry of your pass password store.
 
  New: Measure the performance, IOPS of a disk.
 To measure disk IOPS performance in Linux, you can use the fio tool. Install it with
 apt-get install fio
 Then you need to go to the directory where your disk is mounted. The test is done by performing read/write operations in this directory.
 To do a random read/write operation test an 8 GB file will be created. Then fio will read/write a 4KB block (a standard block size) with the 75/25% by the number of reads and writes operations and measure the performance.
 fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75
 
  New: What is /var/log/tallylog.
 /var/log/tallylog is the file where the PAM linux module (used for authentication of the machine) keeps track of the failed ssh logins in order to temporarily block users.
 
  New: Manage users.
  Change main group of user
 
 usermod -g {{ group_name }} {{ user_name }}
  Add user to group
 
 usermod -a -G {{ group_name }} {{ user_name }}
  Remove user from group.
 
 usermod -G {{ remaining_group_names }} {{ user_name }}
 You have to execute groups {{ user }} get the list and pass the remaining to the above command
  Change uid and gid of the user
 
 usermod -u {{ newuid }} {{ login }}
groupmod -g {{ newgid }} {{ group }}
find / -user {{ olduid }} -exec chown -h {{ newuid }} {} \;
find / -group {{ oldgid }} -exec chgrp -h {{ newgid }} {} \;
usermod -g {{ newgid }} {{ login }}
 
  New: Manage ssh keys.
  Generate ed25519 key
 
 ssh-keygen -t ed25519 -f {{ path_to_keyfile }}
  Generate RSA key
 
 ssh-keygen -t rsa -b 4096 -o -a 100 -f {{ path_to_keyfile }}
  Generate different comment
 
 ssh-keygen -t ed25519 -f {{ path_to_keyfile }} -C {{ email }}
  Generate key headless, batch
 
 ssh-keygen -t ed25519 -f {{ path_to_keyfile }} -q -N ""
  Generate public key from private key
 
 ssh-keygen -y -f {{ path_to_keyfile }} > {{ path_to_public_key_file }}
  Get fingerprint of key ssh-keygen -lf {{ path_to_key }}
 
 
  New: Measure the network performance between two machines.
 Install iperf3 with apt-get install iperf3 on both server and client.
 On the server system run:
 server#: iperf3 -i 10 -s
 Where:
  -i: the interval to provide periodic bandwidth updates
 -s: listen as a server
 
 On the client system:
 client#: iperf3 -i 10 -w 1M -t 60 -c [server hostname or ip address]
 Where:
  -i: the interval to provide periodic bandwidth updates
 -w: the socket buffer size (which affects the TCP Window). The buffer size is also set on the server by this client command.
 -t: the time to run the test in seconds
 -c: connect to a listening server at…
 
 Sometimes is interesting to test both ways as they may return different outcomes
 
  New: Force umount nfs mounted directory.
 umount -l path/to/mounted/dir
 
  New: Configure fstab to mount nfs.
 NFS stands for ‘Network File System’. This mechanism allows Unix machines to share files and directories over the network. Using this feature, a Linux machine can mount a remote directory (residing in a NFS server machine) just like a local directory and can access files from it.
 An NFS share can be mounted on a machine by adding a line to the /etc/fstab file.
 The default syntax for fstab entry of NFS mounts is as follows.
 Server:/path/to/export /local_mountpoint nfs <options> 0 0
 Where:
  Server: The hostname or IP address of the NFS server where the exported directory resides.
 /path/to/export: The shared directory (exported folder) path.
 /local_mountpoint: Existing directory in the host where you want to mount the NFS share.
 
 You can specify a number of options that you want to set on the NFS mount:
  soft/hard: When the mount option hard is set, if the NFS server crashes or becomes unresponsive, the NFS requests will be retried indefinitely. You can set the mount option intr, so that the process can be interrupted. When the NFS server comes back online, the process can be continued from where it was while the server became unresponsive.
 
 When the option soft is set, the process will be reported an error when the NFS server is unresponsive after waiting for a period of time (defined by the timeo option). In certain cases soft option can cause data corruption and loss of data. So, it is recommended to use hard and intr options.
  noexec: Prevents execution of binaries on mounted file systems. This is useful if the system is mounting a non-Linux file system via NFS containing incompatible binaries.
 nosuid: Disables set-user-identifier or set-group-identifier bits. This prevents remote users from gaining higher privileges by running a setuid program.
 tcp: Specifies the NFS mount to use the TCP protocol.
 udp: Specifies the NFS mount to use the UDP protocol.
 nofail: Prevent issues when rebooting the host. The downside is that if you have services that depend on the volume to be mounted they won't behave as expected.
 
 
  New: Fix limit on the number of inotify watches.
 Programs that sync files such as dropbox, git etc use inotify to notice changes to the file system. The limit can be see by -
 cat /proc/sys/fs/inotify/max_user_watches
 For me, it shows 65536. When this limit is not enough to monitor all files inside a directory it throws this error.
 If you want to increase the amount of inotify watchers, run the following in a terminal:
 echo fs.inotify.max_user_watches=100000 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
 Where 100000 is the desired number of inotify watches.
 
  New: Get class of a window.
 Use xprop and click the window.
 Get the current brightness level with cat /sys/class/backlight/intel_backlight/brightness. Imagine it's 1550, then if you want to lower the brightness use:
 sudo echo 500 > /sys/class/backlight/intel_backlight/brightness
 
  New: SSH tunnel.
 ssh -D 9090 -N -f user@host
 
  New: Fix the SSH client kex_exchange_identification: read: Connection reset by peer error.
 Restart the ssh service.
 
  New: Automatic reboot after power failure.
 That's not something you can control in your operating system. That's what the BIOS is for. In most BIOS setups there'll be an option like After power loss with possible values like Power off and Reboot.
 You can also edit /etc/default/grub and add:
 GRUB_RECORDFAIL_TIMEOUT=5
 Then run:
 sudo update-grub
 This will make your machine display the boot options for 5 seconds before it boot the default option (instead of waiting forever for you to choose one).
 
  New: Add sshuttle information link.
 If you need a more powerful ssh tunnel you can try sshuttle
 
  New: Reset failed systemd services.
 Use systemctl to remove the failed status. To reset all units with failed status:
 systemctl reset-failed
 or just your specific unit:
 systemctl reset-failed openvpn-server@intranert.service
 
  New: Get the current git branch.
 git branch --show-current
 
  New: Install latest version of package from backports.
 Add the backports repository:
 vi /etc/apt/sources.list.d/bullseye-backports.list
 deb http://deb.debian.org/debian bullseye-backports main contrib
deb-src http://deb.debian.org/debian bullseye-backports main contrib
 Configure the package to be pulled from backports
 vi /etc/apt/preferences.d/90_zfs
 Package: src:zfs-linux
Pin: release n=bullseye-backports
Pin-Priority: 990
 
  New: Rename multiple files matching a pattern.
 There is rename that looks nice, but you need to install it. Using only find you can do:
 find . -name '*yml' -exec bash -c 'echo mv $0 ${0/yml/yaml}' {} \;
 If it shows what you expect, remove the echo.
 
  New: Force ssh to use password authentication.
 
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no exampleUser@example.com
 feat(linux_snippets#Do a tail -f with grep): Do a tail -f with grep
 tail -f file | grep --line-buffered my_pattern
 
  New: Check if a program exists in the user's PATH.
 command -v <the_command>
 Example use:
 if ! command -v <the_command> &> /dev/null
then
    echo "<the_command> could not be found"
    exit
fi
 
  New: Add interesting tools to explore.
  qbittools: a feature rich CLI for the management of torrents in qBittorrent.
 qbit_manage: tool will help manage tedious tasks in qBittorrent and automate them.
 
 
  New: Limit the resources a docker is using.
 You can either use limits in the docker service itself, see 1 and 2.
 Or/and you can limit it for each docker, see 1 and 2.
 
  New: Wipe a disk.
 Overwrite it many times with badblocks.
 badblocks -wsv -b 4096 /dev/sde | tee disk_wipe_log.txt
 
  New: Impose load on a system to stress it.
 sudo apt-get install stress

stress --cpu 2
 That will fill up the usage of 2 cpus. To run 1 vm stressor using 1GB of virtual memory for 60s, enter:
 stress --vm 1 --vm-bytes 1G --vm-keep -t 60s
 You can also stress io with --io 4, for example to spawn 4 workers.
 
  New: Get the latest tag of a git repository.
 git describe --tags --abbrev=0
 
  New: Configure gpg-agent cache ttl.
 The user configuration (in ~/.gnupg/gpg-agent.conf) can only define the default and maximum caching duration; it can't be disabled.
 The default-cache-ttl option sets the timeout (in seconds) after the last GnuPG activity (so it resets if you use it), the max-cache-ttl option set the timespan (in seconds) it caches after entering your password. The default value is 600 seconds (10 minutes) for default-cache-ttl and 7200 seconds (2 hours) for max-cache-ttl.
 default-cache-ttl 21600
max-cache-ttl 21600
 For this change to take effect, you need to end the session by restarting gpg-agent.
 gpgconf --kill gpg-agent
gpg-agent --daemon --use-standard-socket
 
  New: Get return code of failing find exec.
 When you run find . -exec ls {} \; even if the command run in the exec returns a status code different than 0 you'll get an overall status code of 0 which makes difficult to catch errors in bash scripts.
 You can instead use xargs, for example:
 find /tmp/ -iname '*.sh' -print0 | xargs -0 shellcheck
 This will run shellcheck file_name for each of the files found by the find command.
 
  New: Accept new ssh keys by default.
 While common wisdom is not to disable host key checking, there is a built-in option in SSH itself to do this. It is relatively unknown, since it's new (added in Openssh 6.5).
 This is done with -o StrictHostKeyChecking=accept-new. Or if you want to use it for all hosts you can add the next lines to your ~/.ssh/config:
 Host *
  StrictHostKeyChecking accept-new
 WARNING: use this only if you absolutely trust the IP\hostname you are going to SSH to:
 ssh -o StrictHostKeyChecking=accept-new mynewserver.example.com
 Note, StrictHostKeyChecking=no will add the public key to ~/.ssh/known_hosts even if the key was changed. accept-new is only for new hosts. From the man page:
  If this flag is set to “accept-new” then ssh will automatically add new host keys to the user known hosts files, but will not permit connections to hosts with changed host keys. If this flag is set to “no” or “off”, ssh will automatically add new host keys to the user known hosts files and allow connections to hosts with changed hostkeys to proceed, subject to some restrictions. If this flag is set to ask (the default), new host keys will be added to the user known host files only after the user has confirmed that is what they really want to do, and ssh will refuse to connect to hosts whose host key has changed. The host keys of known hosts will be verified automatically in all cases.
 
 
  New: Do not add trailing / to ls.
 Probably, your ls is aliased or defined as a function in your config files.
 Use the full path to ls like:
 /bin/ls /var/lib/mysql/
 
  New: Convert png to svg.
 Inkscape has got an awesome auto-tracing tool.
  Install Inkscape using sudo apt-get install inkscape
 Import your image
 Select your image
 From the menu bar, select Path > Trace Bitmap Item
 Adjust the tracing parameters as needed
 Save as svg
 
 Check their tracing tutorial for more information.
 Once you are comfortable with the tracing options. You can automate it by using CLI of Inkscape.
 
  New: Redirect stdout and stderr of a cron job to a file.
 */1 * * * * /home/ranveer/vimbackup.sh >> /home/ranveer/vimbackup.log 2>&1
 
  New: Error when unmounting a device Target is busy.
  Check the processes that are using the mountpoint with lsof /path/to/mountpoint
 Kill those processes
 Try the umount again
 
 If that fails, you can use umount -l.
 
 
 Rtorrent
  Correction: Remove unneeded dependencies when installing.
  Correction: Deprecate it in favour of qbittorrent.
 Use qbittorrent instead.
 
 
 aleph
   New: Add application operations.
  How to upgrade it
 Create Aleph admins
 Remove a group
 
 
  New: Ingest gets stuck.
 It looks that Aleph doesn't yet give an easy way to debug it. It can be seen in the next webs:
  Improve the UX for bulk uploading and processing of large number of files
 Document ingestion gets stuck effectively at 100%
 Display detailed ingestion status to see if everything is alright and when the collection is ready
 
 Some interesting ideas I've extracted while diving into these issues is that:
  You can also upload files using the alephclient python command line tool
 Some of the files might fail to be processed without leaving any hint to the uploader or the viewer.
 This results in an incomplete dataset and the users don't get to know that the dataset is incomplete. This is problematic if the completeness of the dataset is crucial for an investigation.
 There is no way to upload only the files that failed to be processed without re-uploading the entire set of documents or manually making a list of the failed documents and re-uploading them
 There is no way for uploaders or Aleph admins to see an overview of processing errors to figure out why some files are failing to be processed without going through docker logs (which is not very user-friendly)
 There was an attempt to improve the way ingest-files manages the pending tasks, it's merged into the release/4.0.0 branch, but it has not yet arrived main.
 
 There are some tickets that attempt to address these issues on the command line:
  Allow users to upload/crawl new files only
 Check if alephclient crawldir was 100% successful or not
 
 I think it's interesting either to contribute to alephclient to solve those issues or if it's complicated create a small python script to detect which files were not uploaded and try to reindex them and/or open issues that will prevent future ingests to fail.
 
 
 gitsigns
   New: Introduce gitsigns.
 Gitsigns is a neovim plugin to create git decorations similar to the vim plugin gitgutter but written purely in Lua.
 Installation:
 Add to your plugins.lua file:
   use {'lewis6991/gitsigns.nvim'}
 Install it with :PackerInstall.
 Configure it in your init.lua with:
 -- Configure gitsigns
require('gitsigns').setup({
  on_attach = function(bufnr)
    local gs = package.loaded.gitsigns

    local function map(mode, l, r, opts)
      opts = opts or {}
      opts.buffer = bufnr
      vim.keymap.set(mode, l, r, opts)
    end

    -- Navigation
    map('n', ']c', function()
      if vim.wo.diff then return ']c' end
      vim.schedule(function() gs.next_hunk() end)
      return '<Ignore>'
    end, {expr=true})

    map('n', '[c', function()
      if vim.wo.diff then return '[c' end
      vim.schedule(function() gs.prev_hunk() end)
      return '<Ignore>'
    end, {expr=true})

    -- Actions
    map('n', '<leader>gs', gs.stage_hunk)
    map('n', '<leader>gr', gs.reset_hunk)
    map('v', '<leader>gs', function() gs.stage_hunk {vim.fn.line('.'), vim.fn.line('v')} end)
    map('v', '<leader>gr', function() gs.reset_hunk {vim.fn.line('.'), vim.fn.line('v')} end)
    map('n', '<leader>gS', gs.stage_buffer)
    map('n', '<leader>gu', gs.undo_stage_hunk)
    map('n', '<leader>gR', gs.reset_buffer)
    map('n', '<leader>gp', gs.preview_hunk)
    map('n', '<leader>gb', function() gs.blame_line{full=true} end)
    map('n', '<leader>gb', gs.toggle_current_line_blame)
    map('n', '<leader>gd', gs.diffthis)
    map('n', '<leader>gD', function() gs.diffthis('~') end)
    map('n', '<leader>ge', gs.toggle_deleted)

    -- Text object
    map({'o', 'x'}, 'ih', ':<C-U>Gitsigns select_hunk<CR>')
  end
})
 Usage:
 Some interesting bindings:
  ]c: Go to next diff chunk
 [c: Go to previous diff chunk
 <leader>gs: Stage chunk, it works both in normal and visual mode
 <leader>gr: Restore chunk from index, it works both in normal and visual mode
 <leader>gp: Preview diff, you can use it with ]c and [c to see all the chunk diffs
 <leader>gb: Show the git blame of the line as a shadowed comment
 
 
 
 Diffview
   New: Introduce DiffView.
 Diffview is a single tabpage interface for easily cycling through diffs for all modified files for any git rev.
 Installation:
 If you're using it with NeoGit and Packer use:
   use {
    'NeogitOrg/neogit',
    requires = {
      'nvim-lua/plenary.nvim',
      'sindrets/diffview.nvim',
      'nvim-tree/nvim-web-devicons'
    }
  }
 Usage:
 Calling :DiffviewOpen with no args opens a new Diffview that compares against the current index. You can also provide any valid git rev to view only changes for that rev.
 Examples:
  :DiffviewOpen
 :DiffviewOpen HEAD~2
 :DiffviewOpen HEAD~4..HEAD~2
 :DiffviewOpen d4a7b0d
 :DiffviewOpen d4a7b0d^!
 :DiffviewOpen d4a7b0d..519b30e
 :DiffviewOpen origin/main...HEAD
 
 You can also provide additional paths to narrow down what files are shown :DiffviewOpen HEAD~2 -- lua/diffview plugin.
 Additional commands for convenience:
  :DiffviewClose: Close the current diffview. You can also use :tabclose.
 :DiffviewToggleFiles: Toggle the file panel.
 :DiffviewFocusFiles: Bring focus to the file panel.
 :DiffviewRefresh: Update stats and entries in the file list of the current Diffview.
 
 With a Diffview open and the default key bindings, you can:
  Cycle through changed files with <tab> and <s-tab>
 You can stage changes with -
 Restore a file with X
 Refresh the diffs with R
 Go to the file panel with <leader>e
 
 
  New: Use the same binding to open and close the diffview windows.
 vim.keymap.set('n', 'dv', function()
  if next(require('diffview.lib').views) == nil then
    vim.cmd('DiffviewOpen')
  else
    vim.cmd('DiffviewClose')
  end
end)
 
 
 Beets
   New: Introduce tridactyl.
 Tridactyl is a Vim-like interface for Firefox, inspired by Vimperator/Pentadactyl.
 In the article you'll also find:
  How to select text on the page
 
 
  New: Guide on how to start using it.
 You’ll want to set a few basic options before you start using beets. The configuration is stored in a text file. You can show its location by running beet config -p, though it may not exist yet. Run beet config -e to edit the configuration in your favorite text editor. The file will start out empty, but here’s good place to start:
 directory: ~/music

library: ~/data/musiclibrary.db
 The default configuration assumes you want to start a new organized music folder (that directory above) and that you’ll copy cleaned-up music into that empty folder using beets’ import command. But you can configure beets to behave many other ways:
   Start with a new empty directory, but move new music in instead of copying it (saving disk space). Put this in your config file:
 import:
    move: yes
 
  Keep your current directory structure; importing should never move or copy files but instead just correct the tags on music. Put the line copy: no under the import: heading in your config file to disable any copying or renaming. Make sure to point directory at the place where your music is currently stored.
 
  Keep your current directory structure and do not correct files’ tags: leave files completely unmodified on your disk. (Corrected tags will still be stored in beets’ database, and you can use them to do renaming or tag changes later.) Put this in your config file:
 import:
    copy: no
    write: no
 to disable renaming and tag-writing.
 
 
 
  New: Importing your library.
 The next step is to import your music files into the beets library database. Because this can involve modifying files and moving them around, data loss is always a possibility, so now would be a good time to make sure you have a recent backup of all your music. We’ll wait.
 There are two good ways to bring your existing library into beets. You can either: (a) quickly bring all your files with all their current metadata into beets’ database, or (b) use beets’ highly-refined autotagger to find canonical metadata for every album you import. Option (a) is really fast, but option (b) makes sure all your songs’ tags are exactly right from the get-go. The point about speed bears repeating: using the autotagger on a large library can take a very long time, and it’s an interactive process. So set aside a good chunk of time if you’re going to go that route.
 If you’ve got time and want to tag all your music right once and for all, do this:
 beet import /path/to/my/music
 (Note that by default, this command will copy music into the directory you specified above. If you want to use your current directory structure, set the import.copy config option.) To take the fast, un-autotagged path, just say:
 beet import -A /my/huge/mp3/library
 Note that you just need to add -A for “don’t autotag”.
 
 
 Gancio
   New: Introduce Gancio.
 Gancio is a shared agenda for local communities.
  Docs
 Source
 Home
 Telegram bridge
 
 
  New: Add radar squat.
 https://radar.squat.net
 
 
 google chrome
   Correction: Update the installation steps.
   Import the GPG key, and use the following command. 
sudo wget -O- https://dl.google.com/linux/linux_signing_key.pub | gpg --dearmor > /usr/share/keyrings/google-chrome.gpg
 
  Once the GPG import is complete, you will need to import the Google Chrome repository.
 
 
 echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] http://dl.google.com/linux/chrome/deb/ stable main' | sudo tee /etc/apt/sources.list.d/google-chrome.list
  Install the program: apt-get update
apt-get install google-chrome-stable
 
 
 
 Gotify
   New: Introduce gotify.
 Gotify is a simple server for sending and receiving messages in real-time per WebSocket.
 Not there yet:
  Reactions on the notifications
 
 
 
 Hard drive health
   New: Check the disk health with smartctl.
 Start with a long self test with smartctl. Assuming the disk to test is /dev/sdd:
 smartctl -t long /dev/sdd
 The command will respond with an estimate of how long it thinks the test will take to complete.
 To check progress use:
 smartctl -A /dev/sdd | grep remaining
smartctl -c /dev/sdd | grep remaining
 Don't check too often because it can abort the test with some drives. If you receive an empty output, examine the reported status with:
 smartctl -l selftest /dev/sdd
 If errors are shown, check the dmesg as there are usually useful traces of the error.
 
  New: Check the health of a disk with badblocks.
 The badblocks command will write and read the disk with different patterns, thus overwriting the whole disk, so you will loose all the data in the disk.
 This test is good for rotational disks as there is no disk degradation on massive writes, do not use it on SSD though.
 WARNING: be sure that you specify the correct disk!!
 badblocks -wsv -b 4096 /dev/sde | tee disk_analysis_log.txt
 If errors are shown is that all of the spare sectors of the disk are used, so you must not use this disk anymore. Again, check dmesg for traces of disk errors.
 
 
 i3wm
   New: Move the focus to a container.
 Get the container identifier with xprop and then:
 i3-msg '[title="khime"]' focus
i3-msg '[class="Firefox"]' focus
 
  New: Interact with Python.
 Install the i3ipc library:
 pip install i3ipc
 Create the connection object:
 from i3ipc import Connection, Event

i3 = Connection()
 Interact with i3:
 focused = i3.get_tree().find_focused()
print('Focused window %s is on workspace %s' %
      (focused.name, focused.workspace().name))

outputs = i3.get_outputs()

print('Active outputs:')

for output in filter(lambda o: o.active, outputs):
    print(output.name)

i3.command('focus left')

for container in i3.get_tree().find_fullscreen():
    container.command('fullscreen')

root = i3.get_tree()
print(root.name)
for con in root:
    print(con.name)

def on_workspace_focus(self, e):
    # The first parameter is the connection to the ipc and the second is an object
    # with the data of the event sent from i3.
    if e.current:
        print('Windows on this workspace:')
        for w in e.current.leaves():
            print(w.name)

def on_window_focus(i3, e):
    focused = i3.get_tree().find_focused()
    ws_name = "%s:%s" % (focused.workspace().num, focused.window_class)
    i3.command('rename workspace to "%s"' % ws_name)

i3.on(Event.WORKSPACE_FOCUS, on_workspace_focus)
i3.on(Event.WINDOW_FOCUS, on_window_focus)

i3.main()
 
 
 Jellyfin
   New: Fix Corrupt: SQLitePCL.pretty.SQLiteException: database disk image is malformed.
 If your server log file shows SQLite errors like the following example your jellyfin.db file needs attention.
 'SQLitePCL.pretty.SQLiteException'
 Typical causes of this are sudden and abrupt terminations of the Emby server process, such as a power loss, operating system crash, force killing the server process, etc.
 To solve it there are many steps:
  Remove Database Locks
 Check Database Integrity and Recover Database
 Reset Library Database & Load Fresh
 
 
  New: Restore watched history.
 Jellyfin stores the watched information in one of the .db files, there are two ways to restore it:
  Using scripts that interact with the API like jelly-jar or jellyfin-backup-watched
 Running sqlite queries on the database itself.
 
 The user data is stored in the table UserDatas table in the library.db database file. The media data is stored in the TypedBaseItems table of the same database.
 Comparing the contents of the tables of the broken database (lost watched content) and a backup database, I've seen that the media content is the same after a full library rescan, so the issue was fixed after injecting the missing user data from the backup to the working database through the importing a table from another database sqlite operation.
 
  New: Fix ReadOnly: SQLitePCL.pretty.SQLiteException: attempt to write a readonly database.
 Some of the database files of Jellyfin is not writable by the jellyfin user, check if you changed the ownership of the files, for example in the process of restoring a database file from backup.
 
  New: Deceptive site ahead.
 It seems that Google is marking the domains that host Jellyfin as deceptive. If it happens to you, your users won't be able to access your instance with Firefox, Chrome nor the Android app. Nice uh? It's kind of scary how google is able to control who can access what in the internet without you signing for it.
 If you search the problem online they suggest that you log in with your google account into the Search Console and see the reasons behind it. Many people did this and reported in the issue that they didn't get any useful information through this process. It's a privacy violation though, as now google is able to tie your identity (as your google account is linked to your phone number) with your Jellyfin domain. Completely disgusting.
 To solve this issue you need to file a case with google and wait for them to unban you. It's like asking them for permission so that they let your users access your system. The disgust levels keep on growing. Don't waste your time being creative in the Comments of the request either, it looks like they don't even read them.
 The problem is that until the people from Jellyfin finds a solution, after following this ugly process, you may be flagged again any time in the future (ranging from days to months).
 A mitigation of the problem is to have an alternative domain that your users can use (for example in duckdns.org). You may be lucky that google doesn't block both domains at the same time.
 For more information follow the Jellyfin issue or the Jellyfin reddit thread.
 
  New: Missing features.
  Hide movie or tv show from my gallery: Tracked by these feature requests 1 and 2
 
 
  New: Introduce Jellyfin Desktop.
  Download the latest deb package from the releases page
 Install the dependencies
 Run dpkg -i
 
 If you're on a TV you may want to enable the TV mode so that the remote keys work as expected. The play/pause/next/prev won't work until this issue is solved, but it's not that bad to use the "Ok" and then navigate with the arrow keys.
 
  New: Introduce Jellycon.
 JellyCon is a lightweight Kodi add-on that lets you browse and play media files directly from your Jellyfin server within the Kodi interface. It can be thought of as a thin frontend for a Jellyfin server.
 It's not very pleasant to use though.
 
  New: Forgot Password. Please try again within your home network to initiate the password reset process.
 If you're an external jellyfin user you can't reset your password unless you are part of the LAN. This is done because the reset password process is simple and insecure.
 If you don't care about that and still think that the internet is a happy and safe place here and here are some instructions on how to bypass the security measure.
 For more information also read 1 and 2.
 
 
 Kitty
   New: How to add fonts to kitty.
  Add your fonts to the ~/.local/share/fonts directory
 Check they are available when you run kitty +list-fonts
 Add them to your config:
 
 
font_family      Operator Mono Book
bold_font        Operator Mono Medium
italic_font      Operator Mono Book Italic
bold_italic_font Operator Mono Medium Italic
 feat(kitty#Screen not working on server with sudo): Troubleshoot the Screen not working on server with sudo issue
 Make sure you're using the ssh alias below
 alias ssh="kitty +kitten ssh"
 And then copy the ~/.terminfo into /root
 sudo copy -r ~/.terminfo /root
 
 
 Kodi
   New: Introduce Kodi.
 Kodi is a entertainment center software. It basically converts your device into a smart tv
 
 
 Koel
   New: Introduce Koel.
 koel is a personal music streaming server.
 Note: Use mopidy instead
 There are docker-compose files to host the service. Although they behave a little bit weird
 For example, you need to specify the DB_PORT. It has had several PR to fix it but weren't merged 1, 2.
 The API is not very well documented:
  Here you can see how to authenticate
 Here are the api docs
 
 
 
 Matrix
   New: How to install matrix.
 sudo apt install -y wget apt-transport-https
sudo wget -O /usr/share/keyrings/element-io-archive-keyring.gpg https://packages.element.io/debian/element-io-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/element-io-archive-keyring.gpg] https://packages.element.io/debian/ default main" | sudo tee /etc/apt/sources.list.d/element-io.list
sudo apt update
sudo apt install element-desktop
 
 
 Mediatracker
   New: Introduce MediaTracker.
 MediaTracker is a self hosted media tracker for movies, tv shows, video games, books and audiobooks
 Installation:
 With docker compose:
 version: "3"
services:
  mediatracker:
    container_name: mediatracker
    ports:
      - 7481:7481
    volumes:
      - /home/YOUR_HOME_DIRECTORY/.config/mediatracker/data:/storage
      - assetsVolume:/assets
    environment:
      SERVER_LANG: en
      TMDB_LANG: en
      AUDIBLE_LANG: us
      TZ: Europe/London
    image: bonukai/mediatracker:latest

volumes:
  assetsVolume: null
 If you attach more than one docker network the container becomes unreachable :S.
 Install the jellyfin plugin:
 They created a Jellyfin plugin so that all scrobs are sent automatically to the mediatracker
  Add new Repository in Jellyfin (Dashboard -> Plugins -> Repositories -> +) from url https://raw.githubusercontent.com/bonukai/jellyfin-plugin-mediatracker/main/manifest.json
 Install MediaTracker plugin from Catalogue (Dashboard -> Plugins -> Catalogue)
 
 Some tips on usage:
  Add the shows you want to watch to the watchlist so that it's easier to find them
  When you're ending an episode, click on the episode number on the watchlist element and then rate the episode itself.
 
  You can create public lists to share with the rest of the users, the way to share it though is a bit archaic so far, it's only through the list link, in the interface they won't be able to see it.
 
 
 
  Correction: Update ryot comparison with mediatracker.
 Ryot has a better web design, it also has a jellyfin scrobbler, although it's not yet stable. There are other UI tweaks that is preventing me from migrating to ryot such as the easier media rating and the percentage over five starts rating system.
 
 
 retroarch
   New: Install retroarch instructions.
 To add the stable branch to your system type:
 sudo add-apt-repository ppa:libretro/stable
sudo apt-get update
sudo apt-get install retroarch
 Go to Main Menu/Online Updater and then update everything you can:
  Update Core Info Files
 Update Assets
 Update controller Profiles
 Update Databases
 Update Overlays
 Update GLSL Shaders
 
 
 
 Rocketchat
   New: Introduce Rocketchat integrations.
 Rocket.Chat supports webhooks to integrate tools and services you like into the platform. Webhooks are simple event notifications via HTTP POST. This way, any webhook application can post a message to a Rocket.Chat instance and much more.
 With scripts, you can point any webhook to Rocket.Chat and process the requests to print customized messages, define the username and avatar of the user of the messages and change the channel for sending messages, or you can cancel the request to prevent undesired messages.
 Available integrations:
  Incoming Webhook: Let an external service send a request to Rocket.Chat to be processed.
 Outgoing Webhook: Let Rocket.Chat trigger and optionally send a request to an external service and process the response.
 
 By default, a webhook is designed to post messages only. The message is part of a JSON structure, which has the same format as that of a .
 Incoming webhook script:
 To create a new incoming webhook:
  Navigate to Administration > Workspace > Integrations.
 Click +New at the top right corner.
 Switch to the Incoming tab.
 Turn on the Enabled toggle.
 Name: Enter a name for your webhook. The name is optional; however, providing a name to manage your integrations easily is advisable.
 Post to Channel: Select the channel (or user) where you prefer to receive the alerts. It is possible to override messages.
 Post as: Choose the username that this integration posts as. The user must already exist.
 Alias: Optionally enter a nickname that appears before the username in messages.
 Avatar URL: Enter a link to an image as the avatar URL if you have one. The avatar URL overrides the default avatar.
 Emoji: Enter an emoji optionally to use the emoji as the avatar. Check the emoji cheat sheet
 Turn on the Script Enabled toggle.
 Paste your script inside the Script field (check below for a sample script)
 Save the integration.
 Use the generated Webhook URL to post messages to Rocket.Chat.
 
 The Rocket.Chat integration script should be written in ES2015 / ECMAScript 6. The script requires a global class named Script, which is instantiated only once during the first execution and kept in memory. This class contains a method called process_incoming_request, which is called by your server each time it receives a new request. The process_incoming_request method takes an object as a parameter with the request property and returns an object with a content property containing a valid Rocket.Chat message, or an object with an error property, which is returned as the response to the request in JSON format with a Code 400 status.
 A valid Rocket.Chat message must contain a text field that serves as the body of the message. If you redirect the message to a channel other than the one indicated by the webhook token, you can specify a channel field that accepts room id or, if prefixed with "#" or "@", channel name or user, respectively.
 You can use the console methods to log information to help debug your script. More information about the console can be found here. . To view the logs, navigate to Administration > Workspace > View Logs.
 /* exported Script */
/* globals console, _, s */

/** Global Helpers
 *
 * console - A normal console instance
 * _       - An underscore instance
 * s       - An underscore string instance
 */

class Script {
  /**
   * @params {object} request
   */
  process_incoming_request({ request }) {
    // request.url.hash
    // request.url.search
    // request.url.query
    // request.url.pathname
    // request.url.path
    // request.url_raw
    // request.url_params
    // request.headers
    // request.user._id
    // request.user.name
    // request.user.username
    // request.content_raw
    // request.content

    // console is a global helper to improve debug
    console.log(request.content);

    return {
      content:{
        text: request.content.text,
        icon_emoji: request.content.icon_emoji,
        channel: request.content.channel,
        // "attachments": [{
        //   "color": "#FF0000",
        //   "author_name": "Rocket.Cat",
        //   "author_link": "https://open.rocket.chat/direct/rocket.cat",
        //   "author_icon": "https://open.rocket.chat/avatar/rocket.cat.jpg",
        //   "title": "Rocket.Chat",
        //   "title_link": "https://rocket.chat",
        //   "text": "Rocket.Chat, the best open source chat",
        //   "fields": [{
        //     "title": "Priority",
        //     "value": "High",
        //     "short": false
        //   }],
        //   "image_url": "https://rocket.chat/images/mockup.png",
        //   "thumb_url": "https://rocket.chat/images/mockup.png"
        // }]
       }
    };

    // return {
    //   error: {
    //     success: false,
    //     message: 'Error example'
    //   }
    // };
  }
}
 To test if your integration works, use curl to make a POST request to the generated webhook URL.
 curl -X POST \
  -H 'Content-Type: application/json' \
  --data '{
      "icon_emoji": ":smirk:",
      "text": "Example message"
  }' \
  https://your-webhook-url
 If you want to send the message to another channel or user use the channel argument with @user or #channel. Keep in mind that the user of the integration needs to be part of those channels if they are private.
 curl -X POST \
  -H 'Content-Type: application/json' \
  --data '{
      "icon_emoji": ":smirk:",
      "channel": "#notifications",
      "text": "Example message"
  }' \
  https://your-webhook-url
 If you want to do more complex things uncomment the part of the attachments.
 
 
 sed
  New: Introduce sed snippets.
 
 Tails
   New: Add interesting operations on tails.
  Upgrading a tails USB
 Change the window manager: Don't do it, they say it it will break Tails although I don't understand why
 
 
 
 vdirsyncer
   Correction: Use SHA256 for the verification.
 Now SHA1 is not allowed
 
  New: Suggest more debuggin steps when connecting to google.
 The code has changed and the fix is now different
 
 
 Vim
   New: Configure nvim with lua.
 Nvim moved away from vimscript and now needs to be configured in lua. You can access the config file in ~/.config/nvim/init.lua. It's not created by default so you need to do it yourself.
 In the article it explains how to do the basic configuration with lua:
  Set variables
 Set key bindings
 Set spelling
 Set test runners: With neotest
 Set the buffer and file management: with Telescope.
 Use Plugin managers: it evaluates the different solutions and then explains how to install and use packer
 What is and how to use Treesitter
 Set git integration: Evaluate the different solutions and configure neogit
 How to run lua snippets
 
 And some troubleshooting:
  How to deal with big files and telescope
 
 
  Correction: Update the leader key section.
 There are different opinions on what key to use as the <leader> key. The <space> is the most comfortable as it's always close to your thumbs, and it works well with both hands. Nevertheless, you can only use it in normal mode, because in insert <space><whatever> will be triggered as you write. An alternative is to use ; which is also comfortable (if you use the english key distribution) and you can use it in insert mode.
 If you want to define more than one leader key you can either:
  Change the mapleader many times in your file: As the value of mapleader is used at the moment the mapping is defined, you can indeed change that while plugins are loading. For that, you have to explicitly :runtime the plugins in your ~/.vimrc (and count on the canonical include guard to prevent redefinition later):
 
 
let mapleader = ','
runtime! plugin/NERD_commenter.vim
runtime! ...
let mapleader = '\'
runime! plugin/mark.vim
...
 * Use the keys directly instead of using <leader>
 " editing mappings
nnoremap ,a <something>
nnoremap ,k <something else>
nnoremap ,d <and something else>

" window management mappings
nnoremap gw <something>
nnoremap gb <something else>
 Defining mapleader and/or using <leader> may be useful if you change your mind often on what key to use a leader but it won't be of any use if your mappings are stable.
 
  New: Configure Telescope to follow symbolic links.
 By default symbolic links are not followed either for files or directories, to enable it use
   require('telescope').setup {
    pickers = {
      find_files = {
        follow = true
      }
    }
  }
 
  New: Run a command when opening vim.
 nvim -c ':DiffViewOpen'
 
  New: Update treesitter language definitions.
 To do so you need to run:
 :TSInstall <language>
 To update the parsers run
 :TSUpdate
 
  New: Telescope changes working directory when opening a file.
 In my case was due to a snippet I have to remember the folds:
 vim.cmd[[
  augroup remember_folds
    autocmd!
    autocmd BufWinLeave * silent! mkview
    autocmd BufWinEnter * silent! loadview
  augroup END
]]
 It looks that it had saved a view with the other working directory so when a file was loaded the cwd changed. To solve it I created a new mkview in the correct directory.
 
  New: Concealment.
 Some plugins allow the conceal of some text, for example in orgmode you will only see the text of the description of a link and not the content, making it more pleasant to read. To enable it set in your config:
 -- Conceal links
-- https://github.com/nvim-orgmode/orgmode#links-are-not-concealed
-- Use visual mode to navigate through the hidden text
vim.opt.conceallevel = 2
vim.opt.concealcursor = 'nc'
 Where:
   conceallevel: Determine how text with the "conceal" syntax attribute is shown:
 
  0: Text is shown normally
 
 1: Each block of concealed text is replaced with one character. If the syntax item does not have a custom replacement character defined the character defined in 'listchars' is used (default is a space). It is highlighted with the "Conceal" highlight group.
 2: Concealed text is completely hidden unless it has a custom replacement character defined.
  3: Concealed text is completely hidden.
 
  concealcursor: Sets the modes in which text in the cursor line can also be concealed. When the current mode is listed then concealing happens just like in other lines.
 
 n: Normal mode
 v: Visual mode
 i: Insert mode
 c: Command line editing, for 'incsearch'
 
 A useful value is nc. So long as you are moving around text is concealed, but when starting to insert text or selecting a Visual area the concealed text is displayed, so that you can see what you are doing.
 
  New: Email inside nvim.
 The best looking one is himalaya
  Home
 Nvim plugin
 Source
 
 
 
 yq
   New: Introduce yq.
 yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor. It uses jq like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn't yet support everything jq does - but it does support the most common operations and functions, and more is being added continuously.
 
  New: Find and update items in an array.
 We have an array and we want to update the elements with a particular name.
 Given a sample.yaml file of:
 - name: Foo
  numBuckets: 0
- name: Bar
  numBuckets: 0
 Then yq '(.[] | select(.name == "Foo") | .numBuckets) |= . + 1' sample.yaml will output:
 - name: Foo
  numBuckets: 1
- name: Bar
  numBuckets: 0
 
  New: Iterate over the elements of a query with a bash loop.
 readarray dependencies < <(yq e -o=j -I=0 '.roles[]' requirements.yaml)
for dependency in "${dependencies[@]}"; do
    source="$(echo "$dependency" | yq e '.src' -)"
done
 
 
 Android
 GrapheneOS
   New: Split the screen.
 Go into app switcher, tap on the app icon above the active app and then select "Split top".
 
 
 LibreTube
   New: Introduce Libretube.
 Libretube is an alternative frontend for YouTube, for Android.
 YouTube has an extremely invasive privacy policy which relies on using user data in unethical ways. They store a lot of your personal data - ranging from ideas, music taste, content, political opinions, and much more than you think.
 This project is aimed at improving the users' privacy by being independent from Google and bypassing their data collection.
 Therefore, the app is using the Piped API, which uses proxies to circumvent Google's data collection and includes some other additional features.
 Differences to NewPipe:
 With NewPipe, the extraction is done locally on your phone, and all the requests sent towards YouTube/Google are done directly from the network you're connected to, which doesn't use a middleman server in between. Therefore, Google can still access information such as the user's IP address. Aside from that, subscriptions can only be stored locally.
 LibreTube takes this one step further and proxies all requests via Piped (which uses the NewPipeExtractor). This prevents Google servers from accessing your IP address or any other personal data. Apart from that, Piped allows syncing your subscriptions between LibreTube and Piped, which can be used on desktop too.
 If the NewPipeExtractor breaks, it only requires an update of Piped and not LibreTube itself. Therefore, fixes usually arrive faster than in NewPipe.
 While LibreTube only supports YouTube, NewPipe also allows the use of other platforms like SoundCloud, PeerTube, Bandcamp and media.ccc.de. Both are great clients for watching YouTube videos. It depends on the individual's use case which one fits their needs better.
 Other software that uses Piped:
  Yattee - an alternative frontend for YouTube, for IOS.
 Hyperpipe - an alternative privacy respecting frontend for YouTube Music.
 Musicale - an alternative to YouTube Music, with style.
 ytify - a complementary minimal audio streaming frontend for YouTube.
 PsTube - Watch and download videos without ads on Android, Linux, Windows, iOS, and Mac OSX.
 Piped-Material - A fork of Piped, focusing on better performance and a more usable design.
 ReacTube - Privacy friendly & distraction free Youtube front-end using Piped API.
 
 
 
 HappyCow
   New: Introduce Happycow.
 Happycow is a web application and android app to search vegan restaurants nearby.
 The android app requires google services to work :(.
 
 
 Orgzly
   New: Introduce Orgzly.
 Orgzly is an android application to interact with orgmode files.
 
  New: Avoid the conflicts in the files edited in two places.
 If you use syncthing you may be seeing conflicts in your files. This happens specially if you use the Orgzly widget to add tasks, this is because it doesn't synchronize the files to the directory when using the widget. If you have a file that changes a lot in a device, for example the inbox.org of my mobile, it's interesting to have a specific file that's edited mainly in the mobile, and when you want to edit it elsewhere, you sync as specified below and then process with the editing. Once it's done manually sync the changes in orgzly again. The rest of the files synced to the mobile are for read only reference, so they rarely change.
 If you want to sync reducing the chance of conflicts then:
  Open Orgzly and press Synchronize
 Open Syncthing.
 
 If that's not enough check these automated solutions:
  Orgzly auto syncronisation for sync tools like syncthing
 watch-for-orgzly
 
 Other interesting solutions:
  org-orgzly: Script to parse a chosen org file or files, check if an entry meets required parameters, and if it does, write the entry in a new file located inside the folder you desire to sync with orgzly.
 Git synchronization: I find it more cumbersome than syncthing but maybe it's interesting for you.
 
 
  New: Add new orgzly fork.
 Alternative fork maintained by the community
 
 
 Seedvault
   New: Introduce seedvault.
 Seedvault is an open-source encrypted backup app for inclusion in Android-based operating systems.
 While every smartphone user wants to be prepared with comprehensive data backups in case their phone is lost or stolen, not every Android user wants to entrust their sensitive data to Google's cloud-based storage. By storing data outside Google's reach, and by using client-side encryption to protect all backed-up data, Seedvault offers users maximum data privacy with minimal hassle.
 Seedvault allows Android users to store their phone data without relying on Google's proprietary cloud storage. Users can decide where their phone's backup will be stored, with options ranging from a USB flash drive to a remote self-hosted cloud storage alternative such as NextCloud. Seedvault also offers an Auto-Restore feature: instead of permanently losing all data for an app when it is uninstalled, Seedvault's Auto-Restore will restore all backed-up data for the app upon reinstallation.
 Seedvault protects users' private data by encrypting it on the device with a key known only to the user. Each Seedvault account is protected by client-side encryption (AES/GCM/NoPadding). This encryption is unlockable only with a 12-word randomly-generated key.
 With Seedvault, backups run automatically in the background of the phone's operating system, ensuring that no data will be left behind if the device is lost or stolen. The Seedvault application requires no technical knowledge to operate, and does not require a rooted device.
 In the article you'll also find:
  How to install it
 How to store the backup remotely
 How to restore a backup
 
 
 
 Signal
   New: Add installation steps.
 These instructions only work for 64 bit Debian-based Linux distributions such as Ubuntu, Mint etc.
  Install our official public software signing key
 
 wget -O- https://updates.signal.org/desktop/apt/keys.asc | gpg --dearmor > signal-desktop-keyring.gpg
cat signal-desktop-keyring.gpg | sudo tee -a /usr/share/keyrings/signal-desktop-keyring.gpg > /dev/null
  Add our repository to your list of repositories
 
 echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main' |\
  sudo tee -a /etc/apt/sources.list.d/signal-xenial.list
  Update your package database and install signal
 
 sudo apt update && sudo apt install signal-desktop
 
 
 Arts
 Dancing
 Shag
   New: Add awesome Shag videos.
  Sandy Lewis and Martí Gasol:
 Warsaw Collegiate Shag Festival 2017
 Warsaw Collegiate Shag Festival 2016
 Cherry Moreno and Filip Gorski:
 BCN Shag Festival 2018
 RTSF 2019
 Stephen and Chandrae:
 Minor Swing
 [RTSF 2013](https://yewtu.be/watch?v=uUGsLoZYXb4
 RTSF 2016 with Arnas and Egle
 MBSW 2013
 Warsaw Shag team
 Dragon Swing 2016
 Warsaw Shag Festival 2018
 Warsaw Shag Festival 2017
 
 
 
 Lindy Hop
   New: New Charleston, lindy and solo jazz videos.
 Charleston:
  The DecaVita Sisters:
 Freestyle Lindy Hop & Charleston
 Moby "Honey"
 
 Solo Jazz:
  Pedro Vieira at Little Big Swing Camp 2022
 
 Lindy Hop:
  The DecaVita Sisters:
 Compromise - agreement in the moment
 Lindy hop improv
 
 
  New: Add Hellzapopping videos.
 Short version Long version
 
 
 Maker
 Vial
   New: Introduce Vial.
 Vial is an open-source cross-platform (Windows, Linux and Mac) GUI and a QMK fork for configuring your keyboard in real time.
 Even though you can use a web version you can install it locally through an AppImage
  Download the latest version
 Give it execution permissions
 Add the file somewhere in your $PATH
 
 On linux you need to configure an udev rule.
 For a universal access rule for any device with Vial firmware, run this in your shell while logged in as your user (this will only work with sudo installed):
 export USER_GID=`id -g`; sudo --preserve-env=USER_GID sh -c 'echo "KERNEL==\"hidraw*\", SUBSYSTEM==\"hidraw\", ATTRS{serial}==\"*vial:f64c2b3c*\", MODE=\"0660\", GROUP=\"$USER_GID\", TAG+=\"uaccess\", TAG+=\"udev-acl\"" > /etc/udev/rules.d/99-vial.rules && udevadm control --reload && udevadm trigger'
 This command will automatically create a udev rule and reload the udev system.
 
 
 Video Gaming
 Age of Empires
   New: Add guides on how to play.
  Fast castle boom
 How to play Arena: Hera's guide, Tatoh game in arena
 How to play Hideout
 How to play Blackforest
 Inside the mind of a pro player: Episode 1, Episode 2
 
 
  New: Strategies against civilisations.
 I'm using only the mongols, and so far I've seen/heard from the pros the next strategies:
  Aztecs: Steppe lancers good against eagle warriors
 Heavy scorpions against eagle warriors and skirms
 
 
 Cumans: Scout, if it drops two TCs in feudal, tower rush into archers
 Put initial pressure: Nice initial pressure
 
 
 Incas: Steppe lancers good against eagle warriors
 Heavy scorpions against eagle warriors and skirms
 
 
 Khmer: boom, map control, monks and albadiers
 Mayans: Steppe lancers good against eagle warriors
 Heavy scorpions against eagle warriors and skirms
 
 
 Romans: Hera guide on how to beat them
 
 
 Tartars: heavy scorpions
 Turks: How to defend against them in Arena
 
 
 
 
  New: Nice games.
 Tournaments:
  2023 Masters of Arena 7 Final Tatoh vs Vinchester: Casted by T90
 Pov by Tatoh
 
 
 
 Showmatches:
  Hera vs TheViper | Battlegrounds 3 | BO5
 The Viper VS Tatoh PA7
 
 1vs1 games:
  Hindustanis vs Portuguese | Arabia | Hera vs Yo
 Dravidians vs Turks | African Clearing | Hera vs Yo
 
 
  New: How to fight Vietnamese with Mongols.
 Gain early map control with scouts, then switch into steppe lancers and front siege, finally castle in the face when you clicked to imperial.
  Example Hera vs Mr.Yo in TCI
 
 
  New: Added references.
  Hera channel: Awesome to learn to play
 Hera gameplay channel: Good 1^st person point of view games
 Tatoh channel: Good spanish content.
 
 
 
 Gardening
   New: Fertilizing with manure.
 Manure is one of the best organic fertilizers for plants. It's made by the accumulation of excrements of bats, sea birds and seals and it usually doesn't contain additives or synthetic chemical components.
 This fertilizer is rich in nitrogen, phosphorus and potassium, which are key minerals for the growth of plants. These components help the regeneration of the soil, the enrichment in terms of nutrients and also acts as fungicide preventing plagues.
 Manure is a fertilizer of slow absorption, which means that it's released to the plants in an efficient, controlled and slow pace. That way the plants take the nutrients when they need them.
 The best moment to use it is at spring and depending on the type of plant you should apply it between each month and a half and three months. It's use in winter is not recommended, as it may burn the plant's roots.
 Manure can be obtained in dust or liquid state. The first is perfect to scatter directly over the earth, while the second is better used on plant pots. You don't need to use much, in fact, with just a pair of spoons per pot is enough. Apply it around the base of the plant, avoiding it's touch with leaves, stem or exposed roots, as it may burn them. After you apply them remember to water them often, keep in mind that it's like a heavy greasy sandwich for the plants, and they need water to digest it.
 For my indoor plants I'm going to apply a small dose (one spoon per plant) at the start of Autumn (first days of September), and two spoons at the start of spring (first days of March).
 
 
 Emojis
   New: Create a list of most used emojis.
 ¯\(°_o)/¯

¯\_(ツ)_/¯

(╯°□°)╯ ┻━┻

＼\ ٩( ᐛ )و /／

(✿◠‿◠)

(/ﾟДﾟ)/

(¬º-°)¬

(╥﹏╥)

ᕕ( ᐛ )ᕗ

ʕ•ᴥ•ʔ

( ˘ ³˘)♥

❤
 
  New: Add new emojis.
 (╥_╥)
(*≧▽≦)ﾉｼ))
 
 
 Science
 Math
   New: Add 3Blue1Brown videos.
 3Blue1Brown. Recomended by Nicky Case
 
 
 Other
   Correction: Update introduction.
 The method was described by David Allen in a book with the same name. It's clear that the book is the corner stone of David's business. He is selling his method on every word, some times to the point of tiresome. It's also repeats the same ideas on different parts of the book, I guess that's good in terms of sticking an idea in the people's mind, but if you're already convinced and are trying to sum up the book it's like, hey, I have 90% of the valuable contents of this chapter already in my summary. It's obvious too the context of the writer, that the book was written a while ago and who does it write to. It talks quite often about assistants, bosses of high firm companies he's helped, preferring low-tech physical solutions over digital ones, a lot of references about parenting... If you're able to ignore all the above, it's actually a very good book. The guy has been polishing the method for more than 30 years, and has pretty nice ideas that can change how you manage your life.
 My idea of this summary is to try to extract the useful ideas removing all those old-fashioned capitalist values from it.
 
  New: Guides on processing your inbox.
 Remember to follow the next rules while processing the items:
  Process the top item first: that way you treat each element equally, so the "least" important ones are not left dangling forever in your inbox thus thwarting it's purpose.
 Process one item at a time.
 Never put anything back into “in.”
 
 For each element you need to ask yourself: "What's the next action?".
 
  New: How to clarify your inbox items.
 If you can do something about the element, you need to think which is the next physical, visible activity that would be required to move the situation towards closure. It's tricky, something like "set meeting" won't do because it's not descriptive of physical behaviour. There is still stuff to decide how, when, with whom, if you don't do it now you won't empty your head and the uncertainty will create a psychological gap that will make you procrastinate, so define the next action now. "Decide what to do about X" doesn't work either, you may need to gather more information on the topic, but deciding doesn't take time.
 Once you have the next action, if it can be done in two minutes or less, do it when you first pick the item up. Even if it is not a high-priority one, do it now if you’re ever going to do it at all. The rationale for the two-minute rule is that it’s more or less the point where it starts taking longer to store and track an item than to deal with it the first time it’s in your hands. Two minutes is just a guideline. If you have a long open window of time in which to process your in-tray, you can extend the cutoff for each item to five or ten minutes. If you’ve got to get to the bottom of all your input rapidly, then you may want to shorten the time to one minute, or even thirty seconds, so you can get through everything a little faster.
 There’s nothing you really need to track about your two-minute actions. Just do them. If, however, you take an action and don’t finish the project with that one action, you’ll need to clarify what’s next on it, and manage that according to the same criteria.
 If the next action is going to take longer than two minutes, ask yourself, “Am I the best person to be doing it?” If not, hand it off to the appropriate person, in order of priority:
  Send an e-mail.
 Write a note or an over-note on paper and route it to that person.
 Send it a instant message.
 Add it as an agenda item on a list for your next real-time conversation with that person.
 Talk with her directly, either face-to-face or by phone.
 
 When you hand it off to someone else, and if you care at all whether something happens as a result, you’ll need to track it. Depending on how active you need to be it can go to your Waiting list or to your tickler.
 
  Correction: Deprecate pydo.
 I'm happy with orgmode so far, so I'm not going to continue it's development
 
  New: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress.
 This error can happen for few reasons, but it most commonly occurs when there is an interruption during the upgrade/install process as you already mentioned.
 To fix this one may need to, first rollback to another version, then reinstall or helm upgrade again.
 Try below command to list the available charts:
 helm ls --namespace <namespace>
 You may note that when running that command ,it may not show any columns with information. If that's the case try to check the history of the previous deployment
 helm history <release> --namespace <namespace>
 This provides with information mostly like the original installation was never completed successfully and is pending state something like STATUS: pending-upgrade state.
 To escape from this state, use the rollback command:
 helm rollback <release> <revision> --namespace <namespace>
 revision is optional, but you should try to provide it.
 You may then try to issue your original command again to upgrade or reinstall.
 
  New: Ansible retry a failed job.
 - command: /usr/bin/false
  retries: 3
  delay: 3
  register: result
  until: result.rc == 0
 
  New: Introduce Bookwyrm.
 Bookwyrm is a social network for tracking your reading, talking about books, writing reviews, and discovering what to read next. Federation allows BookWyrm users to join small, trusted communities that can connect with one another, and with other ActivityPub services like Mastodon and Pleroma.
 
  New: Introduce Elastic security.
 Elastic security is a program to protect, investigate, and respond to complex threats by unifying the capabilities of SIEM, endpoint security, and cloud security.
 
  New: Introduce RSS.
 Note: This post is a shameless direct copy of Nicky beautiful post, read it there as it has beautiful illustrations
 What is RSS (Really Simple Syndication)?
 Imagine an open version of Twitter or Facebook News Feed, with no psy-op ads, owned by no oligopoly, manipulated by no algorithm, and all under your full control.
 Imagine a version of the newsletter where you don't have to worry about them selling your email to scammers, labyrinth-like unsubscribe pages, or stuffing your inbox with ever more crap.
 Now imagine this existed and was extremely popular 15 years ago. Then we got suckered by the shiny walled gardens.
 Well, it's time to make like a tree and go back to the future, baby!
 How does RSS work?
 Unlike newsletters where give each publisher your email (and they may abuse that trust), RSS works on a "don't call me, I'll call you" policy.
 An RSS feed is a text file on a website. It's just a bunch of posts – no tracking or "personalization" – like a printed newspaper:
 Then, whatever RSS reader app you use – you can use any app made by anyone – it'll call the websites for the feeds you specifically opted into, no more or less. The websites can't force it in the other direction.
 Your app then shows you your posts in good ol' reverse chronological order. (Some apps let you add extra filters, but unlike social media algorithms, you control 'em.) Apps also make the posts prettier than raw text:
 Really Simple, indeed!
 Cool, how do I get started?
 First, you need a reader app. Such as the minimalist Inoreader, but Feedly is the most popular, and folks use The Old Reader. See this list of readers.
 To add a feed to your app, just paste a link to the blog/site, and your app will automatically find the feed! RSS also lets you follow creators on YouTube, Substack, Medium, and more.
 Tips for using RSS wisely
  Beware the hoarder instinct. No algorithm can save you from hoarding feeds "just in case", then being overwhelmed. The only cure is to ruthlessly Marie Kondo that crap – if a feed doesn't consistently enrich your life, cut it.
 Some feeds only give you the excerpt of a post, with a link to see the full post at their site. Don't follow those: they break you out of the RSS reading experience, and trick you into losing time on their site. (This is a harsh rule: I used to follow Quanta Magazine's feed, but they switched from full-text to excerpts, so I unsubscribed.)
 Don't follow feeds that update more than once a day. Go for daily digests, or better yet, weekly digests.
 
 If RSS Was So Great, Why'd It Die In The First Place
 Well, Google killed Google Reader in 2013, the #1 RSS reader at the time. This was to make way for Google Plus, which failed. The sacrificial lamb was for nothing.
 But Google only did what nearly everyone – including yours truly – did in 2013: leave the open, decentralized Web 1.0 for the shiny new Web 2.0 platforms. Why? Well, it was more fun & convenient.
 But now in 2021, for most of us, social media is very not fun and not convenient. That's why I went back to the future with RSS, and wrote this post encouraging you to do the same!
 (Ok, RSS had two more problems: 1) Getting overwhelmed with feeds. As said above, the only cure is to trim ruthlessly. 2) RSS lets you serve text/link/image ads, but not the creepy user-tracking ads. In 2013 that was the "best" way make money on the web, but these days ad revenue is dying, and subscriptions like Patreon/Substack are thriving.)
 And that's all, folks! Now you know how to escape the attention-draining, empathy-killing, critical-thought-suffocating siren song of the algorithms. And get your inbox less cluttered with newsletters.
 Here's to a renaissance for a kinder, better web. <3
 
  New: Introduce the analysis of life process.
 It's interesting to do analysis at representative moments of the year. It gives it an emotional weight. You can for example use the solstices or my personal version of the solstices:
  Spring analysis (1^st of March): For me the spring is the real start of the year, it's when life explodes after the stillness of the winter. The sun starts to set later enough so that you have light in the afternoons, the climate gets warmer thus inviting you to be more outside, the nature is blooming new leaves and flowers. It is then a moment to build new projects and set the current year on track.
 Summer analysis (1^st of June): I hate heat, so summer is a moment of retreat. Everyone temporarily stop their lives, we go on holidays and all social projects slow their pace. Even the news have even less interesting things to report. It's so hot outside that some of us seek the cold refuge of home or remote holiday places. Days are long and people love to hang out till late, so usually you wake up later, thus having less time to actually do stuff. Even in the moments when you are alone the heat drains your energy to be productive. It is then a moment to relax and gather forces for the next trimester. It's also perfect to develop easy and chill personal projects that have been forgotten in a drawer. Lower your expectations and just flow with what your body asks you.
 Autumn analysis (1^st of September): September it's another key moment for many people. We have it hardcoded in our life since we were children as it was the start of school. People feel energized after the summer holidays and are eager to get back to their lives and stopped projects. You're already 6 months into the year, so it's a good moment to review your year plan and decide how you want to invest your energy reserves.
 Winter analysis (1^st of December): December is the cue that the year is coming to an end. The days grow shorter and colder, they basically invite you to enjoy a cup of tea under a blanket. It is then a good time to get into your cave and do an introspection analysis on the whole year and prepare the ground for the coming year.
 
 We see then that the year is divided in two sets of an expansion trimester and a retreat one. We can use this information to plan our tasks accordingly. In the expansion trimester we could invest more energies in the planning, and in the retreat ones we can do more throughout reviews.
 
  New: How to create a prometheus exporter with python.
 prometheus-client is the official Python client for Prometheus.
 Installation:
 pip install prometheus-client
 Here is a simple script:
 from prometheus_client import start_http_server, Summary
import random
import time

REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')

@REQUEST_TIME.time()
def process_request(t):
    """A dummy function that takes some time."""
    time.sleep(t)

if __name__ == '__main__':
    # Start up the server to expose the metrics.
    start_http_server(8000)
    # Generate some requests.
    while True:
        process_request(random.random())
 Then you can visit http://localhost:8000/ to view the metrics.
 From one easy to use decorator you get:
  request_processing_seconds_count: Number of times this function was called.
 request_processing_seconds_sum: Total amount of time spent in this function.
 
 Prometheus's rate function allows calculation of both requests per second, and latency over time from this data.
 In addition if you're on Linux the process metrics expose CPU, memory and other information about the process for free.
 
 
      2024-11-27

File or Directory Change	Identifier
File or directory has been modified or file or directory link has changed	M
File or directory is present in the older snapshot but not in the more recent snapshot	—
File or directory is present in the more recent snapshot but not in the older snapshot	+
File or directory has been renamed	R

The Blue Book

2025

Hacktivism

Memoria histórica

Conference organisation

Life navigation

Time navigation

Life chores management

Content Management

Knowledge Management

Torrent management

Health

Technology

Coding

DevSecOps

Operating Systems

Hardware

Software tools

Arts

Languages

Science

Brain food

Galas

Videogames

2024

Hacktivism

Feminism

Life Management

Life chores management

Content Management

Torrent management

Health

Coding

Languages