November of 2023

Activism⚑

Collaborating tools⚑

• New: Introduce collaborating tools.

  Collaborating document creation:

  • https://pad.riseup.net
  • https://rustpad.io . Can be self hosted

  Collaborating through terminals:

  • sshx looks promising although I think it uses their servers to do the connection, which is troublesome.

Life Management⚑

Life Review⚑

• New: Introduce the analysis of life process.

  It's interesting to do analysis at representative moments of the year. It gives it an emotional weight. You can for example use the solstices or my personal version of the solstices:

  • Spring analysis (1^st of March): For me the spring is the real start of the year, it's when life explodes after the stillness of the winter. The sun starts to set later enough so that you have light in the afternoons, the climate gets warmer thus inviting you to be more outside, the nature is blooming new leaves and flowers. It is then a moment to build new projects and set the current year on track.
  • Summer analysis (1^st of June): I hate heat, so summer is a moment of retreat. Everyone temporarily stop their lives, we go on holidays and all social projects slow their pace. Even the news have even less interesting things to report. It's so hot outside that some of us seek the cold refuge of home or remote holiday places. Days are long and people love to hang out till late, so usually you wake up later, thus having less time to actually do stuff. Even in the moments when you are alone the heat drains your energy to be productive. It is then a moment to relax and gather forces for the next trimester. It's also perfect to develop easy and chill personal projects that have been forgotten in a drawer. Lower your expectations and just flow with what your body asks you.
  • Autumn analysis (1^st of September): September it's another key moment for many people. We have it hardcoded in our life since we were children as it was the start of school. People feel energized after the summer holidays and are eager to get back to their lives and stopped projects. You're already 6 months into the year, so it's a good moment to review your year plan and decide how you want to invest your energy reserves.
  • Winter analysis (1^st of December): December is the cue that the year is coming to an end. The days grow shorter and colder, they basically invite you to enjoy a cup of tea under a blanket. It is then a good time to get into your cave and do an introspection analysis on the whole year and prepare the ground for the coming year.

  We see then that the year is divided in two sets of an expansion trimester and a retreat one. We can use this information to plan our tasks accordingly. In the expansion trimester we could invest more energies in the planning, and in the retreat ones we can do more throughout reviews.

Life planning⚑

• New: Introduce the month planning process.

  The objectives of the month plan are:

  • Define the month objectives according to the trimester plan and the insights gathered in the past month review.
  • Make your backlog and todo list match the month objectives
  • Define the philosophical topics to address
  • Define the topics to learn
  • Define the are of habits to incorporate?
  • Define the checks you want to do at the end of the month.
  • Plan when is it going to be the next review.

  It's interesting to do the plannings on meaningful days such as the first one of the month. Usually we don't have enough flexibility in our life to do it exactly that day, so schedule it the closest you can to that date. It's a good idea to do both the review and the planning on the same day.

  We'll divide the planning process in these phases:

  • Prepare
  • Clarify your state
  • Decide the month objectives

  Prepare:

  It's important that you prepare your environment for the planning. You need to be present and fully focused on the process itself. To do so you can:

  • Make sure you don't get interrupted:
    • Check your task manager tools to make sure that you don't have anything urgent to address in the next hour.
    • Disable all notifications
  • Set your analysis environment:
    • Put on the music that helps you get in the zone.
    • Get all the things you may need for the review:
      • The checklist that defines the process of your planning (this document in my case).
      • Somewhere to write down the insights.
      • Your task manager system
      • Your habit manager system
      • Your Objective list.
      • Your Thinking list.
      • Your Reading list.
    • Remove from your environment everything else that may distract you

  Clarify your state:

  To be able to make a good decision on your month's path you need to sort out which is your current state. To do so:

  • Clean your inbox: Refile each item until it's empty
  • Clean your todo: Review each todo element by deciding if they should still be in the todo. If they do and they belong to a month objective, add it. If they don't need to be in the todo, refile it.
  • Clean your someday: Review each relevant someday element (not the ones that are archive at greater levels than month) and decide if they should be refiled elsewhere and if they are part of a month objective that should be dealt with this month.
  • Adress each of the trimester objectives by creating month objectives that get you closer to the desired objective.

  Decide the next steps:

  For each of your month objectives:

  • Decide wheter it makes sense to address it this month. If not, archive it
  • Create a clear plan of action for this month on that objective
  • Tweak your things to think about list.
  • Tweak your reading list.
  • Tweak your habit manager system.

Task Management⚑

Org Mode⚑

• New: The orgmode repository file organization.

  How to structure the different orgmode files is something that has always confused me, each one does it's own way, and there are no good posts on why one structure is better than other, people just state what they do.

  I've started with a typical gtd structure with a directory for the todo another for the calendar then another for the references. In the todo I had a file for personal stuff, another for each of my work clients, and the someday.org. Soon making the internal links was cumbersome so I decided to merge the personal todo.org and the someday.org into the same file and use folds to hide uninteresting parts of the file. The reality is that I feel that orgmode is less responsive and that I often feel lost in the file.

  I'm now more into the idea of having files per project in a flat structure and use an index.org file to give it some sense in the same way I do with the mkdocs repositories. Then I'd use internal links in the todo.org file to organize the priorities of what to do next.

  Benefits:

  • As we're using a flat structure at file level, the links between the files are less cumbersome file:./project.org::*heading. We only need to have unique easy to remember names for the files, instead of having to think on which directory was the file I want to make the link to. The all in one file structure makes links even easier, just *heading, but the disadvantages make it not worth it.
  • You have the liberty to have a generic link like Work on project or if you want to fine grain it, link the specific task of the project
  • The todo file will get smaller.
  • It has been the natural evolution of other knowledge repositories such as blue

  Cons:

  • Filenames must be unique. It hasn't been a problem in blue.
  • Blue won't be flattened into Vida as it's it's own knowledge repository

• New: Syncronize orgmode repositories.

  I use orgmode both at the laptop and the mobile, I want to syncronize some files between both with the next requisites:

  • The files should be available on the devices when I'm not at home
  • The synchronization will be done only on the local network
  • The synchronization mechanism will only be able to see the files that need to be synched.
  • Different files can be synced to different devices. If I have three devices (laptop, mobile, tablet) I want to sync all mobile files to the laptop but just some to the tablet).

  Right now I'm already using syncthing to sync files between the mobile and my server, so it's tempting to use it also to solve this issue. So the first approach is to spawn a syncthing docker at the laptop that connects with the server to sync the files whenever I'm at home.

  I've investigated the next options:

  • Mount the whole orgmode repository with syncthing
  • Mount a specific directory to sync
  • Use the org-orgzly script

Knowledge Management⚑

Anki⚑

• New: What to do with unneeded cards.

  You have three options:

  • Suspend: It stops it from showing up permanently until you reactivate it through the browser.
  • Bury: Just delays it until the next day.
  • Delete: It deletes it forever.

  Unless you're certain that you are not longer going to need it, suspend it.

• New: Configure self hosted synchronization.

  Explain how to install anki-sync-server and how to configure Ankidroid and Anki. In the end I dropped this path and used Ankidroid alone with syncthing as I didn't need to interact with the decks from the computer. Also the ecosystem of synchronization in Anki at 2023-11-10 is confusing as there are many servers available, not all are compatible with the clients and Anki itself has released it's own so some of the community ones will eventually die.

Coding⚑

Languages⚑

Bash snippets⚑

• New: Loop through a list of files found by find.

  For simple loops use the find -exec syntax:

  find . -name '*.txt' -exec process {} \;
  

  For more complex loops use a while read construct:

  find . -name "*.txt" -print0 | while read -r -d $'\0' file
  do
      …code using "$file"
  done
  

  The loop will execute while the find command is executing. Plus, this command will work even if a file name is returned with whitespace in it. And, you won't overflow your command line buffer.

  The -print0 will use the NULL as a file separator instead of a newline and the -d $'\0' will use NULL as the separator while reading.

  How not to do it:

  If you try to run the next snippet:

  for file in $(find . -name "*.txt")
  do
      …code using "$file"
  done
  

  You'll get the next shellcheck warning:

  SC2044: For loops over find output are fragile. Use find -exec or a while read loop.
  

  You should not do this because:

  Three reasons:

  • For the for loop to even start, the find must run to completion.
  • If a file name has any whitespace (including space, tab or newline) in it, it will be treated as two separate names.
  • Although now unlikely, you can overrun your command line buffer. Imagine if your command line buffer holds 32KB, and your for loop returns 40KB of text. That last 8KB will be dropped right off your for loop and you'll never know it.

pytelegrambotapi⚑

• New: Introduce pytelegrambotapi.

  pyTelegramBotAPI is an synchronous and asynchronous implementation of the Telegram Bot API.

  Installation:

  pip install pyTelegramBotAPI
  

• New: Create your bot.

  Use the /newbot command to create a new bot. @BotFather will ask you for a name and username, then generate an authentication token for your new bot.

  • The name of your bot is displayed in contact details and elsewhere.
  • The username is a short name, used in search, mentions and t.me links. Usernames are 5-32 characters long and not case sensitive – but may only include Latin characters, numbers, and underscores. Your bot's username must end in 'bot’, like tetris_bot or TetrisBot.
  • The token is a string, like 110201543:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsaw, which is required to authorize the bot and send requests to the Bot API. Keep your token secure and store it safely, it can be used by anyone to control your bot.

  To edit your bot, you have the next available commands:

  • /setname: change your bot's name.
  • /setdescription: change the bot's description (short text up to 512 characters). Users will see this text at the beginning of the conversation with the bot, titled 'What can this bot do?'.
  • /setabouttext: change the bot's about info, a shorter text up to 120 characters. Users will see this text on the bot's profile page. When they share your bot with someone, this text is sent together with the link.
  • /setuserpic: change the bot's profile picture.
  • /setcommands: change the list of commands supported by your bot. Users will see these commands as suggestions when they type / in the chat with your bot. See commands for more info.
  • /setdomain: link a website domain to your bot. See the login widget section.
  • /deletebot: delete your bot and free its username. Cannot be undone.

• New: Synchronous TeleBot.

  import telebot
  
  API_TOKEN = '<api_token>'
  
  bot = telebot.TeleBot(API_TOKEN)
  
  @bot.message_handler(commands=['help', 'start'])
  def send_welcome(message):
      bot.reply_to(message, """\
  Hi there, I am EchoBot.
  I am here to echo your kind words back to you. Just say anything nice and I'll say the exact same thing to you!\
  """)
  
  @bot.message_handler(func=lambda message: True)
  def echo_message(message):
      bot.reply_to(message, message.text)
  
  bot.infinity_polling()
  

• New: Asynchronous TeleBot.

  from telebot.async_telebot import AsyncTeleBot
  bot = AsyncTeleBot('TOKEN')
  
  @bot.message_handler(commands=['help', 'start'])
  async def send_welcome(message):
      await bot.reply_to(message, """\
  Hi there, I am EchoBot.
  I am here to echo your kind words back to you. Just say anything nice and I'll say the exact same thing to you!\
  """)
  
  @bot.message_handler(func=lambda message: True)
  async def echo_message(message):
      await bot.reply_to(message, message.text)
  
  import asyncio
  asyncio.run(bot.polling())
  

Configure Docker to host the application⚑

• New: Dockerize a PDM application.

  It is possible to use PDM in a multi-stage Dockerfile to first install the project and dependencies into __pypackages__ and then copy this folder into the final stage, adding it to PYTHONPATH.

  FROM python:3.11-slim-bookworm AS builder
  
  RUN pip install pdm
  
  COPY pyproject.toml pdm.lock README.md /project/
  COPY src/ /project/src
  
  WORKDIR /project
  RUN mkdir __pypackages__ && pdm sync --prod --no-editable
  
  FROM python:3.11-slim-bookworm
  
  ENV PYTHONPATH=/project/pkgs
  COPY --from=builder /project/__pypackages__/3.11/lib /project/pkgs
  
  COPY --from=builder /project/__pypackages__/3.11/bin/* /bin/
  
  CMD ["python", "-m", "project"]
  

Pytest⚑

• New: Stop pytest right at the start if condition not met.

  Use the pytest_configure initialization hook.

  In your global conftest.py:

  import requests
  import pytest
  
  def pytest_configure(config):
      try:
          requests.get(f'http://localhost:9200')
      except requests.exceptions.ConnectionError:
          msg = 'FATAL. Connection refused: ES does not appear to be installed as a service (localhost port 9200)'
          pytest.exit(msg)
  

  • Note that the single argument of pytest_configure has to be named config.
  • Using pytest.exit makes it look nicer.

• New: Introduce pytest-xprocess.

  pytest-xprocess is a pytest plugin for managing external processes across test runs.

  Installation:

  pip install pytest-xprocess
  

  Usage:

  Define your process fixture in conftest.py:

  import pytest
  from xprocess import ProcessStarter
  
  @pytest.fixture
  def myserver(xprocess):
      class Starter(ProcessStarter):
          # startup pattern
          pattern = "[Ss]erver has started!"
  
          # command to start process
          args = ['command', 'arg1', 'arg2']
  
      # ensure process is running and return its logfile
      logfile = xprocess.ensure("myserver", Starter)
  
      conn = # create a connection or url/port info to the server
      yield conn
  
      # clean up whole process tree afterwards
      xprocess.getinfo("myserver").terminate()
  

  Now you can use this fixture in any test functions where myserver needs to be up and xprocess will take care of it for you.

  Matching process output with pattern:

  In order to detect that your process is ready to answer queries, pytest-xprocess allows the user to provide a string pattern by setting the class variable pattern in the Starter class. pattern will be waited for in the process logfile for a maximum time defined by timeout before timing out in case the provided pattern is not matched.

  It’s important to note that pattern is a regular expression and will be matched using python re.search.

  Controlling Startup Wait Time with timeout:

  Some processes naturally take longer to start than others. By default, pytest-xprocess will wait for a maximum of 120 seconds for a given process to start before raising a TimeoutError. Changing this value may be useful, for example, when the user knows that a given process would never take longer than a known amount of time to start under normal circunstancies, so if it does go over this known upper boundary, that means something is wrong and the waiting process must be interrupted. The maximum wait time can be controlled through the class variable timeout.

     @pytest.fixture
     def myserver(xprocess):
         class Starter(ProcessStarter):
             # will wait for 10 seconds before timing out
             timeout = 10
  

  Passing command line arguments to your process with args:

  In order to start a process, pytest-xprocess must be given a command to be passed into the subprocess.Popen constructor. Any arguments passed to the process command can also be passed using args. As an example, if I usually use the following command to start a given process:

  $> myproc -name "bacon" -cores 4 <destdir>
  

  That would look like:

  args = ['myproc', '-name', '"bacon"', '-cores', 4, '<destdir>']
  

  when using args in pytest-xprocess to start the same process.

  @pytest.fixture
  def myserver(xprocess):
      class Starter(ProcessStarter):
          # will pass "$> myproc -name "bacon" -cores 4 <destdir>"  to the
          # subprocess.Popen constructor so the process can be started with
          # the given arguments
          args = ['myproc', '-name', '"bacon"', '-cores', 4, '<destdir>']
  
          # ...
  

Python Snippets⚑

• New: Configure the logging of a program to look nice.

  def load_logger(verbose: bool = False) -> None:  # pragma no cover
      """Configure the Logging logger.
  
      Args:
          verbose: Set the logging level to Debug.
      """
      logging.addLevelName(logging.INFO, "\033[36mINFO\033[0m")
      logging.addLevelName(logging.ERROR, "\033[31mERROR\033[0m")
      logging.addLevelName(logging.DEBUG, "\033[32mDEBUG\033[0m")
      logging.addLevelName(logging.WARNING, "\033[33mWARNING\033[0m")
  
      if verbose:
          logging.basicConfig(
              format="%(asctime)s %(levelname)s %(name)s: %(message)s",
              stream=sys.stderr,
              level=logging.DEBUG,
              datefmt="%Y-%m-%d %H:%M:%S",
          )
          telebot.logger.setLevel(logging.DEBUG)  # Outputs debug messages to console.
      else:
          logging.basicConfig(
              stream=sys.stderr, level=logging.INFO, format="%(levelname)s: %(message)s"
          )
  

• New: Get the modified time of a file with Pathlib.

  file_ = Path('/to/some/file')
  file_.stat().st_mtime
  

  You can also access:

  • Created time: with st_ctime
  • Accessed time: with st_atime

  They are timestamps, so if you want to compare it with a datetime object use the timestamp method:

  assert datetime.now().timestamp - file_.stat().st_mtime < 60
  

• New: Show the date in the logging module traces.

  To display the date and time of an event, you would place %(asctime)s in your format string:

  import logging
  logging.basicConfig(format='%(asctime)s %(message)s')
  logging.warning('is when this event was logged.')
  

• New: Remove html url characters.

  To transform an URL string into normal string, for example replacing %20 with space use:

  >>> from urllib.parse import unquote
  >>> print(unquote("%CE%B1%CE%BB%20"))
  αλ
  

mkdocstrings⚑

• Correction: Correct the watch directive.

  watch is a list of directories to watch while serving the documentation. So if any file is changed in those directories, the documentation is rebuilt.

Python Telegram⚑

• New: Analyze the different python libraries to interact with telegram.

  There are two ways to interact with Telegram through python:

  • Client libraries
  • Bot libraries

  Client libraries:

  Client libraries use your account to interact with Telegram itself through a developer API token.

  The most popular to use is Telethon.

  Bot libraries:

  Telegram lists many libraries to interact with the bot API, the most interesting are:

  • python-telegram-bot
  • pyTelegramBotAPI
  • aiogram

  If there comes a moment when we have to create the messages ourselves, telegram-text may be an interesting library to check.

  python-telegram-bot:

  Pros:

  • Popular: 23k stars, 4.9k forks
  • Maintained: last commit 3 days ago
  • They have a developers community to get help in this telegram group
  • I like how they try to minimize third party dependencies, and how you can install the complements if you need them
  • Built on top of asyncio
  • Nice docs
  • Fully supports the Telegram bot API
  • Has many examples

  Cons:

  • Interface is a little verbose and complicated at a first look
  • Only to be run in a single thread (not a problem)

  References:

  • Package documentation is the technical reference for python-telegram-bot. It contains descriptions of all available classes, modules, methods and arguments as well as the changelog.
  • Wiki is home to number of more elaborate introductions of the different features of python-telegram-bot and other useful resources that go beyond the technical documentation.
  • Examples section contains several examples that showcase the different features of both the Bot API and python-telegram-bot
  • Source

  pyTelegramBotAPI:

  Pros:

  • Popular: 7.1k stars, 1.8k forks
  • Maintained: last commit 3 weeks ago
  • Both sync and async
  • Nicer interface with decorators and simpler setup
  • They have an example on how to split long messages
  • Nice docs on how to test
  • They have a developers community to get help in this telegram group
  • Fully supports the Telegram bot API
  • Has examples

  Cons:

  • Uses lambdas inside the decorators, I don't know why it does it.
  • The docs are not as throughout as python-telegram-bot one.

  References:

  • Documentation
  • Source
  • Async Examples

  aiogram:

  Pros:

  • Popular: 3.8k stars, 717k forks
  • Maintained: last commit 4 days ago
  • Async support
  • They have a developers community to get help in this telegram group
  • Has type hints
  • Cleaner interface than python-telegram-bot
  • Fully supports the Telegram bot API
  • Has examples

  Cons:

  • Less popular than python-telegram-bot
  • Docs are written at a developer level, difficult initial barrier to understand how to use it.

  References:

  • Documentation
  • Source
  • Examples

  Conclusion:

  Even if python-telegram-bot is the most popular and with the best docs, I prefer one of the others due to the easier interface. aiograms documentation is kind of crap, and as it's the first time I make a bot I'd rather have somewhere good to look at.

  So I'd say to go first with pyTelegramBotAPI and if it doesn't go well, fall back to python-telegram-bot.

Generic Coding Practices⚑

How to code⚑

• New: Personal evolution on how I code.

  Over the years I've tried different ways of developing my code:

  • Mindless coding: write code as you need to make it work, with no tests, documentation or any quality measure.
  • TDD.
  • Try to abstract everything to minimize the duplication of code between projects.

  Each has it's advantages and disadvantages. After trying them all and given that right now I only have short spikes of energy and time to invest in coding my plan is to:

  • Make the minimum effort to design the minimum program able to solve the problem at hand. This design will be represented in an orgmode task.
  • Write the minimum code to make it work without thinking of tests or generalization, but with the domain driven design concepts so the code remains flexible and maintainable.
  • Once it's working see if I have time to improve it:
  • Create the tests to cover the critical functionality (no more 100% coverage).
  • If I need to make a package or the program evolves into something complex I'd use this scaffold template.

  Once the spike is over I'll wait for a new spike to come either because I have time or because something breaks and I need to fix it.

DevOps⚑

Infrastructure as Code⚑

Ansible Snippets⚑

• New: Ansible lint skip some rules.

  Add a .ansible-lint-ignore file with a line per rule to ignore with the syntax path/to/file rule_to_ignore.

Gitea⚑

• New: Run jobs if other jobs failed.

  This is useful to send notifications if any of the jobs failed.

  Right now you can't run a job if other jobs fail, all you can do is add a last step on each workflow to do the notification on failure:

  - name: Send mail
      if: failure()
      uses: https://github.com/dawidd6/action-send-mail@v3
      with:
          to: ${{ secrets.MAIL_TO }}
          from: Gitea <gitea@hostname>
          subject: ${{ gitea.repository }} ${{gitea.workflow}} ${{ job.status }}
          priority: high
          convert_markdown: true
          html_body: |
              ### Job ${{ job.status }}
  
              ${{ github.repository }}: [${{ github.ref }}@${{ github.sha }}](${{ github.server_url }}/${{ github.repository }}/actions)
  

Infrastructure Solutions⚑

Velero⚑

• New: Create a backup.

  If you already have schedules select the one you want to use:

  velero schedules get
  

  Then create the backup with:

  velero backup create --from-schedule selected-schedule
  

  You can see the other options to create backups in velero backup create --help

Tools⚑

• Correction: Recommend rke2 over k3s.

  A friend told me that it works better.

Storage⚑

OpenZFS storage planning⚑

• New: Analyze the Exos X18 of 16TB disk.

  Specs	IronWolf	IronWolf Pro	Exos 7E8 8TB	Exos 7E10 8TB	Exos X18 16TB
  Technology	CMR	CMR	CMR	SMR	CMR
  Bays	1-8	1-24	?	?	?
  Capacity	1-12TB	2-20TB	8TB	8TB	16 TB
  RPM	5,400 RPM (3-6TB)	7200 RPM	7200 RPM	7200 RPM	7200 RPM
  RPM	5,900 RPM (1-3TB)	7200 RPM	7200 RPM	7200 RPM	7200 RPM
  RPM	7,200 RPM (8-12TB)	7200 RPM	7200 RPM	7200 RPM	7200 RPM
  Speed	180MB/s (1-12TB)	214-260MB/s (4-18TB)	249 MB/s	255 MB/s	258 MB/s
  Cache	64MB (1-4TB)	256 MB	256 MB	256 MB	256 MB
  Cache	256MB (3-12TB)	256 MB	256 MB	256 MB	256 MB
  Power Consumption	10.1 W	10.1 W	12.81 W	11.03 W	9.31 W
  Power Consumption Rest	7.8 W	7.8 W	7.64 W	7.06 W	5.08 W
  Workload	180TB/yr	300TB/yr	550TB/yr	550TB/yr	550TB/yr
  MTBF	1 million	1 million	2 millions	2 millions	2.5 millions
  Warranty	3 years	5 years	5 years	5 years	5 years
  Price	From $60 (2022)	From $83 (2022)	249$ (2022)	249$ (2022)	249$ (2023)

OpenZFS⚑

• New: Remove all snapshots of a dataset.

  zfs list -t snapshot -o name path/to/dataset | tail -n+2 | tac | xargs -n 1 zfs destroy -r
  

Monitoring⚑

Loki⚑

• New: How to install loki.

  There are many ways to install Loki, we're going to do it using docker-compose taking their example as a starting point and complementing our already existent grafana docker-compose.

  It makes use of the environment variables to configure Loki, that's why we have the -config.expand-env=true flag in the command line launch.

  In the grafana datasources directory add loki.yaml:

  ---
  apiVersion: 1
  
  datasources:
    - name: Loki
      type: loki
      access: proxy
      orgId: 1
      url: http://loki:3100
      basicAuth: false
      isDefault: true
      version: 1
      editable: false
  

  Storage configuration:

  Unlike other logging systems, Grafana Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.

  Loki 2.0 brings an index mechanism named ‘boltdb-shipper’ and is what we now call Single Store. This type only requires one store, the object store, for both the index and chunks.

  Loki 2.8 adds TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki as it improves query performance, reduces TCO and has the same feature parity as “boltdb-shipper”.

SIEM⚑

• New: Add Wazuh SIEM.

  Wazuh

Operating Systems⚑

Linux⚑

Linux Snippets⚑

• New: Accept new ssh keys by default.

  While common wisdom is not to disable host key checking, there is a built-in option in SSH itself to do this. It is relatively unknown, since it's new (added in Openssh 6.5).

  This is done with -o StrictHostKeyChecking=accept-new. Or if you want to use it for all hosts you can add the next lines to your ~/.ssh/config:

  Host *
    StrictHostKeyChecking accept-new
  

  WARNING: use this only if you absolutely trust the IP\hostname you are going to SSH to:

  ssh -o StrictHostKeyChecking=accept-new mynewserver.example.com
  

  Note, StrictHostKeyChecking=no will add the public key to ~/.ssh/known_hosts even if the key was changed. accept-new is only for new hosts. From the man page:

  If this flag is set to “accept-new” then ssh will automatically add new host keys to the user known hosts files, but will not permit connections to hosts with changed host keys. If this flag is set to “no” or “off”, ssh will automatically add new host keys to the user known hosts files and allow connections to hosts with changed hostkeys to proceed, subject to some restrictions. If this flag is set to ask (the default), new host keys will be added to the user known host files only after the user has confirmed that is what they really want to do, and ssh will refuse to connect to hosts whose host key has changed. The host keys of known hosts will be verified automatically in all cases.

• New: Do not add trailing / to ls.

  Probably, your ls is aliased or defined as a function in your config files.

  Use the full path to ls like:

  /bin/ls /var/lib/mysql/
  

• New: Convert png to svg.

  Inkscape has got an awesome auto-tracing tool.

  • Install Inkscape using sudo apt-get install inkscape
  • Import your image
  • Select your image
  • From the menu bar, select Path > Trace Bitmap Item
  • Adjust the tracing parameters as needed
  • Save as svg

  Check their tracing tutorial for more information.

  Once you are comfortable with the tracing options. You can automate it by using CLI of Inkscape.

• New: Redirect stdout and stderr of a cron job to a file.

  */1 * * * * /home/ranveer/vimbackup.sh >> /home/ranveer/vimbackup.log 2>&1
  

• New: Error when unmounting a device Target is busy.

  • Check the processes that are using the mountpoint with lsof /path/to/mountpoint
  • Kill those processes
  • Try the umount again

  If that fails, you can use umount -l.

Rtorrent⚑

• Correction: Deprecate it in favour of qbittorrent.

  Use qbittorrent instead.

Rocketchat⚑

• New: Introduce Rocketchat integrations.

  Rocket.Chat supports webhooks to integrate tools and services you like into the platform. Webhooks are simple event notifications via HTTP POST. This way, any webhook application can post a message to a Rocket.Chat instance and much more.

  With scripts, you can point any webhook to Rocket.Chat and process the requests to print customized messages, define the username and avatar of the user of the messages and change the channel for sending messages, or you can cancel the request to prevent undesired messages.

  Available integrations:

  • Incoming Webhook: Let an external service send a request to Rocket.Chat to be processed.
  • Outgoing Webhook: Let Rocket.Chat trigger and optionally send a request to an external service and process the response.

  By default, a webhook is designed to post messages only. The message is part of a JSON structure, which has the same format as that of a .

  Incoming webhook script:

  To create a new incoming webhook:

  • Navigate to Administration > Workspace > Integrations.
  • Click +New at the top right corner.
  • Switch to the Incoming tab.
  • Turn on the Enabled toggle.
  • Name: Enter a name for your webhook. The name is optional; however, providing a name to manage your integrations easily is advisable.
  • Post to Channel: Select the channel (or user) where you prefer to receive the alerts. It is possible to override messages.
  • Post as: Choose the username that this integration posts as. The user must already exist.
  • Alias: Optionally enter a nickname that appears before the username in messages.
  • Avatar URL: Enter a link to an image as the avatar URL if you have one. The avatar URL overrides the default avatar.
  • Emoji: Enter an emoji optionally to use the emoji as the avatar. Check the emoji cheat sheet
  • Turn on the Script Enabled toggle.
  • Paste your script inside the Script field (check below for a sample script)
  • Save the integration.
  • Use the generated Webhook URL to post messages to Rocket.Chat.

  The Rocket.Chat integration script should be written in ES2015 / ECMAScript 6. The script requires a global class named Script, which is instantiated only once during the first execution and kept in memory. This class contains a method called process_incoming_request, which is called by your server each time it receives a new request. The process_incoming_request method takes an object as a parameter with the request property and returns an object with a content property containing a valid Rocket.Chat message, or an object with an error property, which is returned as the response to the request in JSON format with a Code 400 status.

  A valid Rocket.Chat message must contain a text field that serves as the body of the message. If you redirect the message to a channel other than the one indicated by the webhook token, you can specify a channel field that accepts room id or, if prefixed with "#" or "@", channel name or user, respectively.

  You can use the console methods to log information to help debug your script. More information about the console can be found here. . To view the logs, navigate to Administration > Workspace > View Logs.

  /* exported Script */
  /* globals console, _, s */
  
  /** Global Helpers
   *
   * console - A normal console instance
   * _       - An underscore instance
   * s       - An underscore string instance
   */
  
  class Script {
    /**
     * @params {object} request
     */
    process_incoming_request({ request }) {
      // request.url.hash
      // request.url.search
      // request.url.query
      // request.url.pathname
      // request.url.path
      // request.url_raw
      // request.url_params
      // request.headers
      // request.user._id
      // request.user.name
      // request.user.username
      // request.content_raw
      // request.content
  
      // console is a global helper to improve debug
      console.log(request.content);
  
      return {
        content:{
          text: request.content.text,
          icon_emoji: request.content.icon_emoji,
          channel: request.content.channel,
          // "attachments": [{
          //   "color": "#FF0000",
          //   "author_name": "Rocket.Cat",
          //   "author_link": "https://open.rocket.chat/direct/rocket.cat",
          //   "author_icon": "https://open.rocket.chat/avatar/rocket.cat.jpg",
          //   "title": "Rocket.Chat",
          //   "title_link": "https://rocket.chat",
          //   "text": "Rocket.Chat, the best open source chat",
          //   "fields": [{
          //     "title": "Priority",
          //     "value": "High",
          //     "short": false
          //   }],
          //   "image_url": "https://rocket.chat/images/mockup.png",
          //   "thumb_url": "https://rocket.chat/images/mockup.png"
          // }]
         }
      };
  
      // return {
      //   error: {
      //     success: false,
      //     message: 'Error example'
      //   }
      // };
    }
  }
  

  To test if your integration works, use curl to make a POST request to the generated webhook URL.

  curl -X POST \
    -H 'Content-Type: application/json' \
    --data '{
        "icon_emoji": ":smirk:",
        "text": "Example message"
    }' \
    https://your-webhook-url
  

  If you want to send the message to another channel or user use the channel argument with @user or #channel. Keep in mind that the user of the integration needs to be part of those channels if they are private.

  curl -X POST \
    -H 'Content-Type: application/json' \
    --data '{
        "icon_emoji": ":smirk:",
        "channel": "#notifications",
        "text": "Example message"
    }' \
    https://your-webhook-url
  

  If you want to do more complex things uncomment the part of the attachments.

Tails⚑

• New: Add interesting operations on tails.

  • Upgrading a tails USB
  • Change the window manager: Don't do it, they say it it will break Tails although I don't understand why

Vim⚑

• New: Email inside nvim.

  The best looking one is himalaya

  • Home
  • Nvim plugin
  • Source

Android⚑

GrapheneOS⚑

• New: Split the screen.

  Go into app switcher, tap on the app icon above the active app and then select "Split top".

Orgzly⚑

• New: Avoid the conflicts in the files edited in two places.

  If you use syncthing you may be seeing conflicts in your files. This happens specially if you use the Orgzly widget to add tasks, this is because it doesn't synchronize the files to the directory when using the widget. If you have a file that changes a lot in a device, for example the inbox.org of my mobile, it's interesting to have a specific file that's edited mainly in the mobile, and when you want to edit it elsewhere, you sync as specified below and then process with the editing. Once it's done manually sync the changes in orgzly again. The rest of the files synced to the mobile are for read only reference, so they rarely change.

  If you want to sync reducing the chance of conflicts then:

  • Open Orgzly and press Synchronize
  • Open Syncthing.

  If that's not enough check these automated solutions:

  • Orgzly auto syncronisation for sync tools like syncthing
  • watch-for-orgzly

  Other interesting solutions:

  • org-orgzly: Script to parse a chosen org file or files, check if an entry meets required parameters, and if it does, write the entry in a new file located inside the folder you desire to sync with orgzly.
  • Git synchronization: I find it more cumbersome than syncthing but maybe it's interesting for you.

• New: Add new orgzly fork.

  Alternative fork maintained by the community

Arts⚑

Emojis⚑

• New: Create a list of most used emojis.

  ¯\(°_o)/¯
  
  ¯\_(ツ)_/¯
  
  (╯°□°)╯ ┻━┻
  
  ＼\ ٩( ᐛ )و /／
  
  (✿◠‿◠)
  
  (/ﾟДﾟ)/
  
  (¬º-°)¬
  
  (╥﹏╥)
  
  ᕕ( ᐛ )ᕗ
  
  ʕ•ᴥ•ʔ
  
  ( ˘ ³˘)♥
  
  ❤
  

• New: Add new emojis.

  (╥_╥)
  (*≧▽≦)ﾉｼ))
  

Other⚑

• New: How to create a prometheus exporter with python.

  prometheus-client is the official Python client for Prometheus.

  Installation:

  pip install prometheus-client
  

  Here is a simple script:

  from prometheus_client import start_http_server, Summary
  import random
  import time
  
  REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
  
  @REQUEST_TIME.time()
  def process_request(t):
      """A dummy function that takes some time."""
      time.sleep(t)
  
  if __name__ == '__main__':
      # Start up the server to expose the metrics.
      start_http_server(8000)
      # Generate some requests.
      while True:
          process_request(random.random())
  

  Then you can visit http://localhost:8000/ to view the metrics.

  From one easy to use decorator you get:

  • request_processing_seconds_count: Number of times this function was called.
  • request_processing_seconds_sum: Total amount of time spent in this function.

  Prometheus's rate function allows calculation of both requests per second, and latency over time from this data.

  In addition if you're on Linux the process metrics expose CPU, memory and other information about the process for free.