Skip to content

2024

Activism⚑

Antifascism⚑

Hacktivism⚑

Chaos Communication Congress⚑

  • New: Introduce the CCC.

    Chaos Communication Congress is the best gathering of hacktivism in europe.

    Prepare yourself for the congress

    You can follow MacLemon's checklist

    Install the useful apps

    The schedule app

    You can use either the Fahrplan app or giggity, I've been using the second for a while, so is the one I use

    The navigation app

    c3nav is an application to get around the congress. The official F-droid is outdated, so add their repository to get the latest version.

    Reference - [Home](https://events.ccc.de/en/ - Engelsystem - Angel FAQ

  • New: Introduce the Angel's system.

    Angels are participants who volunteer to make the event happen. They are neither getting paid for their work nor do they get free admission.

    Expectation

    Helping at our events also comes with some simple, but important expectations of you:

    • Be on time for your shift or give Heaven early notice.
    • Be well rested, sober and not hungry.
    • Be open-minded and friendly in attitude.
    • Live our moral values:
    • Be excellent to each other.
    • All creatures are welcome.

    Quickstart

    • Create yourself an Engelsystem account
    • Arrive at the venue
    • Find the Heaven and go there.
    • Talk to a welcome angel or a shift coordinator to get your angel badge and get marked as arrived.
    • If you have any questions, you can always ask the shift coordinators behind the counter.
    • Attend an angel meeting
    • Announced in the Engelsystem news
    • Click yourself an interesting shift
    • Read shift descriptions first
    • Participate in your shift
    • Use the navigation to find the right place.
    • Arrive a little bit early at the meeting point
    • Rest for at least one hour
    • Repeat from step 5

    And always, have a lot of fun.

    To get more insights read this article

    The engelsystem

    The Engelsystem is the central place to distribute the work to all the helping angels. It can be a bit overwhelming at the beginning but you will get used to it and find your way around.

    As you might have seen there are many different shifts and roles for angels — some sounding more appealing than others. There are shifts where you need to have some knowledge before you can take them. This knowledge is given in introduction meetings or by taking an unrestricted shift in the team and getting trained on the job. These introduction meetings are announced in the Engelsystem under the tab "Meetings". Heaven and the teams try to make sure that there are only restrictions for shifts in place where they are absolutely needed.

    Most restrictions really only need a meeting or some unrestricted shifts at the team to get lifted. Harder restrictions are in place where volunteers need to have special certification, get access to certain systems with a huge amount of data (e.g. mail-queues with emails from participants) or handling big piles of money. Usually the requirements for joining an angeltype are included in the description of the angeltype.

    Especially the restricted shifts are tempting because after all we want to get the event running, aren’t we? From our personal experience what gets the event running are the most common things: Guarding a door, collecting bottle/trash, washing dishes in the angel kitchen, being on standby to hop in when spontaneous help is needed or check the wrist band at the entrance.

    If there are any further questions about angeltypes, the description of the angeltype usually includes contact data such as a DECT number or an e-mail address that can be used. Alternatively, you can also ask one of the persons of the respective angeltype mentioned under "Supporter".

    Teams

    Congress is organized from different teams, each with its own area of expertise.

    All teams are self-organized and provide their own set of services to the event.

    Teams spawn into existence by a need not fulfilled. They are seldom created by an authority.

    Check out the different teams to see which one suits you best.

    Some poeple suggest not to try to fit into special roles at your first event. The roles will find you – not the other way around. Our com­mu­ni­ty is not about per­so­nal gro­wing but about con­tri­bu­ting to each other and gro­wing by doing this.

    Perks

    Being an angel also comes with some perks. While we hope that participation is reward enough, here is a list of things that are exclusive to angels:

    • Community acknowledgement
    • Hanging out in Heaven and the angel hack center with its chill out area
    • Free coffee and (sparkling) water
    • Warm drinks or similar to make the cold night shifts more bearable

    Rewards

    If you have contributed a certain amount of time, you may receive access to:

    • Fantastic hot vegan and vegetarian meals
    • The famous limited™ angel T-shirt in Congress design
    • Maybe some other perks

Feminism⚑

Privileges⚑

Free Knowledge⚑

  • Correction: Update the way of seeding ill knowledge torrents.

    A good way to contribute is by seeding the ill torrents. You can generate a list of torrents that need seeding up to a limit in TB. If you follow this path, take care of IP leaking, they're

Free Software⚑

Conference organisation⚑

  • New: Software to manage the conference.

    There are some open source software that can make your life easier when hosting a conference:

    In addition to the management of talks from the call for papers till the event itself it can help the users visualise the talks schedule with EventFahrplan which is what's used in the ChaosComputerClub congress.

    If you also want to coordinate helpers and shifts take a look to Engelsystem

Ludditest⚑

Life Management⚑

Time management⚑

  • New: Anticapitalist approach to time management.

    Time management is being used to perpetrate the now hegemonic capitalist values. Its a pity because the underlying concepts are pretty useful and interesting but they are oriented towards improving productivity and being able to deal with an increasing amount of work. Basically they're always telling you to be a better cog. It doesn't matter how good you are, there is always room for improvement. I've fallen on this trap for a long time (I'm still getting my head out of the hole) and I'm trying to amend things by applying the concepts on an anticapitalist mindset. The turning point was to read Four thousand weeks: Time management for mortals by Oliver Burkeman, the article will have book extracts mixed with my way of thinking.

    Some (or most) of what's written in this article may not apply if you're not a male, white, young, cis, hetero, European, university graduated, able-bodied, """wealthy""" person. You need to be at a certain level of the social ladder to even start thinking in these terms. And depending on the number of oppressions you're suffering you'll have more or less room to maneuver. That margin is completely outside our control so by no means we should feel guilty of not being able to manage time. What follows are just guidelines to deal with this time anxiety imposed by the capitalist system with whatever air we have to breath.

    Changing the language

    The easiest way to change the underlying meaning is to change the language. Some substitutions are:

    • work -> focus: Nowadays we use work everywhere even if it's not in the laboral environment. For example I work very hard to achieve my goals. Working is the action of selling your time and energies in order to get the resources you need to live. It has an intrinsic meaning of sacrifice, doing something we don't want to do to get another thing in return. That's a tainted way of thinking about your personal time. I find focus is a great substitute as it doesn't have all those connotations. There are similar substitutions based on the same argument, such as: workspace -> space of focus, workflow -> action flow or just flow.
    • task -> action: Similar to work a task is something you kind of feel obliged to do. It uses a negative mindset to set the perfect scenario to feel guilty when you fail to do them. But you're on your personal time, it should be fine not to do an action for whatever reason. Action on the other side fosters a positive way of thinking, it suggests change, movement in a way that helps you move forward. There are also other derived words such as task manager -> action manager.
    • productivity -> efficiency: Productivy is the measurement of how fast or good you create products. And products are something that is made to be sold. Again this introduces a monetary mindset on all aspects of our life. Efficiency on the other side is the quality of achieving the largest amount of change using as little time, energy or effort as possible (Cambridge doesn't agree with me though :P. It may be because universities are also another important vector of spreading the capitalist values :(). So using efficiency we're focusing more on improving the process itself, so it can be applied for example on how to optimize your enjoyment of doing nothing. Which is completely antagonistic to the concept of productivity.

    Changing the mindset

    There is a widespread feeling that we're always short on time. We're obsessed with our overfilled inboxes and lengthening todo lists, haunted by the guilty feeling that we ought to be getting more done, or different things done, or both. At the same time we're deluged with advice on living the fully optimized life to squeeze the most from your time. And it get's worse as you age because time seems to speed up as you get older, steadily accelerating until months begging to flash by in what feels like minutes.

    The real problem isn't our limited time. It's that we've unwittingly inherited, and feel pressured to live by a troublesome set of ideas about how to use our limited time, all of which are pretty much guaranteed to make things worse. What follows are a list of mindset changes from the traditional time management bibliography that can set the bases of a healthier Anticapitalist one.

    Time is not a resource to spend

    Before timetables life rhythms emerged organically from the tasks they needed to do. You milked the cows when they needed milking and harvested the crops when it was harvest time. Anyone who would tried imposing an external schedule on any of that, for example, doing a month's milking in a single day to get it out of the way would rightly have been considered a lunatic.

    There was no need to think of time as something abstract and separate from life. In those days before clocks, when you did need to explain how long something might take, your only option was to compare it with some other concrete activity. They were untroubled by any notion of time "ticking away" thus living a heightened awareness of the vividness of things, the feeling of timelesness. Also known as living in deep time, or being in the flow, when the boundary separating the self from the rest of reality grows blurry and time stands still.

    There's one huge drawback in giving so little thought to the abstract idea of time, though, which is that it severely limits what you can accomplish. As soon as you want to coordinate the actions of more than a handful of people, you need a reliable, agreed-upon method of measuring time. This is why the first mechanical clocks came to be invented.

    Making time standardized and visible in this fashion inevitably encourages people to think of it as an abstract thing with an independent existence, distinct from the specific activities on which one might spend it. "time" is what ticks away as the hands move around the clock face.

    The next step was to start treating time as a resource, something to be bought and sold and used as efficiently as possible. This mindset shift serves as the precondition for all the uniquely modern ways in which we struggle with time today. Once time is a resource to be used, you start to feel pressure, whether from external forces or from yourself, to use it well, ant to berate yourself when you feel you've wasted it. When you're faced with too many demands, it's easy to assume that the only answer must be to make better use of time, by becoming more efficient, driving yourself harder, or working longer instead of asking whether the demands themselves might be unreasonable.

    Soon your sense of self-worth gets completely bound up with how you're using time: it stops being merely the water in which you swim and turns into something you fell you need to dominate or control if you're to avoid feeling guilty, panicked or overwhelmed.

    The fundamental problem is that this attitude towards time sets up a rigged game in which it's impossible ever to feel as though you're doing well enough. Instead of simply living our lives as they unfold in time it becomes difficult not to value each moment primarily according to its usefulness for some future goal, or for some future oasis of relaxation you hope to reach once your tasks are finally "out of the way".

    Ultimately it backfires. It wrenches us out of the present, leading to a life spent leaning into the future, worrying about whether things will work out, experiencing everything in terms of some later, hoped-for benefit, so that peace of mind never quite arrives. And it makes it all but impossible to experience the flow, that sense of timeless time which depends on forgetting the abstract yardstick and plunging back into the vividness of reality instead.

    If you don't disavow capitalism an increase in efficiency will only make things worse

    All this context makes us eager to believe the promises of time management frameworks (like GTD) that if you improve your efficiency you'll get more time to enjoy your life. If you follow the right time management system, build the right habits, and apply sufficient self-discipline, you will win the struggle with time.

    Reality then kicks in you never win the struggle and only feel more stressed and unhappy. You realize that all the time you've saved is automatically filled up by more things to do in a never ending feedback loop. It's true that you get more done, and yet, paradoxically, you only feel busier, more anxious and somehow emptier as a result. Time feels like an unstoppable conveyor belt, bringing us new actions as fast as we can dispatch the old ones; and becoming more efficient just seems to cause the belt to speed up. Or else, eventually, to break down.

    It also has another side-effect. As life accelerates, everyone grows more impatient. It's somehow vastly more aggravating to wait two minutes for the microwave than two hours for the oven, or ten seconds for a slow loading web page versus three days to receive the same information by mail.

    Denying reality never works though. It may provide some immediate relief, because it allows you to go on thinking that at some point in the future you might, at last, feel totally in control. But it can't ever bring the sense that you're doing enough (that you are enough) because it defines enough as a kind of limitless control that no human can attain. Instead, the endless struggle leads to more anxiety and less fulfilling life. For example, the more you believe yo might succeed in "fitting everything in", the more commitments you naturally take on, and the less you feel the need to ask whether each new commitment is truly worth a portion of your time, and so your days inevitably fill with more activities you don't especially value. The more you hurry, the more frustrating it is to encounter tasks that won't be hurried, the more compulsively you plan for the future, the more anxious you feel about any remaining uncertainties, of which there will always be plenty.

    Time management used this way serves as a distraction to numb our minds:

    • It may hide the sense of precariousness inherent to the capitalist world we live in. If you could meet every boss's demand, while launching various side projects on your own, maybe one day You'd finally feel secure in your career and your finances.
    • Divert your energies from fully experiencing the reality in which you find yourself, holding at bay certain scary questions about what you're doing with your life, and whether major changes might not be needed. As long as you're always just on the cusp of mastering time, you can avoid the thought that what life is really demanding from you might involve surrendering the craving for mastery and diving into the unknown instead.

    Embrace the finitude of time

    We recoil from the notion that this is it. That this life, with all its flaws and inescapable vulnerabilities, its extreme brevity, and our limited influence over how it unfolds, is the only one we'll get a shot at. Instead, we mentally fight against the way things are, so that we don't have to consciously participate in what it's like to feel claustrophobic, imprisoned, powerless, and constrained by reality.

    Our troubled relationship with time arises largely from this same effort to avoid the painful constrains of reality. And most of our strategies for becoming more efficient make things worse, because they're really just ways of furthering the avoidance. After all, it's painful to confront how limited your time is, because it means that tough choices are inevitable and that you won't have time for all you once dreamed you might do. It's also painful to accept the limited control over the time you do get: maybe you simply lack the stamina or talent or other resources to perform well in all the roles you feel you should. And so, rather than face our limitations, we engage in avoidance strategies, in an effort to carry on feeling limitless. We push ourselves harder, chasing fantasies of the perfect work-life balance, or we implement time management systems that promise to make time for everything, so that tough choices won't be required. Or we procrastinate, which is another means of maintaining the feeling of omnipotent control over life, because you needn't risk the upsetting experience of failing at an intimidating project if you never even start it. We fill our minds with busyness and distraction to numb ourselves emotionally. Or we plan compulsively, because the alternative is to confront how little control over the future we really have.

    Heal yourself from FOMO

    In practical terms, a limit-embracing attitude to time means organizing your days with the understanding that you definitely won't have time for everything you want to do, or that other people want you to do, and so, at the very least, you can stop beating yourself up for failing. Since hard choices are unavoidable, what matters is learning to make them consciously, deciding what to focus on and what to neglect, rather than letting them get made by default, or deceiving yourself that, with enough hard work and the right time management tricks, you might not have to make them at all. It also means resisting the temptation to "keep your options open" in favour of deliberately making big, daunting, irreversible commitments, which you can't know in advance will turn out for the best, but which reliably prove more fulfilling in the end. And it means standing firm in the face of FOMO (fear of missing out) because you come to realize that missing out on something (indeed on almost everything) is basically guaranteed. Which isn't actually a problem anyway, it turns to, because "missing out" is what makes your choices meaningful in the first place. Every decision to use a portion of time on anything represents the sacrifice of all the other ways in which you could have spent that time, but didn't, and to willingly make that sacrifice is to take a stand, without reservation, on what matters most to you.

    Embrace your control limits

    The more you try to manage your time with the goal of achieving a feeling of total control and freedom from the inevitable constrains of being human, the more stressful, empty, and frustrating life gets. But the more you confront the facts of finitude instead, and work with them, rather than against them, the more efficient, meaningful and joyful life becomes. Anxiety won't ever completely go away, we're even limited, apparently, in our capacity to embrace our limitations. But I'm aware of no other time management technique that's half as effective as just facing the way things truly are.

    Time pressure comes largely from forces outside our control: from a cutthroat economy; from the loss of the social safety networks that used to help ease the burdens of work and childcare; and from the sexist expectation that women must excel in their careers while assuming most of the responsibilities at home. None of that will be solved with time management. Fully facing the reality of it can only help though. So long as you continue to respond to impossible demands on your time by trying to persuade yourself that you might one day find some way to do the impossible, you're implicitly collaboration with those demands. Whereas once you deeply grasp that they are impossible, you'll stop believing the delusion that any of that is ever going to bring satisfaction and will be newly empowered to resist them, letting you focus instead on building the most meaningful life you can, in whatever situation you're in.

    Seeing and accepting our limited powers over our time can prompt us to question the very idea that time is something you use in the first place. There is an alternative: the notion of letting time use you, approaching life not as an opportunity to implement your predetermined plans for success but as a matter of responding to the needs of your place and your moment in history.

    Embrace the community constrains

    Moreover, most of us seek a specifically individualistic kind of mastery over time. Our culture's ideal is that you alone should control your schedule, doing whatever you prefer, whenever you want, because it's scary to confront the truth that almost everything worth doing depends on cooperating with others, and therefore on exposing yourself to the emotional uncertainties of relationships. In the end the more individual sovereignty you achieve over your time, the lonelier you get. The truth then is that freedom sometimes is to be found not in achieving greater sovereignty over your own schedule but in allowing yourself to be constrained by the rhythms of community. Participating in forms of social life where you don't get to decide exactly what you do or when you doi it. And it leads to the insight that meaningful efficiency often comes not from hurrying things up but from letting them take the time they take.

    Live for today not tomorrow

    It doesn't matter what you do, we all sense that there are always more important and fulfilling ways we could be spending our time, even if we can't say exactly what they are, yet we systematically spend our days doing other things instead. This feeling can take many forms: the desire to devote yourself to some larger cause, continuously demanding more from yourself, desiring to spend more time with your loved ones.

    Our attempts to become more efficient may have the effect of pushing the genuinely important stuff even further over the horizon. Our days are spent trying to "get through" tasks, in order to get them "out of the way", with the result that we live mentally in the future, waiting for when we'll finally get around to what really matters, and worrying in the meantime, that we don't measure up, that we might lack the drive or stamina to keep pace with the speed at which life now seems to move. We live in a constant spirit of joyless urgency.

  • New: Time is not a resource to be tamed.

    You'll see everywhere the concept of time management. I feel it's daring to suggest that you have the power to actually manage time. No you can't as much as you can't tame the sea. Time is not a resource to be spent or be managed, the best we can do is to try to understand its flows and navigate it the best we can.

  • New: Keep on summing up Oliver Burkeman book.

    Efficiency doesn't necessarily give you more time

    We're eager to believe the promises of time management frameworks (like GTD) that if you improve your efficiency you'll get more time to enjoy your life. If you follow the right time management system, build the right habits, and apply sufficient self-discipline, you will win the struggle with time.

    Reality then kicks in you never win the struggle and only feel more stressed and unhappy. You realize that all the time you've saved is automatically filled up by more things to do in a never ending feedback loop. Time feels like an unstoppable conveyor belt, bringing us new actions as fast as we can dispatch the old ones; and becoming more efficient just seems to cause the belt to speed up. Or else, eventually, to break down. It's true that you get more done, and yet, paradoxically, you only feel busier, more anxious and somehow emptier as a result.

    It get's even worse because importance is relative and you may fall into efficiency traps.

    Heal yourself from FOMO

    Another problem that FOMO brings us is that it leads us to lives where you "truly lived" only if you've lived all the experiences you could live. This leads to a frustrating life as the world has infinite of them, so getting a handful of them under your belt brings you no closer to a sense of having feasted on life's possibilities. You lead yourself in another efficiency trap where the more you experience the more additional wonderful experiences you sarta to feel you could have on top of all those you've already had, with the result that the feeling of existential overwhelm gets worse. To fight this existential overwhelm you can resist the urge to consume more and more experiences and embrace the idea that you're going to miss most of them. You'll then be able to focus on fully enjoying the tiny slice of experiences you actually do have time for.

    This FOMO fever is normal given the facts that we're more conscious of the limits of our time (after deterring the after life), the increase of choices that the world has brought us, and the internet amplifier.

    You do what you can do

    It's usual to feel as though you absolutely must do more than you can do. We live overwhelmed in a constant anxiety of fearing, or knowing for certain, that the actions we want to carry out won't fit on our available time. It looks like this feeling arises on every step of the economic ladder (shown in the works of Daniel Markovits).

    The thing is that the idea in itself doesn't make any sense. You can't do more than you can do even if you must. If you truly don't have time for everything you want to do, or feel you ought to do, or that others are badgering you to do, then, well, you don't have time, no matter how grave the consequences of failing to do it all might prove to be. So technically it's irrational to feel troubled by an overwhelming to-do list. You'll do what you can, you won't do what you can't, and the tyrannical inner voice insisting that you must do everything is simply mistaken. We rarely stop to consider things so rationally, though, because that would mean confronting the painful truth of our limitations. We would be forced to acknowledge that there are hard choices to be made: which balls to let drop, which people to disappoint, which ambitions to abandon, which roles to fail at... Instead, in an attempt to avoid these unpleasant truths, we deploy the strategy that dominates most conventional advice on how to deal with busyness: we tell ourselves we'll just have to find a way to do more. So to address our busyness we're making ourselves busier still.

    Importance is relative

    The problem here is that you'll never be able to make time for everything that feels important. A similar mindset of the section Efficiency doesn't give you more time can be applied. The reason isn't that you haven't yet discovered the right time management tricks, or applied sufficient effort, or that you're generally useless. It's that the underlying assumption is unwarranted: there's no reason to believe you'll make time for everything that matters simply by getting more done. For a start, what "matters" is subjective, so you've no grounds for assuming that there will be time for everything that you, or anyone else deems important. But the other exasperating issue is that if you succeed in fitting more in, you'll find the goalposts start to shift: more things will begin to seem important, meaningful or obligatory. Acquire a reputation for doing your work at amazing speed, and you'll be given more of it. An example of this is gathered in Ruth Schwartz's book More work for mother, where it shows that when washing machines and vacuum cleaners appeared no time was saved at all, because society's standards of cleanliness rose to offset the benefits. What needs doing expands so as to fill the time available for its completion.

    Be mindful of the efficiency trap

    Sometimes improving your efficiency may lead you to a worse scenario ("efficiency trap") where you won't generally result in the feeling of having "enough time", because, all else being equal, the demands will increase to offset any benefits. Far from getting things done, you'll be creating new things to do. A clear example of this is email management. Every time you reply to an email, there's a good chance of provoking a reply to that email, which itself may require another reply, and so on and so on. At the same time, you'll become known as someone who responds promptly to email, so more people will consider it worth their while to message you to begin with. So it's not simply that you never get though your email; it's that the process of "getting through your email" actually generates more email.

    For most of us, most of the time, it isn't feasible to avoid the efficiency trap altogether, but you can stop believing you'll ever solve the challenge of busyness by cramming more in, because that just makes matters worse. And once you stop investing in the idea that you might one day achieve peace of mind that way, it becomes easier to find peace of mind in the present, in the midst of overwhelming demands, because you're no longer making your peace of mind dependent on dealing with all the demands. Once you stop believing that it might somehow be possible to avoid hard choices about time, it gets easier to make better ones.

    If you also have the knowledge of the existence of the efficiency traps you may detect them and try to get the benefits without the penalties.

    Do the important stuff

    The worst aspect of the trap is that it's also a matter of quality. The harder you struggle to fit everything in, the more of your time you'll find yourself spending on the least meaningful things. This is because the more firmly you believe it ought to be possible to find time for everything, the less pressure you'll feel to ask whether any given activity sis the best use of a portion of your time. Each time something new shows up, you'll be strongly biased in favor of accepting it, because you'll assume you needn't sacrifice any other tasks or opportunities in order to make space for it. Soon your life will be automatically filled with not just more things but with more trivial or tedious things.

    The important stuff gets postponed because such tasks need your full focus, which means to wait until you have a good chunk of free time and fewer small-but-urgent tasks tugging at your attention. So you spend your energy into clearing the decks, cranking through the smaller stuff to get it out of the way, only to discover that doing so takes the whole day, that the decks are filled up again overnight and that the moment for doing the important stuff never arrives. One can waste years this way, systematically postponing precisely the things one cares the most.

    What's needed in these situations is to resist the urges of being on top of everything and learn to live with the anxiety of feeling overwhelmed without automatically responding by trying to fit more in. Instead of clearing the decks, decline to do so, focusing instead on what's truly of greatest consequence while tolerating the discomfort of knowing that, as you do so, the decks will be filling up further, with emails and errands and other to-dos, many of which you may never get around to at all.

    You'll sometimes still decide to drive yourself hard in an effort to squeeze more in, when circumstances absolutely require it. But that won't be your default mode, because you'll no longer be operating under the illusion of one day making time for everything.

    Evaluate what you miss when you increase your efficiency

    Part of the benefits of efficiency is that you free yourself from tedious experiences, the side effect is that some times we're not conscious of being removing experiences that we valued. So even if everything runs more smoothly, smoothness is a dubious virtue, since it's often the unsmoothed textures of life that makes them livable, helping nurture the relationships that are crucial for mental and physical health, and for the resilience of our communities. For example if you buy online the groceries you miss the chance to regularly meet with your neighbours at your local grocery store.

    Convenience makes things easy, but without regard to whether easiness is truly what's most valuable in any given context. When you render the process more convenient you drain it of its meaning. The effect of convenience isn't just that the given activity starts to feel less valuable, but that we stop engaging in certain valuable activities altogether, in favour of more convenient ones. Because you can stay home, order food online, and watch sitcoms on a streaming service, you find yourself doing so although you might be perfectly aware that you'd have had a better time if you had met with your friends.

    Meanwhile, those aspects of life that resist being made to run more smoothly start to seem repellent. When you can skip the line and buy concert tickets on your phone, waiting in line to vote in an election is irritating. As convenience colonizes everyday life, activities gradually sort themselves into two types: the kind that are now far more convenient, but that feel empty or out of sync with our true preferences; and the kind that now seem intensely annoying because of how inconvenient they remain. Resisting all this is difficult because the Capital is winning this discourse and you'll have more pressure from your environment to stay convenient.

vdirsyncer⚑

Org Mode⚑

  • New: Start working on a task dates.

    SCHEDULED defines when you are plan to start working on that task.

    The headline is listed under the given date. In addition, a reminder that the scheduled date has passed is present in the compilation for today, until the entry is marked as done or disabled.

    *** TODO Call Trillian for a date on New Years Eve.
    SCHEDULED: <2004-12-25 Sat>

    Although is not a good idea (as it promotes the can pushing through the street), if you want to delay the display of this task in the agenda, use SCHEDULED: <2004-12-25 Sat -2d> the task is still scheduled on the 25th but will appear two days later. In case the task contains a repeater, the delay is considered to affect all occurrences; if you want the delay to only affect the first scheduled occurrence of the task, use --2d instead.

    Scheduling an item in Org mode should not be understood in the same way that we understand scheduling a meeting. Setting a date for a meeting is just a simple appointment, you should mark this entry with a simple plain timestamp, to get this item shown on the date where it applies. This is a frequent misunderstanding by Org users. In Org mode, scheduling means setting a date when you want to start working on an action item.

    You can set it with <leader>s (Default: <leader>ois)

  • New: Deadlines.

    DEADLINE are like appointments in the sense that it defines when the task is supposed to be finished on. On the deadline date, the task is listed in the agenda. The difference with appointments is that you also see the task in your agenda if it is overdue and you can set a warning about the approaching deadline, starting org_deadline_warning_days before the due date (14 by default). It's useful then to set DEADLINE for those tasks that you don't want to miss that the deadline is over.

    An example:

    * TODO Do this
DEADLINE: <2023-02-24 Fri>

    You can set it with <leader>d (Default: <leader>oid).

    If you need a different warning period for a special task, you can specify it. For example setting a warning period of 5 days DEADLINE: <2004-02-29 Sun -5d>.

    If you're as me, you may want to remove the warning feature of DEADLINES to be able to keep your agenda clean. Most of times you are able to finish the task in the day, and for those that you can't specify a SCHEDULED date. To do so set the default number of days to 0.

    require('orgmode').setup({
  org_deadline_warning_days = 0,
})

    Using too many tasks with a DEADLINE will clutter your agenda. Use it only for the actions that you need to have a reminder, instead try to using appointment dates instead. The problem of using appointments is that once the date is over you don't get a reminder in the agenda that it's overdue, if you need this, use DEADLINE instead.

  • New: How to deal with overdue SCHEDULED and DEADLINE tasks.

  • New: Introduce org-rw.

    org-rw is a Python library to process your orgmode files.

    Installation:

    pip install org-rw

    Load an orgmode file:

    from org_rw import load

with open('your_file.org', 'r') as f:
    doc = load(f)

  • New: Install using lazyvim.

    return {
  'nvim-orgmode/orgmode',
  ```lua
    {
        'nvim-orgmode/orgmode',
        dependencies = {
      { 'nvim-treesitter/nvim-treesitter', lazy = true },
    },
        event = 'VeryLazy',
        config = function()
    -- Load treesitter grammar for org
    require('orgmode').setup_ts_grammar()

    -- Setup treesitter
    require('nvim-treesitter.configs').setup({
        highlight = {
          enable = true,
          additional_vim_regex_highlighting = { 'org' },
        },
        ensure_installed = { 'org' },
      })

    -- Setup orgmode
    require('orgmode').setup({
        org_agenda_files = '~/orgfiles/**/*',
        org_default_notes_file = '~/orgfiles/refile.org',
      })
  end,
  }
  ```
  dependencies = {
    { 'nvim-treesitter/nvim-treesitter', lazy = true },
  },
  event = 'VeryLazy',
  config = function()
    -- Load treesitter grammar for org
    require('orgmode').setup_ts_grammar()

    -- Setup treesitter
    require('nvim-treesitter.configs').setup({
      highlight = {
        enable = true,
        additional_vim_regex_highlighting = { 'org' },
      },
      ensure_installed = { 'org' },
    })

    -- Setup orgmode
    require('orgmode').setup({
      org_agenda_files = '~/orgfiles/**/*',
      org_default_notes_file = '~/orgfiles/refile.org',
    })
  end,
}

  • New: Troubleshoot orgmode with dap.

    Use the next config and follow the steps of Create an issue in the orgmode repository.

    vim.cmd([[set runtimepath=$VIMRUNTIME]])
vim.cmd([[set packpath=/tmp/nvim/site]])

local package_root = '/tmp/nvim/site/pack'
local install_path = package_root .. '/packer/start/packer.nvim'

local function load_plugins()
  require('packer').startup({
    {
      'wbthomason/packer.nvim',
      { 'nvim-treesitter/nvim-treesitter' },
      { 'nvim-lua/plenary.nvim'},
      { 'nvim-orgmode/orgmode'},
      { 'nvim-telescope/telescope.nvim'},
      { 'lyz-code/telescope-orgmode.nvim' },
      { 'jbyuki/one-small-step-for-vimkind' },
      { 'mfussenegger/nvim-dap' },
      { 'kristijanhusak/orgmode.nvim', branch = 'master' },
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. '/plugin/packer_compiled.lua',
    },
  })
end

_G.load_config = function()
  require('orgmode').setup_ts_grammar()
  require('nvim-treesitter.configs').setup({
    highlight = {
      enable = true,
      additional_vim_regex_highlighting = { 'org' },
    },
  })

  vim.cmd([[packadd nvim-treesitter]])
  vim.cmd([[runtime plugin/nvim-treesitter.lua]])
  vim.cmd([[TSUpdateSync org]])

  -- Close packer after install
  if vim.bo.filetype == 'packer' then
    vim.api.nvim_win_close(0, true)
  end

  require('orgmode').setup({
    org_agenda_files = {
      './*'
    }
  }
  )

  -- Reload current file if it's org file to reload tree-sitter
  if vim.bo.filetype == 'org' then
    vim.cmd([[edit!]])
  end
end
if vim.fn.isdirectory(install_path) == 0 then
  vim.fn.system({ 'git', 'clone', 'https://github.com/wbthomason/packer.nvim', install_path })
  load_plugins()
  require('packer').sync()
  vim.cmd([[autocmd User PackerCompileDone ++once lua load_config()]])
else
  load_plugins()
  load_config()
end

require('telescope').setup{
  defaults = {
    preview = {
     enable = true,
      treesitter = false,
    },
    vimgrep_arguments = {
      "ag",
      "--nocolor",
      "--noheading",
      "--numbers",
      "--column",
      "--smart-case",
      "--silent",
      "--follow",
      "--vimgrep",
    },
    file_ignore_patterns = {
      "%.svg",
      "%.spl",
      "%.sug",
      "%.bmp",
      "%.gpg",
      "%.pub",
      "%.kbx",
      "%.db",
      "%.jpg",
      "%.jpeg",
      "%.gif",
      "%.png",
      "%.org_archive",
      "%.flf",
      ".cache",
      ".git/",
      ".thunderbird",
      ".nas",
    },
    mappings = {
      i = {
        -- Required so that folding works when opening a file in telescope
        -- https://github.com/nvim-telescope/telescope.nvim/issues/559
        ["<CR>"] = function()
            vim.cmd [[:stopinsert]]
            vim.cmd [[call feedkeys("\<CR>")]]
        end,
        ['<C-j>'] = 'move_selection_next',
        ['<C-k>'] = 'move_selection_previous',
      }
    }
  },
  pickers = {
    find_files = {
      find_command = { "rg", "--files", "--hidden", "--glob", "!**/.git/*" },
      hidden = true,
      follow = true,
    }
  },
  extensions = {
    fzf = {
      fuzzy = true,                    -- false will only do exact matching
      override_generic_sorter = true,  -- override the generic sorter
      override_file_sorter = true,     -- override the file sorter
      case_mode = "smart_case",        -- or "ignore_case" or "respect_case"
                                       -- the default case_mode is "smart_case"
    },
    heading = {
      treesitter = true,
    },
  }
}

require('telescope').load_extension('orgmode')

local key = vim.keymap
vim.g.mapleader = ' '

local builtin = require('telescope.builtin')
key.set('n', '<leader>f', builtin.find_files, {})
key.set('n', '<leader>F', ':Telescope file_browser<cr>')

vim.api.nvim_create_autocmd('FileType', {
  pattern = 'org',
  group = vim.api.nvim_create_augroup('orgmode_telescope_nvim', { clear = true }),
  callback = function()
    vim.keymap.set('n', '<leader>r', require('telescope').extensions.orgmode.refile_heading)
    vim.keymap.set('n', '<leader>g', require('telescope').extensions.orgmode.search_headings)
  end,
})

require('orgmode').setup_ts_grammar()
local org = require('orgmode').setup({
  org_agenda_files = {
    "./*"
  },
  org_todo_keywords = { 'TODO(t)', 'CHECK(c)', 'DOING(d)', 'RDEACTIVATED(r)', 'WAITING(w)', '|','DONE(e)', 'REJECTED(j)', 'DUPLICATE(u)' },
  org_hide_leading_stars = true,
  org_deadline_warning_days = 0,
  win_split_mode = "horizontal",
  org_priority_highest = 'A',
  org_priority_default = 'C',
  org_priority_lowest = 'D',
  mappings = {
    global = {
      org_agenda = 'ga',
      org_capture = ';c',
    },
    org = {
      -- Enter new items
      org_meta_return = '<c-cr>',
      org_insert_heading_respect_content = ';<cr>',
      org_insert_todo_heading = "<c-t>",
      org_insert_todo_heading_respect_content = ";t",

      -- Heading promoting and demoting
      org_toggle_heading = '<leader>h',
      org_do_promote = '<h',
      org_do_demote = '>h',
      org_promote_subtree = '<H',
      org_demote_subtree = '>H',

      -- Heading moving
      org_move_subtree_up = "<leader>k",
      org_move_subtree_down = "<leader>j",

      -- Heading navigation
      org_next_visible_heading = '<c-j>',
      org_previous_visible_heading = '<c-k>',
      org_forward_heading_same_level = '<c-n>',
      org_backward_heading_same_level = '<c-p>',
      outline_up_heading = 'gp',
      org_open_at_point = 'gx',

      -- State transition
      org_todo = 't',

      -- Priority change
      org_priority_up = '-',
      org_priority_down = '=',

      -- Date management
      org_deadline = '<leader>d',
      org_schedule = '<leader>s',
      org_time_stamp = ';d',
      org_change_date = '<c-e>',

      -- Tag management
      org_set_tags_command = 'T',

      -- Archive management and refiling
      org_archive_subtree = ';a',
      org_toggle_archive_tag = ';A',
      -- org_refile = '<leader>r',  The refile is being handled below
    },
    agenda = {
      org_agenda_later = '<c-n>',
      org_agenda_earlier = '<c-p>',
      org_agenda_switch_to = '<tab>',
      org_agenda_goto = '<cr>',
      org_agenda_priority_up = '=',
      org_agenda_set_tags = 'T',
      org_agenda_deadline = '<leader>d',
      org_agenda_schedule = '<leader>s',
      org_agenda_archive = 'a',
    },
    capture = {
      org_capture_finalize = ';w',
      org_capture_refile = ';r',
      org_capture_kill = 'qqq',
    },
  }
})
local dap = require"dap"
dap.configurations.lua = {
  {
    type = 'nlua',
    request = 'attach',
    name = "Attach to running Neovim instance",
  }
}

dap.adapters.nlua = function(callback, config)
  callback({ type = 'server', host = config.host or "127.0.0.1", port = config.port or 8086 })
end

vim.api.nvim_set_keymap('n', '<leader>b', [[:lua require"dap".toggle_breakpoint()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>c', [[:lua require"dap".continue()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>n', [[:lua require"dap".step_over()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>m', [[:lua require"dap".repl.open()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>N', [[:lua require"dap".step_into()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<F12>', [[:lua require"dap.ui.widgets".hover()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<F5>', [[:lua require"osv".launch({port = 8086})<CR>]], { noremap = true })

  • New: Hide the state changes in the folds.

    The folding of the recurring tasks iterations is also kind of broken. For the next example

    ** TODO Recurring task
   DEADLINE: <2024-02-08 Thu .+14d -0d>
   :PROPERTIES:
   :LAST_REPEAT: [2024-01-25 Thu 11:53]
   :END:
   - State "DONE" from "TODO" [2024-01-25 Thu 11:53]
   - State "DONE" from "TODO" [2024-01-10 Wed 23:24]
   - State "DONE" from "TODO" [2024-01-03 Wed 19:39]
   - State "DONE" from "TODO" [2023-12-11 Mon 21:30]
   - State "DONE" from "TODO" [2023-11-24 Fri 13:10]

   - [ ] Do X

    When folded the State changes is not added to the Properties fold. It's shown something like:

    ** TODO Recurring task
   DEADLINE: <2024-02-08 Thu .+14d -0d>
   :PROPERTIES:...
   - State "DONE" from "TODO" [2024-01-25 Thu 11:53]
   - State "DONE" from "TODO" [2024-01-10 Wed 23:24]
   - State "DONE" from "TODO" [2024-01-03 Wed 19:39]
   - State "DONE" from "TODO" [2023-12-11 Mon 21:30]
   - State "DONE" from "TODO" [2023-11-24 Fri 13:10]

   - [ ] Do X

    I don't know if this is a bug or a feature, but when you have many iterations it's difficult to see the task description. So it would be awesome if they could be included into the properties fold or have their own fold.

    I've found though that if you set the org_log_into_drawer = "LOGBOOK" in the config this is fixed.

  • New: Things that are still broken or not developed.

  • Correction: Rename Task to Action.

    To remove the productivity capitalist load from the concept

  • New: How to deal with recurring tasks that are not yet ready to be acted upon.

    By default when you mark a recurrent task as DONE it will transition the date (either appointment, SCHEDULED or DEADLINE) to the next date and change the state to TODO. I found it confusing because for me TODO actions are the ones that can be acted upon right now. That's why I'm using the next states instead:

    • INACTIVE: Recurrent task which date is not yet close so you should not take care of it.
    • READY: Recurrent task which date is overdue, we acknowledge the fact and mark the date as inactive (so that it doesn't clobber the agenda).

    The idea is that once an INACTIVE task reaches your agenda, either because the warning days of the DEADLINE make it show up, or because it's the SCHEDULED date you need to decide whether to change it to TODO if it's to be acted upon immediately or to READY and deactivate the date.

    INACTIVE then should be the default state transition for the recurring tasks once you mark it as DONE. To do this, set in your config:

    org_todo_repeat_to_state = "INACTIVE",

    If a project gathers a list of recurrent subprojects or subactions it can have the next states:

    • READY: If there is at least one subelement in state READY and the rest are INACTIVE
    • TODO: If there is at least one subelement in state TODO and the rest may have READY or INACTIVE
    • INACTIVE: The project is not planned to be acted upon soon.
    • WAITING: The project is planned to be acted upon but all its subelements are in INACTIVE state.

  • New: Debug doesn't go up in the jump list.

    It's because is a synonym of , and org_cycle is mapped by default as If you're used to use zc then you can disable the org_cycle by setting the mapping org_cycle = "<nop>".

  • New: Python libraries.

    org-rw

    org-rw is a library designed to handle Org-mode files, offering the ability to modify data and save it back to the disk.

    • Pros:
    • Allows modification of data and saving it back to the disk

    • Includes tests to ensure functionality

    • Cons:

    • Documentation is lacking, making it harder to understand and use
    • The code structure is complex and difficult to read
    • Uses unittest instead of pytest, which some developers may prefer
    • Tests are not easy to read
    • Last commit was made five months ago, indicating potential inactivity
    • Not very popular, with only one contributor, three stars, and no forks

    orgparse

    orgparse is a more popular library for parsing Org-mode files, with better community support and more contributors. However, it has significant limitations in terms of editing and saving changes.

    • Pros:
    • More popular with 13 contributors, 43 forks, and 366 stars
    • Includes tests to ensure functionality

    • Provides some documentation, available here

    • Cons:

    • Documentation is not very comprehensive
    • Cannot write back to Org-mode files, limiting its usefulness for editing content
      • The author suggests using inorganic to convert Org-mode entities to text, with examples available in doctests and the orger library.
      • inorganic is not popular, with one contributor, four forks, 24 stars, and no updates in five years
      • The library is only 200 lines of code
      • The ast is geared towards single-pass document reading. While it is possible to modify the document object tree, writing back changes is more complicated and not a common use case for the author.

    Tree-sitter

    Tree-sitter is a powerful parser generator tool and incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited.

    • Pros:
    • General enough to parse any programming language
    • Fast enough to parse on every keystroke in a text editor
    • Robust enough to provide useful results even in the presence of syntax errors
    • Dependency-free, with a runtime library written in pure C
    • Supports multiple languages through community-maintained parsers
    • Used by Neovim, indicating its reliability and effectiveness
    • Provides good documentation, available here

    • Python library, py-tree-sitter, simplifies the installation process

    • Cons:

    • Requires installation of Tree-sitter and the Org-mode language parser separately
    • The Python library does not handle the Org-mode language parser directly

    To get a better grasp of Tree-sitter you can check their talks:

    lazyblorg orgparser.py

    lazyblorg orgparser.py is another tool for working with Org-mode files. However, I didn't look at it.

  • Correction: Tweak area concept.

    Model a group of projects that follow the same interest, roles or accountabilities. These are not things to finish but rather to use as criteria for analyzing, defining a specific aspect of your life and to prioritize its projects to reach a higher outcome. We'll use areas to maintain balance and sustainability on our responsibilities as we operate in the world. Areas' titles don't contain verbs as they don't model actions. An example of areas can be health, travels or economy.

    To filter the projects by area I set an area tag that propagates downstream. To find the area documents easily I add a section in the index.org of the documentation repository. For example:

  • New: Change the default org-todo-keywords.

    orig = '''* NEW_TODO_STATE First entry

* NEW_DONE_STATE Second entry'''
doc = loads(orig, environment={
  'org-todo-keywords': "NEW_TODO_STATE | NEW_DONE_STATE"
})

Orgzly⚑

  • New: Migrate from Orgzly to Orgzly Revived.

Gancio⚑

  • Correction: Change the concept of Task for Action.

    To remove the capitalist productive mindset from the concept

  • Correction: Action cleaning.

    Marking steps as done make help you get an idea of the evolution of the action. It can also be useful if you want to do some kind of reporting. On the other hand, having a long list of done steps (specially if you have many levels of step indentation may make the finding of the next actionable step difficult. It's a good idea then to often clean up all done items.

    • For non recurring actions use the LOGBOOK to move the done steps. for example: 
      ** DOING Do X
   :LOGBOOK:
   - [x] Done step 1
   - [-] Doing step 2
     - [x] Done substep 1
   :END:
   - [-] Doing step 2
     - [ ] substep 2

    This way the LOGBOOK will be automatically folded so you won't see the progress but it's at hand in case you need it.

    • For recurring actions:
    • Mark the steps as done
    • Archive the todo element.
    • Undo the archive.
    • Clean up the done items.

    This way you have a snapshot of the state of the action in your archive.

  • New: Project cleaning.

    Similar to action cleaning we want to keep the state clean. If there are not that many actions under the project we can leave the done elements as DONE, once they start to get clobbered up we can create a Closed section.

    For recurring projects:

    • Mark the actions as done
    • Archive the project element.
    • Undo the archive.
    • Clean up the done items.

  • New: Trimester review.

    The objectives of the trimester review are:

    • Identify the areas to focus on for the trimester
    • Identify the tactics you want to use on those areas.
    • Review the previous trimester tactics

    The objectives are not:

    • To review what you've done or why you didn't get there.

    When to do the trimester reviews

    As with personal integrity review, it's interesting to do analysis at representative moments. It gives it an emotional weight. You can for example use the solstices or my personal version of the solstices:

    • Spring analysis (1st of March): For me the spring is the real start of the year, it's when life explodes after the stillness of the winter. The sun starts to set later enough so that you have light in the afternoons, the climate gets warmer thus inviting you to be more outside, the nature is blooming new leaves and flowers. It is then a moment to build new projects and set the current year on track.
    • Summer analysis (1st of June): I hate heat, so summer is a moment of retreat. Everyone temporarily stop their lives, we go on holidays and all social projects slow their pace. Even the news have even less interesting things to report. It's so hot outside that some of us seek the cold refuge of home or remote holiday places. Days are long and people love to hang out till late, so usually you wake up later, thus having less time to actually do stuff. Even in the moments when you are alone the heat drains your energy to be productive. It is then a moment to relax and gather forces for the next trimester. It's also perfect to develop easy and chill personal projects that have been forgotten in a drawer. Lower your expectations and just flow with what your body asks you.
    • Autumn analysis (1st of September): September it's another key moment for many people. We have it hardcoded in our life since we were children as it was the start of school. People feel energized after the summer holidays and are eager to get back to their lives and stopped projects. You're already 6 months into the year, so it's a good moment to review your year plan and decide how you want to invest your energy reserves.
    • Winter analysis (1st of December): December is the cue that the year is coming to an end. The days grow shorter and colder, they basically invite you to enjoy a cup of tea under a blanket. It is then a good time to get into your cave and do an introspection analysis on the whole year and prepare the ground for the coming year. Some of the goals of this season are:
    • Think everything you need to guarantee a good, solid and powerful spring start.
    • Do the year review to adjust your principles.

    The year is then divided in two sets of an expansion trimester and a retreat one. We can use this information to adjust our life plan accordingly. In the expansion trimester we could invest more energies in the planning, and in the retreat ones we can do more throughout reviews.

    Listen to your desires

    The trimester review requires an analysis that doesn't fill in a day session. It requires slow thinking over some time. So I'm creating a task 10 days before the actual review to start thinking about the next trimester. Whether it's ideas, plans, desires, objectives, values, or principles.

    Is useful for that document to be available wherever you go, so that in any spare time you can pop it up and continue with the train of thought.

    Doing the reflection without seeing your life path prevents you from being tainted by it, thus representing the real you of right now.

    On the day to actually do the review, follow the steps of the Month review prepare adjusting them to the trimester case.

    Answer some meaningful guided questions

    To be done, until then you can read chapters 13, 14 and the epilogue of the book Four thousand weeks by Oliver Burkman.

    Refactor your gathered thoughts

    If you've followed the prepare steps, you've already been making up your mind on what do you want the next trimester to look like. Now it's the time to refine those thoughts.

    In your roadmap document add a new section for the incoming trimester similar to:

    * Roadmap
** 2024
*** Summer 2024
**** Essential intent
**** Trimester analysis
**** Trimester objectives
***** TODO Objective 1
****** TODO SubObjective 1
    Go one by one (don't peek!) of your gathered items and translate them in the next sections:

    • Trimester analysis: A text with as many paragraphs as you need to order your thoughts
    • Trimester objectives: These can be concrete emotional projects you want to carry through.
    • Essential intent: This is the main headline of your trimester, probably you won't be able to define it until the last parts of the review process. It should be concrete and emotional too, it's going to be the idea that gives you strength on your weak moments and your guide to decide which projects to do and which not to.

    Don't be too concerned on the format of the content of the objectives, this is the first draft, and we'll refine it through the planning.

  • New: Wordpress plugin.

    This plugin allows you to embed a list of events or a single event from your Gancio website using a shortcode. It also allows you to connects a Gancio instance to a your wordpress website to automatically push events published on WordPress: for this to work an event manager plugin is required, Event Organiser and The Events Calendar are supported. Adding another plugin it’s an easy task and you have a guide available in the repo that shows you how to do it.

    The source code of the plugin is in the wp-plugin directory of the official repo

Habit management⚑

  • New: Introduce habit management.

    A habit is a routine of behavior that is repeated regularly and tends to occur subconsciously.

    A 2002 daily experience study found that approximately 43% of daily behaviors are performed out of habit. New behaviours can become automatic through the process of habit formation. Old habits are hard to break and new habits are hard to form because the behavioural patterns that humans repeat become imprinted in neural pathways, but it is possible to form new habits through repetition.

    When behaviors are repeated in a consistent context, there is an incremental increase in the link between the context and the action. This increases the automaticity of the behavior in that context. Features of an automatic behavior are all or some of: efficiency, lack of awareness, unintentionality, and uncontrollability.

    Mastering habit formation can be a powerful tool to change yourself. Usually with small changes you get massive outcomes in the long run. The downside is that it's not for the impatient people as it often appears to make no difference until you cross a critical threshold that unlocks a new level of performance.

  • New: Why are habits interesting.

    Whenever you face a problem repeatedly, your brain begins to automate the process of solving it. Habits are a series of automatic resolutions that solve the problems and stresses you face regularly.

    As habits are created, the level of activity in the brain decreases. You learn to lock in on the cues that predict success and tune out everything else. When a similar situation arises in the future, you know exactly what you look for. There is no longer a need to analyze every angle of a situation. Your brain skips the process of trial and error and creates a mental rule: if this, then that.

    Habit formation is incredibly useful because the conscious mind is the bottleneck of the brain. It can only pay attention to one problem at a time. Habits reduce the cognitive load and free up mental capacity, so they can be carried on with your nonconscious mind and you can allocate your attention to other tasks.

  • New: Identity focused changes.

    Changing our habits is challenging because we try to change the wrong thing in the wrong way.

    There are three levels at which change can occur:

    • Outcomes: Changing your results. Goals fall under this category: publishing a book, run daily
    • Process: Changing your habits and systems: decluttering your desk for a better workflow, developing a meditation practice.
    • Identity: Changing your beliefs, assumptions and biases: your world view, your self-image, your judgments.

    Many people begin the process of changing their habits by focusing on what they want to achieve. This leads to outcome-based habits. The alternative is to build identity-based habits. With this approach, we start by focusing on who we wish to become.

    The first path of change is doomed because maintaining behaviours that are incongruent with the self is expensive and will not last. Even if they make rational sense. Thus it's hard to change your habits if you never change the underlying beliefs that led to your past behaviour. On the other hand it's easy to find motivation once a habit has changed your identity as you may be proud of it and will be willing to maintain all the habits and systems associated with it. For example: The goal is not to read a book, but to become a reader.

    Focusing on outcomes may also bring the next problems:

    • Focusing on the results may lead you to temporal solutions. If you focus on the source of the issue at hand you may solve it with less effort and get you to a more stable one.
    • Goals create an "either-or" conflict: either you achieve your goal and are successful or you fail and you are disappointed. Thus you only get a positive reward if you fulfill a goal. If you instead focus on the process rather than the result, you will be satisfied anytime your system is running.
    • When your hard work is focused on a goal you may feel depleted once you meet it and that could make you loose the condition that made you meet the goal in the first place.

    Research has shown that once a person believes in a particular aspect of their identity, they are more likely to act in alignment with that belief. This of course is a double-edged sword. Identity change can be a powerful force for self-improvement. When working against you, identity change can be a curse.

  • New: Changing your identity.

    Whatever your identity is right now, you only believe it because you have proof of it. The more evidence you have for a belief, the more strongly you will believe it.

    Your habits and systems are how you embody your identity. When you make your bed each day, you embody the identity of an organized person. The more you repeat a behaviour, the more you reinforce the identity associated with that behaviour. To the point that your self-image begins to change. The effect of one-off experiences tends to fade away while the effect of habits gets reinforced with time, which means your habits contribute most of the evidence that shapes your identity.

    Every action you take is a vote for the type of person you wish to become. This is one reason why meaningful change does not require radical change. Small habits can make a meaningful difference by providing evidence of a new identity.

    Once you start the ball rolling things become easier as building habits is a feedback loop. Your habits shape your identity, and your identity shapes your habits.

    The most practical way to change the identity is to:

    Another advantage of focusing in what type of person you want to be is that maybe the outcome you wanted to focus on is not the wisest smallest step to achieve your identity change. Thinking on the identity you want to embrace can make you think outside the box.

  • New: Decide the type of person you want to be.

    One way to decide the person you want to be is to answer big questions like: what do you want to stand for? What are your principles and values? Who do you wish to become?

    As we're more result oriented, another way is to work backwards from them to the person you want to be. Ask yourself: Who is the type of person that could get the outcome I want?

  • New: How to change a habit.

    The process of building a habit from a behaviour can be divided into four stages:

    • Reward is the end goal.
    • Cue is the trigger in your brain that initiate a behaviour. It's contains the information that predicts a reward.
    • Cravings are the motivational force fueled by the desire of the reward. Without motivation we have no reason to act.
    • Response is the thought or action you perform to obtain the reward. The response depends on the amount of motivation you have, how much friction is associated with the behaviour and your ability to actually do it.

    If a behaviour is insufficient in any of the four stages, it will not become a habit. Eliminate the cue and your habit will never start. Reduce the craving and you won't have enough motivation to act. Make the behaviour difficult and you won't be able to do it. And if the reward fails to satisfy your desire, then you'll have no reason to do it again in the future.

    We chase rewards because they:

    • Deliver contentment.
    • Satisfy your craving.
    • Teach us which actions are worth remembering in the future.

    If a reward is met then it becomes associated with the cue, thus closing the habit feedback loop.

    If we keep these stages in mind then:

    • To build good habits we need to:

      • Cue: Make it obvious
      • Craving: Make it attractive
      • Response: Make it easy
      • Reward: Make it satisfying

    • To break bad habits we need to:

      • Cue: Make it invisible
      • Craving: Make it unattractive
      • Response: Make it difficult
      • Reward: Make it unsatisfying

  • New: Select which habits you want to work with.

    Our responses to the cues are so deeply encoded that it may feel like the urge to act comes from nowhere. For this reason, we must begin the process of behavior change with awareness. Before we can effectively build new habits, we need to get a handle on our current ones. The author suggests to do a list of your daily habits and rate them positively, negatively or neutral under the judgement of whether it brings you closer to the desired person you want to be.

    I find this approach expensive time-wise if you already have a huge list of habits to work with. As it's my case I'll skip this part. You can read it in more detail in the chapter "4: The Man Who Didn't Look Right".

  • New: Working with the habit cues.

    The first place to start the habit design is to understand and tweak the triggers that produce them. We'll do it by:

  • New: Clearly formulate the habit you want to change.

    The cues that can trigger an habit can come in a wide range of forms but the two most common are time and location. Being specific about what you want and how you will achieve it helps you say no to things that derail progress, distract your attention and pull you off course. And with enough repetition, you will get the urge to do the right thing at the right time, even if you can't say why. That's why it's interesting to formulate your habits as "I will [behaviour] at [time] in [location]".

    You want the cue to be highly specific and immediately actionable. If there is room for doubt the implementation will suffer. Continuously refine the habit definitions as you catch the exceptions that drift you off.

    If you aren't sure of when to start your habit, try the first day of the week, month or year. People are more likely to take action at those times because hope is usually higher as you get the feeling of a fresh start.

  • New: Habit stacking.

    Many behaviours are linked together where the action of the first is the cue that triggers the next one. You can use this connection to build new habits based on your established ones. This may be called habit stacking. The formulation in this case is "After [current habit], I will [new habit]".

    The key is to tie your desired behaviour into something you already do each day. Once you have mastered this basic structure, you can begin to create larger stacks by chaining small habits together. The catch is that the new habit should have the same frequency as the established one.

    One way to find the right trigger for your habit stack is by brainstorming over:

    • The list of your current habits.
    • A new list of things that always happen to you with that frequency.

    With these two lists, you can begin searching for the best triggers for the stack.

  • New: Use the environment to tweak your cues.

    The cues that trigger a habit can start out very specific, but over time your habits become associated not with a single trigger but with the entire context surrounding the behaviour. This stacks over itself and your habits change depending on the room you are in and the cues in front of you. The context or the environment is then the invisible hand that shapes behaviours. They are not defined by the objects in the environment but by our relationship to them.

    A new environment is a good foundation to make new habits, as you are free from the subtle triggers that nudge you toward your current habits. When you can't manage to get an entirely new environment, you can redefine or rearrange your current one.

    When building good habits you can rearrange the environment to create obvious visual cues that draw your attention towards the desired habit. By sprinkling triggers throughout your surroundings, you increase the odds that you'll think about your habit throughout the day.

    Once a habit has been encoded, the urge to act follows whenever the environmental cues reappear. This is why bad habits reinforce themselves. As you carry through the behaviour you spiral into a situation where the craving keeps growing and points you to keep on going with the same response. For example watching TV makes you feel sluggish, so you watch more television because you don't have the energy to do anything else.

    Even if you manage to break a habit, you are unlikely to forget it's cues even if you don't do it for a while. That means that simply resisting temptation is an ineffective strategy. In the short run it may work. In the long run, as self-control is an exhausting task that consumes willpower, we become a product of the environment we live in. Trying to change a habit with self-control is doomed to fail as you may be able to resist temptation once or twice, but it's unlikely you can muster the willpower to override your desires every time. It's also very hard and frustrating to try to achieve change when you're under the mood influences of a bad habit.

    A more reliable approach is to cut bad habits off at the source. Tweak the environment to make the cue virtually impossible to happen. That way you won't even have the chance to fall for the craving.

  • New: Temptation bundling.

    Dopamine is a neurotransmitter that can be used as the scientific measurement of craving. For years we assumed that it was all about pleasure, but now we know it plays a central role in many neurological processes, including motivation, learning and memory, punishment and aversion and voluntary movement.

    Habits are a dopamine-driven feed back loop. It is released not only when you receive a reward but also when you anticipate it. This anticipation, and not the fulfillment of it, is what gets us to take action.

    If we make a habit more attractive it will release more dopamine which will gives us more motivation to carry it through.

    Temptation bundling works by pairing an action you want to do with an action you need to do. You're more likely to find a behaviour attractive if you get to do one of your favourite things at the same time. In the end you may even look forward to do the habit you need as it's related to the habit you want.

  • New: Align your personal identity change with an existent shared identity.

    We pick up habits from the people around us. As a general rule, the closer we are to someone, the more likely we are to imitate some of their habits. One of the most effective things you can do to build better habits is to join a culture where your desired behaviour is the normal one. This transforms your personal identity transformation into the building of a shared one. Shared identities have great benefits over single ones:

    • They foster belonging. A powerful feeling that creates motivation.
    • They are more resilient: When one falters others will take their place so all together you'll guarantee the maintenance of the identity.
    • They create friendship and community
    • They expose you to an environment where more habits tied to that identity thrive.

    Likewise, if you're trying to run from a bad habit cut your ties to communities that embrace that habit.

  • New: Track your habit management.

    You can have a habits.org file where you prioritize, analyze, track them.

    I'm using the next headings:

    • Habits being implemented: It's subdivided in two:
    • Habits that need attention
    • Habits that don't need attention
    • Unclassified habits: Useful when refiling habits from your inbox. This list will be analyzedwhen you do habit analysis.
    • Backlog of habits: Unrefined and unordered list of habits
    • Implemented habits:
    • Rejected habits:

    Each habit is a TODO item with the usual states: TODO, DOING, DONE, REJECTED. In it's body I keep a log of the evolution and the analysis of the habit.

  • New: Habit management workflow.

    Each month I'm trying to go through the list of habits to:

    • Update the state of the habits: Some will be done, rejected or to register ideas about them.
    • Decide which ones need attention.
    • Do habit analysis on the ones that need attention.

    For each of the habits that need analysis, apply the learnings of the next sections:

Calendar management⚑

  • New: Add calendar event notification system tool.

    Set up a system that notifies you when the next calendar event is about to start to avoid spending mental load on it and to reduce the possibilities of missing the event.

    I've created a small tool that:

    • Tells me the number of pomodoros that I have until the next event.
    • Once a pomodoro finishes it makes me focus on the amount left so that I can prepare for the event
    • Catches my attention when the event is starting.

Life chores management⚑

himalaya⚑

  • New: Doing the inventory review.

    I haven't found a way to make the grocy inventory match the reality because for me it's hard to register when I consume a product. Even more if other people also use them. Therefore I use grocy only to know what to buy without thinking about it. For that use case the inventory needs to meet reality only before doing the groceries. I usually do a big shopping of non-perishable goods at the supermarket once each two or three months, and a weekly shopping of the rest.

    Tracking the goods that are bought each week makes no sense as those are things that are clearly seen and are very variable depending on the season. Once I've automated the ingestion and consumption of products it will, but so far it would mean investing more time than the benefit it gives.

    This doesn't apply to the big shopping, as this one is done infrequently, so we need a better planning.

    To do the inventory review I use a tablet and the android app.

    • Open the stock overview and iterate through the locations to:
    • Make sure that the number of products match the reality
      • Iterate over the list of products checking the quantity
      • Look at the location to see if there are missing products in the inventory
    • Adjust the product properties (default location, minimum amount)
    • Check the resulting shopping list and adjust the minimum values.
    • Check the list of missing products to adjust the minimum values. I have a notepad in the fridge where I write the things I miss.

  • New: Introduce route management.

    To analyze which hiking routes are available in a zone I'm following the next process

    • Check in my trips orgmode directory to see if the zone has already been indexed.
    • Do a first search of routes
    • Identify which books or magazines describe the zone
    • For each of the described routes in each of these books:
      • Create the Routes section with tag :route: if it doesn't exist
      • Fill up the route form in a TODO heading. Something similar to: 
        Reference: Book Page
Source: Where does it start
Distance: X km
Slope: X m
Type: [Lineal/Circular/Semi-lineal]
Difficulty:
Track: URL (only if you don't have to search for it)
      • Add tags of the people I'd like to do it with
      • Put a postit on the book/magazine if it's likely I'm going to do it
      • Open a web maps tab with the source of the route to calculate the time from the different lodgins

    • If there are not enough, repeat the process above for each of your online route reference blogs

    • Choose the routes to do

    • Show the gathered routes to the people you want to go with

    • Select which ones you'll be more likely to do

    • For each of the chosen routes

    • Search the track in wikiloc if it's missing
    • Import the track in OsmAnd+

  • New: Add API and python library docs.

    There is no active python library, although it existed pygrocy

  • New: Introduce himalaya.

    himalaya is a Rust CLI to manage emails.

    Features:

    • Multi-accounting
    • Interactive configuration via wizard (requires wizard feature)
    • Mailbox, envelope, message and flag management
    • Message composition based on $EDITOR
    • IMAP backend (requires imap feature)
    • Maildir backend (requires maildir feature)
    • Notmuch backend (requires notmuch feature)
    • SMTP backend (requires smtp feature)
    • Sendmail backend (requires sendmail feature)
    • Global system keyring for managing secrets (requires keyring feature)
    • OAuth 2.0 authorization (requires oauth2 feature)
    • JSON output via --output json
    • PGP encryption:
    • via shell commands (requires pgp-commands feature)
    • via GPG bindings (requires pgp-gpg feature)
    • via native implementation (requires pgp-native feature)

    Cons:

    • Documentation is inexistent, you have to dive into the --help to understand stuff.

    Installation

    The v1.0.0 is currently being tested on the master branch, and is the prefered version to use. Previous versions (including GitHub beta releases and repositories published versions) are not recommended.

    Himalaya CLI v1.0.0 can be installed with a pre-built binary. Find the latest pre-release GitHub workflow and look for the Artifacts section. You should find a pre-built binary matching your OS.

    Himalaya CLI v1.0.0 can also be installed with cargo:

    $ cargo install --git https://github.com/pimalaya/himalaya.git --force himalaya
    Configuration

    Just run himalaya, the wizard will help you to configure your default account.

    You can also manually edit your own configuration, from scratch:

    • Copy the content of the documented ./config.sample.toml
    • Paste it in a new file ~/.config/himalaya/config.toml
    • Edit, then comment or uncomment the options you want

    If using mbrsync

    My generic configuration for an mbrsync account is:

    [accounts.account_name]

email = "lyz@example.org"
display-name = "lyz"
envelope.list.table.unseen-char = "u"
envelope.list.table.replied-char = "r"
backend.type = "maildir"
backend.root-dir = "/home/lyz/.local/share/mail/lyz-example"
backend.maildirpp = false
message.send.backend.type = "smtp"
message.send.backend.host = "example.org"
message.send.backend.port = 587
message.send.backend.encryption = "start-tls"
message.send.backend.login = "lyz"
message.send.backend.auth.type = "password"
message.send.backend.auth.command = "pass show mail/lyz.example"

    Once you've set it then you need to fix the INBOX directory.

    Then you can check if it works by running himalaya envelopes list -a lyz-example

    Vim plugin installation

    Using lazy:

    return {
  {
    "pimalaya/himalaya-vim",
  },
}

    You can then run :Himalaya account_name and it will open himalaya in your editor.

    Configure the account bindings

    To avoid typing :Himalaya account_name each time you want to check the email you can set some bindings:

    return {
  {
    "pimalaya/himalaya-vim",
    keys = {
      { "<leader>ma", "<cmd>Himalaya account_name<cr>", desc = "Open account_name@example.org" },
      { "<leader>ml", "<cmd>Himalaya lyz<cr>", desc = "Open lyz@example.org" },
    },
  },
}

    Setting the description is useful to see the configured accounts with which-key by typing <leader>m and waiting.

    Configure extra bindings

    The default plugin doesn't yet have all the bindings I'd like so I've added the next ones:

    • In the list of emails view:
    • dd in normal mode or d in visual: Delete emails

    • q: exit the program

    • In the email view:

    • d: Delete email
    • q: Return to the list of emails view

    If you want them too set the next config:

    return {
  {
    "pimalaya/himalaya-vim",
    config = function()
      vim.api.nvim_create_augroup("HimalayaCustomBindings", { clear = true })
      vim.api.nvim_create_autocmd("FileType", {
        group = "HimalayaCustomBindings",
        pattern = "himalaya-email-listing",
        callback = function()
          -- Bindings to delete emails
          vim.api.nvim_buf_set_keymap(0, "n", "dd", "<plug>(himalaya-email-delete)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "x", "d", "<plug>(himalaya-email-delete)", { noremap = true, silent = true })
          -- Bind `q` to close the window
          vim.api.nvim_buf_set_keymap(0, "n", "q", ":bd<CR>", { noremap = true, silent = true })
        end,
      })

      vim.api.nvim_create_augroup("HimalayaEmailCustomBindings", { clear = true })
      vim.api.nvim_create_autocmd("FileType", {
        group = "HimalayaEmailCustomBindings",
        pattern = "mail",
        callback = function()
          -- Bind `q` to close the window
          vim.api.nvim_buf_set_keymap(0, "n", "q", ":q<CR>", { noremap = true, silent = true })
          -- Bind `d` to delete the email and close the window
          vim.api.nvim_buf_set_keymap(
            0,
            "n",
            "d",
            "<plug>(himalaya-email-delete):q<CR>",
            { noremap = true, silent = true }
          )
        end,
      })
    end,
  },
}

    Configure email fetching from within vim

    Fetching emails from within vim is not yet supported, so I'm manually refreshing by account:

    return {
  {
    "pimalaya/himalaya-vim",
    keys = {
      -- Email refreshing bindings
      { "<leader>rj", ':lua FetchEmails("lyz")<CR>', desc = "Fetch lyz@example.org" },
    },
    config = function()
      function FetchEmails(account)
        vim.notify("Fetching emails for " .. account .. ", please wait...", vim.log.levels.INFO)
        vim.cmd("redraw")
        vim.fn.jobstart("mbsync " .. account, {
          on_exit = function(_, exit_code, _)
            if exit_code == 0 then
              vim.notify("Emails for " .. account .. " fetched successfully!", vim.log.levels.INFO)
            else
              vim.notify("Failed to fetch emails for " .. account .. ". Check the logs.", vim.log.levels.ERROR)
            end
          end,
        })
      end
    end,
  },
}

    You still need to open again :Himalaya account_name as the plugin does not reload if there are new emails.

    Show notifications when emails arrive

    You can set up mirador to get those notifications.

    Not there yet

    Troubleshooting

    Cannot find maildir matching name INBOX

    mbrsync uses Inbox instead of the default INBOX so it doesn't find it. In theory you can use folder.alias.inbox = "Inbox" but it didn't work with me, so I finally ended up doing a symbolic link from INBOX to Inbox.

    Cannot find maildir matching name Trash

    That's because the Trash directory does not follow the Maildir structure. I had to create the cur tmp and new directories.

    References - Source - Vim plugin source

  • New: Introduce mailbox.

    mailbox is a python library to work with MailDir and mbox local mailboxes.

    It's part of the core python libraries, so you don't need to install anything.

    Usage

    The docs are not very pleasant to read, so I got most of the usage knowledge from these sources:

    One thing to keep in mind is that an account can have many mailboxes (INBOX, Sent, ...), there is no "root mailbox" that contains all of the other

    initialise a mailbox

    mbox = mailbox.Maildir('path/to/your/mailbox')

    Where the path/to/your/mailbox is the directory that contains the cur, new, and tmp directories.

    Working with mailboxes

    It's not very clear how to work with them, the Maildir mailbox contains the emails in iterators [m for m in mbox], it acts kind of a dictionary, you can get the keys of the emails with [k for k in mbox.iterkeys], and then you can mbox[key] to get an email, you cannot modify those emails (flags, subdir, ...) directly in the mbox object (for example mbox[key].set_flags('P') doesn't work). You need to mail = mbox.pop(key), do the changes in the mail object and then mbox.add(mail) it again, with the downside that after you added it again, the key has changed! But it's the return value of the add method.

    If the program gets interrupted between the pop and the add then you'll loose the email. The best way to work with it would be then:

    • mail = mbox.get(key) the email
    • Do all the process you need to do with the email
    • mbox.pop(key) and key = mbox.add(mail)

    In theory mbox has an update method that does this, but I don't understand it and it doesn't work as expected :S.

    Moving emails around

    You can't just move the files between directories like you'd do with python as each directory contains it's own identifiers.

    Moving a message between the maildir directories

    The Message has a set_subdir

    Creating folders

    Even though you can create folders with mailbox it creates them in a way that mbsync doesn't understand it. It's easier to manually create the cur, tmp, and new directories. I'm using the next function:

    if not (mailbox_dir / "cur").exists():
    for dir in ["cur", "tmp", "new"]:
        (mailbox_dir / dir).mkdir(parents=True)
    log.info(f"Initialized mailbox: {mailbox}")
else:
    log.debug(f"{mailbox} already exists")

    References

  • New: Introduce maildir.

    The Maildir e-mail format is a common way of storing email messages on a file system, rather than in a database. Each message is assigned a file with a unique name, and each mail folder is a file system directory containing these files.

    A Maildir directory (often named Maildir) usually has three subdirectories named tmp, new, and cur.

    • The tmp subdirectory temporarily stores e-mail messages that are in the process of being delivered. This subdirectory may also store other kinds of temporary files.
    • The new subdirectory stores messages that have been delivered, but have not yet been seen by any mail application.
    • The cur subdirectory stores messages that have already been seen by mail applications.

    References

  • New: My emails are not being deleted on the source IMAP server.

    That's the default behavior of mbsync, if you want it to actually delete the emails on the source you need to add:

    Expunge Both
    Under your channel (close to Sync All, Create Both)

  • New: Mbsync error: UID is beyond highest assigned UID.

    If during the sync you receive the following errors:

    mbsync error: UID is 3 beyond highest assigned UID 1

    Go to the place where mbsync is storing the emails and find the file that is giving the error, you need to find the files that contain U=3, imagine that it's something like 1568901502.26338_1.hostname,U=3:2,S. You can strip off everything from the ,U= from that filename and resync and it should be fine, e.g.

    mv '1568901502.26338_1.hostname,U=3:2,S' '1568901502.26338_1.hostname'
    feat(mirador): introduce mirador

    DEPRECATED: as of 2024-11-15 the tool has many errors (1, 2), few stars (4) and few commits (8). use watchdog instead and build your own solution.

    mirador is a CLI to watch mailbox changes made by the maintaner of himalaya.

    Features:

    • Watches and executes actions on mailbox changes
    • Interactive configuration via wizard (requires wizard feature)
    • Supported events: on message added.
    • Supported actions: send system notification, execute shell command.
    • Supports IMAP mailboxes (requires imap feature)
    • Supports Maildir folders (requires maildir feature)
    • Supports global system keyring to manage secrets (requires keyring feature)
    • Supports OAuth 2.0 (requires oauth2 feature)

    Mirador CLI is written in Rust, and relies on cargo features to enable or disable functionalities. Default features can be found in the features section of the Cargo.toml.

    Installation

    The v1.0.0 is currently being tested on the master branch, and is the preferred version to use. Previous versions (including GitHub beta releases and repositories published versions) are not recommended.*

    Cargo (git)

    Mirador CLI v1.0.0 can also be installed with cargo:

    $ cargo install --frozen --force --git https://github.com/pimalaya/mirador.git

    Pre-built binary

    Mirador CLI v1.0.0 can be installed with a pre-built binary. Find the latest pre-release GitHub workflow and look for the Artifacts section. You should find a pre-built binary matching your OS.

    Configuration

    Just run mirador, the wizard will help you to configure your default account.

    You can also manually edit your own configuration, from scratch:

    • Copy the content of the documented ./config.sample.toml
    • Paste it in a new file ~/.config/mirador/config.toml

    • Edit, then comment or uncomment the options you want

    • Source

  • New: Configure navigation bindings.

    The default bindings conflict with my git bindings, and to make them similar to orgmode agenda I'm changing the next and previous page:

    return {
  {
    "pimalaya/himalaya-vim",
    keys = {
      { "b", "<plug>(himalaya-folder-select-previous-page)", desc = "Go to the previous email page" },
      { "f", "<plug>(himalaya-folder-select-next-page)", desc = "Go to the next email page" },
    },
  },
}

  • Correction: Configure the account bindings.

  • Correction: Tweak the bindings.

    Move forward and backwards in the history of emails:

        vim.api.nvim_create_autocmd("FileType", {
      group = "HimalayaCustomBindings",
      pattern = "himalaya-email-listing",
      callback = function()
        vim.api.nvim_buf_set_keymap(0, "n", "b", "<plug>(himalaya-folder-select-previous-page)", { noremap = true, silent = true })
        vim.api.nvim_buf_set_keymap(0, "n", "f", "<plug>(himalaya-folder-select-next-page)", { noremap = true, silent = true })
      end,
    })
    Better bindings for the email list view:

              -- Refresh emails
          vim.api.nvim_buf_set_keymap(0, "n", "r", ":lua FetchEmails()<CR>", { noremap = true, silent = true })
          -- Email list view bindings
          vim.api.nvim_buf_set_keymap(0, "n", "b", "<plug>(himalaya-folder-select-previous-page)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "f", "<plug>(himalaya-folder-select-next-page)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "R", "<plug>(himalaya-email-reply-all)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "F", "<plug>(himalaya-email-forward)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "m", "<plug>(himalaya-folder-select)", { noremap = true, silent = true })
          vim.api.nvim_buf_set_keymap(0, "n", "M", "<plug>(himalaya-email-move)", { noremap = true, silent = true })
    feat(himalaya#Searching emails): Searching emails

    You can use the g/ binding from within nvim to search for emails. The query syntax supports filtering and sorting query.

    I've tried changing it to / without success :'(

    Filters

    A filter query is composed of operators and conditions. There is 3 operators and 8 conditions:

    • not <condition>: filter envelopes that do not match the condition
    • <condition> and <condition>: filter envelopes that match both conditions
    • <condition> or <condition>: filter envelopes that match one of the conditions
    • date <yyyy-mm-dd>: filter envelopes that match the given date
    • before <yyyy-mm-dd>: filter envelopes with date strictly before the given one
    • after <yyyy-mm-dd>: filter envelopes with date stricly after the given one
    • from <pattern>: filter envelopes with senders matching the given pattern
    • to <pattern>: filter envelopes with recipients matching the given pattern
    • subject <pattern>: filter envelopes with subject matching the given pattern
    • body <pattern>: filter envelopes with text bodies matching the given pattern
    • flag <flag>: filter envelopes matching the given flag

    Sorting A sort query starts by "order by", and is composed of kinds and orders. There is 4 kinds and 2 orders:

    • date [order]: sort envelopes by date
    • from [order]: sort envelopes by sender
    • to [order]: sort envelopes by recipient
    • subject [order]: sort envelopes by subject
    • <kind> asc: sort envelopes by the given kind in ascending order
    • <kind> desc: sort envelopes by the given kind in descending order

    Examples

    subject foo and body bar: filter envelopes containing "foo" in their subject and "bar" in their text bodies order by date desc subject: sort envelopes by descending date (most recent first), then by ascending subject subject foo and body bar order by date desc subject: combination of the 2 previous examples

  • New: List more detected issues.

  • New: Troubleshoot cannot install the program.

    Sometimes the installation steps fail as it's still not in stable. A workaround is to download the binary created by the pre-release CI. You can do it by:

    • Click on the latest job
    • Click on jobs
    • Click on the job of your architecture
    • Click on "Upload release"
    • Search for "Artifact download URL" and download the file
    • Unpack it and add it somewhere in your $PATH

alot⚑

  • New: Comments.

    Any text on a line after the character ; is ignored, text like this:

    ; I paid and left the taxi, forgot to take change, it was cold.
2015-01-01 * "Taxi home from concert in Brooklyn"
  Assets:Cash      -20 USD  ; inline comment
  Expenses:Taxi

  • New: Introduce matrix_highlight.

    Matrix Highlight is a decentralized and federated way of annotating the web based on Matrix.

    Think of it as an open source alternative to hypothesis.

    It's similar to Populus but for the web.

    I want to try it and investigate further specially if you can:

    • Easily extract the annotations
    • Activate it by default everywhere

  • New: Get the schema of a table.

    Postgres:

    \d+ table_name

  • New: Get the last row of a table.

    SELECT * FROM Table ORDER BY ID DESC LIMIT 1

  • New: Introduce beanSQL.

    bean-sql is a language to query beancount data.

    References: - Docs - Examples

  • New: Get the quarter of a date.

    Use the quarter(date) selector in the SELECT . For example:

    SELECT quarter(date) as quarter, sum(position) AS value
WHERE
    account ~ '^Expenses:' OR
    account ~ '^Income:'
GROUP BY quarter

    It will return the quarter in the format YYYY-QX.

  • New: Building your own dashboards.

    I was wondering whether to create fava dashboards or to create them directly in grafana.

    Pros of fava dashboards: - They are integrated in fava so it would be easy to browse other beancount data. Although this could be done as well in another window if I used grafana. - There is no need to create the beancount grafana data source logic. - It's already a working project, I would need just to tweak an existent example.

    Cons: - I may need to learn echarts and write JavaScript to tweak some of the dashboards. - I wouldn't have all my dashboards in the same place. - It only solves part of the problem, I'd still need to write the bean-sql queries. But using beanql is probably the best way to extract data from beancount anyway. - It involves more magic than using grafana. - grafana dashboards are prettier. - I wouldn't use the grafana knowledge. - I'd learn a new tool only to use it here instead of taking the chance to improve my grafana skillset.

    I'm going to try with fava dashboards and see how it goes

  • Correction: Deprecate in favour of himalaya.

    DEPRECATED: Use himalaya instead.

  • New: Automatically sync emails.

    I have many emails, and I want to fetch them with different frequencies, in the background and be notified if anything goes wrong.

    For that purpose I've created a python script, a systemd service and some loki rules to monitor it.

    Script to sync emails and calendars with different frequencies

    The script iterates over the configured accounts in accounts_config and runs mbsync for email accounts and vdirsyncer for email accounts based on some cron expressions. It logs the output in logfmt format so that it's easily handled by loki

    To run it you'll first need to create a virtualenv, I use mkvirtualenv account_syncer which creates a virtualenv in ~/.local/share/virtualenv/account_syncer.

    Then install the dependencies:

    pip install aiocron

    Then place this script somewhere, for example (~/.local/bin/account_syncer.py)

    import asyncio
import logging
from datetime import datetime
import asyncio.subprocess
import aiocron

accounts_config = {
    "emails": [
      {
          "account_name": "lyz",
          "cron_expressions": ["*/15 9-23 * * *"],
      },
      {
          "account_name": "work",
          "cron_expressions": ["*/60 8-17 * * 1-5"],  # Monday-Friday
      },
      {
          "account_name": "monitorization",
          "cron_expressions": ["*/5 * * * *"],
      },
    ],
    "calendars": [
      {
          "account_name": "lyz",
          "cron_expressions": ["*/15 9-23 * * *"],
      },
      {
          "account_name": "work",
          "cron_expressions": ["*/60 8-17 * * 1-5"],  # Monday-Friday
      },
    ],
}

class LogfmtFormatter(logging.Formatter):
    """Custom formatter to output logs in logfmt style."""

    def format(self, record: logging.LogRecord) -> str:
        log_message = (
            f"level={record.levelname.lower()} "
            f"logger={record.name} "
            f'msg="{record.getMessage()}"'
        )
        return log_message

def setup_logging(logging_name: str) -> logging.Logger:
    """Configure logging to use logfmt format.
    Args:
        logging_name (str): The logger's name and identifier in the systemd journal.
    Returns:
        Logger: The configured logger.
    """
    console_handler = logging.StreamHandler()
    logfmt_formatter = LogfmtFormatter()
    console_handler.setFormatter(logfmt_formatter)
    logger = logging.getLogger(logging_name)
    logger.setLevel(logging.INFO)
    logger.addHandler(console_handler)
    return logger

log = setup_logging("account_syncer")

async def run_mbsync(account_name: str) -> None:
    """Run mbsync command asynchronously for email accounts.

    Args:
        account_name (str): The name of the email account to sync.
    """
    command = f"mbsync {account_name}"
    log.info(f"Syncing emails for {account_name}...")
    process = await asyncio.create_subprocess_shell(
        command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
    )
    stdout, stderr = await process.communicate()
    if stdout:
        log.info(f"Output for {account_name}: {stdout.decode()}")
    if stderr:
        log.error(f"Error for {account_name}: {stderr.decode()}")

async def run_vdirsyncer(account_name: str) -> None:
    """Run vdirsyncer command asynchronously for calendar accounts.

    Args:
        account_name (str): The name of the calendar account to sync.
    """
    command = f"vdirsyncer sync {account_name}"
    log.info(f"Syncing calendar for {account_name}...")
    process = await asyncio.create_subprocess_shell(
        command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
    )
    _, stderr = await process.communicate()
    if stderr:
        command_log = stderr.decode().strip()
        if "error" in command_log or "critical" in command_log:
            log.error(f"Output for {account_name}: {command_log}")
        elif len(command_log.splitlines()) > 1:
            log.info(f"Output for {account_name}: {command_log}")

def should_i_sync_today(cron_expr: str) -> bool:
    """Check if the current time matches the cron expression day and hour constraints."""
    _, hour, _, _, day_of_week = cron_expr.split()
    now = datetime.now()
    if "*" in hour:
        return True
    elif not (int(hour.split("-")[0]) <= now.hour <= int(hour.split("-")[1])):
        return False
    if day_of_week != "*" and str(now.weekday()) not in day_of_week.split(","):
        return False
    return True

async def main():
    log.info("Starting account syncer for emails and calendars")
    accounts_to_sync = {"emails": [], "calendars": []}

    # Schedule email accounts
    for account in accounts_config["emails"]:
        account_name = account["account_name"]
        for cron_expression in account["cron_expressions"]:
            if (
                should_i_sync_today(cron_expression)
                and account_name not in accounts_to_sync["emails"]
            ):
                accounts_to_sync["emails"].append(account_name)
            aiocron.crontab(cron_expression, func=run_mbsync, args=[account_name])
            log.info(
                f"Scheduled mbsync for {account_name} with cron expression: {cron_expression}"
            )

    # Schedule calendar accounts
    for account in accounts_config["calendars"]:
        account_name = account["account_name"]
        for cron_expression in account["cron_expressions"]:
            if (
                should_i_sync_today(cron_expression)
                and account_name not in accounts_to_sync["calendars"]
            ):
                accounts_to_sync["calendars"].append(account_name)
            aiocron.crontab(cron_expression, func=run_vdirsyncer, args=[account_name])
            log.info(
                f"Scheduled vdirsyncer for {account_name} with cron expression: {cron_expression}"
            )

    log.info("Running an initial fetch on today's accounts")
    for account_name in accounts_to_sync["emails"]:
        await run_mbsync(account_name)
    for account_name in accounts_to_sync["calendars"]:
        await run_vdirsyncer(account_name)

    log.info("Finished loading accounts")
    while True:
        await asyncio.sleep(60)

if __name__ == "__main__":
    asyncio.run(main())

    Where:

    • accounts_config: Holds your account configuration. Each account must contain an account_name which should be the name of the mbsync or vdirsyncer profile, and cron_expressions must be a list of cron valid expressions you want the email to be synced.

    Create the systemd service

    We're using a non-root systemd service. You can follow these instructions to configure this service:

    [Unit]
Description=Account Sync Service for emails and calendars
After=graphical-session.target

[Service]
Type=simple
ExecStart=/home/lyz/.local/share/virtualenvs/account_syncer/bin/python /home/lyz/.local/bin/
WorkingDirectory=/home/lyz/.local/bin
Restart=on-failure
StandardOutput=journal
StandardError=journal
SyslogIdentifier=account_syncer
Environment="PATH=/home/lyz/.local/share/virtualenvs/account_syncer/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="DISPLAY=:0"
Environment="DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"

[Install]
WantedBy=graphical-session.target

    Remember to tweak the service to match your current case and paths.

    As we'll probably need to enter our pass password we need the service to start once we've logged into the graphical interface.

    Monitor the automation

    It's always nice to know if the system is working as expected without adding mental load. To do that I'm creating the next loki rules:

    groups:
  - name: account_sync
    rules:
      - alert: AccountSyncIsNotRunningWarning
        expr: |
          (sum by(hostname) (count_over_time({job="systemd-journal", syslog_identifier="account_syncer"}[15m])) or sum by(hostname) (count_over_time({hostname="my_computer"} [15m])) * 0 ) == 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "The account sync script is not running {{ $labels.hostname}}"
      - alert: AccountSyncIsNotRunningError
        expr: |
          (sum by(hostname) (count_over_time({job="systemd-journal", syslog_identifier="account_syncer"}[3h])) or sum by(hostname) (count_over_time({hostname="my_computer"} [3h])) * 0 ) == 0
        for: 0m
        labels:
          severity: error
        annotations:
          summary: "The account sync script has been down for at least 3 hours {{ $labels.hostname}}"
      - alert: AccountSyncError
        expr: |
          count(rate({job="systemd-journal", syslog_identifier="account_syncer"} |= `` | logfmt | level_extracted=`error` [5m])) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "There are errors in the account sync log at {{ $labels.hostname}}"

      - alert: EmailAccountIsOutOfSyncLyz
        expr: |
          (sum by(hostname) (count_over_time({job="systemd-journal", syslog_identifier="account_syncer"} | logfmt | msg=`Syncing emails for lyz...`[1h])) or sum by(hostname) (count_over_time({hostname="my_computer"} [1h])) * 0 ) == 0
        for: 0m
        labels:
          severity: error
        annotations:
          summary: "The email account lyz has been out of sync for 1h {{ $labels.hostname}}"

      - alert: CalendarAccountIsOutOfSyncLyz
        expr: |
          (sum by(hostname) (count_over_time({job="systemd-journal", syslog_identifier="account_syncer"} | logfmt | msg=`Syncing calendar for lyz...`[3h])) or sum by(hostname) (count_over_time({hostname="my_computer"} [3h])) * 0 ) == 0
        for: 0m
        labels:
          severity: error
        annotations:
          summary: "The calendar account lyz has been out of sync for 3h {{ $labels.hostname}}"
    Where: - You need to change my_computer for the hostname of the device running the service - Tweak the OutOfSync alerts to match your account (change the lyz part).

    These rules will raise: - A warning if the sync has not shown any activity in the last 15 minutes. - An error if the sync has not shown any activity in the last 3 hours. - An error if there is an error in the logs of the automation.

Fava Dashboards⚑

  • New: Disable automatic OMEMO key acceptance.

    Dino automatically accepts new OMEMO keys from your own other devices and your chat partners by default. This default behaviour leads to the fact that the admin of the XMPP server could inject own public OMEMO keys without user verification, which enables the owner of the associated private OMEMO keys to decrypt your OMEMO secured conversation without being easily noticed.

    To prevent this, two actions are required, the second consists of several steps and must be taken for each new chat partner.

    • First, the automatic acceptance of new keys from your own other devices must be deactivated. Configure this in the account settings of your own accounts.
    • Second, the automatic acceptance of new keys from your chat partners must be deactivated. Configure this in the contact details of every chat partner. Be aware that in the case of group chats, the entire communication can be decrypted unnoticed if even one partner does not actively deactivate automatic acceptance of new OMEMO keys.

    Always confirm new keys from your chat partner before accepting them manually

  • New: Dino does not use encryption by default.

    You have to initially enable encryption in the conversation window by clicking the lock-symbol and choose OMEMO. Future messages and file transfers to this contact will be encrypted with OMEMO automatically.

    • Every chat partner has to enable encryption separately.
    • If only one of two chat partner has activated OMEMO, only this part of the communication will be encrypted. The same applies with file transfers.
    • If you get a message "This contact does not support OMEMO" make sure that your chatpartner has accepted the request to add him to your contact list and you accepted vice versa

  • New: Install in Tails.

    If you have more detailed follow this article at the same time as you read this one. That one is more outdated but more detailed.

    • Boot a clean Tails
    • Create and configure the Persistent Storage

    • Restart Tails and open the Persistent Storage

    • Configure the persistence of the directory: 

      echo -e '/home/amnesia/.local/share/dino source=dino' | sudo tee -a /live/persistence/TailsData_unlocked/persistence.conf > /dev/null

    • Restart Tails

    • Install the application: 

      sudo apt-get update
sudo apt-get install dino-im

    • Configure the dino-im alias to use torsocks

      sudo echo 'alias dino="torsocks dino-im &> /dev/null &"' >> /live/persistence/TailsData_unlocked/dotfiles/.bashrc
echo 'alias dino="torsocks dino-im &> /dev/null &"' >> ~/.bashrc

  • New: Introduce Fava Dashboards.

    Installation

    pip install git+https://github.com/andreasgerstmayr/fava-dashboards.git

    Enable this plugin in Fava by adding the following lines to your ledger:

    2010-01-01 custom "fava-extension" "fava_dashboards"

    Then you'll need to create a dashboards.yaml file where your ledger lives.

    Configuration

    The plugin looks by default for a dashboards.yaml file in the directory of the Beancount ledger (e.g. if you run fava personal.beancount, the dashboards.yaml file should be in the same directory as personal.beancount).

    The configuration file can contain multiple dashboards, and a dashboard contains one or more panels. A panel has a relative width (e.g. 50% for 2 columns, or 33.3% for 3 column layouts) and a absolute height.

    The queries field contains one or multiple queries. The Beancount query must be stored in the bql field of the respective query. It can contain Jinja template syntax to access the panel and ledger variables described below (example: use {{ledger.ccy}} to access the first configured operating currency). The query results can be accessed via panel.queries[i].result, where i is the index of the query in the queries field.

    Note: Additionally to the Beancount query, Fava's filter bar further filters the available entries of the ledger.

    Common code for utility functions can be defined in the dashboards configuration file, either inline in utils.inline or in an external file defined in utils.path.

    Start your configuration

    It's best to tweak the example than to start from scratch. Get the example by: 

    cd $(mktemp -d)
git clone https://github.com/andreasgerstmayr/fava-dashboards
cd fava-dashboards/example
fava example.beancount

    Configuration reference

    HTML, echarts and d3-sankey panels: The script field must contain valid JavaScript code. It must return a valid configuration depending on the panel type. The following variables and functions are available: * ext: the Fava ExtensionContext * ext.api.get("query", {bql: "SELECT ..."}: executes the specified BQL query * panel: the current (augmented) panel definition. The results of the BQL queries can be accessed with panel.queries[i].result. * ledger.dateFirst: first date in the current date filter * ledger.dateLast: last date in the current date filter * ledger.operatingCurrencies: configured operating currencies of the ledger * ledger.ccy: shortcut for the first configured operating currency of the ledger * ledger.accounts: declared accounts of the ledger * ledger.commodities: declared commodities of the ledger * helpers.urlFor(url): add current Fava filter parameters to url * utils: the return value of the utils code of the dashboard configuration

    Jinja2 panels: The template field must contain valid Jinja2 template code. The following variables are available: * panel: see above * ledger: see above * favaledger: a reference to the FavaLedger object

    Common Panel Properties * title: title of the panel. Default: unset * width: width of the panel. Default: 100% * height: height of the panel. Default: 400px * link: optional link target of the panel header. * queries: a list of dicts with a bql attribute. * type: panel type. Must be one of html, echarts, d3_sankey or jinja2.

    HTML panel The script code of HTML panels must return valid HTML. The HTML code will be rendered in the panel.

    ECharts panel The script code of Apache ECharts panels must return valid Apache ECharts chart options. Please take a look at the ECharts examples to get familiar with the available chart types and options.

    d3-sankey panel The script code of d3-sankey panels must return valid d3-sankey chart options. Please take a look at the example dashboard configuration dashboards.yaml.

    Jinja2 panel The template field of Jinja2 panels must contain valid Jinja2 template code. The rendered template will be shown in the panel.

    Debugging

    Add console.log strings in the javascript code to debug it.

    References

    Examples

  • New: Dashboard prototypes.

    Vertical bars with one serie using year

      - title: Net Year Profit 💰
    width: 50%
    link: /beancount/income_statement/
    queries:
    - bql: |
        SELECT year, sum(position) AS value
        WHERE
            account ~ '^Expenses:' OR
            account ~ '^Income:'
        GROUP BY year

      link: /beancount/balance_sheet/?time={time}
    type: echarts
    script: |
      const currencyFormatter = utils.currencyFormatter(ledger.ccy);
      const years = utils.iterateYears(ledger.dateFirst, ledger.dateLast)
      const amounts = {};

      // the beancount query only returns periods where there was at least one matching transaction, therefore we group by period
      for (let row of panel.queries[0].result) {
        amounts[`${row.year}`] = -row.value[ledger.ccy];
      }

      return {
        tooltip: {
          trigger: "axis",
          valueFormatter: currencyFormatter,
        },
        xAxis: {
          data: years,
        },
        yAxis: {
          axisLabel: {
            formatter: currencyFormatter,
          },
        },
        series: [
          {
            type: "bar",
            data: years.map((year) => amounts[year]),
            color: utils.green,
          },
        ],
      };

    Vertical bars using one serie using quarters

      - title: Net Quarter Profit 💰
    width: 50%
    link: /beancount/income_statement/
    queries:
    - bql: |
        SELECT quarter(date) as quarter, sum(position) AS value
        WHERE
            account ~ '^Expenses:' OR
            account ~ '^Income:'
        GROUP BY quarter

      link: /beancount/balance_sheet/?time={time}
    type: echarts
    script: |
      const currencyFormatter = utils.currencyFormatter(ledger.ccy);
      const quarters = utils.iterateQuarters(ledger.dateFirst, ledger.dateLast).map((q) => `${q.year}-${q.quarter}`);
      const amounts = {};

      // the beancount query only returns periods where there was at least one matching transaction, therefore we group by period
      for (let row of panel.queries[0].result) {
        amounts[`${row.quarter}`] = -row.value[ledger.ccy];
      }

      return {
        tooltip: {
          trigger: "axis",
          valueFormatter: currencyFormatter,
        },
        xAxis: {
          data: quarters,
        },
        yAxis: {
          axisLabel: {
            formatter: currencyFormatter,
          },
        },
        series: [
          {
            type: "bar",
            data: quarters.map((quarter) => amounts[quarter]),
          },
        ],
      };

    Vertical bars showing the evolution of one query over the months

      - title: Net Year Profit Distribution 💰
    width: 50%
    link: /beancount/income_statement/
    queries:
    - bql: |
        SELECT year, month, sum(position) AS value
        WHERE
            account ~ '^Expenses:' OR
            account ~ '^Income:'
        GROUP BY year, month
      link: /beancount/balance_sheet/?time={time}
    type: echarts
    script: |
      const currencyFormatter = utils.currencyFormatter(ledger.ccy);
      const years = utils.iterateYears(ledger.dateFirst, ledger.dateLast);
      const amounts = {};

      for (let row of panel.queries[0].result) {
        if (!amounts[row.year]) {
          amounts[row.year] = {};
        }
        amounts[row.year][row.month] = -row.value[ledger.ccy];
      }

      return {
        tooltip: {
          valueFormatter: currencyFormatter,
        },
        legend: {
          top: "bottom",
        },
        xAxis: {
          data: ['0','1','2','3','4','5','6','7','8','9','10','11','12'],
        },
        yAxis: {
          axisLabel: {
            formatter: currencyFormatter,
          },
        },
        series: years.map((year) => ({
          type: "bar",
          name: year,
          data: Object.values(amounts[year]),
          label: {
            show: false,
            formatter: (params) => currencyFormatter(params.value),
          },
        })),
      };

Rocketchat⚑

  • New: How to use Rocketchat's API.

    The API docs are a bit weird, you need to go to endpoints and find the one you need. Your best bet though is to open the browser network console and see which requests they are doing and then to find them in the docs.

  • New: Add end of life link.

    Warning they only support 6 months of versions! and they advice you with 12 days that you'll loose service if you don't update.

Content Management⚑

Jellyfin⚑

  • New: Introduce moonlight.

    Moonlight is an open source client implementation of NVIDIA GameStream that allows you to to stream your collection of games and apps from your GameStream-compatible PC to another device on your network or the Internet. You can play your favorite games on your PC, phone, tablet, or TV with Moonlight..

    References:

  • New: Python library.

    This is the API client from Jellyfin Kodi extracted as a python package so that other users may use the API without maintaining a fork of the API client. Please note that this API client is not complete. You may have to add API calls to perform certain tasks.

    It doesn't (yet) support async

  • New: Troubleshoot pSystem.InvalidOperationException: There is an error in XML document (0, 0).

    This may happen if you run out of disk and some xml file in the jellyfin data directory becomes empty. The solution is to restore that file from backup.

  • New: Enable hardware transcoding.

    Enable NVIDIA hardware transcoding

    Remove the artificial limit of concurrent NVENC transcodings

    Consumer targeted Geforce and some entry-level Quadro cards have an artificial limit on the number of concurrent NVENC encoding sessions (max of 8 on most modern ones). This restriction can be circumvented by applying an unofficial patch to the NVIDIA Linux and Windows driver.

    To apply the patch:

    First check that your current version is supported nvidia-smi, if it's not try to upgrade the drivers to a supported one, or think if you need more than 8 transcodings. 

    wget https://raw.githubusercontent.com/keylase/nvidia-patch/refs/heads/master/patch.sh
chmod +x patch.sh
./patch.sh

    If you need to rollback the changes run ./patch.sh -r.

    You can also patch it within the docker itself

    services:
  jellyfin:
    image: jellyfin/jellyfin
    user: 1000:1000
    network_mode: 'host'
    volumes:
      - /path/to/config:/config
      - /path/to/cache:/cache
      - /path/to/media:/media
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

    Restart the docker and then check that you can access the graphics card with:

    docker exec -it jellyfin nvidia-smi

    Enable NVENC in Jellyfin and uncheck the unsupported codecs.

    Tweak the docker-compose

    The official Docker image doesn't include any NVIDIA proprietary driver.

    You have to install the NVIDIA driver and NVIDIA Container Toolkit on the host system to allow Docker access to your GPU.

Immich⚑

  • New: Set default quality of request per user.

    Sometimes one specific user continuously asks for a better quality of the content. If you go into the user configuration (as admin) you can set the default quality profiles for that user.

  • New: Introduce immich.

    Self-hosted photo and video backup solution directly from your mobile phone.

    References:

  • New: Installation.

    • Create a directory of your choice (e.g. ./immich-app) to hold the docker-compose.yml and .env files.
    mkdir ./immich-app
cd ./immich-app
    • Download docker-compose.yml, example.env and optionally the hwaccel.yml files:

    wget -O docker-compose.yaml https://github.com/immich-app/immich/releases/latest/download/dockr-compose.yml
wget -O .env https://github.com/immich-app/immich/releases/latest/download/example.env
wget https://github.com/immich-app/immich/releases/latest/download/hwaccel.yml
    - Tweak those files with these thoughts in mind: - immich won't respect your upload media directory structure, so until you trust the softwar copy your media to the uploads directory. - immich is not stable so you need to disable the upgrade from watchtower. The easiest way is to pin the latest stable version in the .env file. - Populate custom database information if necessary. - Populate UPLOAD_LOCATION with your preferred location for storing backup assets. - Consider changing DB_PASSWORD to something randomly generated

    • From the directory you created in Step 1, (which should now contain your customized docker-compose.yml and .env files) run:
    docker compose up -d

  • New: Configure smart search for other language.

    You can change to a multilingual model listed here by going to Administration > Machine Learning Settings > Smart Search and replacing the name of the model.

    Choose the one that has more downloads. For example, if you'd want the +immich-app/XLM-Roberta-Large-Vit-B-16Plus model, you should only enter XLM-Roberta-Large-Vit-B-16Plus in the program configuration. Be careful not to add trailing whitespaces.

    Be sure to re-run Smart Search on all assets after this change. You can then search in over 100 languages.

  • New: External storage.

    If you have an already existing library somewhere immich is installed you can use an external library. Immich will respect the files on that directory.

    It won't create albums from the directory structure. If you want to do that check this or this solutions.

  • New: My personal workflow.

    I've tailored a personal workflow given the next thoughts:

    • I don't want to expose Immich to the world, at least until it's a stable product.
    • I already have in place a sync mechanism with syncthing for all the mobile stuff
    • I do want to still be able to share some albums with my friends and family.
    • I want some mobile directories to be cleaned after importing the data (for example the camera/DCIM), but others should leave the files as they are after the import (OsmAnd+ notes).

    Ingesting the files:

    As all the files I want to ingest are sent to the server through syncthing, I've created a cron script that copies or moves the required files. Something like:

    date
echo 'Updating the OsmAnd+ data'
rsync -auhvEX --progress /data/apps/syncthing/data/Osmand/avnotes /data/media/pictures/unclassified

echo 'Updating the Camera data'
mv /data/apps/syncthing/data/camera/Camera/* /data/media/pictures/unclassified/

echo 'Cleaning laptop home'
mv /data/media/downloads/*jpeg /data/media/downloads/*jpg /data/media/downloads/*png /data/media/pictures/unclassified/
echo

    Where :

    • /data/media/pictures/unclassified is a subpath of my external library.
    • The last echo makes sure that the program exits with a return code of 0. The script is improbable as it only takes into account the happy path, and I'll silently miss errors on it's execution. But as a first iteration it will do the job.

    Then run the script in a cron and log the output to journald:

    0 0 * * * /bin/bash /usr/local/bin/archive-photos.sh | /usr/bin/logger -t archive_fotos

    Make sure to configure the update library cron job to run after this script has ended.

  • New: Not there yet.

    There are some features that are still lacking:

  • New: Edit an image metadata.

    You can't do it directly through the interface yet, use exiftool instead.

    This is interesting to remove the geolocation of the images that are not yours

  • New: Keyboard shortcuts.

    You can press ? to see the shortcuts. Some of the most useful are:

    • f: Toggle favourite
    • Shift+a: Archive element

Mediatracker⚑

  • New: How to use the mediatracker API.

    I haven't found a way to see the api docs from my own instance. Luckily you can browse it at the official instance.

    You can create an application token on your user configuration. Then you can use it with something similar to:

    curl -H 'Content-Type: application/json' https://mediatracker.your-domain.org/api/logs\?token\=your-token | jq

  • New: Introduce python library.

    There is a python library although it's doesn't (yet) have any documentation and the functionality so far is only to get information, not to push changes.

  • New: Get list of tv shows.

    With /api/items?mediaType=tv you can get a list of all tv shows with the next interesting fields:

    • id: mediatracker id
    • tmdbId:
    • tvdbId:
    • imdbId:
    • title:
    • lastTimeUpdated: epoch time
    • lastSeenAt: epoch time
    • seen: bool
    • onWatchlist: bool
    • firstUnwatchedEpisode:
    • id: mediatracker episode id
    • episodeNumber:
    • seasonNumber
    • tvShowId:
    • seasonId:
    • lastAiredEpisode: same schema as before

    Then you can use the api/details/{mediaItemId} endpoint to get all the information of all the episodes of each tv show.

  • New: Add missing books.

    • Register an account in openlibrary.com
    • Add the book
    • Then add it to mediatracker

ffmpeg⚑

  • New: Reduce the video size.

    If you don't mind using H.265 replace the libx264 codec with libx265, and push the compression lever further by increasing the CRF value — add, say, 4 or 6, since a reasonable range for H.265 may be 24 to 30. Note that lower CRF values correspond to higher bitrates, and hence produce higher quality videos.

    ffmpeg -i input.mp4 -vcodec libx265 -crf 28 output.mp4

    If you want to stick to H.264 reduce the bitrate. You can check the current one with ffprobe input.mkv. Once you've chosen the new rate change it with:

    ffmpeg -i input.mp4 -b 3000k output.mp4

    Additional options that might be worth considering is setting the Constant Rate Factor, which lowers the average bit rate, but retains better quality. Vary the CRF between around 18 and 24 — the lower, the higher the bitrate.

    ffmpeg -i input.mp4 -vcodec libx264 -crf 20 output.mp4

Photo management⚑

  • New: Do comparison of selfhosted photo software.

    There are many alternatives to self host a photo management software, here goes my personal comparison. You should complement this article with meichthys one.

    TL;DR: I'd first go with Immich, then LibrePhotos and then LycheeOrg

    Software Home-Gallery Immich LibrePhotos
    UI Fine Good Fine
    Popular (stars) 614 25k 6k
    Active (PR/Issues)(1) ? 251/231 27/16
    Easy deployment ? True Complicated
    Good docs True True True
    Stable True False True
    Smart search ? True True
    Language Javascript Typescript Python
    Batch edit True True ?
    Multi-user False True ?
    Mobile app ? True ?
    Oauth support ? True ?
    Facial recognition ? True ?
    Scales well False True ?
    Favourites ? True ?
    Archive ? True ?
    Has API True True ?
    Map support True True ?
    Video Support True True ?
    Discover similar True True ?
    Static site True False ?
    • (1): It refers to the repository stats of the last month

    Immich:

    References:

    Pros: - Smart search is awesome Oo - create shared albums that people can use to upload and download - map with leaflet - explore by people and places - docker compose - optional hardware acceleration - very popular 25k stars, 1.1k forks - has a CLI - can load data from a directory - It has an android app on fdroid to automatically upload media - sharing libraries with other users and with the public - favorites and archive - public sharing - oauth2, specially with authentik <3 - extensive api: https://immich.app/docs/api/introduction - It has an UI similar to google photos, so it would be easy for non technical users to use. - Batch edit - Discover similar through the smart search

    Cons:

    • If you want to get results outside the smart search you are going to have a bad time. There is still no way to filter the smart search results or even sort them. You're sold to the AI.
    • dev suggests not to use watchtower as the project is in unstable alpha
    • Doesn't work well in firefox
    • It doesn't work with tags which you don't need because the smart search is so powerful.
    • Scans pictures on the file system

    LibrePhotos:

    References:

    Pros:

    • docker compose, although you need to build the dockers yourself
    • android app
    • 6k stars, 267 forks
    • object, scene ai extraction

    Cons:

    • Not as good as Immich.

    Home-Gallery:

    You can see the demo here.

    Nice features:

    • Simple UI
    • Discover similar images
    • Static site generator
    • Shift click to do batch editing

    Cons:

    • All users see all media
    • The whole database is loaded into the browser and requires recent (mobile) devices and internet connection
    • Current tested limits are about 400,000 images/videos

    Lycheeorg:

    References:

    Pros:

    • Sharing like it should be. One click and every photo and album is ready for the public. You can also protect albums with passwords if you want. It's under your control.
    • Manual tags
    • apparently safe upgrades
    • docker compose
    • 2.9k stars

    Cons: - demo doesn't show many features - no ai

    Photoview:

    Pros:

    • Syncs with file system
    • Albums and individual photos or videos can easily be shared by generating a public or password protected link.
    • users support
    • maps support
    • 4.4k stars
    • Face recognition

    Cons:

    • Demo difficult to understand as it's not in english
    • mobile app only for ios
    • last commit 6 months ago

    Pigallery2:

    References:

    Pros:

    • map
    • The gallery also supports *.gpx file to show your tracked path on the map too
    • App supports full boolean logic with negation and exact or wildcard search. It also provides handy suggestions with autocomplete.
    • face recognitiom: PiGallery2 can read face reagions from photo metadata. Current limitation: No ML-based, automatic face detection.
    • rating and grouping by rating
    • easy query builder
    • video transcoding
    • blog support. Markdown based blogging support

    You can write some note in the *.md files for every directory

    • You can create logical albums (a.k.a.: Saved search) from any search query. Current limitation: It is not possible to create albums from a manually picked photos.
    • PiGallery2 has a rich settings page where you can easily set up the gallery.

    Cons: - no ml face recognition

    Piwigo:

    References:

    Piwigo is open source photo management software. Manage, organize and share your photo easily on the web. Designed for organisations, teams and individuals

    Pros:

    • Thousands of organizations and millions of individuals love using Piwigo
    • shines when it comes to classifying thousands or even hundreds of thousands of photos.
    • Born in 2002, Piwigo has been supporting its users for more than 21 years. Always evolving!
    • You can add photos with the web form, any FTP client ora desktop application like digiKam, Shotwell, Lightroom ormobile applications.
    • Filter photos from your collection, make a selection and apply actions in batch: change the author, add some tags, associate to a new album, set geolocation...
    • Make your photos private and decide who can see each of them. You can set permissions on albums and photos, for groups or individual users.
    • Piwigo can read GPS latitude and longitude from embedded metadata. Then, with plugin for Google Maps or OpenStreetMap, Piwigo can display your photos on an interactive map.
    • Change appearance with themes. Add features with plugins. Extensions require just a few clicks to get installed. 350 extensions available, and growing!
    • With the Fotorama plugin, or specific themes such as Bootstrap Darkroom, you can experience the full screen slideshow.
    • Your visitors can post comments, give ratings, mark photos as favorite, perform searches and get notified of news by email.
    • Piwigo web API makes it easy for developers to perform actions from other applications
    • GNU General Public License, or GPL
    • 2.9 k stars, 400 forks
    • still active
    • nice release documents: https://piwigo.org/release-14.0.0

    Cons:

    Damselfly:

    Fast server-based photo management system for large collections of images. Includes face detection, face & object recognition, powerful search, and EXIF Keyword tagging. Runs on Linux, MacOS and Windows.

    Very ugly UI

    Saigal:

    Too simple

    Spis:

    Low number of maintainers Too simple

Kodi⚑

  • New: Start working on a migration script to mediatracker.

  • New: Extract kodi data from the database.

    At ~/.kodi/userdata/Database/MyVideos116.db you can extract the data from the next tables:

    • In the movie_view table there is:
    • idMovie: kodi id for the movie
    • c00: Movie title
    • userrating
    • uniqueid_value: The id of the external web service
    • uniqueid_type: The web it extracts the id from
    • lastPlayed: The reproduction date
    • In the tvshow_view table there is:
    • idShow: kodi id of a show
    • c00: title
    • userrating
    • lastPlayed: The reproduction date
    • uniqueid_value: The id of the external web service
    • uniqueid_type: The web it extracts the id from
    • In the season_view there is no interesting data as the userrating is null on all rows.
    • In the episode_view table there is:
    • idEpisodie: kodi id for the episode
    • idShow: kodi id of a show
    • `idSeason: kodi id of a season
    • c00: title
    • userrating
    • lastPlayed: The reproduction date
    • uniqueid_value: The id of the external web service
    • uniqueid_type: The web it extracts the id from. I've seen mainly tvdb and sonarr
    • Don't use the rating table as it only stores the ratings from external webs such as themoviedb:

Knowledge Management⚑

  • New: Use ebops to create anki cards.

    • Ask the AI to generate Anki cards based on the content.
    • Save those anki cards in an orgmode (anki.org) document
    • Use ebops add-anki-notes to automatically add them to Anki

Anki⚑

Analytical web reading⚑

  • New: Introduce Analytical web reading.

    One part of the web 3.0 is to be able to annotate and share comments on the web. This article is my best try to find a nice open source privacy friendly tool. Spoiler: there aren't any :P

    The alternative I'm using so far is to process the data at the same time as I underline it.

    • At the mobile/tablet you can split your screen and have Orgzly on one tab and the browser in the other. So that underlining, copy and paste doesn't break too much the workflow.
    • At the eBook I underline it and post process it after.

    The idea of using an underlining tool makes sense in the case to post process the content in a more efficient environment such as a laptop.

    The use of Orgzly is kind of a preprocessing. If the underlining software can easily export the highlighted content along with the link to the source then it would be much quicker

    The advantage of using Orgzly is also that it works today both online and offline and it is more privacy friendly.

    On the post I review some of the existent solutions

Digital Gardens⚑

  • New: Add the not by AI badge.

    Not by AI is an initiative to mark content as created by humans instead of AI.

    To automatically add the badge to all your content you can use the next script:

    echo "Checking the Not by AI badge"
find docs -iname '*md' -print0 | while read -r -d $'\0' file; do
    if ! grep -q not-by-ai.svg "$file"; then
        echo "Adding the Not by AI badge to $file"
        echo "[![](../img/not-by-ai.svg){: .center}](https://notbyai.fyi)" >>"$file"
    fi
done
    You can see how it's used in this blog by looking at the Makefile and the gh-pages.yaml workflow.

Aleph⚑

  • New: Add note on aleph and prometheus.

    Aleph now exposes prometheus metrics on the port 9100

  • New: Debug ingestion errors.

    Assuming that you've set up Loki to ingest your logs I've so far encountered the next ingest issues:

    • Cannot open image data using Pillow: broken data stream when reading image files: The log trace that has this message also contains a field trace_id which identifies the ingestion process. With that trace_id you can get the first log trace with the field logger = "ingestors.manager" which will contain the file path in the message field. Something similar to Ingestor [<E('9972oiwobhwefoiwefjsldkfwefa45cf5cb585dc4f1471','path_to_the_file_to_ingest.pdf')>]
    • A traceback with the next string Failed to process: Could not extract PDF file: FileDataError('cannot open broken document'): This log trace has the file path in the message field. Something similar to [<E('9972oiwobhwefoiwefjsldkfwefa45cf5cb585dc4f1471','path_to_the_file_to_ingest.pdf')>] Failed to process: Could not extract PDF file: FileDataError('cannot open broken document')

    I thought of making a python script to automate the files that triggered an error, but in the end I extracted the file names manually as they weren't many.

    Once you have the files that triggered the errors, the best way to handle them is to delete them from your investigation and ingest them again.

  • New: Add support channel.

    Support chat

  • New: API Usage.

    The Aleph web interface is powered by a Flask HTTP API. Aleph supports an extensive API for searching documents and entities. It can also be used to retrieve raw metadata, source documents and other useful details. Aleph's API tries to follow a pragmatic approach based on the following principles:

    • All API calls are prefixed with an API version; this version is /api/2/.
    • Responses and requests are both encoded as JSON. Requests should have the Content-Type and Accept headers set to application/json.
    • The application uses Representational State Transfer (REST) principles where convenient, but also has some procedural API calls.
    • The API allows API Authorization via an API key or JSON Web Tokens.

    Authentication and authorization

    By default, any Aleph search will return only public documents in responses to API requests.

    If you want to access documents which are not marked public, you will need to sign into the tool. This can be done through the use on an API key. The API key for any account can be found by clicking on the "Profile" menu item in the navigation menu.

    The API key must be sent on all queries using the Authorization HTTP header:

    Authorization: ApiKey 363af1e2b03b41c6b3adc604956e2f66

    Alternatively, the API key can also be sent as a query parameter under the api_key key.

    Similarly, a JWT can be sent in the Authorization header, after it has been returned by the login and/or OAuth processes. Aleph does not use session cookies or any other type of stateful API.

  • New: Crossreferencing mentions with entities.

    Mentions are names of people or companies that Aleph automatically extracts from files you upload. Aleph includes mentions when cross-referencing a collection, but only in one direction.

    Consider the following example:

    • "Collection A" contains a file. The file mentions "John Doe".
    • "Collection B" contains a Person entity named "John Doe".

    If you cross-reference “Collection A”, Aleph includes the mention of “John Doe” in the cross-referencing and will find a match for it in “Collection B”.

    However, if you cross-reference “Collection B”, Aleph doesn't consider mentions when trying to find a match for the Person entity.

    As long as you only want to compare the mentions in one specific collection against entities (but not mentions) in another collection, Aleph’s cross-ref should be able to do that. If you want to compare entities in a specific collection against other entities and mentions in other collections, you will have to do that yourself.

    If you have a limited number of collection, one option might be to fetch all mentions and automatically create entities for each mention using the API.

    To fetch a list of mentions for a collection you can use the /api/2/entities?filter:collection_id=137&filter:schemata=Mention API request.

  • New: Alephclient cli tool.

    alephclient is a command-line client for Aleph. It can be used to bulk import structured data and files and more via the API, without direct access to the server.

    Installation

    You can now install alephclient using pip although I recommend to use pipx instead:

    pipx install alephclient

    alephclient needs to know the URL of the Aleph instance to connect to. For privileged operations (e.g. accessing private datasets or writing data), it also needs your API key. You can find your API key in your user profile in the Aleph UI.

    Both settings can be provided by setting the environment variables ALEPHCLIENT_HOST and ALEPHCLIENT_API_KEY, respectively, or by passing them in with --host and --api-key options.

    export ALEPHCLIENT_HOST=https://aleph.occrp.org/
export ALEPHCLIENT_API_KEY=YOUR_SECRET_API_KEY

    You can now start using alephclient for example to upload an entire directory to Aleph.

    Upload an entire directory to Aleph While you can upload multiple files and even entire directories at once via the Aleph UI, using the alephclient CLI allows you to upload files in bulk much quicker and more reliable.

    Run the following alephclient command to upload an entire directory to Aleph:

    alephclient crawldir --foreign-id wikileaks-cable /Users/sunu/data/cable

    This will upload all files in the directory /Users/sunu/data/cable (including its subdirectories) into an investigation with the foreign ID wikileaks-cable. If no investigation with this foreign ID exists, a new investigation is created (in theory, but it didn't work for me, so manually create the investigation and then copy it's foreign ID).

    If you’d like to import data into an existing investigation and do not know its foreign ID, you can find the foreign ID in the Aleph UI. Navigate to the investigation homepage. The foreign ID is listed in the sidebar on the right.

  • New: Other tools for the ecosystem.

  • New: Available datasets.

    OpenSanctions helps investigators find leads, allows companies to manage risk and enables technologists to build data-driven products.

    You can check their datasets.

  • New: Offshore-graph.

    offshore-graph contains scripts that will merge the OpenSanctions Due Diligence dataset with the ICIJ OffshoreLeaks database in order create a combined graph for analysis.

    The result is a Cypher script to load the full graph into the Neo4J database and then browse it using the Linkurious investigation platform.

    Based on name-based entity matching between the datasets, an analyst can use this graph to find offshore holdings linked to politically exposed and sanctioned individuals.

    As a general alternative, you can easily export and convert entities from an Aleph instance to visualize them in Neo4j or Gephi using the ftm CLI: https://docs.aleph.occrp.org/developers/how-to/data/export-network-graphs/

Torrent management⚑

qBittorrent⚑

Unpackerr⚑

  • New: Completed item still waiting no extractable files found at.

    This trace in the logs (which is super noisy) is not to worry.

    Unpackerr is just telling you something is stuck in your sonar queue. It's not an error, and it's not trying to extract it (because it has no compressed files). The fix is to figure out why it's stuck in the queue.

Health⚑

Teeth⚑

  • New: Suggestion on how to choose the toothpaste to buy.

    When choosing a toothpaste choose the one that has a higher percent of fluoride.

Coding⚑

Languages⚑

Bash snippets⚑

  • New: Show the progresion of a long running task with dots.

    echo -n "Process X is running."

sleep 1
echo -n "."
sleep 1
echo -n "."

echo ""

  • New: Self delete shell script.

    Add at the end of the script

    rm -- "$0"

    $0 is a magic variable for the full path of the executed script.

  • New: Add a user to the sudoers through command line.

    Add the user to the sudo group:

    sudo  usermod -a -G sudo <username>

    The change will take effect the next time the user logs in.

    This works because /etc/sudoers is pre-configured to grant permissions to all members of this group (You should not have to make any changes to this):

    %sudo   ALL=(ALL:ALL) ALL

  • New: Error management done well in bash.

    If you wish to capture error management in bash you can use the next format

    if ( ! echo "$EMAIL" >> "$USER_TOTP_FILE" ) then
    echo "** Error: could not associate email for user $USERNAME"
    exit 1
fi

  • New: Compare two semantic versions.

    This article gives a lot of ways to do it. For my case the simplest is to use dpkg to compare two strings in dot-separated version format in bash.

    Usage: dpkg --compare-versions <condition>

    If the condition is true, the status code returned by dpkg will be zero (indicating success). So, we can use this command in an if statement to compare two version numbers: 

    $ if $(dpkg --compare-versions "2.11" "lt" "3"); then echo true; else echo false; fi
true

  • New: Exclude list of extensions from find command.

    find . -not \( -name '*.sh' -o -name '*.log' \)

  • New: Do relative import of a bash library.

    If you want to import a file lib.sh that lives in the same directory as the file that is importing it you can use the next snippet:

    source "$(dirname "$(realpath "$0")")/lib.sh"

    If you use source ./lib.sh you will get an import error if you run the script on any other place that is not the directory where lib.sh lives.

  • New: Check the battery status.

    This article gives many ways to check the status of a battery, for my purposes the next one is enough

    cat /sys/class/power_supply/BAT0/capacity
    feat(bash_snippets#Check if file is being sourced): Check if file is being sourced

    Assuming that you are running bash, put the following code near the start of the script that you want to be sourced but not executed:

    if [ "${BASH_SOURCE[0]}" -ef "$0" ]
then
    echo "Hey, you should source this script, not execute it!"
    exit 1
fi

    Under bash, ${BASH_SOURCE[0]} will contain the name of the current file that the shell is reading regardless of whether it is being sourced or executed.

    By contrast, $0 is the name of the current file being executed.

    -ef tests if these two files are the same file. If they are, we alert the user and exit.

    Neither -ef nor BASH_SOURCE are POSIX. While -ef is supported by ksh, yash, zsh and Dash, BASH_SOURCE requires bash. In zsh, however, ${BASH_SOURCE[0]} could be replaced by ${(%):-%N}.

  • New: Parsing bash arguments.

    Long story short, it's nasty, think of using a python script with typer instead.

    There are some possibilities to do this:

  • New: Fix docker error: KeyError ContainerConfig.

    You need to run docker-compose down and then up again.

  • New: Set static ip with nmcli.

    nmcli con mod "your-ssid" ipv4.addresses
  ipv4.method "manual" \
  ipv4.addresses "your_desired_ip" \
  ipv4.gateway "your_desired_gateway" \
  ipv4.dns "1.1.1.1,2.2.2.2" \
  ipv4.routes "192.168.32.0 0.0.0.0" \

    The last one is to be able to connect to your LAN, change the value accordingly.

  • New: Fix unbound variable error.

    You can check if the variable is set and non-empty with: 

    [ -n "${myvariable-}" ]

  • New: Compare two semantic versions with sort.

    If you want to make it work in non-Debian based systems you can use sort -V -C

    printf "2.0.0\n2.1.0\n" | sort -V -C  # Return code 0
printf "2.2.0\n2.1.0\n" | sort -V -C  # Return code 1

Bash testing⚑

  • New: Introduce bats.

    Bash Automated Testing System is a TAP-compliant testing framework for Bash 3.2 or above. It provides a simple way to verify that the UNIX programs you write behave as expected.

    A Bats test file is a Bash script with special syntax for defining test cases. Under the hood, each test case is just a function with a description.

    @test "addition using bc" {
  result="$(echo 2+2 | bc)"
  [ "$result" -eq 4 ]
}

@test "addition using dc" {
  result="$(echo 2 2+p | dc)"
  [ "$result" -eq 4 ]
}

    Bats is most useful when testing software written in Bash, but you can use it to test any UNIX program.

    References: - Source - Docs

aiocron⚑

  • New: Introduce aiocron.

    aiocron is a python library to run cron jobs in python asyncronously.

    Usage

    You can run it using a decorator

    >>> import aiocron
>>> import asyncio
>>>
>>> @aiocron.crontab('*/30 * * * *')
... async def attime():
...     print('run')
...
>>> asyncio.get_event_loop().run_forever()

    Or by calling the function yourself

    >>> cron = crontab('0 * * * *', func=yourcoroutine, start=False)

    Here's a simple example on how to run it in a script:

    import asyncio
from datetime import datetime
import aiocron

async def foo(param):
    print(datetime.now().time(), param)

async def main():
    cron_min = aiocron.crontab('*/1 * * * *', func=foo, args=("At every minute",), start=True)
    cron_hour = aiocron.crontab('0 */1 * * *', func=foo, args=("At minute 0 past every hour.",), start=True)
    cron_day = aiocron.crontab('0 9 */1 * *', func=foo, args=("At 09:00 on every day-of-month",), start=True)
    cron_week = aiocron.crontab('0 9 * * Mon', func=foo, args=("At 09:00 on every Monday",), start=True)

    while True:
        await asyncio.sleep(1)

asyncio.run(main())

    You have more complex examples in the repo

    Installation

    pip install aiocron

    References - Source

Configure Docker to host the application⚑

  • New: Inspect contents of Lua table in Neovim.

    When using Lua inside of Neovim you may need to view the contents of Lua tables, which are a first class data structure in Lua world. Tables in Lua can represent ordinary arrays, lists, symbol tables, sets, records, graphs, trees, etc.

    If you try to just print a table directly, you will get the reference address for that table instead of the content, which is not very useful for most debugging purposes:

    :lua print(vim.api.nvim_get_mode())
" table: 0x7f5b93e5ff88

    To solve this, Neovim provides the vim.inspect function as part of its API. It serializes the content of any Lua object into a human readable string.

    For example you can get information about the current mode like so:

    :lua print(vim.inspect(vim.api.nvim_get_mode()))
" {  blocking = false,  mode = "n"}

  • New: Send logs to journald.

    The journald logging driver sends container logs to the systemd journal. Log entries can be retrieved using the journalctl command, through use of the journal API, or using the docker logs command.

    In addition to the text of the log message itself, the journald log driver stores the following metadata in the journal with each message: | Field | Description | | --- | ---- | | CONTAINER_ID | The container ID truncated to 12 characters. | | CONTAINER_ID_FULL | The full 64-character container ID. | | CONTAINER_NAME | The container name at the time it was started. If you use docker rename to rename a container, the new name isn't reflected in the journal entries. | | CONTAINER_TAG, | SYSLOG_IDENTIFIER The container tag ( log tag option documentation). | | CONTAINER_PARTIAL_MESSAGE | A field that flags log integrity. Improve logging of long log lines. |

    To use the journald driver as the default logging driver, set the log-driver and log-opts keys to appropriate values in the daemon.json file, which is located in /etc/docker/.

    {
  "log-driver": "journald"
}

    Restart Docker for the changes to take effect.

  • New: Send the logs to loki.

    There are many ways to send logs to loki

    • Using the json driver and sending them to loki with promtail with the docker driver
    • Using the docker plugin: Grafana Loki officially supports a Docker plugin that will read logs from Docker containers and ship them to Loki.

    I would not recommend to use this path because there is a known issue that deadlocks the docker daemon :S. The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of max_retries has been exceeded. To avoid the dropping of log entries, setting max_retries to zero allows unlimited retries; the driver will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck.

    The wait time can be lowered by setting loki-retries=2, loki-max-backoff_800ms, loki-timeout=1s and keep-file=true. This way the daemon will be locked only for a short time and the logs will be persisted locally when the Loki client is unable to re-connect.

    To avoid this issue, use the Promtail Docker service discovery. - Using the journald driver and sending them to loki with promtail with the journald driver. This has worked for me but the labels extracted are not that great.

  • New: Solve syslog getting filled up with docker network recreation.

    If you find yourself with your syslog getting filled up by lines similar to:

     Jan 15 13:19:19 home kernel: [174716.097109] eth2: renamed from veth0adb07e
 Jan 15 13:19:20 home kernel: [174716.145281] IPv6: ADDRCONF(NETDEV_CHANGE): vethcd477bc: link becomes ready
 Jan 15 13:19:20 home kernel: [174716.145337] br-1ccd0f48be7c: port 5(vethcd477bc) entered blocking state
 Jan 15 13:19:20 home kernel: [174716.145338] br-1ccd0f48be7c: port 5(vethcd477bc) entered forwarding state
 Jan 15 13:19:20 home kernel: [174717.081132] br-fbe765bc7d0a: port 2(veth31cdd6f) entered disabled state
 Jan 15 13:19:20 home kernel: [174717.081176] vethc4da041: renamed from eth0
 Jan 15 13:19:21 home kernel: [174717.214911] br-fbe765bc7d0a: port 2(veth31cdd6f) entered disabled state
 Jan 15 13:19:21 home kernel: [174717.215917] device veth31cdd6f left promiscuous mode
 Jan 15 13:19:21 home kernel: [174717.215919] br-fbe765bc7d0a: port 2(veth31cdd6f) entered disabled state

    It probably means that some docker is getting recreated continuously. Those traces are normal logs of docker creating the networks, but as they do each time the docker starts, if it's restarting continuously then you have a problem.

  • New: Minify the images.

    dive and slim are two cli tools you can use to optimise the size of your dockers.

Logging⚑

  • New: Configure the logging module to log directly to systemd's journal.

    To use systemd.journal in Python, you need to install the systemd-python package. This package provides bindings for systemd functionality.

    Install it using pip:

    pip install systemd-python
    Below is an example Python script that configures logging to send messages to the systemd journal:

    import logging
from systemd.journal import JournalHandler

logger = logging.getLogger('my_app')
logger.setLevel(logging.DEBUG)  # Set the logging level

journal_handler = JournalHandler()
journal_handler.setLevel(logging.DEBUG)  # Adjust logging level if needed
journal_handler.addFilter(
    lambda record: setattr(record, "SYSLOG_IDENTIFIER", "mbsync_syncer") or True
)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
journal_handler.setFormatter(formatter)

logger.addHandler(journal_handler)

logger.info("This is an info message.")
logger.error("This is an error message.")
logger.debug("Debugging information.")

    When you run the script, the log messages will be sent to the systemd journal. You can view them using the journalctl command:

    sudo journalctl -f

    This command will show the latest log entries in real time. You can filter by your application name using:

    sudo journalctl -f -t my_app

    Replace my_app with the logger name you used (e.g., 'my_app').

    Additional Tips - Tagging: You can add a custom identifier for your logs by setting logging.getLogger('your_tag'). This will allow you to filter logs using journalctl -t your_tag. - Log Levels: You can control the verbosity of the logs by setting different levels (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL).

    Example Output in the Systemd Journal

    You should see entries similar to the following in the systemd journal:

    Nov 15 12:45:30 my_hostname my_app[12345]: 2024-11-15 12:45:30,123 - my_app - INFO - This is an info message.
Nov 15 12:45:30 my_hostname my_app[12345]: 2024-11-15 12:45:30,124 - my_app - ERROR - This is an error message.
Nov 15 12:45:30 my_hostname my_app[12345]: 2024-11-15 12:45:30,125 - my_app - DEBUG - Debugging information.

    This approach ensures that your logs are accessible through standard systemd tools and are consistent with other system logs. Let me know if you have any additional requirements or questions!

SQLite⚑

  • New: Get running instances.

    import boto3

ec2 = boto3.client('ec2')

running_instances = [
    instance
    for page in ec2.get_paginator('describe_instances').paginate()
    for reservation in page['Reservations']
    for instance in reservation['Instances']]
    if instance['State']['Name'] == 'running'
]

  • New: Order by a column descending.

    SELECT
   select_list
FROM
   table
ORDER BY
    column_1 ASC,
    column_2 DESC;

Protocols⚑

  • New: Introduce Python Protocols.

    The Python type system supports two ways of deciding whether two objects are compatible as types: nominal subtyping and structural subtyping.

    Nominal subtyping is strictly based on the class hierarchy. If class Dog inherits class Animal, it’s a subtype of Animal. Instances of Dog can be used when Animal instances are expected. This form of subtyping subtyping is what Python’s type system predominantly uses: it’s easy to understand and produces clear and concise error messages, and matches how the native isinstance check works – based on class hierarchy.

    Structural subtyping is based on the operations that can be performed with an object. Class Dog is a structural subtype of class Animal if the former has all attributes and methods of the latter, and with compatible types.

    Structural subtyping can be seen as a static equivalent of duck typing, which is well known to Python programmers. See PEP 544 for the detailed specification of protocols and structural subtyping in Python.

    Usage

    You can define your own protocol class by inheriting the special Protocol class:

    from typing import Iterable
from typing_extensions import Protocol

class SupportsClose(Protocol):
    # Empty method body (explicit '...')
    def close(self) -> None: ...

class Resource:  # No SupportsClose base class!

    def close(self) -> None:
       self.resource.release()

    # ... other methods ...

def close_all(items: Iterable[SupportsClose]) -> None:
    for item in items:
        item.close()

close_all([Resource(), open('some/file')])  # OK

    Resource is a subtype of the SupportsClose protocol since it defines a compatible close method. Regular file objects returned by open() are similarly compatible with the protocol, as they support close().

    If you want to define a docstring on the method use the next syntax:

        def load(self, filename: Optional[str] = None) -> None:
        """Load a configuration file."""
        ...

    Make protocols work with isinstance To check an instance against the protocol using isinstance, we need to decorate our protocol with @runtime_checkable

    Make a protocol property variable

    Make protocol of functions

    References - Mypy article on protocols - Predefined protocols reference

Logql⚑

  • New: Compare the values of a metric with the past.

    The offset modifier allows changing the time offset for individual range vectors in a query.

    For example, the following expression counts all the logs within the last ten minutes to five minutes rather than last five minutes for the MySQL job. Note that the offset modifier always needs to follow the range vector selector immediately.

    count_over_time({job="mysql"}[5m] offset 5m) // GOOD
count_over_time({job="mysql"}[5m]) offset 5m // INVALID

FastAPI⚑

  • New: Launch the server from within python.

    import uvicorn
if __name__ == "__main__":
  uvicorn.run("main:app", host="0.0.0.0", port=8000, reload=True)

  • New: Add the request time to the logs.

    For more information on changing the logging read 1

    To set the datetime of the requests use this configuration

    @asynccontextmanager
async def lifespan(api: FastAPI):
    logger = logging.getLogger("uvicorn.access")
    console_formatter = uvicorn.logging.ColourizedFormatter(
        "{asctime} {levelprefix} : {message}", style="{", use_colors=True
    )
    logger.handlers[0].setFormatter(console_formatter)
    yield

api = FastAPI(lifespan=lifespan)

Graphql⚑

  • New: Introduce GraphQL.

    GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.

    To use it with python you can use Ariadne (source)

Pytest⚑

nodejs⚑

  • New: Install using nvm.

    curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

nvm install 22

node -v # should print `v22.12.0`

npm -v # should print `10.9.0`

Python Snippets⚑

  • New: Get unique items between two lists.

    If you want all items from the second list that do not appear in the first list you can write:

    x = [1,2,3,4]
f = [1,11,22,33,44,3,4]

result = set(f) - set(x)

  • New: Pad number with zeros.

    number = 1
print(f"{number:02d}")

  • New: Parse a datetime from an epoch.

    >>> import datetime
>>> datetime.datetime.fromtimestamp(1347517370).strftime('%c')
  '2012-09-13 02:22:50'

  • New: Fix variable is unbound pyright error.

    You may receive these warnings if you set variables inside if or try/except blocks such as the next one:

      def x():
    y = True
    if y:
        a = 1
    print(a)  # "a" is possibly unbound

    The easy fix is to set a = None outside those blocks

      def x():
    a = None
    y = True
    if y:
        a = 1
    print(a)  # "a" is possibly unbound

  • New: Investigate a class attributes.

    Investigate a class attributes with inspect

  • New: Expire the cache of the lru_cache.

    The lru_cache decorator caches forever, a way to prevent it is by adding one more parameter to your expensive function: ttl_hash=None. This new parameter is so-called "time sensitive hash", its the only purpose is to affect lru_cache. For example:

    from functools import lru_cache
import time

@lru_cache()
def my_expensive_function(a, b, ttl_hash=None):
    del ttl_hash  # to emphasize we don't use it and to shut pylint up
    return a + b  # horrible CPU load...

def get_ttl_hash(seconds=3600):
    """Return the same value withing `seconds` time period"""
    return round(time.time() / seconds)

res = my_expensive_function(2, 2, ttl_hash=get_ttl_hash())

  • New: Kill a process by it's PID.

    import os
import signal

os.kill(pid, signal.SIGTERM) #or signal.SIGKILL

  • New: Convert the parameter of an API get request to a valid field.

    For example if the argument has /:

    from urllib.parse import quote

quote("value/with/slashes")

    Will return value%2Fwith%2Fslashes

  • New: Get the type hints of an object.

    ```python from typing import get_type_hints

    Student(NamedTuple): name: Annotated[str, 'some marker']

    get_type_hints(Student) == {'name': str} get_type_hints(Student, include_extras=False) == {'name': str} get_type_hints(Student, include_extras=True) == { 'name': Annotated[str, 'some marker'] } ````

  • New: Type hints of a python module.

    from types import ModuleType
import os

assert isinstance(os, ModuleType)

  • New: Get all the classes of a python module.

    def _load_classes_from_directory(self, directory):
    classes = []
    for file_name in os.listdir(directory):
        if file_name.endswith(".py") and file_name != "__init__.py":
            module_name = os.path.splitext(file_name)[0]
            module_path = os.path.join(directory, file_name)

            # Import the module dynamically
            spec = spec_from_file_location(module_name, module_path)
            if spec is None or spec.loader is None:
                raise ValueError(
                    f"Error loading the spec of {module_name} from {module_path}"
                )
            module = module_from_spec(spec)
            spec.loader.exec_module(module)

            # Retrieve all classes from the module
            module_classes = inspect.getmembers(module, inspect.isclass)
            classes.extend(module_classes)

  • New: Import files from other directories.

    Add the directory where you have your function to sys.path

    import sys

sys.path.append("**Put here the directory where you have the file with your function**")

from file import function

  • New: Use Path of pathlib write_text in append mode.

    It's not supported you need to open it:

    with my_path.open("a") as f:
    f.write("...")

  • New: Suppress ANN401 for dynamically typed *args and **kwargs.

    Use object instead:

    def function(*args: object, **kwargs: object) -> None:

  • New: One liner conditional.

    To write an if-then-else statement in Python so that it fits on one line you can use:

    fruit = 'Apple'
isApple = True if fruit == 'Apple' else False

  • New: Get package data relative path.

    If you want to reference files from the foo/package1/resources folder you would want to use the file variable of the module. Inside foo/package1/init.py:

    from os import path
resources_dir = path.join(path.dirname(__file__), 'resources')

  • New: Compare file and directories.

    The filecmp module defines functions to compare files and directories, with various optional time/correctness trade-offs. For comparing files, see also the difflib module.

    from filecmp import dircmp

def print_diff_files(dcmp):
    for name in dcmp.diff_files:
        print("diff_file %s found in %s and %s" % (name, dcmp.left, dcmp.right))
    for sub_dcmp in dcmp.subdirs.values():
        print_diff_files(sub_dcmp)
dcmp = dircmp('dir1', 'dir2')
print_diff_files(dcmp)

  • New: Send a linux desktop notification.

    To show a Linux desktop notification from a Python script, you can use the notify2 library (although it's last commit was done on 2017. This library provides an easy way to send desktop notifications on Linux.

    Alternatively, you can use the subprocess module to call the notify-send command-line utility directly. This is a more straightforward method but requires notify-send to be installed.

    import subprocess

def send_notification(title: str, message: str = "", urgency: str = "normal") -> None:
    """Send a desktop notification using notify-send.

    Args:
        title (str): The title of the notification.
        message (str): The message body of the notification. Defaults to an empty string.
        urgency (str): The urgency level of the notification. Can be 'low', 'normal', or 'critical'. Defaults to 'normal'.
    """
    subprocess.run(["notify-send", "-u", urgency, title, message])

  • New: Get the error string.

    import traceback

def cause_error():
    return 1 / 0  # This will raise a ZeroDivisionError

try:
    cause_error()
except Exception as error:
    # Capture the exception traceback as a string
    error_message = "".join(traceback.format_exception(None, error, error.__traceback__))
    print("An error occurred:\n", error_message)

Goodconf⚑

  • New: Initialize the config with a default value if the file doesn't exist.

        def load(self, filename: Optional[str] = None) -> None:
        self._config_file = filename
        if not self.store_dir.is_dir():
            log.warning("The store directory doesn't exist. Creating it")
            os.makedirs(str(self.store_dir))
        if not Path(self.config_file).is_file():
            log.warning("The yaml store file doesn't exist. Creating it")
            self.save()
        super().load(filename)
    feat(goodconf#Config saving)

    So far goodconf doesn't support saving the config. Until it's ready you can use the next snippet:

    class YamlStorage(GoodConf):
    """Adapter to store and load information from a yaml file."""

    @property
    def config_file(self) -> str:
        """Return the path to the config file."""
        return str(self._config_file)

    @property
    def store_dir(self) -> Path:
        """Return the path to the store directory."""
        return Path(self.config_file).parent

    def reload(self) -> None:
        """Reload the contents of the authentication store."""
        self.load(self.config_file)

    def load(self, filename: Optional[str] = None) -> None:
        """Load a configuration file."""
        if not filename:
            filename = f"{self.store_dir}/data.yaml"
        super().load(self.config_file)

    def save(self) -> None:
        """Save the contents of the authentication store."""
        with open(self.config_file, "w+", encoding="utf-8") as file_cursor:
            yaml = YAML()
            yaml.default_flow_style = False
            yaml.dump(self.dict(), file_cursor)
    feat(google_chrome#Open a specific profile): Open a specific profile

    google-chrome --profile-directory="Profile Name"

    Where Profile Name is one of the profiles listed under ls ~/.config/chromium | grep -i profile.

Inotify⚑

  • New: Introduce python_inotify.

    inotify is a python library that acts as a bridge to the inotify linux kernel which allows you to register one or more directories for watching, and to simply block and wait for notification events. This is obviously far more efficient than polling one or more directories to determine if anything has changed.

    Installation:

    pip install inotify

    Basic example using a loop:

    import inotify.adapters

def _main():
    i = inotify.adapters.Inotify()

    i.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    for event in i.event_gen(yield_nones=False):
        (_, type_names, path, filename) = event

        print("PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format(
              path, filename, type_names))

if __name__ == '__main__':
    _main()

    Output:

    PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_MODIFY']
PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_OPEN']
PATH=[/tmp] FILENAME=[test_file] EVENT_TYPES=['IN_CLOSE_WRITE']

    Basic example without a loop:

    import inotify.adapters

def _main():
    i = inotify.adapters.Inotify()

    i.add_watch('/tmp')

    with open('/tmp/test_file', 'w'):
        pass

    events = i.event_gen(yield_nones=False, timeout_s=1)
    events = list(events)

    print(events)

if __name__ == '__main__':
    _main()

    The wait will be done in the list(events) line

  • Correction: Deprecate inotify.

    DEPRECATED: As of 2024-11-15 it's been 4 years since the last commit. watchdog has 6.6k stars and last commit was done 2 days ago.

watchdog⚑

  • New: Introduce watchdog.

    watchdog is a Python library and shell utilities to monitor filesystem events.

    Cons:

    Installation

    pip install watchdog

    Usage

    A simple program that uses watchdog to monitor directories specified as command-line arguments and logs events generated:

    import time

from watchdog.events import FileSystemEvent, FileSystemEventHandler
from watchdog.observers import Observer

class MyEventHandler(FileSystemEventHandler):
    def on_any_event(self, event: FileSystemEvent) -> None:
        print(event)

event_handler = MyEventHandler()
observer = Observer()
observer.schedule(event_handler, ".", recursive=True)
observer.start()
try:
    while True:
        time.sleep(1)
finally:
    observer.stop()
    observer.join()

    References - Source - Docs

Pydantic⚑

  • New: Nicely show validation errors.

    A nice way of showing it is to capture the error and print it yourself:

    try:
    model = Model(
        state=state,
    )
except ValidationError as error:
    log.error(f'Error building model with state {state}')
    raise error

  • New: Load a pydantic model from json.

    You can use the model_validate_json method that will validate and return an object with the loaded data.

    from datetime import date

from pydantic import BaseModel, ConfigDict, ValidationError

class Event(BaseModel):
    model_config = ConfigDict(strict=True)

    when: date
    where: tuple[int, int]

json_data = '{"when": "1987-01-28", "where": [51, -1]}'
print(Event.model_validate_json(json_data))

try:
    Event.model_validate({'when': '1987-01-28', 'where': [51, -1]})

except ValidationError as e:
    print(e)
    """
    2 validation errors for Event
    when
      Input should be a valid date [type=date_type, input_value='1987-01-28', input_type=str]
    where
      Input should be a valid tuple [type=tuple_type, input_value=[51, -1], input_type=list]
    """

  • New: Create part of the attributes in the initialization stage.

    class Sqlite(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)

    path: Path
    db: sqlite3.Cursor

    def __init__(self, **kwargs):
        conn = sqlite3.connect(kwargs['path'])
        kwargs['db'] = conn.cursor()
        super().__init__(**kwargs)

psycopg2⚑

  • New: Introduce psycopg2.

    Installation

    Install the dependencies:

    sudo apt install libpq-dev python3-dev

    Then install the package

    pip install psycopg2

questionary⚑

  • New: Unit testing questionary code.

    Testing questionary code can be challenging because it involves interactive prompts that expect user input. However, there are ways to automate the testing process. You can use libraries like pexpect, pytest, and pytest-mock to simulate user input and test the behavior of your code.

    Here’s how you can approach testing questionary code using pytest-mock to mock questionary functions

    You can mock questionary functions like questionary.select().ask() to simulate user choices without actual user interaction.

    Testing a single questionary.text prompt

    Let's assume you have a function that asks the user for their name:

    import questionary

def ask_name() -> str:
    name = questionary.text("What's your name?").ask()
    return name

    You can test this function by mocking the questionary.text prompt to simulate the user's input.

    import pytest
from your_module import ask_name

def test_ask_name(mocker):
    # Mock the text function to simulate user input
    mock_text = mocker.patch('questionary.text')

    # Define the response for the prompt
    mock_text.return_value.ask.return_value = "Alice"

    result = ask_name()

    assert result == "Alice"

    Test a function that has many questions

    Here’s an example of how to test a function that contains two questionary.text prompts using pytest-mock.

    Let's assume you have a function that asks for the first and last names of a user:

    import questionary

def ask_full_name() -> dict:
    first_name = questionary.text("What's your first name?").ask()
    last_name = questionary.text("What's your last name?").ask()
    return {"first_name": first_name, "last_name": last_name}

    You can mock both questionary.text calls to simulate user input for both the first and last names:

    import pytest
from your_module import ask_full_name

def test_ask_full_name(mocker):
    # Mock the text function for the first name prompt
    mock_text_first = mocker.patch('questionary.text')
    # Define the response for the first name prompt
    mock_text_first.side_effect = ["Alice", "Smith"]

    result = ask_full_name()

    assert result == {"first_name": "Alice", "last_name": "Smith"}

rich⚑

  • New: Adding a footer to a table.

    Adding a footer is not easy task. This answer doesn't work anymore as table doesn't have the add_footer. You need to create the footer in the add_column so you need to have the data that needs to go to the footer before building the rows.

    You would do something like:

    table = Table(title="Star Wars Movies", show_footer=True)
table.add_column("Title", style="magenta", footer='2342')

Coding tools⚑

Singer⚑

  • New: Introduce neotree.

    General keymaps:

    • <cr>: Open the file in the current buffer
    • <s>: Open in a vertical split
    • <S>: Open in an horizontal split
    • <bs>: Navigate one directory up (even if it's the root of the cwd)

    File and directory management:

    • a: Create a new file or directory. Add a / to the end of the name to make a directory.
    • d: Delete the selected file or directory
    • r: Rename the selected file or directory
    • y: Mark file to be copied (supports visual selection)
    • x: Mark file to be cut (supports visual selection)
    • m: Move the selected file or directory
    • c: Copy the selected file or directory

    References:

  • New: Show hidden files.

    return {
  "nvim-neo-tree/neo-tree.nvim",
  opts = {
    filesystem = {
      filtered_items = {
        visible = true,
        show_hidden_count = true,
        hide_dotfiles = false,
        hide_gitignored = true,
        hide_by_name = {
          '.git',
        },
        never_show = {},
      },
    }
  }
}

  • New: Autoclose on open file.

    This example uses the file_open event to close the Neo-tree window when a file is opened. This applies to all windows and all sources at once.

    require("neo-tree").setup({
  event_handlers = {

    {
      event = "file_opened",
      handler = function(file_path)
        -- auto close
        -- vimc.cmd("Neotree close")
        -- OR
        require("neo-tree.command").execute({ action = "close" })
      end
    },

  }
})

  • New: Configuring vim folds.

    Copy the code under implementation in your config file.

  • New: Can't copy file/directory to itself.

    If you want to copy a directory you need to assume that the prompt is done from within the directory. So if you want to copy it to a new name at the same level you need to use ../new-name instead of new-name.

  • New: Introduce the vim foldings workflow.

    ne way to easily work with folds is by using the fold-cycle plugin to be able to press <tab> or <enter> to toggle a fold.

    If you're using lazyvim you can use the next configuration:

    return {
  {
    "jghauser/fold-cycle.nvim",
    config = function()
      require("fold-cycle").setup()
    end,
    keys = {
      {
        "<tab>",
        function()
          return require("fold-cycle").open()
        end,
        desc = "Fold-cycle: open folds",
        silent = true,
      },
      {
        "<cr>",
        function()
          return require("fold-cycle").open()
        end,
        desc = "Fold-cycle: open folds",
        silent = true,
      },
      {
        "<s-tab>",
        function()
          return require("fold-cycle").close()
        end,
        desc = "Fold-cycle: close folds",
        silent = true,
      },
      {
        "zC",
        function()
          return require("fold-cycle").close_all()
        end,
        remap = true,
        silent = true,
        desc = "Fold-cycle: close all folds",
      },
    },
  },
}

  • New: Introduce singer.

    Singer is an open-source standard for writing scripts that move data.

    It describes how data extraction scripts—called “taps” —and data loading scripts—called “targets”— should communicate, allowing them to be used in any combination to move data from any source to any destination. Send data between databases, web APIs, files, queues, and just about anything else you can think of.

    It has many "taps" and "targets" that can help you interact with third party tools without needing to write the code.

    References - Home

  • New: is not well mapped.

    It's because <c-i> is a synonym of <TAB>.

Coding with AI⚑

  • New: Introduce LazyVim.

  • New: Adding plugins configuration.

    Configuring LazyVim plugins is exactly the same as using lazy.nvim to build a config from scratch.

    For the full plugin spec documentation please check the lazy.nvim readme.

    LazyVim comes with a list of preconfigured plugins, check them here before diving on your own.

  • New: Adding a plugin.

    Adding a plugin is as simple as adding the plugin spec to one of the files under `lua/plugins/*.lua``. You can create as many files there as you want.

    You can structure your lua/plugins`` folder with a file per plugin, or a separate file containing all the plugin specs for some functionality. For example:lua/plugins/lsp.lua`

    return {
  -- add symbols-outline
  {
    "simrat39/symbols-outline.nvim",
    cmd = "SymbolsOutline",
    keys = { { "<leader>cs", "<cmd>SymbolsOutline<cr>", desc = "Symbols Outline" } },
    opts = {
      -- add your options that should be passed to the setup() function here
      position = "right",
    },
  },
}

    Customizing plugin specs. Defaults merging rules:

    • cmd: the list of commands will be extended with your custom commands
    • event: the list of events will be extended with your custom events
    • ft: the list of filetypes will be extended with your custom filetypes
    • keys: the list of keymaps will be extended with your custom keymaps
    • opts: your custom opts will be merged with the default opts
    • dependencies: the list of dependencies will be extended with your custom dependencies
    • any other property will override the defaults

    For ft, event, keys, cmd and opts you can instead also specify a values function that can make changes to the default values, or return new values to be used instead.

    -- change trouble config
{
  "folke/trouble.nvim",
  -- opts will be merged with the parent spec
  opts = { use_diagnostic_signs = true },
}

-- add cmp-emoji
{
  "hrsh7th/nvim-cmp",
  dependencies = { "hrsh7th/cmp-emoji" },
  ---@param opts cmp.ConfigSchema
  opts = function(_, opts)
    table.insert(opts.sources, { name = "emoji" })
  end,
}

    Defining the plugin keymaps:

    Adding keys= follows the rules as explained above. You don't have to specify a mode for normal mode keymaps.

    You can also disable a default keymap by setting it to false. To override a keymap, simply add one with the same lhs and a new rhs. For example lua/plugins/telescope.lua

    return {
  "nvim-telescope/telescope.nvim",
  keys = {
    -- disable the keymap to grep files
    {"<leader>/", false},
    -- change a keymap
    { "<leader>ff", "<cmd>Telescope find_files<cr>", desc = "Find Files" },
    -- add a keymap to browse plugin files
    {
      "<leader>fp",
      function() require("telescope.builtin").find_files({ cwd = require("lazy.core.config").options.root }) end,
      desc = "Find Plugin File",
    },
  },
},

    Make sure to use the exact same mode as the keymap you want to disable.

    return {
  "folke/flash.nvim",
  keys = {
    -- disable the default flash keymap
    { "s", mode = { "n", "x", "o" }, false },
  },
}
    You can also return a whole new set of keymaps to be used instead. Or return {} to disable all keymaps for a plugin.

    return {
  "nvim-telescope/telescope.nvim",
  -- replace all Telescope keymaps with only one mapping
  keys = function()
    return {
      { "<leader>ff", "<cmd>Telescope find_files<cr>", desc = "Find Files" },
    }
  end,
},

  • New: Auto update plugins.

    Add this to ~/.config/nvim/lua/config/autocomds.lua

    local function augroup(name)
  return vim.api.nvim_create_augroup("lazyvim_" .. name, { clear = true })
end

vim.api.nvim_create_autocmd("VimEnter", {
  group = augroup("autoupdate"),
  callback = function()
    if require("lazy.status").has_updates then
      require("lazy").update({ show = false })
    end
  end,
})

  • New: Introduce vim keymaps.

    LazyVim comes with some sane default keybindings, you can see them here. You don't need to remember them all, it also comes with which-key to help you remember your keymaps. Just press any key like and you'll see a popup with all possible keymaps starting with .

    • default <leader> is <space>
    • default <localleader> is \

    General editor bindings:

    • Save file: <C-s>
    • Quit all: <leader>qq
    • Open a floating terminal: <C-/>

    Movement keybindings:

    • Split the windows:
    • Vertically: <C-|
    • Horizontally: <C--
    • Delete window: <leader>wd
    • To move around the windows you can use: , , , .
    • To resize the windows use: , , ,
    • To move between buffers:
    • Next and previous with ,
    • Switch to the previously opened buffer: <leader>bb

    Coding keybindings:

    Diagnostics:

    • <leader>cd>: Shows you the diagnostics message of the current line in a floating window
    • ]d and [d: iterates over all diagnostics
    • ]e and [e: iterates over all error diagnostics
    • ]w and [w: iterates over all warning diagnostics

  • New: Setting keymaps in lua.

    If you need to set keymaps in lua you can use vim.keymap.set. For example:

    vim.keymap.set('n', '<space>w', '<cmd>write<cr>', {desc = 'Save'})

    After executing this, the sequence Space + w will call the write command. Basically, we can save changes made to a file with Space + w.

    Let's dive into what does the vim.keymap.set parameters mean.

    vim.keymap.set({mode}, {lhs}, {rhs}, {opts})
    • {mode}: mode where the keybinding should execute. It can be a list of modes. We need to speify the mode's short name. Here are some of the most common.
    • n: Normal mode.
    • i: Insert mode.
    • x: Visual mode.
    • s: Selection mode.
    • v: Visual + Selection.
    • t: Terminal mode.
    • o: Operator-pending.

    • '': Yes, an empty string. Is the equivalent of n + v + o.

    • {lhs}: is the key we want to bind.

    • {rhs} is the action we want to execute. It can be a string with a command or an expression. You can also provide a lua function.

    • {opts} this must be a lua table. If you don't know what is a "lua table" just think is a way of storing several values in one place. Anyway, it can have these properties.

    • desc: A string that describes what the keybinding does. You can write anything you want.

    • remap: A boolean that determines if our keybinding can be recursive. The default value is false. Recursive keybindings can cause some conflicts if used incorrectly. Don't enable it unless you know what you're doing.
    • buffer: It can be a boolean or a number. If we assign the boolean true it means the keybinding will only be effective in the current file. If we assign a number, it needs to be the "id" of an open buffer.
    • silent: A boolean. Determines whether or not the keybindings can show a message. The default value is false.
    • expr: A boolean. If enabled it gives the chance to use vimscript or lua to calculate the value of {rhs}. The default value is false.

  • New: The leader key.

    When creating keybindings we can use the special sequence <leader> in the {lhs} parameter, it'll take the value of the global variable mapleader.

    So mapleader is a global variable in vimscript that can be string. For example.

    vim.g.mapleader = ' '

    After defining it we can use it as a prefix in our keybindings.

    vim.keymap.set('n', '<leader>w', '<cmd>write<cr>')

    This will make <space key> + w save the current file.

    There are different opinions on what key to use as the <leader> key. The <space> is the most comfortable as it's always close to your thumbs, and it works well with both hands. Nevertheless, you can only use it in normal mode, because in insert <space><whatever> will be triggered as you write. An alternative is to use ; which is also comfortable (if you use the english key distribution) and you can use it in insert mode.

    If you want to define more than one leader key you can either:

    • Change the mapleader many times in your file: As the value of mapleader is used at the moment the mapping is defined, you can indeed change that while plugins are loading. For that, you have to explicitly :runtime the plugins in your ~/.vimrc (and count on the canonical include guard to prevent redefinition later):

    let mapleader = ','
runtime! plugin/NERD_commenter.vim
runtime! ...
let mapleader = '\'
runime! plugin/mark.vim
...
    * Use the keys directly instead of using <leader>

    " editing mappings
nnoremap ,a <something>
nnoremap ,k <something else>
nnoremap ,d <and something else>

" window management mappings
nnoremap gw <something>
nnoremap gb <something else>

    Defining mapleader and/or using <leader> may be useful if you change your mind often on what key to use a leader but it won't be of any use if your mappings are stable.

  • New: Configure vim from scratch.

    Neovim configuration is a complex thing to do, both to start and to maintain. The configurations are endless, the plugins are too. Be ready to spend a lot of energy on it and to get lost reading a lot.

    If I'm scaring you, you are right to be scared! xD. Once you manage to get it configured to your liking you'll think that in the end it doesn't even matter spending all that time. However if you're searching for something that is plug and play try vscodium.

    To make things worse, the configuration is done in lua, so you may need a small refreshment to understand what are you doing.

  • New: Vim distributions.

    One way to make vim's configuration more bearable is to use vim distributions. These are projects that maintain configurations with sane defaults and that work with the whole ecosystem of plugins.

    Using them is the best way to:

    • Have something usable fast
    • Minimize the maintenance efforts as others are doing it for you (plugin changes, breaking changes, ..)
    • Keep updated with the neovim ecosystem, as you can see what is the community adding to the default config.

    However, there are so many good Neovim configuration distributions that it becomes difficult for a Neovim user to decide which distribution to use and how to tailor it for their use case.

    By far, the top 5 Neovim configuration distributions are AstroNvim, kickstart, LazyVim, LunarVim, and NvChad. That is not to say these are the “best” configuration distributions, simply that they are the most popular.

    Each of these configuration distributions has value. They all provide excellent starting points for crafting your own custom configuration, they are all extensible and fairly easy to learn, and they all provide an out-of-the-box setup that can be used effectively without modification.

    Distinguishing features of the top Neovim configuration distributions are:

    • AstroNvim:

      • An excellent community repository
      • Fully featured out-of-the-box
      • Good documentation

    • kickstart

      • Minimal out-of-the-box setup
      • Easy to extend and widely used as a starting point
      • A good choice if your goal is hand-crafting your own config

    • LazyVim

      • Very well maintained by the author of lazy.nvim
      • Nice architecture, it’s a plugin with which you can import preconfigured plugins
      • Good documentation

    • LunarVim

      • Well maintained and mature
      • Custom installation processs installs LunarVim in an isolated location
      • Been around a while, large community, widespread presence on the web

    • NvChad

      • Really great base46 plugin enables easy theme/colorscheme management
      • Includes an impressive mappings cheatsheet
      • ui plugin and nvim-colorizer

    Personally I tried LunarVim and finally ended up with LazyVim because:

    • It's more popular
    • I like the file structure
    • It's being maintained by folke one of the best developers of neovim plugins.

  • New: Starting your configuration with LazyVim.

    Installing the requirements:

    LazyVim needs the next tools to be able to work:

    • Neovim >= 0.9.0 (needs to be built with LuaJIT). Follow these instructions
    • Git >= 2.19.0 (for partial clones support). sudo apt-get install git.
    • a Nerd Font(v3.0 or greater) (optional, but strongly suggested as they rae needed to display some icons). Follow these instructions if you're using kitty.
    • lazygit (optional and I didn't like it)
    • a C compiler for nvim-treesitter. apt-get install gcc
    • for telescope.nvim (optional)
    • live grep: ripgrep
    • find files: fd
    • a terminal that support true color and undercurl:
    • kitty (Linux & Macos)
    • wezterm (Linux, Macos & Windows)
    • alacritty (Linux, Macos & Windows)
    • iterm2 (Macos)

    Install the starter:

    • Make a backup of your current Neovim files: 
      # required
mv ~/.config/nvim{,.old}

# optional but recommended
mv ~/.local/share/nvim{,.old}
mv ~/.local/state/nvim{,.old}
mv ~/.cache/nvim{,.old}

    • Clone the starter

      git clone https://github.com/LazyVim/starter ~/.config/nvim

    • Remove the .git folder, so you can add it to your own repo later

      rm -rf ~/.config/nvim/.git

    • Start Neovim!

      nvim
      - It is recommended to run :LazyHealth after installation. This will load all plugins and check if everything is working correctly.

    Understanding the file structure:

    The files under config will be automatically loaded at the appropriate time, so you don't need to require those files manually.

    You can add your custom plugin specs under lua/plugins/. All files there will be automatically loaded by lazy.nvim.

    ~/.config/nvim
├── lua
│   ├── config
│   │   ├── autocmds.lua
│   │   ├── keymaps.lua
│   │   ├── lazy.lua
│   │   └── options.lua
│   └── plugins
│       ├── spec1.lua
│       ├── **
│       └── spec2.lua
└── init.toml
    The files autocmds.lua, keymaps.lua, lazy.lua and options.lua under lua/config will be automatically loaded at the appropriate time, so you don't need to require those files manually. LazyVim comes with a set of default config files that will be loaded before your own.

    You can continue your config by adding plugins.

  • New: Introduce the vim movement workflow.

    Moving around vim can be done in many ways, which an lead to being lost on how to do it well.

    LazyVim has a very nice way to deal with buffers - Use H and L if the buffer you want to go to is visually close to where you are. - Otherwise, if the buffer is open, use <leader>, - For other files, use <leader><space> - Close buffers you no longer need with <leader>bd - <leader>ss to quickly jump to a function in the buffer you're on - Using the jump list with <c-o>, <c-i> and gd to navigate the code - You can pin buffers with <leader>bp and delete all non pinned buffers with <leader>bP

  • New: Using the jump list.

    Vim has a feature called the “Jump List”, which saves all the locations you’ve recently visited, including their line number, column number, and what else not in the .viminfo file, to help you get exactly the position you were last in. Not only does it save the locations in your current buffer, but also previous buffers you may have edited in other Vim sessions. Which means, if you’re currently working on a file, and there aren’t many last-location saves in this one, you’ll be redirected to the previous file you had edited. But how do you do that? Simply press Ctrl + O, and it’ll get you back to the previous location you were in, or more specifically, your cursor was in.

    If you want to go back to the newer positions, after you’re done with what you wanted to do, you can then press Ctrl + i to go back to the newer position. This is exceptionally useful when you’re working with a lot of project files at a time, and you need to go back and forth between multiple blocks in different files. This could instantly give you a boost, as you won’t need to have separate buffers opened up or windows to be setted up, you can simply jump between the files and edit them.

    Ctrl + O is probably not meant for a single task, as far as Vim’s philosophy is concerned. The jumping mentioned in the previous section only works when you’re in Normal Mode, and not when you’re in Insert Mode. When you press Ctrl + O in Insert Mode, what happens instead is that you’ll enter Normal Mode, and be able to execute a single command, after which Vim will automatically switch back to Insert Mode.

  • New: Install using Lazyvim.

    return {
  {
    "sindrets/diffview.nvim",
    dependencies = {
      { "nvim-tree/nvim-web-devicons", lazy = true },
    },

    keys = {
      {
        "dv",
        function()
          if next(require("diffview.lib").views) == nil then
            vim.cmd("DiffviewOpen")
          else
            vim.cmd("DiffviewClose")
          end
        end,
        desc = "Toggle Diffview window",
      },
    },
  },
}

    Which sets the next bindings: - dv: Toggle the opening and closing of the diffview windows

  • New: Use diffview as merge tool.

    Add to your ~/.gitconfig:

    [alias]
  mergetool = "!nvim -c DiffviewOpen"

  • New: Resolve merge conflicts.

    If you call :DiffviewOpen during a merge or a rebase, the view will list the conflicted files in their own section. When opening a conflicted file, it will open in a 3-way diff allowing you to resolve the conflict with the context of the target branch's version (OURS, left), and the version from the branch which is being merged (THEIRS, right).

    The conflicted file's entry in the file panel will show the remaining number of conflict markers (the number following the file name). If what follows the file name is instead an exclamation mark (!), this indicates that the file has not yet been opened, and the number of conflicts is unknown. If the sign is a check mark, this indicates that there are no more conflicts in the file.

    You can interact with the merge tool with the next bindings:

    • ]x and [x: Jump between conflict markers. This works from the file panel as well.
    • dp: Put the contents on the other buffer
    • do: Get the contents from the other buffer
    • 2do: to obtain the hunk from the OURS side of the diff
    • 3do to obtain the hunk from the THEIRS side of the diff
    • 1do to obtain the hunk from the BASE in a 4-way diff

    Additionally there are mappings for operating directly on the conflict markers:

    • <leader>co: Choose the OURS version of the conflict.
    • <leader>ct: Choose the THEIRS version of the conflict.
    • <leader>cb: Choose the BASE version of the conflict.
    • <leader>ca: Choose all versions of the conflict (effectively just deletes the markers, leaving all the content).
    • dx: Choose none of the versions of the conflict (delete the conflict region).

  • New: Introduce ai coding prompts.

    These are some useful AI prompts to help you while you code:

    • create a function with type hints and docstring using google style called { } that { }
    • create the tests for the function { } adding type hints and following the AAA style where the Act section is represented contains a returns = (thing to test) line or if the function to test doesn't return any value append an # act comment at the end of the line. Use paragraphs to separate the AAA blocks and don't add comments inside the tests for the sections

    If you use espanso you can simplify the filling up of these prompts on the AI chats. For example:

    ---
matches:
  - trigger: :function
    form: |
      Create a function with type hints and docstring using google style called [[name]] that:
      [[text]]
    form_fields:
      text:
        multiline: true
  - trigger: :tweak
    form: |
      Tweak the next code:
      [[code]]

      So that:

      [[text]]
    form_fields:
      text:
        multiline: true
      code:
        multiline: true
  - trigger: :test
    form: |
      create the tests for the function:
      [[text]]

      Following the next guidelines:

      - Add type hints
      - Follow the AAA style
      - In the Act section if the function to test returns a value always name that variable returns. If the function to test doesn't return any value append an # act comment at the end of the line.
      - Use paragraphs to separate the AAA blocks and don't add comments like # Arrange or # Act or # Act/Assert or # Assert

    form_fields:
      text:
        multiline: true
  - trigger: :refactor
    form: |
     Refactor the next code
     [[code]]
     with the next conditions
     [[conditions]]
    form_fields:
      code:
        multiline: true
      conditions:
        multiline: true

  • New: Introduce Kestra.

    Kestra is an open-source orchestrator designed to bring Infrastructure as Code (IaC) best practices to all workflows — from those orchestrating mission-critical operations, business processes, and data pipelines to simple Zapier-style automation. Built with an API-first philosophy, Kestra enables users to define and manage data pipelines through a simple YAML configuration file. This approach frees you from being tied to a specific client implementation, allowing for greater flexibility and easier integration with various tools and services.

    Look at this 4 minute video for a visual introduction

    References - Docs - Home - 4 minute introduction video

  • New: Add new prompts for developers.

      - trigger: :polish
    form: |
     Polish the next code
     [[code]]
     with the next conditions:
     - Use type hints on all functions and methods
     - Add or update the docstring using google style on all functions and methods
    form_fields:
      code:
        multiline: true
  - trigger: :commit
    form: |
     Act as an expert developer. Create a message commit with the next conditions:
     - follow semantic versioning
     - create a semantic version comment per change
     - include all comments in a raw code block so that it's easy to copy

     for the following diff
     [[text]]
    form_fields:
      text:
        multiline: true

  • Correction: Update the ai prompts.

    ```yaml matches: - trigger: :function form: | Create a function with: - type hints - docstrings for all classes, functions and methods - docstring using google style with line length less than 89 characters - adding logging traces using the log variable log = logging.getLogger(name) - Use fstrings instead of %s - If you need to open or write a file always set the encoding to utf8 - If possible add an example in the docstring - Just give the code, don't explain anything

      Called [[name]] that:
  [[text]]
form_fields:
  text:
    multiline: true

    • trigger: :class form: | Create a class with:

      • type hints
      • docstring using google style with line length less than 89 characters
      • use docstrings on the class and each methods
      • adding logging traces using the log variable log = logging.getLogger(name)
      • Use fstrings instead of %s
      • If you need to open or write a file always set the encoding to utf8
      • If possible add an example in the docstring
      • Just give the code, don't explain anything

      Called [[name]] that: [[text]] form_fields: text: - trigger: :class form: | ... - Use paragraphs to separate the AAA blocks and don't add comments like # Arrange or # Act or # Act/Assert or # Assert. So the test will only have black lines between sections - In the Act section if the function to test returns a value always name that variable result. If the function to test doesn't return any value append an # act comment at the end of the line. - If the test uses a pytest.raises there is no need to add the # act comment - Don't use mocks - Use fstrings instead of %s - Gather all tests over the same function on a common class - If you need to open or write a file always set the encoding to utf8 - Just give the code, don't explain anything

      form_fields: text: - trigger: :polish form: | ... - Add or update the docstring using google style on all classes, functions and methods - Wrap the docstring lines so they are smaller than 89 characters - All docstrings must start in the same line as the """ - Add logging traces using the log variable log = logging.getLogger(name) - Use f-strings instead of %s - Just give the code, don't explain anything form_fields: code: multiline: true - trigger: :text form: | Polish the next text by:

      • Summarising each section without losing relevant data
      • Tweak the markdown format
      • Improve the wording

      [[text]] form_fields: text: multiline: true

    • trigger: :readme form: | Create the README.md taking into account:

      • Use GPLv3 for the license
      • Add Lyz as the author
      • Add an installation section
      • Add an usage section

      of: [[text]]

      form_fields: text: multiline: true ``` feat(aleph#Get all documents of a collection): Get all documents of a collection

    list_aleph_collection_documents.py is a Python script designed to interact with an API to retrieve and analyze documents from specified collections. It offers a command-line interface (CLI) to list and check documents within a specified collection.

    Features

    • Retrieve documents from a specified collection.
    • Analyze document processing statuses and warn if any are not marked as successful.
    • Return a list of filenames from the retrieved documents.
    • Supports verbose output for detailed logging.
    • Environment variable support for API key management.

    Installation

    To install the required dependencies, use pip:

    pip install typer requests

    Ensure you have Python 3.6 or higher installed.

    Create the file list_aleph_collection_documents.py with the next contents:

    import logging
import requests
from typing import List, Dict, Any, Optional
import logging
import typer
from typing import List, Dict, Any

log = logging.getLogger(__name__)
app = typer.Typer()

@app.command()
def get_documents(
    collection_name: str = typer.Argument(...),
    api_key: Optional[str] = typer.Option(None, envvar="API_KEY"),
    base_url: str = typer.Option("https://your.aleph.org"),
    verbose: bool = typer.Option(
        False, "--verbose", "-v", help="Enable verbose output"
    ),
):
    """CLI command to retrieve documents from a specified collection."""
    if verbose:
        logging.basicConfig(level=logging.DEBUG)
        log.debug("Verbose mode enabled.")
    else:
        logging.basicConfig(level=logging.INFO)
    if api_key is None:
        log.error(
            "Please specify your api key either through the --api-key argument "
            "or through the API_KEY environment variable"
        )
        raise typer.Exit(code=1)
    try:
        documents = list_collection_documents(api_key, base_url, collection_name)
        filenames = check_documents(documents)
        if filenames:
            print("\n".join(filenames))
        else:
            log.warning("No documents found.")
    except Exception as e:
        log.error(f"Failed to retrieve documents: {e}")
        raise typer.Exit(code=1)

def list_collection_documents(
    api_key: str, base_url: str, collection_name: str
) -> List[Dict[str, Any]]:
    """
    Retrieve documents from a specified collection using pagination.

    Args:
        api_key (str): The API key for authentication.
        base_url (str): The base URL of the API.
        collection_name (str): The name of the collection to retrieve documents from.

    Returns:
        List[Dict[str, Any]]: A list of documents from the specified collection.

    Example:
        >>> docs = list_collection_documents("your_api_key", "https://api.example.com", "my_collection")
        >>> print(len(docs))
        1000
    """
    headers = {
        "Authorization": f"ApiKey {api_key}",
        "Accept": "application/json",
        "Content-Type": "application/json",
    }

    collections_url = f"{base_url}/api/2/collections"
    documents_url = f"{base_url}/api/2/entities"
    log.debug(f"Requesting collections list from {collections_url}")
    collections = []
    params = {"limit": 300}

    while True:
        response = requests.get(collections_url, headers=headers, params=params)
        response.raise_for_status()
        data = response.json()
        collections.extend(data["results"])
        log.debug(
            f"Fetched {len(data['results'])} collections, "
            f"page {data['page']} of {data['pages']}"
        )
        if not data["next"]:
            break
        params["offset"] = params.get("offset", 0) + data["limit"]

    collection_id = next(
        (c["id"] for c in collections if c["label"] == collection_name), None
    )
    if not collection_id:
        log.error(f"Collection {collection_name} not found.")
        return []

    log.info(f"Found collection '{collection_name}' with ID {collection_id}")

    documents = []
    params = {
        "q": "",
        "filter:collection_id": collection_id,
        "filter:schemata": "Document",
        "limit": 300,
    }

    while True:
        log.debug(f"Requesting documents from collection {collection_id}")
        response = requests.get(documents_url, headers=headers, params=params)
        response.raise_for_status()
        data = response.json()
        documents.extend(data["results"])
        log.info(
            f"Fetched {len(data['results'])} documents, "
            f"page {data['page']} of {data['pages']}"
        )
        if not data["next"]:
            break
        params["offset"] = params.get("offset", 0) + data["limit"]

    log.info(f"Retrieved {len(documents)} documents from collection {collection_name}")

    return documents

def check_documents(documents: List[Dict[str, Any]]) -> List[str]:
    """Analyze the processing status of documents and return a list of filenames.

    Args:
        documents (List[Dict[str, Any]]): A list of documents in JSON format.

    Returns:
        List[str]: A list of filenames from documents with a successful processing status.

    Raises:
        None, but logs warnings if a document's processing status is not 'success'.

    Example:
        >>> docs = [{"properties": {"processingStatus": ["success"], "fileName": ["file1.txt"]}},
        >>>         {"properties": {"processingStatus": ["failed"], "fileName": ["file2.txt"]}}]
        >>> filenames = check_documents(docs)
        >>> print(filenames)
        ['file1.txt']
    """
    filenames = []

    for doc in documents:
        status = doc.get("properties", {}).get("processingStatus")[0]
        filename = doc.get("properties", {}).get("fileName")[0]

        if status != "success":
            log.warning(
                f"Document with filename {filename} has processing status: {status}"
            )

        if filename:
            filenames.append(filename)

    log.debug(f"Collected filenames: {filenames}")
    return filenames

if __name__ == "__main__":
    app()

    Get your API key

    By default, any Aleph search will return only public documents in responses to API requests.

    If you want to access documents which are not marked public, you will need to sign into the tool. This can be done through the use on an API key. The API key for any account can be found by clicking on the "Settings" menu item in the navigation menu.

    Usage

    You can run the script directly from the command line. Below are examples of usage:

    Retrieve and list documents from a collection:

    python list_aleph_collection_documents.py --api-key "your-api-key" 'Name of your collection'

    Using an Environment Variable for the API Key

    This is better from a security perspective. 

    export API_KEY=your_api_key
python list_aleph_collection_documents.py 'Name of your collection'

    Enabling Verbose Logging

    To enable detailed debug logs, use the --verbose or -v flag:

    python list_aleph_collection_documents.py -v 'Name of your collection'

    Getting help

    python list_aleph_collection_documents.py --help

memorious⚑

  • New: Switch to the previous opened buffer.

    Often the buffer that you want to edit is the buffer that you have just left. Vim provides a couple of convenient commands to switch back to the previous buffer. These are <C-^> (or <C-6>) and :b#. All of them are inconvenient so I use the next mapping:

    nnoremap <Leader><Tab> :b#<CR>

  • New: Troubleshoot Undefined global vim warning.

    Added to my lua/plugins directory:

      {
    "neovim/nvim-lspconfig",
    opts = {
      servers = {
        lua_ls = {
          settings = {
            Lua = {
              diagnostics = {
                globals = { "vim" },
              },
            },
          },
        },
      },
    },
  },

  • New: Get the version of the packages installed by Packer.

    Go into the plugin directory cd ~/.local/share/nvim/site/pack/packer/start/your-plugin and check it with git

  • New: Introduce memorious.

    Memorious is a light-weight web scraping toolkit. It supports scrapers that collect structured or un-structured data. This includes the following use cases:

    • Make crawlers modular and simple tasks re-usable
    • Provide utility functions to do common tasks such as data storage, HTTP session management
    • Integrate crawlers with the Aleph and FollowTheMoney ecosystem

    References

Data orchestrators⚑

  • Correction: Update disable regular login with oauth.

    The last signin_inner.tmpl failed with the latest version. I've uploaded the new working one.

  • New: Configure vim to work with markdown.

    Markdown specific plugins:

  • New: Enable folds.

    If you have set the foldmethod to indent by default you won't be able to use folds in markdown.

    To fix this you can create the next autocommand (in lua/config/autocmds.lua if you're using lazyvim).

    vim.api.nvim_create_autocmd("FileType", {
  pattern = "markdown",
  callback = function()
    vim.wo.foldmethod = "expr"
    vim.wo.foldexpr = "v:lua.vim.treesitter.foldexpr()"
  end,
})

  • New: Aligning tables in markdown.

    In the past I used Tabular but it doesn't work with the latest neovim and the project didn't have any update in the last 5 years.

    A good way to achieve this without installing any plugin is to:

    • select the table, including the header and footer lines (with shift V, for example).
    • Prettify the table with :!column -t -s '|' -o '|'

    If you don't want to remember that command you can bind it to a key:

    vim.keymap.set("v", "<leader>tf", "!column -t -s '|' -o '|'<cr>", { desc = "Format table" })

    How the hell this works?

    • shift V switches to Visual mode linewise. This is to select all the lines of the table.
    • : switches to Command line mode, to type commands.
    • ! specifies a filter command. This means we will send data to a command to modify it (or to filter) and replace the original lines. In this case we are in Visual mode, we defined the input text (the selected lines) and we will use an external command to modify the data.
    • column is the filter command we are using, from the util-linux package. column’s purpose is to “columnate”. The -t flag tells column to use the Table mode. The -s flag specifies the delimiters in the input data (the default is whitespace). And the -o flag is to specify the output delimiter to use (we need that because the default is two whitespaces).

  • New: Fix Server does not allow request for unadvertised object error.

    Fetching the whole history with fetch-depth: 0 worked for us:

    - name: Checkout the codebase
  uses: actions/checkout@v3
  with:
    fetch-depth: 0

  • New: Introduce data orchestrators.

    Data orchestration is the process of moving siloed data from multiple storage locations into a centralized repository where it can then be combined, cleaned, and enriched for activation.

    Data orchestrators are web applications that make this process easy. The most popular right now are:

    • Apache Airflow
    • Kestra
    • Prefect

    There are several comparison pages:

    When looking at the return on investment when choosing an orchestration tool, there are several points to consider:

    • Time of installation/maintenance
    • Time to write pipeline
    • Time to execute (performance)

    Kestra

    Pros:

    Cons:

    Kestra offers a higher ROI globally compared to Airflow:

    • Installing Kestra is easier than Airflow; it doesn’t require Python dependencies, and it comes with a ready-to-use docker-compose file using few services and without the need to understand what’s an executor to run task in parallel.
    • Creating pipelines with Kestra is simple, thanks to its syntax. You don’t need knowledge of a specific programming language because Kestra is designed to be agnostic. The declarative YAML design makes Kestra flows more readable compared to Airflow’s DAG equivalent, allowing developers to significantly reduce development time.
    • In this benchmark, Kestra demonstrates better execution time than Airflow under any configuration setup.

Scrapers⚑

  • New: Introduce Debug Adapter Protocol.

    nvim-dap](https://github.com/mfussenegger/nvim-dap) implements a client for the Debug Adapter Protocol. This allows a client to control a debugger over a documented API. That allows us to control the debugger from inside neovim, being able to set breakpoints, evaluate runtime values of variables, and much more.

    nvim-dap is not configured for any language by default. You will need to set up a configuration for each language. For the configurations you will need adapters to run.

    I would suggest starting with 2 actions. Setting breakpoints and “running” the debugger. The debugger allows us to stop execution and look at the current state of the program. Setting breakpoints will allow us to stop execution and see what the current state is.

    vim.api.nvim_set_keymap('n', '<leader>b', [[:lua require"dap".toggle_breakpoint()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>c', [[:lua require"dap".continue()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>n', [[:lua require"dap".step_over()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>N', [[:lua require"dap".step_into()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<F5>', [[:lua require"osv".launch({port = 8086})<CR>]], { noremap = true })

    Go to a line where a conditional or value is set and toggle a breakpoint. Then, we’ll start the debugger. If done correctly, you’ll see an arrow next to your line of code you set a breakpoint at.

    There is no UI with dap by default. You have a few options for UI nvim-dap-ui

    In the dap repl you can use the next operations:

    • .exit: Closes the REPL
    • .c or .continue: Same as |dap.continue|
    • .n or .next: Same as |dap.step_over|
    • .into: Same as |dap.step_into|
    • .into_target: Same as |dap.step_into{askForTargets=true}|
    • .out: Same as |dap.step_out|
    • .up: Same as |dap.up|
    • .down: Same as |dap.down|
    • .goto: Same as |dap.goto_|
    • .scopes: Prints the variables in the current scopes
    • .threads: Prints all threads
    • .frames: Print the stack frames
    • .capabilities: Print the capabilities of the debug adapter
    • .b or .back: Same as |dap.step_back|
    • .rc or .reverse-continue: Same as |dap.reverse_continue|

  • New: Introduce nvim-dap-ui.

    Install with packer:

    use { "rcarriga/nvim-dap-ui", requires = {"mfussenegger/nvim-dap"} }

    It is highly recommended to use neodev.nvim to enable type checking for nvim-dap-ui to get type checking, documentation and autocompletion for all API functions.

    require("neodev").setup({
  library = { plugins = { "nvim-dap-ui" }, types = true },
  ...
})
    nvim-dap-ui is built on the idea of "elements". These elements are windows which provide different features.

    Elements are grouped into layouts which can be placed on any side of the screen. There can be any number of layouts, containing whichever elements desired.

    Elements can also be displayed temporarily in a floating window.

    Each element has a set of mappings for element-specific possible actions, detailed below for each element. The total set of actions/mappings and their default shortcuts are:

    • edit: e
    • expand: <CR> or left click
    • open: o
    • remove: d
    • repl: r
    • toggle: t

    See :h dapui.setup() for configuration options and defaults.

    To get started simply call the setup method on startup, optionally providing custom settings.

    require("dapui").setup()

    You can open, close and toggle the windows with corresponding functions:

    require("dapui").open()
require("dapui").close()
require("dapui").toggle()

  • New: Debug neovim plugins with DAP.

    one-small-step-for-vimkind is an adapter for the Neovim lua language. It allows you to debug any lua code running in a Neovim instance.

    Install it with Packer:

    use 'mfussenegger/nvim-dap'
    After installing one-small-step-for-vimkind, you will also need a DAP plugin which will allow you to interact with the adapter. Check the install instructions here.

    Then add these lines to your config:

    local dap = require"dap"
dap.configurations.lua = {
  {
    type = 'nlua',
    request = 'attach',
    name = "Attach to running Neovim instance",
  }
}

dap.adapters.nlua = function(callback, config)
  callback({ type = 'server', host = config.host or "127.0.0.1", port = config.port or 8086 })
end

  • New: Debugging with DAP.

    You can debug Lua code running in a separate Neovim instance with jbyuki/one-small-step-for-vimkind.

    The plugin uses the Debug Adapter Protocol. Connecting to a debug adapter requires a DAP client like mfussenegger/nvim-dap. Check how to configure here

    Once you have all set up and assuming you're using the next keybindings for nvim-dap:

    vim.api.nvim_set_keymap('n', '<leader>b', [[:lua require"dap".toggle_breakpoint()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>c', [[:lua require"dap".continue()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>n', [[:lua require"dap".step_over()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>N', [[:lua require"dap".step_into()<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<F5>', [[:lua require"osv".launch({port = 8086})<CR>]], { noremap = true })
vim.api.nvim_set_keymap('n', '<leader>B', [[:lua require"dapui".toggle()<CR>]], { noremap = true })

    You will debug the plugin by:

    • Launch the server in the debuggee using F5.
    • Open another Neovim instance with the source file (the debugger).
    • Place breakpoint with <leader>b.
    • On the debugger connect to the DAP client with <leader>c.
    • Optionally open the nvim-dap-ui with <leader>B in the debugger.
    • Run your script/plugin in the debuggee
    • Interact in the debugger using <leader>n to step to the next step, and <leader>N to step into. Then use the dap console to inspect and change the values of the state.

  • New: Introduce morph.io.

    morph.io is a web service that runs your scrapers for you.

    Write your scraper in the language you know and love, push your code to GitHub, and they take care of the boring bits. Things like running your scraper regularly, alerting you if there's a problem, storing your data, and making your data available for download or through a super-simple API.

    To sign in you'll need a GitHub account. This is where your scraper code is stored.

    The data is stored in an sqlite

    Usage limits

    Right now there are very few limits. They are trusting you that you won't abuse this.

    However, they do impose a couple of hard limits on running scrapers so they don't take up too many resources

    • max 512 MB memory
    • max 24 hours run time for a single run

    If a scraper runs out of memory or runs too long it will get killed automatically.

    There's also a soft limit:

    • max 10,000 lines of log output

    If a scraper generates more than 10,000 lines of log output the scraper will continue running uninterrupted. You just won't see any more output than that. To avoid this happening simply print less stuff to the screen.

    Note that they are keeping track of the amount of cpu time (and a whole bunch of other metrics) that you and your scrapers are using. So, if they do find that you are using too much they reserve the right to kick you out. In reality first they'll ask you nicely to stop.

    References

Vim autosave⚑

  • Correction: Search for alternatives to git-sweep.

    The tool is no longer maintained but there is still no good alternative. I've found some but are either not popular and/or not maintained:

  • New: Update all git submodules.

    If it's the first time you check-out a repo you need to use --init first:

    git submodule update --init --recursive

    To update to latest tips of remote branches use:

    git submodule update --recursive --remote

  • New: Manually toggle the autosave function.

    Besides running auto-save at startup (if you have enabled = true in your config), you may as well:

    • ASToggle: toggle auto-save

Espanso⚑

  • New: Introduce espanso.

    Espanso is a cross-platform Text Expander written in Rust.

    A text expander is a program that detects when you type a specific keyword and replaces it with something else. This is useful in many ways:

    • Save a lot of typing, expanding common sentences or fixing common typos.
    • Create system-wide code snippets.
    • Execute custom scripts
    • Use emojis like a pro.

    Installation Espanso ships with a .deb package, making the installation convenient on Debian-based systems.

    Start by downloading the package by running the following command inside a terminal:

    wget https://github.com/federico-terzi/espanso/releases/download/v2.2.1/espanso-debian-x11-amd64.deb

    You can now install the package using:

    sudo apt install ./espanso-debian-x11-amd64.deb

    From now on, you should have the espanso command available in the terminal (you can verify by running espanso --version).

    At this point, you are ready to use espanso by registering it first as a Systemd service and then starting it with:

    espanso service register

    Start espanso

    espanso start

    Espanso ships with very few built-in matches to give you the maximum flexibility, but you can expand its capabilities in two ways: creating your own custom matches or installing packages.

    Configuration

    Your configuration lives at ~/.config/espanso. A quick way to find the path of your configuration folder is by using the following command espanso path.

    • The files contained in the match directory define what Espanso should do. In other words, this is where you should specify all the custom snippets and actions (aka Matches). The match/base.yml file is where you might want to start adding your matches.
    • The files contained in the config directory define how Espanso should perform its expansions. In other words, this is were you should specify all Espanso's parameters and options. The config/default.yml file defines the options that will be applied to all applications by default, unless an app-specific configuration is present for the current app.

    Using packages

    Custom matches are great, but sometimes it can be tedious to define them for every common operation, especially when you want to share them with other people.

    Espanso offers an easy way to share and reuse matches with other people, packages. In fact, they are so important that Espanso includes a built-in package manager and a store, the Espanso Hub.

    Installing a package

    Get the id of the package from the Espanso Hub and then run espanso install <<package_name>>.

    Of all the packages, I've found the next ones the most useful:

    Overwriting the snippets of a package

    For example the typofixer-en replaces si to is, although si is a valid spanish word. To override the fix you can create your own file on ~/.config/espanso/match/typofix_overwrite.yml with the next content:

    matches:
  # Simple text replacement
  - trigger: "si"
    replace: "si"

    Creating a package

    Auto-restart on config changes

    Set auto_restart: true on ~/.config/espanso/config/default.yml.

    Changing the search bar shortcut

    If the default search bar shortcut conflicts with your i3 configuration set it with:

    search_shortcut: CTRL+SHIFT+e

    Hiding the notifications

    You can hide the notifications by adding the following option to your $CONFIG/config/default.yml config:

    show_notifications: false

    Usage

    Just type and you'll see the text expanded.

    You can use the search bar if you don't remember your snippets.

    References - Code - Docs

  • New: Desktop application to add words easily.

    Going into the espanso config files to add words is cumbersome, to make things easier you can use the espansadd Python script.

    I'm going to assume that you have the following prerequisites:

    • A Linux distribution with i3 window manager installed.
    • Python 3 installed.
    • Espanso installed and configured.
    • ruyaml and tkinter Python libraries installed.
    • notify-send installed.
    • Basic knowledge of editing configuration files in i3.

    Installation

    Create a new Python script named espansadd.py with the following content:

    import tkinter as tk
from tkinter import simpledialog
import traceback
import subprocess
import os
import sys

from ruyaml import YAML
from ruyaml.scanner import ScannerError

file_path = os.path.expanduser("~/.config/espanso/match/typofixer_overwrite.yml")

def append_to_yaml(file_path: str, trigger: str, replace: str) -> None:
    """Appends a new entry to the YAML file.

    Args:ath
        file_path (str): The file to append the new entry.
        trigger (str): The trigger string to be added.
        replace (str): The replacement string to be added.
    """

    # Define the new snippet
    new_entry = {
        "trigger": trigger,
        "replace": replace,
        "propagate_case": True,
        "word": True,
    }

    # Load existing data or initialize an empty list
    try:
        with open(os.path.expanduser(file_path), "r") as f:
            try:
                data = YAML().load(f)
            except ScannerError as e:
                send_notification(
                    f"Error parsing yaml of configuration file {file_path}",
                    f"{e.problem_mark}: {e.problem}",
                    "critical",
                )
                sys.exit(1)
    except FileNotFoundError:
        send_notification(
            f"Error opening the espanso file {file_path}", urgency="critical"
        )
        sys.exit(1)

    data["matches"].append(new_entry)

    # Write the updated data back to the file
    with open(os.path.expanduser(file_path), "w+") as f:
        yaml = YAML()
        yaml.default_flow_style = False
        yaml.dump(data, f)

def send_notification(title: str, message: str = "", urgency: str = "normal") -> None:
    """Send a desktop notification using notify-send.

    Args:
        title (str): The title of the notification.
        message (str): The message body of the notification. Defaults to an empty string.
        urgency (str): The urgency level of the notification. Can be 'low', 'normal', or 'critical'. Defaults to 'normal'.
    """
    subprocess.run(["notify-send", "-u", urgency, title, message])

def main() -> None:
    """Main function to prompt user for input and append to the YAML file."""
    # Create the main Tkinter window (it won't be shown)
    window = tk.Tk()
    window.withdraw()  # Hide the main window

    # Prompt the user for input
    trigger = simpledialog.askstring("Espanso add input", "Enter trigger:")
    replace = simpledialog.askstring("Espanso add input", "Enter replace:")

    # Check if both inputs were provided
    try:
        if trigger and replace:
            append_to_yaml(file_path, trigger, replace)
            send_notification("Espanso snippet added successfully")
        else:
            send_notification(
                "Both trigger and replace are required", urgency="critical"
            )
    except Exception as error:
        error_message = "".join(
            traceback.format_exception(None, error, error.__traceback__)
        )
        send_notification(
            "There was an unknown error adding the espanso entry",
            error_message,
            urgency="critical",
        )

if __name__ == "__main__":
    main()

    Ensure the script has executable permissions. Run the following command:

    chmod +x espansadd.py

    To make the espansadd script easily accessible, we can configure a key binding in i3 to run the script. Open your i3 configuration file, typically located at ~/.config/i3/config or ~/.i3/config, and add the following lines:

    bindsym $mod+Shift+e exec --no-startup-id /path/to/your/espansadd.py

    Replace /path/to/your/espansadd.py with the actual path to your script.

    If you also want the popup windows to be in floating mode add

    for_window [title="Espanso add input"] floating enable

    After editing the configuration file, reload i3 to apply the changes. You can do this by pressing Mod + Shift + R (where Mod is typically the Super or Windows key) or by running the following command:

    i3-msg reload

    Usage

    Now that everything is set up, you can use the espansadd script by pressing Mod + Shift + E. This will open a dialog where you can enter the trigger and replacement text for the new Espanso snippet. After entering the information and pressing Enter, a notification will appear confirming the snippet has been added, or showing an error message if something went wrong.

Generic Coding Practices⚑

Writing good documentation⚑

Conventional comments⚑

  • New: Introduce conventional comments.

    Conventional comments is the practice to use a specific format in the review comments to express your intent and tone more clearly. It's strongly inspired by semantic versioning.

    Let's take the next comment:

    This is not worded correctly.

    Adding labels you can tell the difference on your intent:

    **suggestion:** This is not worded correctly.
    Or 
    **issue (non-blocking):** This is not worded correctly.

    Labels also prompt the reviewer to give more actionable comments.

    **suggestion:** This is not worded correctly.

Can we change this to match the wording of the marketing page?

    Labeling comments encourages collaboration and saves hours of undercommunication and misunderstandings. They are also parseable by machines!

    Format

    Adhering to a consistent format improves reader's expectations and machine readability. Here's the format we propose: 

    <label> [decorations]: <subject>

[discussion]
    - label - This is a single label that signifies what kind of comment is being left. - subject - This is the main message of the comment. - decorations (optional) - These are extra decorating labels for the comment. They are surrounded by parentheses and comma-separated. - discussion (optional) - This contains supporting statements, context, reasoning, and anything else to help communicate the "why" and "next steps" for resolving the comment. For example: 
    **question (non-blocking):** At this point, does it matter which thread has won?

Maybe to prevent a race condition we should keep looping until they've all won?

    Can be automatically parsed into:

    {
  "label": "question",
  "subject": "At this point, does it matter which thread has won?",
  "decorations": ["non-blocking"],
  "discussion": "Maybe to prevent a race condition we should keep looping until they've all won?"
}
    Labels

    We strongly suggest using the following labels: | | | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | praise: | Praises highlight something positive. Try to leave at least one of these comments per review. Do not leave false praise (which can actually be damaging). Do look for something to sincerely praise. | | quibble: | Quibbles are trivial preference-based requests. These should be non-blocking by nature. Similar to polish but clearly preference-based.| | suggestion: | Suggestions propose improvements to the current subject. It's important to be explicit and clear on what is being suggested and why it is an improvement. These are non-blocking proposals. If it's blocking use todo instead.| | todo: | TODO's are necessary changes. Distinguishing todo comments from issues or suggestions helps direct the reader's attention to comments requiring more involvement. | | issue: | Issues highlight specific problems with the subject under review. These problems can be user-facing or behind the scenes. It is strongly recommended to pair this comment with a suggestion. If you are not sure if a problem exists or not, consider leaving a question. | | question: | Questions are appropriate if you have a potential concern but are not quite sure if it's relevant or not. Asking the author for clarification or investigation can lead to a quick resolution. | | thought: | Thoughts represent an idea that popped up from reviewing. These comments are non-blocking by nature, but they are extremely valuable and can lead to more focused initiatives and mentoring opportunities. | | chore: | Chores are simple tasks that must be done before the subject can be "officially" accepted. Usually, these comments reference some common process. Try to leave a link to the process description so that the reader knows how to resolve the chore. | | note: | Notes are always non-blocking and simply highlight something the reader should take note of. |

    If you like to be a bit more expressive with your labels, you may also consider:
    typo: Typo comments are like todo:, where the main issue is a misspelling.
    polish: Polish comments are like a suggestion, where there is nothing necessarily wrong with the relevant content, there's just some ways to immediately improve the quality. Similar but not exactly the same as quibble.

    Decorations

    Decorations give additional context for a comment. They help further classify comments which have the same label (for example, a security suggestion as opposed to a test suggestion). 

    **suggestion (security):** I'm a bit concerned that we are implementing our own DOM purifying function here...
Could we consider using the framework instead?

    **suggestion (test,if-minor):** It looks like we're missing some unit test coverage that the cat disappears completely.

    Decorations may be specific to each organization. If needed, we recommend establishing a minimal set of decorations (leaving room for discretion) with no ambiguity. Possible decorations include: | | | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | (non-blocking) | A comment with this decoration should not prevent the subject under review from being accepted. This is helpful for organizations that consider comments blocking by default. | | (blocking) | A comment with this decoration should prevent the subject under review from being accepted, until it is resolved. This is helpful for organizations that consider comments non-blocking by default. | | (if-minor) | This decoration gives some freedom to the author that they should resolve the comment only if the changes ends up being minor or trivial. |

    Adding a decoration to a comment should improve understandability and maintain readability. Having a list of many decorations in one comment would conflict with this goal.

    More examples 

    **quibble:** `little star` => `little bat`

Can we update the other references as well?

    **chore:** Let's run the `jabber-walk` CI job to make sure this doesn't break any known references.
Here are [the docs](https://en.wikipedia.org/wiki/Jabberwocky) for running this job. Feel free to reach out if you need any help!

    **praise:** Beautiful test!

    Best Practices

    Some best practices for writing helpful review feedback:

    • Mentoring pays off exponentially
    • Leave actionable comments
    • Combine similar comments
    • Replace "you" with "we"
    • Replace "should" with "could"

    References - Home

OCR⚑

Camelot⚑

  • New: Introduce Camelot.

    Camelot is a Python library that can help you extract tables from PDFs

    import camelot

tables = camelot.read_pdf('foo.pdf')

tables
<TableList n=1>

tables.export('foo.csv', f='csv', compress=True) # json, excel, html, markdown, sqlite

tables[0]
<Table shape=(7, 7)>

tables[0].parsing_report
{
    'accuracy': 99.02,
    'whitespace': 12.24,
    'order': 1,
    'page': 1
}

tables[0].to_csv('foo.csv') # to_json, to_excel, to_html, to_markdown, to_sqlite

tables[0].df # get a pandas DataFrame!

    Installation

    To install Camelot from PyPI using pip, please include the extra cv requirement as shown:

    $ pip install "camelot-py[base]"

    It requires Ghostscript to be able to use the lattice mode. Which is better than using tabular-py that requires java to be installed.

    Usage

    Process background lines

    To detect line segments, Lattice needs the lines that make the table to be in the foreground. To process background lines, you can pass process_background=True.

    tables = camelot.read_pdf('background_lines.pdf', process_background=True)

    tables[1].df

    References - Docs

DevSecOps⚑

Infrastructure as Code⚑

Ansible Snippets⚑

  • New: Fix the ERROR! 'become' is not a valid attribute for a IncludeRole error.

    If you're trying to do something like: 

    tasks:
  - name: "Install nfs"
    become: true
    ansible.builtin.include_role:
      name: nfs

    You need to use this other syntax:

    tasks:
  - name: "Install nfs"
    ansible.builtin.include_role:
      name: nfs
      apply:
        become: true

  • New: Set host variables using a dynamic inventory.

    As with a normal inventory you can use the host_vars files with the proper name.

  • New: Avoid arbitrary disk mount.

    Instead of using /dev/sda use /dev/disk/by-id/whatever

  • New: Get the user running ansible in the host.

    If you gather_facts use the ansible_user_id variable.

  • New: Filter json data.

    To select a single element or a data subset from a complex data structure in JSON format (for example, Ansible facts), use the community.general.json_query filter. The community.general.json_query filter lets you query a complex JSON structure and iterate over it using a loop structure.

    This filter is built upon jmespath, and you can use the same syntax. For examples, see jmespath examples.

    A complex example would be:

    "{{ ec2_facts | json_query('instances[0].block_device_mappings[?device_name!=`/dev/sda1` && device_name!=`/dev/xvda`].{device_name: device_name, id: ebs.volume_id}') }}"

    This snippet:

    • Gets all dictionaries under the block_device_mappings list which device_name is not equal to /dev/sda1 or /dev/xvda
    • From those results it extracts and flattens only the desired values. In this case device_name and the id which is at the key ebs.volume_id of each of the items of the block_device_mappings list.

  • New: Do asserts.

    - name: After version 2.7 both 'msg' and 'fail_msg' can customize failing assertion message
  ansible.builtin.assert:
    that:
      - my_param <= 100
      - my_param >= 0
    fail_msg: "'my_param' must be between 0 and 100"
    success_msg: "'my_param' is between 0 and 100"

  • New: Split a variable in ansible.

    {{ item | split ('@') | last }}

  • New: Get a list of EC2 volumes mounted on an instance an their mount points.

    Assuming that each volume has a tag mount_point you could:

    - name: Gather EC2 instance metadata facts
  amazon.aws.ec2_metadata_facts:

- name: Gather info on the mounted disks
  delegate_to: localhost
  block:
    - name: Gather information about the instance
      amazon.aws.ec2_instance_info:
        instance_ids:
          - "{{ ansible_ec2_instance_id }}"
      register: ec2_facts

    - name: Gather volume tags
      amazon.aws.ec2_vol_info:
        filters:
          volume-id: "{{ item.id }}"
      # We exclude the root disk as they are already mounted and formatted
      loop: "{{ ec2_facts | json_query('instances[0].block_device_mappings[?device_name!=`/dev/sda1` && device_name!=`/dev/xvda`].{device_name: device_name, id: ebs.volume_id}') }}"
      register: volume_tags_data

    - name: Save the required volume data
      set_fact:
        volumes: "{{ volume_tags_data | json_query('results[0].volumes[].{id: id, mount_point: tags.mount_point}') }}"

    - name: Display volumes data
      debug:
        msg: "{{ volumes }}"

    - name: Make sure that all volumes have a mount point
      assert:
        that:
          - item.mount_point is defined
          - item.mount_point|length > 0
        fail_msg: "Configure the 'mount_point' tag on the volume {{ item.id }} on the instance {{ ansible_ec2_instance_id }}"
        success_msg: "The volume {{ item.id }} has the mount_point tag well set"
      loop: "{{ volumes }}"

  • New: Create a list of dictionaries using ansible.

    - name: Create and Add items to dictionary
  set_fact:
      userdata: "{{ userdata | default({}) | combine ({ item.key : item.value }) }}"
  with_items:
    - { 'key': 'Name' , 'value': 'SaravAK'}
    - { 'key': 'Email' , 'value': 'sarav@gritfy.com'}
    - { 'key': 'Location' , 'value': 'Coimbatore'}
    - { 'key': 'Nationality' , 'value': 'Indian'}

  • New: Merge two dictionaries on a key.

    If you have these two lists:

    "list1": [
  { "a": "b", "c": "d" },
  { "a": "e", "c": "f" }
]

"list2": [
  { "a": "e", "g": "h" },
  { "a": "b", "g": "i" }
]
    And want to merge them using the value of key "a":

    "list3": [
  { "a": "b", "c": "d", "g": "i" },
  { "a": "e", "c": "f", "g": "h" }
]

    If you can install the collection community.general use the filter lists_mergeby. The expression below gives the same result

    list3: "{{ list1|community.general.lists_mergeby(list2, 'a') }}"

  • New: Loop over dict fails when only one element detected.

    If you see the If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup." error in an Ansible log it means that the content of the variable is not the type you expect it to be. This can happen for example for lists that have only one or zero elements, which gets translated into a string thus breaking the loop structure.

    So instead of: 

    - name: Create filesystem on device
  community.general.filesystem:
    fstype: ext4
    dev: "/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol{{ item.id | split('-') | last }}"
  loop: "{{ volumes }}"

    You can use:

    - name: Create filesystem on device
  community.general.filesystem:
    fstype: ext4
    dev: "/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol{{ item.id | split('-') | last }}"
  loop: "{{ lookup('list', volumes, wantlist=True) }}"

    If that gives you issues you can use this other construction instead:

    - name: Save the required volume data
  set_fact:
    volumes: "{{ volume_tags_data | json_query('results[0].volumes[].{id: id, mount_point: tags.mount_point}') }}"

- name: Get result type for the volumes
  set_fact:
    volumes_type: "{{ volumes | type_debug }}"

- name: Display volumes type
  debug:
    msg: "{{ volumes_type }}"

- name: Force list of volumes if it's a string
  set_fact:
    volumes: "{{ [] }}"
  when:
    - volumes_type == 'str'

- name: Force list of volumes if it's a dictionary
  set_fact:
    volumes: "{{ [volumes] }}"
  when:
    - volumes_type == 'dict'

- name: Create filesystem on device
  community.general.filesystem:
    fstype: ext4
    dev: "/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol{{ item.id | split('-') | last }}"
  loop: "{{ volumes }}"

  • New: Set the ssh connection port using dynamic inventories.

    To specify a custom SSH port, you can use a host_vars or group_vars file. For example, create a group_vars directory and a file named all.yaml inside it:

    ansible_port: 2222

Infrastructure Solutions⚑

Kubernetes⚑

  • New: Introduce IceKube.

    IceKube tool for finding complex attack paths in Kubernetes clusters. It's like Bloodhound for Kubernetes. It uses Neo4j to store & analyze Kubernetes resource relationships → identify attack paths & security misconfigs

  • New: Add reloader to tools to test.

    stakater/reloader: A Kubernetes controller to watch changes in ConfigMap and Secrets and do rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet and DeploymentConfig. Useful for not that clever applications that need a reboot when a configmap changes.

AWS⚑

Kubectl Commands⚑

  • New: Understanding how reserved instances are applied.

    A Reserved Instance that is purchased for a Region is called a regional Reserved Instance, and provides Availability Zone and instance size flexibility.

    • The Reserved Instance discount applies to instance usage in any Availability Zone in that Region.
    • The Reserved Instance discount applies to instance usage within the instance family, regardless of size—this is known as instance size flexibility.

    With instance size flexibility, the Reserved Instance discount applies to instance usage for instances that have the same family, generation, and attribute. The Reserved Instance is applied from the smallest to the largest instance size within the instance family based on the normalization factor.

    The discount applies either fully or partially to running instances of the same instance family, depending on the instance size of the reservation, in any Availability Zone in the Region. The only attributes that must be matched are the instance family, tenancy, and platform.

    The following table lists the different sizes within an instance family, and the corresponding normalization factor. This scale is used to apply the discounted rate of Reserved Instances to the normalized usage of the instance family.

    Instance size Normalization factor
    nano 0.25
    micro 0.5
    small 1
    medium 2
    large 4
    xlarge 8
    2xlarge 16
    3xlarge 24
    4xlarge 32
    6xlarge 48
    8xlarge 64
    9xlarge 72
    10xlarge 80
    12xlarge 96
    16xlarge 128
    18xlarge 144
    24xlarge 192
    32xlarge 256
    48xlarge 384
    56xlarge 448
    112xlarge 896

    For example, a t2.medium instance has a normalization factor of 2. If you purchase a t2.medium default tenancy Amazon Linux/Unix Reserved Instance in the US East (N. Virginia) and you have two running t2.small instances in your account in that Region, the billing benefit is applied in full to both instances.

    Or, if you have one t2.large instance running in your account in the US East (N. Virginia) Region, the billing benefit is applied to 50% of the usage of the instance.

    Limitations:

    • Supported: Instance size flexibility is only supported for Regional Reserved Instances.
    • Not supported: Instance size flexibility is not supported for the following Reserved Instances:
      • Reserved Instances that are purchased for a specific Availability Zone (zonal Reserved Instances)
      • Reserved Instances for G4ad, G4dn, G5, G5g, and Inf1 instances
      • Reserved Instances for Windows Server, Windows Server with SQL Standard, Windows Server with SQL Server Enterprise, Windows Server with SQL Server Web, RHEL, and SUSE Linux Enterprise Server
      • Reserved Instances with dedicated tenancy

  • New: EC2 Instance savings plan versus reserved instances.

    I've been comparing the EC2 Reserved Instances and of the EC2 instance family savings plans and decided to go with the second because:

    • They both have almost the same rates. Reserved instances round the price at the 3rd decimal and the savings plan at the fourth, but this difference is neglegible.
    • Savings plan are easier to calculate, as you just need to multiply the number of instances you want times the current rate and add them all up.
    • Easier to understand: To reserve instances you need to take into account the instance flexibility and the normalization factors which makes it difficult both to make the plans and also to audit how well you're using it.
    • Easier to audit: In addition to the above point, you have nice dashboards to see the coverage and utilization over time of your ec2 instance savings plans, which are at the same place as the other savings plans.

  • New: Important notes when doing a savings plan.

    • Always use the reservation rates instead of the on-demand rates!
    • Analyze your coverage reports. You don't want to have many points of 100% coverage as it means that you're using less resources than you've reserved. On the other hand it's fine to sometimes use less resources than the reserved if that will mean a greater overall savings. It's a tight balance.
    • The Savings plan reservation is taken into account at hour level, not at month or year level. That means that if you reserve 1/hour of an instance type and you use for example 2/hour half the day and 0/hour half the day, you'll have a 100% coverage of your plan the first hour and another 1/hour of on-demand infrastructure cost for the first part of the day. On the second part of the day you'll have a 0% coverage. This means that you should only reserve the amount of resources you plan to be using 100% of the time throughout your savings plan. Again you may want to overcommit a little bit, reducing the utilization percentage of a plan but getting better savings in the end.

  • New: Upload a file to a pod.

    kubectl cp {{ path_to_local_file }} {{ container_id }}:{{ path_to_file }}

  • New: Delete pods that are stuck in terminating state for a while.

    kubectl delete pod <pod-name> --grace-period=0 --force

Volumes⚑

  • New: Troubleshoot don't have enough permissions to start restore from backup.

    That may be because the default EFS backup policy doesn't let you do that (stupid, I know).

    To fix it go into the backup policy and remove the next line from the Deny policy:

    backup:StartRestoreJob

  • New: Specify a path of a configmap.

    If you have a configmap with a key ssh-known-hosts and you want to mount it's content in a file, in the deployment volumeMounts section you can use the subPath field:

          - mountPath: /home/argocd/.ssh/known_hosts
        name: ssh-known-hosts
        subPath: ssh_known_hosts
        readOnly: true

  • New: List the size of the recovery points.

    BACKUP_VAULT_NAME="your-vault-name"

RECOVERY_POINTS=$(aws backup list-recovery-points-by-backup-vault --backup-vault-name $BACKUP_VAULT_NAME --query 'RecoveryPoints[*].[RecoveryPointArn,BackupSizeInBytes,CreationDate]' --output text)
echo -e "Creation Date\t\tRecovery Point ARN\t\t\t\t\t\t\t\t\tSize (TB)"
echo "---------------------------------------------------------------------------------------------------------------------"

while read -r RECOVERY_POINT_ARN BACKUP_SIZE_BYTES CREATION_DATE; do
    # Remove the decimal part from the epoch time
    EPOCH_TIME=$(echo $CREATION_DATE | cut -d'.' -f1)
    # Convert the creation date from epoch time to YYYY-MM-DD format
    FORMATTED_DATE=$(date -d @$EPOCH_TIME +"%Y-%m-%d")
    SIZE_TB=$(echo "scale=6; $BACKUP_SIZE_BYTES / (1024^4)" | bc)
    # echo -e "$FORMATTED_DATE\t$RECOVERY_POINT_ARN\t$SIZE_TB"
    printf "%-16s %-80s %10.6f\n" "$FORMATTED_DATE" "$RECOVERY_POINT_ARN" "$SIZE_TB"
done <<< "$RECOVERY_POINTS"

  • New: List the size of the jobs.

    To list AWS Backup jobs and display their completion dates and sizes in a human-readable format, you can use the following AWS CLI command combined with jq for parsing and formatting the output. This command handles cases where the backup size might be null and rounds the size to the nearest whole number in gigabytes.

    aws backup list-backup-jobs --output json | jq -r '
  .BackupJobs[] |
  [
    (.CompletionDate | strftime("%Y-%m-%d %H:%M:%S")),
    (if .BackupSizeInBytes == null then "0GB" else ((.BackupSizeInBytes / 1024 / 1024 / 1024) | floor | tostring + " GB") end)
  ] |
  @tsv' | column -t -s$'\t'
    Explanation:

    • aws backup list-backup-jobs --output json: Lists all AWS Backup jobs in JSON format.
    • .BackupJobs[]: Iterates over each backup job.
    • (.CompletionDate | strftime("%Y-%m-%d %H:%M:%S")): Converts the Unix timestamp in CompletionDate to a human-readable date format (YYYY-MM-DD HH:MM:SS).
    • (if .BackupSizeInBytes == null then "0GB" else ((.BackupSizeInBytes / 1024 / 1024 / 1024) | floor | tostring + " GB") end): Checks if BackupSizeInBytes is null. If it is, outputs "0GB". Otherwise, converts the size from bytes to gigabytes, rounds it down to the nearest whole number, and appends " GB".
    • | @tsv: Formats the output as tab-separated values.
    • column -t -s$'\t': Formats the TSV output into a table with columns aligned.

Continuous Deployment⚑

ArgoCD⚑

Continuous Integration⚑

Bandit⚑

  • New: Solving warning B603: subprocess_without_shell_equals_true.

    The B603: subprocess_without_shell_equals_true issue in Bandit is raised when the subprocess module is used without setting shell=True. Bandit flags this because using shell=True can be a security risk if the command includes user-supplied input, as it opens the door to shell injection attacks.

    To fix it:

    1. Avoid shell=True if possible: Instead, pass the command and its arguments as a list to subprocess.Popen (or subprocess.run, subprocess.call, etc.). This way, the command is executed directly without invoking the shell, reducing the risk of injection attacks.

    Here's an example:

    import subprocess

# Instead of this:
# subprocess.Popen("ls -l", shell=True)

# Do this:
subprocess.Popen(["ls", "-l"])
    1. When you must use shell=True: - If you absolutely need to use shell=True (e.g., because you are running a complex shell command or using shell features like wildcards), ensure that the command is either hardcoded or sanitized to avoid security risks.

    Example with shell=True:

    import subprocess

# Command is hardcoded and safe
command = "ls -l | grep py"
subprocess.Popen(command, shell=True)

    If the command includes user input, sanitize the input carefully:

    import subprocess

user_input = "some_directory"
command = f"ls -l {subprocess.list2cmdline([user_input])}"
subprocess.Popen(command, shell=True)

    Note: Even with precautions, using shell=True is risky with user input, so avoid it if possible.

    1. Explicitly tell bandit you have considered the risk: If you have reviewed the code and are confident that the code is safe in your particular case, you can mark the line with a # nosec comment to tell Bandit to ignore the issue:
    import subprocess

command = "ls -l | grep py"
subprocess.Popen(command, shell=True)  # nosec

Safety⚑

  • New: Add deprecation warning.

    Since 2024-05-27 it requires an account to work, use pip-audit instead.

Security Checkers⚑

  • New: Introduce pip-audit.

    pip-audit is the official pypa tool for scanning Python environments for packages with known vulnerabilities. It uses the Python Packaging Advisory Database (https://github.com/pypa/advisory-database) via the PyPI JSON API as a source of vulnerability reports.

    Installation

    pip install pip-audit

    Usage

    pip-audit
    On completion, pip-audit will exit with a code indicating its status.

    The current codes are:

    • 0: No known vulnerabilities were detected.
    • 1: One or more known vulnerabilities were found.

    pip-audit's exit code cannot be suppressed. See Suppressing exit codes from pip-audit for supported alternatives.

    References

Yamllint⚑

  • New: Ignore certain files.

    It is possible to exclude specific files or directories, so that the linter doesn’t process them. They can be provided either as a list of paths, or as a bulk string.

    You can either totally ignore files (they won’t be looked at):

    extends: default

ignore: |
  /this/specific/file.yaml
  all/this/directory/
  *.template.yaml

ignore:
  - /this/specific/file.yaml
  - all/this/directory/
  - '*.template.yaml'

Automating Processes⚑

letsencrypt⚑

  • New: Manually renew a certificate.

    Linuxserver swag container renews the certificates at night. If you don't have your server up at those hours your certificate won't be renewed automatically and you need to react to the prometheus alert manually. To do so get into the container and run certbot renew.

Storage⚑

NAS⚑

OpenZFS⚑

  • New: Solve the pool or dataset is busy error.

    If you get an error of pool or dataset is busy run the next command to see which process is still running on the pool:

    lsof 2>/dev/null | grep dataset-name

  • New: Monitor dbgmsg with loki.

    If you use loki remember to monitor the /proc/spl/kstat/zfs/dbgmsg file:

    - job_name: zfs
    static_configs:
      - targets:
          - localhost
        labels:
          job: zfs
          __path__: /proc/spl/kstat/zfs/dbgmsg

  • Correction: Add loki alerts on the kernel panic error.

    You can monitor this issue with loki using the next alerts:

    groups:
  - name: zfs
    rules:
      - alert: SlowSpaSyncZFSError
       expr: |
          count_over_time({job="zfs"} |~ `spa_deadman.*slow spa_sync` [5m])
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Slow sync traces found in the ZFS debug logs at {{ $labels.hostname}}"
          message: "This usually happens before the ZFS becomes unresponsible"

  • New: Monitorization.

    You can monitor this issue with loki using the next alerts:

    groups:
  - name: zfs
    rules:
      - alert: ErrorInSanoidLogs
        expr: |
          count_over_time({job="systemd-journal", syslog_identifier="sanoid"} |= `ERROR` [5m])
        for: 1m
        labels:
            severity: critical
        annotations:
            summary: "Errors found on sanoid log at {{ $labels.hostname}}"

  • New: Set zfs module parameters or options.

    Most of the ZFS kernel module parameters are accessible in the SysFS /sys/module/zfs/parameters directory. Current values can be observed by

    cat /sys/module/zfs/parameters/PARAMETER

    Many of these can be changed by writing new values. These are denoted by Change|Dynamic in the PARAMETER details below.

    echo NEWVALUE >> /sys/module/zfs/parameters/PARAMETER

    If the parameter is not dynamically adjustable, an error can occur and the value will not be set. It can be helpful to check the permissions for the PARAMETER file in SysFS.

    In some cases, the parameter must be set prior to loading the kernel modules or it is desired to have the parameters set automatically at boot time. For many distros, this can be accomplished by creating a file named /etc/modprobe.d/zfs.conf containing a text line for each module parameter using the format:

    options zfs PARAMETER=VALUE

    Some parameters related to ZFS operations are located in module parameters other than in the zfs kernel module. These are documented in the individual parameter description. Unless otherwise noted, the tunable applies to the zfs kernel module. For example, the icp kernel module parameters are visible in the /sys/module/icp/parameters directory and can be set by default at boot time by changing the /etc/modprobe.d/icp.conf file.

  • New: Configure the deadman failsafe measure.

    ZFS has a safety measure called the zfs_deadman_failmode. When a pool sync operation takes longer than zfs_deadman_synctime_ms, or when an individual I/O operation takes longer than zfs_deadman_ziotime_ms, then the operation is considered to be "hung". If zfs_deadman_enabled is set, then the deadman behavior is invoked as described by zfs_deadman_failmode. By default, the deadman is enabled and set to wait which results in "hung" I/O operations only being logged. The deadman is automatically disabled when a pool gets suspended.

    zfs_deadman_failmode configuration can have the next values:

    • wait: Wait for a "hung" operation to complete. For each "hung" operation a "deadman" event will be posted describing that operation.
    • continue: Attempt to recover from a "hung" operation by re-dispatching it to the I/O pipeline if possible.
    • panic: Panic the system. This can be used to facilitate automatic fail-over to a properly configured fail-over partner.

    Follow the guides under Set zfs module parameters or options to change this value.

  • New: Monitor the ZFS events.

    You can see the ZFS events using zpool events -v. If you want to be alerted on these events you can use this service to ingest them into Loki and raise alerts.

  • New: Manually create a backup.

    To create a snapshot of tank/home/ahrens that is named friday run:

    zfs snapshot tank/home/ahrens@friday

  • New: Tweak loki alerts.

          - alert: SyncoidCorruptedSnapshotSendError
        expr: |
          count_over_time({syslog_identifier="syncoid_send_backups"} |= `cannot receive incremental stream: invalid backup stream` [15m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Error trying to send a corrupted snapshot at {{ $labels.hostname}}"
          message: "Look at the context on loki to identify the snapshot in question. Delete it and then run the sync again"
      - alert: SanoidNotRunningError
        expr: |
          sum by (hostname) (count_over_time({job="systemd-journal", syslog_identifier="sanoid"}[1h])) or sum by (hostname) (count_over_time({job="systemd-journal"}[1h]) * 0)
        for: 0m
        labels:
            severity: critical
        annotations:
            summary: "Sanoid has not shown signs to be alive for the last hour at least in arva and helm"
      - alert: SlowSpaSyncZFSError
        expr: |
          count_over_time({job="zfs"} |~ `spa_deadman.*slow spa_sync` [10m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Slow sync traces found in the ZFS debug logs at {{ $labels.hostname}}"
          message: "This usually happens before the ZFS becomes unresponsible"

    The SanoidNotRunningError alert uses a broader search that ensures that all hosts are included and multiplies it to 0 to raise the alert if none is shown for the sanoid service.

  • New: Sync an already created cold backup.

    Mount the existent pool

    Imagine your pool is at /dev/sdf2:

    • Connect your device
    • Check for available ZFS pools: First, check if the system detects any ZFS pools that can be imported:
    sudo zpool import

    This command will list all pools that are available for import, including the one stored in /dev/sdf2. Look for the pool name you want to import.

    • Import the pool: If you see the pool listed and you know its name (let's say the pool name is mypool), you can import it with:
    sudo zpool import mypool
    • Import the pool from a specific device: If the pool isn't showing up or you want to specify the device directly, you can use:
    sudo zpool import -d /dev/sdf2

    This tells ZFS to look specifically at /dev/sdf2 for any pools. If you don't know the name of the pool this is also the command to run.

    This should list any pools found on the device. If it shows a pool, import it using:

    sudo zpool import -d /dev/sdf2 <pool_name>
    • Mount the pool: Once the pool is imported, ZFS should automatically mount any datasets associated with the pool. You can check the status of the pool with:
    sudo zpool status

    Additional options:

    • If the pool was exported cleanly, you can use zpool import without additional flags.
    • If the pool wasn’t properly exported or was interrupted, you might need to use -f (force) to import it:

    sudo zpool import -f mypool
    feat(linux_snippets#Create a systemd service for a non-root user): Create a systemd service for a non-root user

    To set up a systemd service as a non-root user, you can create a user-specific service file under your home directory. User services are defined in ~/.config/systemd/user/ and can be managed without root privileges.

    1. Create the service file:

    Open a terminal and create a new service file in ~/.config/systemd/user/. For example, if you want to create a service for a script named my_script.py, follow these steps:

    mkdir -p ~/.config/systemd/user
nano ~/.config/systemd/user/my_script.service
    1. Edit the service file:

    In the my_script.service file, add the following configuration:

    [Unit]
Description=My Python Script Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /path/to/your/script/my_script.py
WorkingDirectory=/path/to/your/script/
SyslogIdentifier=my_script
Restart=on-failure
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target
    • Description: A short description of what the service does.

    • ExecStart: The command to run your script. Replace /path/to/your/script/my_script.py with the full path to your Python script. If you want to run the script within a virtualenv you can use /path/to/virtualenv/bin/python instead of /usr/bin/python3.

      You'll need to add the virtualenv path to Path 

      # Add virtualenv's bin directory to PATH
Environment="PATH=/path/to/virtualenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      - WorkingDirectory: Set the working directory to where your script is located (optional). - Restart: Restart the service if it fails. - StandardOutput and StandardError: This ensures that the output is captured in the systemd journal. - WantedBy: Specifies the target to which this service belongs. default.target is commonly used for user services.

    • Reload systemd to recognize the new service:

    Run the following command to reload systemd's user service files:

    systemctl --user daemon-reload
    1. Enable and start the service:

    To start the service immediately and enable it to run on boot (for your user session), use the following commands:

    systemctl --user start my_script.service
systemctl --user enable my_script.service

    1. Check the status and logs:

    2. To check if the service is running:

      systemctl --user status my_script.service

    3. To view logs specific to your service:

      journalctl --user -u my_script.service -f

    If you need to use the graphical interface

    If your script requires user interaction (like entering a GPG passphrase), it’s crucial to ensure that the service is tied to your graphical user session, which ensures that prompts can be displayed and interacted with.

    To handle this situation, you should make a few adjustments to your systemd service:

    Ensure service is bound to graphical session

    Change the WantedBy target to graphical-session.target instead of default.target. This makes sure the service waits for the full graphical environment to be available.

    Use Type=forking instead of Type=simple (optional)

    If you need the service to wait until the user is logged in and has a desktop session ready, you might need to tweak the service type. Usually, Type=simple is fine, but you can also experiment with Type=forking if you notice any issues with user prompts.

    Here’s how you should modify your mbsync_syncer.service file:

    [Unit]
Description=My Python Script Service
After=graphical-session.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /path/to/your/script/my_script.py
WorkingDirectory=/path/to/your/script/
Restart=on-failure
StandardOutput=journal
StandardError=journal
SyslogIdentifier=my_script
Environment="DISPLAY=:0"
Environment="DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"

[Install]
WantedBy=graphical-session.target

    After modifying the service, reload and restart it:

    systemctl --user daemon-reload
systemctl --user restart my_script.service

ZFS Prometheus exporter⚑

  • Correction: Tweak the zfs_exporter target not available error.

    Remember to set the scrape_timeout to at least of 60s as the exporter is sometimes slow to answer, specially on low hardware resources.

     - job_name: zfs_exporter
   metrics_path: /metrics
   scrape_timeout: 60s
   static_configs:
   - targets: [192.168.3.236:9134]
   metric_relabel_configs:
   ...

  • Correction: Tweak the ZfsPoolUnhealthy alert.

       - alert: ZfsPoolUnhealthy
     expr: last_over_time(zfs_pool_health[1h]) > 0
     for: 5m
     labels:
       severity: critical

Resilience⚑

  • New: Introduce linux resilience.

    Increasing the resilience of the servers is critical when hosting services for others. This is the roadmap I'm following for my servers.

    Autostart services if the system reboots Using init system services to manage your services

    **Get basic metrics traceability and alerts ** Set up Prometheus with:

    • The blackbox exporter to track if the services are available to your users and to monitor SSL certificates health.
    • The node exporter to keep track on the resource usage of your machines and set alerts to get notified when concerning events happen (disks are getting filled, CPU usage is too high)

    **Get basic logs traceability and alerts **

    Set up Loki and clear up your system log errors.

    Improve the resilience of your data If you're still using ext4 for your filesystems instead of zfs you're missing a big improvement. To set it up:

    Automatically react on system failures - Kernel panics - watchdog

    Future undeveloped improvements - Handle the system reboots after kernel upgrades

Memtest⚑

  • New: Introduce memtest.

    memtest86 is a testing software for RAM.

    Installation 

    apt-get install memtest86+

    After the installation you'll get Memtest entries in grub which you can spawn.

    For some unknown reason the memtest of the boot menu didn't work for me. So I downloaded the latest free version of memtest (It's at the bottom of the screen), burnt it in a usb and booted from there.

    Usage It will run by itself. For 64GB of ECC RAM it took aproximately 100 minutes to run all the tests.

    Check ECC errors MemTest86 directly polls ECC errors logged in the chipset/memory controller registers and displays it to the user on-screen. In addition, ECC errors are written to the log and report file.

watchdog⚑

  • New: Introduce the watchdog.

    A watchdog timer (WDT, or simply a watchdog), sometimes called a computer operating properly timer (COP timer), is an electronic or software timer that is used to detect and recover from computer malfunctions. Watchdog timers are widely used in computers to facilitate automatic correction of temporary hardware faults, and to prevent errant or malevolent software from disrupting system operation.

    During normal operation, the computer regularly restarts the watchdog timer to prevent it from elapsing, or "timing out". If, due to a hardware fault or program error, the computer fails to restart the watchdog, the timer will elapse and generate a timeout signal. The timeout signal is used to initiate corrective actions. The corrective actions typically include placing the computer and associated hardware in a safe state and invoking a computer reboot.

    Microcontrollers often include an integrated, on-chip watchdog. In other computers the watchdog may reside in a nearby chip that connects directly to the CPU, or it may be located on an external expansion card in the computer's chassis.

    Hardware watchdog

    Before you start using the hardware watchdog you need to check if your hardware actually supports it.

    If you see Watchdog hardware is disabled error on boot things are not looking good.

    Check if the hardware watchdog is enabled You can see if hardware watchdog is loaded by running wdctl. For example for a machine that has it enabled you'll see:

    Device:        /dev/watchdog0
Identity:      iTCO_wdt [version 0]
Timeout:       30 seconds
Pre-timeout:    0 seconds
Timeleft:      30 seconds
FLAG           DESCRIPTION               STATUS BOOT-STATUS
KEEPALIVEPING  Keep alive ping reply          1           0
MAGICCLOSE     Supports magic close char      0           0
SETTIMEOUT     Set timeout (in seconds)       0           0

    On a machine that doesn't you'll see:

    wdctl: No default device is available.: No such file or directory

    Another option is to run dmesg | grep wd or dmesg | grep watc -i. For example for a machine that has enabled the hardware watchdog you'll see something like:

    [   20.708839] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   20.708894] iTCO_wdt: Found a Intel PCH TCO device (Version=4, TCOBASE=0x0400)
[   20.709009] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)

    For one that is not you'll see:

    [    1.934999] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver
[    1.935057] sp5100-tco sp5100-tco: Using 0xfed80b00 for watchdog MMIO address
[    1.935062] sp5100-tco sp5100-tco: Watchdog hardware is disabled

    If you're out of luck and your hardware doesn't support it you can delegate the task to the software watchdog or get some usb watchdog

    Systemd watchdog

    Starting with version 183 systemd provides full support for hardware watchdogs (as exposed in /dev/watchdog to userspace), as well as supervisor (software) watchdog support for invidual system services. The basic idea is the following: if enabled, systemd will regularly ping the watchdog hardware. If systemd or the kernel hang this ping will not happen anymore and the hardware will automatically reset the system. This way systemd and the kernel are protected from boundless hangs -- by the hardware. To make the chain complete, systemd then exposes a software watchdog interface for individual services so that they can also be restarted (or some other action taken) if they begin to hang. This software watchdog logic can be configured individually for each service in the ping frequency and the action to take. Putting both parts together (i.e. hardware watchdogs supervising systemd and the kernel, as well as systemd supervising all other services) we have a reliable way to watchdog every single component of the system.

    Configuring the watchdog To make use of the hardware watchdog it is sufficient to set the RuntimeWatchdogSec= option in /etc/systemd/system.conf. It defaults to 0 (i.e. no hardware watchdog use). Set it to a value like 20s and the watchdog is enabled. After 20s of no keep-alive pings the hardware will reset itself. Note that systemd will send a ping to the hardware at half the specified interval, i.e. every 10s.

    Note that the hardware watchdog device (/dev/watchdog) is single-user only. That means that you can either enable this functionality in systemd, or use a separate external watchdog daemon, such as the aptly named watchdog. Although the built-in hardware watchdog support of systemd does not conflict with other watchdog software by default. systemd does not make use of /dev/watchdog by default, and you are welcome to use external watchdog daemons in conjunction with systemd, if this better suits your needs.

    ShutdownWatchdogSec=`` is another option that can be configured in/etc/systemd/system.conf`. It controls the watchdog interval to use during reboots. It defaults to 10min, and adds extra reliability to the system reboot logic: if a clean reboot is not possible and shutdown hangs, we rely on the watchdog hardware to reset the system abruptly, as extra safety net.

    Now, let's have a look how to add watchdog logic to individual services.

    First of all, to make software watchdog-supervisable it needs to be patched to send out "I am alive" signals in regular intervals in its event loop. Patching this is relatively easy. First, a daemon needs to read the WATCHDOG_USEC= environment variable. If it is set, it will contain the watchdog interval in usec formatted as ASCII text string, as it is configured for the service. The daemon should then issue sd_notify("WATCHDOG=1") calls every half of that interval. A daemon patched this way should transparently support watchdog functionality by checking whether the environment variable is set and honouring the value it is set to.

    To enable the software watchdog logic for a service (which has been patched to support the logic pointed out above) it is sufficient to set the WatchdogSec= to the desired failure latency. See systemd.service(5) for details on this setting. This causes WATCHDOG_USEC= to be set for the service's processes and will cause the service to enter a failure state as soon as no keep-alive ping is received within the configured interval.

    The next step is to configure whether the service shall be restarted and how often, and what to do if it then still fails. To enable automatic service restarts on failure set Restart=on-failure for the service. To configure how many times a service shall be attempted to be restarted use the combination of StartLimitBurst= and StartLimitInterval= which allow you to configure how often a service may restart within a time interval. If that limit is reached, a special action can be taken. This action is configured with StartLimitAction=. The default is a none, i.e. that no further action is taken and the service simply remains in the failure state without any further attempted restarts. The other three possible values are reboot, reboot-force and reboot-immediate.

    • reboot attempts a clean reboot, going through the usual, clean shutdown logic.
    • reboot-force is more abrupt: it will not actually try to cleanly shutdown any services, but immediately kills all remaining services and unmounts all file systems and then forcibly reboots (this way all file systems will be clean but reboot will still be very fast).
    • reboot-immediate does not attempt to kill any process or unmount any file systems. Instead it just hard reboots the machine without delay. reboot-immediate hence comes closest to a reboot triggered by a hardware watchdog. All these settings are documented in systemd.service(5).

    Putting this all together we now have pretty flexible options to watchdog-supervise a specific service and configure automatic restarts of the service if it hangs, plus take ultimate action if that doesn't help.

    Here's an example unit file:

    [Unit]
Description=My Little Daemon
Documentation=man:mylittled(8)

[Service]
ExecStart=/usr/bin/mylittled
WatchdogSec=30s
Restart=on-failure
StartLimitInterval=5min
StartLimitBurst=4
StartLimitAction=reboot-force
````

This service will automatically be restarted if it hasn't pinged the system manager for longer than 30s or if it fails otherwise. If it is restarted this way more often than 4 times in 5min action is taken and the system quickly rebooted, with all file systems being clean when it comes up again.

To write the code of the watchdog service you can follow one of these guides:

- [Python based watchdog](https://sleeplessbeastie.eu/2022/08/15/how-to-create-watchdog-for-systemd-service/)
- [Bash based watchdog](https://www.medo64.com/2019/01/systemd-watchdog-for-any-service/)

**[Testing a watchdog](https://serverfault.com/questions/375220/how-to-check-what-if-hardware-watchdogs-are-available-in-linux)**
One simple way to test a watchdog is to trigger a kernel panic. This can be done as root with:

```bash
echo c > /proc/sysrq-trigger

    The kernel will stop responding to the watchdog pings, so the watchdog will trigger.

    SysRq is a 'magical' key combo you can hit which the kernel will respond to regardless of whatever else it is doing, unless it is completely locked up. It can also be used by echoing letters to /proc/sysrq-trigger, like we're doing here.

    In this case, the letter c means perform a system crash and take a crashdump if configured.

    Troubleshooting

    Watchdog hardware is disabled error on boot

    According to the discussion at the kernel mailing list it means that the system contains hardware watchdog but it has been disabled (probably by BIOS) and Linux cannot enable the hardware.

    If your BIOS doesn't have a switch to enable it, consider the watchdog hardware broken for your system.

    Some people are blacklisting the module so that it's not loaded and therefore it doesn't return the error (1, 2

    References

Magic keys⚑

  • New: Introduce the Magic Keys.

    The magic SysRq key is a key combination understood by the Linux kernel, which allows the user to perform various low-level commands regardless of the system's state. It is often used to recover from freezes, or to reboot a computer without corrupting the filesystem.[1] Its effect is similar to the computer's hardware reset button (or power switch) but with many more options and much more control.

    This key combination provides access to powerful features for software development and disaster recovery. In this sense, it can be considered a form of escape sequence. Principal among the offered commands are means to forcibly unmount file systems, kill processes, recover keyboard state, and write unwritten data to disk. With respect to these tasks, this feature serves as a tool of last resort.

    The magic SysRq key cannot work under certain conditions, such as a kernel panic[2] or a hardware failure preventing the kernel from running properly.

    The key combination consists of Alt+Sys Req and another key, which controls the command issued.

    On some devices, notably laptops, the Fn key may need to be pressed to use the magic SysRq key.

    Reboot the machine

    A common use of the magic SysRq key is to perform a safe reboot of a Linux computer which has otherwise locked up (abbr. REISUB). This can prevent a fsck being required on reboot and gives some programs a chance to save emergency backups of unsaved work. The QWERTY (or AZERTY) mnemonics: "Raising Elephants Is So Utterly Boring", "Reboot Even If System Utterly Broken" or simply the word "BUSIER" read backwards, are often used to remember the following SysRq-keys sequence:

    • unRaw (take control of keyboard back from X),
    • tErminate (send SIGTERM to all processes, allowing them to terminate gracefully),
    • kIll (send SIGKILL to all processes, forcing them to terminate immediately),
    • Sync (flush data to disk),
    • Unmount (remount all filesystems read-only),
    • reBoot.

    When magic SysRq keys are used to kill a frozen graphical program, the program has no chance to restore text mode. This can make everything unreadable. The commands textmode (part of SVGAlib) and the reset command can restore text mode and make the console readable again.

    On distributions that do not include a textmode executable, the key command Ctrl+Alt+F1 may sometimes be able to force a return to a text console. (Use F1, F2, F3,..., F(n), where n is the highest number of text consoles set up by the distribution. Ctrl+Alt+F(n+1) would normally be used to reenter GUI mode on a system on which the X server has not crashed.)

    Interact with the sysrq through the commandline It can also be used by echoing letters to /proc/sysrq-trigger, for example to trigger a system crash and take a crashdump you can:

    echo c > /proc/sysrq-trigger

Monitoring⚑

Loki⚑

  • New: Prevent the too many outstanding requests error.

    Add to your loki config the next options 

    limits_config:
  split_queries_by_interval: 24h
  max_query_parallelism: 100

query_scheduler:
  max_outstanding_requests_per_tenant: 4096

frontend:
  max_outstanding_per_tenant: 4096

  • Correction: Use fake when using one loki instance in docker.

    If you only have one Loki instance you need to save the rule yaml files in the /etc/loki/rules/fake/ otherwise Loki will silently ignore them (it took me a lot of time to figure this out -.-).

  • New: Add alerts.

    Surprisingly I haven't found any compilation of Loki alerts. I'll gather here the ones I create.

    There are two kinds of rules: alerting rules and recording rules.

  • New: Alert when query returns no data.

    Sometimes the queries you want to alert happen when the return value is NaN or No Data. For example if you want to monitory the happy path by setting an alert if a string is not found in some logs in a period of time.

    count_over_time({filename="/var/log/mail.log"} |= `Mail is sent` [24h]) < 1

    This won't trigger the alert because the count_over_time doesn't return a 0 but a NaN. One way to solve it is to use the vector(0) operator with the operation or on() vector(0) 

    (count_over_time({filename="/var/log/mail.log"} |= `Mail is sent` [24h]) or on() vector(0)) < 1

  • New: Monitor loki metrics.

    Since Loki reuses the Prometheus code for recording rules and WALs, it also gains all of Prometheus’ observability.

    To scrape loki metrics with prometheus add the next snippet to the prometheus configuration:

      - job_name: loki
    metrics_path: /metrics
    static_configs:
    - targets:
      - loki:3100

    This assumes that loki is a docker in the same network as prometheus.

    There are some rules in the awesome prometheus alerts repo

    ---
groups:
- name: Awesome Prometheus loki alert rules
  # https://samber.github.io/awesome-prometheus-alerts/rules#loki
  rules:
  - alert: LokiProcessTooManyRestarts
    expr: changes(process_start_time_seconds{job=~".*loki.*"}[15m]) > 2
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: Loki process too many restarts (instance {{ $labels.instance }})
      description: "A loki process had too many restarts (target {{ $labels.instance }})\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - alert: LokiRequestErrors
    expr: 100 * sum(rate(loki_request_duration_seconds_count{status_code=~"5.."}[1m])) by (namespace, job, route) / sum(rate(loki_request_duration_seconds_count[1m])) by (namespace, job, route) > 10
    for: 15m
    labels:
      severity: critical
    annotations:
      summary: Loki request errors (instance {{ $labels.instance }})
      description: "The {{ $labels.job }} and {{ $labels.route }} are experiencing errors\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - alert: LokiRequestPanic
    expr: sum(increase(loki_panic_total[10m])) by (namespace, job) > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: Loki request panic (instance {{ $labels.instance }})
      description: "The {{ $labels.job }} is experiencing {{ printf \"%.2f\" $value }}% increase of panics\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - alert: LokiRequestLatency
    expr: (histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket{route!~"(?i).*tail.*"}[5m])) by (le)))  > 1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: Loki request latency (instance {{ $labels.instance }})
      description: "The {{ $labels.job }} {{ $labels.route }} is experiencing {{ printf \"%.2f\" $value }}s 99th percentile latency\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

    And there are some guidelines on the rest of the metrics in the grafana documentation

    Monitor the ruler

    Prometheus exposes a number of metrics for its WAL implementation, and these have all been prefixed with loki_ruler_wal_.

    For example: prometheus_remote_storage_bytes_total → loki_ruler_wal_prometheus_remote_storage_bytes_total

    Additional metrics are exposed, also with the prefix loki_ruler_wal_. All per-tenant metrics contain a tenant label, so be aware that cardinality could begin to be a concern if the number of tenants grows sufficiently large.

    Some key metrics to note are:

    • loki_ruler_wal_appender_ready: whether a WAL appender is ready to accept samples (1) or not (0)
    • loki_ruler_wal_prometheus_remote_storage_samples_total: number of samples sent per tenant to remote storage
    • loki_ruler_wal_prometheus_remote_storage_samples_pending_total: samples buffered in memory, waiting to be sent to remote storage
    • loki_ruler_wal_prometheus_remote_storage_samples_failed_total: samples that failed when sent to remote storage
    • loki_ruler_wal_prometheus_remote_storage_samples_dropped_total: samples dropped by relabel configurations
    • loki_ruler_wal_prometheus_remote_storage_samples_retried_total: samples re-resent to remote storage
    • loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds: highest timestamp of sample appended to WAL
    • loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds: highest timestamp of sample sent to remote storage.

  • New: Get a useful Source link in the alertmanager.

    This still doesn't work. Currently for the ruler external_url if you use the URL of your Grafana installation: e.g. external_url: "https://grafana.example.com" it creates a Source link in alertmanager similar to https://grafana.example.com/graph?g0.expr=%28sum+by%28thing%29%28count_over_time%28%7Bnamespace%3D%22foo%22%7D+%7C+json+%7C+bar%3D%22maxRetries%22%5B5m%5D%29%29+%3E+0%29&g0.tab=1, which isn't valid.

    This url templating (via /graph?g0.expr=%s&g0.tab=1) appears to be coming from prometheus. There is not a workaround yet

  • New: Maximum of series reached for a single query.

    Go to the loki-local-config.yaml, then find the limits_config configuration. Then modify this to the limits_config:

    limits_config:
   max_query_series: 100000

    But probably you're doing something wrong. feat(orgzly#Both local and remote notebook have been modified): Both local and remote notebook have been modified

    You can force load or force save a single note with a long tap.

  • Correction: Don't use vector(0) on aggregation over labels.

    If you're doing an aggregation over a label this approach won't work because it will add a new time series with value 0. In those cases use a broader search that includes other logs from the label you're trying to aggregate and multiply it by 0. For example:

    (
sum by (hostname) (
  count_over_time({job="systemd-journal", syslog_identifier="sanoid"}[1h])
)
or
sum by (hostname) (
  count_over_time({job="systemd-journal"}[1h]) * 0
)
) < 1

    The first part of the query returns all log lines of the service sanoid for each hostname. If one hostname were not to return any line that query alone won't show anything for that host. The second part of the query counts all the log lines of each hostname, so if it's up it will probably be sending at least one line per hour. As we're not interested in those number of lines we multiply it by 0, so that the target is shown.

  • New: Interact with loki through python.

    There is no client library for python (1, 2) they suggest to interact with the API with requests. Although I'd rather use logcli with the sh library.

  • New: Download the logs.

    The web UI only allows you to download the logs that are loaded in the view, if you want to download big amounts of logs you need to either use logcli or interact with the API.

    One user did a query on loop:

    set -x

JOB_ID=9079dc54-2f5c-4d74-a9aa-1d9eb39dd3c2

for I in `seq 0 655`; do
    FILE=logs_$I.txt
    ID="$JOB_ID:$I"
    QUERY="{aws_job_id=\"$ID\",job=\"varlogs\"}"
    docker run grafana/logcli:main-1b6d0bf-amd64 --addr=http://localhost:3100/ -o raw -q query $QUERY --limit 100000 --batch 100 --forward --from "2022-09-25T10:00:00Z" > $FILE
done

Logcli⚑

  • New: Introduce Promtail.

    Promtail is an agent which ships the contents of local logs to a Loki instance.

    It is usually deployed to every machine that runs applications which need to be monitored.

    It primarily:

    • Discovers targets
    • Attaches labels to log streams
    • Pushes them to the Loki instance.

  • New: Scrape journald logs.

    On systems with systemd, Promtail also supports reading from the journal. Unlike file scraping which is defined in the static_configs stanza, journal scraping is defined in a journal stanza:

    scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 12h
      path: /var/log/journal
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: unit
      - source_labels: ['__journal__hostname']
        target_label: hostname
      - source_labels: ['__journal_syslog_identifier']
        target_label: syslog_identifier
      - source_labels: ['__journal_transport']
        target_label: transport
      - source_labels: ['__journal_priority_keyword']
        target_label: keyword

    All fields defined in the journal section are optional, and are just provided here for reference.

    • max_age ensures that no older entry than the time specified will be sent to Loki; this circumvents entry too old errors.
    • path tells Promtail where to read journal entries from.
    • labels map defines a constant list of labels to add to every journal entry that Promtail reads.
    • matches field adds journal filters. If multiple filters are specified matching different fields, the log entries are filtered by both, if two filters apply to the same field, then they are automatically matched as alternatives.
    • When the json field is set to true, messages from the journal will be passed through the pipeline as JSON, keeping all of the original fields from the journal entry. This is useful when you don’t want to index some fields but you still want to know what values they contained.
    • When Promtail reads from the journal, it brings in all fields prefixed with __journal_ as internal labels. Like in the example above, the _SYSTEMD_UNIT field from the journal was transformed into a label called unit through relabel_configs. Keep in mind that labels prefixed with __ will be dropped, so relabeling is required to keep these labels. Look at the systemd man pages for a list of fields exposed by the journal.

    By default, Promtail reads from the journal by looking in the /var/log/journal and /run/log/journal paths. If running Promtail inside of a Docker container, the path appropriate to your distribution should be bind mounted inside of Promtail along with binding /etc/machine-id. Bind mounting /etc/machine-id to the path of the same name is required for the journal reader to know which specific journal to read from.

    docker run \
  -v /var/log/journal/:/var/log/journal/ \
  -v /run/log/journal/:/run/log/journal/ \
  -v /etc/machine-id:/etc/machine-id \
  grafana/promtail:latest \
  -config.file=/path/to/config/file.yaml

  • New: Scrape docker logs.

    Docker service discovery allows retrieving targets from a Docker daemon. It will only watch containers of the Docker daemon referenced with the host parameter. Docker service discovery should run on each node in a distributed setup. The containers must run with either the json-file or journald logging driver.

    Note that the discovery will not pick up finished containers. That means Promtail will not scrape the remaining logs from finished containers after a restart.

    scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_id']
        target_label: docker_id
      - source_labels: ['__meta_docker_container_name']
        target_label: docker_name
    The available meta labels are:

    • __meta_docker_container_id: the ID of the container
    • __meta_docker_container_name: the name of the container
    • __meta_docker_container_network_mode: the network mode of the container
    • __meta_docker_container_label_<labelname>: each label of the container
    • __meta_docker_container_log_stream: the log stream type stdout or stderr
    • __meta_docker_network_id: the ID of the network
    • __meta_docker_network_name: the name of the network
    • __meta_docker_network_ingress: whether the network is ingress
    • __meta_docker_network_internal: whether the network is internal
    • __meta_docker_network_label_<labelname>: each label of the network
    • __meta_docker_network_scope: the scope of the network
    • __meta_docker_network_ip: the IP of the container in this network
    • __meta_docker_port_private: the port on the container
    • __meta_docker_port_public: the external port if a port-mapping exists
    • __meta_docker_port_public_ip: the public IP if a port-mapping exists

    These labels can be used during relabeling. For instance, the following configuration scrapes the container named flog and removes the leading slash (/) from the container name. yaml

    scrape_configs:
  - job_name: flog_scrape
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: name
            values: [flog]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

  • New: Pipeline building.

    In this issue there are nice examples on different pipelines.

  • New: Drop logs.

    If you don't want the logs that have the keyword systemd-journal and value docker-compose you can add the next pipeline stage:

    pipeline_stages:
  - drop:
      source: syslog_identifier
      value: docker-compose

  • New: Install with ansible.

    Use patrickjahns ansible role. Some interesting variables are:

    loki_url: localhost
promtail_system_user: root

promtail_config_clients:
  - url: "http://{{ loki_url }}:3100/loki/api/v1/push"
    external_labels:
      hostname: "{{ ansible_hostname }}"

  • New: Troubleshooting promtail.

    Find where is the positions.yaml file and see if it evolves.

    Sometimes if you are not seeing the logs in loki it's because the query you're running is not correct.

  • New: Set the hostname label on all logs.

    There are many ways to do it:

    This won't work if you're using promtail within a docker-compose because you can't use bash expansion in the docker-compose.yaml file - Allowing env expansion and setting it in the promtail conf. You can launch the promtail command with -config.expand-env and then set in each scrape jobs: 

    labels:
    host: ${HOSTNAME}
    This won't work either if you're using promtail within a docker as it will give you the ID of the docker - Set it in the promtail_config_clients field as external_labels of each promtail config: 
    promtail_config_clients:
  - url: "http://{{ loki_url }}:3100/loki/api/v1/push"
    external_labels:
      hostname: "{{ ansible_hostname }}"
    - Hardcode it for each promtail config scraping config as static labels. If you're using ansible or any deployment method that supports jinja expansion set it that way 
    labels:
    host: {{ ansible_hostname }}

  • New: Introduce logcli.

    logcli is the command-line interface to Grafana Loki. It facilitates running LogQL queries against a Loki instance.

    Installation Download the logcli binary from the Loki releases page and install it somewhere in your $PATH.

    Usage logcli points to the local instance http://localhost:3100 directly, if you want another one export the LOKI_ADDR environment variable.

    Run a query:

    logcli query '{job="loki-ops/consul"}'

    You can also set the time range and output format

    logcli query \
     --timezone=UTC  \
     --from="2024-06-10T07:23:36Z" \
     --to="2024-06-12T16:23:58Z" \
     --output=jsonl \
     '{job="docker", container="aleph_ingest-file_1"} | json | __error__=`` | severity =~ `WARNING|ERROR` | message !~ `Queueing failed task for retry.*` | logger!=`ingestors.manager`'

    References

Grafana⚑

AlertManager⚑

  • Correction: Add another source on how to silence alerts.

    If previous guidelines don't work for you, you can use the sleep peacefully guidelines to tackle it at query level.

  • New: Using time intervals.

    The values of time_intervals can be:

    - times:
  [ - <time_range> ...]
  weekdays:
  [ - <weekday_range> ...]
  days_of_month:
  [ - <days_of_month_range> ...]
  months:
  [ - <month_range> ...]
  years:
  [ - <year_range> ...]
  location: <string>

    All fields are lists. Within each non-empty list, at least one element must be satisfied to match the field. If a field is left unspecified, any value will match the field. For an instant of time to match a complete time interval, all fields must match. Some fields support ranges and negative indices, and are detailed below. If a time zone is not specified, then the times are taken to be in UTC.

    • time_range: Ranges inclusive of the starting time and exclusive of the end time to make it easy to represent times that start/end on hour boundaries. For example, start_time: '17:00' and end_time: '24:00' will begin at 17:00 and finish immediately before 24:00. They are specified like so: 

      times:
- start_time: HH:MM
  end_time: HH:MM

    • weekday_range: A list of days of the week, where the week begins on Sunday and ends on Saturday. Days should be specified by name (e.g. 'Sunday'). For convenience, ranges are also accepted of the form <start_day>:<end_day> and are inclusive on both ends. For example: ['monday:wednesday','saturday', 'sunday']

    • days_of_month_range: A list of numerical days in the month. Days begin at 1. Negative values are also accepted which begin at the end of the month, e.g. -1 during January would represent January 31. For example: ['1:5', '-3:-1']. Extending past the start or end of the month will cause it to be clamped. E.g. specifying ['1:31'] during February will clamp the actual end date to 28 or 29 depending on leap years. Inclusive on both ends.

    • month_range: A list of calendar months identified by a case-insensitive name (e.g. 'January') or by number, where January = 1. Ranges are also accepted. For example, ['1:3', 'may:august', 'december']. Inclusive on both ends.

    • year_range: A numerical list of years. Ranges are accepted. For example, ['2020:2022', '2030']. Inclusive on both ends.

    • location: A string that matches a location in the IANA time zone database. For example, 'Australia/Sydney'. The location provides the time zone for the time interval. For example, a time interval with a location of 'Australia/Sydney' that contained something like:

    times:
- start_time: 09:00
  end_time: 17:00
weekdays: ['monday:friday']

    Would include any time that fell between the hours of 9:00AM and 5:00PM, between Monday and Friday, using the local time in Sydney, Australia. You may also use 'Local' as a location to use the local time of the machine where Alertmanager is running, or 'UTC' for UTC time. If no timezone is provided, the time interval is taken to be in UTC.

Process Exporter⚑

Databases⚑

PostgreSQL⚑

  • New: Store expensive calculation values in a postgresql database.

    First you need to think if you actually need to store the calculations or you can do them on the fly with views. If views are too slow you can either use materialized views or triggers over calculation tables.

    Materialized views are simpler to maintain but have some disadvantages such as outdated data or unneeded processing of data. If you need totally current information or if you don't want to periodically do the calculations on all the rows then triggers are probably the better solution.

  • New: Drop all tables of a database.

    drop schema public cascade;
create schema public;

  • New: Views.

    A view is a named query stored in the PostgreSQL database server. A view is defined based on one or more tables which are known as base tables, and the query that defines the view is referred to as a defining query.

    After creating a view, you can query data from it as you would from a regular table. Behind the scenes, PostgreSQL will rewrite the query against the view and its defining query, executing it to retrieve data from the base tables.

    Views do not store data except the materialized views. In PostgreSQL, you can create special views called materialized views that store data physically and periodically refresh it from the base tables.

    Simple views can be updatable.

    Advantages of views - Simplifying complex queries: Views help simplify complex queries. Instead of dealing with joins, aggregations, or filtering conditions, you can query from views as if they were regular tables.

    Typically, first, you create views based on complex queries and store them in the database. Then, you can use simple queries based on views instead of using complex queries.

    • Logical data independence: If your applications use views, you can freely modify the structure of the base tables. In other words, views enable you to create a layer of abstraction over the underlying tables.

    • Security and access control: Views enable fine-grained control over data access. You can create views that expose subsets of data in the base tables, hiding sensitive information.

    This is particularly useful when you have applications that require access to distinct portions of the data.

    Creating a view In PostgreSQL, a view is a named query stored in the database server. To create a new view, you can use the CREATE VIEW statement.

    CREATE VIEW view_name
AS
  query;

    In this syntax:

    • Specify the name of the view after the CREATE VIEW keywords.
    • Specify a SELECT statement (query) that defines the view. The query is often referred to as the defining query of the view.

    Creating a view examples

    We’ll use the customer table from the sample database:

    Basic CREATE VIEW statement example

    The following example uses the CREATE VIEW statement to create a view based on the customer table:

    CREATE VIEW contact AS
SELECT
  first_name,
  last_name,
  email
FROM
  customer;

    Output:

    CREATE VIEW

    The following query data from the contact view:

    SELECT * FROM contact;

    Output:

     first_name  |  last_name   |                  email
-------------+--------------+------------------------------------------
 Jared       | Ely          | jared.ely@sakilacustomer.org
 Mary        | Smith        | mary.smith@sakilacustomer.org
 Patricia    | Johnson      | patricia.johnson@sakilacustomer.org
...

    Using the CREATE VIEW statement to create a view based on a complex query

    The following example creates a view based on the tables customer, address, city, and country:

    CREATE VIEW customer_info AS
SELECT
  first_name,
  last_name,
  email,
  phone,
  city,
  postal_code,
  country
FROM
  customer
  INNER JOIN address USING (address_id)
  INNER JOIN city USING (city_id)
  INNER JOIN country USING (country_id);

    The following query retrieves data from the customer_info view:

    SELECT * FROM customer_info;

    Output:

     first_name  |  last_name   |                  email                   |    phone     |            city            | postal_code |                country
-------------+--------------+------------------------------------------+--------------+----------------------------+-------------+---------------------------------------
 Jared       | Ely          | jared.ely@sakilacustomer.org             | 35533115997  | Purwakarta                 | 25972       | Indonesia
 Mary        | Smith        | mary.smith@sakilacustomer.org            | 28303384290  | Sasebo                     | 35200       | Japan
 Patricia    | Johnson      | patricia.johnson@sakilacustomer.org      | 838635286649 | San Bernardino             | 17886       | United States
...

    Creating a view based on another view

    The following statement creates a view called customer_usa based on the customer_info view. The customer_usa returns the customers who are in the United States:

    CREATE VIEW customer_usa
AS
SELECT
  *
FROM
  customer_info
WHERE
  country = 'United States';

    Here’s the query that retrieves data from the customer_usa view:

    SELECT * FROM customer_usa;

    Output:

     first_name | last_name  |                email                 |    phone     |          city           | postal_code |    country
------------+------------+--------------------------------------+--------------+-------------------------+-------------+---------------
 Zachary    | Hite       | zachary.hite@sakilacustomer.org      | 191958435142 | Akron                   | 88749       | United States
 Richard    | Mccrary    | richard.mccrary@sakilacustomer.org   | 262088367001 | Arlington               | 42141       | United States
 Diana      | Alexander  | diana.alexander@sakilacustomer.org   | 6171054059   | Augusta-Richmond County | 30695       | United States
...

    Replacing a view Note: for simple changes check alter views

    To change the defining query of a view, you use the CREATE OR REPLACE VIEW statement:

    CREATE OR REPLACE VIEW view_name
AS
  query;

    In this syntax, you add the OR REPLACE between the CREATE and VIEW keywords. If the view already exists, the statement replaces the existing view; otherwise, it creates a new view.

    For example, the following statement changes the defining query of the contact view to include the phone information from the address table:

    CREATE OR REPLACE VIEW contact AS
SELECT
  first_name,
  last_name,
  email,
  phone
FROM
  customer
INNER JOIN address USING (address_id);

    Display a view on psql

    To display a view on psql, you follow these steps:

    First, open the Command Prompt on Windows or Terminal on Unix-like systems and connect to the PostgreSQL server:

    psql -U postgres

    Second, change the current database to dvdrental:

    \c dvdrental

    Third, display the view information using the \d+ view_name command. For example, the following shows the contact view:

    \d+ contact

    Output:

                                        View "public.contact"
   Column   |         Type          | Collation | Nullable | Default | Storage  | Description
------------+-----------------------+-----------+----------+---------+----------+-------------
 first_name | character varying(45) |           |          |         | extended |
 last_name  | character varying(45) |           |          |         | extended |
 email      | character varying(50) |           |          |         | extended |
 phone      | character varying(20) |           |          |         | extended |
View definition:
 SELECT customer.first_name,
    customer.last_name,
    customer.email,
    address.phone
   FROM customer
     JOIN address USING (address_id);

    Updatable views Recursive views Alter views

  • New: Materialized Views.

    PostgreSQL extends the view concept to the next level which allows views to store data physically. These views are called materialized views.

    Materialized views cache the result set of an expensive query and allow you to refresh data periodically.

    The materialized views can be useful in many cases that require fast data access. Therefore, you often find them in data warehouses and business intelligence applications.

    Benefits of materialized views

    • Improve query efficiency: If a query takes a long time to run, it could be because there are a lot of transformations being done to the data: subqueries, functions, and joins, for example.

    A materialized view can combine all of that into a single result set that’s stored like a table.

    This means that any user or application that needs to get this data can just query the materialized view itself, as though all of the data is in the one table, rather than running the expensive query that uses joins, functions, or subqueries.

    Calculations can also be added to materialized views for any fields you may need, which can save time, and are often not stored in the database.

    • Simplify a query: Like a regular view, a materialized view can also be used to simplify a query. If a query is using a lot of logic such as joins and functions, using a materialized view can help remove some of that logic and place it into the materialized view.

    Disadvantages of a Materialized View - Updates to data need to be set up: The main disadvantage to using materialized views is that the data needs to be refreshed.

    The data that’s used to populate the materialized view is stored in the database tables. These tables can have their data updated, inserted, or deleted. When that happens, the data in the materialized view needs to be updated.

    This can be done manually, but it should be done automatically.

    • Incremental updates are not supported: So the whole view is generated on each refresh.
    • Data may be inconsistent: Because the data is stored separately in the materialized view, the data in the materialized view may be inconsistent with the data in the underlying tables.

    This may be an issue if you are expecting or relying on data to be consistent.

    However, for scenarios where it doesn’t matter (e.g. monthly reporting on months in the past), then it may be OK.

    • Storage Requirements: Materialized Views can consume significant storage space, depending on the size of your dataset. This consideration is crucial, especially in resource-limited environments.

    Creating materialized views

    To create a materialized view, you use the CREATE MATERIALIZED VIEW statement as follows:

    CREATE MATERIALIZED VIEW [IF NOT EXISTS] view_name
AS
query
WITH [NO] DATA;

    How it works.

    • First, specify the view_name after the CREATE MATERIALIZED VIEW clause
    • Second, add the query that retrieves data from the underlying tables after the AS keyword.
    • Third, if you want to load data into the materialized view at the creation time, use the WITH DATA option; otherwise, you use WITH NO DATA option. If you use the WITH NO DATA option, the view is flagged as unreadable. It means that you cannot query data from the view until you load data into it.
    • Finally, use the IF NOT EXISTS option to conditionally create a view only if it does not exist.

    Refreshing data for materialized views

    Postgresql will never refresh the data by it's own, you need to define the processes that will update it.

    To load or update the data into a materialized view, you use the REFRESH MATERIALIZED VIEW statement:

    REFRESH MATERIALIZED VIEW view_name;

    When you refresh data for a materialized view, PostgreSQL locks the underlying tables. Consequently, you will not be able to retrieve data from underlying tables while data is loading into the view.

    To avoid this, you can use the CONCURRENTLY option.

    REFRESH MATERIALIZED VIEW CONCURRENTLY view_name;

    With the CONCURRENTLY option, PostgreSQL creates a temporary updated version of the materialized view, compares two versions, and performs INSERT and UPDATE only the differences.

    PostgreSQL allows you to retrieve data from a materialized view while it is being updated. One requirement for using CONCURRENTLY option is that the materialized view must have a UNIQUE index.

    Automatic update of materialized views

    Removing materialized views

    To remove a materialized view, you use the DROP MATERIALIZED VIEW statement:

    DROP MATERIALIZED VIEW view_name;

    In this syntax, you specify the name of the materialized view that you want to drop after the DROP MATERIALIZED VIEW keywords.

    Materialized view example We’ll use the tables in the sample database for creating a materialized view.

    First, create a materialized view named rental_by_category using the CREATE MATERIALIZED VIEW statement:

    CREATE MATERIALIZED VIEW rental_by_category
AS
 SELECT c.name AS category,
    sum(p.amount) AS total_sales
   FROM (((((payment p
     JOIN rental r ON ((p.rental_id = r.rental_id)))
     JOIN inventory i ON ((r.inventory_id = i.inventory_id)))
     JOIN film f ON ((i.film_id = f.film_id)))
     JOIN film_category fc ON ((f.film_id = fc.film_id)))
     JOIN category c ON ((fc.category_id = c.category_id)))
  GROUP BY c.name
  ORDER BY sum(p.amount) DESC
WITH NO DATA;

    Because of the WITH NO DATA option, you cannot query data from the view. If you attempt to do so, you’ll get the following error message:

    SELECT * FROM rental_by_category;

    Output:

    [Err] ERROR: materialized view "rental_by_category" has not been populated
HINT: Use the REFRESH MATERIALIZED VIEW command.

    PostgreSQL is helpful to give you a hint to ask for loading data into the view.

    Second, load data into the materialized view using the REFRESH MATERIALIZED VIEW statement:

    REFRESH MATERIALIZED VIEW rental_by_category;

    Third, retrieve data from the materialized view:

    SELECT * FROM rental_by_category;

    Output:

     category   | total_sales
-------------+-------------
 Sports      |     4892.19
 Sci-Fi      |     4336.01
 Animation   |     4245.31
 Drama       |     4118.46
 Comedy      |     4002.48
 New         |     3966.38
 Action      |     3951.84
 Foreign     |     3934.47
 Games       |     3922.18
 Family      |     3830.15
 Documentary |     3749.65
 Horror      |     3401.27
 Classics    |     3353.38
 Children    |     3309.39
 Travel      |     3227.36
 Music       |     3071.52
(16 rows)

    From now on, you can refresh the data in the rental_by_category view using the REFRESH MATERIALIZED VIEW statement.

    However, to refresh it with CONCURRENTLY option, you need to create a UNIQUE index for the view first.

    CREATE UNIQUE INDEX rental_category
ON rental_by_category (category);

    Let’s refresh data concurrently for the rental_by_category view.

    REFRESH MATERIALIZED VIEW CONCURRENTLY rental_by_category;

Hardware⚑

ECC RAM⚑

  • New: Introduce ECC RAM.

    Error Correction Code (ECC) is a mechanism used to detect and correct errors in memory data due to environmental interference and physical defects. ECC memory is used in high-reliability applications that cannot tolerate failure due to corrupted data.

    Installation: Due to additional circuitry required for ECC protection, specialized ECC hardware support is required by the CPU chipset, motherboard and DRAM module. This includes the following:

    • Server-grade CPU chipset with ECC support (Intel Xeon, AMD Ryzen)
    • Motherboard supporting ECC operation
    • ECC RAM

    Consult the motherboard and/or CPU documentation for the specific model to verify whether the hardware supports ECC. Use vendor-supplied list of certified ECC RAM, if provided.

    Most ECC-supported motherboards allow you to configure ECC settings from the BIOS setup. They are usually on the Advanced tab. The specific option depends on the motherboard vendor or model such as the following:

    • DRAM ECC Enable (American Megatrends, ASUS, ASRock, MSI)
    • ECC Mode (ASUS)

    Monitorization

    The mechanism for how ECC errors are logged and reported to the end-user depends on the BIOS and operating system. In most cases, corrected ECC errors are written to system/event logs. Uncorrected ECC errors may result in kernel panic or blue screen.

    The Linux kernel supports reporting ECC errors for ECC memory via the EDAC (Error Detection And Correction) driver subsystem. Depending on the Linux distribution, ECC errors may be reported by the following:

    • rasdaemon: monitor ECC memory and report both correctable and uncorrectable memory errors on recent Linux kernels.
    • mcelog (Deprecated): collects and decodes MCA error events on x86.
    • edac-utils (Deprecated): fills DIMM labels data and summarizes memory errors.

    To configure rasdaemon follow this article

    Confusion on boards supporting ECC

    I've read that even if some motherboards say that they "Support ECC" some of them don't do anything with it.

    On this post and the kernel docs show that you should see references to ACPI/WHEA in the specs manual. Ideally ACPI5 support.

    From the ) EINJ provides a hardware error injection mechanism. It is very useful for debugging and testing APEI and RAS features in general.

    You need to check whether your BIOS supports EINJ first. For that, look for early boot messages similar to this one:

    ACPI: EINJ 0x000000007370A000 000150 (v01 INTEL           00000001 INTL 00000001)

    Which shows that the BIOS is exposing an EINJ table - it is the mechanism through which the injection is done.

    Alternatively, look in /sys/firmware/acpi/tables for an "EINJ" file, which is a different representation of the same thing.

    It doesn't necessarily mean that EINJ is not supported if those above don't exist: before you give up, go into BIOS setup to see if the BIOS has an option to enable error injection. Look for something called WHEA or similar. Often, you need to enable an ACPI5 support option prior, in order to see the APEI,EINJ,... functionality supported and exposed by the BIOS menu.

    To use EINJ, make sure the following are options enabled in your kernel configuration:

    CONFIG_DEBUG_FS
CONFIG_ACPI_APEI
CONFIG_ACPI_APEI_EINJ

    One way to test it can be to run memtest as it sometimes shows ECC errors such as ** Warning** ECC injection may be disabled for AMD Ryzen (70h-7fh).

    Other people (1, 2 say that there are a lot of motherboards that NEVER report any corrected errors to the OS. In order to see corrected errors, PFEH (Platform First Error Handling) has to be disabled. On some motherboards and FW versions this setting is hidden from the user and always enabled, thus resulting in zero correctable errors getting reported.

    They also suggest to disable "Quick Boot". In order to initialize ECC, memory has to be written before it can be used. Usually this is done by BIOS, but with some motherboards this step is skipped if "Quick Boot" is enabled.

    The people behind memtest have a paid tool to test ECC

  • New: Introduce rasdaemon the ECC monitor.

    rasdaemon is a RAS (Reliability, Availability and Serviceability) logging tool. It records memory errors, using the EDAC tracing events. EDAC is a Linux kernel subsystem with handles detection of ECC errors from memory controllers for most chipsets on i386 and x86_64 architectures. EDAC drivers for other architectures like arm also exists.

    Installation 

    apt-get install rasdaemon

    The output will be available via syslog but you can show it to the foreground (-f) or to an sqlite3 database (-r)

    To post-process and decode received MCA errors on AMD SMCA systems, run:

    rasdaemon -p --status <STATUS_reg> --ipid <IPID_reg> --smca --family <CPU Family> --model <CPU Model> --bank <BANK_NUM>

    Status and IPID Register values (in hex) are mandatory. The smca flag with family and model are required if not decoding locally. Bank parameter is optional.

    You may also start it via systemd:

    systemctl start rasdaemon

    The rasdaemon will then output the messages to journald.

    Usage

    At this point rasdaemon should already be running on your system. You can now use the ras-mc-ctl tool to query the errors that have been detected. If everything is well configured you'll see something like:

    $: ras-mc-ctl --error-count
Label                 CE      UE
mc#0csrow#2channel#0  0   0
mc#0csrow#2channel#1  0   0
mc#0csrow#3channel#1  0   0
mc#0csrow#3channel#0  0   0

    If it's not you'll see:

    ras-mc-ctl: Error: No DIMMs found in /sys or new sysfs EDAC interface not found.

    The CE column represents the number of corrected errors for a given DIMM, UE represents uncorrectable errors that were detected. The label on the left shows the EDAC path under /sys/devices/system/edac/mc/ of every DIMM. This is not very readable, if you wish to improve the labeling read this article

    More ways to check is to run:

    $: ras-mc-ctl --status
ras-mc-ctl: drivers are loaded.

    You can also see a summary of the state with:

    $: ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1183.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1184.

    Monitorization

    You can use loki to monitor ECC errors shown in the logs with the next alerts:

    groups:
  - name: ecc
    rules:
      - alert: ECCError
        expr: |
          count_over_time({job="systemd-journal", unit="rasdaemon.service", level="error"} [5m])  > 0
        for: 1m
        labels:
            severity: critical
        annotations:
            summary: "Possible ECC error detected in {{ $labels.hostname}}"

      - alert: ECCWarning
        expr: |
          count_over_time({job="systemd-journal", unit="rasdaemon.service", level="warning"} [5m])  > 0
        for: 1m
        labels:
            severity: warning
        annotations:
            summary: "Possible ECC warning detected in {{ $labels.hostname}}"
      - alert: ECCAlert
        expr: |
          count_over_time({job="systemd-journal", unit="rasdaemon.service", level!~"info|error|warning"} [5m]) > 0
        for: 1m
        labels:
            severity: info
        annotations:
            summary: "ECC log trace with unknown severity level detected in {{ $labels.hostname}}"
    References

  • New: Check if system is actually using ECC.

    Another way is to run dmidecode. For ECC support you'll see: 

    $: dmidecode -t memory | grep ECC
  Error Correction Type: Single-bit ECC
  # or
  Error Correction Type: Multi-bit ECC

    No ECC:

    $: dmidecode -t memory | grep ECC
  Error Correction Type: None

    You can also test it with rasdaemon

GPU⚑

  • New: Introduce GPU.

    GPU or Graphic Processing Unit is a specialized electronic circuit initially designed to accelerate computer graphics and image processing (either on a video card or embedded on motherboards, mobile phones, personal computers, workstations, and game consoles).

    For years I've wanted to buy a graphic card but I've been stuck in the problem that I don't have a desktop. I have a X280 lenovo laptop used to work and personal use with an integrated card that has let me so far to play old games such as King Arthur Gold or Age of Empires II, but has hard times playing "newer" games such as It takes two. Last year I also bought a NAS with awesome hardware. So it makes no sense to buy a desktop just for playing.

    Now that I host Jellyfin on the NAS and that machine learning is on the hype with a lot of interesting solutions that can be self-hosted (whisper, chatgpt similar solutions...), it starts to make sense to add a GPU to the server. What made me give the step is that you can also self-host a gaming server to stream to any device! It makes so much sense to have all the big guns inside the NAS and stream the content to the less powerful devices.

    That way if you host services, you make the most use of the hardware.

  • New: Install cuda.

    CUDA is a parallel computing platform and programming model invented by NVIDIA®. It enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU). If you're not using Debian 11 follow these instructions

    Base Installer

    wget https://developer.download.nvidia.com/compute/cuda/12.5.1/local_installers/cuda-repo-debian11-12-5-local_12.5.1-555.42.06-1_amd64.deb
sudo dpkg -i cuda-repo-debian11-12-5-local_12.5.1-555.42.06-1_amd64.deb
sudo cp /var/cuda-repo-debian11-12-5-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo add-apt-repository contrib
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-5

    Additional installation options are detailed here.

    Driver Installer

    To install the open kernel module flavor:

    sudo apt-get install -y nvidia-kernel-open-dkms
sudo apt-get install -y cuda-drivers

    Install cuda:

    apt-get install cuda
reboot

    Install nvidia card

    Check if your card is supported in the releases supported by your OS - If it's supported - If it's not supported

    Ensure the GPUs are Installed

    Install pciutils:

    Ensure that the lspci command is installed (which lists the PCI devices connected to the server):

    sudo apt-get -y install pciutils

    Check Installed Nvidia Cards: Perform a quick check to determine what Nvidia cards have been installed:

    lspci | grep VGA

    The output of the lspci command above should be something similar to:

    00:02.0 VGA compatible controller: Intel Corporation 4th Gen ...
01:00.0 VGA compatible controller: Nvidia Corporation ...

    If you do not see a line that includes Nvidia, then the GPU is not properly installed. Otherwise, you should see the make and model of the GPU devices that are installed.

    Disable Nouveau

    Blacklist Nouveau in Modprobe: The nouveau driver is an alternative to the Nvidia drivers generally installed on the server. It does not work with CUDA and must be disabled. The first step is to edit the file at /etc/modprobe.d/blacklist-nouveau.conf.

    Create the file with the following content:

    cat <<EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off
EOF

    Then, run the following commands:

    echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
sudo update-initramfs -u

    Update Grub to Blacklist Nouveau:

    Backup your grub config template:

    sudo cp /etc/default/grub /etc/default/grub.bak

    Then, update your grub config template at /etc/default/grub. Add rd.driver.blacklist=nouveau and rcutree.rcu_idle_gp_delay=1 to the GRUB_CMDLINE_LINUX variable. For example, change:

    GRUB_CMDLINE_LINUX="quiet"

    to:

    GRUB_CMDLINE_LINUX="quiet rd.driver.blacklist=nouveau rcutree.rcu_idle_gp_delay=1"

    Then, rebuild your grub config:

    sudo grub2-mkconfig -o /boot/grub/grub.cfg

    Install prerequisites

    The following prerequisites should be installed before installing the Nvidia drivers:

    sudo apt-get -y install linux-headers-$(uname -r) make gcc-4.8
sudo apt-get -y install acpid dkms

    Close X Server:

    Before running the install, you should exit out of any X environment, such as Gnome, KDE, or XFCE. To exit the X session, switch to a TTY console using Ctrl-Alt-F1 and then determine whether you are running lightdm or gdm by running:

    sudo ps aux | grep "lightdm|gdm|kdm"

    Depending on which is running, stop the service, running the following commands (substitute gdm or kdm for lightdm as appropriate):

    sudo service lightdm stop
sudo init 3

    Install Drivers Only:

    To accommodate GL-accelerated rendering, OpenGL and GL Vendor Neutral Dispatch (GLVND) are now required and should be installed with the Nvidia drivers. OpenGL is an installation option in the *.run type of drivers. In other types of the drivers, OpenGL is enabled by default in most modern versions (dated 2016 and later). GLVND can be installed using the installer menus or via the --glvnd-glx-client command line flag.

    This section deals with installing the drivers via the *.run executables provided by Nvidia.

    To download only the drivers, navigate to http://www.nvidia.com/object/unix.html and click the Latest Long Lived Branch version under the appropriate CPU architecture. On the ensuing page, click Download and then click Agree and Download on the page that follows.

    The Unix drivers found in the link above are also compatible with all Nvidia Tesla models.

    If you'd prefer to download the full driver repository, Nvidia provides a tool to recommend the most recent available driver for your graphics card at http://www.Nvidia.com/Download/index.aspx?lang=en-us.

    If you are unsure which Nvidia devices are installed, the lspci command should give you that information:

    lspci | grep -i "nvidia"

    Download the recommended driver executable. Change the file permissions to allow execution:

    chmod +x ./NVIDIA-Linux-$(uname -m)-*.run

    Run the install.

    To check that the GPU is well installed and functioning properly, you can use the nvidia-smi command. This command provides detailed information about the installed Nvidia GPUs, including their status, utilization, and driver version.

    First, ensure the Nvidia drivers are installed. Then, run:

    nvidia-smi

    If the GPU is properly installed, you should see an output that includes information about the GPU, such as its model, memory usage, and driver version. The output will look something like this:

    +-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66       Driver Version: 450.66       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   38C    P8    29W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

    If you encounter any errors or the GPU is not listed, there may be an issue with the installation or configuration of the GPU drivers.

    Measure usage

    For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU.

    Load test the gpu

    First make sure you have CUDA installed, then install the gpu_burn tool 

    git clone https://github.com/wilicc/gpu-burn
cd gpu-burn
make

    To run a test for 60 seconds run: 

    ./gpu_burn 60

    Monitor it with Prometheus

    NVIDIA DCGM is a set of tools for managing and monitoring NVIDIA GPUs in large-scale, Linux-based cluster environments. It’s a low overhead tool that can perform a variety of functions including active health monitoring, diagnostics, system validation, policies, power and clock management, group configuration, and accounting. For more information, see the DCGM User Guide.

    You can use DCGM to expose GPU metrics to Prometheus using dcgm-exporter.

    • Install NVIDIA Container Kit: The NVIDIA Container Toolkit allows users to build and run GPU accelerated containers. The toolkit includes a container runtime library and utilities to automatically configure containers to leverage NVIDIA GPUs.
    sudo apt-get install -y nvidia-container-toolkit
    • Configure the container runtime by using the nvidia-ctk command:

    sudo nvidia-ctk runtime configure --runtime=docker
    - Restart the Docker daemon:

    sudo systemctl restart docker

    Determine the distribution name:

    distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')

    Download the meta-package to set up the CUDA network repository:

    wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.1-1_all.deb

    Install the repository meta-data and the CUDA GPG key:

    sudo dpkg -i cuda-keyring_1.1-1_all.deb

    Update the Apt repository cache:

    sudo apt-get update

    Now, install DCGM:

    sudo apt-get install -y datacenter-gpu-manager

    Enable the DCGM systemd service (on reboot) and start it now:

    sudo systemctl --now enable nvidia-dcgm

    You should see output similar to this:

    ● dcgm.service - DCGM service
  Loaded: loaded (/usr/lib/systemd/system/dcgm.service; disabled; vendor preset: enabled)
  Active: active (running) since Mon 2020-10-12 12:18:57 PDT; 14s ago
Main PID: 32847 (nv-hostengine)
    Tasks: 7 (limit: 39321)
  CGroup: /system.slice/dcgm.service
          └─32847 /usr/bin/nv-hostengine -n

Oct 12 12:18:57 ubuntu1804 systemd[1]: Started DCGM service.
Oct 12 12:18:58 ubuntu1804 nv-hostengine[32847]: DCGM initialized
Oct 12 12:18:58 ubuntu1804 nv-hostengine[32847]: Host Engine Listener Started

    To verify installation, use dcgmi to query the system. You should see a listing of all supported GPUs (and any NVSwitches) found in the system:

    dcgmi discovery -l

    Output:

    8 GPUs found.
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information                                                   |
+--------+----------------------------------------------------------------------+
| 0      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:07:00.0                                         |
|        | Device UUID: GPU-1d82f4df-3cf9-150d-088b-52f18f8654e1                |
+--------+----------------------------------------------------------------------+
| 1      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:0F:00.0                                         |
|        | Device UUID: GPU-94168100-c5d5-1c05-9005-26953dd598e7                |
+--------+----------------------------------------------------------------------+
| 2      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:47:00.0                                         |
|        | Device UUID: GPU-9387e4b3-3640-0064-6b80-5ace1ee535f6                |
+--------+----------------------------------------------------------------------+
| 3      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:4E:00.0                                         |
|        | Device UUID: GPU-cefd0e59-c486-c12f-418c-84ccd7a12bb2                |
+--------+----------------------------------------------------------------------+
| 4      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:87:00.0                                         |
|        | Device UUID: GPU-1501b26d-f3e4-8501-421d-5a444b17eda8                |
+--------+----------------------------------------------------------------------+
| 5      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:90:00.0                                         |
|        | Device UUID: GPU-f4180a63-1978-6c56-9903-ca5aac8af020                |
+--------+----------------------------------------------------------------------+
| 6      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:B7:00.0                                         |
|        | Device UUID: GPU-8b354e3e-0145-6cfc-aec6-db2c28dae134                |
+--------+----------------------------------------------------------------------+
| 7      | Name: A100-SXM4-40GB                                                 |
|        | PCI Bus ID: 00000000:BD:00.0                                         |
|        | Device UUID: GPU-a16e3b98-8be2-6a0c-7fac-9cb024dbc2df                |
+--------+----------------------------------------------------------------------+
6 NvSwitches found.
+-----------+
| Switch ID |
+-----------+
| 11        |
| 10        |
| 13        |
| 9         |
| 12        |
| 8         |
+-----------+

    Install the dcgm-exporter

    As it doesn't need any persistence I've added it to the prometheus docker compose:

      dcgm-exporter:
    # latest didn't work
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.6-3.4.2-ubuntu22.04
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    restart: unless-stopped
    container_name: dcgm-exporter

    And added the next scraping config in prometheus.yml

      - job_name: dcgm-exporter
    metrics_path: /metrics
    static_configs:
    - targets:
      - dcgm-exporter:9400

    Adding alerts

    Tweak the next alerts for your use case.

    ---
groups:
- name: dcgm-alerts
  rules:
  - alert: GPUHighTemperature
    expr: DCGM_FI_DEV_GPU_TEMP > 80
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "GPU High Temperature (instance {{ $labels.instance }})"
      description: "The GPU temperature is above 80°C for more than 5 minutes.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: GPUMemoryUtilizationHigh
    expr: DCGM_FI_DEV_MEM_COPY_UTIL > 90
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "GPU Memory Utilization High (instance {{ $labels.instance }})"
      description: "The GPU memory utilization is above 90% for more than 10 minutes.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: GPUComputeUtilizationHigh
    expr: DCGM_FI_DEV_GPU_UTIL > 90
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "GPU Compute Utilization High (instance {{ $labels.instance }})"
      description: "The GPU compute utilization is above 90% for more than 10 minutes.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: GPUPowerUsageHigh
    expr: DCGM_FI_DEV_POWER_USAGE > 160
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "GPU Power Usage High (instance {{ $labels.instance }})"
      description: "The GPU power usage is above 160W for more than 5 minutes.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: GPUUnavailable
    expr: up{job="dcgm-exporter"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "GPU Unavailable (instance {{ $labels.instance }})"
      description: "The DCGM Exporter instance is down or unreachable for more than 5 minutes.\n  LABELS: {{ $labels }}"

    Adding a dashboard

    I've tweaked this dashboard to simplify it. Check the article for the full json

Operating Systems⚑

Linux⚑

  • New: Linux loki alerts.

          - alert: TooManyLogs
        expr: |
          sum by(hostname) (count_over_time({job="systemd-journal"} [1d])) / sum by(hostname) (count_over_time({job="systemd-journal"} [1d] offset 1d)) > 1.5
        for: 0m
        labels:
            severity: warning
        annotations:
            summary: "The server {{ $labels.hostname}} is generating too many logs"

      - alert: TooFewLogs
        expr: |
          sum by(hostname) (count_over_time({job="systemd-journal"} [1d])) / sum by(hostname) (count_over_time({job="systemd-journal"} [1d] offset 1d)) < 0.5
        for: 0m
        labels:
            severity: warning
        annotations:
            summary: "The server {{ $labels.hostname}} is generating too few logs"

Linux Snippets⚑

  • New: Makefile use bash instead of sh.

    The program used as the shell is taken from the variable SHELL. If this variable is not set in your makefile, the program /bin/sh is used as the shell.

    So put SHELL := /bin/bash at the top of your makefile, and you should be good to go.

  • New: Recover the message of a commit if the command failed.

    git commit can fail for reasons such as gpg.commitsign = true && gpg fails, or when running a pre-commit. Retrying the command opens a blank editor and the message seems to be lost.

    The message is saved though in .git/COMMIT_EDITMSG, so you can:

    git commit -m "$(cat .git/COMMIT_EDITMSG)"

    Or in general (suitable for an alias for example):

    git commit -m "$(cat "$(git rev-parse --git-dir)/COMMIT_EDITMSG)")"

  • New: Configure nginx to restrict methods.

    server {
    listen 80;
    server_name yourdomain.com;

    location / {
        if ($request_method !~ ^(GET|POST)$ ) {
            return 405;
        }

        try_files $uri $uri/ =404;
    }
}

  • New: Configure nginx location regexp to accept dashes.

    location ~* /share/[\w-]+ {
  root /home/project_root;
}

  • New: Configure nginx location to accept many paths.

    location ~ ^/(static|media)/ {
  root /home/project_root;
}

  • New: Remove image metadata.

    exiftool -all:all= /path/to/file

  • New: Get the size increment of a directory between two dates.

    To see how much has a directory grown between two dates you can use:

    find /path/to/directory -type f -newerat 2022-12-31 ! -newerat 2024-01-01 -printf "%s\\n" | awk '{s+=$1} END {print s}'

    It finds all the files in that directory that were created in the 2023, it only prints their size in bytes and then it adds them all up.

  • New: Introduce detox.

    detox cleans up filenames from the command line.

    Installation:

    apt-get install detox

    Usage:

    detox *

  • New: Rotate image with the command line.

    If you want to overwrite in-place, mogrify from the ImageMagick suite seems to be the easiest way to achieve this:

    mogrify -rotate -90 *.jpg

mogrify -rotate 90 *.jpg

  • New: Configure desktop icons in gnome.

    Latest versions of gnome dont have desktop icons read this article to fix this

  • New: Send multiline messages with notify-send.

    The title can't have new lines, but the body can.

    notify-send "Title" "This is the first line.\nAnd this is the second.")
    feat(linux_snippets#Find BIOS version): Find BIOS version

    dmidecode | less

  • New: Reboot server on kernel panic.

    The proc/sys/kernel/panic file gives read/write access to the kernel variable panic_timeout. If this is zero, the kernel will loop on a panic; if nonzero it indicates that the kernel should autoreboot after this number of seconds. When you use the software watchdog device driver, the recommended setting is 60.

    To set the value add the next contents to the /etc/sysctl.d/99-panic.conf

    kernel.panic = 60

    Or with an ansible task:

    - name: Configure reboot on kernel panic
  become: true
  lineinfile:
    path: /etc/sysctl.d/99-panic.conf
    line: kernel.panic = 60
    create: true
    state: present

  • New: Share a calculated value between github actions steps.

    You need to set a step's output parameter. Note that the step will need an id to be defined to later retrieve the output value.

    echo "{name}={value}" >> "$GITHUB_OUTPUT"

    For example:

    - name: Set color
  id: color-selector
  run: echo "SELECTED_COLOR=green" >> "$GITHUB_OUTPUT"
- name: Get color
  env:
    SELECTED_COLOR: ${{ steps.color-selector.outputs.SELECTED_COLOR }}
  run: echo "The selected color is $SELECTED_COLOR"

  • New: Split a zip into sizes with restricted size.

    Something like:

    zip -9 myfile.zip *
zipsplit -n 250000000 myfile.zip

    Would produce myfile1.zip, myfile2.zip, etc., all independent of each other, and none larger than 250MB (in powers of ten). zipsplit will even try to organize the contents so that each resulting archive is as close as possible to the maximum size.

  • New: Find files that were modified between dates.

    The best option is the -newerXY. The m and t flags can be used.

    • m The modification time of the file reference
    • t reference is interpreted directly as a time

    So the solution is

    find . -type f -newermt 20111222 \! -newermt 20111225

    The lower bound in inclusive, and upper bound is exclusive, so I added 1 day to it. And it is recursive.

  • New: Add rofi launcher.

    pass is a command line password store

    Configure rofi launcher

    • Save this script somewhere in your $PATH
    • Configure your window manager to launch it whenever you need a password.

  • New: Get info of a mkv file.

    ffprobe file.mkv

  • New: Clean the logs of a unit of the journal.

    journalctl --vacuum-time=1s --unit=your.service

    If you wish to clear all logs use journalctl --vacuum-time=1s

  • New: Introduce Alacritty.

    Alacritty is a modern terminal emulator that comes with sensible defaults, but allows for extensive configuration. By integrating with other applications, rather than reimplementing their functionality, it manages to provide a flexible set of features with high performance.

    Installation

    • Clone the repo 
      git clone https://github.com/alacritty/alacritty.git
cd alacritty
    • Install rustup 
      curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
    • To make sure you have the right Rust compiler installed, run 
      rustup override set stable
rustup update stable
    • Install the dependencies 
      apt install cmake pkg-config libfreetype6-dev libfontconfig1-dev libxcb-xfixes0-dev libxkbcommon-dev python3
    • Build the release 
      cargo build --release
      If all goes well, this should place a binary at target/release/alacritty
    • Move the binary to somewhere in your PATH

    mv target/release/alacritty ~/.local/bin
    - Check the terminfo: To make sure Alacritty works correctly, either the alacritty or alacritty-direct terminfo must be used. The alacritty terminfo will be picked up automatically if it is installed. If the following command returns without any errors, the alacritty terminfo is already installed:

    infocmp alacritty

    If it is not present already, you can install it globally with the following command:

    sudo tic -xe alacritty,alacritty-direct extra/alacritty.info

    Configuration

    Alacritty's configuration file uses the TOML format. It doesn't create the config file for you, but it looks for one in ~/.config/alacrity/alacritty.toml

    Not there yet - Support for ligatures

    References - Homepage - Source - Docs

  • New: Set the vim filetype syntax in a comment.

    Add somewhere in your file:

  • New: Export environment variables in a crontab.

    If you need to expand the PATH in theory you can do it like this:

    PATH=$PATH:/usr/local/bin

* * * * * /path/to/my/script

    I've found however that sometimes this doesn't work and you need to specify it in the crontab line:

    * * * * * PATH=$PATH:/usr/local/bin /path/to/my/script

  • Correction: Docker prune without removing the manual networks.

    If you run the command docker system prune in conjunction with watchtower and manually defined networks you may run into the issue that the docker system prune acts just when the dockers are stopped and thus removing the networks, which will prevent the dockers to start. In those cases you can either make sure that docker system prune is never run when watchtower is doing the updates or you can split the command into the next script:

    date
echo "Pruning the containers"
docker container prune -f --filter "label!=prune=false"
echo "Pruning the images"
docker image prune -f --filter "label!=prune=false"
echo "Pruning the volumes"
docker volume prune -f

  • New: Debugging high IOwait.

    High I/O wait (iowait) on the CPU, especially at 50%, typically indicates that your system is spending a large portion of its time waiting for I/O operations (such as disk access) to complete. This can be caused by a variety of factors, including disk bottlenecks, overloaded storage systems, or inefficient applications making disk-intensive operations.

    Here’s a structured approach to debug and analyze high I/O wait on your server:

    ** Monitor disk I/O**

    First, verify if disk I/O is indeed the cause. Tools like iostat, iotop, and dstat can give you an overview of disk activity:

    • iostat: This tool reports CPU and I/O statistics. You can install it with apt-get install sysstat. Run the following command to check disk I/O stats:

    iostat -x 1
    The -x flag provides extended statistics, and 1 means it will report every second. Look for high values in the %util and await columns, which represent: - %util: Percentage of time the disk is busy (ideally should be below 90% for most systems). - await: Average time for I/O requests to complete.

    If either of these values is unusually high, it indicates that the disk subsystem is likely overloaded.

    • iotop: If you want a more granular look at which processes are consuming disk I/O, use iotop:
    sudo iotop -o

    This will show you the processes that are actively performing I/O operations.

    • dstat: Another useful tool for monitoring disk I/O in real-time:
    dstat -cdl 1

    This shows CPU, disk, and load stats, refreshing every second. Pay attention to the dsk/await value.

    Check disk health Disk issues such as bad sectors or failing drives can also lead to high I/O wait times. To check the health of your disks:

    • Use smartctl: This tool can give you a health check of your disks if they support S.M.A.R.T.
    sudo smartctl -a /dev/sda

    Check for any errors or warnings in the output. Particularly look for things like reallocated sectors or increasing "pending sectors."

    • dmesg logs: Look at the system logs for disk errors or warnings:
    dmesg | grep -i "error"

    If there are frequent disk errors, it may be time to replace the disk or investigate hardware issues.

    Look for disk saturation If the disk is saturated, no matter how fast the CPU is, it will be stuck waiting for data to come back from the disk. To further investigate disk saturation:

    • df -h: Check if your disk partitions are full or close to full.
    df -h
    • lsblk: Check how your disks are partitioned and how much data is written to each partition:
    lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
    • blktrace: For advanced debugging, you can use blktrace, which traces block layer events on your system.
    sudo blktrace -d /dev/sda -o - | blkparse -i -

    This will give you very detailed insights into how the system is interacting with the block device.

    Check for heavy disk-intensive processes Identify processes that might be using excessive disk I/O. You can use tools like iotop (as mentioned earlier) or pidstat to look for processes with high disk usage:

    • pidstat: Track per-process disk activity:
    pidstat -d 1

    This command will give you I/O statistics per process every second. Look for processes with high I/O values (r/s and w/s).

    • top or htop: While top or htop can show CPU usage, they can also show process-level disk activity. Focus on processes consuming high CPU or memory, as they might also be performing heavy I/O operations.

    check file system issues Sometimes the file system itself can be the source of I/O bottlenecks. Check for any file system issues that might be causing high I/O wait.

    • Check file system consistency: If you suspect the file system is causing issues (e.g., due to corruption), run a file system check. For ext4:
    sudo fsck /dev/sda1

    Ensure you unmount the disk first or do this in single-user mode.

    • Check disk scheduling: Some disk schedulers (like cfq or deadline) might perform poorly depending on your workload. You can check the scheduler used by your disk with:
    cat /sys/block/sda/queue/scheduler

    You can change the scheduler with:

    echo deadline > /sys/block/sda/queue/scheduler

    This might improve disk performance, especially for certain workloads.

    Examine system logs The system logs (/var/log/syslog or /var/log/messages) may contain additional information about hardware issues, I/O bottlenecks, or kernel-related warnings:

    sudo tail -f /var/log/syslog

    or

    sudo tail -f /var/log/messages

    Look for I/O or disk-related warnings or errors.

    Consider hardware upgrades or tuning

    • SSD vs HDD: If you're using HDDs, consider upgrading to SSDs. HDDs can be much slower in terms of I/O, especially if you have a high number of random read/write operations.
    • RAID Configuration: If you are using RAID, check the RAID configuration and ensure it's properly tuned for performance (e.g., using RAID-10 for a good balance of speed and redundancy).
    • Memory and CPU Tuning: If the server is swapping due to insufficient RAM, it can result in increased I/O wait. You might need to add more RAM or optimize the system to avoid excessive swapping.

    Check for swapping issues Excessive swapping can contribute to high I/O wait times. If your system is swapping (which happens when physical RAM is exhausted), I/O wait spikes as the system reads from and writes to swap space on disk.

    • Check swap usage:
    free -h

    If swap usage is high, you may need to add more physical RAM or optimize applications to reduce memory pressure.

  • New: Create a file with random data.

    Of 3.5 GB

    dd if=/dev/urandom of=random_file.bin bs=1M count=3584

  • New: Convert an html to a pdf.

    Using weasyprint

    Install it with pip install weasyprint PyMuPDF

    weasyprint input.html output.pdf

    It gave me better result than wkhtmltopdf

    Using wkhtmltopdf To convert the given HTML into a PDF with proper styling and formatting using a simple method on Linux, you can use wkhtmltopdf with some custom options.

    First, ensure that you have wkhtmltopdf installed on your system. If not, install it using your package manager (e.g., Debian: sudo apt-get install wkhtmltopdf).

    Then, convert the HTML to PDF using wkhtmltopdf with the following command:

    wkhtmltopdf --page-size A4 --margin-top 15mm --margin-bottom 15mm --encoding utf8 input.html output.pdf

    In this command: - --page-size A4: Sets the paper size to A4. - --margin-top 15mm and --margin-bottom 15mm: Adds top and bottom margins of 15 mm to the PDF.

    After running the command, you should have a nicely formatted output.pdf file in your current directory. This method preserves most of the original HTML styling while providing a simple way to export it as a PDF on Linux.

    If you need to zoom in, you can use the --zoom 1.2 flag. For this to work you need your css to be using the em sizes.

  • New: Format a drive to use a FAT32 system.

    sudo mkfs.vfat -F 32 /dev/sdX

    Replace /dev/sdX with your actual drive identifier

  • New: Get the newest file of a directory with nested directories and files.

    find . -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" "

  • Correction: How to debug a CPU Throttling high alert.

    If the docker is using less resources than the limits but they are still small (for example 0.1 CPUs) the issue may be that the CPU spikes are being throttle before they are shown in the CPU usage, the solution is then to increase the CPU limits

    # Create a systemd service for a non-root user

    To set up a systemd service as a non-root user, you can create a user-specific service file under your home directory. User services are defined in ~/.config/systemd/user/ and can be managed without root privileges.

  • New: Check the docker images sorted by size.

    docker images --format "{{.Repository}}:{{.Tag}}\t{{.Size}}" | sort -k2 -h

    You can also use the builtin docker system df -v to get a better understanding of the usage of disk space.

Wezterm⚑

  • New: Introduce wezterm.

    WezTerm is a powerful cross-platform terminal emulator and multiplexer implemented in Rust.

    Installation

    You can configure your system to use that APT repo by following these steps:

    curl -fsSL https://apt.fury.io/wez/gpg.key | sudo gpg --yes --dearmor -o /usr/share/keyrings/wezterm-fury.gpg
echo 'deb [signed-by=/usr/share/keyrings/wezterm-fury.gpg] https://apt.fury.io/wez/ * *' | sudo tee /etc/apt/sources.list.d/wezterm.list

    Update your dependencies:

    sudo apt update

    Now you can install wezterm:

    sudo apt install wezterm

    or to install a nightly build:

    sudo apt install wezterm-nightly
    Troubleshooting

    Install in Debian 12 error

    Install from nightly.

    References

Terminals⚑

  • New: Suggest to use nerd fonts.

    If you're thinking of adding a new font, you should take a look at the nerd fonts as they patch developer targeted fonts with a high number of glyphs (icons). Specifically to add a high number of extra glyphs from popular ‘iconic fonts’ such as Font Awesome, Devicons, Octicons, and others.

    For example you can get the FiraCode Nerd Font

  • New: Introduce Rofi.

    Rofi is a window switcher, application launcher and dmenu replacement.

    Installation

    sudo apt-get install rofi

    Usage To launch rofi directly in a certain mode, specify a mode with rofi -show <mode>. To show the run dialog:

    rofi -show run

    Or get the options from a script:

    ~/my_script.sh | rofi -dmenu

    Specify an ordered, comma-separated list of modes to enable. Enabled modes can be changed at runtime. Default key is Ctrl+Tab. If no modes are specified, all configured modes will be enabled. To only show the run and ssh launcher:

    rofi -modes "run,ssh" -show run

    The modes to combine in combi mode. For syntax to -combi-modes , see -modes. To get one merge view, of window,run, and ssh:

    rofi -show combi -combi-modes "window,run,ssh" -modes combi

    Configuration

    The configuration lives at ~/.config/rofi/config.rasi to create this file with the default conf run:

    rofi -dump-config > ~/.config/rofi/config.rasi

    Use fzf to do the matching

    To run once:

    rofi -show run -sorting-method fzf -matching fuzzy

    To persist them change those same values in the configuration.

    Theme changing To change the theme: - Choose the one you like most looking here - Run rofi-theme-selector to select it - Accept it with Alt + a

    Keybindings change

    Plugins You can write your custom plugins. If you're on python using python-rofi seems to be the best option although it looks unmaintained.

    Some interesting examples are:

    Other interesting references are:

    References - Source - Docs - Plugins

  • New: Do terminal comparison.

    Alacritty

    Pros:

    Cons:

    • The installation is difficult if you're not used to Rust.
    • Doesn't support for ligatures so Fira Code with ligatures looks weird.
    • Awful docs
    • Difficult to keep updated

    Kitty

    Pros:

    • Built in python
    • Supports ligatures
    • Nice docs
    • Easy installation

    Cons:

    • Vim bindings to move around and copy the buffer don't work well
    • When you sudo su on a server you need to copy the ~/.terminfo otherwise the shell is broken
    • You need to replace ssh so that they copy that file.

    Wezterm

    Pros:

    • Nice docs

    Cons:

    • Dev is enough narcissistic enough to not only add it in the name of the terminal but also to say his name as the main developer.

Gotify⚑

  • New: Complete installation.

    • Create the data directories: 
      mkdir -p /data/config/gotify/ /data/gotify
    • Assuming you're using an external proxy create the next docker compose in /data/config/gotify.
    ---
version: "3"

services:
  gotify:
    image: gotify/server
    container_name: gotify
    networks:
      - swag
    env_file:
      - .env
    volumes:
      - gotify-data:/app/data

networks:
  swag:
    external:
      name: swag

volumes:
  gotify-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/gotify

    With the next .env file:

      GOTIFY_SERVER_SSL_ENABLED=false

  GOTIFY_DATABASE_DIALECT=sqlite3
  GOTIFY_DATABASE_CONNECTION=data/gotify.db

  GOTIFY_DEFAULTUSER_NAME=admin
  GOTIFY_DEFAULTUSER_PASS=changeme

  GOTIFY_PASSSTRENGTH=10
  GOTIFY_UPLOADEDIMAGESDIR=data/images
  GOTIFY_PLUGINSDIR=data/plugins
  GOTIFY_REGISTRATION=false
  ```

* Create the service by adding a file `gotify.service` into `/etc/systemd/system/`
    [Unit] Description=gotify Requires=docker.service After=docker.service

    [Service] Restart=always User=root Group=docker WorkingDirectory=/data/config/gotify TimeoutStartSec=100 RestartSec=2s ExecStart=/usr/bin/docker-compose -f docker-compose.yaml up ExecStop=/usr/bin/docker-compose -f docker-compose.yaml down

    [Install] WantedBy=multi-user.target 

    * Copy the nginx configuration in your `site-confs`

  ```

  server {
      listen 443 ssl;
      listen [::]:443 ssl;

      server_name gotify.*;

      include /config/nginx/ssl.conf;

      client_max_body_size 0;

      # enable for ldap auth (requires ldap-location.conf in the location block)
      #include /config/nginx/ldap-server.conf;

      # enable for Authelia (requires authelia-location.conf in the location block)
      #include /config/nginx/authelia-server.conf;

      location / {
          # enable the next two lines for http auth
          #auth_basic "Restricted";
          #auth_basic_user_file /config/nginx/.htpasswd;

          # enable for ldap auth (requires ldap-server.conf in the server block)
          #include /config/nginx/ldap-location.conf;

          # enable for Authelia (requires authelia-server.conf in the server block)
          #include /config/nginx/authelia-location.conf;

          include /config/nginx/proxy.conf;
          include /config/nginx/resolver.conf;
          set $upstream_app gotify;
          set $upstream_port 80;
          set $upstream_proto http;
          proxy_pass $upstream_proto://$upstream_app:$upstream_port;
      }
  }
  ```
* Start the service `systemctl start gotify`
* Restart the nginx service `systemctl restart swag`
* Enable the service `systemctl enable gotify`.
* Login with the `admin` user
* Create a new user with admin permissions
* Delete the `admin` user

**Configuration**

- [Android client](https://github.com/gotify/android)
- Linux clients
  - [command line client](#command-line-client)
  - [Dunst client](https://github.com/ztpnk/gotify-dunst)
  - [gotify-desktop](https://github.com/desbma/gotify-desktop)
  - [rofi client](https://github.com/diddypod/rotify)

**Connect it with Alertmanager**

It's not trivial to connect it to Alertmanager([1](https://github.com/prometheus/alertmanager/issues/2120), [2](https://github.com/gotify/contrib/issues/21), [3](https://github.com/prometheus/alertmanager/issues/3729), [4](https://github.com/prometheus/alertmanager/issues/2120). The most popular way is to use [`alertmanager_gotify_bridge`](https://github.com/DRuggeri/alertmanager_gotify_bridge?tab=readme-ov-file).

We need to tweak the docker-compose to add the bridge:

```yaml

    Connect it with Authentik

    Here are some guides to connect it to authentik. The problem is that the clients you want to use must support it

    References

HAProxy⚑

journald⚑

  • New: Introduce journald.

    journald is a system service that collects and stores logging data. It creates and maintains structured, indexed journals based on logging information that is received from a variety of sources:

    • Kernel log messages, via kmsg
    • Simple system log messages, via the libc syslog call
    • Structured system log messages via the native Journal API.
    • Standard output and standard error of service units.
    • Audit records, originating from the kernel audit subsystem.

    The daemon will implicitly collect numerous metadata fields for each log messages in a secure and unfakeable way.

    Journald provides a good out-of-the-box logging experience for systemd. The trade-off is, journald is a bit of a monolith, having everything from log storage and rotation, to log transport and search. Some would argue that syslog is more UNIX-y: more lenient, easier to integrate with other tools. Which was its main criticism to begin with. When the change was made not everyone agreed with the migration from syslog or the general approach systemd took with journald. But by now, systemd is adopted by most Linux distributions, and it includes journald as well. journald happily coexists with syslog daemons, as:

    • Some syslog daemons can both read from and write to the journal
    • journald exposes the syslog API

    It provides lots of features, most importantly:

    • Indexing. journald uses a binary storage for logs, where data is indexed. Lookups are much faster than with plain text files.
    • Structured logging. Though it’s possible with syslog, too, it’s enforced here. Combined with indexing, it means you can easily filter specific logs (e.g. with a set priority, in a set timeframe).
    • Access control. By default, storage files are split by user, with different permissions to each. As a regular user, you won’t see everything root sees, but you’ll see your own logs.
    • Automatic log rotation. You can configure journald to keep logs only up to a space limit, or based on free space.

Syncthing⚑

  • New: Change the path of a folder.

    • Shutdown Syncthing
    • Edit the config file (~/.config/syncthing/config.xml)
    • Search and replace the path
    • Start again syncthing

Wireguard⚑

  • New: Improve logging.

    WireGuard doesn’t do any logging by default. If you use the WireGuard Linux kernel module (on kernel versions 5.6 or newer), you can turn on WireGuard’s dyndbg logging, which sends log messages to the kernel message buffer, kmsg. You can then use the standard dmesg utility to read these messages. Also, many Linux systems have a logging daemon like rsyslogd or journald that automatically captures and stores these messages.

    First, enable WireGuard dyndbg logging with the following commands:

    modprobe wireguard
echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

    Once you do that, you’ll be able to see WireGuard log messages in the kernel message facility, if your system is set up with rsyslogd, journald, or a similar logging daemon. With rsyslogd, check the /var/log/kern.log or /var/log/messages file. With journald, run journalctl -ek.

  • New: Monitor wireguard.

  • New: Check the status of the tunnel.

    One method is to do ping between VPN IP addresses or run command wg show`` from the server or from the client. Below you can seewg show`` command output where VPN is not up.

    $: wg show
interface: wg0
  public key: qZ7+xNeXCjKdRNM33Diohj2Y/KSOXwvFfgTS1LRx+EE=
  private key: (hidden)
  listening port: 45703

peer: mhLzGkqD1JujPjEfZ6gkbusf3sfFzy+1KXBwVNBRBHs=
  endpoint: 3.133.147.235:51820
  allowed ips: 10.100.100.1/32
  transfer: 0 B received, 592 B sent
  persistent keepalive: every 21 seconds

    The below output from the wg show command indicates the VPN link is up. See the line with last handshake time

    $: wg show
interface: wg0
  public key: qZ7+xNeXCjKdRNM33Diohj2Y/KSOXwvFfgTS1LRx+EE=
  private key: (hidden)
  listening port: 49785

peer: 6lf4SymMbY+WboI4jEsM+P9DhogzebSULrkFowDTt0M=
  endpoint: 3.133.147.235:51820
  allowed ips: 10.100.100.1/32
  latest handshake: 14 seconds ago
  transfer: 732 B received, 820 B sent
  persistent keepalive: every 21 seconds

Android⚑

GrapheneOS⚑

  • New: Disable Bluetooth and Wifi once it's disconnected.

    If you don't want to go spreading your SSIDs you can configure graphene to disable wifi and bluetooth X minutes after loosing connection.

    For Wifi:

    • Go to Settings > network & internet > internet > network preferences
    • Select Turn off Wi-Fi automatically, for example after 2 minutes

    For Bluetooth:

    • Go to Settings > connected devices > connection preferences> bluetooth
    • Select Turn Bluetooth off automatically, for example after 2 minutes

GadgetBridge⚑

Android SDK Platform tools⚑

  • New: Introduce android_sdk.

    Android SDK Platform tools is a component for the Android SDK. It includes tools that interface with the Android platform, primarily adb and fastboot.

    Installation

    While many Linux distributions already package Android Platform Tools (for example android-platform-tools-base on Debian), it is preferable to install the most recent version from the official website. Packaged versions might be outdated and incompatible with most recent Android handsets.

    • Download the latest toolset
    • Extract it somewhere in your filesystem
    • Create links to the programs you want to use in your $PATH

    Next you will need to enable debugging on the Android device you are testing. Please follow the official instructions on how to do so.

    Usage

    Connecting over USB

    To use adb with a device connected over USB, you must enable USB debugging in the device system settings, under Developer options. On Android 4.2 (API level 17) and higher, the Developer options screen is hidden by default.

    Enable the Developer options

    To make it visible, enable Developer options. On Android 4.1 and lower, the Developer options screen is available by default. On Android 4.2 and higher, you must enable this screen.

    • On your device, find the Build number option (Settings > About phone > Build number)
    • Tap the Build Number option seven times until you see the message You are now a developer! This enables developer options on your device.
    • Return to the previous screen to find Developer options at the bottom.

    Enable USB debugging

    Before you can use the debugger and other tools, you need to enable USB debugging, which allows Android Studio and other SDK tools to recognize your device when connected via USB.

    Enable USB debugging in the device system settings under Developer options. You can find this option in one of the following locations, depending on your Android version:

    • Android 9 (API level 28) and higher: Settings > System > Advanced > Developer Options > USB debugging
    • Android 8.0.0 (API level 26) and Android 8.1.0 (API level 27): Settings > System > Developer Options > USB debugging
    • Android 7.1 (API level 25) and lower: Settings > Developer Options > USB debugging

    Test it works

    If everything is configured appropriately you should see your device when launching the command adb devices.

    Create udev rules if it fails

    If you see the next error:

    failed to open device: Access denied (insufficient permissions)

* failed to start daemon
adb: failed to check server version: cannot connect to daemon

    It indicates an issue with permissions when adb tries to communicate with the device via USB. Here are some steps you can take to resolve this issue:

    • Check USB permissions
    • Ensure that you have the necessary permissions to access the USB device. If you're running on Linux, check if the device has appropriate udev rules.

    • You can try adding your user to the plugdev group:

      sudo usermod -aG plugdev $USER

    • Make sure you have a udev rule for Android devices in /etc/udev/rules.d/. If not, you can create one by adding a file like 51-android.rules:

      sudo touch /etc/udev/rules.d/51-android.rules

    • Add this line to the file to grant access to Android devices:

      SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0666", GROUP="plugdev"

    • Reload the udev rules:

      sudo udevadm control --reload-rules
sudo service udev restart

    • Unplug and reconnect the USB device. References

    • Home

ICSx5⚑

  • New: Introduce ICSx5.

    ICSx5 is an android app to sync calendars.

    References

Signal⚑

  • New: Use the Molly FOSS android client.

    Molly is an independent Signal fork for Android. The advantages are:

    • Contains no proprietary blobs, unlike Signal.
    • Protects database with passphrase encryption.
    • Locks down the app automatically when you are gone for a set period of time.
    • Securely shreds sensitive data from RAM.
    • Automatic backups on a daily or weekly basis.
    • Supports SOCKS proxy and Tor via Orbot.

    Migrate from Signal

    Note, the migration should be done when the available Molly version is equal to or later than the currently installed Signal app version.

    • Verify your Signal backup passphrase. In the Signal app: Settings > Chats > Chat backups > Verify backup passphrase.
    • Optionally, put your phone offline (enable airplane mode or disable data services) until after Signal is uninstalled in step 5. This will prevent the possibility of losing any Signal messages that are received during or after the backup is created.
    • Create a Signal backup. In the Signal app, go to Settings > Chats > Chat backups > Create backup.
    • Uninstall the Signal app. Now you can put your phone back online (disable airplane mode or re-enable data services).
    • Install the Molly or Molly-FOSS app.
    • Open the Molly app. Enable database encryption if desired. As soon as the option is given, tap Transfer or restore account. Answer any permissions questions.
    • Choose to Restore from backup and tap Choose backup. Navigate to your Signal backup location (Signal/Backups/, by default) and choose the backup that was created in step 3.
    • Check the backup details and then tap Restore backup to confirm. Enter the backup passphrase when requested.
    • If asked, choose a new folder for backup storage. Or choose Not Now and do it later.

    Consider also:

    • Any previously linked devices will need to be re-linked. Go to Settings > Linked devices in the Molly app. If Signal Desktop is not detecting that it is no longer linked, try restarting it.
    • Verify your Molly backup settings and passphrase at Settings > Chats > Chat backups (to change the backup folder, disable and then enable backups). Tap Create backup to create your first Molly backup.
    • When you are satisfied that Molly is working, you may want to delete the old Signal backups (in Signal/Backups, by default).

Arts⚑

Cooking⚑

  • New: Comer naranja por la noche.

    "Por la mañana oro, al mediodía plata y por la noche mata". Así de tajante se muestra el refranero español con respecto a la naranja y el melón. La naranja es una fruta ácida y por esta razón es recomendable evitar su consumo por la noche. Y es que este alto nivel de acidez puede provocar una digestión lenta y sensación de acidez y ardor en el estómago. Directamente relacionado con estos síntomas puede sobrevenir dolor de estómago y esto conducirnos a un descanso poco placentero durante la noche. Así, lo mejor es evitar su consumo a última hora del día. Si encima le añades unos trocitos de chocolate negro dificultarás el tránsito intestinal (por las grasas) y harás que te cueste más conciliar el sueño ya que es estimulante. Pero qué rico sabe de postre de cena xD.

    Estas consecuencias varían en función de las personas y de su sistema digestivo. Así, este alimento sí puede resultar saludable para ciertas personas sin problemas gástricos y un alto nivel de tolerancia a este alimento.

    Lo más aconsejable es consumirla por la mañana. Tomar una naranja a primera hora del día te aportará una buena dosis de vitamina C y fibra.

    Siempre es recomendable tomar la fruta directamente y evitar los zumos, aunque sean naturales. De esta forma nos sentiremos más saciados y añadiremos más fibra y menos azúcares a nuestra dieta.

    Además, la acidez de las naranjas que podría perjudicarnos por la noche es perfectamente tolerable por la mañana incluso si la tomamos en ayunas. De esta forma, podremos ir asimilando sus propiedades durante todo el día.

    Estas frutas tienen un gran efecto antioxidante. Además, su gran contenido en vitamina C es un refuerzo inigualable para el sistema inmunológico y ayuda a combatir la anemia. También es un remedio ideal para tratar los resfriados y las gripes.

  • New: Por qué no vale con quitar la parte del moho de los alimentos.

    Artículo

Cleaning⚑

Cleaning tips⚑

  • New: Cleaning car headlights.

    If you need to clean the car headlights you can use a mixture of one squeezed lemon and two spoonfuls of baking soda

Parkour⚑

  • New: Warming up.

    Never do static stretches if you're cold, it's better to do dynamic stretches.

    Take the joints through rotations

    • Head:
    • Nod 10 times
    • Say no 10 times
    • Ear shoulder 10 times

    • Circles 10 times each direction

    • Shoulders

    • Circles back 10 times

    • Circles forward 10 times

    • Elbows

    • Circles 10 each direction

    • Wrists:

    • Circle 10 each direction

    • Chest:

    • Chest out/in 10 times
    • Chest one side to the other 10 times

    • Chest in circles

    • Hips:

    • Circles 10 each direction

    • Figure eight 10 times each direction

    • Knees:

    • Circular rotations 10 each direction feet and knees together
    • 10 ups and downs with knees together

    • Circular rotations 10 each direction feet waist width

    • Ankles:

    • Circular 10 rotations each direction

    Light exercises

    • 10 steps forward of walking on your toes, 10 back
    • 10 steps forward of walking on your toes feet rotated outwards, 10 back
    • 10 steps forward of walking on your toes feet rotated inwards, 10 back
    • 10 steps forward of walking on your heels feet rotated outwards, 10 back

    • 10 steps forward of walking on your heels feet rotated inwards, 10 back

    • 2 x 10 x Side step, carry the leg up (from out to in) while you turn 180, keep on moving on that direction

    • 2 x 10 x Front step carrying the leg up (from in to out)while you turn 45, then side step, keep on moving on that direction
    • 10 light skips on one leg: while walking forward lift your knee and arms and do a slight jump
    • 10 steps with high knees
    • 10 steps with heel to butt

    • 10 side shuffles (like basketball defense)

    • 5 lunges forward, 5 backwards

    • 10 rollups and downs from standing position

    • 5 push-ups
    • 10 rotations from the pushup position on each direction with straigth arms
    • 5 push-ups
    • 10 rotations from the pushup position on each direction with shoulders at ankle level

    • 3 downward monkeys: from piramid do a low pushup and go to cobra, then a pushup

    • 10 steps forward walking on all fours

    Strengthen your knees

    Follow there steps

    Transit to the parkour place

    Go by bike, skate, jogging to the parkour place

Music⚑

Sister Rosetta Tharpe⚑

  • New: Introduce Sister Rosetta Tharpe.

    Sister Rosetta Tharpe was a visionary, born in 1915 she started shredding the guitar in ways that did not exist in that time. Yes, she founded Rock and Roll. It's lovely to see a gospel singer with an electrical guitar.

    In this video you'll be able to understand how awesome she ws.

    Videos:

Drawing⚑

Languages⚑

Castellano⚑

  • New: El agua o la agua?.

    El sustantivo agua es de género femenino, pero tiene la particularidad de comenzar por /a/ tónica (la vocal tónica de una palabra es aquella en la que recae el acento de intensidad: [água]). Por razones de fonética histórica, este tipo de palabras seleccionan en singular la forma el del artículo, en lugar de la forma femenina normal la. Esta regla solo opera cuando el artículo antecede inmediatamente al sustantivo, de ahí que digamos el agua, el área, el hacha; pero, si entre el artículo y el sustantivo se interpone otra palabra, la regla queda sin efecto, de ahí que digamos la misma agua, la extensa área, la afilada hacha. Puesto que estas palabras son femeninas, los adjetivos deben concordar siempre en femenino: el agua clara, el área extensa, el hacha afilada (y no el agua claro, el área extenso, el hacha afilado).

    Por su parte, el indefinido una toma generalmente la forma un cuando antecede inmediatamente a sustantivos femeninos que comienzan por /a/ tónica: un área, un hacha, un águila (si bien no es incorrecto, aunque sí poco frecuente, utilizar la forma plena una: una área, una hacha, una águila). Asimismo, los indefinidos alguna y ninguna pueden adoptar en estos casos las formas apocopadas (algún alma, ningún alma) o mantener las formas plenas (alguna alma, ninguna alma).

    Al tratarse de sustantivos femeninos, con los demostrativos este, ese, aquel o con cualquier otro adjetivo determinativo, como todo, mucho, poco, otro, etc., deben usarse las formas femeninas correspondientes: esta hacha, aquella misma arma, toda el agua, mucha hambre, etc. (y no este hacha, aquel mismo arma, todo el agua, mucho hambre, etc.)

Galego⚑

  • New: Add some galego vocabulary.

  • New: Introduce galego.

    O galego é unha lingua indoeuropea que pertence á póla de linguas románicas. É a lingua propia de Galiza, onde é falada por uns 2.4 millóns de galegas. Á parte de en Galiza, a lingua falase tamén en territórios limítrofes con esta comunidade, ainda que sen estatuto de oficialidade, asi como pola diáspora galega que emigrou a outras partes do estado español, América latina, os Estados Unidos, Suíza e outros países do Europa.

  • New: Te e che. Trucos para saber diferencialos.

    En galego temos dúas formas para o pronome átono da segunda persoa do singular: te e che.

    O pronome te ten a función de complemento directo (CD) e o pronome che de complemento indirecto (CI).

    Cando se utiliza o pronome te?

    O pronome te utilízase cando ten a función de CD, propio dos verbos transitivos, xa que alude ao ser ou ao obxecto sobre o que recae a acción verbal.

    Se convertemos a oración en pasiva, o CD pasa a ser o suxeito. Por exemplo:

    Vinte na cafetería / Ti fuches visto por min na cafetería.

    Cando se utiliza o pronome che?

    O pronome che utilízase cando ten a función de CI, xa que indica o destinatario da acción expresada polo verbo. Por exemplo:

    Díxenche a verdade.

    Compreiche unhas lambonadas.

    Truco para saber diferencialos

    Un truco moi rápido para diferenciarmos os pronomes te e che é substituír eses pronomes de segunda persoa polos de terceira.

    Se podemos cambiar ese pronome por o/lo/no ou a/la/na, quere dicir que o pronome vai ser de CD. Polo tanto, temos que poñer te.

    Saudeite onte pola rúa / Saudeino onte pola rúa.

    Chameite por teléfono / Chameina por teléfono.

    Se podemos substituílo por un lle, significa que é un pronome de CI e que debemos utilizar o che.

    Lévoche mañá os apuntamentos / Lévolle mañá os apuntamentos.

    Collinche as entradas do concerto / Collinlle as entradas do concerto.

  • New: Uso de asemade.

    Asemade pode utilizarse como adverbio cando ten o significado de ‘ao mesmo tempo’ ou ‘simultaneamente’. Ainda que normalmente úsase no registro culto, non utilizalo na fala.

    • Non se pode comer e falar asemade.
    • Non podes facer os deberes e ver a televisión asemade, pois non te concentras.

    Tamén se pode utilizar como conxunción co significado de ‘tan pronto como’.

    • Foi o primeiro que vimos asemade entramos.
    • Recoñecino asemade o vin.

    É incorrecto empregar asemade como sinónimo de tamén, ademais ou igualmente.

  • New: Referencias e libros de gramática.

    Referencias:

    Libros gramática:

Science⚑

Artificial Intelligence⚑

  • New: Add aider tool.

    • Aider lets you pair program with LLMs, to edit code in your local git repository. Start a new project or work with an existing git repo. Aider works best with GPT-4o & Claude 3.5 Sonnet and can connect to almost any LLM.

Whisper⚑

Coding by Voice⚑

  • New: Introduce Coding by voice.

    Coding by voice command requires two kinds of software: a speech-recognition engine and a platform for voice coding. Dragon from Nuance, a speech-recognition software developer in Burlington, Massachusetts, is an advanced engine and is widely used for programming by voice. On the platform side, VoiceCode by Ben Meyer and Talon by Ryan Hileman (both are for Mac OS only) are popular.

    Coding by voice platforms:

    Two platforms for voice programming are Caster and Aenea, the latter of which runs on Linux. Both are free and open source, and enable voice-programming functionality in Dragonfly, which is an open-source Python framework that links actions with voice commands detected by a speech-recognition engine. Saphra tried Dragonfly, but found that setup required more use of her hands than she could tolerate.

    All of these platforms for voice command work independently of coding language and text editor, and so can also be used for tasks outside programming. Pimentel, for instance, uses voice recognition to write e-mails, which he finds easier, faster and more natural than typing.

    To the untrained ear, coding by voice command sounds like staccato bursts of a secret language. Rudd’s video is full of terms like ‘slap’ (hit return), ‘sup’ (search up) and ‘mara’ (mark paragraph).

    Unlike virtual personal assistants such as Apple’s Siri or Google’s Alexa, VoiceCode and Talon don’t do natural-language processing, so spoken instructions have to precisely match the commands that the system already knows. But both platforms use continuous command recognition, so users needn’t pause between commands, as Siri and Alexa require.

    VoiceCode commands typically use words not in the English language, because if you use an English word as a command, such as ‘return’, it means you can never type out that word. By contrast, Talon, Aenea and Caster feature dynamic grammar, a tool that constantly updates which words the software can recognize on the basis of which applications are open. This means users can give English words as commands without causing confusion.

    In addition to voice recognition, Talon can also replace a computer mouse with eye tracking, which requires a Tobii 4c eye tracker (US$150). Other eye-mousing systems generally require both the eye tracker and head-tracking hardware, such as the TrackIR from NaturalPoint. “I want to make fully hands-free use of every part of a desktop computer a thing,” says Hileman. Other mouse replacements also exist; Pimentel uses one called SmartNav.

    Voice command requires at least a decent headset or microphone. Many users choose a unidirectional microphone so that others can talk to them while they are dictating code. One such mic, a cardioid mic, requires special equipment to supply power, and hardware costs can reach $400, says Pimentel.

    The software can cost several hundred dollars too. The speech-recognition engine Dragon Professional costs $300, as does VoiceCode. Caster and Aenea are free and open source. Talon is available for free, but requires a separate speech-recognition engine. A beta version of Talon that includes a built-in speech-recognition engine is currently available to Hileman’s Patreon supporters for $15 per month. 

    Whether or not users have RSI, it can be difficult and frustrating to start programming by voice. It took a month and a half for Pimentel to get up to speed, he says, and there were days when he was ready to throw in the towel. He printed out 40 pages of commands and forced himself to look at them until he learnt them. Saphra needed two months of coding, a little every day, before she felt that it was a “perfectly enjoyable experience and I could see myself doing this for a living”.

    After the initial learning curve, users often create custom prompts for commonly used commands as the need arises. 

Data Analysis⚑

Parsers⚑

  • New: Learning about parsers.

    Parsers are a whole world. I kind of feel a bit lost right now and I'm searching for good books on the topic. So far I've found:

    Pros: - Pleasant to read - Doesn't use external tools, you implement it from scratch. - Multiple format: EPUB, PDF, web - You can read it for free - Cute drawings <3

    Cons: - Code snippets are on Java and C - Doesn't use external tools, you implement it from scratch - It's long

    • Compilers: Principles, Techniques, and Tools by Aho, Alfred V. & Monica S. Lam & Ravi Sethi & Jeffrey D. Ullman

    Pros: - EPUB

    Cons: - Code snippets are on C++

    • Parsing Techniques: A Practical Guide by Dick Grune and Ceriel J.H Jacobs

    Pros: - Gives an overview of many grammars and parsers

    Cons: - Only in PDF - It's long - Too focused on the theory, despite the name xD

Other⚑

  • New: Inhibit rules between times.

    To prevent some alerts to be sent between some hours you can use the time_intervals alertmanager configuration.

    This can be useful for example if your backup system triggers some alerts that you don't need to act on.

    route:
  receiver: 'email'
  group_by: [job, alertname, severity]
  group_wait: 5m
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - receiver: 'email'
      matchers:
        - alertname =~ "HostCpuHighIowait|HostContextSwitching|HostUnusualDiskWriteRate"
        - hostname = backup_server
      mute_time_intervals:
        - night
time_intervals:
  - name: night
    time_intervals:
      - times:
          - start_time: 02:00
            end_time: 07:00

  • New: Thoughts on the reviews themselves.

    • Keep It Simple: It's important for the process to be light enough that you want to actually do it, so you see it as a help instead of a burden. It's always better to do a small and quick review rather than nothing at all. At the start of the review analyze yourself to assess how much energy do you have and decide which steps of the review you want to do.

    • Review approaches: In the past I used the life logging tools to analyze the past in order to understand what I achieved and take it as a base to learn from my mistakes. It was useful when I needed the endorphines boost of seeing all the progress done. Once I assumed that progress speed and understood that we always do the best we can given how we are, I started to feel that the review process was too cumbersome and that it was holding me into the past.

    Nowadays I try not to look back but forward, analyze the present: how I feel, how's the environment around me, and how can I tweak both to fulfill my life goals. This approach leads to less reviewing of achievements and logs and more introspection, thinking and imagining. Which although may be slower to correct mistakes of the past, will surely make you live closer to the utopy.

    The reviews below then follow that second approach.

    • Personal alive reviews: Reviews have to reflect ourselves, and we change continuously, so take for granted that your review is going to change.

    I've gone for full blown reviews of locking myself up for a week to not doing reviews for months.

    This article represent the guidelines I follow to do my life review. It may seem a lot to you or may be very simple. Please take it as a base or maybe to get some ideas and then create your own that fits your needs.

  • New: Update the Month review process.

  • New: When to do the trimester reviews.

    As with moth reviews, it's interesting to do analysis at representative moments. It gives it an emotional weight. You can for example use the solstices or my personal version of the solstices:

    • Spring analysis (1st of March): For me the spring is the real start of the year, it's when life explodes after the stillness of the winter. The sun starts to set later enough so that you have light in the afternoons, the climate gets warmer thus inviting you to be more outside, the nature is blooming new leaves and flowers. It is then a moment to build new projects and set the current year on track.

    • Summer analysis (1st of June): I hate heat, so summer is a moment of retreat. Everyone temporarily stop their lives, we go on holidays and all social projects slow their pace. Even the news have even less interesting things to report. It's so hot outside that some of us seek the cold refuge of home or remote holiday places. Days are long and people love to hang out till late, so usually you wake up later, thus having less time to actually do stuff. Even in the moments when you are alone the heat drains your energy to be productive. It is then a moment to relax and gather forces for the next trimester. It's also perfect to develop easy and chill personal projects that have been forgotten in a drawer. Lower your expectations and just flow with what your body asks you.

    • Autumn analysis (1st of September): September it's another key moment for many people. We have it hardcoded in our life since we were children as it was the start of school. People feel energized after the summer holidays and are eager to get back to their lives and stopped projects. You're already 6 months into the year, so it's a good moment to review your year plan and decide how you want to invest your energy reserves.

    • Winter analysis (1st of December): December is the cue that the year is coming to an end. The days grow shorter and colder, they basically invite you to enjoy a cup of tea under a blanket. It is then a good time to get into your cave and do an introspection analysis on the whole year and prepare the ground for the coming year. Some of the goals of this season are:

    • Think everything you need to guarantee a good, solid and powerful spring start.
    • Do the year review to adjust your principles.

    The year is then divided in two sets of an expansion trimester and a retreat one. We can use this information to adjust our life plan accordingly. In the expansion trimester we could invest more energies in the planning, and in the retreat ones we can do more throughout reviews.

  • New: The principle documents.

    Principle documents for me are orgmode documents where I think about the principle itself. It acts both as a way of understanding it and evolving my idea around it, and to build the roadmap to materialize the principle's path.

    Without ever having created one I feel that it makes sense to make the reflection part public in the blue book, while I keep for myself the private one. This may also change between principles.

  • New: The life path document.

    The life path document is an orgmode document where I think about what I want to do with my life and how. It's the highest level of abstraction of the life management system.

    The structure so far is as follows:

    * Life path
** {year}
*** Principles of {season} {year}
    {Notes on the season}
    - Principle 1
    - Principle 2
    ...

**** Objectives of {month} {year}
     - [-] Objective 1
       - [X] SubObjective 1
       - [ ] SubObjective 2
     - [ ] Objective 2
     - [ ] ...

    Where the principles are usually links to principle documents and the objectives links to tasks.

  • New: Trimester prepare.

    The trimester review requires an analysis that doesn't fill in a day session. It requires slow thinking over some time. So I'm creating a task 10 days before the actual review to start thinking about the next trimester. Whether it's ideas, plans, desires, objectives, or principles.

    Is useful for that document to be available wherever you go, so that in any spare time you can pop it up and continue with the train of thought.

    Doing the reflection without seeing your life path prevents you from being tainted by it, thus representing the real you of right now.

    On the day to actually do the review, follow the steps of the Month review prepare adjusting them to the trimester case.

  • New: Update the month planning.

    Decide the month objectives:

    Create the month objectives in your roadmap file after addressing each element of:

    • Your last month review document.
    • The trimester objectives of your roadmap.

    Once they are ready, copy them to the description of your todo.org file. That way you'll see it every day.

    For each of your month objectives:

    • Decide whether it makes sense to address it this month. If not, archive it
    • Create a clear plan of action for this month on that objective
    • Define the todo of each device (mobile, tablet, laptop)
    • Tweak your things to think about list.
    • Tweak your reading list.
    • Tweak your week distribution (what to do in each day)
    • If you selected maintenance week days tweak the priorities on your maintenance list.
    • If you selected improvement week days tweak the priorities on your improvements list.
    • Tweak your habit manager system.

  • New: Use deadlines.

    Identify hard deadlines: Add a warning days before the deadline to make sure you're reminded until it's done.

  • New: Add 2024 Hidden Cup 5 awesome match.

    Semifinal Viper vs Lierey

  • New: Prometheus metrics.

    Use prometheus-fastapi-instrumentator

  • Correction: Tweak the month planning.

    Add the next steps:

    • Clean your agenda and get an feeling of the busyness of the month:
    • Open the orgmode month view agenda and clean it
    • Read the rest of your calendars

    Then reorder the objectives in order of priority. Try to have at least one objective that improves your life.

    • For each of your month and trimester objectives:
    • Decide whether it makes sense to address it this month. If not, mark it as inactive

    • Create a clear plan of action for this month on that objective.

      • Reorder the projects as needed
      • Mark as INACTIVE the ones that you don't feel need to be focused on this month.

    • Refine the roadmap of each of the selected areas (change this to the trimestral planning)

    • Select at least one coding project in case you enter in programming mode
    • Clean your mobile browser tabs

  • Correction: Deprecate not-by-ai.

    As they have introduced pricing, which makes no sense, and we had a discussion that using that badge it's a nice way to tell the AI which content to use and which not to

  • Reorganization: Into roadmap_adjustment.