Skip to content

Blog Entries

Welcome! This page lists all my technical articles, notes, and findings.

Subscribe via RSS

Here are the latest entries:

The Long-Term Damage Windows Has Done to the Tech Industry

Production Runs on Linux. Development Should Have Too.

Here's a fact that should make every tech executive uncomfortable: virtually all production infrastructure today runs on Linux. Cloud servers, containers, Kubernetes clusters, CI/CD pipelines, embedded systems, networking equipment, supercomputers — Linux, all the way down. Yet for decades, the industry trained its developers on Windows.

That mismatch has cost us enormously.

A Generation Trained on the Wrong OS

For roughly 25 years (mid-1990s through the mid-2010s), the default development environment in most companies and universities was Windows. Developers wrote code on Windows, tested on Windows, and then deployed to Linux servers where things behaved differently. This created an entire class of bugs and inefficiencies that simply shouldn't exist:

  • Path separators and case sensitivity — Windows uses backslashes and case-insensitive filenames. Linux uses forward slashes and is case-sensitive. How many production bugs have been caused by this mismatch alone? Too many to count.

  • Line endings — CR+LF vs LF. Decades of tooling, git configs, and workarounds for a problem that only exists because developers use a different OS than production.

  • Shell scripting illiteracy — Windows developers grew up with CMD and later PowerShell, neither of which translates to the Bash/POSIX shell that runs every production script, Dockerfile, and CI pipeline. This created a skills gap that persists to this day.

  • Permission models — Windows ACLs and Linux POSIX permissions are fundamentally different. Developers who never used Linux often don't understand file permissions, ownership, or the principle of least privilege as implemented in production systems.

  • Process management — Signals, daemons, systemd, cgroups, namespaces — the building blocks of modern containerization — are all Linux concepts that Windows developers had to learn from scratch when the industry moved to Docker and Kubernetes.

The Cultural Damage

Beyond technical skills, Windows dominance created a cultural problem. It taught developers that:

  • GUIs are primary, CLIs are secondary. In production, it's the opposite. You SSH into servers. You write automation scripts. You read logs with grep, awk, and sed. The GUI-first mindset made developers less effective at operations.

  • You don't need to understand the OS. Windows actively hides its internals. Linux exposes everything as files and processes. The Windows mindset of "don't worry about what's underneath" produces developers who can't debug production issues because they never learned how an OS actually works.

  • Proprietary formats are normal. The Windows ecosystem normalized closed formats, closed protocols, and vendor lock-in. This slowed adoption of open standards and made interoperability harder than it needed to be.

The Tooling Tax

The industry spent enormous effort building bridges between the Windows development world and the Linux production world:

  • Vagrant and later Docker Desktop for Windows — entire projects that exist primarily to let Windows developers run Linux environments locally.

  • WSL (Windows Subsystem for Linux) — Microsoft itself eventually admitted the problem by embedding Linux inside Windows. Think about that: the solution to developing on Windows was to run Linux inside it.

  • Cross-platform build systems — CMake, various CI abstractions, and countless Makefiles with Windows-specific branches. Complexity that exists solely because development and production environments didn't match.

  • Cygwin and MSYS — heroic efforts to bring POSIX tools to Windows, used by millions of developers who needed Unix tools but were stuck on Windows machines.

The Wasted Years

Universities taught computer science on Windows for years. Students graduated without knowing how to use a terminal effectively, how to write a shell script, or how Linux package management works. Their first job required all of these skills.

Companies then spent months onboarding these developers into Linux-based production environments. Senior engineers became full-time translators, explaining Linux concepts that would have been obvious had the developers learned on Linux from the start.

What We Should Learn From This

The lesson isn't "Windows is bad" — it serves its purpose for desktop users, gamers, and certain enterprise workflows. The lesson is:

Your development environment should match your production environment.

This principle, so obvious in hindsight, was ignored for decades because of market momentum, licensing deals with universities, and the assumption that the OS you develop on doesn't matter. It does. It always did.

Today, the industry is finally converging. Linux desktops are viable for developers. macOS provides a Unix-like environment. WSL exists for those who stay on Windows. Cloud-based development environments run Linux natively. New developers are more likely to encounter Linux early.

But let's not forget the cost. Decades of reduced productivity, entire categories of bugs that shouldn't have existed, a generation of developers who had to relearn fundamental skills, and billions of dollars spent on tooling to bridge a gap that was self-inflicted.

The tech industry chose the wrong default OS for developers, and we're still paying for it.

MkDocs with GitHub Pages: File Layout That Works

If you use MkDocs to build a site hosted on GitHub Pages, and you also have static files (HTML, JS, CSS) that aren't part of the blog, getting the file layout right can be tricky. Here's what I learned.

The Problem

MkDocs wipes its output directory (site_dir) on every build. If you put your static files directly in docs/ (the default GitHub Pages root), mkdocs build deletes them.

The Solution

Put everything in the MkDocs source directory (docs_dir). MkDocs copies non-Markdown files through as-is.

My mkdocs.yml:

docs_dir: "blog"
site_dir: "docs"

My layout:

blog/               # MkDocs source (docs_dir)
  index.md          # Blog home page
  about.md
  posts/            # Blog posts (Markdown)
  media.html        # Static HTML page (passed through)
  calendar.html     # Static HTML page (passed through)
  keys.js           # Static JS (passed through)
  data/             # Static data files (passed through)
docs/               # MkDocs output (site_dir) - don't edit manually

On mkdocs build, everything in blog/ ends up in docs/. Markdown files get rendered with the theme. HTML, JS, CSS, and other files are copied unchanged. GitHub Pages serves docs/.

Key Points

  • Never manually edit files in docs/ — they'll be overwritten on next build.
  • Put all static assets in blog/ alongside your Markdown.
  • Add a .nojekyll file in blog/ to prevent GitHub from running Jekyll.
  • Reference static pages in nav without a leading slash:
nav:
  - 'Home': 'index.md'
  - 'Media': 'media.html'
  - 'Calendar': 'calendar.html'

Using a leading / makes MkDocs treat the path as an external URL and it won't validate the file exists.

How to upgrade Ubuntu without their upgrade tool

The upgrade problem

The heart of the problem is that sometimes when you try to upgrade ubuntu the upgrade fails. This happened to me when trying to upgrade to plucky (25.04). The tool would just fail and I tried waiting it out hoping that ubuntu will solve the bug. No such luck. Finally I decided to upgrade it myself manually and it worked like a charm.

The manual upgrade solution

Sync up

The first thing you need to do is sync up with the previous release:

$ sudo apt update
$ sudo apt dist-upgrade

Disable third party repos

The next thing is to manually disable any non ubuntu source of packages from /etc/apt/sources.list.d. I usually just create a folder called /etc/apt/sources.list.moved and move all but ubuntu there.

Setup the ubuntu source to the new distribution

update /etc/apt/sources.list.d/ubuntu.sources to the following content (replace your distro name):

Enabled: yes
Types: deb
URIs: http://us.archive.ubuntu.com/ubuntu
Suites: plucky plucky-updates plucky-security plucky-backports
Components: main restricted universe multiverse
Architectures: amd64
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Upgrade and solve all issues

$ sudo apt update
$ sudo apt dist-upgrade

You will need to solve issues along the way but they are standard things.

Reboot

And that's it.

Move from Google-Chrome to Firefox on Linux

The problems of Google-Chrome

There are several issues with Google-Chrome, some specific to Linux some not

  • Google-Chrome is spying on you and sends way too much information to google and advertisers.
  • Google-Chrome is using way too much CPU on Linux and it's responsiveness is much worse than that of Firefox. I've actually seen this in actual touch typing sites.
  • Google-Chrome is writing too much and wears out your disk in Linux. This is a known issue https://unix.stackexchange.com/questions/438456/google-chrome-high-i-o-writes

The result of all of this is that I recommend Firefox on Linux rather than Google-Chrome.

I wrote a script called browser_move_to_firefox.sh where you can see all the configs that need to be changed when moving to a different browser.

Problems with Netflix web and Netflix webos clients

I've had some issues with the Netflix service recently.

Here is my grievance list:

  • The UI is too intrusive, starts preview of videos/shows when you are just browsing. Cannot turn this behaviour off. Video plays from the start even though it clearly shows that the video is in mid viewing. Very annoying since I have to find the right position again.
  • Items disappear from “My List” with no heads up warning. Very annoying. This is sometimes because Netflix remove shows from the platform which is also annoying and what’s more – they don’t clearly state what is going to go away and when inside the app. I have to go online and find out for myself.
  • Things disappear from the “Continue watching as…” list with no heads up. Very annoying and forces me to maintain my own list of stuff I’m in the middle of watching. Sometimes this happens because a show is going off the platform (again, no heads up) and sometimes for no apparent reason at all.
  • The UI does not allow me to store more than one list. I need one for things I’ve seen and for things I want to see as well as things I’m in the middle of seeing (see above why Netflix support for studd you are in the middle of watching is terrible).
  • The site doesn’t provide an API for getting your data from Netflix. This may be a problem shared by a small minority of programming inclined users but it is important to me.

These problems are endemic to Netflix in general not just to a certain Netflix app or it’s website, so they cannot be solved at the application level. Netflix really needs to fix core issues to make progress on any of these issues.

As a result of all this I decided to leave Netflix. Bye bye.

Open heart surgery on a Fatar StdioLogic SL880

This one is for all of you who have a Fatar keyboard of version StudioLogic SL880 or similar ones. If one of your keys stops working and slumps down it may be that an inner plastic has broken in which case you will need to either send it to the shop or do surgery on it. This one is for the brave of heart who want to take the surgery road. Why should you do it? Because you are brave, because you don't want to haul the heavy keyboard to an expensive lab to fix it for lots of money. In any case the idea is to get a plastic from one of the unused keys (I used the lowest notes) and put it instead of the broken one on the broken notes. One piece of advice: no fear - and read the entire guide before starting!. Photos were taken using my iPhone and you can click on them to get a more detailed image.

Here are the stages:

First gut out the keyboard. You'll have to open 6 deep screws (hidden in trenches), 3 on either side at the bottom of the case. It's hard but it's doable. I have also released 6 more screws at the bottom and gutted the keyboard totally. You really don't have to do that but I wanted to clean the inside while I'm at it.

The SL-880 case

The keyboard gutted — case with PCB exposed

Now find the key(s) that cause(d) the problem. You need to use a small flat screwdriver in order to free the keys. Just insert the screwdriver into the back of the key and press on the small plastic. Once it's pushed the key could be pulled upwards and released. You will now see the problem.

The broken green plastic piece — and a good one for comparison

Close-up of the broken plastic

In order to fix the problem you will have to release all keys!. Yes - I know this hurts but there is a long steel rod that runs through all of them. As long as the keys are clicked into place they apply pressure on the rod and you will not be able to pull it out or, if you happen to pull it out, to get it back in again. So, release all the keys with the screw driver as before. You can either put them on the side or keep them in their place. I started with the former and ended up with the latter since it is better. Since you will be releasing all the keys this is your chance to clean the keys as well.

Keys released — the internal mechanism exposed

Another angle of the exposed mechanism

Close-up of the key mechanism

During the whole process watch out for the small springs. Each key has one and the spring is not held by anything once you release the keys...

Now you will get to a situation where there is no iron bar for the key you want to work on...

Keys with the steel rod visible

Close-up showing the key labels

Get the bad plastic out and put in a good piece of plastic from an unused key. I used the bottom most notes.

Keys close-up — the green plastic holders

Another view of the key mechanism

Some keys on the side. I pulled out a couple only to realize that it is better to keep them in place to avoid having to reconstruct exactly where each key goes. In any case, if you do pull them out, it is not a big deal since the keys are all numbered. White keys are "A B C D E F G" and black ones are numbered "1 2 3 4 5" and stand for C#, D#, F#/Gb, Ab, Bb. It looks like the black keys are interchangeable so you their numbers are not as important as those of the white keys. The ends of the keyboard have special keys. Keep an eye on those.

Removed keys and screwdriver on a table

Another view of the removed keys

Keys leaning with the steel rod pulled out

Another angle of keys with rod

If you do decide to gut out the keyboard completely by removing the extra set of 6 screws at the bottom then you will be able to clean the case itself. If you decide on this remember to release the keyboard only after you disengage the 4 data cables (two fat, two thin) that connect the keyboard to the case. Here is an image of the case after the cleanup...

The cleaned case with PCB

The SL-880 label on the back

The whole procedure took me about 3 hours and some. Well worth it.

more links about Fatar fixes: bad sounds electronics (original link dead) hardware issues (original link dead)

The official owners guide (from my site): fatar-sl880.pdf

Reviews of the Fatar SL-880: Harmony Central (original link dead)

Java runtime environment control

There are four ways to control Java environment for runtime:

- _JAVA_OPTIONS environment variable.
- Command line when running the java virtual machine.
- Java source code. In this case you must make sure to set the option before it is picked up by whatever subsystem it is intended for.
  • In Java web start you can also use the JNLP file to control the environment passed over to the executing JVM.

Examples of them can be:

  • export _JAVA_OPTIONS='-Dawt.useSystemAAFontSettings=lcd'
  • java -Dawt.useSystemAAFontSettings=lcd [arguments...]
  • System.setProperty("awt.useSystemAAFontSettings","lcd");
  • property name="awt.useSystemAAFontSettings" value="lcd" (under the resources element)

Each of these methods naturally has it's own advantages and disadvantages. In Java web start you have a hard time controlling the environment variables or the command line but two options (the JNLP file and the source code itself) are still open to you.

Some properties, like the anti-aliasing option, is notoriously bad by default and setting it (as shown above) will give you much better look and feel.

The values of the awt.useSystemAAFontSettings key are as follows:

  • false corresponds to disabling font smoothing on the desktop.

  • on corresponds to Gnome Best shapes/Best contrast (no equivalent Windows setting).

  • gasp corresponds to Windows Standard font smoothing (no equivalent Gnome desktop setting).

  • lcd corresponds to Gnome's subpixel smoothing and Windows ClearType.

What is the best option to choose? Well - I really don't know. On my laptop lcd looks best. Let me know about your own experience...

Debugging shared library problems

A tip: sometimes you install stuff from source and library search order makes analyzing which library you are actually using a mess. A useful tool is ldconfig -p that will print the cache of the dynamic linker for you allowing you to understand which libraries are actually being used.

Using gpg-agent to write authenticating scripts

Sometimes you want to write a shell or other script, and that script is going to have to run under sudo. Under such conditions if the script does anything that requires authentication it will not act as expected. In plain terms it means that the regular popup for authentication will not appear. The tool maybe written in a way which deals with the problem and falls back on other authentication methods, and yet it may not. In any case what you really want is for your own authentication agent (the little program called gpg-agent which is running on almost every Linux distribution from the time you log in till the time you log out) will do the authentication. This saves you lots of clicking. Imagine that the script has to do something which requires authentication X number of times. If the script does not use an agent it will not be able to cache the pass-phrases and so you will have to retype the pass-phrase several times. It can also be the case that your authenticating agent already has your pass-phrase in it's cache and you can save typing it yet another time.

Ok. So how do you do it? Well, in your original environment you have a variable called GPG_AGENT_INFO. This variable holds the details of how to connect to your authenticating agent. If you are running regular scripts then this variable, which is an environment variable, is automatically available to them. But if you run your scripts via ssh or sudo then it is not. Just make the variable available to those scripts. Obviously the users that these scripts will be running under will have to have the right level of permission to talk to your gpg agent. How do you make them available? One way is to pass this variable over the command line and turn it into an environment variable as soon as the script starts.

Producing MySQL dates from Perl

Ever written the occasional Perl script and wanted to insert the current date and time into a MySQL database? Here is the function to do it. This works for a column of type 'datetime'.

# function to return the current time in mysql format
sub mysql_now() {
        my($sec,$min,$hour,$mday,$mon,$year,$wday, $yday,$isdst)=localtime(time);
        my($result)=sprintf("%4d-%02d-%02d %02d:%02d:%02d",$year+1900,$mon+1,$mday,$hour,$min,$sec);
        return $result;
}