←
Michael McGreal

Michael McGreal

Highlight lines with comments

The frustrations of code sytax blocks

When embedding code blocks in articles, it's useful to be able to highlight specific lines, blur or de-emphasize irrelevant lines, or even initialize some code blocks in a collapsed state.

These simple features can drastically increase the quality of reading a blog. For context, here is a superb article visualizing best practices for presenting code.

Highlight lines 3-4:

1console.log('line 1')
2console.log('line 2')
3console.log('line 3. highlight me.') 
4console.log('line 4. highlight me too.') 
5console.log('line 5')

Focus lines 3-4, blur the rest:

1console.log('line 1')
2console.log('line 2')
3console.log('line 3. highlight me.') 
4console.log('line 4. focus me too.') 
5console.log('line 5')

Collapsable sections:

(Click the "..." in the code block to expand)

 1console.log('some code.')
 2const collapsableSection = () => {
 3 ...
 4  console.log('long section')
 5  console.log('long section')
 6  console.log('long section')
 7  console.log('long section')
 8  console.log('long section')
 9  console.log('long section')
10  console.log('long section')
11  console.log('long section')
12  console.log('long section')
13  console.log('long section')
14  console.log('long section')
15  console.log('long section')
16  console.log('long section') 
17}
18 
19console.log('more code.')

And diffs!:

1console.log('line 1')
2console.log('line 2')
3console.log('line 3. remove me.') 
4console.log('line 4. add me.') 
5console.log('line 5')

However, after intense research, these features are rare.

Many packages support some but not all.

And the ones that do support them are implemented in non-flexible ways.

For example, most packages require you to specify exact line numbers as a prop, which requires constantly updating them as your code evoles. The ability to specify highlighting meta data as inline comments is brilliant! And retains part of what torchlight accurately describes as the "Joy of Authoring".

I've read many code tutorials that show small snippets of code without any context. Sometimes it's fine, but when trying to explain how a complicate program connects, by using small snippets, the author is asking the reader to make a lot of discrete connections. However, what are their options? The alterante of pasting 100 lines of code, just to show 2 new lines is undesireable too.

This can sometimes be solved by 'diff' viewers, such as the ones used by github or gitlab, to visualize changes between code on pull or merge requests. And there are some useful packages to help implement these in an article, such as 'diff2html'. But these packages have their own problems.

Some issues with common syntax highlighters:

  • Require pasting code as a string, which is difficult to maintain as code evoles. e.g.:
    • Changing a function that is used in multiple snippets requires updating each snippet
    • Prettier doesn't format code well when it's in a string. So you'll have to format it elsewhere before pasting.
  • Many syntax highlighting tools don't support emphasizing lines at all. The ones that do, require a 'lineNumber' prop which is inflexible as code evolves. I.e. adding a line will shift all your line numbers by 1, requiring you to update each components 'lineNumber' prop.
    • Since highlighting lines in general is rare, other features such as 'de-emphasizing non-highlighted lines' or 'collapsable code blocks' are even more rare

What about diff UI tools?

  • Don't git review tools like github and gitlab already have beautiful looking UI's? (i.e. when reviewing a pull or merge request). They even offer "+" and "-" buttons to expand/collapse surrounded lines of code, to reveal context. And when showing a diff, they stylize individual words in addition to individual (so it's clear which word in the line changed). You can move blog comments from a section on the bottom, to inline as contextual PR comments! Some even allow jumping around the code by command clicking declarations, similar to Vscode (but this is getting into codesandbox territory, at which point a different category of tools becomes worth exploring, such as codesandbox sandpack)
  • But the primary problem is that these wonderful git review tools don't offer their UI as a package.
  • the closest they offer are git gists and gitlab snippets for example.
  • These have some benefits such as automatic (and high quality) syntax highlighting for a multitude of lanaguages. Some even allow emphasizing specific lines! For example, to style a specific line as yellow background in git gists, you can click the line number in the UI, or simply append something like '#L2L5' to the URL, which will tell git gist to highlight lines 2 through 5. However, none have features to highlight multiple segments of lines, i.e. '#L2L5_L8L11'.
  • Further, they suffer similar limitations to other syntaz highlighting packages, such as: you still have to paste code from your editor, and it should be pre-formatted. You have to manage a bunch of external urls now. And if the code evolves, each snippet has to be updated (their line numbers too). There are some great vscode extensions that make publishing and managing git gist urls easier, such as GistPad, however, the other problems remain.

There has to be some git diff package though right?

  • Yes! There are a few. For example, diff2html and react-diff-viewer-continued
  • They even allow passing .diff or .patch files, instead of pasting code as strings!
  • However, each has their own limitations too (of course)
  • Challenges such as server-side-rendering, lack of UI features such as expanding lines, non-mobile friendly side-by-side layouts, etc.
  • some have some, but not all. And you still have to generate the .diff
  • And showing all your code as diffs is not enjoyable to readers either. A mix of diffs and simple code blocks would be best.

Generating an article from git

  • After all of this research, I had an idea...
  • I want to be able to write articles with ease. I don't want to have to worry about:
    • formatting with prettier
    • a change in 1 snippet requiring editing all previous snippets
    • manually managing a bunch of snippet urls
    • manually inputting emphasise lineNumber props which will likely change
    • installing additional packages
    • it should not require any external databases or web app interfaces. Everything should be doable from within vscode.
  • I just want to walk a user through code as it changes. For example, show the code broken. Then show them how to fix it, step by step. As if they were watching me in vscode, except more structured and with a summarized text at each commit.
  • Wait a second... can I generate an article from git?!
  • Actually, this would be a superb UX! To write an article, I could just code in vscode and commit with messages.
  • The commit message would become the paragraph text shown above each code snippet. Gitlab already has a commits page, showing each commit message in a vertical list, however, it requires opening each commits page to view it's diff. Rather, in an article format, the git diff should be shown directly inline, below the message.
  • Then, I would get the exact github or gitlab pull or merge request UI that we all know and love! But, since these tools don't allow using their UI as a package, I would be re-directing users off my blog and onto github or gitlab. Or embedded those urls, which is not the best UI, and loses SEO benefits.
  • I tried this first, using the github REST API, and then their oktokit js client, and found that it's diff generation method only returned a few lines of code surrounding the changes (minimal context). Additionally, it required users to upload their code to a cloud repository.
  • So, I changed my approach to using a command line tool that used the git command directly (which can return full length .diffs, and stores all code local-first).
  • But without using gitlab or github review UI, a major benefit had been lost, and I was back to using git diff packages.
  • However, this tool could still be useful to at least generate the structure of an article as a json file that could be passed to another git diff package. (or it could handle this for the user and simply return them a url to a hosted article page)
  • The idea of using git for this was not wilted yet though! It could still satisfy my other core problems:
    • To edit the paragraph text, I just edit the commit message!
    • If I make a mistake later on, I edit the code, and rebase.
  • well, already some problems are appearing with this solution
    • Editing the last commit message is easy. Editing further back requires re-writing git history using some git ammend commands that are not known by most beginners
    • Futher, editing a code from an earlier commit and having it update all future commits is doable with a git rebase, but again this is an advanced git feature. (and more importantly, it generates new commit sha's which became a problem)
    • what if I commit 3 files, but only want to show 1 in the article?
    • what if I want to intialize a long file as collapsed, so it doesn't consume 3000px height of the article?
    • what if I want to stylize some parts of the commit message, like using itlaics?
  • One idea would be similar to markdown files, where metadata is stored at the top of the file, and parsed using a tool like front-matter. But this defeats a huge goal of the tool which is to not install additional packages. If I just want to share a quick article about some small topic, I don't want to setup front-matter.
  • Okay, could this metadata be stored in the git commit itself? That could work! But git commit messages are not parsed for type safety. I guess they could be with some package, but surely, this can't be the best solution.
  • It became clear that the easiest way to achieve this was likely using a vscode extension. Yes, it's installing something, but it's a lot easier than installing a package.
  • I started hacking away at a vscode extension that displayed commit messages in a TreeView. Each commit message would have a buttons: 1) edit commit message (which would git ammend on their behalf); 2) exclude from article (i.e. to exclude a long yarn.lock file); 3) re-arrange (i.e. set order number); 4) initializeAsCollapsed; 5) edit file (which would git rebase on their behalf)
  • The settings would be saved as an article.config.json file that is committed to the repository.
  • When ready, user could invoke a vscode command pallete to generate the article, with options to output as json, HTML, or hosted static html page on vercel.
  • I first wrote this article config generator script as bash files, using the jq library to map over data structures
  • later, I rewrote it in javascript using { spawn } form the node module child_process. This was more verbose, but more maintainable, and easier to debug.
  • Then, with the help of chatGPT, I got a basic vscode extension working.
  • The article config file was beautiful! And mapping over the article-output.json on my blog site was easy.
  • However, even with a great VScode extension, this method of generating articles as problems:
    • writing article text and coding are 2 different mindsets. Crafting article text in git commit messages were sloppy. I had to edit each one later. And when I edited them, I often found changes to the code I wanted to make.
    • editing files in old commit and triggering a rebase worked – but with one detrimental problem: it (of course) changed all the git commit shas! And since my article-config was using commit shas as a primary id, this required completely refactoring the article config. I then had to map new commit to old commits using custom ids, and it was messy.
    • Furthermore, even if this vscode extension was perfected – it is still affected by all the other limitations of diff tools. Without customization by the developer, it's cannot show a mix of simple syntax highlighted code-blocks and git diffs – it's 'all-or-nothing'. The git diff features were still limited by the git diff package (such as diff2html or react-diff-viewer continued) – this tool was simply an easier way to generate and maintain .diffs. But as it turns out re-writing git history is not as simple as it seems.

After immense research (dev tools inspecting every code block I found!), I finally found torchlight.dev, via a backlink on the shiki highlighters's github.

It has all the features I imagined and more. The problem? It's mostly built for non javascript languages. There are some beta clients for a node-CLI and remark plugin, but they had issues too.

Contributing to issues in torchlight

It is remarkable that this package is so unknown by the javascript ecosystem. The reason is clearly because the working clients are more php and laravel focused. The node-cli is beta, but works in some cases (it causes issues with nextjs server side rendering, specifically related to hydration mismatches). And there is a beta remark-torchlight package which has a known bug, but is workable. Actually, all the code snippets you see on this site are using it with the workarounds.

So it is exciting to find such a high quality package, that is unknown, and specifically needs javascript contributors!

← Back to home