Steeph's Web Site

Go To Navigation
Show/Hide Navigation

Entries tagged 'cat:Bash'

SBWG 0.12.8

I've recently published a new version of SBWG, my web site generating bash script. The time that I'm able to allocate to working on SBWG fluctuated a lot in the last year and when I do work on it, I often just feel like doing one type of thing. So, when that is starting a new feature, all the other types of tasks can take on a pile that takes months to resolve. But eventually I managed to test and stabilise the features that I've added and changed, edited the README file and tried out the new version with real web sites. Well, testing could be more professionel, but it's okay for a hobby project, I think.

It's been well over a year since I wrote a blog entry about what's new in SBWG. This entry goes through the main new things since version 0.11.1.

A big new thing, at least from the view of the author/me, is that certain options where that makes sense now support multiple arguments. For example if you've edited three entries and want to re-generate just those three, you don't have to run the script three times but rather define the entries like this: sbwg -e ENTRYNAME_1 ENTRYNAME_2 ENTRYNAME_3. The same goes for pages, tagpages, entry attachments and galleries. This doesn't just make it easier and quicker to generate just some changed pieces of content. It also allows the usage of shell globbing (like *, ? and […]) and brace expansion ({…}). For example you can now regenerate all entries that are stored in one subdirectory or all entries whose names start with a vertain string of characters. Another practicle use case is to add external entries and/or to the site that are not stored in the respective directories in the web site's input directory. Using option -E/-P on the contents of an entire directory creates HTML files in the web site's output directory that look like any other of the web site's pages, but without integrating them in the structure of categories and other tags.

There is now a way to create a custom menu in the navigation bar without writing any code in the settings file of a web site. By using the new 'menu:' tag type in the header of a page or entry source file, the page or entry will get added to the menu. This allows for a list of pages you want to link to from the site's navbar, or a nested tree of interesting blog entries. The tag can be used similarly to the 'topic:' tag with the main difference that it doesn't add the entry or page to a list of entries with that topic, but rather directly in the navigation bar. In the default style set that is a drop down menu like the list of categories, authors, languages and topics. But it is just an unordered list, so it can be styled like any web site menu.

The default style set has changed a bit over the year. It basically looks the same but it's a bit cleaner now and is split into more file more logically. It will become even cleaner in the future though. It now also makes use of the new possibility to present the list of categories in the navigation bar in the form of a tag cloud. Audio attachments are better to look at now, especially when there are several audio files attached to an entry. Image attachments are can now be previewed in a modal without loading the (large) original file and without leaving the page. This almost gallery-like display is about as far as I'd like to go without starting to use JavaScript in the default styleset. Some parts of a web site generated with SBWG are now collapsible/expandable. The parts with this new feature are: entry attachments, entire entries on tagpages, entire entries on their own pages, the custom menu, the category list, topic list, language list and author list in the navbar and the entire navbar. By default all of those things are extanded upon page lead and collapsible by clicking/tapping on their titles. But you can add a setting in your settings file for each of those types of things to be collapsed upon page load and expandable by cliking/tapping on them.

Another thing that behaves similarly is content warnings. Hiding the content of an entry, or parts of it, could always be done by hand, e.g. by adding a <details> and <summary> tag pair to the body of the entry. But it is now easily possible by adding a warn: tag to the header of an entry. For example adding warn:This entry contains spoilers. to an entry header results in the entire content being hidden behind a collapsed <details> tag. Initially visible is only the summary "This entry contains spoilers. (click to open)". The default style set makes this warning line very visible but dunking it into a strong red. If the entry has attachments, those will be collapsed and their title marked in red, too. Both the entry content and its attachments will be collapsed by default if the entry has a warn: tag, independently of what your settings for those parts is in the settins file.

New special tagpages: If there are entries that have a language tag and others don't have one, 'nolang.html' will be created that lists all entries that don't have a language tags. Similarly, if at least one entry on the blog has an author tag, but other entries don't, 'noauthor.html' is created and linked to from the navigation bar.

RSS feeds are now created for every tagpage. That means visitors can now subscribe to individual topics or categories or authors or languages. It also means a longer generation time for a complete regeneration process. But permanent caching reduces that to an acceptable amount. Other feed formats are still not created because I reckon that writing those will be particularly fun and satisfying. So I want to get to that when more of the less interesting todo bullet points are done.

Many small changes make web sites with different structures than mine cleaner. Empty directories or menu entries aren't created. There are new hooks for the various different new functions and loops which a user might want to hook into. Various changes around the default language being used for RSS feeds and HTML documents in order to abide by the standards. Many bugs around all sorts of things have been fixed. The logging option works pretty reliable now. It may still be completely removed one day because it's a large chunk in the code, makes the script slower when enabled and can now be entirely replaced by redirecting output from the script on the shell level. The new flags 'hideinfeeds', 'noshow', 'nogal', 'noatts' and 'noheader' control how and where entries are presented and which parts are visible. See the README for descriptions of those flags.

Reading these update blog posts shouldn't be seen as a replacement for reading the CHANGELOG file. If you have a SBWG web site and consider updating SBWG, at least check the CHANGELOG for lines marked with ! since your current version. That makes it much much less likely that you miss something that you should change upon updating SBWG. I mean, I don't think anybody except me uses any version of SBWG. But I've always approached this project working as if it would be used by others in order to produce something that is practically usable without reading and understanding the codebase first.

The list of things that I'd like to do with SBWG is still long and includes heavy changes on how content is placed in the website's input directory. I reckon that it would take me something like 10 - 16 years to get there is I would continue advancing through the todo list at the pace at which I have in the last year. But if I would loose interest at any point along this path, I could feel okay for at least have gotton as far as testing and publishing version 0.12.8 because it's getting closer to looking how I want it to look in regards to the results it produces.

SBWG 0.11.1

I'm glad to have this version done and published. Even though it has new features compared to the last published version, I've used it enough to feel relatively comfortable saying it is also more stable and has fewer bugs than the last version. There are known bugs still left. But those aren't new.

Caching options and settings are slowly getting a bit elaborate, but also close to what I imagined them being able to do. There are new cachegroups and parts of entries can now be chosed as individual cachegroups, giving the user more control over what is cached and what caches are used. The lifetime of caches can now be set to make sure that no matter how the caching options are used, no outdated content, tags, attachments, etc. will stay on the web site for too long because cache files have been forgotten to be purges. The cache lifetime can be chosen from 1 second to unfinite. The directory used for persistant caches can now be chosen via command line option. I think the caching options are now in a state in which they can seriously be used reliably. Changes to existing cachegroups, like in previous versions, will probably not be made anymore. Just more cachegroups will be added.

The README file has grown a lot. Not only from new features and options. It's now also more complete. A lot of bugs have been fixed. And some general little code quality improvements took place. The stylesheets of the example web site have been improved a bit.

The messaging and logging system has been completely rewritten. It's not really an important part of the script and strictly doesn't matter for its functionality at all. But I decided to have a messaging system and a logging option that doesn't rely on shell redirects. So I did want to redo it properly. Different message types can now be redirected using file descriptors. The option parser has also been party re-written. It's approaching a state in which it's close to what I think personally a complete option parser for Bash should be like.

Alternate styles can now be defined in a web site's settings file. That way alternate style sheets will be offered to the web browser. If the browser supports alternate style sheets, the visitor can select from them to view the page in a different style. There is now an option to change the tagstyle, enabling a web site to have tags of the same type grouped for a more readable look of the entries' headers. Tags now show in parenthesis the number of entries that carry that tag. The number of file attachments is also displayed below entries.

The options for generating a single entry, page or gallery that is not located in the input directory/isn't actually part of the web site can now be used. They were very buggy or nonexistant before. Entries (and other items) with duplicate names are handled in defined ways now. Text files can now be used as file attachments to entries. But if the script determines one to look like a SBWG source file, it is omitted/not displayed/not linked to.

There are still big plans for future versions. Small and big features as well as a complete re-thinking of how the input directory is structured. Nicer looking web sites, more flexible command line options, more options for speeding up the script in cases where a complete website is updated/re-built. A pause like the one before this release will probably occur more often in the future.

Bash Tip

In a script, if you open a file descriptor like this: eval exec 3>&1 and redirect output from certain commands to it like this: echo blub >&3 then you can redirect only that certain output to a different file if needed when calling the script. By default all goes to stdout, but when called like this: ./script 3>>file1 >>file2 then you can check the separated output in file1 and the rest of stdout in file2.

When I started to really use Linux I followed the common suggestion to read all the relevant man pages. I skipped a lot. But I still sometimes feel like it is useful that I've read about a thing or another before. I attempted to read the bash manual back then, but it was just too much at once.

Many years later I started to use more of my shell's potential, which was still bash. Now I'm used to write scripts and have written a relatively large application in bash, but had never read the entire bash manual.

Until this year. I've read the entire bash man page. And yes, I did learn useful new things.

SBWG 0.10.11

It's been two months since I published a new version of SBWG (and therefore since I worked enough on it to make me feel that an increase in the version number may be justified). I hardly get around to working on it, lately. But when I do, I try to make some solid small steps towards the improvements I wish do be done to SBWG.

So, now there is a new version (0.10.11) but I haven't published it yet because before I do I want to test it a bit more when I'm not as fucking tired as I am right now. Writing this entry could actually be considered part of the testing that I still want to do. But as I said, I'm fucking tired right now. So I'll continue this entry another time.

Edit 2022-09-12: I did some tests with fake content offline and feel confident that I fixed more than I broke in this version. I'll add it to the project page soon.

Two long procrastinate areas that have been overdue to get fixed for a while now are permanent caching and parallelisation. For permanent caching, the bugs that I found are fixed. Aggressive caching with option `--cache` (`-C`) works and certain groups can be selected (excluding others) for permanent caching. In my test this speeds up everyday web site re-generations by up to 100%. But it also means the user has to be aware of the cache files that persist after SBWG is done and will be used the next time. Changes to existing entries will not take effect without removing the relevant cache files. For parallelisation, the bugs that I found and that affect the generated HTML are fixed. I still consider parallel mode to be experimental. Sometimes messages on stdout are weird or cut off. But who uses very verbose mode and looks at every message anyway? Parallelisation is functional and usable is what I mean to say. In my test on a Core i5 with 4 cores (8 threads) it speeds up everyday web site re-generation by around 380% with default settings. It depends a lot on how much of web site generation is generating thumbnails and other image sizes of file attachments or gallery images. Those aren't sped up that much because the CPU already is the bottleneck for those. HTML file generation profits more from having more parallel threads than CPU cores (or threads) running.