Steeph's Web Site


Entries tagged 'cat:Linux'

SBWG 0.10.11

It's been two months since I published a new version of SBWG (and therefore since I worked enough on it to make me feel that an increase in the version number may be justified). I hardly get around to working on it, lately. But when I do, I try to make some solid small steps towards the improvements I wish do be done to SBWG.

So, now there is a new version (0.10.11) but I haven't published it yet because before I do I want to test it a bit more when I'm not as fucking tired as I am right now. Writing this entry could actually be considered part of the testing that I still want to do. But as I said, I'm fucking tired right now. So I'll continue this entry another time.

Edit 2022-09-12: I did some tests with fake content offline and feel confident that I fixed more than I broke in this version. I'll add it to the project page soon.

Two long procrastinate areas that have been overdue to get fixed for a while now are permanent caching and parallelisation. For permanent caching, the bugs that I found are fixed. Aggressive caching with option `--cache` (`-C`) works and it certain groups can be selected (excluding others) for permanent caching. In my test this speeds up everyday web site re-generations by up to 100%. But it also means the user has to be aware of the cache files that persist after SBWG is done and will be used the next time. Changes to existing entries will not take effect without removing the relevant cache files. For parallelisation, the bugs that I found and that affect the generated HTML are fixed. I still consider parallel mode to be experimental. Sometimes messages on stdout are weird or cut off. But who used very verbose mode and looks at every message anyway? Parallelisation is functional and usable is what I mean to say. In my test on a Core i5 with 4 cores (8 threads) it speeds up everyday web site re-generation by around 380% with default settings. It depends a lot on how much of web site generation is generating thumbnails and other image sizes of file attachments or gallery images. Those aren't sped up that much because the CPU already is the bottleneck for those. HTML file generation profits more from having more parallel threads than CPU cores (or threads) running.

SBWG 0.10.10

I decided to publish a new version of SBWG. I hardly got around to editing it in the last few months. So it doesn't really contain the new things that I had for a while intended for v0.10.10. But there were a few bugs that I wanted to have fixed in the latest published version. There still are known bugs. But now there are a few less.

Well, since it doesn't really contain anything new that's relevant, I didn't have to create this entry. But I did, so I'll use it to say I also fixed the RSS feed of draft0 and the menu link of steephlog on mobile. Bugs that I didn't know I had introduced in the last update and that were very happy that I didn't test things beforehand.

The things that did change since my last SBWG update just aren't that interesting. But I'll list some here, nevertheless: Entries can no have custom notes (using the 'note:' tag; Topic tagpages now also show the author names of entries; Srickied entries are now excluded from the RSS feed because they're jsut used as notices at the top of tagpages and don't belong context-less in the feed (there is a better solution planned); Error reporting inside opf the script now includes information on the basis of which the clean-up routine can decide how to handle things differently for different errors and stuff; SBWG now locks directories; There is now a flag that hides attachments to an individual entry that are of a specific mime type, and, you could just read the CHANGELOG file instead of this if you really are interested in these little changes.

It'll probably continue to be slow in the foreseeable future. There are a lot of ideas in my head and several bugs in the code and probably vice versa. But time… not so much.

Edit: Oh, and another thing: I started to test updates and changes that I made to SBWG live on this web site because it's more fun if something breaks every other day.

Linus Torvalds in 1992 on comp.os.minix

Aparently this is somewhat a famous thread at least in the Linux community. But I'm neither a frequent reader nor a subscriber of any usenet content. So I didn't see it before.

Linus Torvalds in 1992 on comp.os.minix, apologizing for overreacting and ignoring netiquette: "my first, and hopefully last flamefest"

I guess hope wasn't enough. :)

SBWG 0.10.7

So many ideas, so little time. So many todos, no little time. So little time, so much rest of my life. But the todo list has actually started to become a little shorter for the first time in a long time. As with the previous entries about my progress with SBWG, I'm not going to address or list everything that I've improved or added. But I feel like it's time for another update.

File attachments pretty much work as intended now. Text files still can't be file attachments to entries, because text files are entries. But I think that will stay as it is for now because I don't need to attach text files to entries and if I would, I could gzip, bzip or 7z them. I mainly wanted file attachments for entry images and podcast support. Both works as well as I need it to. But some things could still be improved in the future.

Referencing tags are improved, their notes look better and there is now a way to mark an entry as an update of an other entry, giving both entries a note that tells the reader about the newer/older version. And since this works now, I dropped the idea to introduce any form of content revisioning. An entry revision system would be over the top for a feature that I hardly use. This could still be implemented just with hooks.

SBWG was relying on a feature of ImageMagick's mogrify that turned out to be a bug and was fixed. So I had to switch to convert with worse tolerance/support for broken image files. That's annoying, but I'm over it now. I mean, I wasn't complaining. Those tools are amazing and I'm so glad that they are free.

There were a number of other bugs that I've fixed. Some of them old and overlooked. But most of them were introduced not long ago. Well, I guess that's what happens if you don't do proper tests every time. SBWG is starting to become a feature-rich tool and every new feature that gets implemented could cause trouble with other features or combinations of features. In early versions of SBWG I didn't always test everything I changed. I now do, and I now also try to think of some related things the change could have effected unintentionally and test those basically. But proper tests of all sorts of scenarios every time I publish a new version? That would be professional but not worth the effort in this case. If I'm not using a feature, it doesn't matter much if I break it. I think I'll probably do more extensive tests every now and then.

Entries can now have comment links. If there is an email address available for an author, all their entries get an email link below them to allow readers to reply to them. For a public blog it might be best to use throwaway addresses that can quickly be replaced if they start to receive spam. I'd have some ideas to improve the email composed by this comment feature. But they would have to become HTML emails. So I forgot about them again.

New tag types, expanded README file, improved navigation by adding parent links and things like this, improved CSS a bit here and there, very small improvements to the code style. I didn't make any relevant improvements or corrections in terms of code quality. I think this is where SBWG could improve most. It's not a professional thing, so I declare myself fine with the way it is (which isn't that bad compared to some other software, I think) but I could learn a lot there.

Command line options can now be grouped to make typing commands easier. Long options (starting with --) didn't change (I may make those things POSIX-like, too, some other time) but short options (starting with a -) now don't need to be separated and all have their own dash. So sbwg -v -p -e blub can now be shortened to sbwg -vpe blub. And if you want to run the command again but with added verbosity, you can add the second -v wherever you want.

There is now a waiting animation that starts to play if there is no update on stdout for three seconds. Both the animation frames and the time delay can be changed. The animation is only enabled if SBWG runs with any verbosity. Then it is enabled by default. It will be disabled by default in future versions though, I think, because it's such an unnecessary messing around with terminal output. It does work pretty well though. I thought it may be harder to make it work as smoothly as it does now (which isn't 100% perfect, but pretty good).

The new default style set 'elth' is a working hybrid with different styles for small phone screens and larger screens of phablets, tablets and laptops. It's composed of several CSS files, which should make it easier to create a custom style now, since just the basics (without any colors and fancy embolishment and without any tag-specific stuff) can be copied for a template, or just the mobile style can be copied and used to create a new mobile style. Style sets can now also include files whose name don't end in '.css'. That means they can contain any file type. So JS or background images or whatever will be copied as part of the style set and placed in the 'styles/' directory of the generated web site.

More things are now cached that may be used more than once in one run of SBWG. Permament caches (option -C/--cache) can now be enabled for only parts of the web site. I still need to decide how permanent caches should be managed. The way I had/have it in mind does not seem very intuitive. It is to me because I know how it's built. But... well, maybe it's enough that I think that it makes sense. Maybe I should just start using it and if it works well, write it down in the README file.

Added new hooks, gave the helper scripts makeovers, made how HTML footers are made logical, sorted much of the remaining todos. Overall: progress.

So, I've published SBWG version 0.10.7 now. I wish I'll find the time to create and publish new versions more frequently again in the near future. But I doubt I will.

SBWG 0.10.2

I know, my entries on SBWG are too chaotic to follow what's happening with the project in detail. But who cares, really? I'll just continue to write broadly about the progress that I make with the script.

So, after re-deciding that I'm not going to make a version 1.0 because it's just not the kind of software that can ever claim for itself the thoroughness that I myself would expect from a piece of software that's labeled v1.0, I finished the testing and bug squashing that I, for a short while, had intended to lead to version 1.0 to my satisfaction, and started implementing new features again. Great, now I can make the script take longer to finish again. :)

During my testing and bug squashing spree I already implemented a routine that I call the pathreducer. If enabled through command line option, it reduces paths to conform with the file and directory name restrictions of most file systems out there. I took this a bit far and it can now even conform to filenames of the 6.3 format, if the user wishes. I have ideas to improve this further, including making it automatically conform to the exact limitations of individual file systems instead of stripping and replacing characters that aren't allowed in file names in many file systems whether it would have been necessary in the present case or not. But that's for another time. For now it works well for what it is supposed to be: a workaround for too long and otherwise invalid file names that would be generated by the script when naming files after categories, topics and other tags, as well as a solution to problems that might occur if SBWG would be used to generate a web site onto a file system that has more restrictions than those commonly used on Unix-like systems.

As part of the improvements made during the no-new-feature time, I fixed many bugs that I didn't expect at all when I started (which is why I prohibited myself from implementing new features in the first place), learned a lot about Bash again, and made some small improvements that I wouldn't call new features, but are still neat to have. One of them makes tag lines more intuitive. Forbidden characters are removed or replaced and leading and trailing whitespaces are removed, leading coincidentally to the fact that you can use a space after the tag type (e.g. 'cat: Example' instead of 'cat:Example') or not and the tag will be recognised as being identical in both cases. I'm mentioning this because the first person to type a SBWG tag line besides me typed a space intuitively and I had to tell them that that's not how that works. Well, now it is, if you prefer it. Maybe case-insensitivity will be next. I'm considering it. Wouldn't be hard to add. I'm not sure if I want it, though.

Another thing that improved drastically in that time is the time it takes to generate huge web sites. SBWG never had problems with giant files, even entry files with gigabytes of text, because the content body isn't parsed. But with thousands of entries or thousands of tags in one entry, generation time would previously go into hundreds of years (theoretically/calculated, of course). By improving some routines and caching some parts during the generation process I got the generation time of an example torture web site that I created, from over 300 years down to just short of 72 hours on my Core-i5 ultrabook onto an NVMe drive. Any reasonably sized web site even with a relatively large weblog is completely generated much quicker. So it would be possible to e.g. let SBWG re-generate a web site automatically multiple times a day without problems. I have other ideas that will speed up the generation process significantly, namely parallel processing and skipping of existing unchanged content. But both those features are not mature enough, yet. I will probably attend to them again when I have packed SBWG with so many features that slow down the process compared to the current version that I feel more parallelisation and more drastic caching will be necessary.

So, the first new feature on the list were entry pictures. A way to bind pictures, image files, to entries without having to include <img> tags. This feature now already developed into a general way to attach files to entries. Images are resized, embedded and linked to their originals, audio files are embedded as an audio player if the browser supports this, and other files are linked to allow visitors to download them. Video files will also be embedded as a player in future version. The file attachments feature is not done, yet. Enclosures in the RSS feed, presentation of the download links and many small things have to be improved. But the initial idea of entry pictures already works. There is a naming convention that makes a file an attachment to a certain entry. (The same naming convention as is used for style sheet sets.) See the README of version 0.10.2 or above for detailed information.

The next feature on the list is comment links below blog entries. Simple mailto links to let readers e-mail comments on entries to the author. I will probably start to work on multiple features at the same time again, as I did before.

So, I've published SBWG version 0.10.2 now. I hope I'll find the time to create and publish new versions more frequently again in the near future. But I doubt I will.