Steeph's Web Site

Go To Navigation
Show/Hide Navigation

Entries tagged 'cat:Code'

SBWG 0.11.1

I'm glad to have this version done and published. Even though it has new features compared to the last published version, I've used it enough to feel relatively comfortable saying it is also more stable and has fewer bugs than the last version. There are known bugs still left. But those aren't new.

Caching options and settings are slowly getting a bit elaborate, but also close to what I imagined them being able to do. There are new cachegroups and parts of entries can now be chosed as individual cachegroups, giving the user more control over what is cached and what caches are used. The lifetime of caches can now be set to make sure that no matter how the caching options are used, no outdated content, tags, attachments, etc. will stay on the web site for too long because cache files have been forgotten to be purges. The cache lifetime can be chosen from 1 second to unfinite. The directory used for persistant caches can now be chosen via command line option. I think the caching options are now in a state in which they can seriously be used reliably. Changes to existing cachegroups, like in previous versions, will probably not be made anymore. Just more cachegroups will be added.

The README file has grown a lot. Not only from new features and options. It's now also more complete. A lot of bugs have been fixed. And some general little code quality improvements took place. The stylesheets of the example web site have been improved a bit.

The messaging and logging system has been completely rewritten. It's not really an important part of the script and strictly doesn't matter for its functionality at all. But I decided to have a messaging system and a logging option that doesn't rely on shell redirects. So I did want to redo it properly. Different message types can now be redirected using file descriptors. The option parser has also been party re-written. It's approaching a state in which it's close to what I think personally a complete option parser for Bash should be like.

Alternate styles can now be defined in a web site's settings file. That way alternate style sheets will be offered to the web browser. If the browser supports alternate style sheets, the visitor can select from them to view the page in a different style. There is now an option to change the tagstyle, enabling a web site to have tags of the same type grouped for a more readable look of the entries' headers. Tags now show in parenthesis the number of entries that carry that tag. The number of file attachments is also displayed below entries.

The options for generating a single entry, page or gallery that is not located in the input directory/isn't actually part of the web site can now be used. They were very buggy or nonexistant before. Entries (and other items) with duplicate names are handled in defined ways now. Text files can now be used as file attachments to entries. But if the script determines one to look like a SBWG source file, it is omitted/not displayed/not linked to.

There are still big plans for future versions. Small and big features as well as a complete re-thinking of how the input directory is structured. Nicer looking web sites, more flexible command line options, more options for speeding up the script in cases where a complete website is updated/re-built. A pause like the one before this release will probably occur more often in the future.

SBWG 0.10.11

It's been two months since I published a new version of SBWG (and therefore since I worked enough on it to make me feel that an increase in the version number may be justified). I hardly get around to working on it, lately. But when I do, I try to make some solid small steps towards the improvements I wish do be done to SBWG.

So, now there is a new version (0.10.11) but I haven't published it yet because before I do I want to test it a bit more when I'm not as fucking tired as I am right now. Writing this entry could actually be considered part of the testing that I still want to do. But as I said, I'm fucking tired right now. So I'll continue this entry another time.

Edit 2022-09-12: I did some tests with fake content offline and feel confident that I fixed more than I broke in this version. I'll add it to the project page soon.

Two long procrastinate areas that have been overdue to get fixed for a while now are permanent caching and parallelisation. For permanent caching, the bugs that I found are fixed. Aggressive caching with option `--cache` (`-C`) works and certain groups can be selected (excluding others) for permanent caching. In my test this speeds up everyday web site re-generations by up to 100%. But it also means the user has to be aware of the cache files that persist after SBWG is done and will be used the next time. Changes to existing entries will not take effect without removing the relevant cache files. For parallelisation, the bugs that I found and that affect the generated HTML are fixed. I still consider parallel mode to be experimental. Sometimes messages on stdout are weird or cut off. But who uses very verbose mode and looks at every message anyway? Parallelisation is functional and usable is what I mean to say. In my test on a Core i5 with 4 cores (8 threads) it speeds up everyday web site re-generation by around 380% with default settings. It depends a lot on how much of web site generation is generating thumbnails and other image sizes of file attachments or gallery images. Those aren't sped up that much because the CPU already is the bottleneck for those. HTML file generation profits more from having more parallel threads than CPU cores (or threads) running.

SBWG 0.10.10

I decided to publish a new version of SBWG. I hardly got around to editing it in the last few months. So it doesn't really contain the new things that I had for a while intended for v0.10.10. But there were a few bugs that I wanted to have fixed in the latest published version. There still are known bugs. But now there are a few less.

Well, since it doesn't really contain anything new that's relevant, I didn't have to create this entry. But I did, so I'll use it to say I also fixed the RSS feed of draft0 and the menu link of steephlog on mobile. Bugs that I didn't know I had introduced in the last update and that were very happy that I didn't test things beforehand.

The things that did change since my last SBWG update just aren't that interesting. But I'll list some here, nevertheless: Entries can now have custom notes (using the 'note:' tag; Topic tagpages now also show the author names of entries; Srickied entries are now excluded from the RSS feed because they're just used as notices at the top of tagpages and don't belong context-less in the feed (there is a better solution planned); Error reporting inside of the script now includes information on the basis of which the clean-up routine can decide how to handle things differently for different errors and stuff; SBWG now locks directories; There is now a flag that hides attachments to an individual entry that are of a specific mime type, and, you could just read the CHANGELOG file instead of this if you really are interested in these little changes.

It'll probably continue to be slow in the foreseeable future. There are a lot of ideas in my head and several bugs in the code and probably vice versa. But time… not so much.

Edit: Oh, and another thing: I started to test updates and changes that I made to SBWG live on this web site because it's more fun if something breaks every other day. I'll pretend I don't mind.

PIN Changer

I had to change the default PINs of over 200 SIM cards once. And such a situation could arise again. So I've built a PIN changer in which I just have to insert the card, wait a few seconds and it's done.

The Card Slots

SIM cards in their natural form factor aren't as fiddly to handle as they are in the form factors most people know, which is Mini SIM, Micro SIM and Nano SIM. Classical SIM cards are the same size as other smart cards. I found a card slot with an end switch on eBay. I like it when I find industry grade parts for cheap on eBay as part of some remaining stock. Additionally I've used a slide-in mini SIM slot and another, separate end switch from my parts collection in case I to change the PINs of smaller cards.

The Baseband Processor

Other parts that I've used is an Arduino Nano sized Arduino Nano nearly-clone and an A6 modem module. There are many similar modem modules designed around different but similar ICs. Many of them are cheap and widely used for DIY IoT projects. So example code for the Arduino and other help can be easily found on the web. I don't know why I went with a module with an A6. But it works fine and there are an Arduino library for it as well as cheap modem modules with it available.

(tba:voltage supply)

The Controller

Yes, Arduino may be kind of the noob go-to board and could look up how to use microcontrollers on their own finally and even if I don't want to I could finally start to use ESP32s like everybody else. But I know Arduinos and by now I'm familiar with it and it works, so, whatever. Arduino Nano is kind of my go-to form factor now because they have integraded USB, are Uno compatible and small. Unless I need more or something very specific I use Arduino Nano almost-clones with USB-C port.

The Code

The code is a real mess. It had been a long time since I had written any even halfway serious C. It may have been the first time, actually. The sketch surely is very easily improved by somebody who knows what they are doing. I intended to improve it myself. But the project is currently abandoned and The code is doing what it should in a way I initially had in mind as the goal. But I'll leave the mess of the comments in for the case that somebody wants to make out what I was thinking.


// Funktionen umschreiben: Beim Empfangen wird erwartet: 1. der AT-Command zurück, 2. eine Antwort, 3. OK oder ein ERROR.
//                         Daher sollte abgefragt werden, bis entweder OK oder ERROR kam oder 20/50/9001(?) Abfragen lang weder OK noch ERROR an kam.
//                         Die Antwort in Variable speichern? Naja, String zurückgeben halt.
//                         Antworten, die mit "^" anfangen brauchen nicht gehandlet zu werden, da keine Kommandos, die mit AT^ beginnen gesendet werden.


#include  

SoftwareSerial A6MODULE(6,7);
int intled = 13; // Internal LED
int successled = 8; // Green LED
int failled = 9; // Red LED
int wrongpinled = 10; // Orange LED

boolean debug=true;

//String commands[5] = { "AT", "AT+CPINC2", "AT+CPIN?", "AT+CLCK=\"SC\",2", "AT+CPIN=\"3010\"" };
//int command = 0;

const byte maxmsglength = 32;
char received[maxmsglength];
boolean newData = false;
String response = "";

int i=0;

/*
To check/do:
1  AT: OK?
2  is PUK required - abort
3  are less than 3 PIN attempts left? - abort AT^CPINC=?
4  is PIN disabled
       5  enable it: 0000
6  is PIN enabled
       7  unlock
8  is card unlocked
       9  change PIN: 1996
              was PIN wrong - report and leave it
*/


void setup() {
  pinMode(intled, OUTPUT);
  pinMode(successled, OUTPUT);
  pinMode(failled, OUTPUT);
  pinMode(wrongpinled, OUTPUT);
  // All LEDs turn on at the beginning and stay on during the wait period at the beginning, then turn off before communication with the A6 module starts.
  digitalWrite(intled, HIGH);
  digitalWrite(successled, HIGH);
  digitalWrite(failled, HIGH);
  digitalWrite(wrongpinled, HIGH);
  Serial.begin(9600);
  delay(500);
  digitalWrite(intled, LOW);
  digitalWrite(successled, LOW);
  digitalWrite(failled, LOW);
  digitalWrite(wrongpinled, LOW);
  A6MODULE.begin(9600);
  delay(500);
  digitalWrite(intled, LOW);
}

void loop() {
  sendtoa6("AT");
  if(getfroma6("OK")) {
//    sendtoa6("AT+CPIN?");
//    if(getfroma6("+CPIN:SIM PUK")) { fail; }         // If the required password is PUK, abort.
//    sendtoa6("AT+CPIN?");
//    if(getfroma6("+CPIN:SIM PIN2")) { fail; }        // If the required password is PIN2, abort.
//    sendtoa6("AT+CPIN?");
//    if(getfroma6("+CPIN:SIM PUK2")) { fail; }        // If the required password is PUK2, abort.

// WAIT FOR SWITCH RELEASE FIRST
    sendtoa6("AT+CPIN?");
    getfroma6("+CPIN:SIM PIN");                      // The last non-empty response will be stored in the global response variable. Problem with this: If the A6 module sends an unsolicitated message before the response to the CPIN command, nothing gets done and the card needs to be re-inserted again.
    if(strcmp(response, "OK") == 0) {                // If already no PIN is required

//      sendtoa6("AT+CLCK=\"SC\",2"); // Ist PIN-Abfrage eingeschaltet? Oder ist es "SC"?
//      if(getfroma6(???)) { PIN-Abfrage einschalten mit 0000; }

      // ENABLE PIN HERE
      d("I don't know how to enable the PIN.");

    }
    if(response = "+CPIN:SIM PIN") {                 // If the required password is PIN, continue.
      sendtoa6("AT^CPINC=?");
      if(getfroma6("^CPINC: 3")) {                   // If not exactly 3 attempts are left, abort. (should be larger than or equal to 3, shouldn't it?)
        sendtoa6("AT+CPIN=\"0000\"");                // Freischalten mit 0000
        delay(50);
        if(!getfroma6("OK")) { fail; }               // If that was not the right password, abort.
        sendtoa6("AT+CPWD=\"SC\",\"0000\",\"1996\"");// PIN ändern
        delay(500);
        if(getfroma6("OK")) { Serial.println("Looking good."); }

//      sendtoa6("AT+CMGD=0,4"); // Should delete all SMS
//      if(getfroma6(???)) { ; }
        d("I don't know how to delete SMS.");

        sendtoa6("AT+CPIN?");
        if(getfroma6("+CPIN:READY")) {
          
          // TURN OFF A& MODULE FOR SAFE CARD REMOVAL

          digitalWrite(successled, HIGH);
        }
      } else {                                         // If not exactly 3 times left
        fail;
      }
      delay(1000);
    }
  }
}

void fail() {
  digitalWrite(failled, HIGH);               // Turn red fail LED on and ...
  d("Something failed! Ending programme.");
  while(1);                                  // ... don't do anything anymore.
}

void wrongpin() {
  digitalWrite(wrongpinled, HIGH);           // Turn yellow LED on and ...
  d("Wrong PIN! Ending programme.");
  while(1);                                  // ... don't do anything anymore.
}

//boolean getfroma6(char str[32], char str1[32], char str1[32], char str1[32], char str1[32], char str1[32]) { // Returns true if the passed (expected) message was received, false if anything else was received.
boolean getfroma6(char str[32]) { // Returns true if the passed (expected) message was received, false if anything else was received.
  boolean asexpected = false;
    for (i = 1; i < 9; ++i) {
    receivelinefroma6();
//    d("d1 "+response);

    if(received[0] == '\0') {                            // If the received message is empty
      continue;
    } else {
      response = received;
    }
    if(strstr(received, "ERROR") != NULL) {              // If the received message contains "ERROR"
      d("Received an error: "+response);
      fail;
    }
    if(received[0] == '+') {                             // If the received message starts with a "+" sign
      d("Reseived response: "+response);
    } else {
      if(strstr(received, "OK") != NULL) {               // If the received message is "OK"
        d("Received OK.");
        asexpected = true;                               // Also treat OK like the expected message. No unexpected OK should ever be sent from the A6 module. So this is fine. No, it is, really.
//        if(asexpected) { return true; }
      } else {
        if(strncmp(received,"AT",2) == 0) {              // If the received message starts with "AT"
          d("Received AT: "+response);
        } else {                                         // For any other received message
          d("Received: "+response);
        }
      }
    }
    newData = false;
    if(strstr(received, str) != NULL) {
      d("Received expected message: "+response);
      asexpected = true;
    }
  }
  if(asexpected) { return true; }
  return false;
}


//void handleresponse() {
//  response = received;
//  if(received[0] == '+') {
//    d("               Response: "+response);
//  } else {
//    if(strstr(received, "OK") != NULL) {
//      d("               It's okay.");
//    } else {
//      if(strncmp(received,"AT",2) == 0) {
//        d("               I've sent: "+response);
//      }
//    }
//  }
//  newData = false;
//}



void receivelinefroma6() {
  delay(80);
  static byte counter = 0;
  char rc;
  
  received[0] = '\0';

  while (A6MODULE.available() > 0 && newData == false) {
    rc = A6MODULE.read();

    if (rc != '\n') {
      received[counter] = rc;
      counter++;
      if (counter >= maxmsglength) {
          counter = maxmsglength - 1;
      }
    }
    else {
      received[counter] = '\0'; // terminate the string
      counter = 0;
      newData = true;
    }
  }
}

void sendtoa6(String command) {
//  Serial.println(command);
  A6MODULE.println(command);
  d("Sent: "+command);
}

void d(String line) {
  if (debug == true) { Serial.println(line); }
}

(tba:connections,assembly,photos?)

SBWG 0.10.7

So many ideas, so little time. So many todos, no little time. So little time, so much rest of my life. But the todo list has actually started to become a little shorter for the first time in a long time. As with the previous entries about my progress with SBWG, I'm not going to address or list everything that I've improved or added. But I feel like it's time for another update.

File attachments pretty much work as intended now. Text files still can't be file attachments to entries, because text files are entries. But I think that will stay as it is for now because I don't need to attach text files to entries and if I would, I could gzip, bzip or 7z them. I mainly wanted file attachments for entry images and podcast support. Both works as well as I need it to. But some things could still be improved in the future.

Referencing tags are improved, their notes look better and there is now a way to mark an entry as an update of an other entry, giving both entries a note that tells the reader about the newer/older version. And since this works now, I dropped the idea to introduce any form of content revisioning. An entry revision system would be over the top for a feature that I hardly use. This could still be implemented just with hooks.

SBWG was relying on a feature of ImageMagick's mogrify that turned out to be a bug and was fixed. So I had to switch to convert with worse tolerance/support for broken image files. That's annoying, but I'm over it now. I mean, I wasn't complaining. Those tools are amazing and I'm so glad that they are free.

There were a number of other bugs that I've fixed. Some of them old and overlooked. But most of them were introduced not long ago. Well, I guess that's what happens if you don't do proper tests every time. SBWG is starting to become a feature-rich tool and every new feature that gets implemented could cause trouble with other features or combinations of features. In early versions of SBWG I didn't always test everything I changed. I now do, and I now also try to think of some related things the change could have effected unintentionally and test those basically. But proper tests of all sorts of scenarios every time I publish a new version? That would be professional but not worth the effort in this case. If I'm not using a feature, it doesn't matter much if I break it. I think I'll probably do more extensive tests every now and then.

Entries can now have comment links. If there is an email address available for an author, all their entries get an email link below them to allow readers to reply to them. For a public blog it might be best to use throwaway addresses that can quickly be replaced if they start to receive spam. I'd have some ideas to improve the email composed by this comment feature. But they would have to become HTML emails. So I forgot about them again.

New tag types, expanded README file, improved navigation by adding parent links and things like this, improved CSS a bit here and there, very small improvements to the code style. I didn't make any relevant improvements or corrections in terms of code quality. I think this is where SBWG could improve most. It's not a professional thing, so I declare myself fine with the way it is (which isn't that bad compared to some other software, I think) but I could learn a lot there.

Command line options can now be grouped to make typing commands easier. Long options (starting with --) didn't change (I may make those things POSIX-like, too, some other time) but short options (starting with a -) now don't need to be separated and all have their own dash. So sbwg -v -p -e blub can now be shortened to sbwg -vpe blub. And if you want to run the command again but with added verbosity, you can add the second -v wherever you want.

There is now a waiting animation that starts to play if there is no update on stdout for three seconds. Both the animation frames and the time delay can be changed. The animation is only enabled if SBWG runs with any verbosity. Then it is enabled by default. It will be disabled by default in future versions though, I think, because it's such an unnecessary messing around with terminal output. It does work pretty well though. I thought it may be harder to make it work as smoothly as it does now (which isn't 100% perfect, but pretty good).

The new default style set 'elth' is a working hybrid with different styles for small phone screens and larger screens of phablets, tablets and laptops. It's composed of several CSS files, which should make it easier to create a custom style now, since just the basics (without any colors and fancy embolishment and without any tag-specific stuff) can be copied for a template, or just the mobile style can be copied and used to create a new mobile style. Style sets can now also include files whose name don't end in '.css'. That means they can contain any file type. So JS or background images or whatever will be copied as part of the style set and placed in the 'styles/' directory of the generated web site.

More things are now cached that may be used more than once in one run of SBWG. Permament caches (option -C/--cache) can now be enabled for only parts of the web site. I still need to decide how permanent caches should be managed. The way I had/have it in mind does not seem very intuitive. It is to me because I know how it's built. But... well, maybe it's enough that I think that it makes sense. Maybe I should just start using it and if it works well, write it down in the README file.

Added new hooks, gave the helper scripts makeovers, made how HTML footers are made logical, sorted much of the remaining todos. Overall: progress.

So, I've published SBWG version 0.10.7 now. I wish I'll find the time to create and publish new versions more frequently again in the near future. But I doubt I will.