Tom Pike's Web log. The incoherent ramblings of a web developer. Tom Pike tom.pike@xiven.com urn:uuid:81fc386c-a5b8-1029-a3f9-0015f24852f1 2018-08-05T10:14:49Z
Recreating Atoms urn:uuid:ff59dc31-981f-11e8-be17-001999d4f28f 2018-08-05T10:14:49Z Earlier this year, one of my cousins reminded me about a game that we used to play on my old Amiga 500. The name of this game was “Atoms”, and it was a simple but somewhat addictive game created by Tom Kuhn that was given away on the coverdisk of iss…

Earlier this year, one of my cousins reminded me about a game that we used to play on my old Amiga 500. The name of this game was “Atoms”, and it was a simple but somewhat addictive game created by Tom Kuhn that was given away on the coverdisk of issue 46 of the Amiga Format magazine back in 1993 (and based on an old 1989 Atari game of the same name).

Having been reminded of this game, I boldly claimed that I could probably write a clone of that game myself. Ideas were already forming in my head as to exactly how I was going to do this. The main thought that I had was that I wanted to do it purely using web technologies.

Initially the project sat on the metaphorical back burner, but having been inspired by looking at some of the games that my niece had created in Scratch, I decided I had to get started. Progress was slow, but I had set myself a deadline that I should have this game in an at-least-playable state by the next time I saw my nieces in England.

Naturally I had to first research how exactly the original game played since I hadn't played it myself in over 20 years. Unfortunately not having an Amiga to play it on I had to resort to internet research. With some searching online I did find that someone had already implemented a version of this game for Android phones which was pretty good (and even had the original music), but it didn't quite have the feel of the original, and I did still want to do my own version of it for the web. Another useful source I found was a (poor quality) YouTube video of the original game. Armed with this information I set out to create the atoms-www project.

In creating the game I got the chance to play with some web technologies that I hadn't had much experience with yet. Scripting CSS Transitions was fun, if a little challenging - at least until a friend explained the weirdness of double rAF (requestAnimationFrame). Combining the transitions with the transform property allowed for some nice animations mimicking the original game. CSS Grid Layout was very pleasant to work with, avoiding the need to use a <table> element for the game board, or another similarly evil hack. The atom visuals themselves were simple CSS border-radius and CSS radial gradients with the aforementioned scripted CSS transitions to move them into the right places each time an atom is added to a cell. The only places SVG was used was for the logo and the coloured mouse pointer images. No bitmap images were used at all, aside from the favicon / shortcut image.

The game was in a just-about playable state by the time my self-imposed deadline arrived, and I uploaded the work so far to a GitHub project.

Since then I've worked on polishing the game experience, adding the ability to play against computer players (or even just set 4 computer players fighting each other). I added a debug mode to allow me to compare my implementation with the video of the original (for which Vivaldi's tiled tabs feature was incredibly useful) which led me to fix a bug related to conservation of mass (and rediscover a bug in the original game in the process).

Although it's still missing one feature that I want to include (the music!), I think it's now just about ready to share with the rest of the world properly. So here it is:

Play atoms-www

End scene urn:uuid:f60e1047-59d0-11e8-bece-001999d4f28f 2018-05-17T12:51:54Z After more than 11 years working at Opera Software I have decided to move on. A difficult decision to make to be sure, but I'm really looking forward to my new job. As of June 2018 I start work as a Backend Developer at Vivaldi Technologies AS.

After more than 11 years working at Opera Software I have decided to move on. A difficult decision to make to be sure, but I'm really looking forward to my new job.

As of June 2018 I start work as a Backend Developer at Vivaldi Technologies AS.

SSH Key Authority demo server urn:uuid:64971a1a-8e98-11e7-b16f-001999d4f28f 2017-08-31T22:05:17Z Following up on the recent open source release of our SSH Key Authority tool, there is now a demonstration server available. You can find it here: SSH Key Authority demonstration server Use one of the following sets of username / password credentials t…

Following up on the recent open source release of our SSH Key Authority tool, there is now a demonstration server available. You can find it here:

SSH Key Authority demonstration server

Use one of the following sets of username / password credentials to log in:

  • testuser / testuser - normal user with admin access granted to a few servers
  • testadmin / testadmin - admin user
Restoring a deleted custom field in JIRA from backups urn:uuid:5c070a79-8e6d-11e7-b16f-001999d4f28f 2017-08-31T17:15:36Z Imagine if you will, the hypothetical scenario where your newest JIRA admin has just managed to delete a custom field from your JIRA instance. This of course means that all data stored in that field has been deleted across all issues in all of your proje…

Imagine if you will, the hypothetical scenario where your newest JIRA admin has just managed to delete a custom field from your JIRA instance. This of course means that all data stored in that field has been deleted across all issues in all of your projects, and all the schemes and configurations it was referenced by will no longer reference it.

So, after revoking this hypothetical admin's administration privileges, what is your next step?

Like any good system administrator, you obviously have full database backups running on at least a nightly basis, so almost all of the data is there and can (at least theoretically) be recovered. But you don't want to restore the entire database, otherwise you would lose all data for today, which would be very bad.

You could re-create the field through the JIRA UI and then restore the data into this new field, but that has a couple of caveats:

  • You have to manually re-integrate the field into all the field configurations and screens that it used to be in.
  • The ID of the field will have changed, so if the field is referenced by any workflows, scripts or filters by ID then all of these things will need to be fixed manually.

It would be much better if you could restore the field and all of the configuration surrounding it directly as it was before. But how will you know which data to restore, and how will you actually perform this restoration? Allow me to be your guide...

Step-by-step guide

This guide is written for JIRA admins who are using a PostgreSQL database as the backend of their JIRA instance. If you're using a different database engine you will need to make various adjustments.

  1. First things first, get that database backup file and copy it to some server or workstation where you can safely mess around with a database without consequence.
  2. On this safe place, import the file into your database: psql jira < 2017-08-31.sql
  3. Connect to postgres: psql jira
  4. Find the ID of your now-deleted field: select id from customfield where cfname = 'My custom field'; (for this hypothetical scenario, let's assume the ID was 10141).
  5. You'll also need to find the ID of any entries in the fieldconfiguration table that reference your field: select * from fieldconfiguration where fieldid = 'customfield_10141'; (we'll assume we got an ID from this query of 10131)
  6. Copy out the data from the relevant tables, filtered specifically to your custom field:
    
    create table cf_temp as select * from customfield where id = 10141;
    create table cfo_temp as select * from customfieldoption where customfield = 10141;
    create table cfv_temp as select * from customfieldvalue where customfield = 10141;
    create table fsli_temp as select * from fieldscreenlayoutitem where fieldidentifier = 'customfield_10141';
    create table cli_temp as select * from columnlayoutitem where fieldidentifier = 'customfield_10141';
    create table fli_temp as select * from fieldlayoutitem where fieldidentifier = 'customfield_10141';
    create table cc_temp as select * from configurationcontext where customfield = 'customfield_10141';
    create table fc_temp as select * from fieldconfiguration where fieldid = 'customfield_10141';
    create table gc_temp as select * from genericconfiguration where datatype = 'DefaultValue' and datakey = '10131';
    create table fcsit as select * from fieldconfigschemeissuetype where fieldconfigscheme = 10131;
    create table fcsit_temp as select * from fieldconfigschemeissuetype where fieldconfigscheme = 10131;
    
  7. Now you'll want to extract this data from postgres in the form of SQL INSERT statements. From your shell command-line:
    
    pg_dump --data-only --inserts -t cf_temp jira > cf.sql
    pg_dump --data-only --inserts -t cfo_temp jira > cfo.sql
    pg_dump --data-only --inserts -t cfv_temp jira > cfv.sql
    pg_dump --data-only --inserts -t fsli_temp jira > fsli.sql
    pg_dump --data-only --inserts -t cli_temp jira > cli.sql
    pg_dump --data-only --inserts -t fli_temp jira > fli.sql
    pg_dump --data-only --inserts -t cc_temp jira > cc.sql
    pg_dump --data-only --inserts -t gc_temp jira > gc.sql
    pg_dump --data-only --inserts -t fc_temp jira > fc.sql
    pg_dump --data-only --inserts -t fcs_temp jira > fcs.sql
    pg_dump --data-only --inserts -t fcsit_temp jira > fcsit.sql
    
  8. These SQL files are almost what you need, but they'll be using the wrong table names. Use some simple sed magic to clear those right up:
    
    sed s/cf_temp/customfield/ < cf.sql > cf-out.sql
    sed s/cfo_temp/customfieldoption/ < cfo.sql > cfo-out.sql
    sed s/cfv_temp/customfieldvalue/ < cfv.sql > cfv-out.sql
    sed s/fsli_temp/fieldscreenlayoutitem/ < fsli.sql > fsli-out.sql
    sed s/cli_temp/columnlayoutitem/ < cli.sql > cli-out.sql
    sed s/fli_temp/fieldlayoutitem/ < fli.sql > fli-out.sql
    sed s/cc_temp/configurationcontext/ < cc.sql > cc-out.sql
    sed s/gc_temp/genericconfiguration/ < gc.sql > gc-out.sql
    sed s/fc_temp/fieldconfiguration/ < fc.sql > fc-out.sql
    sed s/fcs_temp/fieldconfigscheme/ < fcs.sql > fcs-out.sql
    sed s/fcsit_temp/fieldconfigschemeissuetype/ < fcsit.sql > fcsit-out.sql
    
  9. Copy these SQL files over to your JIRA server.
  10. Back up your JIRA database! Trigger an extra special run of your trusty database backup script.
  11. Stop your JIRA instance.
  12. Run all of these shiny SQL files:
    
    psql jira < cf-out.sql
    psql jira < cfo-out.sql
    psql jira < cfv-out.sql
    psql jira < fsli-out.sql
    psql jira < cli-out.sql
    psql jira < fli-out.sql
    psql jira < cc-out.sql
    psql jira < gc-out.sql
    psql jira < fc-out.sql
    psql jira < fcs-out.sql
    psql jira < fcsit-out.sql
    
  13. Start your JIRA instance up again.
  14. Check to see if everything looks okay.
  15. Hopefully breathe a huge sigh of relief.
  16. Set a re-index going, just for good measure. This is JIRA we're talking about after all.
The Keys System urn:uuid:99643bd3-8751-11e7-b16f-001999d4f28f 2017-08-22T16:10:16Z Several years ago, when I was still working in the Core department at Opera Software, I became aware that the Information Services team had started using a web-based tool to manage SSH key access to root accounts on their servers (I discovered this when…

Several years ago, when I was still working in the Core department at Opera Software, I became aware that the Information Services team had started using a web-based tool to manage SSH key access to root accounts on their servers (I discovered this when I needed to upload my key there to get access to a server that was shared with them).

A few reorganizations and ownership shifts later, I found myself in the team that owned this tool, known only as "the Keys system", or by its hostname "keys".

In November 2013 I embarked on a project to rewrite the Keys system, adding the ability to manage more than just the root account, and improving the user interface. Other features were added later such as group access management, SSH access options, and immediate syncing. It soon became quite well-liked within the company as a tool for people to manage access to their servers.

We've been hoping to open source this for quite some time, and after the successful open-sourcing of our PowerDNS management tool (which was actually developed after the Keys system and shares a lot of core code in common) we've been getting ready to go ahead with this one.

One of the big challenges was to come up with an actual name for the project. In the end we settled on "SSH Key Authority". Today we have released SSH Key Authority to the public on GitHub!

DNS UI demo urn:uuid:6badb262-fe9f-11e6-b805-001999d4f28f 2017-03-01T16:52:47Z A month ago I announced the release of the Opera DNS UI tool. The GitHub page is not really much to look at on its own though, and it's a bit difficult to get a feel for what it does and how it works without seeing it in action, so I've now set up a demo…

A month ago I announced the release of the Opera DNS UI tool. The GitHub page is not really much to look at on its own though, and it's a bit difficult to get a feel for what it does and how it works without seeing it in action, so I've now set up a demonstration server so people can actually see it in action. You can find it here:

Opera DNS UI demonstration server

Use one of the following sets of username / password credentials to log in:

  • testuser / testuser - normal user with admin access granted to a few domains
  • testadmin / testadmin - admin user
Let's distrust StartCom. Let's encrypt instead urn:uuid:ed9046e3-fba9-11e6-b863-001999d4f28f 2017-02-25T22:32:19Z Recently I noticed that the Opera developer browser had started rejecting the SSL certificate used by my server for this website. It worked fine in Opera, Opera beta, Chrome and Firefox on my PC, and Qualsys SSL Labs still reported the domain with an A+…

Recently I noticed that the Opera developer browser had started rejecting the SSL certificate used by my server for this website. It worked fine in Opera, Opera beta, Chrome and Firefox on my PC, and Qualsys SSL Labs still reported the domain with an A+ rating, but for some reason the latest Opera Developer builds simply rejected it. As I would discover later, so too did the latest Chrome Canary builds.

Why?

I certainly wondered that, so after initially coming up dry I asked for some help from colleagues at work. One very helpful person soon pinpointed the problem: the StartCom CA that my certificate was issued by has now been distrusted by Google (with good reason). Mozilla had already done this too.

Although initial reports were that certificates issued before October 21st 2016 would not be affected by this distrust, due to a number of technical limitations and concerns, Google Chrome is unable to trust all pre-existing certificates while ensuring our users are sufficiently protected from further misissuance - hence my certificate no longer being accepted.

Despite this distrust being fairly widely reported in the tech press, I had somehow managed to miss this crucial information, and I certainly didn't receive any notifications about it from StartCom themselves either. Regardless, it needed fixing so I had to find a new (ideally also free) SSL CA.

I had heard about the Let's Encrypt project before, but I hadn't really looked into it in depth. It turns out that they really have made it about as painless as it is possible to be to obtain and install SSL certificates. I opted for using the Certbot ACME client in "certonly" mode and configuring Apache manually both for the verification mechanism and installing the certificate, which was a little more work but still very easy. I now have a brand new multi-domain certificate in use by Apache, Dovecot and Postfix on my server. It should get renewed automatically when needed too with a simple cron job.

Then today I wanted to add another subdomain to the certificate. Absolutely no problem - just reissue the command with the new domain added and then reload the services. Literally as simple as that.

Open sauce urn:uuid:368d002e-e22e-11e6-b97f-001999d4f28f 2017-01-24T12:11:58Z First post since 2013, better be for something good, right? Well no, not really. But hey it's something: my employer (Opera Software AS) has released as open source one of the projects that I've worked on during my time working here. Find it on GitHub…

First post since 2013, better be for something good, right?

Well no, not really. But hey it's something: my employer (Opera Software AS) has released as open source one of the projects that I've worked on during my time working here. Find it on GitHub here:

Opera DNS UI

It's probably not much use to you unless you happen to be looking for a user interface to manage a PowerDNS authoritative server (and maybe have a particular desire to have it authenticate with LDAP and allow multiple users to be assigned as admins to specific zones). But if you are, then maybe it could be really useful. Here's the feature list:

  • Connects to PowerDNS via its JSON API.
  • Allows login managed by LDAP server.
  • Create zones; add, edit and delete records.
  • Grant multiple users access to administer a zone.
  • Lower access level that allows to view a zone and request changes.
  • Provides its own JSON API for making changes to DNS records.
  • Keeps a changelog of all DNS changes done through it.
  • (Optionally) export all zones as bind-format zone files and store changes in git.
PHP: Invalid UTF-8 characters in XML, revisited urn:uuid:c2b30e8e-62d1-1031-b176-001999635b44 2013-08-30T14:37:40Z Back in 2008, I wrote a blog post with a function to clean up UTF-8 characters in PHP that were not valid in XML. Some lines of that function no longer work in newer PHP versions due to my use of a blacklist rather than a whitelist, and PHP still doesn'…

Back in 2008, I wrote a blog post with a function to clean up UTF-8 characters in PHP that were not valid in XML. Some lines of that function no longer work in newer PHP versions due to my use of a blacklist rather than a whitelist, and PHP still doesn't seem to have a proper built-in function for this.

Various proposed solutions to this problem can be found on the net (1, 2, 3, 4), but none of those I found actually do it right (some are in fact quite badly wrong).

Here's one that I believe should handle all cases correctly, and it's a fair bit cleaner than my original one.

[code][/code]
It's About Time urn:uuid:4258a43c-152b-1030-a094-001999635b44 2012-07-02T00:16:50Z Our awesome sysadmin team did some serious overtime over the weekend, thanks to a fun little leap second bug. It took down a scary number of servers, though fortunately our most important external public services escaped largely unscathed (mostly thank…

Our awesome sysadmin team did some serious overtime over the weekend, thanks to a fun little leap second bug. It took down a scary number of servers, though fortunately our most important external public services escaped largely unscathed (mostly thanks to a high level of redundancy). I too lost a server to this bug and had to spend a little while dealing with the fallout.

Things like this do serve as an important reminder of the sometimes startling effects of invalid assumptions when applied to computers, eg. the assumption that there are always 60 seconds in a minute (though in this particular case the actual kernel bug was far more complicated than that).

Addendum: Bron Gondwana of our FastMail team has now written an excellent write-up of the leap-second incident.