xiven.com stating the blatantly obvious since 2002

SSH Key Authority demo server

Following up on the recent open source release of our SSH Key Authority tool, there is now a demonstration server available. You can find it here:

SSH Key Authority demonstration server

Use one of the following sets of username / password credentials to log in:

Posted: 2017-08-31 23:05:17 UTC by Xiven | Cross-references (0) | Comments (0)

Restoring a deleted custom field in JIRA from backups

Imagine if you will, the hypothetical scenario where your newest JIRA admin has just managed to delete a custom field from your JIRA instance. This of course means that all data stored in that field has been deleted across all issues in all of your projects, and all the schemes and configurations it was referenced by will no longer reference it.

So, after revoking this hypothetical admin's administration privileges, what is your next step?

Like any good system administrator, you obviously have full database backups running on at least a nightly basis, so almost all of the data is there and can (at least theoretically) be recovered. But you don't want to restore the entire database, otherwise you would lose all data for today, which would be very bad.

You could re-create the field through the JIRA UI and then restore the data into this new field, but that has a couple of caveats:

It would be much better if you could restore the field and all of the configuration surrounding it directly as it was before. But how will you know which data to restore, and how will you actually perform this restoration? Allow me to be your guide...

Step-by-step guide

This guide is written for JIRA admins who are using a PostgreSQL database as the backend of their JIRA instance. If you're using a different database engine you will need to make various adjustments.

  1. First things first, get that database backup file and copy it to some server or workstation where you can safely mess around with a database without consequence.
  2. On this safe place, import the file into your database: psql jira < 2017-08-31.sql
  3. Connect to postgres: psql jira
  4. Find the ID of your now-deleted field: select id from customfield where cfname = 'My custom field'; (for this hypothetical scenario, let's assume the ID was 10141).
  5. You'll also need to find the ID of any entries in the fieldconfiguration table that reference your field: select * from fieldconfiguration where fieldid = 'customfield_10141'; (we'll assume we got an ID from this query of 10131)
  6. Copy out the data from the relevant tables, filtered specifically to your custom field:
    
    create table cf_temp as select * from customfield where id = 10141;
    create table cfo_temp as select * from customfieldoption where customfield = 10141;
    create table cfv_temp as select * from customfieldvalue where customfield = 10141;
    create table fsli_temp as select * from fieldscreenlayoutitem where fieldidentifier = 'customfield_10141';
    create table cli_temp as select * from columnlayoutitem where fieldidentifier = 'customfield_10141';
    create table fli_temp as select * from fieldlayoutitem where fieldidentifier = 'customfield_10141';
    create table cc_temp as select * from configurationcontext where customfield = 'customfield_10141';
    create table fc_temp as select * from fieldconfiguration where fieldid = 'customfield_10141';
    create table gc_temp as select * from genericconfiguration where datatype = 'DefaultValue' and datakey = '10131';
    create table fcsit as select * from fieldconfigschemeissuetype where fieldconfigscheme = 10131;
    create table fcsit_temp as select * from fieldconfigschemeissuetype where fieldconfigscheme = 10131;
    
  7. Now you'll want to extract this data from postgres in the form of SQL INSERT statements. From your shell command-line:
    
    pg_dump --data-only --inserts -t cf_temp jira > cf.sql
    pg_dump --data-only --inserts -t cfo_temp jira > cfo.sql
    pg_dump --data-only --inserts -t cfv_temp jira > cfv.sql
    pg_dump --data-only --inserts -t fsli_temp jira > fsli.sql
    pg_dump --data-only --inserts -t cli_temp jira > cli.sql
    pg_dump --data-only --inserts -t fli_temp jira > fli.sql
    pg_dump --data-only --inserts -t cc_temp jira > cc.sql
    pg_dump --data-only --inserts -t gc_temp jira > gc.sql
    pg_dump --data-only --inserts -t fc_temp jira > fc.sql
    pg_dump --data-only --inserts -t fcs_temp jira > fcs.sql
    pg_dump --data-only --inserts -t fcsit_temp jira > fcsit.sql
    
  8. These SQL files are almost what you need, but they'll be using the wrong table names. Use some simple sed magic to clear those right up:
    
    sed s/cf_temp/customfield/ < cf.sql > cf-out.sql
    sed s/cfo_temp/customfieldoption/ < cfo.sql > cfo-out.sql
    sed s/cfv_temp/customfieldvalue/ < cfv.sql > cfv-out.sql
    sed s/fsli_temp/fieldscreenlayoutitem/ < fsli.sql > fsli-out.sql
    sed s/cli_temp/columnlayoutitem/ < cli.sql > cli-out.sql
    sed s/fli_temp/fieldlayoutitem/ < fli.sql > fli-out.sql
    sed s/cc_temp/configurationcontext/ < cc.sql > cc-out.sql
    sed s/gc_temp/genericconfiguration/ < gc.sql > gc-out.sql
    sed s/fc_temp/fieldconfiguration/ < fc.sql > fc-out.sql
    sed s/fcs_temp/fieldconfigscheme/ < fcs.sql > fcs-out.sql
    sed s/fcsit_temp/fieldconfigschemeissuetype/ < fcsit.sql > fcsit-out.sql
    
  9. Copy these SQL files over to your JIRA server.
  10. Back up your JIRA database! Trigger an extra special run of your trusty database backup script.
  11. Stop your JIRA instance.
  12. Run all of these shiny SQL files:
    
    psql jira < cf-out.sql
    psql jira < cfo-out.sql
    psql jira < cfv-out.sql
    psql jira < fsli-out.sql
    psql jira < cli-out.sql
    psql jira < fli-out.sql
    psql jira < cc-out.sql
    psql jira < gc-out.sql
    psql jira < fc-out.sql
    psql jira < fcs-out.sql
    psql jira < fcsit-out.sql
    
  13. Start your JIRA instance up again.
  14. Check to see if everything looks okay.
  15. Hopefully breathe a huge sigh of relief.
  16. Set a re-index going, just for good measure. This is JIRA we're talking about after all.

Posted: 2017-08-31 18:15:36 UTC by Xiven | Cross-references (0) | Comments (0)

The Keys System

Several years ago, when I was still working in the Core department at Opera Software, I became aware that the Information Services team had started using a web-based tool to manage SSH key access to root accounts on their servers (I discovered this when I needed to upload my key there to get access to a server that was shared with them).

A few reorganizations and ownership shifts later, I found myself in the team that owned this tool, known only as "the Keys system", or by its hostname "keys".

In November 2013 I embarked on a project to rewrite the Keys system, adding the ability to manage more than just the root account, and improving the user interface. Other features were added later such as group access management, SSH access options, and immediate syncing. It soon became quite well-liked within the company as a tool for people to manage access to their servers.

We've been hoping to open source this for quite some time, and after the successful open-sourcing of our PowerDNS management tool (which was actually developed after the Keys system and shares a lot of core code in common) we've been getting ready to go ahead with this one.

One of the big challenges was to come up with an actual name for the project. In the end we settled on "SSH Key Authority". Today we have released SSH Key Authority to the public on GitHub!

Posted: 2017-08-22 17:10:16 UTC by Xiven | Cross-references (1) | Comments (0)

DNS UI demo

A month ago I announced the release of the Opera DNS UI tool. The GitHub page is not really much to look at on its own though, and it's a bit difficult to get a feel for what it does and how it works without seeing it in action, so I've now set up a demonstration server so people can actually see it in action. You can find it here:

Opera DNS UI demonstration server

Use one of the following sets of username / password credentials to log in:

Posted: 2017-03-01 16:52:47 UTC by Xiven | Cross-references (0) | Comments (0)

Let's distrust StartCom. Let's encrypt instead

Recently I noticed that the Opera developer browser had started rejecting the SSL certificate used by my server for this website. It worked fine in Opera, Opera beta, Chrome and Firefox on my PC, and Qualsys SSL Labs still reported the domain with an A+ rating, but for some reason the latest Opera Developer builds simply rejected it. As I would discover later, so too did the latest Chrome Canary builds.

Why?

I certainly wondered that, so after initially coming up dry I asked for some help from colleagues at work. One very helpful person soon pinpointed the problem: the StartCom CA that my certificate was issued by has now been distrusted by Google (with good reason). Mozilla had already done this too.

Although initial reports were that certificates issued before October 21st 2016 would not be affected by this distrust, due to a number of technical limitations and concerns, Google Chrome is unable to trust all pre-existing certificates while ensuring our users are sufficiently protected from further misissuance - hence my certificate no longer being accepted.

Despite this distrust being fairly widely reported in the tech press, I had somehow managed to miss this crucial information, and I certainly didn't receive any notifications about it from StartCom themselves either. Regardless, it needed fixing so I had to find a new (ideally also free) SSL CA.

I had heard about the Let's Encrypt project before, but I hadn't really looked into it in depth. It turns out that they really have made it about as painless as it is possible to be to obtain and install SSL certificates. I opted for using the Certbot ACME client in "certonly" mode and configuring Apache manually both for the verification mechanism and installing the certificate, which was a little more work but still very easy. I now have a brand new multi-domain certificate in use by Apache, Dovecot and Postfix on my server. It should get renewed automatically when needed too with a simple cron job.

Then today I wanted to add another subdomain to the certificate. Absolutely no problem - just reissue the command with the new domain added and then reload the services. Literally as simple as that.

Posted: 2017-02-25 22:32:19 UTC by Xiven | Cross-references (0) | Comments (0)