Kwartik's Blog

March 30, 2011

Use your screensaver to display your favorite quotes !

Filed under: General — Tags: , , , , — kwartik @ 9:16 pm

Xscreensaver used to display quotes randomly Screenshot of my new screensaver

The “challenge” I faced was to configure my screensaver to display randomly my favorite quotes, and this in my Linux KDE4 based environment.Unfortunately none of the screensavers shipped with KDE4 provides this option. I was ready to develop my own screensaver when I realized that the xscreensaver project provides several screensavers designed to display text coming from the output of a custom command. Xscreensaver has the great advantage of not being distribution dependant as it is shipped on most Linux and Unix systems running the X11 Window System. The good news also was that the Mandriva distribution I use provides an xcreensaver rpm that once installed integrates the different screensavers shipped with xcreensavers directly in the KDE4 list of screensavers so there is no need to manually modify the X11 configuration.

So the remaining problem was to create a custom command to print randomly one of my favorite quotes. I first defined a simple XML format to save my quotes. Below an example of an XML file that I named quotes.xml and in which I saved two quotes to illustrate the concept:

<?xml version="1.0" encoding="UTF-8"?>

<collection>

<quote id='orwell-01' category="literature" reference="George Orwell - Animal Farm - 1945">
Man is the only creature that consumes without producing. He does not give milk, he does not lay eggs, he is too weak to pull the plough, he cannot run fast enough to catch rabbits. Yet he is lord of all the animals. He sets them to work, he gives back to them the bare minimum that will prevent them from starving, and the rest he keeps for himself
</quote>

<quote id='shakespeare-01' category="literature" reference="William Shakespeare - Hamlet - ~1600">
To be, or not to be, that is the question:
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles,
And by opposing end them? To die, to sleep,
No more; and by a sleep to say we end
The heart-ache, and the thousand natural shocks
That flesh is heir to: 'tis a consummation
Devoutly to be wished. To die, to sleep;
To sleep, perchance to dream – ay, there's the rub:
For in that sleep of death what dreams may come,
When we have shuffled off this mortal coil,
Must give us pause – there's the respect
That makes calamity of so long life.
For who would bear the whips and scorns of time,
The oppressor's wrong, the proud man's contumely,
The pangs of disprized love, the law’s delay,
The insolence of office, and the spurns
That patient merit of the unworthy takes,
When he himself might his quietus make
With a bare bodkin? Who would fardels bear,
To grunt and sweat under a weary life,
But that the dread of something after death,
The undiscovered country from whose bourn
No traveller returns, puzzles the will,
And makes us rather bear those ills we have
Than fly to others that we know not of?
Thus conscience does make cowards of us all,
And thus the native hue of resolution
Is sicklied o'er with the pale cast of thought,
And enterprises of great pith and moment,
With this regard their currents turn awry,
And lose the name of action. Soft you now,
The fair Ophelia! Nymph, in thy orisons
Be all my sins remembered.
</quote>

</collection>

Then I created a simple PERL script named gl that parses this XML file (it requires for this purpose the excellent XML::Twig perl module usually shipped with the perl-XML-Twig rpm), chooses randomly one quote and prints it to the standard output :

#!/usr/bin/perl

use strict;
use utf8;
use Encode qw(encode decode);
use Getopt::Long;
use XML::Twig; #PERL module usually shipped in the perl-XML-Twig rpms

#---------------- BEGINNING OF CONFIGURABLE SECTION -----------------------------#

my $xml_collection = '/home/kwartik/quotes.xml';

my $encoding='utf8';

#---------------- END OF CONFIGURABLE SECTION -----------------------------------#

GetOptions(
    "c|collection=s" => \$xml_collection,
    "e|encoding=s"  => \$encoding,
);

if ( $xml_collection eq "" )
{
  print
     "SYNTAX : gl [-c|--collection path_to_xml_collection] [-e|--encoding encoding]\n\n"
    ."encoding option examples : 'utf8' 'iso-8859-1'\n";
  exit 1;
}

(-f $xml_collection) or die ("Error: collection $xml_collection is not a file.\n");

my $nb_entries = 0;
my %hash_collection = ();

sub quote
{
    my( $twig, $item )= @_;
    my $id = $item->att('id');
    my $category  = $item->att('category');
    my $reference  = $item->att('reference');
    $hash_collection {$id} {'category'}  =  $category;
    $hash_collection {$id} {'reference'}  =  $reference;
    $hash_collection {$id} {'quote'}  =  $item->text;
    $twig->purge;
    $nb_entries ++;
}

my $twig= new XML::Twig( twig_handlers => { 'quote' => \&quote }   );
$twig->parsefile( $xml_collection );

sub print_quote( $ )
{
	my $entry = $_[0];
    my $category = $hash_collection{$entry}{'category'};
    my $quote = $hash_collection{$entry}{'quote'};
    my $reference = $hash_collection{$entry}{'reference'};
    printf("%s (%s)\n", encode($encoding, $quote) , encode($encoding, $reference) );
}

my $choice = int(rand($nb_entries));
my @entry = ( sort {$a cmp $b} keys %hash_collection );
print_quote($entry[$choice]);
exit 0;

print "\n";

The final step is to link xscreensaver to gl. I first chose the Phosphor screensaver in the KDE4 screensaver list (accessible in “Configure your desktop/desktop/screensaver”).

Phosphor screensaver appearing in the KDE4 screensaver list once xscreensaver installedSelection of Phosphor screensaver in the KDE4 screensaver list once the xscreensaver rpm has been installed

Then I used the screensaver-demo command to configure the Phosphor screensaver.

Phosphor configuration with sxcreensaver-demoConfiguration of the phosphor screensaver with the xscreensaver-demo command

The advanced tab allows to specify the custom command that xscreensaver must call to display text.

Configuration of the phosphor screensaver with the xscreensaver-demo command (advanced tab)

In the Program field, I have set the value sleep3;/usr/local/bin/gl -e iso-8859-1 to configure Phosphor to print every 3 seconds the output of gl. The -e iso-8859-1 option is used to make sure that gl prints the content in ISO-8859-1 (otherwise accents and other diacritics are badly displayed by Phosphor). It is important to notice in this example that both my XML collection file and gl script have been saved with in UTF-8. I could have also set sleep3;clear;/usr/local/bin/gl -e iso-8859-1 to ask xscreensaver to clear the screen between each quote. I have also discovered that the fortune command (usually shipped with the fortune-mod rpm) is a tool that provides many possibilities to randomly display quotes and it was indeed proposed by default by xscreensaver.

November 12, 2010

Have your repetitive tasks reminded by Check-Tasks

Filed under: Bash — Tags: , , , , — kwartik @ 11:56 pm

I have released CHECK-TASKS which is a simple BASH script intended to be used as a reminder for repetitive tasks. The tasks to remind are listed in a syntax inspired from the crontab syntax. When CHECK-TASKS is executed, it identifies the tasks to remind based on the current date and sends an email notification ONLY IF THE NOTIFICATION HAS NOT ALREADY BEEN SENT ON THE SAME DAY. The idea is to execute CHECK-TASKS AT BOOT TIME (OR SESSION START TIME for users that do not have administrative privileges), so in case of several boots in one single day, the notifications are only sent at the first boot. It is also possible to configure CHECK-TASKS to execute scripts.

An example to illustrate the concept

Let consider the following task list definition

task_list=(
*:*:7:“Jogging time”:
01:*:*:“Pay child’s nurse”:
12:4:*:“Bob’s birthday”:
*:*:*::/path/to/backup-script.sh)

CHECK-TASKS will send a reminder :
- every Sunday to notify that it is jogging time
- every first day of the month to notify that it is time to pay the  child’s nurse
- every 12th of April to notify that it is Bob’s birthday

CHECK-TASKS will also execute :
- every day the backup script named backup-script.sh

How to configure and use Check-Tasks ?

  1. Download the script from sourceforge.
  2. Save this file in a local directory and make sure that it is executable (use chmod +x /path/to/check-tasks.sh).
  3. Edit the script with a text editor of your choice and adapt the configurable section at the beginning of the script.
  4. Execute manually check-tasks.sh to see if it works.
  5. Configure one of your boot scripts or session startup scripts to execute check-tasks.sh  (See examples in the next paragraph).

Examples to configure an automatic execution of Check-Tasks

An easy solution is to configure your window manager to execute check-tasks.sh during the session start. For example, under KDE, simply create a link to check-tasks.sh in your ~/.kde/Autostart directory (ln -s /path/to/check-tasks.sh  ~/.kde/Autostart/check-tasks.sh).

An other possible approach is to create a service script in the /etc/rc.d/init.d directory. An example named ‘after_boot’ operational on a Linux Mandriva 2010 system is given below. Use the chkconfig command to configure your system to execute this script (chkconfig –add after_boot).

#! /bin/sh
# chkconfig: 5 99 99
### BEGIN INIT INFO
# Provides: after_boot
# Required-Start:
# Required-Stop:
# Default-Start: 5
# Description: procedure executed at the end of the boot
### END INIT INFO

_service_name="after_boot"

# Source function library.
. /etc/init.d/functions

start() {
        gprintf "Starting $_service_name..."
        sleep 120
        /bin/su -c "/usr/local/bin/check-tasks.sh" kwartik
        success "%s startup" "$base"
        return 0
}

stop() {
    gprintf "Stopping $_service_name..."
    success "%s startup" "$base"
    return 0
}

case "$1" in

    'start')
        start
        ;;

    'stop')
        stop
        ;;

    'restart')
        stop
        start
        ;;

    *)
        echo "Usage: $0 { start | stop | restart }"
        exit 1
        ;;
esac

exit 0
<pre style=’color:#231f20;background-color:#ffffff;’>
<i><span style=’color:#888786;’>#! /bin/sh</span></i>
<i><span style=’color:#888786;’># chkconfig: 5 99 99</span></i>
<i><span style=’color:#888786;’>### BEGIN INIT INFO</span></i>
<i><span style=’color:#888786;’># Provides: after_boot</span></i>
<i><span style=’color:#888786;’># Required-Start:</span></i>
<i><span style=’color:#888786;’># Required-Stop:</span></i>
<i><span style=’color:#888786;’># Default-Start: 5</span></i>
<i><span style=’color:#888786;’># Description: procedure executed at the end of the boot</span></i>
<i><span style=’color:#888786;’>### END INIT INFO</span></i> 

<span style=’color:#006e28;’>_service_name=</span><span style=’color:#bf0303;’>&quot;after_boot&quot;</span>

<i><span style=’color:#888786;’># Source function library.</span></i>
<b><span style=’color:#880088;’>.</span></b> /etc/init.d/functions

<span style=’color:#800080;’>start()</span> <b>{</b>
gprintf <span style=’color:#bf0303;’>&quot;Starting </span><span style=’color:#006e28;’>$_service_name</span><span style=’color:#bf0303;’>…&quot;</span>
/bin/date <b><span style=’color:#223388;’>&gt;&gt;</span></b> /var/log/after_boot.log
<b><span style=’color:#cc00cc;’>sleep</span></b> 120
/bin/su -c <span style=’color:#bf0303;’>&quot;/data/bin/check-tasks.sh&quot;</span> kwartik
success <span style=’color:#bf0303;’>&quot;%s startup&quot;</span> <span style=’color:#bf0303;’>&quot;</span><span style=’color:#006e28;’>$base</span><span style=’color:#bf0303;’>&quot;</span>
<b><span style=’color:#880088;’>return</span></b> 0
<b>}</b>

<span style=’color:#800080;’>stop()</span> <b>{</b>
gprintf <span style=’color:#bf0303;’>&quot;Stopping </span><span style=’color:#006e28;’>$_service_name</span><span style=’color:#bf0303;’>…&quot;</span>
success <span style=’color:#bf0303;’>&quot;%s startup&quot;</span> <span style=’color:#bf0303;’>&quot;</span><span style=’color:#006e28;’>$base</span><span style=’color:#bf0303;’>&quot;</span>
<b><span style=’color:#880088;’>return</span></b> 0
<b>}</b>

<b>case</b> <span style=’color:#bf0303;’>&quot;</span><span style=’color:#006e28;’>$1</span><span style=’color:#bf0303;’>&quot;</span><b> in</b>

<span style=’color:#bf0303;’>’start’</span><b>)</b>
start
<b>;;</b>

<span style=’color:#bf0303;’>’stop’</span><b>)</b>
stop
<b>;;</b>

<span style=’color:#bf0303;’>’restart’</span><b>)</b>
stop
start
<b>;;</b>

*<b>)</b>
<b><span style=’color:#880088;’>echo</span></b> <span style=’color:#bf0303;’>&quot;Usage: </span><span style=’color:#006e28;’>$0</span><span style=’color:#bf0303;’> { start | stop | restart }&quot;</span>
<b><span style=’color:#880088;’>exit</span></b> 1
<b>;;</b>
<b>esac</b>

<b><span style=’color:#880088;’>exit</span></b> 0
</pre>

September 22, 2010

TrivialConfig – a generic configuration file parser

Filed under: Perl — Tags: , , — kwartik @ 7:51 pm

We will see in this article how it is possible to create a powerful and extendable configuration file parser with less than one hundred lines of PERL code.

There are at least two advantages of using a generic parser :

  1. There is no need to reinvent the wheel for each new program.
  2. If the parser has been designed as an object, it’s reference can be easily passed to different components of a code.

It is for these two reasons that I have created the TrivialConfig PERL module.

Let start with an example to illustrate the advantages. Let consider a program designed to manage a set of remote servers. This program may require as input hardware details on all the servers managed. Moreover, all the program input may be organized in several profiles and categories. For example, a pre-defined profile for the production servers and an other one for the development servers may be created. A specific category may also be defined to save the system parameters that should be used by the program.

All these configuration details can be saved in the following configuration file (let name it config.conf) using the TrivialConfig format :

## BEGINNING OF GENERIC SYSTEM PARAMETERS ###
newcategory system
    # run_as: idendity under witch the program should be run
    run_as = root

    # ssh_path: path to SSH binary
    ssh_path = /usr/bin/ssh
endcategory system
## END OF GENERIC SYSTEM PARAMETERS ###

## BEGINNING OF CONFIGURATION PROFILES ###
newcategory profile_prod
    memory = 1024
    timeout = 300
endcategory profile_prod

newcategory profile_dev
    memory = 512
    timeout = 200
endcategory profile_dev
## END OF CONFIGURATION PROFILES ###

## BEGINNING OF SERVER CONFIGURATIONS ###
newcategory conf_server_alpha
    profile = profile_dev
endcategory conf_server_alpha

newcategory conf_server_beta
    profile = profile_dev
endcategory conf_server_beta

newcategory conf_server_gamma
    profile = profile_prod
endcategory conf_server_gamma
## END OF SERVER CONFIGURATIONS ###

Once this file defined, it can be easily parsed by just creating a new TrivialConfig object and invoking the parse() method :

my $config = new TrivialConfig(/path/to/config.conf );
$config->parse();

At this stage, the file has been parsed, which means that its content has been loaded in memory. Once this first parsing step done, the access to a parameter is trivial. For this, the get_category_param method must be called. For example the ‘run_as’ parameter of the ‘system’ category can be fetched with the following call :

my $run_as = $config->get_category_param(system, run_as);

Another great thing with TrivialConfig is that it is possible to extend the basic generic feature by subclassing the TrivialConfig class. It may be useful in our example to define methods to access the configuration data in a program specific way. For example two custom methods, get_memory_for_server and get_timeout_for_server, may be useful to access directly to the memory or timeout parameters of a server without having first to identify the profile it belongs to. A subclass implementing these two methods can be created in just 15 lines of code as shown in the example below :

package ExampleConfig;
use TrivialConfig;
use warnings;
our @ISA = qw(TrivialConfig);

sub get_memory_for_server ( $ $ ) {

    my ($this, $server) = @_;
    my $server_profile = $this->get_category_param('conf_server_'.$server, 'profile');
    return $this->get_category_param($server_profile, 'memory');
}

sub get_timeout_for_server ( $ $ ) {

    my ($this, $server) = @_;
    my $server_profile = $this->get_category_param('conf_server_'.$server, 'profile');
    return $this->get_category_param($server_profile, 'timeout');
}

# DON'T SUPPRESS THE FOLLOWING LINE => REQUIRED BY PERL OBJECT SEMANTIC CONVENTIONS.
1;

The following parse.pl script shows how this new subclass can be used :

#!/usr/bin/perl -w
use strict;

use lib '/path/to/parser/directory';
use ExampleConfig;

my $config = new ExampleConfig(
    '/path/to/config.conf'
);
$config->parse();

my $run_as = $config->get_category_param('system', 'run_as');
print ">> system::run_as=$run_as\n";

my @servers = ('alpha', 'beta', 'gamma');

for my $server (@servers) {
    print '-' x 20, ' ', uc($server), ' ', '-' x 20, "\n";

    my $memory = $config->get_memory_for_server($server);
    printf " %10s = %s\n", 'memory', $memory;

    my $timeout = $config->get_timeout_for_server($server);
    printf " %10s = %s\n", 'timeout', $timeout;
}

Here is the output that will produce  parse.pl when executed :
$ ./parse.pl
>> system::run_as=root
——————– ALPHA ——————–
memory = 512
timeout = 200
——————– BETA ——————–
memory = 512
timeout = 200
——————– GAMMA ——————–
memory = 1024
timeout = 300

The code of TrivialConfig.pm is given below. Enjoy !

# DESCRIPTION:  
# 
# TrivialConfig is a generic class to parse a configuration file using
# the "TrivialConfig" format described below. 
#
#         The TrivialConfig class provides two methods:
#             -   parse() to parse the configuration file and load its
#                 content in memory
#             -   get_category_param($category, $param) to access to the 
#                 content of a specific parameter.
#         It is very easy to create a custom class extending
#         TrivialConfig class, and providing custom methods.
#         See http://kwartik.wordpress.com/trivial-config-parser for a detailed example.
#
#
# TRIVIALCONFIG FORMAT:
#
# Each parameter defined in this configuration file must belong
# to a category. Each category must have :
#   - an opening tag (newgategory category_name)
#   - a closing tag (endcategory category_name)
#
# It is not possible to have subcategories.
#
# The two possible syntaxes to specify a parameter are :
#   parameter = value
#   parameter = "value whith spaces"
#
# All lines starting with a # are considered by the parser as comments.
#
# EXAMPLE:
# newcategory system
#	# run_as: idendity under witch the program should be run
#	run_as = root
#
#	# ssh_path: path to SSH binary
#	ssh_path = /usr/bin/ssh
#
#       # email_subjects_refix: prefix that must be used in the email notifications
#       email_subjects_refix = "[My Prefix] "
#
# endcategory system
#
#
# Version: 1.0
#
# Author : kwartik@gmail.com
# Date of creation : 2010-09-22
#
# History of modifications :
# 2010-09-22: v1.0, initial release
##############################################################################

package TrivialConfig;
use warnings;
my $debug = 0;

sub new()
{
    my ($class) = @_;
    my $this = {};
    bless($this, $class);

    my %hash_categories = ();

    $this->{CONFIGURATION_FILE} =  $_[1];
    $this->{HASH_CATEGORIES} = \%hash_categories;

    return $this;
}

sub parse ()
{
    my ($this) = @_;
    my $configuration_file = $this->{CONFIGURATION_FILE};

    my $nb_lines = 0;
    my $context = '';
    my $sub_context = '';
    if ( $configuration_file ne '' and -f $configuration_file ) {
        open(FICCONF, "<$configuration_file")
            or die "impossible to open $configuration_file\"\"", $!;
        while ( <FICCONF> ) {
            my $line=$_;
            $nb_lines ++;
            chomp($line);

            next if ( $line =~ /^\s*#|^\s*$/ );

            if ( $line =~ /^\s*newcategory\s*\S*$/ ) {
                $context = 'category';
                if ( $line =~ /^\s*newcategory\s*\S+$/ ) {
                    ($sub_context) = ( $line =~ /^\s*newcategory\s*(\S+)$/ );
                        printf "\n>> NEW CATEGORY: |$sub_context|\n"
                            if ( $debug );
                }
                else {
                    print 
                         "FATAL ERROR in configuration file at line $nb_lines : "
                        ,"name of category not specified";
                    exit 3;
                }

            }
            elsif ( $line =~ /^\s*endcategory\s*\S*$/ ) {
                $context = '';
                my $end_sub_context = '';
                if ( $line =~ /^\s*endcategory\s*\S+$/ ) {
                    ($end_sub_context) = ( $line =~ /^\s*endcategory\s*(\S+)$/ );
                        printf ">> END CATEGORY: |$end_sub_context|\n" if ( $debug );
                }
                else {
                    print
                        "FATAL ERROR in configuration file at line $nb_lines : "
                        ,"name of category not specified\n";
                    exit 3;
                }
                if ( $end_sub_context ne $sub_context ) {
                    print
                        "FATAL ERROR in configuration file at line $nb_lines : "
                        ,"wrong endcategory (expecting $sub_context "
                        ,"and not $end_sub_context)\n";
                    exit 4;
                }
                $sub_context = '';
            }
            elsif  (
                $context eq 'category' 
                and $line =~ /^\s*\S+\s*=\s*".*"$|^\s*\S+\s*=\s*.*$/
            ) {
                printf ">>> $line\n" if ( $debug );
                my ($param, $value) = ('', '');
                if ( $line =~ /^\s*\S+\s*=\s*".*"$/ ) {
                    ($param, $value) = ( $line =~ /^\s*(\S+)\s*=\s*"(.*)"$/ );
                }
                elsif ( $line =~ /^\s*\S+\s*=\s*.*$/ ) {
                    ($param, $value) = ( $line =~ /^\s*(\S+)\s*=\s*(.*)$/ );
                }
                printf "===>$context:$sub_context:$param=|$value|\n" if ( $debug );
                $this->{HASH_CATEGORIES}{$sub_context}{$param} = $value;
            }

        }
        close(FICCONF);
    }

}

sub get_category_param ( $ $ )
{
    my ($this, $category, $param) = @_;
    return undef if ( ! defined $this->{HASH_CATEGORIES}{$category});
    return undef if ( ! defined $this->{HASH_CATEGORIES}{$category}{$param});
    return $this->{HASH_CATEGORIES}{$category}{$param};
}

# DON'T SUPPRESS THE FOLLOWING LINE => REQUIRED BY PERL OBJECT SEMANTIC CONVENTIONS.
1;

June 21, 2010

Editing a MySQL table with full historization using stored routines

Filed under: MySQL — Tags: , , , — kwartik @ 8:59 pm

MySQL offers a great feature which is the possibility to implement stored routines. Stored routines allows to code all the logic related to the database model directly in SQL. Stored routines offer :

  • Security : Stored routines are in general not vulnerable to SQL injection attacks.
  • Data coherency : All the database update logic is located within the stored routines. Thus, the application layer don’t need to have the knowledge of the database internals like the relation between tables or the different tables that need to be updated for a given operation.The database is exposed to the application as a set of procedures. This concept is very close to the API (Application Programming Interface) one.
  • Performances: as the SQL logic is directly embedded inside the database, performances are maximal.

For all these reasons, it is generally a very good idea to use as much as possible stored routines, at least for all the UPDATE operations. We will see in this article how stored routines can be very useful and practical to edit a table and to keep track of all the modifications (who modified a record, how and when ?) in a separate historization table.

I. A bit of ternary logic

The main difficulty I faced to establish a historization logic is the ternary logic of MySQL. MySQL distinguishes indeed the ZERO value (0 for INTEGER, ” for VARCHAR, ’0000-00-00′ for DATES) from the NULL value. If A is equal to NULL and B is not equal to NULL, than the test A!=B will return FALSE !

Therefore, the test A!=B has to be translated in SQL to:

( A != B ) OR (A IS NOT NULL AND B IS NULL) OR (A IS NULL AND B IS NOT NULL)

II. Use case

The following use case will serve as a basis to establish the historization logic. Let consider a simple SQL table named users having five different fields :

mysql> desc users;
+------------+------------------+------+-----+---------+----------------+
| Field      | Type             | Null | Key | Default | Extra          |
+------------+------------------+------+-----+---------+----------------+
| id         | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| first_name | varchar(50)      | YES  |     | NULL    |                |
| last_name  | varchar(50)      | NO   |     | NULL    |                |
| is_staff   | tinyint(1)       | NO   |     | 0       |                |
| birth_date | date             | YES  |     | NULL    |                |
+------------+------------------+------+-----+---------+----------------+

Id is the primary key, first_name and last_name are of type VARCHAR, is_staff is boolean, birthday is a date.

The goal is to define a stored routine named edit_users that will be the unique procedure to call to perform modifications on table users. This same procedure will historize all the modifications performed  on users in a table called users_history :

mysql> desc users_history;
+-----------------+------------------+------+-----+---------+----------------+
| Field           | Type             | Null | Key | Default | Extra          |
+-----------------+------------------+------+-----+---------+----------------+
| id              | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| requestor_id    | int(10) unsigned | NO   |     | NULL    |                |
| user_id         | int(11)          | NO   |     | NULL    |                |
| action_flag     | varchar(40)      | NO   |     | NULL    |                |
| order_date      | datetime         | NO   |     | NULL    |                |
| processing_date | datetime         | YES  |     | NULL    |                |
| first_name      | varchar(50)      | YES  |     | NULL    |                |
| last_name       | varchar(50)      | YES  |     | NULL    |                |
| is_staff        | tinyint(1)       | YES  |     | NULL    |                |
| birth_date      | date             | YES  |     | NULL    |                |
+-----------------+------------------+------+-----+---------+----------------+

We can find in this table the five fields of the first table : users_id (and not id which is the primary key of users_history), first_name, last_name, is_staff, birth_date. The other fields will be used to save the “historization context” :

  • requestor_id is the id that identifies the person that performed the modification
  • action_flag is used to save a keyword that identifies the type of modification performed on users (‘User_Create’, ‘User_Modify’)
  • order_date is used to save the date of the modification
  • processing_date will not be used in this example, but the idea is to use this field to manage the case when a modification request requires a additional action from a piece of code external to the database.

The idea is to have a edit_user procedure clever enough to manage transparently users and users_history tables ; edit_users should take six parameters as input :

  • parameter 1 : key (id) of the users entry to modify. If NULL is set, than users_history will consider that a new user should be created (A new id will be automatically created for the user)
  • parameter 2 : new first_name value to possibly update in users and historize in users_history. The term possibly means that edit_users should be clever enough to ignore the update and historization if users(id)->firstname is already set to parameter 2. If the value NULL is used for parameter 2, edit_users should ignore completely the check : the first_name will not be updated.
  • parameter 3 : new last_name value to possibly update in users and historize in users_history.
  • parameter 4 : new is_staff value to possibly update in users and historize in users_history.
  • parameter 5 : new birth_date value to possibly update in users and historize in users_history.
  • parameter 6 : id of the person that performed the modification (needed by users_history).

So with the above logic a creation of a user should be done with only one call :

mysql> CALL edit_user(NULL, 'John', 'Doe', 1, '1978-04-03', 2);
mysql> select * from users;
+----+------------+-----------+----------+------------+
| id | first_name | last_name | is_staff | birth_date |
+----+------------+-----------+----------+------------+
|  1 | John       | DOE       |        1 | 1978-04-03 |
+----+------------+-----------+----------+------------+

mysql> select * from users_history;
+----+--------------+---------+-------------+---------------------+---------------------+------------+-----------+----------+------------+
| id | requestor_id | user_id | action_flag | order_date          | processing_date     | first_name | last_name | is_staff | birth_date |
+----+--------------+---------+-------------+---------------------+---------------------+------------+-----------+----------+------------+
|  1 |            2 |       1 | User_Create | 2010-06-28 20:40:24 | 2010-06-28 20:40:24 | John       | DOE       |        1 | 1978-04-03 |
+----+--------------+---------+-------------+---------------------+---------------------+------------+-----------+----------+------------+

A modification of  this entry should be as simple as this :

mysql> CALL edit_user(1, 'Johny', 'Doe', 1, '1978-04-03', 2);

mysql> select * from users;
+----+------------+-----------+----------+------------+
| id | first_name | last_name | is_staff | birth_date |
+----+------------+-----------+----------+------------+
|  1 | Johny      | DOE       |        1 | 1978-04-03 |
+----+------------+-----------+----------+------------+

mysql> select * from users_history;
+----+--------------+---------+-------------+---------------------+---------------------+------------+-----------+----------+------------+
| id | requestor_id | user_id | action_flag | order_date          | processing_date     | first_name | last_name | is_staff | birth_date |
+----+--------------+---------+-------------+---------------------+---------------------+------------+-----------+----------+------------+
|  1 |            2 |       1 | User_Create | 2010-06-28 20:40:24 | 2010-06-28 20:40:24 | John       | DOE       |        1 | 1978-04-03 |
|  2 |            2 |       1 | User_Modify | 2010-06-28 21:31:17 | 2010-06-28 21:31:17 | Johny      | NULL      |     NULL | NULL       |
+----+--------------+---------+-------------+---------------------+---------------------+------------+-----------+----------+------------+
2 rows in set (0.00 sec)

We can see in the previous example that edit_users has correctly updated the modified fields (first_name) in users. It has also historized in users_history all the fields that have changed. As the NULL value is used to indicate that the update should be ignored (except for the first parameter where NULL means create a new entry), this last example could have also been successfully performed with the following call :

mysql> CALL edit_user(1, 'Johny', NULL, NULL, NULL, 2);

We have now defined the concept of a convenient procedure that can be used to create and historize a new entry as well as to modify and historize an arbitrary number of fields. We will formalize a little bit more the concept in the next section to establish the logic that will finally be implemented as a MySQL stored routine.

III. Specification of the historization logic

Based on the above example, let consider all the possibilities that we would like to have for the couples (initial value L, new value N) and let list for each couple, the update value U that we would like to update in the data table (users) and the differential value H to historize in the historization table (users_history). As in the example above, we will consider that N=NULL means that the field should be ignored. We will also consider that H=NULL means ‘historization is not needed’ and that C or C’ means ‘any value not NULL and not equal to ZERO.

List of (L, N) couples with update and historization actions wanted :

(NULL, NULL) => Nothing to do
(NULL, ZERO) => U=NULL (*), H=NULL (**)
(NULL, C) => U=C , H=C
(ZERO, NULL) => Nothing to do
(ZERO, ZERO) => Nothing to do
(ZERO, C) => U=C , H=C
(ZERO, NULL) => Nothing to do
(C, ZERO) => U=NULL (*), H=ZERO
(C, NULL) => Nothing to do
(C, C) => Nothing to do
(C, C') => U=C', H=C'

(*) If NULL value is not authorised by the table schema, MySQL set automatically the ZERO value and emits a warning.

(**) It seems preferable here not to historize because we would often have in practise a useless historization entry. It is indeed often the case that a field is not set in an entry.

The test IF ( N IS NOT NULL AND L != N  ) allows us to identify the cases where an update and historization action is required :

(NULL, ZERO) => U=NULL (***), H=NULL
(C, ZERO) => U=NULL (***), H=ZERO
(NULL, C) => U=C , H=C
(ZERO, C) => U=C , H=C
(C, C') => U=C', H=C'

(***) If NULL value is not authorised by the table schema, MySQL set automatically the ZERO value and emits a warning

We can therefore establish the following logic :

IF ( N IS NOT NULL AND L != N  ) THEN
  IF ( N = ZERO AND L IS NULL ) THEN
    U=NULL, H=NULL
  ELSEIF (N = ZERO AND L IS NOT NULL)  THEN
    U=NULL, H=ZERO
  ELSE THEN
    U=N, H=N
  END IF;
END IF;

In SQL ternary logic , the first line of this logic should be translated to :

N IS NOT NULL AND (( L != N ) OR (L IS NOT NULL AND N IS NULL) OR (L IS NULL
AND N IS NOT NULL) ))

Which hopefully can be simplified to :
N IS NOT NULL AND ( L != N OR L IS NULL )

Finally, we have established :

IF ( N IS NOT NULL AND ( L != N OR L IS NULL ) )
  IF ( N = ZERO AND L IS NULL ) THEN
    U=NULL, H=NULL
  ELSEIF (N = ZERO AND L IS NOT NULL)  THEN
    U=NULL, H=ZERO
  ELSE THEN
    U=N,H=N
  END IF;
END IF;

IV. Implementation of this logic

DELIMITER $$

/*******************************************************/
-- > Emulation of an error signal. This emulation will not be required anymore 
-- in MySQL 5.2 that is expected to implement the SIGNAL directive.
SELECT 'emulate_signal' AS Proc
$$
DROP PROCEDURE IF EXISTS emulate_signal
$$
CREATE PROCEDURE emulate_signal (IN in_errortext VARCHAR(255))
BEGIN
     SET @sql=CONCAT('UPDATE `', in_errortext, '` SET xyz=1');
     PREPARE my_signal_stmt FROM @sql;
     EXECUTE my_signal_stmt;
     DEALLOCATE PREPARE my_signal_stmt;
 END
$$

/********************************
 * Procedure called by edit_user
 ********************************/
SELECT 'check_diff_varchar' AS Proc
$$
DROP PROCEDURE IF EXISTS check_diff_varchar
$$
CREATE PROCEDURE check_diff_varchar (
    IN in_var_old VARCHAR(1024),
    IN in_var_new VARCHAR(1024),
    OUT out_var_update VARCHAR(1024),
    OUT out_var_diff VARCHAR(1024),
    OUT out_is_update_required BOOL
)
BEGIN
    SET out_is_update_required = 0;
    IF ( in_var_new IS NOT NULL AND ( in_var_new != in_var_old OR in_var_old IS NULL ) ) THEN
        IF ( in_var_new = '' ) THEN
            SET out_is_update_required = 1;
            SET out_var_update = NULL; 
            IF ( in_var_old IS NULL ) THEN
               SET out_var_diff = NULL;
            ELSE
                SET out_var_diff = '';
            END IF;

        ELSE
            SET out_is_update_required = 1;
            SET out_var_update = in_var_new;
            SET out_var_diff = in_var_new;
        END IF;
    END IF;
END
$$
SHOW WARNINGS
$$

/********************************
 * Procedure called by edit_user
 ********************************/
SELECT 'check_diff_date' AS Proc
$$
DROP PROCEDURE IF EXISTS check_diff_date
$$
CREATE PROCEDURE check_diff_date (
    IN in_var_old DATE,
    IN in_var_new DATE,
    OUT out_var_update DATE,
    OUT out_var_diff DATE,
    OUT out_is_update_required BOOL
)
BEGIN
    SET out_is_update_required = 0;
    IF ( in_var_new IS NOT NULL AND ( in_var_new != in_var_old OR in_var_old IS NULL ) ) THEN
        IF ( in_var_new = '000-00-00' ) THEN
            SET out_is_update_required = 1;
            SET out_var_update = NULL; 
            IF ( in_var_old IS NULL ) THEN
               SET out_var_diff = NULL;
            ELSE
                SET out_var_diff = '0000-00-00';
            END IF;

        ELSE
            SET out_is_update_required = 1;
            SET out_var_update = in_var_new;
            SET out_var_diff = in_var_new;
        END IF;
    END IF;
END
$$
SHOW WARNINGS
$$

/********************************
 * Procedure called by edit_user
 ********************************/
SELECT 'check_diff_integer' AS Proc
$$
DROP PROCEDURE IF EXISTS check_diff_integer
$$
CREATE PROCEDURE check_diff_integer (
    IN in_var_old INTEGER,
    IN in_var_new INTEGER,
    OUT out_var_update INTEGER,
    OUT out_var_diff INTEGER,
    OUT out_is_update_required BOOL
)
BEGIN
    SET out_is_update_required = 0;
    IF ( in_var_new IS NOT NULL AND ( in_var_new != in_var_old OR in_var_old IS NULL ) ) THEN
        IF ( in_var_new = 0 ) THEN
            SET out_is_update_required = 1;
            SET out_var_update = NULL; 
            IF ( in_var_old IS NULL ) THEN
               SET out_var_diff = NULL;
            ELSE
                SET out_var_diff = 0;
            END IF;

        ELSE
            SET out_is_update_required = 1;
            SET out_var_update = in_var_new;
            SET out_var_diff = in_var_new;
        END IF;
    END IF;
END
$$
SHOW WARNINGS
$$

/********************************
 * Procedure edit_user
 ********************************/
SELECT 'edit_user' AS Proc
$$
DROP PROCEDURE IF EXISTS edit_user
$$
CREATE PROCEDURE edit_user (
    IN in_id INTEGER,
    IN in_first_name VARCHAR(50),
    IN in_last_name VARCHAR(50),
    IN in_is_staff BOOL,
    IN in_birth_date DATE,
    IN in_requestor_id INTEGER
)
BEGIN
    DECLARE v_id INTEGER;
    DECLARE v_first_name VARCHAR(50);
    DECLARE v_first_name_update VARCHAR(50);
    DECLARE v_first_name_diff VARCHAR(50);
    DECLARE v_last_name VARCHAR(50);
    DECLARE v_last_name_update VARCHAR(50);
    DECLARE v_last_name_diff VARCHAR(50);
    DECLARE v_is_staff BOOL;
    DECLARE v_is_staff_update BOOL;
    DECLARE v_is_staff_diff BOOL;
    DECLARE v_birth_date DATE;
    DECLARE v_birth_date_update DATE;
    DECLARE v_birth_date_diff DATE;

    DECLARE v_is_update_required BOOL;
    DECLARE v_new_id INTEGER;

    DECLARE v_error_msg VARCHAR(255);
    DECLARE EXIT HANDLER FOR SQLEXCEPTION
    BEGIN
        ROLLBACK;
        CALL emulate_signal(v_error_msg);
    END;
    START TRANSACTION;
    SET v_error_msg = 'edit_user : rollback';

    --  modification of en entry
    IF ( in_id IS NOT NULL ) THEN
        SELECT id, first_name, last_name, is_staff, birth_date INTO v_id, v_first_name, v_last_name, v_is_staff, v_birth_date FROM users WHERE id=in_id;

        CALL check_diff_varchar(v_first_name, in_first_name, v_first_name_update, v_first_name_diff, v_is_update_required);
        IF ( v_is_update_required = 1 ) THEN
            UPDATE users SET first_name=v_first_name_update WHERE id = in_id;
        END IF;

        CALL check_diff_varchar(v_last_name, UPPER(in_last_name), v_last_name_update, v_last_name_diff, v_is_update_required);
        IF ( v_is_update_required = 1 ) THEN
            UPDATE users SET last_name=v_last_name_update WHERE id = in_id;
        END IF;

        CALL check_diff_date(v_birth_date, in_birth_date, v_birth_date_update, v_birth_date_diff, v_is_update_required);
        IF ( v_is_update_required = 1 ) THEN
            UPDATE users SET birth_date=v_birth_date_update WHERE id = in_id;
        END IF;

        CALL check_diff_integer(v_is_staff, in_is_staff, v_is_staff_update, v_is_staff_diff, v_is_update_required);
        IF ( v_is_update_required = 1 ) THEN
            UPDATE users SET is_staff=v_is_staff_update WHERE id = in_id;
        END IF;

        -- Differential historization
        IF ( v_first_name_diff IS NOT NULL OR v_last_name_diff IS NOT NULL OR v_is_staff_diff IS NOT NULL OR v_birth_date_diff IS NOT NULL ) THEN
                INSERT INTO users_history (requestor_id, user_id, order_date, processing_date, action_flag, first_name, last_name, is_staff, birth_date) VALUES(in_requestor_id, in_id, NOW(), NOW(), 'User_Modify', v_first_name_diff, UPPER(v_last_name_diff), v_is_staff_diff, v_birth_date_diff);
        END IF;

    -- Creation of an entry
    ELSE

        INSERT INTO users (first_name, last_name, is_staff, birth_date) VALUES(in_first_name,  UPPER(in_last_name), in_is_staff, in_birth_date );

        SELECT MAX(id) INTO v_new_id FROM users;

        INSERT INTO users_history (requestor_id, user_id, order_date, processing_date, action_flag, first_name, last_name, is_staff, birth_date) VALUES(in_requestor_id, v_new_id, NOW(), NOW(), 'User_Create', in_first_name, UPPER(in_last_name), in_is_staff, in_birth_date);

    END IF;

    COMMIT;

END
$$
SHOW WARNINGS
$$

April 13, 2010

Protect your data with a blowfish

Filed under: Bash — Tags: , , , — kwartik @ 8:45 pm

I have just released safe-archive, a command line based tool for any LINUX/UNIX system providing a simple and convenient interface to manage secured archives. It allows to create/extract/modify encrypted and compressed tar files. The syntax is very much inspired by the tar command. It uses an extremely strong encryption algorithm, the Blowfish cipher (CBC mode) provided by Openssl.

A blowfishA blowfish by saragoldsmith.

The following example shows how easy it is to create a secured archive, list its content and extract (decrypt) it :

$ safe-archive.sh -h

safe-archive.sh: management of secured archives

Syntax:

To create an archive of a file or direcory: safe-archive.sh -c -f archive_name file_or_directory

To list the content of an archive: safe-archive.sh -t -f archive_name

To extract the content of an archive: safe-archive.sh -x -f archive_name

To add a file or a directory to an archive: safe-archive.sh -r -f archive_name file_or_directory

To display this help: safe-archive.sh -h

$ safe-archive.sh -cf test.arc dir1

>> Creation of archive test.arc …

dir1/

dir1/fic2

dir1/fic3

dir1/fic1

enter bf-cbc encryption password:

Verifying – enter bf-cbc encryption password:

>> Creation of archive test.arc OK.

$ safe-archive.sh -tf test.arc

>> Listing of archive content test.arc …

enter bf-cbc decryption password:

drwxr-xr-x root/root         0 2010-04-13 21:48 dir1/

-rw-r–r– root/root        33 2010-04-13 21:48 dir1/fic2

-rw-r–r– root/root        33 2010-04-13 21:48 dir1/fic3

-rw-r–r– root/root        33 2010-04-13 21:48 dir1/fic1

>> Listing of archive content test.arc OK.

$ rm -rf dir1

$ safe-archive.sh -xf test.arc

>> Extraction of archive content test.arc …

enter bf-cbc decryption password:

dir1/

dir1/fic2

dir1/fic3

dir1/fic1

>> Extraction of archive content test.arc OK.

$ find . -ls

133988    4 drwxr-xr-x   3 root     root         4096 avril 13 21:51 .

133990    4 -rw-r–r–   1 root     root          248 avril 13 21:50 ./test.arc

137002    4 drwxr-xr-x   2 root     root         4096 avril 13 21:48 ./dir1

137006    4 -rw-r–r–   1 root     root           33 avril 13 21:48 ./dir1/fic2

137007    4 -rw-r–r–   1 root     root           33 avril 13 21:48 ./dir1/fic3

137008    4 -rw-r–r–   1 root     root           33 avril 13 21:48 ./dir1/fic1

Safe-archive is a simple Bash script (less than 175 lines of code). If you want to test it, just copy-paste the code below in a file named safe-archive.sh and make sure that it is executable on your system (chmod 755 safe-archive.sh).

#!/bin/bash
#
# Description :  safe-archive is a tool to manage secured archives. It allows
#                to create/extract/modify encrypted and compressed tar files.
#                The syntax is very much inspired by the tar command.
#                It uses an extremely strong encryption algorithm (Blowfish cipher,
#                CBC mode) provided by openssl.
#
# Dependencies : All the commands should be present on any LINUX/UNIX system.
# - GNU TAR
# - OpenSSL
# - GZIP
# - cp, mv, rm
# Paths may need to be adjusted in the configurable section.
#
# Syntax : see help (option -h)
#
# Return code : 0 in case of success.
#
# Author : kwartik@gmail.com 
# Creation date : 2010-04-13
# Revision : 001
#
# History of modifications :
# 2010-04-13: rev001, initial release
_pn=`basename $0`

### BEGINNING OF CONFIGURABLE SECTION ###

#_debug=1: mode debug activated 
#_debug=0: mode debug deacivated
_debug=0

# list of commands used by the script. For security reason, it is preferrable to sepcify full absolute paths
_command_tar='/bin/tar';
_command_openssl='/usr/bin/openssl';
_command_gzip='/usr/bin/gzip';
_command_cp='/bin/cp';
_command_mv='/bin/mv';
_command_rm='/bin/rm';
_command_gzip='/usr/bin/gzip';

### END OF CONFIGURABLE SECTION ###

_src=''
_file=''
_out=''
_create=''
_list=''
_extract=''
_append=''

help()
{
    echo "${_pn}: management of secured archives"
    echo
    echo "Syntax:"
    echo
    echo " To create an archive of a file or direcory: ${_pn} -c -f archive_name file_or_directory"
    echo
    echo " To list the content of an archive: ${_pn} -t -f archive_name"
    echo
    echo " To extract the content of an archive: ${_pn} -x -f archive_name"
    echo
    echo " To add a file or a directory to an archive: ${_pn} -r -f archive_name file_or_directory"
    echo
    echo " To display this help: ${_pn} -h"
    echo
    exit 1
}

exit_on_error()
{
	echo "FATAL ERROR: $*"
	exit 1
}

while getopts "f:ctxrh" _flag
do
    case ${_flag} in
        f) _file=$OPTARG ;;
        c) _create=1 ;;
        t) _list=1 ;;
        x) _extract=1 ;;
        r) _append=1 ;;
        h) help ;;
    esac
done

_args=("$@")
_src=${_args[$OPTIND - 1]}

[ ${_debug} -eq 1 ] && echo "_directory=${_directory} _list=${_list} _create=${_create} _extract=${_extract} _append=${_append} _file=${_file} OPTIND=${OPTIND} _src=${_src}"

create()
{
    [ ! -f ${_src} -a ! -d ${_src} ] && exit_on_error ${_src} is not a file or a directory
    echo ">> Creation of archive ${_file} ...   "
    ${_command_tar} czvpf ${_file} ${_src}
    [ $? -ne 0 ] && exit_on_error ${_command_tar} czvpf ${_file} ${_src} failed
    ${_command_openssl} enc -bf-cbc -salt -in ${_file} -out ${_file}.tmp
    [ $? -ne 0 ] && exit_on_error ${_command_openssl} enc -bf-cbc -salt -in ${_file} -out ${_file}.tmp failed
    ${_command_mv} ${_file}.tmp ${_file}
    [ $? -ne 0 ] && exit_on_error ${_command_mv} ${_file}.tmp ${_file}
    echo ">> Creation of archive ${_file} OK."
    exit 0
}

list()
{
    [ ! -f ${_file} ] && exit_on_error archive ${_file} is not a file
    echo ">> Listing of archive content ${_file} ...   "
    ${_command_openssl} enc -bf-cbc -d -salt -in ${_file} -out ${_file}.tmp
    [ $? -ne 0 ] && exit_on_error enc -bf-cbc -d -salt -in ${_file} -out ${_file}.tmp failed
    ${_command_tar} ztvf ${_file}.tmp
    [ $? -ne 0 ] && exit_on_error ${_command_tar} ztvf ${_file}.tmp failed
    ${_command_rm} ${_file}.tmp
    [ $? -ne 0 ] && exit_on_error ${_command_rm} ${_file}.tmp
    echo ">> Listing of archive content ${_file} OK."
    exit 0
}

extract()
{
    [ ! -f ${_file} ] && exit_on_error archive ${_file} is not a file
    echo ">> Extraction of archive content ${_file} ...   "
    ${_command_openssl} enc -bf-cbc -d -salt -in ${_file} -out ${_file}.tmp
    [ $? -ne 0 ] && exit_on_error enc -bf-cbc -d -salt -in ${_file} -out ${_file}.tmp failed
    ${_command_tar} zxpvf ${_file}.tmp
    [ $? -ne 0 ] && exit_on_error ${_command_tar} zxpvf ${_file}.tmp failed
    ${_command_rm} ${_file}.tmp
    [ $? -ne 0 ] && exit_on_error ${_command_rm} ${_file}.tmp
    echo ">> Extraction of archive content ${_file} OK."
    exit 0
}

append()
{
    # Append case is not so straight forward because tar command throws an error if
    # option z an r are used simultaneously ...
    [ ! -f ${_src} -a ! -d ${_src} ] && exit_on_error ${_src} is not a file or a directory
    echo ">> Adding ${_src} to archive ${_file} ...   "
    echo ">>> archive decryption ..."
    ${_command_cp} -p ${_file} ${_file}.bk
    [ $? -ne 0 ] && exit_on_error ${_command_cp} -p ${_file} ${_file}.bk failed
    ${_command_openssl} enc -bf-cbc -d -salt -in ${_file} -out ${_file}.tmp.gz
    [ $? -ne 0 ] && exit_on_error enc -bf-cbc -d -salt -in ${_file} -out ${_file}.tmp.gz failed
    ${_command_gzip} -d ${_file}.tmp.gz
    [ $? -ne 0 ] && exit_on_error ${_command_gzip} -d ${_file}.tmp.gz
    ${_command_mv} ${_file}.tmp ${_file}
    [ $? -ne 0 ] && exit_on_error ${_command_mv} ${_file}.tmp ${_file}
    echo ">>> adding ${_file} ..."
    ${_command_tar} rvpf ${_file} ${_src}
    [ $? -ne 0 ] && exit_on_error ${_command_tar} rvpf ${_file} ${_src} failed
    ${_command_gzip} ${_file}
    [ $? -ne 0 ] && exit_on_error ${_command_gzip} ${_file}
    echo ">>> archive re-encryption ..."
    ${_command_openssl} enc -bf-cbc -salt -in ${_file}.gz -out ${_file}
    [ $? -ne 0 ] && exit_on_error ${_command_openssl} enc -bf-cbc -salt -in ${_file}.gz -out ${_file}
    ${_command_rm} -f ${_file}.gz
    [ $? -ne 0 ] && exit_on_error ${_command_rm} ${_file}.gz
    ${_command_rm} -f ${_file}.bk
    [ $? -ne 0 ] && exit_on_error ${_command_rm} -f ${_file}.bk
    echo ">> Adding of ${_src} to archive ${_file} OK."
    exit 0
}

[ "x${_create}" != x -a "x${_file}" != x -a "x${_src}" != x ] && create
[ "x${_append}" != x -a "x${_file}" != x -a "x${_src}" != x ] && append
[ "x${_list}" != x -a "x${_file}" != x ] && list
[ "x${_extract}" != x -a "x${_file}" != x ] && extract
help

Scripts.com

Rate this script!

March 28, 2010

Chess Openings : a simple PERL script to practise chess openings

Filed under: Perl — Tags: , — kwartik @ 1:43 pm

I have just released CHESS OPENINGS, a simple PERL script to practise chess openings. It is provided with a default configuration file containing a list of 437 famous openings. CHESS OPENINGS chooses randomly one opening in the configuration file and displays it as a nice UTF-8 matrix (UTF-8 do support chess characters). CHESS OPENINGS may be simply used through the command line but may be also be called by other tools like screen savers.

This script is intended to be used in a UTF-8 environment supporting chess characters. It has been successfully tested in a Linux terminal. Unfortunately, under Windows, the “terminal” (cmd.exe) is not able to print UTF-8 characters, so Windows users might need to use a two steps approach :

  1. redirection of the output in a file: perl chess-openings > opening.out
  2. edition of the output file with a UTF-8 compliant editor providing a nice font for chess characters

Chess fans may want to improve the code to add features (comments on the moves, …).

The script can be downloaded HERE. Enjoy !

January 28, 2010

Soliloquy or Talk to Yourself problem

Filed under: Perl — kwartik @ 9:23 pm

As a Perl programmer, I have been often faced to the following problem : execute an external code that may if the external condition are bad (network problem) not give the hands back to Perl. The challenge is to be able after a predefined time to kill the external process and all its subprocess and to continue the main Perl program execution. This can be achieved in Perl without too much difficulties.

Solution

The proposed solution uses a standard IPC (Interprocess Communication) design where two process communicate through a pipe. The open system call is used to fork the current process. This allows to create a pipe between the father and the child. The child is then asked to test the SSH connectivity and to send back the result to the father. If after N seconds, the child has not sent the return code, then the father assumes that the connectivity is bad and kills its son and all its sub process. The system concepts of this technique are explained in Programming Perl ( Larry Wall, Tom Christiansen, John Orwant – O’REILLY, chapter 16)

Detailed Code

# testSSH: SSH connectivity test. If after N seconds there is still no answer,
#          the SSH process group is killed
# 
# Parameter : hostname of the host to test
#
# Return code : 1 if test succeeds, 0 if test fails                                      
#                                                                                                
sub testSSH ( $ ) {                                                                                                                                 
        my ($host) =  @_;                                                                                                                                  
        my $debug = 0; #0: normal mode, 1: debug mode                                                                                                       
        my $test_command = "/usr/bin/ssh -l login $host /bin/ls /etc/passwd 1>/dev/null 2>&1";                                                              
        my $timeout = 5;                                                                                                                                      

        my $res_soliloquy = open(SOLI, "-|");
        if ( $res_soliloquy ) {              
                # We are in the father        
                print "|Father: pid of son returned by open=$res_soliloquy|\n" if ( $debug );
                my $res_command = -2;                                                           
                eval {                                                                       
                        local $SIG{ALRM} = sub { die "alarm\n" }; # NB: \n required          
                        alarm $timeout;                                                      
                        $res_command = <SOLI>;                                                  
                        chomp($res_command);                                                    
                        print "Father: $res_command received\n" if ($debug);                          

                        alarm 0;
                };              

                if ( $@ ne "alarm\n" ) {
                        print "\n NO ALARM (RC=$res_command)" if ( $debug );
                        if ( $res_command == 0) {                            
                                return 1;                                 
                        }                                                 
                        else {                                            
                                return 0;                                 
                        }                                                 
                }
                else {
			# As the son may have created some sub process (The ssh command is usually identified
			# by several process), we need to identify the process group of the son. The 
			# problem is that the process group is usually identified (at least on Linux) with 
			# the father process id which should not be killed in our case. We grab the ids 
			# of all the process running of the system, then for each of them we do a 
			# getgrgid($id) and we keep in a list of process to kill all the process that have 
			# the same process group of the father. In order to get the list of process ids 
			# running on the system, we simply execute `ps axho pid`. This could also be 
			# achieved with other PERL modules...
                        my $parent_pid=$$;
                        my $pgrp = getpgrp($res_soliloquy);
                        print "\n ALARM: Will kill the son's branch (father_pid=$parent_pid, pid=$res_soliloquy,
                        pgrp=$pgrp)" if ( $debug );
                        my $all_pid_output = `ps axho pid`;
                        $all_pid_output =~ s/^\s*//gm;
                        my @all_pid_list = split('\n',$all_pid_output);
                        my @array_pid_to_kill = ();
                        foreach (@all_pid_list) {
                                my $current_pid=$_;
                                my $current_pgrp = getpgrp($current_pid);
                                print "\n-> $current_pid $current_pgrp"  if ( $debug );
                                if ( $current_pgrp eq $pgrp and $current_pid ne $parent_pid ) {
                                        print "\n-> $current_pid eligible for kill" if ( $debug );
                                        push(@array_pid_to_kill,$current_pid);
                                }
                        }
                        # All the identified process are killed
                        kill 9, @array_pid_to_kill;

                        return 0;
                }

        }
        else {
                # We are in the son (NO PRINT IN THE SON OTHER THAN THE RETURN CODE)
                my $res_command = system( "$test_command");
                print STDOUT $res_command;
                exit;
        }
}

my $res_test = testSSH('server');

December 7, 2009

How to set up a free and practical parental control filter with Procon Latte

Filed under: General — Tags: , — kwartik @ 11:50 pm

I was looking for a simple parental control filter solution in order to be sure that my young daughter (4 years old who of course likes to click on any link she sees !) will not accidentally find herself on one of these numerous undesirable sites. My requirements were the following :

- white list feature (I want to block all sites by default and allow explicitly a small number of sites)

- configuration protected with a master password

- filtering at application level, ideally as a Firefox module (add-on). I consider as totally sufficient this solution for my daughter, as I am sure that she is too young to think of using a different browser. She would have indeed quite some difficulties to find one on the Linux system I have configured for her !

- free solution

I queried the Firefox module catalog ( Tools / Complementary Modules / Catalog ) and it basically proposed me two parental control modules :

- foxfilter, promoted as “Simple and effective! FoxFilter is a personal content filter that helps block pornographic and other inappropriate content using customizable features. All filtering features are free! Premium features require a small support fee.”,

- broozi, promoted as the “first Firefox add-on which enable kids to take their first steps on Internet, so simply and in full safety”.

I was amazed by the philosophy of FoxFilter. “All filtering features free”, but the most important one which is the master password to guarantee that your kid will not be able to add a site to a white list is a Premium feature ! More over, I had the impression that a lot of the configuration is done online as if the foxfilter is insterested by your parental configuration settings ! A few minutes of testing were sufficient to add the product to my black list!

- The philosophy of broozi to radically change the user interface was not at all what I was looking for. Moreover, control parental feature simply did not work, as it was possible to click on the firefox icon in the tasks bar to leave the broozi interface and to go back to the firefox interface and my daughter is clearly able to do this ! Once installed I was encouraged to purchase Rooki, which adds a real _safe_ parental control feature to broozi.

- Finally, I googled a little bit and found ProCon Latte described as a content filter for Forefox allowing to “filter any kind of material (pornography, gambling, hacking, cracking, etc…), it can also block all traffic, making sure that only desired websites (set in the Whitelist) can be accessed, and includes a profanity filter, all *like* a parental control filter.” I tested only the “block all traffic” and white list features and I really appreciated this tool. I found the interface really intuitive to use and not at all intrusive.

I have posted below a few screenshots showing how easy it was to set a configuration compliant to my requirements. My system uses french locales but of course internationalization is well managed by ProCon.

First, a master password needs to be set, it will be required at each settings modifications as shown on the picture below :

ProCon master password prompt

Then all sites can be blocked by default by clicking on the “Block all traffic checkbox” :

ProCon : activation of the "all sites are blocked by default" rule

Finally, a white list can be easily created by clicking on the “Activate white list” check box :

ProCon: activating the white list

And that’s it ! After this straight-forward configuration step, I can safely leave my daughter in front of the screen. ProCon implements also much more elaborated filtering logic that is detailed on its web site.

I was also very impressed by the support provided by corvineum, the author of ProCon. After a ProCon minor release update (v1.7.9.7 => 1.7.9.8), I was not able to use ProCon anymore and an error related to the french accents was displayed in a popup. I sent a email to corvineum and twelve hours latter he sent me the fix ! I am not even sure that if I had experimented a similar problem with a proprietary solution, I would have observed the same reactivity !

November 28, 2009

Designing a safe and practical long-term backup solution

Filed under: SysAdmin — Tags: , , , — kwartik @ 10:26 pm

1. The paradox

A worrying tendency has been growing exponentially in the last two decades, worrying because it can give a totally false impression of security to its adepts. This tendency has several denominations like digitization or de-materialization.

All the knowledge that has been archived on traditional support (stone, paper) by our ancestors since the invention of writing, approximately 5300 years ago, is being migrated onto this new so-called “numerical” support, fundamentally different. Functionally fundamentally different because it gives data a new dimension which can be defined as the intersection of ubiquity and instantaneousness and which literally revolutionizes sharing capabilities. But also fundamentally different by essence. Data becomes virtual, its physical representations and locations obscure for most of the humanity, the knowledge of the machinery of this new technology is shared only by a relatively small community of skilled technicians. A characteristic of this new support is its dependency to energy. Depending of the numerical support, absence of electrical power, means at least impossibility to read the data, and at most total loss of the data. We know that energy management is going to be a key challenge in this millennium. What would be the consequences if energy resources were to be one day insufficient to keep alive this gigantic amount of virtual data which is growing at such an incredible speed ?

Data destruction sinistrality takes also a new dimension with this technology revolution. History has been marked by several acts of intentional or accidental data destruction (acts often materialized as book burnings) and such regrettable events continue and will for sure continue to happen in this new numerical era. The two main differences that can be noticed in the new destruction events are their magnitude and instantaneousness. Who has not experimented a data loss or corruption due to a virus (criminal event) or hard disk failure (accidental event) ? How much data was present on this hard disk that just failed ? Today, 1 Tera bytes disks are common, they can easily host the equivalent of a million of books ! The next generation saving units will  be more and more capacitive, probably in the order of the Peta bytes in 2025 if we consider that Moore’s Law also applies to hard disks!  Risks take definitely a new magnitude. The reasoning can also be pushed at organization level. Consequences for an organization that would lost its data after an electromagnetic attack are disastrous. Electromagnetic weapons do exist and have already been successfully tested on real companies ! It is not an hazard if companies that are totally dependent on their information systems choose, if they can afford it, to bury their electronic equipments in bunkers !

I have the feeling to observe a paradox in my daily life. On the one hand we are (or can be if we decide to) aware of the dimension of these new risks, on the other hand we blindly archive under this new format every single event of our daily life ! The next part of this article tries to define the security measures that should be taken to limit the risk of personal data lost and proposes a practical technical implementation.

2. Risks Analysis

Data loss causes are either accidental or criminal.

Accidental causes are events such as :

  • hardware failures (electronic or mechanicals components)
  • human error
  • software bugs

Criminal causes are events such as :

  • hardware thefts
  • hacking (viruses, system compromising ..)
  • Electromagnetic attack

3. Requirements for a safe backup policy

Backup (or more generally redundancy) is certainly the best and only parade to data loss but it has to be done carefully as it can give a real false impression of security if not done properly. Indeed, defining and implementing a back-up policy is quite challenging as all the different risks should be taken into consideration. We will focus here on the use case concerning individuals willing to deploy a practical solution to secure their personal data. The case of data backup management within an organization is more complex to address as there are real-time security requirements, also because of its important size and the fact that it is usually spread on many physical locations, but the general principles exposed here are still valid.

A crucial parameter to take into consideration in the backup policy is the fact that damages after any of the previous causes are not necessarily immediately visible – part of the data may be in a incoherent state while the other isn’t. If the concerned incoherent files are diagnosed as corrupted too late, restoring the back-up may not help if this back-up has been done after the compromising. This is why it is fundamental to have a data backup technology allowing to restore the state of a file at different points in time. The earlier state it is possible to restore, the safer the backup is.

An other crucial principle is that the backup should be performed on a non-rewritable medium. It is indeed very easy to imagine a nasty virus capable of deleting in a few seconds all the data present on a rewritable backup medium like a external hard drive as soon as it is plugged into the system.

4. Solution

Delta-backup is a simple and efficient tool for cross system backup with differential archiving. I have designed it with the two principles listed above in mind. It is a simple PERL script invoking the powerful RSYNC utility which is the reference tool for file systems synchronization in the UNIX/LINUX world.

The principle of delta-backup is the following: When run for the first time, it performs a backup of a reference directory (the “source directory”) into a backup directory (“the destination directory”). Both directories can be physically located on different hosts. After this initial full backup, which depending of the source directory size can be quite time consuming, each time the script is executed, the destination directory is synchronized with the source directory. This synchronization step between the source and destination directory is much faster than the initial full backup as only the differential (files that have been created and modified since its last execution) is copied from the source to the destination. Delta-backup also duplicates the differential by archiving it (using the tar format) in a third directory, the differential directory (the “delta directory”). The list of deleted files is also archived in the delta directory. It is highly encouraged to regularly save the content of the delta directory on a non rewritable medium (typically an optical support). In case of a major incident (lost of both source and destination directories) it is therefore possible to restore an “acceptable” state of the data by successfully restoring the differentials archived (and deleting the files listed in the list of deleted files). The older the initial differential archive will be, the more acceptable the restored state will be. The script provides two optional features for the archiving of the differential which are compression and high-level encryption. It is therefore conceivable to safely ask someone to keep the differentials in a different physical location than the source or remote directories. Delta-backup can be downloaded here : https://sourceforge.net/projects/delta-backup/.

Backup (or more generally redundancy ) is certainly the best and only parade to data loss but it has to be done carefully as it can give a real false impression of security if not done properly. Indeed, defining and implementing a back-up policy is quite challenging as all the different risks should be taken into consideration.

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.