Linux filesystems for simple storage

Newly bought USB keys or memory cards usually come with patent-encumbered FAT or exFAT filesystems on them. While exFAT does away with size limitations, it is still not a first-class citizen under Linux. Only a fuse-based solution is available for mounting so far, which is not even in repositories such as rpmfusion due to the licensing issues.

If the volumes aren't to be used in devices such as digital cameras or PVRs, and use in Windows-PCs is also not an issue, one can of course opt for a native Linux filesystem.
However, standard mkfs.extX will use a lot of space for some things that is well-spent on a large system harddrive, but much less so on a relatively small flash device.

So here are two commands to make neat little filesystems for these:

mkfs.ext2 -T largefile -m 0 -L <some_name> /dev/sdX1
tune2fs -c 0 -i 0 -O large_file /dev/sdX1

Choosing mkfs.ext2 will result in maximum usable capacity. However it will not create a journal like ext3 / ext4 would. A journal does away with lengthy filesystem checks in case the medium was not properly unmounted. Given that today's USB keys provide multi-gigabytes of storage and that Linux tends not to mount such devices in synchronous mode by default, you may still choose to keep the journal for removable media. Extents are actually a nice thing that can save space for large files, but it is a relatively new feature that should be avoided for portable media at the moment so they can also be used with older Linux versions, YMMV.

The -T largefile option gives a better ratio of space reserved for management structures vs. actual data to be stored. Assuming you will usually not store thousands of files with only a dozend bytes in them this makes sense, but don't try this with an unpacked gentoo portage tree!

The large_file flag is normally set automatically and allows storing files >2GiB on the filesystem.

The -m 0 flag makes sure there is no space set aside for the superuser, which would make no sense on a data-only volume.

The -c 0 -i 0 flags prevent the filesystem from asking to be checked from time to time. This normally just puts warnings in your logs; no system I'm aware of runs fsck on hotplugged storage devices, and hardly anyone checks their USB keys manually as far as I can tell.

Getting the serial number of a certificate stored in pkcs12

When saving a client certificate from a Mozilla application to disk – like getting it out initially after it's been installed from the cacert website – the certificate is inside a PKCS12 container.

To uniquely identify a certificate, it is useful to know its serial number, especially if you have several certificates that are similar and have gotten renewed several times, so there is more than one version of "the same" certificate around.

To learn the serial number from a p12 file, I use this line:

openssl pkcs12 -in <container>.p12 -clcerts -passout pass:"" | openssl x509 -serial -noout

Usually the certificate will be encrypted, so you have to type in the password that was used on export.

Setting up a cyberJack chipcard reader under Fedora 17 x86_64

• Get the latest driver source code release from Reiner's web site, at the time of writing the file is pcsc-cyberjack-3.99.5final.SP03.tar.gz.
• Install needed packages
yum install gcc gcc-c++ libusb1-devel pcsc-lite-devel
• Unpack the driver tarball
• Edit the source file libcyberjack/checksuite/cm_distri.cpp to add #include <unistd.h> at line 22, otherwise the compile will fail with the error message error: ‘unlink’ was not declared in this scope
• From within the top level directory of the unpacked and modified source run
• ./configure && make
• Then from the same directory run as root
• mkdir -p /usr/lib64/pcsc/drivers/libifd-cyberjack.bundle/Contents/Linux
• install -m 644 ifd/Info.plist /usr/lib64/pcsc/drivers/libifd-cyberjack.bundle/Contents
• ./libtool --mode=install /usr/bin/install -c ifd/libifd-cyberjack.la '/usr/lib64/pcsc/drivers/libifd-cyberjack.bundle/Contents/Linux'
• install -s tools/cyberjack/cyberjack /usr/local/bin/
• Now make sure that pcscd is not running as a background service on your system then start it as root: pcscd -f -d
• If it worked the reader should flash its LEDs and there should be lots of output to the terminal, with this somewhere in the middle

00000011 hotplug_libudev.c:311:HPAddDevice() Adding USB device: REINER SCT cyberJack pp_a
00000023 readerfactory.c:936:RFInitializeReader() Attempting startup of REINER SCT cyberJack pp_a (5745748205) 00 00 using /usr/lib64/pcsc/drivers/libifd-cyberjack.bund
CYBERJACK: Started
00000615 readerfactory.c:826:RFBindFunctions() Loading IFD Handler 3.0
00006834 readerfactory.c:291:RFAddReader() Using the pcscd polling thread

Processing HD DVR recordings

As it so happens, I own a DVR device that will record DVB-S and DVB-S2 TV into segments of a MPEG transport stream.

For DVB-S2 HD programs, the stream already contains h.264 compressed video. However since we don't live in the 80's anymore where there used to be a thing called VPS that would make sure your VCR starts and stops when the program does – even if that does not match the time printed in the program guide – the recording will be enclosed by parts of the surrounding broadcast.

Getting rid of that "garbage" is a bit tedious, but here's how I managed.

The receiver writes files 00001.ts, 00002.ts...00012.ts of one gigabyte each to the attached USB drive. So first turn that into one nice big file by using cat (the target better be ext4 or something else that can handle 2G+ files).

cat *.ts > <name>.ts

Now the problem is we can't just use a media player and seek to the cut points because the data in this transport stream is not properly timestamped.

But first we need to figure out which streams (tracks) are present in the raw material. Often broadcasters transmit a bunch of redundant audio streams. Just in a single language of course, apparently to make sure viewers are not getting too much value for their license fee.

$ ffprobe <name>.ts
[...]
    Stream #0:0[0x18a6]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 100 tbc
    Stream #0:1[0x18b0](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 256 kb/s
    Stream #0:2[0x18b1](mis): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 192 kb/s
    Stream #0:3[0x18b3](mul): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 192 kb/s
    Stream #0:4[0x18b2](deu): Audio: ac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, s16, 448 kb/s
    Stream #0:5[0x18ba](deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)

In this case, I'm only going to keep streams 0:0 (the h.264 video) and 0:4 (the AC3 audio). You can use mplayer with the -aid/-vid options to check out the various streams. To play the file with audio stream 0:3 for example, run as

mplayer -aid 0x18b2 <name>.ts

So now to get a seekable file with only the wanted streams, do this:

ffmpeg -i <name>.ts -f mpegts -c copy -bsf:v h264_mp4toannexb -map 0:0 -map 0:4 <intermediate>.mkv

Then open the file e.g. in gnome-mplayer, seek to the start and the end of  the actual content and make note of the timestamps.

Finally, fire up mmg (mkvmerge GUI), and add the freshly created <intermediate>.mkv file on the Input tab.
Then switch to the Global tab and enter the title of the recording into the File/segment title field.
Now check the Enable splitting... checkbox and select ...by parts.
Enter the begin/end timestamps noted down earlier into that field, separated by dash like 00:06:45-2:19:31.
Now verify that the Output filename is to your liking and hit the Start muxing button.

Autostarting screen on a server with programs running

There are some things I want my little home server to do permanently in the background. The corresponding programs may or should run in the context of the regular, non-privileged user. They may be interactive / full-screen console applications that I want to access from different computers around my network.

A way to accomplish this is to set up screen with an autostart configuration. The screen command is part of all modern Linux distributions, although it may not be installed by default.

To start the jobs at boot time without manual intervention, screen must be hooked up with the boot process. For gentoo this happens through an executable file /etc/local.d/*.start. Fedora uses /etc/rc.d/rc.local, which must be created in contemporary releases of the distribution. In both cases, the files must be set executable.

The call to screen from this system boot hook needs to be prefixed with an appropriate sudo to run in the context of the non-privileged user:

sudo -u <user> /usr/bin/screen -wipe

sudo -u <user> -i /usr/bin/screen -dm

From the screen manpage:

-d -m   Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts.

When the system was not shut down properly, screen session files may have been left behind which can prevent screen from autostarting on the next boot. Since this is in bootup and no other screen sessions are yet running, we can just use the -wipe option preventively to make sure no such leftovers remain.

There is an additional catch with Fedora as it sets Defaults requiretty in /etc/sudoers which prevents sudo from working here:

sudo: sorry, you must have a tty to run sudo

The solution is to add another line using visudo as follows:

/etc/sudoers

Defaults:root   !requiretty

Now create a directory .screen and a file .screenrc in the user's home directory. The directory holds a number of scripts containing the commands to be run in various screen windows, while the .screenrc file assigns these scripts to windows.

.screenrc

defflow off
screen -t shell0 0 $HOME/.screen/s0
screen -t pinger 1 $HOME/.screen/s1
screen -t nload 2 $HOME/.screen/s2

The defflow off directive turns off screen's flow control handling, which allows fullscreen console applications to receive and process the keystrokes Ctrl-S / Ctrl-Q. This can also be set per window (-f / -fn).
The -t option to the screen directive sets the window title; it is followed by the window number and the command to run.

.screen/s0

#!/bin/sh
fetchmail -d 3600
bash

This starts fetchmail in the background to retrieve mail each hour, and also gives me a shell in screen window 0.

 

.screen/s1

#!/bin/sh
pinger

Window 1 runs a tool that regularly pings a number of other computers and presents statistics about their availability.

 

.screen/s2

#!/bin/sh
nload -i 100000 -o 100000 -t 4000 -u K -U K eth0

This tool visualizes the utilization of the eth0 interface as an ascii-art graph.

See here for how to connect to such a screen session, and here for a general introduction to screen.

Readable MySQL dumps

Reading a text dump produced with mysqldump is normally hampered by the fact that the default options include --opt (for optimize), which is shorthand for a bunch of other options that make the resulting sql more concise and quicker to parse, but also very hard to read for humans as inserts are munched together into ridiculously long lines without any visual structure.

To prevent this from happening, use --skip-opt:
mysqldump -u root -p --skip-opt <dbname> > /tmp/sql.txt

For reading with less, the -S option is useful to prevent line breaking, which can be very slow.
less -S /tmp/sql.txt

Running ExtraDB on a Linux block device

Since the future of free MySQL appears a bit uncertain due to the fact that its owner happens to be well recognized for its self-developed proprietary database, I decided to have a look at the fork that is led by the original main developer of MySQL: MariaDB.

Interestingly, things have not changed much once you get your head around the fact that the database itself as well as the engines have been renamed in the documentation, but still use their familiar names in the config files.

So the state-of-the-art storage engine that used to to be InnoDB is now called ExtraDB, but all its config options still start out with innodb_. Also the engine names in table type specifications still use the old names. So mostly, you just replace MySQL with MariaDB and keep going.

There is one catch however when using a raw block device for ExtraDB tablespace: It no longer works to pass the absolute name of the block device's /dev file (eg. /dev/sdb1). Examining mysqld.err reveals that the daemon actually tried to access a file by the name .//dev/sdb1, which looks to me like somebody took Windows support quite a bit too far.

120116 15:21:21  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name .//dev/sdb1
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.

Fortunately there is a simple workaround for this bug, simply create a symlink or a replica of the respective device node inside the /var/lib/mysql directory and point the daemon to plain sdb1.

cp -a /dev/sdb1 /var/lib/mysql/


/etc/mysql/my.cnf:

innodb_data_file_path           = sdb1:10000269312raw

Bulk-shrink images with ImageMagick

I'm one of those (apparently few) people that are annoyed by emails showing me e.g. the location of a screw on a piece of metal with 3 pictures of 2 Megabytes each.

So as I'm trying to lead by example (uber-smug, ain't I?), when sending pictures I took with a digital camera myself I reduce those images to around 600 pixels along their larger dimension, which will do nicely in most cases and save the other party the trouble not only of downloading huge files but also of staring at a tiny irrelevant detail of the picture that happens to be in the top left corner after the image viewer has ceased up their entire screen.

The best interactive tool for the occasional view-crop-correct-resize-compress task is obviously gthumb, but when faced with a larger number of images, it's commandline-time:

mogrify -quality 82 -resize 600x600\> *.jpg

This will shrink all jpg files in the current directory to fit within a 600x600 pixel bounding box while preserving aspect ratio (so the images are not distorted as one might first think). The closing angle bracket at the end ensures that images that are smaller than the target size are not enlarged. Note that the mogrify command changes the images in place, so you may either want to make sure to have a backup of the original files, or go for find/convert instead.
For more information, here is the resize operator's official documentation.

Another little-known but very excellent feature of the ImageMagick suite is the -strip operator, which will remove all EXIF data from an image. This can be useful especially when the picture was taken on a modern smartphone which will normally embed all sorts of information in the image's metadata such as location and time of the shot, as well as the model name of the device the image was taken with. Maybe you wouldn't want to share that with everyone that should see the picture.

Lowering the (re)bar

Although the Erlang programming language has been on my radar for many years now, I never quite grasped the concept of how to put things together such that a bundle of functionality might be properly deployed to and run on a target system. The target system would of course be Linux :)

Fortunately, in the last couple of years an effort has emerged that promises to make things easier. It's a tool called rebar, and its canonical use seems to serve as the build system for a database application by the name riak. Some of the participant's names on the rebar mailing list ring familiar, so it seems possible that it may even be adopted into OTP at some point.

There are quite a number of "getting started" tutorials around for rebar, one right inside the rebar github wiki, and also this one being still quite recent early in 2012. rebar is (or at least has been) a moving target, and some tutorials you may find are badly outdated.

However even following the supposedly applicable instructions, I invariably ran into this error at the generate stage:
$ ./rebar generate
==> rel (generate)
{"init terminating in do_boot","Release exemplar uses non existing application exemplar"}

Crash dump was written to: erl_crash.dump
init terminating in do_boot (Release exemplar uses non existing application exemplar)

It took me quite a while to get past this problem, and the key here is that the Erlang runtime needs to accept your working directory as an application, which requires two things to be true:

1. The directory that contains the src and ebin directories, as well as rebar.config (not rebar.conf btw!) needs to have exactly the same name as the Erlang application you are creating.
2. The directory which comprises that application directory needs to be part of Erlang's lib_dirs (an extension searchpath for /usr/lib/erlang/lib) so it will be considered by the runtime when it is looking for applications.

What is left out by all the tutorials I found is that for #2 to become true, one needs to manually edit the reltool.config file that is created by the rebar create-node step and edit the lib_dirs assignment to include that parent directory (can be an absolute path, or given relative from the rel directory).

So if you are trying this out in /tmp/exemplar like

/tmp/exemplar
|-- deps
|   `-- exemplar
|       |-- ebin
|       `-- src
|           |-- exemplar_app.erl
|           |-- exemplar.app.src
|           `-- exemplar_sup.erl
|-- rebar.config
`-- rel
    |-- exemplar
[...]
    |-- files
    |   |-- erl
    |   |-- exemplar
    |   |-- exemplar.cmd
    |   |-- nodetool
    |   |-- start_erl.cmd
    |   |-- sys.config
    |   `-- vm.args
    `-- reltool.config

then your /tmp/exemplar/rel/reltool.config file should be changed to start like

{sys, [
       {lib_dirs, ["/tmp/exemplar/deps"]},
[...]

At first I was wondering whether this manual edit is really the right thing to do since this file is generated, but checking out the riak rel directory, the reltool.config file there is checked into git and also contains a manual lib_dirs assignment.

Another mistake to avoid is naming your node (rebar create-node nodeid=<name>) anything different from your main application, which will result in an obscure error message

Command 'generate' not understood or not applicable

Adding a user in mysql for a specific database

Whenever installing some new webapp on a LAMP stack, the typical procedure goes such that the app gets its own database in mysql/mariadb/drizzle/whatever and needs a user to access it.

The database is of course created like

create database <database>;

To add the user, one usually uses a command like this in the mysql client shell:

grant all privileges on <database>.* to '<new_user>'@'localhost' identified by '<some_password>';

In which I like to use a randomly generated password (e.g. using makepasswd) for each user.

Besides assigning rights, the latter command also creates <newuser> as a side effect, and even flushes privileges in one fell swoop.

One might further want to prune the granted rights to only work on the actual data, not allowing modification of the tables/schema, but most webapps come with an installer built in that creates/updates the app's tables so this is left as an exercise for the reader.

autossh / screen without the hassle

$ autossh -M 0 <host> -t 'screen -x'

Coping with gnome3

The most important thing to know is of course gnome-tweak-tool, which is pretty much self-explaining once you know it's there.

On less capable machines, fallback mode can be quite cozy too, provided you can do with the limited number applets available. For laptops where vertical screen estate is scarce, know that the panel (bar at the bottom) can still be set to autohide like in the olden days, however it must be clicked with Alt+RightMouseButton to bring up a preferences menu that actually includes this option. Same goes for the top menu, which can hold application icons in case you forgot.

To get rid of the annoyance that is edge tiling (the fact that a window goes fullscreen whenever it happens to touch the top of the screen while you're trying to move it around), one can use gconf-editor and remove the tickmark from /desktop/gnome/shell/windows/edge_tiling.

A great summary of Tips'n'Tricks on this topic is found here. It is based on Mint Linux which deviates somewhat from stock gnome3, but the guide is still very useful.