Processing HD DVR recordings

As it so happens, I own a DVR device that will record DVB-S and DVB-S2 TV into segments of a MPEG transport stream.

For DVB-S2 HD programs, the stream already contains h.264 compressed video. However since we don't live in the 80's anymore where there used to be a thing called VPS that would make sure your VCR starts and stops when the program does – even if that does not match the time printed in the program guide – the recording will be enclosed by parts of the surrounding broadcast.

Getting rid of that "garbage" is a bit tedious, but here's how I managed.

The receiver writes files 00001.ts, 00002.ts...00012.ts of one gigabyte each to the attached USB drive. So first turn that into one nice big file by using cat (the target better be ext4 or something else that can handle 2G+ files).

cat *.ts > <name>.ts

Now the problem is we can't just use a media player and seek to the cut points because the data in this transport stream is not properly timestamped.

But first we need to figure out which streams (tracks) are present in the raw material. Often broadcasters transmit a bunch of redundant audio streams. Just in a single language of course, apparently to make sure viewers are not getting too much value for their license fee.

$ ffprobe <name>.ts
[...]
    Stream #0:0[0x18a6]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 100 tbc
    Stream #0:1[0x18b0](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 256 kb/s
    Stream #0:2[0x18b1](mis): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 192 kb/s
    Stream #0:3[0x18b3](mul): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 192 kb/s
    Stream #0:4[0x18b2](deu): Audio: ac3 ([6][0][0][0] / 0x0006), 48000 Hz, stereo, s16, 448 kb/s
    Stream #0:5[0x18ba](deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)

In this case, I'm only going to keep streams 0:0 (the h.264 video) and 0:4 (the AC3 audio). You can use mplayer with the -aid/-vid options to check out the various streams. To play the file with audio stream 0:3 for example, run as

mplayer -aid 0x18b2 <name>.ts

So now to get a seekable file with only the wanted streams, do this:

ffmpeg -i <name>.ts -f mpegts -c copy -bsf:v h264_mp4toannexb -map 0:0 -map 0:4 <intermediate>.mkv

Then open the file e.g. in gnome-mplayer, seek to the start and the end of  the actual content and make note of the timestamps.

Finally, fire up mmg (mkvmerge GUI), and add the freshly created <intermediate>.mkv file on the Input tab.
Then switch to the Global tab and enter the title of the recording into the File/segment title field.
Now check the Enable splitting... checkbox and select ...by parts.
Enter the begin/end timestamps noted down earlier into that field, separated by dash like 00:06:45-2:19:31.
Now verify that the Output filename is to your liking and hit the Start muxing button.

Autostarting screen on a server with programs running

There are some things I want my little home server to do permanently in the background. The corresponding programs may or should run in the context of the regular, non-privileged user. They may be interactive / full-screen console applications that I want to access from different computers around my network.

A way to accomplish this is to set up screen with an autostart configuration. The screen command is part of all modern Linux distributions, although it may not be installed by default.

To start the jobs at boot time without manual intervention, screen must be hooked up with the boot process. For gentoo this happens through an executable file /etc/local.d/*.start. Fedora uses /etc/rc.d/rc.local, which must be created in contemporary releases of the distribution. In both cases, the files must be set executable.

The call to screen from this system boot hook needs to be prefixed with an appropriate sudo to run in the context of the non-privileged user:

sudo -u <user> /usr/bin/screen -wipe

sudo -u <user> -i /usr/bin/screen -dm

From the screen manpage:

-d -m   Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts.

When the system was not shut down properly, screen session files may have been left behind which can prevent screen from autostarting on the next boot. Since this is in bootup and no other screen sessions are yet running, we can just use the -wipe option preventively to make sure no such leftovers remain.

There is an additional catch with Fedora as it sets Defaults requiretty in /etc/sudoers which prevents sudo from working here:

sudo: sorry, you must have a tty to run sudo

The solution is to add another line using visudo as follows:

/etc/sudoers

Defaults:root   !requiretty

Now create a directory .screen and a file .screenrc in the user's home directory. The directory holds a number of scripts containing the commands to be run in various screen windows, while the .screenrc file assigns these scripts to windows.

.screenrc

defflow off
screen -t shell0 0 $HOME/.screen/s0
screen -t pinger 1 $HOME/.screen/s1
screen -t nload 2 $HOME/.screen/s2

The defflow off directive turns off screen's flow control handling, which allows fullscreen console applications to receive and process the keystrokes Ctrl-S / Ctrl-Q. This can also be set per window (-f / -fn).
The -t option to the screen directive sets the window title; it is followed by the window number and the command to run.

.screen/s0

#!/bin/sh
fetchmail -d 3600
bash

This starts fetchmail in the background to retrieve mail each hour, and also gives me a shell in screen window 0.

 

.screen/s1

#!/bin/sh
pinger

Window 1 runs a tool that regularly pings a number of other computers and presents statistics about their availability.

 

.screen/s2

#!/bin/sh
nload -i 100000 -o 100000 -t 4000 -u K -U K eth0

This tool visualizes the utilization of the eth0 interface as an ascii-art graph.

See here for how to connect to such a screen session, and here for a general introduction to screen.

Readable MySQL dumps

Reading a text dump produced with mysqldump is normally hampered by the fact that the default options include --opt (for optimize), which is shorthand for a bunch of other options that make the resulting sql more concise and quicker to parse, but also very hard to read for humans as inserts are munched together into ridiculously long lines without any visual structure.

To prevent this from happening, use --skip-opt:
mysqldump -u root -p --skip-opt <dbname> > /tmp/sql.txt

For reading with less, the -S option is useful to prevent line breaking, which can be very slow.
less -S /tmp/sql.txt

Running ExtraDB on a Linux block device

Since the future of free MySQL appears a bit uncertain due to the fact that its owner happens to be well recognized for its self-developed proprietary database, I decided to have a look at the fork that is led by the original main developer of MySQL: MariaDB.

Interestingly, things have not changed much once you get your head around the fact that the database itself as well as the engines have been renamed in the documentation, but still use their familiar names in the config files.

So the state-of-the-art storage engine that used to to be InnoDB is now called ExtraDB, but all its config options still start out with innodb_. Also the engine names in table type specifications still use the old names. So mostly, you just replace MySQL with MariaDB and keep going.

There is one catch however when using a raw block device for ExtraDB tablespace: It no longer works to pass the absolute name of the block device's /dev file (eg. /dev/sdb1). Examining mysqld.err reveals that the daemon actually tried to access a file by the name .//dev/sdb1, which looks to me like somebody took Windows support quite a bit too far.

120116 15:21:21  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name .//dev/sdb1
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.

Fortunately there is a simple workaround for this bug, simply create a symlink or a replica of the respective device node inside the /var/lib/mysql directory and point the daemon to plain sdb1.

cp -a /dev/sdb1 /var/lib/mysql/


/etc/mysql/my.cnf:

innodb_data_file_path           = sdb1:10000269312raw

Bulk-shrink images with ImageMagick

I'm one of those (apparently few) people that are annoyed by emails showing me e.g. the location of a screw on a piece of metal with 3 pictures of 2 Megabytes each.

So as I'm trying to lead by example (uber-smug, ain't I?), when sending pictures I took with a digital camera myself I reduce those images to around 600 pixels along their larger dimension, which will do nicely in most cases and save the other party the trouble not only of downloading huge files but also of staring at a tiny irrelevant detail of the picture that happens to be in the top left corner after the image viewer has ceased up their entire screen.

The best interactive tool for the occasional view-crop-correct-resize-compress task is obviously gthumb, but when faced with a larger number of images, it's commandline-time:

mogrify -quality 82 -resize 600x600\> *.jpg

This will shrink all jpg files in the current directory to fit within a 600x600 pixel bounding box while preserving aspect ratio (so the images are not distorted as one might first think). The closing angle bracket at the end ensures that images that are smaller than the target size are not enlarged. Note that the mogrify command changes the images in place, so you may either want to make sure to have a backup of the original files, or go for find/convert instead.
For more information, here is the resize operator's official documentation.

Another little-known but very excellent feature of the ImageMagick suite is the -strip operator, which will remove all EXIF data from an image. This can be useful especially when the picture was taken on a modern smartphone which will normally embed all sorts of information in the image's metadata such as location and time of the shot, as well as the model name of the device the image was taken with. Maybe you wouldn't want to share that with everyone that should see the picture.

Lowering the (re)bar

Although the Erlang programming language has been on my radar for many years now, I never quite grasped the concept of how to put things together such that a bundle of functionality might be properly deployed to and run on a target system. The target system would of course be Linux :)

Fortunately, in the last couple of years an effort has emerged that promises to make things easier. It's a tool called rebar, and its canonical use seems to serve as the build system for a database application by the name riak. Some of the participant's names on the rebar mailing list ring familiar, so it seems possible that it may even be adopted into OTP at some point.

There are quite a number of "getting started" tutorials around for rebar, one right inside the rebar github wiki, and also this one being still quite recent early in 2012. rebar is (or at least has been) a moving target, and some tutorials you may find are badly outdated.

However even following the supposedly applicable instructions, I invariably ran into this error at the generate stage:
$ ./rebar generate
==> rel (generate)
{"init terminating in do_boot","Release exemplar uses non existing application exemplar"}

Crash dump was written to: erl_crash.dump
init terminating in do_boot (Release exemplar uses non existing application exemplar)

It took me quite a while to get past this problem, and the key here is that the Erlang runtime needs to accept your working directory as an application, which requires two things to be true:

1. The directory that contains the src and ebin directories, as well as rebar.config (not rebar.conf btw!) needs to have exactly the same name as the Erlang application you are creating.
2. The directory which comprises that application directory needs to be part of Erlang's lib_dirs (an extension searchpath for /usr/lib/erlang/lib) so it will be considered by the runtime when it is looking for applications.

What is left out by all the tutorials I found is that for #2 to become true, one needs to manually edit the reltool.config file that is created by the rebar create-node step and edit the lib_dirs assignment to include that parent directory (can be an absolute path, or given relative from the rel directory).

So if you are trying this out in /tmp/exemplar like

/tmp/exemplar
|-- deps
|   `-- exemplar
|       |-- ebin
|       `-- src
|           |-- exemplar_app.erl
|           |-- exemplar.app.src
|           `-- exemplar_sup.erl
|-- rebar.config
`-- rel
    |-- exemplar
[...]
    |-- files
    |   |-- erl
    |   |-- exemplar
    |   |-- exemplar.cmd
    |   |-- nodetool
    |   |-- start_erl.cmd
    |   |-- sys.config
    |   `-- vm.args
    `-- reltool.config

then your /tmp/exemplar/rel/reltool.config file should be changed to start like

{sys, [
       {lib_dirs, ["/tmp/exemplar/deps"]},
[...]

At first I was wondering whether this manual edit is really the right thing to do since this file is generated, but checking out the riak rel directory, the reltool.config file there is checked into git and also contains a manual lib_dirs assignment.

Another mistake to avoid is naming your node (rebar create-node nodeid=<name>) anything different from your main application, which will result in an obscure error message

Command 'generate' not understood or not applicable

Adding a user in mysql for a specific database

Whenever installing some new webapp on a LAMP stack, the typical procedure goes such that the app gets its own database in mysql/mariadb/drizzle/whatever and needs a user to access it.

The database is of course created like

create database <database>;

To add the user, one usually uses a command like this in the mysql client shell:

grant all privileges on <database>.* to '<new_user>'@'localhost' identified by '<some_password>';

In which I like to use a randomly generated password (e.g. using makepasswd) for each user.

Besides assigning rights, the latter command also creates <newuser> as a side effect, and even flushes privileges in one fell swoop.

One might further want to prune the granted rights to only work on the actual data, not allowing modification of the tables/schema, but most webapps come with an installer built in that creates/updates the app's tables so this is left as an exercise for the reader.