Transferring Picasa photo captions / descriptions to iPhoto and Aperture

If you ever find yourself trying to transition from Google Picasa to iPhoto, Aperture or indeed many other photo management packages, you may find that your meticulously entered captions and descriptions are lost. The reason is that, as with most photo management tools, Picasa stores this information in the metadata of the image file. However, different tools use different metadata fields to store captions with Picasa using the IPTC Caption-Abstract metadata field. Unfortunately, iPhoto and Aperture don't read this field (they use the Exif Image Description field) and so importing pictures from Picasa will normally result in you losing this information.

However, as has been pointed out elsewhere there is a way you can fairly easily copy your existing captions to the right field.

1. Get and install exiftool which is a command line tool for manipulating photo metadata 
2. Copy the IPTC Caption-Abstract metadata to the exif Image Description:
   exiftool "-exif:ImageDescription<IPTC:Caption-Abstract" -r -P --progress <targetdir>
   Just replace <targetdir> with the directory containing all your photos and this command will recursively process all your photos (reasonably quickly too!)
3. Import you photos into either iPhoto or Aperture

exiftool makes a backup of all the images it modifies and will give you a nice summary of how it went.
Even so, it is probably a good idea to make a backup of your valuable photos before running it on your whole library.

How to: make use of a locked VoIP ATA with asterisk

So you've got a VoIP ATA but it's locked to one provider and you want to make better use of it?

You've searched all over the net on how to crack or hack the firmware to unlock it but given up all hope?

Well, I've worked out an interesting way to get around this problem. This works with my Netgear TA612V that is locked to MyNetFone here in Australia. Essentially, what you can do is set up an asterisk server to pretend to be your provider. Here is roughly how I did it:

You'll need:

- a DNS server such as bind (and know how to set it up)

- to be able to redirect DNS requests to your bind server (some ATA's don't use your network's DNS server provided by DHCP) - i.e. you'll need to be able to add a port redirect on your router

- asterisk and the desire to learn how it works and to set it up

A note, my server has the IP address 10.0.0.250

1) Set up bind to redirect the domain name of your provider's sip server to your own. I did this by setting up a hosts file in bind for sip01.mynetfone.com.au and adding the following host:

sip01.mynetfone.com.au. IN A 10.0.0.250

2) Redirect all UDP port 53 requests from your local network (but not your server!) to your server. I use shorewall firewall and did that with the following line:

REDIRECT loc 53 udp 53 - !10.0.0.250 # Transparent DNS

3) Add accounts in your asterisk sip.conf for your ATA to log into. The following is what worked with my TA612V:

[09*****4] ; ATA line 1

canreinvite=yes

type=friend

secret=********

host=dynamic

dtmfmode=rfc2833

context=ta612v

[09*****5] ; ATA line 2

canreinvite=yes

type=friend

secret=********

host=dynamic

dtmfmode=rfc2833

context=ta612v

Naturally, I've *'d out my username and password. A note, I originally found that this didn't work if I had both lines set up in Asterisk (there was a bug with the TA612V and old versions of asterisk) however, with 1.6.2 this seems to be fine now.

You'll then need to edit extensions.conf in asterisk to create dialling plans. Unfortunately, as the ATA will be programmed with you'r providers dialing plans, you won't be able to set up extensions (dial 1 to 9 for each extension or dial 0 for an outside line). What I mean by this is that (for example) my ATA is set up to accept either a 10 digit number if the first digit is 0 (eg 02xxxxxxxx or 04xxxxxxxx) a 10 digit number for numbers starting with 1300 or 1800, a 6 digit number for other 13xxxx numbers and also to send 121 to voicemail. This is all needed because the ATA needs to know how many number presses to wait for before it sends the dialled number and create the call.

This is how I set up my extensions.conf

[ta612v]

exten => _X.,1,Dial(SIP/${EXTEN}@othersipprovider)

exten => _121,1,Dial(SIP/111@othersipprovider)

exten => _09X.,1,Dial(SIP/${EXTEN}@mynetfone)

You can see I've set up outgoing calls to go via an alternative sip provider, I've configured dialling 121 to go to my other sip provider's voicemail (which is 111) and I've configured all 09 numbers to go via mynetfone (as 09XXXXXX is the format of mynetfone's direct numbers). In theory, you could set up an extension (like 121) to connect you to an asterisk IVR extension where you could select which extension to dial or which provider to dial out on. I'm thinking of doing this but haven't had the time to look into this. Asterisk is an immensely powerful voice service and the possibilities are endless once you've got your ATA working with it.

Legalities: Whilst hacking or cracking the firmware of a locked ATA is definitely a grey area legally, (whilst I'm no lawyer) I feel that the above approach is generally OK. In doing the above at no stage have you had to reverse engineer any form of encryption or even modify the firmware of your ATA.

What surprised me is that there is no challenge/handshake authentication that occurs between these ATA's and the provider. In theory it would be incredibly easy to intercept these authentication details on a network and hack into someone's SIP account (just use some packet sniffing software and you can see for yourself the SIP authentication process with your username and password there in plain text) - however this would be illegal, I'm just pointing out how insecure I found the whole system to be!

Finally, I know I haven't been very detailed with this how-to but the reasons are:

1) I'm short on time at the moment

2) There are a huge number of combinations of the tools and devices and providers - I've provided you with the idea on how to do it, it's up to you to work out how to do this with your setup

SATA AHCI mode on systems without BIOS option

Background: I've got a Dell Poweredge SC430 I bought a couple of years back.

I recently put 4 new WD 1TB drives in it which support native command queuing and installed the latest Ubuntu 9.04 whilst I was rebuilding my server.

Anyway, I noticed linux was only using the ata_piix module for my onboard SATA ICH7 family controller (integrated in the Intel Chipset). As some background, many SATA controllers offer a legacy mode (ATA/IDE emulation mode) and native mode (AHCI or RAID). Normally you can change which mode the controller is in via the BIOS hoever for whatever reason Dell chose not to include this option in my system.

Problem: The hardware ID that the controller reports to the system is dependant on which mode the controller is in. This means that the OS can load the appropriate driver depending on the mode of the controller. For Windows users the solution is aparently easy, you can force Windows to use the AHCI driver when the controller is still in the legacy mode. I presume the driver then detects that the controller is in legacy mode and instructs it to change to AHCI mode on the fly.

Unfortunately for linux this isn't exactly easy to do. To force linux to use a particular module with hardware that it doesn't have it must have it's hardware ID listed in the driver when the module is built. This means compiling the kernel from source. This also means re-compiling the kernel from source whenever you want to update your kernel, not something I want to have to deal with.

Fortunately some clever Mac Pro users have found another way. Instead of modifying the linux kernel, they edited the stage1 part of the grub bootloader to add some assembly commands to instruct the controller to switch modes. This is a fantastic hack for several reasons:

1. As grub can boot virtually any OS, this hack will ensure AHCI mode is available for any OS, not just linux
2. Grub, and especially the stage1 file, is relatively static and doesn't get updated too often, so no need to worry about patching new versions continually
3. No need to worry about kernel updates clobbering the patch and no need to compile the kernel from source every time there is an update

Solution: Note, at the time of writing this, I used grub 0.97, you might need to change some directory paths appropriately if you use a different version.

This patch, whilst originally written for a Mac Pro with an Intel 631xESB/632xESB Controller just happens to work perfectly on my Dell with an ICH7 Controller. Note that I did indeed verify that the commands to switch SATA mode were identical by comparing the Intel documentation here: http://www.intel.com/Assets/PDF/datasheet/313082.pdf Port Mapping Register (SATA–D31:F2) on page 764 and here: http://www.intel.com/Assets/PDF/datasheet/307013.pdf Sub Class Code Register (SATA–D31:F2) on page 495.

You should verify at least two things before trying this patch:

• ensure your chipset supports AHCI (not all ICH7 variants do)
• ensure your chipset has the same format command at the same offset to switch modes (if it is an Intel, I'd hope that they're consistent, they seem to be so far)

If you're happy to try this out, here's how you do it:

1. Get the source of grub, in ubuntu:

   sudo apt-get source grub

2. If you're using and x64 OS you'll need the 32bit libc dev files:

   sudo apt-get install libc6-dev-i386

3. Get the patch:

   http://boeglin.org/static/macpro/grub-0.97_macpro_esb2_ahci_stage1_new.patch

4. Apply the patch

   patch grub-0.97/stage1/stage1.S grub-0.97_macpro_esb2_ahci_stage1_new.patch

5. Configure grub

   cd grub-0.97; ./configure

6. Build grub:

   make all

7. Find your grub installed libraries. eg:

   cd /usr/lib/grub/x86_64-pc

8. Backup your grub existing grub stage1 file
9. Copy the compiled stage1 to your grub library directory
10. Update your grub bootloader. eg:

    grub-install hd0

11. Note: grub-install will overwrite /boot/grub/stage1 with the stage1 from your library directory so you'll need to make sure you're putting the patched stage1 in the library directory NOT the /boot/grub or equivalent directory
12. Reboot and verify your controller is now using the ahci module!

pppd persist not so persist with udev

OK so this is my first official rant.

Background: I have a dell server which (amongst other things) shares my Internet connection here at home. It essentially shares a persistent pppoe connection with the rest of my network. Great simple use of linux you agree? Well in the pre-udev world it was!

Up until a couple of weeks ago this machine happily ran Debian however I was getting a little tired of Unstable's slow release schedule and Testing breaking things every update. So I decided to switch to Ubuntu 6.10 which I've been using on my test machines at Uni and was quite happy with.

Problem: After setting everything up I noticed that whenever my ppp link got disconnected it didn't automatically reconnect. After a quick examination I discovered that the pppd daemon was being killed once it disconnected:

pppd[15890]: No response to 4 echo-requests
pppd[15890]: Serial link appears to be disconnected.
pppd[15890]: Connect time 2436.6 minutes.
pppd[15890]: Sent 410764979 bytes, received 645991881 bytes.
pppd[15890]: Connection terminated.
pppd[15890]: Modem hangup
pppd[15890]: Terminating on signal 15
pppd[15890]: Exit.

Solution: After a LOT of googling and stumbling on other users with similar problems a SuSe bugzilla bug report suggested that there was a problem caused by udev. The Novel bug report is here: https://bugzilla.novell.com/show_bug.cgi?id=211936

and a copy (Novell seem to remove public access to bugs once they are solved... "thanks for fixing this bug in our open source software, now we won't tell anyone about the fix") here:
http://lists.opensuse.org/opensuse-bugs/2006-10/msg02573.html

OK now after looking into this further I found reports that under Debian based distros one of the udev scripts was broken. Specifically: /etc/udev/rules.d/85-ifupdown.rules.

A note to users of other distros, this may be a different script in udev for you, perhaps the best way to find it is to: grep "ifup" /etc/udev/rules.d/*

My original version of this file contained the following:

# This file causes network devices to be brought up or down as a result
# of hardware being added or removed, including that which isn't ordinarily
# removable.
# See udev(8) for syntax.

SUBSYSTEM!="net", GOTO="net_end"

# Bring devices up and down only if they're marked auto.
# Use start-stop-daemon so we don't wait on dhcp
ACTION=="add", RUN+="/sbin/start-stop-daemon --start --background --pidfile /var/run/network/bogus --startas /sbin/ifup -- --allow auto $env{INTERFACE}"
ACTION=="remove", RUN+="/sbin/start-stop-daemon --start --background --pidfile /var/run/network/bogus --startas /sbin/ifdown -- --allow auto $env{INTERFACE}"

LABEL="net_end"

The first fix (which was found by googling) was to remove "--" after the ifup and ifdown commands. A little explanation of what's going on here. The "--allow auto" option is meant to be set so that the interface specified by $env{INTERFACE} will only go up/down if it is prefixed AUTO in /etc/network/interfaces like this: auto eth0 or this: allow-auto eth0. The problem is the extra "--" breaks this.

Now this is just the start of the problem. The real issue here is that the udev system is essentially running "ifdown ppp0" whenever my pppoe connection gets disconnected. As you may know, running ifdown on a pppd based interface sends a TERM signal to pppd (and so pppd closes without being able to reconnect). The SuSe fix to this is to put a line something like the following in the udev script so that special interfaces (such as ppp) are not ifup/ifdown by udev:

SUBSYSTEM=="net", ENV{INTERFACE}=="ppp*|ippp*|isdn*|plip*|lo*|irda*|dummy*|ipsec*|tun*|tap*|bond*|vlan*|modem*|dsl*", GOTO="net_end"

This basically will skip the add and remove actions in the udev script mentioned earlier. I, however, disagree with this fix and propose the following (and IMHO cleaner and more cautious) fix. I propose the following two lines are changed accordingly:

ACTION=="add", RUN+="/sbin/start-stop-daemon --start --background --pidfile /var/run/network/bogus --startas /sbin/ifup --allow=hotplug $env{INTERFACE}"
ACTION=="remove", RUN+="/sbin/start-stop-daemon --start --background --pidfile /var/run/network/bogus --startas /sbin/ifdown --allow=hotplug $env{INTERFACE}"

And also the following line be put in /etc/network/interfaces for each and every interface that the user indeed wants udev to ifup/ifdown:

allow-hotplug eth0

What my solution does is to bring back the hotplug class (I think which came from the hotplug days before udev) so that udev will only ifup/ifdown interfaces that are marked as hotplug(able). To me this seems to be the original intent of the --allow=auto option. Why don't we just make ppp0 not auto? Because it is a persistent connection that needs to come up on boot.

Rant: Now I'm very pleased that this is all fixed. BUT I am very pissed off that a crappy bug like this can make its way into a release that has been out for so long! Also, who the hell wrote the udev script and OBVIOUSLY didn't test it?! The thing that gets me is that I'm not doing a really obscure task here, I'm just trying to create a persistent ppp connection! Isn't this one of linux's niche markets (cheap simple internet gateway)?! In Windows I tick a box in my dial-up connection properties and IT JUST WORKS.

If people want linux to be taken seriously a bit more professionalism over issues like this is needed. Why the hell an update hasn't been released to fix this simple yet very silly bug is beyond me. Also, when udev replaced hotplug, who on earth tested it? One of linux's biggest problems is that their testing and quality assurance seems to be "leave it in beta until people stop posting bug reports, then presume it's been tested enough". Even Microsoft has worked out that this just doesn't work. Some sort of system needs to be in place which documents how much (if any) testing has been actually performed on each section of code (and I mean testing where you sit down and test for the sake of testing, not "I'll just use it in one of my production machines for a week and see if it falls over").

OK I feel better now, and I sincerely hope this helps those of you I've seen posting problems related to this. I feel for you!

Update (4/4/2007): Good news! After 4 months of the bug sitting in the Ubuntu bug tracker I have just received word that it has been fixed. Not sure what package and version the patch will be in though...

Fixed rules to only take affect for devices with drivers

** Changed in: ifupdown (Ubuntu)
Sourcepackagename: udev => ifupdown
Status: Unconfirmed => Fix Released

Update (18/11/2009): Just to clarify current behaviour of udev and pppd...

The current behaviour with udev is that interfaces marked auto in /etc/network/interfaces are told to ifdown when udev detects that the connection has terminated. With pppd this causes it to be sent a TERM signal causing pppd to end and breaking the persistent properties of the connection.

Due to this it is best not to mark ppp interfaces auto.

Let the rants begin

I always thought myself too lazy to blog but I've realised there are a few things in life that give me enough motivation. So, here in this blog you'll find my rants and raves about things that piss me off enough for me to come on here and let you know. Considering I'm a graduate Computer Engineer/Scientist (and I'm currently doing a PhD) (un)fortunately you'll probably find most of these rants focused on technology.

I aim for this blog not only for me to vent frustration but also share the solutions that I hopefully find in the end. Feel free to reply with a comment to any of my posts.

Thanks and good luck everyone!

Alan