Monthly Archives: January 2007

Internet Review

Yahoo search does not obey robots.txt

January 25, 2007 –

Since Dec 1st last year I have the following robots.txt in my webroot:

User-agent: *
Disallow: /

And still after 6 weeks I receive hits on my website from for example lj612201.inktomisearch.com. This is the Yahoo search-engine webspider.
I find this unacceptable search-engine behaviour. For that reason I have blocked 74.6.0.0/16 on my firewall.

How hard is it to build a search-engine which takes the effort to fetch the robots.txt regularly? Google does this. Although I can still find my website when I search on Google. Does anyone know how long it takes for Google to refresh its contents?

Tags , | Permalink | Comment (0)
Review Software

TODO in Desktop Linux III: an ode to Unison

January 23, 2007 –

This is the third post in a series about open issues in the current offerings of Desktop Linux. In this post I write about my favourite Open Source application which is Unison.

In my daily work I use two laptops; one at home and one at work. I need to have my document repository available on both machines. Moreover I am sometimes in a situation where I connect to a network which is not my office network or in some rare occasions I am even offline. In all these situations I need up to date access to my documents and email. In all these occasions email has an easy solution. I use an imap server to store my mail and use Thunderbird to sync my mail tree. But the document repository is more of a challenge. Try to solve this with the available Gnome tools on a standard installation of Ubuntu. It is simply not possible. Here comes Unison to the rescue.

All my documents are stored on my home server. That could easily be an backoffice server but this has historical reasons. It is essential I have my documents available on both laptops and for that I need reliable two-way synchronization of the document repository. The synchronization needs to be able to work in different locations with different types of internet access. In some cases I am on my office LAN but in other cases my documents need to flow over the public internet via public internet gateways. Therefor security is essential. The second requirement is reliability. The synchronization can take a few minutes to an hour depending on the size of the accumulate update of the repository and the available bandwidth. The third and last requirement is that the synchronization needs to take place in the background. I am just too busy to remember to synchronize. It needs to happen transparently.

The solution to this problem is a combination of OpenSSH with a password protected private/public key combination and Unison. SSH provides the essential data encryption combined with solid access protection of my document repository. Unison takes care of the two-way synchronization. In my 5 years of use it proves to be a very reliable tool which survives unreliable and slow links, mid-way disconnections and large updates.

For background synchronization I use cron which can login without a password to my server when I enable ssh-agent key forwarding. In my Gnome session I autostart ssh-askpass which asks me for my private key passphrase during logon. During my session cron can start a script I wrote to access my ssh-agent to use the private SSH key to login to my server and start Unison.

To give you an idea on the reliability and scalability of Unison. A few years ago I used Kontact/Kmail for my email. And although Kmail supported IMAP, it did not synchronize my email. When I was offline I could not access it. I solved this using Unison. At that time I used Courier IMAP which stores the user’s email in a directory called Maildir in their home directory. I synchronized this email repository containing 10.000+ emails with my laptop and ran a local Courier IMAP server to access my emails through Kmail. This worked very reliable. A synchronization run only took 10 seconds when no changes were detected which is an amazing feat in itself by the people who wrote this great tool.

When I find the time I will write a small howto on the setup of Unison together with OpenSSH and ssh-agent key forwarding.

But to come back to the original theme of this series of posts I am disappointed that Unison is still not integrated in the Desktop environments of Linux. Often users find themselves in a situation they need to have reliable and scalable two-way synchronization of data. The tools are available but the Desktop Environment does not give easy access to these tools. For me personally this would be a killer feature and I am not alone.

Tags , , | Permalink | Comment (0)
Review Software

TODO in Desktop Linux II: networking

January 22, 2007 –

This is the second post in this series. The subject in this post is networking for a laptop. Like in the previous post I use Ubuntu Edgy 6.10 as a reference platform.

One of the main characteristics of a laptop is that it is used in different locations. Different locations implies different network connections. In my daily use I connect to the internet through UTP, Wifi and a Bluetooth/GPRS combination. The bluetooth/GPRS connection is a connection to my phone using bluetooth which is connected to the internet through GPRS, a 2.5G mobile network service here in Europe.

The main setup of my laptop is to have UTP enabled with DHCP combined with a fixed wifi setup for my home network using a WEP-key. This works reliable. The problems start when I want to use another Wifi network or Bluetooth/GPRS.

For connecting to a different Wifi network I need to go in the networking tools of Gnome and often need to issue commandline commands to find the available Wifi networks at that location. This is not what I expect. I expect that my Operating Systems tells me it cannot find any default (or configured) networks to connect to with a question if I would like to search for a new network. I expect to have a list of available Wifi networks with their characteristics like their SSID and encryption setting. Based on that network I would like to choose which network to connect to.

Likewise with the Bluetooth/GPRS combination. This is a dialup type of network which should be an option I can select and configure from my network configuration dialog. It is possible to setup such a connection using the available commandline tools and utilities. But there are no configuration dialogs available. Without the expert knowledge on how to setup such a connection it is virtually impossible for the average user. Even though the configuration in itself is not very complicated. It is possible to capture all the information in a configuration dialog and let the user click through the setup. Just look at the Maemo / Nokia N770. They have that in their distribution.

For me the creation and maintenance of these kind of configuration dialogs is one of the primary add-on functionalities distribution can bring. Even though that the development would benefit all Linux distributions it is essential that some distribution vendor picks it up and develops it. I will leave my vision on the community and distribution interacting scenarios for another post because this topic deserves an article on its own. It is the foundation of the Open Source / Linux ecosystem.

Tags , , , , , | Permalink | Comment (0)
Hardware Review Software

TODO in Desktop Linux: monitor configuration

January 20, 2007 –

This is the first post in a series. I use Linux daily as my workhorse but I also use Windows (currently XP). There are differences between Linux and Windows. And though I can do most of things I want in Linux there are still specific issues with running Linux as a desktop. In this series I highlight a specific issue per post. For your information I use Ubuntu Edgy 6.10 as a reference platform.

In this first post I start with one of the major problems I encounter during business use of Linux. The situation is as follows. I have a laptop. The laptop is mostly used in three configurations:

  1. Standalone
  2. Connected to a desktop LCD monitor
  3. Connected to a projector

The issue is that X does not recognize any monitor which was not connected during startup. So when I resume my laptop from standalone use and I hook it up to an external monitor, the monitor is not recognized or used by X. When I start up the laptop with the external monitor in depends on my X configuration what happens. In some cases the external monitor is used exclusively and in other cases the external monitor and the laptop LCD are showing the same picture (“cloning”). And can you imagine how this looks when you have an widescreen laptop LCD screen and a 4:3 LCD external monitor?
The expected behaviour is that whenever you connect an external monitor a popup is shown asking what I would like to do. Much in the same way as the desktop handles the insertion of a DVD or CDROM. Moreover I need to be able to change these settings during runtime. Although not perfect Windows is at least capable of doing this. It does not recognize the monitor on connecting but at least it enables you to configure when it is available.

The development of X.org even introduced a regression issue with regard to my Thinkpad Fn+F7 button. With this button I could change the graphics card output selection. But since X.org the radeon driver is not able to this anymore.

In some internet post I read that dynamic configuration of monitors is part of the feature roadmap of X.org R7.3. So there is hope that this issue is finally resolved sometime in the next years. I truely hope so!

Tags , | Permalink | Comment (0)