iftop - display bandwidth usage on an interface by host
on 25.01.2009, 05:00
in packages-news
atop: an ASCII full-screen performance monitor
on 18.01.2009, 05:00
in packages-news
A Few Questions For Jeremy Malcolm
on 12.01.2009, 18:21
in interviews
Debian Project News 2009/01
on 12.01.2009, 00:00
in weekly-news

iftop - display bandwidth usage on an interface by host

published on Sun Jan 25 05:00:24 2009 in packages-news

Article submitted by András Horváth. We’re running out of articles! If you like Debian Package of the Day please submit good articles about software you like!

Sometimes it’s hard to guess what is eating up our bandwidth the most - on a home computer with an xDSL connection, for example. We can run a lot of programs simultaneously while they can eat as much bandwidth as they want independently of their CPU usage.

iftopWithout by-host statistics, it would be hard to guess what is slowing down our internet connection the most. iftop helps us to find out with a simple curses based interface and real-time statistics output calculated on a by-host basis. When starting it from a terminal, it starts collecting data and printing them on the screen in lines separated by host name pairs or IP addresses, showing the highest network usage. It refreshes the screen every 2 seconds.

iftop works by reading the host names out of the network packets traversing the interface and then associating them together.

iftop screenshotAfter identifying the host name, we can easily guess which program is generating this particular traffic — that’s why iftop must be run with root privileges.

Though ifstat can be used with complex filtering rules, running it without parameters gives enough statistics in most cases. iftop tries to listen on eth0 by default which might not exist at all. To specify an interface on which to listen, use the -i parameter: iftop -i wlan1, iftop -i eth1.

Pressing the h key while running will display a nice on-screen help, showing the commands that can be triggered interactively in running mode. Some other examples of key commands changing the default state:

  • n turns off name resolving.
  • p turns on port number display.

Pros and Cons

  • Pros
    • Per-host statistics of network usage.
    • Graphical representation of statistics with character bars.
  • Cons
    • Host names are not bound to process names in the statistic, so you have to figure out by yourself which processes they belong to.

Official packages have been available in both Debian (since Etch) and Ubuntu (universe) for a long time now.

atop: an ASCII full-screen performance monitor

published on Sun Jan 18 05:00:44 2009 in packages-news

Article submitted by András Horváth. We’re running out of articles! If you like Debian Package of the Day please submit good articles about software you like!

Every system administrator must be familiar with the top(1) command that shows the most active running processes in a Linux environment.

atop is different than that: it shows the most active processes not only related to CPU or memory, but disk and network usage are also calculated into the resource usage statistics. It also calculates incremental statistics, like VGROW (virtual size grow) and RGROW (resident set size grow). For example, if a process is leaking memory, its RGROW will always show positive values.

It can happen that a process starts using the hard disk of the computer making a heavy I/O traffic that takes less CPU resource though, still making the computer response slowly. With the atop utility we can discover this process easily.

atop screenshot For example, if we start the ls -R / command and then run atop, plus we type Shift+A after to get statistics about the processes that use the most system resources, we will see the ls command in the first place with a ‘D’ flag next to it —showing that its mostly used resource is related to Disk and not CPU. You can see this in the screenshot. atop highlights in red the resources if their usage is in critical limits. In the same example, both CPU and disk are being heavily used.

It is recommended to run atop with root privileges in order to see information from all processes. Moreover, when running with root privileges, atop can report about recently finished processes and their exit status.

atop refreshes the information every 10 seconds giving precise statistics. If you want it sooner, just press t to trigger a manual refresh. Pressing h key will give a detail-full on-screen help.

atop installs a daemon service by default and starts himself at boot time. The daemon stores system activity data in /var/log/atop.log. Using the -r command-line option, you can analyze the system activity recorded. Using keys t and T you move forward and backwards in time respectively. You can generate more system activity reports with the atopsar utility, included in the atop package. You don’t need this daemon if all you want is a real-time monitor.

atop can also report about network and disk utilization per process, but you need a patched kernel. Debian packages kernel-patch-atopacct and kernel-patch-atopcnt provide them. Sadly, they are not very well maintained. They were removed from Debian Etch and, in Lenny, the patches included only cover until version 2.6.24 of the Linux kernel (the same happens in Ubuntu). More up to date patches can be found in atop’s webpage. A guide on how to patch and rebuild the kernel is beyond the scope of this article.

  • Pros
    • per-process statistics related to CPU, memory, disk and network.
    • Ability to analyze past system behaviour.
  • Cons
    • Per process disk and network statistics requires patching the kernel.
    • atop installs a daemon service by default. Some people think that the atop daemon should be optional.

You can find atop both in Debian (since Etch) and Ubuntu (since Dapper).

A Few Questions For Jeremy Malcolm

published on Mon Jan 12 18:21:55 2009 in interviews

A Few Questions For Jeremy Malcolm

How did you end up using Debian and becoming a DD?

I became interested in using Linux in around 1994, even before I had an Internet connection at home. I think I first heard of it from a mail-order shareware vendor (yes, that's how people like me obtained software in those days — by mail order!) I ended up ordering a CD pack that contained a number of distributions, and trying it out. Debian 0.93 was included, along with Slackware and Red Hat. I tried Red Hat for a year or two — I remember it used fvwm95 as the default Window Manager which seemed to me to defeat the purpose a bit.

In 1995 I became involved in the administration of a local community-based ISP, the Australian Public Access Network Association (APANA), and began to improve my Linux skills from the use of my Slackware shell account. It must have been in 1997 that decided to switch to Debian because it seemed to have the strongest commitment to the ideals of open source. I ordered a CD of Debian 1.3 (bo) from another CD vendor and at first installed it as a dual-boot with Windows 95.

In 1998 through a friend I had originally met through APANA who ran a hosting company, I set up a co-located server mostly for personal use and to host Web sites for clubs I was involved with. I installed Debian 2.0 (hamm) on it, and it became the first server for an IT consultancy that I established called Terminus Network Services. It still exists today under a different name and different management.

A couple of years later I joined the Perth Linux User Group, PLUG, and heard from James Bromberger about his experiences in joining Debian as a developer and packaging libapache-mod-backhand. I decided that there were packages that I would like to see in Debian too — not least one that I had written in gtk-perl called gtkgrepmail. So I applied to become a DD myself, and was accepted in 2002, with gtkgrepmail as my first package.

Next I packaged dbengine, which was one of the first Web-based interfaces to PostgreSQL and MySQL, preceding tools like phpmyadmin. Neither it nor gtkgrepmail are in Debian any more. In 2003 I also started a project called Debian-Lex to make a customised distribution for lawyers (because I was a lawyer in my day job), but nothing came of that unfortunately because of my lack of time and inability to attract a community of helpers.

How are you currently involved in the Debian project?

I currently maintain a Python Jabber and IRC bot called gozerbot and a DocBook-based wiki, called — er — docbookwiki. I can't say that I spend much time maintaining them! Normally the most that I do is to upload new versions and translations, which takes only about an hour a month.

How do you currently use Debian?

I run lenny/sid on the virtual server that hosts my personal domain malcolm.id.au. At home I now run Ubuntu — I have a MythTV server, a print and file server, and a virtual machine on my MacBook. At work I convinced our management to install a new Ubuntu server in place of the Windows server they had planned, and I have also installed an Ubuntu virtual server to host a Web site for one of the projects I'm coordinating.

What do you do when you're not working on Debian?

I work for an organisation called Consumers International, based out of its Asia-Pacific office in Kuala Lumpur, coordinating its projects on Access to Knowledge or A2K. I gave a talk at the inaugural foss.my conference here in Malaysia about the linkages between free and open source software and A2K. In fact, the ideals of the open source movement and the A2K movement are very closely aligned. We are trying to work for fairer access to intellectual property for consumers. Of course, open source is a big part of that.

Debian Project News 2009/01

published on Mon Jan 12 00:00:00 2009 in weekly-news

Welcome to this year's 1st issue of DPN, the newsletter for the Debian community. This issue is dedicated to Thiemo Seufer, who died on 26 December 2008 in a tragic car accident.

Bits from the Debian CD team

Frans Pop sent Bits from the Debian CD team, summarizing various changes affecting the installation media for the upcoming release of Debian GNU/Linux 5.0 Lenny.

The most notable change is the replacement of the Xfce installation disc with a "light desktop" installation disc for installing Xfce or LXDE. Furthermore, all four major desktop environments (GNOME, KDE, Xfce and LXDE) are directly installable from the first DVD image; the desktop environment to be installed can now be chosen via the boot menu.

Bits from the Debian Installer team

Christian Perrier sent bits from the Debian Installer team. With the upcoming release of Debian GNU/Linux 5.0 Lenny in mind, he summarized the improvements made during this release cycle, including (but not limited to) support for loading firmware during installation, SATA RAID support, early upgrade of packages for security fixes, support for volatile, a new boot menu for i386/amd64, and support for installation from Windows through win32-loader.

Results from the Lenny release GR

Bdale Garbee posted the results of the Lenny release GR, a general resolution about the handling of firmware issues and similar bugs affecting Debian GNU/Linux 5.0 Lenny.

New maintainer for search.debian.org needed

Thomas Viehmann is seeking a new maintainer for search.debian.org. The search engine is a slightly patched xapian-omega instance. You should be a Debian Developer or be willing to be part of a team for access to the installation. Some more improvements to the language support are desirable, which involve some C++, but Thomas would try to help out there.

Security support for next testing (Squeeze)

The Testing Security Team announced that security support for the next Debian testing (codenamed Squeeze) will not begin immediately after the release of Lenny as stable. Users of Debian testing who need security support should stay with Lenny until the beginning of security support for Squeeze is announced.

Internationalization support for planet.debian.org

Jörg Jaspert announced that planet.debian.org, a service collecting the personal blogs of various Debian Developers, Maintainers and other Debian-associated people, is now capable of hosting web pages and RSS/Atom-feeds in different languages, as demonstrated by the Spanish Planet Debian.

Personnel change of the Debian Secretary

Due to criticism while conducting the recent Lenny release GR vote, Manoj Srivastava, long term secretary of the Debian Project, decided to resign from this position. Following his resignation many developers have thanked him for his work.

Personnel changes in Debian's Technical Committee

After being member of Debian's Technical Committee for three years, Anthony Towns resigned from that position to get fresh blood into the committee. We would like to thank Anthony for the work he has done over the past years.

Personnel changes in the New Maintainer front desk

Christoph Berg announced some personnel changes in the New Maintainer front desk, handling applications of people interested in becoming Debian Developers. Marc Brockschmidt, who has been a member since 2005, has resigned from this position, while Bernd Zeimetz has joined the team.

Other news

The 12th issue of the miscellaneous news for developers has been released and covers the following topics:

New Developers and Maintainer

Three applicants have been accepted as Debian Developers and one applicant has been accepted as Debian Maintainer since the previous issue of the Debian Project News. Please welcome Michael Casadevall, Arthur Loiret, Jelmer Vernooij and Tiago Bortoletto Vaz into our project!

Release-critical bugs statistics for the upcoming release

According to the unofficial RC-bugs count, the upcoming release Debian GNU/Linux 5.0 Lenny is currently affected by 99 release critical bugs. 26 of them have already been fixed in Debian's unstable branch. Of the remaining 73 release critical bugs, 15 already have a patch (which might need testing) and 12 are marked as pending.

Important Debian Security Advisories

Debian's Security Team recently released advisories for these packages (among others): courier-authlib, proftpd-dfsg, avahi, moodle, phppgadmin, xterm, ruby1.8 and ruby1.9, icedove, iceape and gforge. Please read them carefully and take the proper measures.

New and noteworthy packages

The following packages were added to the unstable Debian archive recently (among others):

Work-needing packages

Currently 482 packages are orphaned and 107 packages are up for adoption. Please take a look at the recent reports to see if there are packages you are interested in or view the complete list of packages which need your help.

Want to continue reading DPN?

Please help us create this newsletter. We still need more volunteer writers to watch the Debian community and report about what is going on. Please see the contributing page to find out how to help. We're looking forward to receiving your mail at debian-publicity@lists.debian.org.

dstat: versatile tool for generating system resource statistics

published on Sun Jan 11 05:00:54 2009 in packages-news

Article submitted by András Horváth. We’re running out of articles! If you like Debian Package of the Day please submit good articles about software you like!

During my work with computers, I like to check the usage of system resources in my network. Sometimes a running process takes up too much CPU load, or the disk I/O goes too high. To get a clean picture of how much resources are being used by a client, I used ifstat, top(1) and iostat(1).

Since I have found out about dstat, I can cleanly check out all the system resources used by my computers. dstat prints all the different type of resources in separate columns on a single line, so it is very easy to see the system load globally.

Quoting from the website:

Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.

Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).

Here is a sample output that I made on my computer:

screenshot of dstat

Though dstat gives global statistics about the currently used system resources, it might replace several tools in one. Mostly you would run it without any parameters, that makes it very easy to remember too :)

Pros (compared to other programs):

  • All kinds of resource statistics in one single line.
  • No parameters needed in most cases.
  • CSV files can be generated easily to create charts in OpenOffice or Gnumeric.

Cons:

  • No per-process statistics

There are official packages available in both Debian and Ubuntu since very long time.

tellico: collection manager for books, videos, music, and a whole lot more

published on Sun Jan 4 05:00:12 2009 in packages-news

Article submitted by Dean Serenevy. We are running out of articles! Please submit good articles about software you like!

You’ve heard of book and movie collection organizers, but Robby Stephenson’s tellico is a general purpose collection manager. This application can be used to store information about arbitrary collections of whatever tickles your fancy. Tellico is available from the tellico package in Debian since Sarge and in Ubuntu since Dapper. Tellico is a KDE application, but works fine in other desktop environments.

The Basics

Like any good book or movie collection application, tellico presents the user with a multi-pane window that groups entries by some customizable criterion (I’ve grouped by director below), lists entries by some fields (customizable), and shows thumbnails.

Typical movie database

Selecting a list entry shows a more detailed view and a larger thumbnail. Clicking the image in this view launches your image editor.

Viewing an entry

Most of the built-in collection types include search sources to make adding new entries easy. Tellico has default search sources for Amazon.com (US, Japan, Germany, United Kingdom, France, and Canada), IMDb (movie database), z39.50 servers (bibliographic database), SRU servers (bibliographic database), PubMed (Medicine bibliography), CrossRef.org (bibliographic database developed by a consortium of publishers), and some others. You can also write your own script that performs the search and returns entries in a supported format.

Searching for every Debian user's favorite movie

The search box will filter results based on regular expression queries. Complex filters can be named and will be saved with the collection file.

So many movies, so little time

Beyond Books and Movies

Tellico’s built-in list of collection templates is already quite impressive. It provides default templates for books, bibliographies, videos, music, video games, coins, stamps, trading cards, comic books, and wines. However, users are free to modify, add, or remove fields in these collections or even create custom collections with arbitrary fields.

For example, I keep a collection of hyperplane arrangement examples in a custom tellico file. Tellico happily keeps a fully group-able and search-able record of my coefficient fields, polynomials, and other fields.

Arrangements of hyperplanes

Editing a custom entry looks just like editing a standard record type. Fields are grouped by customizable categories.

Editing a record

Modifying the collection fields is wonderfully simple. Your fields may be any of several field types including: text, paragraph, choice, checkbox, table, URL, date, and image. Field upgrading is supported between compatible field types.

Fields may be auto-formatted as names or titles if you wish. You can also control whether the field should support auto-completion (using existing entries in your collection), multiple values, or whether the field should appear in the grouping combo box.

Editing the fields in a collection

The paragraph field type supports basic HTML markup (used here in my bibliography collection). The red letters are KDE’s spell-check attempting to be useful.

HTML markup in a paragraph field

I use the table field type in my recipe collection.

Using a table for an ingredient listing

Beyond the Application

Tellico can import and export data to and from many sources (Bibtex, CSV, PDF metadata, Alexandria, …). It can export your collections (even custom collections) to HTML and generate HTML reports in several styles. Tellico even has limited support for sending citations to OpenOffice.org Writer (though I have never used this feature).

Moreover, since Tellico stores its data in a fully documented XML file you can write XSLT or use any XML parser to transform the data file however you like.

Tellico supports loan tracking for any collection type. It also translated into more than ten languages.

The not so good

Tellico is somewhat laggy when loading hundreds or thousands of images from disk and occasionally when switching from thumbnail view to entry view. However, switching between entries is always fine and collections with fewer images are quick and responsive.

Alternatives

There are many special-purpose collection managers (most of which are listed on the tellico homepage), but tellico is one of the earlier general purpose managers. Some applications (such as GCstar) are becoming more general-purpose as they mature. Others (such as Stuffkeeper) are simply younger applications and are not yet stable. Tellico is a well-designed application and therefore can give even the special-purpose collection managers a run for their money.