Sunday, January 31, 2010

Shell Scripts and Here Documents

A here document (or heredoc) is a way of getting text input into a script without having to feed it in from a separate file.

If you've got a small quantity of data you don't expect to change often, it's a quick and tidy way of keeping script and data together.

For more complicated data or scripts, separating the two is still usually a better idea.

The basic version of this in bash looks like this:
cat << EOF
several lines of 
my data
listed here

which will output

several lines of 
my data
listed here

You can use any label (although EOF is standard) — just make sure it matches at both start and end.

There are a couple of useful variations on this format. Just using a plain label, as above, means the shell will interpret any special characters. This would substitute your current directory for $PWD:

cat << EOF
current directory is:

However, with some data sets this can cause problems, especially if you don't remember that it happens! To disable interpretation, use single or double quotes:
cat << 'EOF'
current directory variable is:

This will output $PWD verbatim:
current directory variable is:

By and large, it's best to disable shell interpretation by default and to take the quotes out if your data definitely requires it.

Another useful trick is to use a minus sign after the <<. This will ignore tab characters (Note: not all whitespace!) at the start of a line, so you can indent your data to increase the readability of your script:
cat <<- 'EOF'
 lots of my 
 data will be printed 
 out without tabs!

You can also use heredocs in other programming languages, including Perl, PHP and Python. Be aware, however, that quotes may affect escaping of special characters in different ways in different languages.

Create an Ubuntu Repository

There are several reasons you may want to create a local repository.  The first is that you want to save on bandwidth if you have multiple Ubuntu machines to update.

For example if you had 25 Ubuntu machines that all needed updating at least once a week, you would significantly save bandwidth because you could do all but the repository locally.

Most organizations have decent bandwidth for their network gateways but this bandwidth is a precious commodity that needs to be used wisely.

Many organizations still have routers with 10MB or 100MB limits at the gateway but 1 GB network connections internally so bandwidth could be better used internally.

The second reason for creating your own repository is that you can control what applications are loaded on your internal Ubuntu machines.

You can remove any applications your organization does not want to use on the local network from the repository that updates the machines.

Even better, you can create a test box and test applications and versions before you allow them to roll out into your network assuring security and stability.

# apt-get install apt-mirror
############# config ##################
# set base_path    /var/spool/apt-mirror
# if you change the base path you must create the directories below with write privileges
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/
# set defaultarch 
# set postmirror_script $var_path/
set run_postmirror 0
set nthreads     20
set _tilde 0
############# end config ##############

deb karmic main restricted universe multiverse
deb karmic-security main restricted universe multiverse
deb karmic-updates main restricted universe multiverse
#deb karmic-proposed main restricted universe multiverse
#deb karmic-backports main restricted universe multiverse
deb-src karmic main restricted universe multiverse
deb-src karmic-security main restricted universe multiverse
deb-src karmic-updates main restricted universe multiverse
#deb-src karmic-proposed main restricted universe multiverse
#deb-src karmic-backports main restricted universe multiverse

Need help configuring an Ubuntu Server….take a look at the Online Ubuntu Server Course.

Files in apt-mirror
The main configuration file is /etc/apt/mirror.list which contains a list of files that you should be aware of in case you have any issues.


This is the  cron configuration template which you can modify.  The example which is not active as it is commented out, starts a cron job at 4 AM each day and sends the activity into the log as listed.

Some administrators may want to only update once a week and others will prefer twice a day to be up to date.
# Regular cron jobs for the apt-mirror package
#0 4    * * *   apt-mirror      /usr/bin/apt-mirror > /var/spool/apt-mirror/var/cron.log

This is the location of the mirrors that you create. Be aware, that the /var directory can fill up and these mirrors will take 15-20 GB each so verify you have the space to set them up.


The location for the indexes which will list the applications that are available.


The log files, MD5 sums, URLS and are placed here as you can see below.

ALL             archive-log.15  archive-log.5    archive-urls.11  archive-urls.19  archive-urls.9  index-log.6   MD5
archive-log.0   archive-log.16  archive-log.6    archive-urls.12  archive-urls.2        index-urls.0  NEW
archive-log.1   archive-log.17  archive-log.7    archive-urls.13  archive-urls.3   index-log.0     index-urls.1
archive-log.10  archive-log.18  archive-log.8    archive-urls.14  archive-urls.4   index-log.1     index-urls.2
archive-log.11  archive-log.19  archive-log.9    archive-urls.15  archive-urls.5   index-log.2     index-urls.3
archive-log.12  archive-log.2   archive-urls.0   archive-urls.16  archive-urls.6   index-log.3     index-urls.4
archive-log.13  archive-log.3   archive-urls.1   archive-urls.17  archive-urls.7   index-log.4     index-urls.5
archive-log.14  archive-log.4   archive-urls.10  archive-urls.18  archive-urls.8   index-log.5     index-urls.6

Download Repositories
Once you have your set up apt-mirror you can start your download of the repository with this command.

# apt-mirror /etc/apt/mirror.list

Downloading 7 index files using 7 threads…
Begin time: Tue Jan 19 20:21:07 2010
[7]… [6]… [5]… [4]… [3]… [2]… [1]… [0]…
End time: Tue Jan 19 20:23:17 2010
Proceed indexes: [P]
6.6 GiB will be downloaded into archive.
Downloading 6612 archive files using 20 threads…
Begin time: Tue Jan 19 20:23:19 2010

The apt-mirror provides multithreaded downloads and allows for multiple architectures for your repositories.

As the download progresses you will see applications saved and space being used.

# ls mirror/
a  b  d  f  g  h  k  l  libc  libm  libx  m  o  p  r  t  u

Link Repository to Web Interface
Because apt-mirror saves everything in /var/spool/apt-mirror you need to create a link to make it available to users as the /var/spool is not available to the public.

The first link created provides a link to the Packages.gz which lists the available packages in the repository.

This is the line you will place in the /etc/apt/sources.list for the client to access this list.

# sudo ln -s /var/spool/apt-mirror/skel/ /var/www/ubuntu

The additional symlinks provide access to the public to individual applications to download. Of course the symlinks are representative of the repositories you will use.

# sudo ln -s /var/spool/apt-mirror/mirror/ /var/www/archive-ubuntu
# sudo ln -s /var/spool/apt-mirror/mirror/ /var/www/archive-canonical
# sudo ln -s /var/spool/apt-mirror/mirror/ /var/www/security-ubuntu

Lock File Problem
There is a bug in apt-mirror so that it runs fine the first time but then it does not remove the lock file so it will not run the second time, telling you it is already running.

If you see that remove the lock file and it will work fine.

# sudo rm  /var/spool/apt-mirror/var/apt-mirror.lock

Client Set Up for apt
You will need to modify the /etc/apt/sources.list file in order to have the client connect only to your local repository.

deb karmic main

Note that the example above has only the main repository installed so as to provide an evaluation of the set up.

Remember, this takes a lot of space for these repositories.

Now you can update and then use apt-get to install programs.

# sudo apt-get update

Get:1 karmic Release.gpg [189B]
Ign karmic/main Translation-en_US
Get:2 karmic Release [65.9kB]
Get:3 karmic/main Packages [1,353kB]
Fetched 1,419kB in 1s (1,322kB/s)
Reading package lists… Done

Here is an example of a website address set up for a local repository.

If you want to download manual individual packages you can use this URL.

Further control of Linux files with ACL

If you read my article “Get to know Linux: File permissions” you know that it’s possible, out of the box, to control who can access a file and what they can do with it. This helps to make Linux a fairly secure system.

But did you know you can take even further control of that system with the help of Access Control Lists?

Access Control Lists allow you to provide different levels of access to files and folders. Say, for instance, user jlwallen creates a file but doesn’t want to allow anyone to do anything with this file but he and another user, wookie (even though there are other users that belong to the group jlwallen).

ACL can handle this task.
In this article you will learn how to install and use ACL to further enchance your file permissions on a Linux system.

Let’s install ACL on a Ubuntu system. Since this is a command line tool, we’ll do the installation from the command line. And since ACL will be found in the standard repository, you won’t have to monkey with your /etc/apt/sources.list file.

From the command line enter the following:

# sudo apt-get install acl

Type your user password, hit the Enter key and the install, and the installation will begin and end fairly quickly.

You are now ready to start with ACL.

Using ACL
Before you use the commands for ACL you actually have to mount your partition such that ACL is available.

By default this is not the case. In order to set this you have to edit your /etc/fstab file. Open that file up and look for the line that mounts your data partition.

In my case, this line is:

UUID=c7812a34-3ec1-4451-aace-02d122b6c454 /   ext4  errors=remount-ro 0 1

You need to edit this line to look something like:

UUID=c7812a34-3ec1-4451-aace-02d122b6c454 /   ext4 errors=remount-ro,acl 0 1

After you make this edit, save the file and then either issue the command:

# sudo mount -o remount,acl /

or reboot your machine.

There are two commands you will use for ACL:
  • setfacl – Set file access control list.
  • getfacl – Get file access control list.
You can probably guess that the first command sets the the ACL and the second lists the ACL for the file.

Using ACL
So let’s say you have the file test and you want only two users on your system to be able to read that file, jlwallen and wookie.

You want to exclude all users in the group jlwallen as well.  What you want to do is use the setfacl command like so (as the user jlwallen):

# setfacl -m u:wookie:rw- test

Now when you run the command:

# getfacl test

you will see something like:

# file: test
# owner: jlwallen
# group: jlwallen

As you can see both users jlwallen and wookie can read and write to the file test, whereas all others can only read the file.

You can verify that a file has had ACL modifications done to it by using the ls command like so:

# ls -l test

which should produce results like:

-rw-rw-r–+ jlwallen jlwallen

What gives this away is the “+” character.

Final thoughts
Although this is just a cursory glance as using ACL, it will get you started with gaining even further control of the security of your Linux files. We’ll revisit ACL later and take the security of Linux files even further.

Linux Audio Plugin Update

Audio processing and synthesis plugins are always a lively topic for musicians. Many contemporary music-makers rely completely upon their plugin collection for all their sound sources and processing routines, and it is not at all uncommon to discover that some of these composers have never learned to play a traditional instrument.

However you feel about audio plugins they are a fact of life in modern music production.

In the Win/Mac worlds the VST standard rules, thanks to the Steinberg Company's liberal policy regarding the use of their VST SDK (systems development kit), but the VST flag is not the only one waving over those worlds.

Apple offers the AU (Audio Unit) format for Mac audio plugins, Digidesign has their proprietary RTAS (Real-Time Audio Suite) for their products, and Microsoft's DirectX includes a plugin programming interface.

In an apparent bid to compete with this array of formats, Linux can claim its own variety of standards and acronyms, including LADSPA, LV2, MESS, DSSI, and native VST. This article presents a summary of recent activity in Linux audio plugin development.

LADSPA is the senior player in the Linux audio plugin field. Its API (applications programming interface) is stable, but development is no longer as lively as when it first appeared.

Nevertheless, some significant LADSPA plugins have shown up in the last few months.

Figure 1. Ardour as the Harrison Mixbus

The Harrison Mixbus (Figure 1) is the Big News in LADSPA plugin development. Mixbus is a special commercially-available version of Ardour customized by Ardour's chief developer Paul Davis and the engineers at Harrison.

According to the announcement on the Ardour site the Mixbus DSP core is a single closed-source LADSPA plugin, which is certainly a vindication of the continuing viability of the LADSPA API (applications programming interface).

Reports from users are positive, the screenshot is hot, but alas, Mixbus is currently available only for OSX.

Plans are in the works for a Linux version, so you can bet that I'll be watching the Ardour site for the announcement.

I've written about Guitarix here already, but I must admit that it's hard keeping up with its development. The latest release contains an improved GUI, free re-ordering of the processing modules (hooray!), a new multiband EQ, and some helpful additions to the jconv widget.

More to the point, all of the Guitarix processors are available as LADSPA plugins. If you build Guitarix the installation process will automatically place the module plugins in the default LADSPA directory (/usr/lib/ladspa or /usr/local/lib/ladspa).

I compile most of the software I use, and I assume that a repo installation of Guitarix will also install the plugins in the expected locations.

The LEET collection is not new, but I discovered it only recently. Programmer socaldan3000 has selected some existing problematic LADSPA plugins and has worked on improvements to their internal processing operations.

The collection includes only three plugins - two EQs from the TAP set and one chorus from the MCP bundle - but they have been reprogrammed to accept 2-channel I/O for easier accommodations in Ardour.

Linux audio software developers continue to provide impressive examples of the LV2 API. Krzysztof Foltman has updated the GUIs for his CALF plugin collection.

Figure 2 shows off the new look of the plugins and the project's neat calfjackhost launcher. In previous articles I've mentioned the excellent LV2 collections from linuxDSP and the Invada group.

The list expands once more with the addition of the Russolo Suite and EQ10Q. The Russolo Suite is named after Luigi Russolo, a musician in the Futurist movement who is widely regarded as one of the first composers focused on using noise as a musical resource.

As might be expected from its namesake, the Suite is dedicated to software for creating noisy sounds. Pere Ràfols Soler's EQ10Q (Figure 3) is dedicated to shaping the frequency bands of that noise (or any other sound, noisy or not).

EQ10Q is a 10-band parametric equalizer with a variety of filters and per-band resonance (Q) control, packaged in a neat GtkMM interface. It's also another good example of the capabilities of the LV2 API.

Figure 2. CALF plugins get shiny

Figure 3. The EQ10Q parametric equalizer

Linux offers two methods for VST plugin support. One method provides bridge software such as dssi-vst and fst running VST plugins in an emulation environment.

The other method provides development tools for compiling VST plugins as native Linux software. Both methods depend upon the capabilities of the Wine project.

Until recently these methods also employed code from the Steinberg VST development package. Doing so presented a problem for Linux developers - the code can't be distributed freely and thus conflicts with software protected by the GPL.

This conflict resulted in source-only availability, i.e. you could build the software yourself but you could not distribute the binary legally.

Today, the situation has changed, thanks to the VeSTige project, and bridge software and native plugs alike can be built and distributed without the Steinberg software.

Figure 4. Festige at work

An announcement from user funkmuscle on a Linux Musicians forum led me to a neat GUI front-end for fst. Festige (Figure 4) is a handy tool for launching your VST/VSTi plugins under Linux.

The program is simple, but it allows multiple instances of any plugin, uses JackMIDI for MIDI connectivity, and supports MIDI-learn for MIDI controllers.

Festige can use a little more tuning - I'm getting too many xruns in an optimized system - but at version 0.0.3 it's already usable.

It's also just great fun. For one of my tests I launched the dfx Transverb processor and two instances of the Crystal synthesizer, then I routed the audio of one synth through the Transverb while the other went directly to the system audio output in QJackCtl.

I fired up Paul Davis's great Softwerk MIDI sequencer, configured it for random pitch, rhythm, and program change, and connected it to the synths.

Crystal receives on all MIDI channels, so I used holborn's useful midirgui to route the MIDI data coming from Softwerk.

Thus, one instance of Crystal received data over channel 1, the other instance received data on channel 2.

With all systems go I started Softwerk and let the good times roll (Figure 5). Softwerk happily generated patterns for hours while I occasionally reset the randomization controls to vary the orchestration and pitch sequence. Like I said, it was great fun.

Figure 5. Softwerk plays with Festige

In another recent article I listed some new native VST plugins, including Christian Borß's HybridReverb2, KResearch's KR-Reverb FS, and Loomer's String synthesizer.

Not long after that article was published I read a comment stating that native VSTs weren't supported by many programs because there simply aren't many of them.

In response to that comment I composed a list of the plugins populating my $HOME/vst-linux folder. I was surprised to find that I have more than a hundred native Linux VST plugins in that folder.

The collection encompasses audio processing, sound synthesis, utility plugins, sampling, and MIDI processing, with notable work from Lucio Asnaghi's Jucetice project, the Loomer group, mucoder, discoDSP, and the pizmidi project.

Their quality varies, but some of them have become regular players in my virtual bands (Figure 6). Their quantity at least refutes the assertion that there aren't many of them.

Figure 6. Some native Linux VST/VSTi plugins


Werner Schweer's MusE Experimental Soft-Synth (MESS) interface was designed to accommodate virtual instrument plugins.

The API saw some early adoption for the MusE sequencer, but after an initial spate of production it appears that the format is currently moribund.

Chris Cannam's Disposable Soft-Synth Interface (DSSI) was also designed for virtual instruments. While there have been few notable recent releases for its plugin interface the system is supported by various Linux hosts, including Rosegarden, Csound5, the Aldrin tracker, and the QTractor DAW.

The API is stable at version 1.0, there are some good plugins available for it now, and its dssi-vst software (see above) remains popular with users running Windows VST plugins on Linux music production boxes.

LinuxDSP Update
Just after I submitted this article I discovered that the linuxDSP LV2 collection has expanded to include a new compressor, a vintage delay, a stereo reverb, and a graphic equalizer (Figure 7).

These plugins are donation-ware, i.e. you must donate some amount in order to acquire the key to unlock the encrypted zip file.

The author suggests a modest figure, but trust me, these plugins are worth ten times what he suggests. As this article demonstrates, LV2 is certainly up to the task of allowing classy new interfaces, but looks are meaningless in the world of sound.

The linuxDSP collection is exceptionally well-engineered, the plugins sound great, and the author is constantly improving their capabilities.

So go ahead, make a donation, and check out some of the best audio plugins available for Linux.

Figure 7. LinuxDSP LV2 plugins in Ardour 2.8.6

That's all for this brief update. I hope you'll check out some of the plugins mentioned here, have some fun, and be sure to make useful reports to the developers.

Saturday, January 30, 2010

Personal Finances on Linux with KMyMoney

Linux has several excellent personal finance applications. Today we're going to look at KMyMoney. Though specifically developed with the KDE desktop in mind, KMyMoney works with any Linux desktop.

How useful KMyMoney will be to you and how easy it will be to use depends on two factors: One, whether or not your bank uses a supported online banking protocol, like OFX or HBCI, which applies when using any type of money management program. Two, if you can get a set of plugins working, which includes check printing support.

KMyMoney Summary view
KMyMoney Summary view

If your bank and KMyMoney are compatible, your account history and balances are automatically updated in KMyMoney.

If they are incompatible for any reason, you'll be stuck with having to manually update your accounts. That isn't great if you deal with a great amount of money or transactions.

If you can successfully install a set of plugins, you can bring check printing support into KMyMoney, among a few other features.

This can also be a deciding factor for if KMyMoney is a feasible option for you. With check printing support and a stack of compatible blank check paper, you can have KMyMoney print and log all your checks.

If you don't go that route, you must manually input the check details to have a balanced account.

Getting started
The first time you open KMyMoney, the New File Setup Wizard should appear. Follow this wizard to setup the basic settings.

You'll input your personal data (name, address, etc), set the currency type, and enter checking account details. You'll also choose the account type(s) you want to track with KMyMoney, so it can create the proper database and structure.

Near the end of the wizard, you can optionally hit the Configure Preferences Now button. In the Preferences, you might consider enabling Autosave so you're better protected if the application or your computer crashes in the future.

You should also consider enabling GPG encryption, so your data and sensitive information is secure if your computer is stolen or hacked into.

Finally, at the end, you'll be prompted to define a filename and location. Make sure you store the KMyMoney data file in a safe location.

You might even edit the file or folder permissions. You could remove all permissions for Others and Groups, just giving yourself access to the data file.

Adding more plugins
As discussed, you can install plugins for additional features, such as check printing. If desired, download and extract the compressed folder to an accessible location, such as the desktop, and install using the following steps:

Create a new directory somewhere and then move to that directory:

mkdir kmymoney-plugins-build
cd kmymoney-plugins-build
Then install the plugins with the following lines of commands:

cmake /home/YOUR_USERNAME/Desktop/kmymoney-plugins-x.x.x/cmake
make install

Configure savings account
Most likely you have a savings account--though, like many others, it might be running low. So first you should probably create an account in KMyMoney for it. Here's how:

  1. Click Account > New Account.
  2. On the wizard, select the Institution or create a new one if it isn't listed by clicking the button.
  3. Then type in the Account Number and, if one exists, a IBAN number.
  4. Then click Next. Type in an Account Name, such as Savings, configure the other settings, and click Next.
  5. Select the main account you want to place this new one in, if multiple parent accounts exist, and click Next. Verify the details and click Next.
Configure additional accounts
To get the most out of the KMyMoney application, you should input details for all your accounts.

This includes credit cards and loans, investments, and any other assets or liabilities. Follow similar steps as you did when configuring your savings account:

Click Account > New Account. For each different banking organization you must create an Institution entry.

Enabling online banking
Keep in mind, most banks require a separate username and password to access your accounts through an application. So before proceeding, check with your bank to see if they support OFX connections.

They likely won't know what KMyMoney is particularly, so you might mention a more popular application, like Microsoft Money or Quicken.

Click the Accounts icon, right-click an Account, and select Map to Online Account. If more than one plugin exists, it will prompt you to choose one.

To use the integrated feature, select KMyMoney OFX. Then use the wizard to search for the bank and input your login credentials.

If you can't find your bank, you can hit the Manual tab and input the details you get from your bank.

Once you have it configured, you can go to the Accounts or Ledger page, select the Account and select Update Account to download updates from your bank.

Adding and modifying payees and payers
When you input transactions, KMyMoney automatically creates payee entries when you pay money out or payer entries when you receive money.

That way you can see and analyze your expenses and income per the companies/persons you're giving your money to or receiving your money from.

If you haven't had a transaction come through for a particular payee or payer yet, you might want to create payee entries manually. Additionally, you might want to do this if setting up payee/payer matching.

To see the list of payees/payers, click the Payee icon on the left pane of the application. To add an entry, right-click on any existing entry and select New Payee.

Remember, you must already have at least one entry before you can manually create one. To create the first entry, you can enter a transaction.

If you have an account linked with your bank or you manually import transactions, and the payee/payer name or description isn't exactly the same for each of its own transactions, consider using payee/payer matching.

This could group all the transactions from the same company but at different locations, such as fast food locations. For instance, both "Fast Food Corp Store #123" and "Fast Food Corp Store #987" could be automatically placed into the "Fast Food Corp" payee.

Inputting transactions
If transactions aren't being automatically logged into KMyMoney, you'll want to manually input them.

Simply click the Ledger icon on the left pane and click the New button on the bottom of the window.

Then choose the appropriate tab: Deposit, Transfer, or Withdraw. Then fill in all the details and click the Enter button.

Looking into the other features
We reviewed the basic tasks in getting KMyMoney up and running. Don't forget about the remaining features.

If you expect a transaction or have reoccurring transactions, consider setting up scheduled transactions.

This can be useful to input all your fixed expenses and income. If you have stocks or retirement money, you might want to look into the investment features.

Don't over-look the reports and budgeting features; these can be a big help.

Discovering ncurses, the GUI for the Linux Console

The CLI versus GUI debate is well-known, but sometimes, drawing a hard line between text-only interfaces and graphical interfaces is too restrictive.

Most of the time, if you're using a terminal (whether over SSH or on the console), the CLI and text-only interface works great.

Sometimes, though, it can be a real advantage to be able to use at least some graphical logic, without having to fire up an X server.

Not all servers run X; you don't always want to forward X to your remote terminal; or you may be at a stage of system or hardware install that prevents you from running a regular graphical interface.


This is where ncurses comes in. The ncurses library allows you to write programs that work in a GUI-like way, but which will run within a normal terminal emulator. You've almost certainly already encountered programs that use ncurses, including:

  • screen, which allows you to run multiple screens in a single terminal window (this uses the terminfo information from ncurses but does its own screen-handling).
  • make menuconfig, a tool which gives you a menu-driven user interface when configuring the Linux kernel before building it. This is a massive improvement over the older question-and-answer style configuration tool, as you'll know if you've ever missed a question with the older tool and had to start all over again.
  • The email program mutt.
  • The terminal-based web browsers lynx and w3m.
  • aptitude, which provides a menu-driven interface to apt-get on Debian and Ubuntu systems.
  • The visual file manager GNU Midnight Commander.
  • The SuSE system management tool yast, when run in ncurses mode.
  • A variety of MP3-player and IRC interfaces available via Sourceforge and other sites.

Midnight Commander, the powerful ncurses-based file manager
Midnight Commander, the powerful ncurses-based file manager

ncurses is a C library which provides a variety of functions for handling screen drawing, cursor management, mouse management, and so on.

It isn't released under the GPL, but under a specific free software copyright notice which is discussed in the FAQ.

As a programmer, using ncurses means that you don't have to deal with the massive variety of slightly different terminal codes that exist. Instead, you can just deal with the more abstract concept of a display with multiple windows, and ncurses will handle the translation into control codes for whichever terminal emulator it detects that the program is running under. It can also handle mouse events, so if you have a mouse handler running (either on console or via X), you can use links and clickable menus within your terminal.

w3m, the ncurses Web browser. It take the same colors as your X 

w3m, the ncurses Web browser.

It take the same colors as your X terminal

There are ways of interacting with ncurses in numerous languages other than C itself (including Python, Perl, and C++), so anything you learn in one language will transfer with reasonable ease to another.

If you're interested in writing your own ncurses programs, there's an introduction here (technically obsolete, but still useful as a starting point), and a HOWTO.

It's entirely possible to run an ncurses-based program within a terminal running in an X environment.

The main thing to bear in mind when doing this is that the ncurses mouse-handling overrides the X mouse-handling while the mouse pointer is within the window that the ncurses application is running in.

So you can use the mouse to click on menus and links, but you can't (unless the program implements this itself) copy-and-paste using the normal X methods. 

In terms of a straight comparison between ncurses and X as ways of providing a graphical interface, X certainly provides much more scope for things like moving between terminals, and cutting-and-pasting between terminals.

Fundamentally, X provides a way of managing multiple windows within a desktop environment, and ncurses doesn't. screen, which uses part of the ncurses code, does provide multiple terminals within one session, which you can paste between, but you can't natively cut-and-paste between two separate instances of screen, or even use X to do this between ncurses programs in separate terminals.

ncurses applications are also less aesthetically pleasing, by and large, than X applications, due to the limitations imposed by operating within a terminal; they tend to look quite blocky and very text-based.

The extent to which ncurses applications are GUI-like is more about their logic and internal navigation than it is about their aesthetics. However, this 'graphical logic' can be incredibly useful. 

Being able to use the mouse for browsing within w3m is useful; and being able to use a menu-driven system for configuring a new Linux kernel (with make menuconfig) is incredibly useful and makes the task significantly easier.

(There is also an X kernel configuration tool available, but you don't always have X running when you're rebuilding a kernel!).

Even in a modern system, it's not necessarily the case that a full-on GUI will actually give you significant improvements.

Depending on what you're doing, the graphical login of ncurses may be enough of an improvement over a text-only CLI; and ncurses applications are way more lightweight than X applications.

Most importantly, they don't require you to be running an X server, which isn't always either desirable or practical.

If your focus is on aesthetic beauty, or you need a desktop-style environment, then by all means stick with X and regular GUIs.

If you want ease-of-use and low overheads, and you're either not bothered about graphical bells and whistles, or running in a terminal or console environment, ncurses-based programs can make that terminal environment far more usable, fast, and productive.

Linux performance basics

I want to write about Cassandra performance tuning, but first I need to cover some basics: how to use vmstat, iostat, and top to understand what part of your system is the bottleneck -- not just for Cassandra but for any system.

You will typically run vmstat with "vmstat sampling-period", e.g., "vmstat 5." The output looks like this:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
20  0 195540  32772   6952 576752    0    0    11    12   38   43  1  0 99  0
22  2 195536  35988   6680 575132    6    0  2952    14  959 16375 72 21  4  3

The first line is your total system average since boot; typically this will not be very useful, since you are interested in what is causing problems NOW.

Then you will get one line per sample period; most of the output is self explanatory. The reason to start with vmstat is the "swap" section: si and so are swap in (memory read from disk) and swap out (memory written to disk).

Remember that a little swapping is normal, particularly during application startup: by default, Linux will swap infrequently used pages of application memory to disk to free up more room for disk caching, even if there is enough ram to accommodate all applications.

To get more details of io, use iostat -x. Again, you want to give it a sampling interval, and ignore the first set of output. iostat also gives you some cpu information but top does that better; let's focus on the Device section:
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               9.80     0.20   36.60    0.40  5326.40     4.80   144.09     0.06    1.62   1.41   5.20

There are 3 easy ways to tell if a disk is a probable bottleneck here, and none of them show up without the -x flag, so get in the habit of using that. "avgqu-sz" is the size of the io request queue; if it is large, there are lots of requests waiting in line. "await" is how long (in ms) the average request took to be satisfied (including time enqueued); recall that on non-SSDs, a single seek is between 5 and 10ms.

Finally, "%util" is Linux's guess at how fully saturated the device is.

To learn more about per-process CPU and memory usage, use "top." I won't paste top output here because everyone is so familiar with it, but I will mention a few useful things to know:
  • "P" and "M" toggle between sorting by cpu usage and sorting by memory usage
  • "1" toggles breaking down the CPU summary by CPU core
  • SHR (shared memory) is included in RES (resident memory)
  • Amount of memory belonging to a process that has been swapped out is VIRT - RES
  • a state (S column) of D means the process (or thread, see below) is waiting for disk or network i/o
  • "steal" is how much CPU the hypervisor is giving to another VM in a virtual environment; as virtual provisioning becomes more common, avoiding noisy neighbors is increasingly important
"top -H" will split out individual threads into their own lines; both per-process and per-thread views are useful.

The per-thread view is particularly useful when dealing with Java applications since you can easily correlate them with thread names from the JVM to see which threads are consuming your CPU.

Briefly, you take the PID (thread ID) from top, convert it to hex -- e.g., "python -c 'print hex(12345)'" -- and match it with the corresponding thread ID from jstack.

Now you can troubleshoot with a process like: "Am I swapping? If so, what processes are using all the memory?

If my application makes a lot of disk read requests, are my reads being cached or are they actually hitting the disk?

If I am hitting the disk, is it saturated?

How much 'hot data' can I have before I run out of cache room?

Are any/all of my cpu cores maxed?

Which threads are actually using the CPU?

Which threads spend most of their time waiting for i/o?"

Then if you go to ask for help tuning something, you can show that you've done your homework.

65 Open Source Downloads That Could Change Your Life

Can an open-source application really change your life? To find out, we went looking for apps that users described as "indispensable."

What we found were tools that truly can help you significantly change the way you live or the way you use your computer every day. And as we put the list together, we noticed that a lot of these tools fall into categories that corresponded with some of the most popular New Year's resolutions.

For example, a recent survey by Franklin Covey found that the top New Year's resolution for 2010 was "improve my financial situation." We found a whole host of open source apps that can help you accomplish that goal by creating a budget, managing your portfolio, or even launching your own e-commerce site.

In fact, we found open source apps for most of the most popular resolutions: apps to help you eat better and exercise more, apps to help you stop smoking or break other bad habits, lots of apps to help you get organized and make better use of your time.

We found open source apps that could help you get started in a new hobby, read more, improve your mind, and become more spiritual. And for the (admittedly small) group of people whose New Year's resolution is to try open source for the first time, we assembled a small group of apps for open-source neophytes.

Unfortunately, no matter how hard we looked, we simply couldn't find any open source tools for one of the other most popular New Year's resolutions: improving your love life.

So until some creative open source developers come up with an app that can help you find your true love, you'll have to make due with the rest of the tools of our list of 65 open source applications that can change your life.

Download open source apps to:

Page 1: Improve Your Financial Situation, Lose Weight (Diet/Exercise), Stop a Bad Habit
Page 2: Get Organized, Make Better Use of Your Time

Page 3: Start a New Hobby, Learn a New Language, Read More
Page 4: Improve Your Mind, Become More Spiritual, Give Open Source a Try

Improve Your Financial Situation
1. iFreeBudget
One of the simplest budgeting apps available, iFreeBudget offers home and small business users a free alternative to Quicken or MS Money.

It stores data in a password-protected, encrypted file, and it generates reports, updates stocks, and does most of the other things you'd expect a personal finance program to do.

Operating System: OS Independent

2. jGnash
This java-based, double-entry accounting system tracks budgets, creates pdf reports, prints checks, and imports data from Quicken files.

It's fairly robust, but may take a while to figure out if you've never used personal finance software before.

Operating System: Windows, Linux, OS X.

3. Buddi
The “personal budget software for the rest of us,” Buddi is an excellent option for people who have never used personal financial management software before.

It includes helpful tutorials and a QuickStart guide when using it for the first time. Like many of the other options in this category, it also imports data from Quicken and MS Money.

Operating System: Windows, Linux, OS X

4. LightWallet
If you have trouble remembering to save receipts or enter transactions in your budgeting software, LightWallet is for you.

This Java-based personal finance package installs on your mobile phone, so you can enter transactions while you're on the go instead of having to remember what you bought when you get back home.

Operating System: OS Independent.

5. StockManiac
Manage your own investments? StockManiac incorporates a feed reader/comment system with a stock tracker so that you can track your thoughts about your stocks while you track your stocks.

It's not suitable for day trading, but works well for average personal investors.

Operating System: OS Independent.

6. JStock
Another portfolio management tool, JStock provides you with near real-time data on 24 world stock markets, in addition to providing you with plenty of reporting and graphing tools.

It also allows you to store your data in the cloud so that you can access it online from anywhere.

Operating System: Windows, Linux, OS X.

7. Market Analysis System
Ready to get really serious about investing? This app tracks a huge number of metrics and analyzes market data, flagging you when certain criteria are met.

You can run it on a single PC or in a client/server setup.

Operating System: Windows, Linux.

8. PrestaShop
Considering changing your financial system by setting up your own e-commerce site? PrestaShop offers a handy demo tool on its Web site, so that you can see exactly how it works.

It's lightweight, fast, and easy to use, even if you've never managed a Web site before.

Operating System: Windows, Linux, OS X.

9. Magento
Downloaded more than 1.5 million times, Magento calls itself the "world's fastest-growing eCommerce platform."

The free community edition offers plenty of features to get you started with a basic site, and the paid enterprise edition adds support and features like strong encryption, gift certificates, enhanced content management and more.

Operating System: Windows, Linux.

10. Zen Cart
Unlike most other e-commerce platforms, ZenCart was designed by actual businesspeople. As a result, it meets the needs of business owners well and includes features like XHTML template system, multiple ad banner controller, newsletter manager, discount coupons, gift certificates, and quantity discounts.

Operating System: Windows, Linux.

Lose Weight (Diet/Exercise)
11. iDiet
Trying to decide which diet to try? iDiet makes it easy to compare a lot of different diets, including Atkins, Summer Fresh, The Zone, Body for Life, Jenny Craig, based on nutritional guidelines. Once you pick a diet, the Java-based app helps you set goals and track your progress. Operating System: OS Independent

12. SportsTracker
No matter what sport you participate in (running, cycling, lifting weights, tennis, etc.), SportsTracker can help you track your training plan, type of activity, body weight, time spent exercising, etc.

It also integrates with several brands of heart rate monitors, so you can track your heart rate data as well.

Operating System: Windows, Linux, OS X

Stop a Bad Habit
13. QuitCount
Need some motivation to help you stop smoking? Quit Count adds up how much money you've saved and how much time you've added to your life expectancy.

Operating System: Linux.

14. No Smoke Counter
Like QuitCount, the No Smoke Counter tracks your money saved by not smoking. However, this version is designed for your smartphone, so you can take it along with you. Operating System: OS Independent.

15. Areca Backup
Even if you don't have a bad health habit, like smoking, you probably have a least one bad computing habit. If you're bad habit is neglecting to back up your files, Areca Backup is for you.

It makes it fast and easy to back up and encrypt your files, so you won't have to worry about a hard drive crash.

Operating System: Windows, Linux.

1. BleachBit
Another bad computing habit: forgetting to maintain your system. BleachBit frees up disk space, deletes cache and cookies, cleans up your browser, and generally frees up your system for better performance.

It can also guard your privacy by "shredding" files so that they can't be recovered after deletion.

Operating System: Windows, Linux.

2. AxCrypt
Yet another bad habit—storing personal or sensitive data on your system without encrypting it—is especially bad if you use a laptop.

AxCrypt makes it fast and easy to encrypt files with Windows. Just right-click files or folders in Windows Explorer, and they'll be encrypted automatically.

Operating System: Windows.

3. TrueCrypt
Another encryption option, TrueCrypt is particularly helpful if you want to encrypt an entire drive, not just a couple of files or folders.

You can also use it to encrypt data on a thumb drive so that you can take sensitive information with you without worrying about losing your drive.

Operating System: Windows, Linux, OS X.

4. KeePass
The last bad computing habit on our list is bad password selection, as in picking passwords that are easy to guess or using the same password for all of your accounts.

KeePass makes it easy to choose different passwords for every login, because it remembers the passwords for you.

All you have to remember is the one master password to unlock the encrypted database.

Operating System: Windows.

5. PWGen
Have trouble thinking of strong passwords? PWGen thinks of random strings of letters and numbers for you, so that you don't end up choosing something easy to guess, like "123456."

Operating System: Windows.

Get Organized
6. Data Crow
If you'd like to organize your collection of books, CDs, DVDs, or anything else, Data Crow simplifies the process and gives you a tool for tracking when your friends borrow your stuff.

When last we checked, the project's main site was down, but you can still download the app from Sourceforge at the link above.

Operating System: OS Independent

7. eLibrary
If you have a growing collection of ebooks, you may have trouble keeping track of what you have. It automatically downloads your books' covers from the Internet and makes it easy to browse or search your collection to find the eBook you want.

Operating System: Windows.

8. Krecipes
Krecipes combines several New Year's resolutions into one app: it lets you try a new hobby (cooking), manage your caloric intake, and organize your recipes.

It also lets you track and plan meals on a calendar and creates shopping lists.

Operating System: Linux.

9. FreeMind
The FreeMind mind-mapping software makes it easy to create free-flowing diagrams like you would in a brainstorming session. Because mind-mappers are so different from other types of applications, it sometimes takes a while to get used to them.

However, often people get hooked and use FreeMind on a daily basis to organize their thoughts.

Operating System: Windows, Linux, OS X.

10. Dia
For those who are really serious about organizing their lives, Dia can help you create flowcharts for any process. It's very similar to Visio, and has extensive help for new users.

Operating System: Windows, Linux/Unix.

11. GanttProject
If you're undertaking a large, multi-step project, say organizing a PTA fundraiser or managing your own home remodel, project management software like GanttProject can be extremely helpful.

Within just minutes you'll be breaking your project into tasks, assigning tasks to particular people, and scheduling when you want everything to be done.

Operating System: Windows, Linux, OS X.

12. OpenProj
Similar to GanttProject, OpenProj was specifically designed as a replacement for Microsoft Project and other commercially available project managers.

It claims to offer "the most advanced scheduling, resource and cost algorithms in the industry."

Operating System: Windows, Linux, Unix, OS X.

13. PNotes
Always forgetting things? PNotes lets you leave yourself electronic "sticky notes" on your desktop. You pick the size and color of the notes.

Operating System: Windows

14. DocFetcher
Can't remember where you put a file or what you called it? DocFetcher searches the text of your files to help you find the document you were wanted much faster than you would with the built-in Windows search feature.

Also, because it only searches documents (not photos, music, videos, applications, etc.), DocFetcher usually works faster than most other desktop search apps too, including Google Desktop.

Operating System: Windows, Linux

15. Launchy
Another desktop enhancement that makes it easier to find files on your system, Launchy starts applications or open files when you start typing the name.

With this utility, you'll never need desktop icons, your Windows start menu, or even a file manager.

Operating System: Windows, Linux

Make Better Use of Your Time
1. Rachota
This app can track the amount of time you spend on any task—whether it's work-related or a project you're working on around the house.

Plus, it's portable, so you can take it with you on a USB drive or a handheld device.

Operating System: OS Independent.

2. RedNotebook
This unique app combines a calendar/scheduler with a text editor. It's great for keeping a diary, tracking your thoughts, or leaving yourself notes about upcoming events and tasks.

Operating System: Windows, Linux

1. Makagiga
Like RedNotebook, Makagiga includes a to-do list, calendar, and text editor, but it also adds a feed reader and a sticky-note widget.

You can also import and export documents from other applications.

Operating System: Windows, Linux

2. ThinkingRock
This app helps you put into practice the "Getting Things Done" methodology featured in books by David Allen.

Different screens help you collect your thoughts, process thoughts, then organize, review, and do.

Operating System: Windows, Linux

3. Task Coach
Most to-do list programs lack the ability to break a task into smaller components, but not this one. It's also portable so that you can take it with you on a smartphone, iPod, or other device.

Operating System: Windows, Linux, OS X, and others.

4. eHour
While the other projects in this category can help a single person become more efficient, eHour can help an entire office staff track their time.

Designed for consultants, freelancers, lawyers, and other small businesses that charge by the hour, eHour can track multiple employees, multiple clients, and multiple projects.

Operating System: Windows, Linux, OS X.

Take Up a New Hobby
5. Stellarium
If you've always wanted to learn more about astronomy, Stellarium can help. It allows you to view the night skies from any point on earth at any time. It's so accurate, it's even used to power many planetariums.

Operating System: Windows, Linux, OS X.

6. Celestia
Celestia takes the capabilities of Stellarium one step further. Instead of just viewing the night sky from earth, you can also fly through the virtual heavens to view the sky as seen from Mars, Jupiter, or any other point in the known galaxy.

Operating System: Windows, Linux, OS X.

7. Gramps
Gramps (short for "Genealogical Research and Analysis Management Programming System") makes it easy to get started on the popular hobby of genealogy. It boasts a very active user community and nearly 1,000 pages of online help.

Operating System: Linux, Windows

8. GenealogyJ
This Java-based genealogy app lets you view family information as a family tree, in a table, on a timeline, or by geographic location. It's appropriate both for people looking for a hobby and more serious historians.

Operating System: OS Independent

9. Sweet Home 3D
For those interested in interior decorating as a hobby, Sweet Home 3D lets you create a 2D floorplan and then see what your room will look like in 3D.

If you don't want to download the software, you can also use the app online and even save your data in the cloud for free.

Operating System: Windows, Linux, OS X.

10. LenMus
Whether you're an accomplished musician or just a beginner, LenMus can help you improve your musical skills.

It covers the fundamentals of music theory and site reading, with plenty of exercises for practice.

Operating System: Windows, Linux, OS X.

11. GNU Solfege
Even if you weren't born with perfect pitch, you can improve your "ear" for music. This app offers ear training exercises that teach you to identify intervals, sing intervals, identify rhythm patterns, sing scales, sing chords, and learn harmonic progressions.

Operating System: Windows, Linux, OS X.

12. Numismatic
Looking for a hobby with an impressive-sounding name? Become a numismatist, otherwise known as a coin collector. This app can help you track your collection in a MySQL database.

Operating System: OS Independent.

13. JDiveLog
Want to try a more adventurous hobby? How about Scuba? JDiveLog is a digital logbook for scuba divers with a huge list of features.

Operating System: Windows, Linux, OS X.

Learn a New Language
14. jVLT
Short for "Java Vocabulary Learning Tool," jVLT combines flashcard, dictionary, and quizzing capabilities in a single app.

You can make up your own vocab list or use one of the built-in libraries which include German-French, Thai-English, French-English, practical Chinese, and several other lists.

Operating System: Windows, Linux, OS X.

15. Zkanji
If you've always wanted to learn Japanese, Zkanji can help. In essence, it's a very elaborate Japanese-English dictionary, with a number of special tools to help you read, write, and recognize kanji symbols.

Operating System: Windows.

16. ZWDisplay
Practicing Chinese on your own can be very difficult, but ZWDisplay makes it easier by pronouncing Chinese words for you.

You simply input a Chinese word, and the app pronounces it for you and shows you the pronunciation in pinyin.

Operating System: Linux.

Read More
17. FBReader
Reading is more convenient when you can take ebooks with you wherever you go. This app lets you use a netbook, laptop, or other Windows- or Linux-based devices as an ebook reader.

Operating System: Windows, Linux

1. RSS Owl
You have no excuse for not keeping up with the news when you have a tool like RSS Owl. One of the most popular feed readers available, RSS Owl aggregates headlines from all of your favorite media sites and makes them easy to browse.

It has a ton of advanced features, like saved searches, news filters, labels, and news bins that set it apart from other similar apps.

Operating System: Windows, Linux, OS X.

2. Skim
If you need to read a lot of pdf files, for example technical or scientific papers, Skim makes it easier to read and take notes digitally instead of printing out pages and making notes by hand.

You can also use it to give presentations using a pdf file instead of a traditional presentation program.

Operating System: OS X.

Improve Your Mind
3. Brain Workshop
Give "Dual N-Back" memory exercises a try with Brain Workshop. These simple tasks have been shown to improve short-term memory and fluid intelligence, and it may help reduce the symptoms of ADD/ADHD.

Operating System: Windows, Linux, OS X.

4. Genius Genius can help you memorize almost anything—foreign language phrases, vocabulary words, historical events, legal definitions, even formal speeches.

It uses a "spaced repetition" method that takes your previous answers into account when quizzing you on the material to be learned.

Operating System: OS X.

5. The Mnemosyne Project
Like Genius, The Mnemosyne Project can help you learn almost anything using a flashcard-type interface. It uses a complicated algorithm to determine which card should come next in order to maximize the effectiveness of your study time.

Operating System: Windows, Linux, OS X.

6. Parley
This program from the KDE Education Project can be used to learn any material, but many pre-made card sets are available.

Unlike some of the other "flashcard" programs, it offers several different types of testing: multiple choice, fill in the blank, etc.

Operating System: Windows, Linux.

7. Pauker
Pauker aims to strengthen your ultra-short-term, short-term, and long-term memory as you learn facts in a flashcard-like fashion.

It's Java-based and available in a mobile version, so you can use it on almost any device.

Operating System: Windows, Linux, OS X.

8. OpenCourseWare
Many universities have "open-sourced" the content of their classes. The link above isn't exactly an app, but it will connect you with free courses that you can take from dozens of leading institutions around the world.

Operating System: OS Independent.

Become More Spiritual
9. Zekr Qu'ran
Available in several different languages, Zekr aims to make it easy to read and study the Qu'ran. It also includes a number of supported audio files, so you can hear the Qu'ran read aloud.

Operating System: Windows, Linux, OS X.

10. BibleTime
This Bible study software offers access to more than 200 books, including different versions of the Bible, commentaries, dictionaries, and other Christian books.

The Web site also offers a guide on how to study the Bible on your own.

Operating System: Windows, Linux, BSD

11. Xiphos
Xiphons, formerly known as GnomeSword, offers many of the same features as BibleTime, as well as prayer list and journaling capabilities.

Operating System: Windows, Linux

Give Open Source a Try
OK, we'll admit it—trying open-source software isn't at the top of the list of most popular New Year's Resolutions.

But just in case your resolution is to try open source for the first time, these are some of the best apps for open-source newbies.

12. Firefox
Quickly becoming one of the most popular browsers available, Firefox attracts many new users with its speed and advanced security features.

As an added bonus, it also offers a lot of personalization features and add-ons, so you can have a browser that's customized specifically for you.

Operating System: Windows, Linux, OS X

You don't have to purchase Microsoft Office in order to open or save files in Microsoft formats. OpenOffice gives you word processor, spreadsheet, presentation, database, and other office productivity software with many of the same features as their Microsoft counterparts.

Operating System: Windows, Linux, OS X

14. OpenDisc
OpenDisc combines a lot of the most popular open-source apps for Windows in a single download. It includes apps like Dia, The Gimp, Inkscape, Scribus, Enigma, Firefox, Pidgin, RSSOwl,, and many others. If you'd rather not download such a big file, you can also pay a fee for a disk-based version.

Operating System: Windows

Like OpenDisc, combines many, many open-source apps in a single package. In this case, the selected apps are lightweight enough that you can take them with you on a thumb drive.

That means you'll always have access to your favorite apps and settings no matter what system you're using.

Operating System: Windows.

16. Ubuntu
If you're ready to make the big switch from Windows or Mac to Linux, consider Ubuntu. This version of Linux offers a particularly user-friendly look and feel, and it gives you instant access to all of the apps you're likely to need on a daily basis.