Thursday, February 28, 2013

Unix: Under the spell of magic numbers

http://www.itworld.com/operating-systems/344491/unix-under-spell-magic-numbers

If you've ever wondered how Unix systems identify files, you might be surprised to learn that file names really aren't an important factor. Unix systems reach into files looking for special codes called "magic numbers" to figure out what your files really are. They also examine other file content to pick out additional details. - See more at: http://www.itworld.com/operating-systems/344491/unix-under-spell-magic-numbers#sthash.QJPMtyqh.dpuf

Magic numbers play a very important role on Unix systems. They help the OS recognize details about files that aren't obvious even to Unix wizards.
What are magic numbers?

Magic numbers are values, not generally visible unless you dump file contents using a command like od that displays file contents in hex, that serve as signatures for file type identification. Some are fairly obvious once you see them -- like if you spot "\x89PNG" in a file dump. Others won't offer much of a clue.

In general, they are special values set up at or near the beginning or nearby in files that allow command like file to distinguish files by their type so even if you call a jpg file my.image, the OS can still figure out what it is.
How are they used?

The tool for file identification is a command named "file". Use the file command to examine your oddly named my.image file and might just tell you something like this:

$ file my.image
my.image: JPEG image data, JFIF standard 1.01

The "JFIF" tag in this description stands for JPEG File Interchange Format. This output tells you that this complies. This tells you that the magic number that is associated with JPEG files is both stored in this file and stored in the right location to identify the file. The same value at some other location would have no effect and would likely be coincidental.

Use file to examine a file that isn't a JPEG file, but has been assigned a .jpg file extension and the file command comes through and identifies this file as an executable.

$ file bash.jpg
binfile.jpg: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/
Linux 2.6.9, dynamically linked (uses shared libs), stripped

If we follow up this inquiry by examing the beginning of the file with the od command, we can see what appears to be an identifier for ELF files strikingly present in the first four bytes.

$ od -bc bash.jpg | head -4
0000000 177 105 114 106 001 001 001 000 000 000 000 000 000 000 000 000
177 E L F 001 001 001 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000020 002 000 003 000 001 000 000 000 340 307 005 010 064 000 000 000
002 \0 003 \0 001 \0 \0 \0 340 307 005 \b 4 \0 \0 \0

Do the same thing with out odd my.image file and we see the "JFIF" tag within the magic number fairly clearly.

$ od -bc my.image | head -4
0000000 377 330 377 340 000 020 112 106 111 106 000 001 001 001 001 054
377 330 377 340 \0 020 J F I F \0 001 001 001 001 ,
0000020 001 054 000 000 377 341 035 122 105 170 151 146 000 000 111 111
001 , \0 \0 377 341 035 R E x i f \0 \0 I I

Most magic numbers aren't readable unless you use a tool such as od to examine the file's contents. The exception is the shebang line that we use in scripts to identify the shell to be used when they are run. This serves as a kind of magic number on its own and is used by the file command as such.

For most files, lines in the /usr/share/file/magic file describe what the file command needs to look for. The line for a PNG file, for example, looks like this:

0 string \x89PNG PNG image data,

This line tells us that there's no offset (the magic number appears at the beginning of the file),

Looking at tux.png, an image of the beloved penguin, file tells us:

$ file tux.png
tux.png: PNG image data, 400 x 479, 8-bit/color RGBA, non-interlaced

Now that's a lot more than just the magic number indicates. The file command, once determining that the file is a PNG file, can look at other content in the file to determine its size and color depth.

$ od -bc tux.png | head -4
0000000 211 120 116 107 015 012 032 012 000 000 000 015 111 110 104 122
211 P N G \r \n 032 \n \0 \0 \0 \r I H D R
0000020 000 000 001 220 000 000 001 337 010 006 000 000 000 176 261 216
\0 \0 001 220 \0 \0 001 337 \b 006 \0 \0 \0 ~ 261 216

PNG files begin with this 8-byte signature which is very obvious in the output shown above.

\211 P N G \r \n \032 \n

Magic numbers help ensure that Unix systems have more to go on than file names when trying to identify file content. Examine the /usr/share/file/magic on your Linux system and you'll notice that there's a lot of information there about different types of files.
- See more at: http://www.itworld.com/operating-systems/344491/unix-under-spell-magic-numbers#sthash.QJPMtyqh.dpuf

75 Open Source Replacements for Popular Education Apps

http://www.datamation.com/open-source/75-open-source-replacements-for-popular-education-apps.html

School budgets never seem to get any larger, but one way educational institutions may be able to cut costs is by deploying open source software. The open source community has developed applications that educators can use directly in the classroom, apps that are great for use at home and tools that administrators can use for school management.
This month, we've updated our list of top open source applications. And for reference sake, we've included comparable closed-source software that these applications can replace. However, note that when we say "replace," we don't mean that these open source applications have exactly the same features as the closed-source versions — only that the programs serve a similar function.
As always, if you know of additional open source applications that you think should be on our list, please let us know in the comments section below.

Alphabet

1. KLettres
Replaces School Zone Alphabet Express, Reader Rabbit Toddler
KLettres teaches the very youngest computer users to recognize the appearance and sound of letters and syllables. It's also useful for older computer users who are learning a new language, such as Arabic, Czech, Brazilian Portuguese, Danish, Dutch, British English, English, English Phonix, French, German, Hebrew, Hungarian, Italian, Kannada, Hebrew, Hindi Romanized, Low Saxon, Luganda, Malayalam, Norwegian Bokmål, Punjabi, Spanish, Slovak, Ukrainian or Telugu. (Note that in order to use KLettres on Windows, you'll have to download KDE for Windows.) Operating System: Windows, Linux.

Astronomy

2. Celestia
Replaces Starry Night, Seeker
A fabulous tool for home or the classroom, Celestia contains an interactive map of the known universe built with actual imagery of objects in space whenever possible. Not only can you view the sky from any point on earth, this app lets you "fly" throughout the galaxy and see how the stars would look from Mars, Jupiter, Pluto or beyond. Operating System: Windows, Linux, OS X.
3. Stellarium
Replaces The SkyX, Perseus, SkyMap Lite
While Stellarium doesn't include Celestia's "flying" capabilities, it does present an incredibly accurate and detailed picture of the night sky from any point on earth at any time. In fact, it's so well-done that many planetariums use it to power their shows. Operating System: Windows, Linux, OS X.
4. KStars
Replaces The SkyX, Perseus, SkyMap Lite
Similar to Stellarium, KStars lets users view "up to 100 million stars, 13,000 deep-sky objects, all 8 planets, the sun and moon, and thousands of comets and asteroids." It also includes a number of tools helpful for amateur astronomers, such as an observation list, an FOV editor, a sky calendar, supernova alerts and a glossary of technical terms. (Note that in order to use KStars on Windows, you'll have to download KDE for Windows.) Operating System: Windows, Linux.
5. PP3
Replaces Starry Night, SkyMap Software
Science teachers often need to reproduce star charts for use during class lectures, as well as for handouts and tests. This app works with LaTeX, to simplify the process of creating star charts for use in presentation slides or for printing. Operating System: Windows, Linux.
6. StarChart
Replaces Starry Night, SkyMap Software
StarChart describes itself simply as a program that "draws maps of things in the sky." Like PP3, it does a good job of producing high-quality star charts for study purposes. Operating System: Linux.

Art

7. Tux Paint
Replaces Crayola Art Studio
This basic drawing program features a kid-friendly interface, numerous drawing tools, special effects and Tux the Linux penguin. Although the website says it's for children ages 3 to 12, it's probably best for those 7 and under. Operating System: Windows, Linux, OS X.
8. GPaint
Replaces Microsoft Paint
Students who are beyond Tux Paint but not quite ready for professional drawing and photo manipulation tools might enjoy this Gnu app. It offers an easy-to-use interface and a wide variety of drawing tools. Operating System: Linux.
9. Inkscape
Replaces Illustrator, CorelDraw
This professional-caliber vector drawing program is suitable for advanced high school and college students or adults who want to experiment with digital art. The site also features a library of open source clip art to help you get started with your own creations. Operating System: Windows, Linux, OS X
10. Alchemy
Replaces Microsoft Paint
Designed to help artists get their creative juices flowing, Alchemy has a deliberately limited feature set--no undo, no selecting, no editing. Art teachers can use it as a starting point to help students "sketch" new ideas before going on to create finished pieces of art using real-world media or more advanced applications. Operating System: Windows, Linux, OS X

Chemistry

11. Kalzium
Replaces Periodic Table of Chemical Elements
Need help with introductory chemistry? This KDE app allows students to explore the periodic table, and it comes complete with a molecular weight calculator, an isotope table, a 3D molecule editor and an equation solver for stoichiometric problems. (Note that in order to use Kalzium on Windows, you'll have to download KDE for Windows.) Operating System: Windows, Linux.
12. Avogadro
Replaces ChemDraw
For more advanced students and professional chemists, Avogrado offers an intuitive interface for creating visualizations of molecules. The website also includes some tips for educators on integrating Avogadro into the classroom. Operating System: Windows, Linux.

Educational Games

13. GCompris
Replaces Various JumpStart Packages, Various Reader Rabbit Packages
Children 10 and under will enjoy this collection of more than 100 educational games. In addition to activities designed to teach lessons about reading, history, math, science and other subjects, it also includes "fun stuff" like chess, memory, sudoku and similar activities. Operating System: Windows, Linux.
14. ChildsPlay
Replaces Various JumpStart Packages, Various Reader Rabbit Packages
ChildsPlay is also a collection of apps, although it doesn't include nearly as many activities as GCompris. It offers 11 games, including memory games, alphabet and number recognition activities and puzzles. Operating System: Windows, Linux, OS X.

Foreign Language

15. ZWDisplay
Replaces Declan's ReadWrite Chinese
A combination dictionary/flashcard program, ZWDisplay helps Mandarin students learn to read and pronounce Chinese characters. It uses a unique coloring scheme and a wide variety of options to simplify the language learning process. Operating System: Linux.
16. Step Into Chinese
Replaces Declan's ReadWrite Chinese
Step Into Chinese describes itself as a "language mining tool" that helps native English speaker understand Mandarin in context. It includes 8,300 Chinese characters that correspond to 26,000 modern Chinese words and concepts. Operating System: Windows, Linux, OS X.
17. Zkanji
Replaces JISHOP
This app for students learning Japanese includes a dictionary with more than 190,000 words, as well as a flashcard program. It also includes helpful information about how to write kanji characters. Operating System: Windows.

Flashcards

18. FlashQard
Replaces Flash Card Manager, studyPerfect, WinFlash
Using the Leitner method of presenting flashcards, this app optimizes your study time for maximum efficiency by focusing on the information you don't yet know. Create your own cards for any subject or use one of the pre-made card sets for learning languages or geography facts. Operating System: Windows, Linux.
19. jVLT
Replaces Flash Card Manager, studyPerfect, WinFlash
Very similar to FlashQard, Java-based jVLT also uses the Leitner method to help students master concepts. However, this app focuses primarily on vocabulary, making it ideal for students who are studying a foreign language. Operating System: Windows, Linux, OS X.

20. The Mnemosyne Project
Replaces Flash Card Manager, studyPerfect, WinFlash
One of the better flashcard apps available, Mnemosyne was born from a research project that is studying the nature of long-term memory. It supports multiple card formats and uses an algorithm to determine which card should appear next. Users who choose to do so can upload their progress anonymously in order to provide data for the research team. Operating System: Windows, Linux, OS X, Android/
21. Pauker
Replaces Flash Card Manager, studyPerfect, WinFlash
Java-based Pauker lets you take your flashcards with you on almost any device. Like many of the other apps in this category, it uses the Leitner method of flashcard presentation to strengthen your ultra-short-term, short-term, and long-term memory. Operating System: OS Independent.
22. Parley
Replaces Flash Card Manager, studyPerfect, WinFlash
In addition to traditional flashcards, Parley also utilizes anagrams, multiple choice, fill-in-the-blank, synonym/antonym and other types of tests to improve your vocabulary. Many user-created card sets are available at KDE for Windows. Operating System: Windows, Linux.
23. Anki
Replaces Flash Card Manager, studyPerfect, WinFlash
This app boasts that it can help students learn just about any type of information, whether they are learning a new language, studying for medical exams or trying to remember guitar chords. It comes in iOS and Android versions, making it ideal for learning on the go. Operating System: Windows, Linux, OS X, iOS, Android.

Geography

24. WorldWind
Replaces Google Earth
Very similar to Google Earth, this NASA-developed tool lets you view satellite imagery and maps for any location on earth. It also includes an API for developers who want to include WorldWind functionality within their own apps. Operating System: OS Independent.
25. Marble
Replaces Google Earth
This "virtual globe" goes far beyond basic mapping, integrating weather, photos, traffic and Wikipedia information. With the mobile versions (currently only for select Nokia phones, an Android version is in the works), it also becomes a turn-by-turn navigation system. Operating System: Windows, Linux, OS X.

Library Automation

26. VuFind
Replaces Alexandria, ResourceMate, Surpass, Mandarin Library Automation
This library management system boasts Web 2.0 capabilities and design done "by libraries for libraries." The modular architecture means it's easy to install just the features you need--or to add or modify modules as necessary. Operating System: OS Independent.
27. LibLime Koha
Replaces Alexandria, ResourceMate, Surpass, Mandarin Library Automation
Used by thousands of libraries around the world, Koha describes itself as "the most advanced and cost effective open source automation solution on the market." Paid hosting, support, consulting, implementation and development services are also available on the site. Operating System: OS Independent.
28. OpenBiblio
Replaces Alexandria, ResourceMate, Surpass, Mandarin Library Automation
OpenBiblio features include an online public access catalog (OPAC), circulation, cataloging, and staff administration. See the site for extensive documentation and templates. Operating System: OS Independent.
29. Evergreen
Replaces Alexandria, ResourceMate, Surpass, Mandarin Library Automation
This library automation solution prides itself on its scalability. The Evergreen community is unusually active, with regular newsletters and updates, and you can find demos of the software and an interactive of libraries using Evergreen at the website. Operating System: OS Independent.
30. OPALS
Replaces Alexandria, ResourceMate, Surpass, Mandarin Library Automation
Short for "OPen-source Automated Library System," OPALS is one of the most highly rated library automation solutions available. It provides Web-based access to information databases, library collections and digital archives. Paid hosting and other services are available. Operating System: OS Independent.
31. NewGenLib
Replaces Alexandria, ResourceMate, Surpass, Mandarin Library Automation
NewGenLib proudly proclaims itself the "best open source library system." Notable features include support for Android smartphones and tablets, standards compliance, an intuitive interface and much more. Operating System: OS Independent.

Logic/Debate

32. Argumentative
Replaces Rationale
Ideal for students taking logic, debate, law, writing, speech and similar classes, Argumentative makes it easy to create a visual representation of an argument. It tracks premises, reasons, objections, helpers, reasoning, analysis and more to help users create well-reasoned papers and speeches. Operating System: Windows.
33. Argunet
Replaces Rationale
Argunet offers both an argument editor (similar to Argumentative) and a library of arguments that students can browse and analyze. You can use it on a standalone system or in a client-server setup to encourage collaboration among students or debate teams. Operating System: Windows, Linux, OS X.
34. iLogos
Replaces Rationale
Java-based iLogos isn't quite as fancy as some of the other argument mappers, but it does make it fairly easy to diagram arguments with a combination of words and pictures. The site also includes a helpful list of similar tools. Operating System: Windows, Linux, OS X.

Math

35. Sage
Replaces Visual Math, Mathematica
Sage brings together more than 100 other open source math projects under a single unifying interface. It can handle everything from basic algebra to calculus to cryptography to group theory and everything in between. However, the interface is very basic and will feel most comfortable to users with some background in programming. Operating System: Windows, Linux, OS X.
36. Genius
Replaces Visual Math, Mathematica
This app functions both as a calculator and as a research/educational tool. To use it, you'll have to enter a mathematical expression using the Genius Extension Language (GEL), but the language is designed to look as much like regular math syntax as possible. Operating System: Linux, OS X.
37. Scilab
Replaces Visual Math, Mathematica
Designed for engineers and scientists, Scilab can handle maths and simulation, 2D and 3D visualizations, optimization, statistics, control system design and analysis, signal processing and more. Paid support, training and services are available through Scilab Enterprises. Operating System: Windows, Linux, OS X.
38. Maxima
Replaces Visual Math, Mathematica
A descendant of MIT's Macsyma, Maxima describes itself as "a system for the manipulation of symbolic and numerical expressions, including differentiation, integration, Taylor series, Laplace transforms, ordinary differential equations, systems of linear equations, polynomials, and sets, lists, vectors, matrices, and tensors." It can also create both 2D and 3D graphs. Operating System: Windows, Linux, OS X.
39. GeoGebra
Replaces Visual Math, Mathematica
With its wide range of capabilities, GeoGebra is suitable for elementary students learning arithmetic, high school students learning algebra and geometry, and even for college students studying calculus and advanced mathematics. Specifically designed as an educational tool, it is much more user-friendly than many of the other math apps on our list, and the website also includes helpful resources for educators. Operating System: OS Independent.
40. GraphCalc
Replaces Visual Math
GraphCalc isn't so much as replacement for other software as a replacement for handheld graphing calculators. The site boasts, "GraphCalc can be your first, last, and only line of offense against the mathematics that threaten to push you over the brink of insanity. It slices, dices, shreds and purees functions that leave other calculators wondering what hit them." The project hasn't been updated in a while, but it remains a helpful alternative to graphing calculators. Operating System: Windows, Linux.

41. gnuplot
Replaces Visual Math, Mathematica
Unlike many of the other math apps on our list, Gnuplot does just one thing--create graphs. It works from the command line and supports both 2D and 3D plots. Operating System: Windows, Linux, Unix, OS X, and others.
42. Dr. Geo
Replaces Cabri
This award-winning app invites people age 7 to 77 to "be a geometer." It allows students to create and interact with sketches of geometric objects so that they can better understand geometry lessons. Operating System: Windows, Linux, OS X.
43. Kig
Replaces Cabri
Similar to Dr. Geo, KDE's geometry app also provides an interface for working with and learning more about geometric figures. (Note that in order to use Kig on Windows, you'll have to download KDE for Windows.) Operating System: Windows, Linux.

Music

44. MuseScore
Replaces Sibelius, Finale
There's no need to spend hundreds of dollars on music notation software or to spend hundreds of hours writing scores by hand. MuseScore creates, prints and even plays musical scores--and it's free. Operating System: Windows, Linux, OS X.
45. LenMus/Phonascus Replaces Practica Musica
Music theory students will likely find this app helpful. It includes lessons and interactive games that quiz you on your theory knowledge and ear training. Operating System: Windows, Linux, OS X.
46. GNU Solfege
Replaces Earmaster
This ear training app has exercises designed to teach students to identify intervals, chords, scales and rhythmic patterns. It also includes some theory and sight singing activities. Operating System: Windows, Linux, OS X.

Physics

47. Step
Replaces Yenka Science
Step is an interactive physics simulator that allows students to explore classical mechanics, particles, springs, gravity, coulomb forces, collisions, sound waves and much more. It also includes an equation solver and a unit conversion tool. (Note that in order to use Step on Windows, you'll have to download KDE for Windows.) Operating System: Windows, Linux.

Online Course Management Systems/E-Learning

48. Moodle
Replaces Pearson Learning Studio, Articulate Storyline, TrainCaster LMS
Incredibly popular, Moodle is a highly scalable course management system that currently powers nearly 7 million online courses for more than 65 million users. The website offers helpful demonstrations so that you can see this CMS in action from both an instructor's and a student's point of view. Operating System: Windows, Linux, OS X.
49. Sakai
Replaces Pearson Learning Studio, Articulate Storyline, TrainCaster LMS
Sakai offers two separate open source products: Sakai CLE, which is similar to other online learning systems, and Sakai OAE which is a collaboration and social sharing platform for academics, researchers and students. Several commercial affiliates offer hosting and related services. Operating System: OS Independent.
50. Claroline
Replaces Pearson Learning Studio, Articulate Storyline, TrainCaster LMS
Available in many different languages, Claroline is used by organizations in more than 100 different countries. It boasts a highly intuitive interface, and it's flexible enough to meet the needs of businesses and associations, as well as colleges and universities. Operating System: Windows, Linux, OS X.
51. eFront
Replaces Pearson Learning Studio, Articulate Storyline, TrainCaster LMS
Unlike most of the other learning management systems on our list, eFront places particular emphasis on meeting the training needs of enterprises. It's available in paid hosted and enterprise versions, as well as the open source edition. Operating System: Windows, Linux.
52. ILIAS
Replaces Pearson Learning Studio, Articulate Storyline, TrainCaster LMS
ILIAS claims to be the first open source learning management system in the world to achieve SCORM 2004 (3rd Edition) compliance. It also enjoys a very active community with conferences and user events regularly scheduled in many different countries. Operating System: Windows, Linux.
53. ATutor
Replaces Pearson Learning Studio, Articulate Storyline, TrainCaster LMS
ATutor was designed to comply with industry standards, such as SCORM Content Packaging, SCORM 1.2 LMS RTE3, OAuth Authentication Protocol and others, and accessibility standards, such as W3C WCAG 2.0, W3C ATAG 2.0, US Section 508 and many others. Try the online demo to see it in action. Operating System: OS Independent.

Operating System

54. Edubuntu
Replaces Microsoft Windows
This Ubuntu-based Linux distribution aims to collect all the best in open source educational software and make it available along with an easy-to-use, free operating system. It's developed by a community committed to the idea that "learning and knowledge should be available to everyone who wants to improve themselves and the world around them."
55. Qimo for Kids
Replaces Microsoft Windows
As you might guess from the name, this Ubuntu-based operating system was designed to be used by kids, specifically by those as young as three years of age. It features an extensive library of educational games, and its large icons make it easy for young children to find what they're looking for.
56. Debian Edu/Skolelinux
Replaces Microsoft Windows
This Debian variation incorporates many educational games and apps, and it's available in multiple languages. The name "Skolelinux" is Norwegian for "school Linux."
57. Fedora Education Spin
Replaces Microsoft Windows
Fedora, the community version of Red Hat's Linux distribution, comes in a special education "spin." It includes the Sugar desktop environment, the KDE education apps and a version of the Eclipse integrated development environment so that students can learn to write their own code.

Religion

58. BibleTime
Replaces Logos
Top-of-the-line Bible study tools like Logos can cost hundreds or even thousands of dollars, but BibleTime offers many of the same capabilities for free. It provides searchable access to more than 200 Bibles, commentaries and related resources. Operating System: Windows, Linux, OS X.
59. Xiphos
Replaces Logos
Xiphos provides access to the same Biblical texts and resources as BibleTime—it just has a different interface. It also provides the capabilities for users to create their own modules such as journals, prayer lists and personal commentaries that they can share with friends. Operating System: Windows, Linux.
60. Zekr
Replaces Al-Qari Plus
This app provides access to the Quran and commentaries in a number of different translations. It's completely free and open source because its developers follow a very simple rule: "Never Profit Off The Prophet." Operating System: Windows, Linux, OS X.

Research/Bibliography

61. WIKINDX
Replaces EndNote, Biblioscape
WIKINDX combines tools for tracking bibliographic information with tools for collaboratively authoring papers. It can be deployed on a single system or on a Web server, and it knows how to format entries for the most popular style guides, including the Chicago Manual, APA and others. Operating System: OS Independent.
62. Zotero
Replaces EndNote, Biblioscape
Zotero stores research information where you found it—in your browser. It's a Firefox plug-in that automatically collects and organizes your research so that you can easily search it later. Operating System: OS Independent.

Student Information System

63. openSIS
Replaces PowerSchool, Schoolmaster, RenWeb
Downloaded more than 50,000 times and used by more than 12,000 schools, openSIS bills itself as "the only open source Global SIS in the world." In addition to the free community edition it comes in commercially supported school and district versions. Operating System: Windows, Linux.
64. ClaSS
Replaces PowerSchool, Schoolmaster, RenWeb
Used by some of the largest international schools in Europe, ClaSS describes itself as "an information management system which puts students first." It includes a student database, messaging, reporting and school management features.
65. OpenAdmin
Replaces PowerSchool, Schoolmaster, RenWeb
Downloaded nearly 25,000 times, this SIS has been deployed by schools all around the world—in fact, you can see a count of how many schools in each country use it right on the project home page. It includes modules for demographics, attendance, discipline, report cards, fees, online gradebook, transcripts, special needs and more. Operating System: OS Independent.
66. SchoolTool
Replaces PowerSchool, Schoolmaster, RenWeb
Designed for schools in the developing world, SchoolTool runs on Ubuntu and has been translated into numerous languages. The latest update can track skills, outcomes and standards, which is helpful for complying with regulations in many areas. Operating System: Linux.
67. Open-School
Replaces PowerSchool, Schoolmaster, RenWeb
This very polished SIS describes boasts user-friendliness, security and scalability. A free community version is available, but the website primarily promotes the supported commercial version. Operating System: Windows, Linux.
68. Akura
Replaces PowerSchool, Schoolmaster, RenWeb
Akura was designed for a "college" (which equates to a K-12 school in the U.S.) in Sri Lanka and serves many Asian institutions. It's won many awards, and it's particularly popular with religious schools. Operating System: Windows, Linux.

Testing

69. TCExam
Replaces Exam Pro, QuestionMark
This computer-based assessment software aims to simplify the creation, delivery and scoring of tests. It's available under an open source license for non-commercial use or with a paid commercial license. Operating System: OS Independent.
70. iTest
Replaces Exam Pro, QuestionMark
This client-server based software makes it easy to give each student a different version of a test because it can pull questions from a pre-established database. It's available in nine different languages. Operating System: Windows, Linux, OS X.
71. Safe Exam Browser
Replaces Exam Pro, QuestionMark
If you're giving a browser-based test (for example, through a learning management system), this tool locks down the student's system so they can't browse the Internet or access other applications while taking the exam. It can be used with student-owned PCs, as well as with school-owned, networked systems. Operating System: Windows.

Typing

72. TuxType
Replaces Maevis Beacon Teaches Typing, Typing Instructor
For younger kids who are just learning their way around the keyboard, this app uses fun games to help students learn where the various letters are. Type quickly to help Tux the Linux penguin eat all the fish or save the earth from falling asteroids. Operating System: Windows, Linux, OS X.
73. TypeFaster Typing Tutor
Replaces Maevis Beacon Teaches Typing, Typing Instructor
Unlike TuxType, TypeFaster is a full-featured touch-typing learning system. It includes a 3D typing game, and it has a teacher mode that allows instructors to track students' progress. Operating System: Windows, Linux.
74. Klavaro
Replaces Maevis Beacon Teaches Typing, Typing Instructor
Humble Klavaro describes itself as "just another free touch typing tutor program," but it's actually quite unique for its keyboard- and language-independent operation. In addition to the familiar "QWERTY" keyboard layout, it supports six other built-in layouts, or you can use the keyboard editor to create your own. It's also available for 25 different languages, making this a good choice for users outside the U.S. Operating System: Windows, Linux.
75. WriteType
Replaces Microsoft Word
This option doesn't teach students to type—it makes typing easier, particularly for younger users. It includes features like word completion, read aloud mode, grammar checking and auto-correction that make it easier for young people to create documents on their own. Operating System: Windows, Linux.

An overview of GNU R programming language

http://how-to.linuxcareer.com/an-overview-of-gnu-r-programming-language

1. Introduction

The aim of this article is to provide an overview of the GNU R programming language. It starts a series of articles devoted to programming with R. Its objective is to present, in an organized and concise manner, the elementary components of the R programming language. It is designed to help you understand R code and write your own. It is assumed that the reader has already some basic programming knowledge of R. If you are not familiar with any of R features it is recommended that you first read A quick GNU R tutorial to basic operations, functions and data structures.

2. Expressions

An R expression is an elementary component of R code. Expression in R can be:

assignment statement;
conditional statement;
arithmetic expression.

Examples of R expressions:

> y<-100 br="">> if (1==1) 1 else 0
[1] 1
> 100/5
[1] 20

R expression are constructed from objects or functions. It is common to separate them with a new line, however, you can also separate expressions with semicolons as below.

 > "LinuxCareer.com";sin(pi);5^7
[1] "LinuxCareer.com"
[1] 1.224647e-16
[1] 78125

3. Objects

An R object can be thought of as an elementary component ("thing") of R programming language. For instance, the R objects are:

numeric vector;
character vector;
list;
function.

Examples of objects in R:

> c(1,5,2,7,9,0)
[1] 1 5 2 7 9 0
> c("GNU R programming tutorial","LinuxCareer.com")
[1] "GNU R programming tutorial" "LinuxCareer.com"           
> list("GNU R programming tutoial",c(1:5),"this is also an object in R")
[[1]]
[1] "GNU R programming tutoial"

[[2]]
[1] 1 2 3 4 5

[[3]]
[1] "this is also an object in R"

> function(a,b) {a/b}
function(a,b) {a/b}

4. Symbols

A symbol in R is a variable name. So if you assign an object to a variable name you actually assigning an object to a symbol. An environment in R, then, is a set of such symbols that are created for a given purpose.
Example of symbols in R:

> x<-3 br="">> y<- pre="" tutorial="">
Above, x and y are symbols.



5. Functions
A function in R is an object that takes as arguments
 other objects and returns an object as a result. Did you know that the 
assignment operator '<- a="" can="" function="" in="" instead="" is="" of="" p="" r="" writing="" you="">
> a<-1 pre="">
call '<- a="" and="" arguments="" as="" below="" function="" indicated="" p="" with="">
> '<- a="" pre="">
Some examples of functions in R include:



'<- assignment="" li="" operator="">
'+' summation;
'if' statement;
'[' vector reference.


Examples of functions in R:


> '+'(1,1)
[1] 2
> 'if'(1>3,"one greater than three", "one less than three")
[1] "one less than three"
> '['(a,1)
[1] 1

6. Other properties of R language

6.1. Objects are immutable in R
In R objects are immutable. This means that R will copy the object 
not just reference to the object. Consider the following example. We 
define a function that sets the ith element of vector 'x' to 4 as below


> f<-function i="" pre="" x="">
Let's see what happens if we define a vector 'w' and pass it as argument to the function 'f'.


> w<-c br="">> f(w,1)
> w
[1] 1 3 6 7
What we just observed above is that the vector 'w' was copied when it
 was passed to the function so that the function does not modify this 
vector.



6.2. Everything in R is an object 
Everything
 in R is an object. Objects are not only defined to store data as in the
 case of vectors, lists or other data structures. Other examples of 
objects in R are functions, symbols or R expressions. For instance, 
function names in R are symbol objects that point to function objects as
 indicated below


> functionname<-function br="" x="" y="">> functionname
function(x,y) x+y

6.3. Special values in R
There is a number of special values used in R. These are:



NA, used to represent missing values, means "not available";
Inf and -Inf, resulting in a calculation when the output number is too big or too small or when dividing by zero;
NaN, resulting in a calculation that is not possible to compute such as division of zero by zero, means "not a number";
NULL, used often as an argument in functions, means that no value was assigned to that argument.


6.4. Coercion
R often coerces values from one type to another. For instance, when 
you call a function with an argument of a wrong type, R will try to 
convert this argument to a different type so the function can work. 
Another example might be when we define a vector with numeric values, R 
will assign it a type "integer" as below


> x<-c br="">> typeof(x)
[1] "integer"
Now, when we change the forth element of vector 'x' to four, R will 
automatically change the type of the vector to 'double' as indicated 
below


> x[4]<-4 .1="" br="">> typeof(x)
[1] "double"

6.5. The R interpreter
An interpreter is a program that executes the 
written code. There is no need to compile R code into an object language
 as in the case of C, C++ or Java. This means that R is an interpreted 
language.


R interpreter evaluates R expressions in few steps. First, it parses 
an expression changing it into an appropriate functional form. Let's 
call the quote() function to see how this happens.


> typeof(quote(if(1>3) "one is greater than three" else "one is less than three"))
[1] "language"
The R expression above returned a "language" object. To see how R evaluates an expression we produce a parse tree.


> as(quote(if(1>3) "one is greater than three" else "one is less than three"),"list")
[[1]]
`if`

[[2]]
1 > 3

[[3]]
[1] "one is greater than three"

[[4]]
[1] "one is less than three"
Let's also apply the typeof() function to the elements in such list, which shows how the expression is interpreted by R.


> lapply(quote(if(1>3) "one is greater than three" else "one is less than three"),typeof)
[[1]]
[1] "symbol"

[[2]]
[1] "language"

[[3]]
[1] "character"

[[4]]
[1] "character"
As you can see some parts of the if statement where not included in the parsed expression. That is, the else element. Additionally, it is interesting to note that the first item in the list is a symbol, which points to the if() function. Even though the syntax for the if
 statement differs from the function call, the R interpreter translates 
the expression into the function call with the name of the function as 
its first argument and other arguments as in the list above.



7. Conclusion
This article is an introduction to the R programming language. In the
 forthcoming articles we will focus in detail on the defined here 
elements of the R language.


Make sure you tune in to our RSS and Linux jobs portal to stay informed about the latest opportunities in the field. Also visit our Linux Forum if you want to share your Linux experiences with us or require additional help.

Ubuntu Linux – Apache and Self Signed Certificates

http://pinehead.tv/linux/ubuntu-linux-apache-and-self-signed-certificates

Whether it is on your desktop or server installation of Ubuntu, there will come a time that you may need to work with Apache and certificates. We will go into full certificates from Certificate Authorities (like Verisign or Entrust) as well as exploring some of the ‘Open Source’ Certificate Authorities (read: free) in a later article. Today we are discussing how to prepare Apache to answer HTTPS requests in the VHOSTS as well as installing and configuring the pieces. Finally, we will install a self signed certificate and access our system over HTTPS to verify it all works.

Assembling All the Pieces
The first thing we need to do (assuming that Apache 2 is already installed) is to make sure we have all the pieces. Let’s install the SSL package and module we need:
sudo apt-get install openssl
This will install the parts needed to generate our certificate and the module that apache needs to enable SSL support (NOTE: Apache 2 must be installed prior to running this command in order for the module to be installed in the proper location). Once that is installed, apache needs to be told to enable the module:
sudo a2enmod ssl
If this is the first time you have used apache’s shortcut scripts, you may not be familiar with the most common one’s – you will want to be familiar with the following scripts when using apache in any Debian based distribution (including Ubuntu):

a2enmod: Shortcut for ‘Available To Enabled Module’, takes an installed module and creates a link from ‘/etc/apache2/mods-available’ to ‘/etc/apache2/mods-enabled’ so that when apache is restarted, the module will be enabled in the live configuration
a2dismod: Shortcut for ‘Available to Disabled Module’, removes the link from ‘/etc/apache2/mods-enabled’ created when the module was enabled (see above), the module will then be disabled after apache is restarted
a2ensite: Shortcut for ‘Available to Enabled Site’, takes an installed site (vhost) and creates a link from ‘/etc/apache2/sites-available’ to ‘/etc/apache2/sites-enabled’ so that when apache is restarted, the site will be enabled in the live configuration
a2dissite: Shortcut for ‘Available to Disabled Site’, removes the link from ‘/etc/apache2/sites-enabled’ created when the site was enabled (see above), the site will then be disabled after apache is restarted

Generating the Certificate
Our module is installed and active (at least the next time we restart apache), so now we need to generate a certificate and then note their locations (we will place our resulting certificate files and keys in the most ‘standard’ locations, but they can be changed to whatever is appropriate in your circumstance as long as you note where they are during our vhost setup later). First, we need to generate the ‘Certificate Request’ file:

Creating Our Certificate Request

As you can see, you will be asked for a number of pieces of information. During the generation and installation of a ‘Self Signed’ certificate, like we are installing, this information is unimportant. However, in our next article on the topic, it will be important since some of these items will generate the key embedded in your certificate and that is used by the Certificate Authority who issues the final certificate to validate your identity and secure your site. At this point, as you can see I did, you can put in almost anything you want.
A couple more things to complete the creation of our certificate and move the files into place. See the following screen shot:

Creating Our Certificate and Moving Files

Informing Our Web Server
A couple more steps and we are ready to test. We need to grab a vhost for apache to use for SSL requests and then we need to make some changes to it in order to use our key and certificate file. The easiest way to do this is to copy the following:
sudo cp /etc/apache2/sites-available/default-ssl /etc/apache2/sites-available/my-ssl
Then edit the ‘my-ssl’ file and be sure to add the following (comment out or remove existing entries if they are different than this):

SSLEngine on
SSLOptions +StrictRequire
SSLCertificateFile /etc/ssl/certs/server.crt
SSLCertificateKeyFile /etc/ssl/private/server.key

NOTE: change the path for the certificate and/or key if you saved or named them something else in our earlier steps.
Finally, we need to restart apache so that the new virtual host is picked up:
sudo service apache2 restart
You can then go to your browser and enter ‘https://localhost’ to see your new certificate in action. You WILL get a browser security warning since this is a SELF SIGNED certificate and not issued by a Certificate Authority. This is normal and in our next article, we will cover installing a CA signed certificate which will fix that.
Final Thoughts
This will give your system a bit of extra security. For testing or local use, a self signed certificate is enough protection to secure basic authentication, database traffic or other web traffic from anyone sniffing your connection. Total time to implement this is about 20 minutes and is worth the effort.
Leave me a comment with any questions or problems and I will help out if I can.

Why you should not use .htaccess (AllowOverride All) in production

http://www.eschrade.com/page/why-you-should-not-use-htaccess-allowoverride-all-in-production

Commonly known as .htaccess, AllowOverride is a neat little feature that allows you to tweak the server’s behavior without modifying the configuration file or restarting the server. Personally, I think this is great for development purposes. It allows you to quickly test various server configurations without needing to mess with restarting the server. It helps you be more (buzzword alert!) agile.
Beyond the obvious security problems of allowing configuration modifications in a public document root there is also a performance impact. What happens with AllowOverride is that Apache will do an open() call on each parent directory from the requested file onward.
To demonstrate this I used a program called strace which checks for system calls and gives you a list of each system call that is made.
First we’ll take a look at the strace with AllowOverride set to None.

semop(1638426, {{0, -1, SEM_UNDO}}, 1) = 0
epoll_wait(42, {{EPOLLIN, {u32=3507213864, u64=139813282633256}}}, 2, 10000) = 1
accept4(4, {sa_family=AF_INET6, sin6_port=htons(55755), inet_pton(AF_INET6, "::ffff:192.168.0.212", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 43
semop(1638426, {{0, 1, SEM_UNDO}}, 1) = 0
getsockname(43, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "::ffff:192.168.0.212", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
fcntl(43, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(43, F_SETFL, O_RDWR|O_NONBLOCK) = 0
read(43, "GET /test.txt HTTP/1.0\r\nHost: ma"..., 8000) = 87
gettimeofday({1361542861, 683952}, NULL) = 0
stat("/var/www/magento.loc/test.txt", {st_mode=S_IFREG|0644, st_size=12, ...}) = 0
open("/var/www/magento.loc/test.txt", O_RDONLY|O_CLOEXEC) = 44
fcntl(44, F_GETFD) = 0x1 (flags FD_CLOEXEC)
fcntl(44, F_SETFD, FD_CLOEXEC) = 0
mmap(NULL, 12, PROT_READ, MAP_SHARED, 44, 0) = 0x7f28cfc74000
writev(43, [{"HTTP/1.1 200 OK\r\nDate: Fri, 22 F"..., 267}, {"hello world\n", 12}], 2) = 279
munmap(0x7f28cfc74000, 12) = 0
write(12, "192.168.0.212 - - [22/Feb/2013:0"..., 79) = 79
shutdown(43, 1 /* send */) = 0
poll([{fd=43, events=POLLIN}], 1, 2000) = 1 ([{fd=43, revents=POLLIN|POLLHUP}])
read(43, "", 512) = 0
close(43) = 0
read(6, 0x7fff79e2d7cf, 1) = -1 EAGAIN (Resource temporarily unavailable)
close(44) = 0
semop(1638426, {{0, -1, SEM_UNDO}}, 1

Now let’s take a look at the strace results with AllowOverride set to All.

semop(1736730, {{0, -1, SEM_UNDO}}, 1) = 0
epoll_wait(42, {{EPOLLIN, {u32=3392874024, u64=140410168747560}}}, 2, 10000) = 1
accept4(4, {sa_family=AF_INET6, sin6_port=htons(55795), inet_pton(AF_INET6, "::ffff:192.168.0.212", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 43
semop(1736730, {{0, 1, SEM_UNDO}}, 1) = 0
getsockname(43, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "::ffff:192.168.0.212", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
fcntl(43, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(43, F_SETFL, O_RDWR|O_NONBLOCK) = 0
read(43, "GET /test.txt HTTP/1.0\r\nHost: ma"..., 8000) = 87
gettimeofday({1361543373, 140117}, NULL) = 0
stat("/var/www/magento.loc/test.txt", {st_mode=S_IFREG|0644, st_size=12, ...}) = 0
open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/magento.loc/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/magento.loc/test.txt/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOTDIR (Not a directory)
open("/var/www/magento.loc/test.txt", O_RDONLY|O_CLOEXEC) = 44
fcntl(44, F_GETFD) = 0x1 (flags FD_CLOEXEC)
fcntl(44, F_SETFD, FD_CLOEXEC) = 0
mmap(NULL, 12, PROT_READ, MAP_SHARED, 44, 0) = 0x7fb3c8bf9000
writev(43, [{"HTTP/1.1 200 OK\r\nDate: Fri, 22 F"..., 267}, {"hello world\n", 12}], 2) = 279
munmap(0x7fb3c8bf9000, 12) = 0
write(12, "192.168.0.212 - - [22/Feb/2013:0"..., 79) = 79
shutdown(43, 1 /* send */) = 0
poll([{fd=43, events=POLLIN}], 1, 2000) = 1 ([{fd=43, revents=POLLIN|POLLHUP}])
read(43, "", 512) = 0
close(43) = 0
read(6, 0x7fff95abfc1f, 1) = -1 EAGAIN (Resource temporarily unavailable)
close(44) = 0
semop(1736730, {{0, -1, SEM_UNDO}}, 1

You can clearly see the additional open() calls being made to try and discover the .htaccess file. In this case the calls are completely superfluous because we have nothing there. But even so we have a significant impact on static file throughput.
AllowOverride None

Concurrency Level: 10
Time taken for tests: 2.441 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 2790279 bytes
HTML transferred: 120012 bytes
Requests per second: 4096.02 [#/sec] (mean)
Time per request: 2.441 [ms] (mean)
Time per request: 0.244 [ms] (mean, across all concurrent requests)
Transfer rate: 1116.12 [Kbytes/sec] received

AllowOverride All

Concurrency Level: 10
Time taken for tests: 3.922 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 2790558 bytes
HTML transferred: 120024 bytes
Requests per second: 2549.42 [#/sec] (mean)
Time per request: 3.922 [ms] (mean)
Time per request: 0.392 [ms] (mean, across all concurrent requests)
Transfer rate: 694.76 [Kbytes/sec] received

The requests where AllowOverride was turned off were executed at 60% of the time of the ones where AllowOverride was turned on.
And remember, this is just the impact of file operations and does not take into account the time to reconfigure Apache during the course of these requests.
So the data would clearly show that there is a negative impact to having AllowOverride turned on in a production environment. Instead it will generally be better to take those changes in .htaccess and place them in your httpd configuration file.
[UPDATE]
In fact Mike Willbanks says you should never do it. I agree with him, but I wouldn’t make as big a stink in dev as I would in prod.

Sunday, February 24, 2013

Detailed Error Handling In Bash

http://www.howtoforge.com/detailed-error-handling-in-bash

Summary

Shell scripts are often running as background processes, doing useful things without running in a visible shell. Think, for example, of cron jobs or scripts that are fired from a program on a web server. To write such scripts can be quite painful, as all errors occur out of sight as well. Off course you can make use of a log file, but the ideal level of logging is hard to find. You often log way too much when the script is running fine and way too little when it unexpectedly fails. While log files can hold a lot of information, finding the relevant information is a bit trickier.
My solution is to log only the errors with all the details to a small database. This database contains tables for the message, the corresponding stack trace and the important environment variables. I have chosen for an SQLite database in this howto, but the same principle works with other databases as well.

The database

SQLite needs some settings to work as I expect it to, and these settings can be put in an initializing script. These settings include the error behaviour of SQLite itself and its foreign key handling:

.bail ON
.echo OFF
PRAGMA foreign_keys = TRUE;

Off course, we also need a database and I do not want to rely on one to exist. Therefore, the first thing the bash script will do is to run an SQL "revive" script on the database file: if the database did not exist, it will be created and if it did, it will do nothing:

CREATE TABLE IF NOT EXISTS ErrorLog
      (intErrorLogId INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
       strMessage TEXT NOT NULL,
       tstOccurredAt DATE NOT NULL DEFAULT(CURRENT_TIMESTAMP) );
CREATE INDEX IF NOT EXISTS idxELOccurredAt ON ErrorLog(tstOccurredAt);

CREATE TABLE IF NOT EXISTS ErrorStackTrace
      (intErrorStackTraceId INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
       intErrorLogId INTEGER NOT NULL,
       strSourceFile TEXT NOT NULL,
       strFunction TEXT NOT NULL,
       intLine INTEGER NOT NULL,
       FOREIGN KEY(intErrorLogId) REFERENCES ErrorLog(intErrorLogId)
               ON DELETE CASCADE
               ON UPDATE CASCADE );
CREATE INDEX IF NOT EXISTS idxESTTraceErrorLogId ON ErrorStackTrace(intErrorLogId);

CREATE TABLE IF NOT EXISTS ErrorEnvironment
      (intErrorEnvironmentId INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
       intErrorLogId INTEGER NOT NULL,
       strVariable TEXT NOT NULL,
       strValue TEXT NULL,
       FOREIGN KEY(intErrorLogId) REFERENCES ErrorLog(intErrorLogId)
               ON DELETE CASCADE
               ON UPDATE CASCADE );
CREATE INDEX IF NOT EXISTS idxEEErrorLogId ON ErrorEnvironment(intErrorLogId);

The mechanism

Now we have the database defined, we can focus on the error handler itself. Bash has a trap command that, among other things, can be used to trap non-zero exit codes of commands. These non-zero exit codes usually denote an error. Within a "trapped" section of a script, you can still react to a non-zero exit code without the error handler to take over by using the "or construct":

false || echo "non-zero exit"

The above line will not invoke the error handler, but any of the following lines will (the command false will always exit with a non-zero exit code, the command true will exit with an exit code of zero):

false
false;true

If we define a function that will log all error data, we can pass that function to the trap command. From that function, we can access the call stack by invoking the caller command. The caller command will return an array containing the line number, the subroutine name and the file name.

I will read the environment from a predefined list of variables that I am interested in, because my (Xubuntu) system has so many pre-defined variables that logging them all would drown me in useless information. I am also interested in the messages that are written to STDERR, so I will define a file location that I can use to send the STDERR stream to. After logging the error, I will still write the contents of that file to STDERR, so the user is not left in the dark with scripts that do run interactively. Now I have all the information I want to log.

Caveat

The commands logged are the commands at the level of the trap command. This means that if you put a trap statement in your main script and call a function, that function will complete, and only after its completion, the error handler may be invoked if the function ended in an error state. If you want error handling within the function as well, put the trap command also inside it.

Invoking the error handler

To make my error handling script available as a kind of library, I will call it from the scripts I want to monitor, with the source command. This will effectively include it in the scripts I want to monitor. The calling scripts need to define a few settings before including the error handling script:

ERROR_CLEANUP_ACTION: Optional. A command to be executed after the error handler has run, for example to clean up temporary files.
ERROR_ENVIRONMENT_VARIABLES: A list of variables to log. This list will be augmented with EXITCODE BASH_COMMAND BASH_LINENO BASH_ARGV
ERRORDB: The location of the database. The database file does not need to exist, but the location does need to be writeable by the user running the script.
ERROROUTPUT: The location of a file to catch the contents of STDERR. Optional. Defaults to "/var/tmp/$$.err".
SQLITE3_EXECUTABLE: The location of the SQLite3 command line client. Optional. Defaults to `which sqlite3`.

Because escaping all possible error messages for an INSERT query can be quite a challenge, I define two auxiliary functions Error_Hexit and Error_Hexit_File. This allows me to "escape" all exotic strings to hexadecimal strings. SQLite interprets all hexadecimal strings as binary objects, so I cast them to TEXT in my INSERT statements.

The full script

After putting it all together, I have this script that I can include:

#!/bin/bash
# Needed variable: ERRORDB
# Optional variables: ERROROUTPUT, SQLITE3_EXECUTABLE, ERROR_CLEANUP_ACTION, ERROR_ENVIRONMENT_VARIABLES

if [ -z $ERROROUTPUT ];then
   ERROROUTPUT=/var/tmp/$$.err
fi
if [ -e $SQLITE3_EXECUTABLE ] ;then
   SQLITE3_EXECUTABLE=`which sqlite3`
fi
# The settings file for SQLite3:
ERROR_INIT_SQLITE=/var/tmp/$$_init.sql
# For convenience, a function to create the settings for SQLite3:
function Error_Create_Init_File
        {
         cat > $ERROR_INIT_SQLITE <<'ERROR_SQLITE_SETTINGS'
.bail ON
.echo OFF
PRAGMA foreign_keys = TRUE;
ERROR_SQLITE_SETTINGS
        }
# Create the database:
if [ -z $ERRORDB ] ;then
   echo "Error database is not defined."
   exit 1
else
   Error_Create_Init_File
   $SQLITE3_EXECUTABLE -batch -init $ERROR_INIT_SQLITE $ERRORDB <<'ERROR_TABLE_DEFINITION'
CREATE TABLE IF NOT EXISTS ErrorLog
      (intErrorLogId INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
       strMessage TEXT NOT NULL,
       tstOccurredAt DATE NOT NULL DEFAULT(CURRENT_TIMESTAMP) );
CREATE INDEX IF NOT EXISTS idxELOccurredAt ON ErrorLog(tstOccurredAt);

CREATE TABLE IF NOT EXISTS ErrorStackTrace
      (intErrorStackTraceId INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
       intErrorLogId INTEGER NOT NULL,
       strSourceFile TEXT NOT NULL,
       strFunction TEXT NOT NULL,
       intLine INTEGER NOT NULL,
       FOREIGN KEY(intErrorLogId) REFERENCES ErrorLog(intErrorLogId)
               ON DELETE CASCADE
               ON UPDATE CASCADE );
CREATE INDEX IF NOT EXISTS idxESTTraceErrorLogId ON ErrorStackTrace(intErrorLogId);

CREATE TABLE IF NOT EXISTS ErrorEnvironment
      (intErrorEnvironmentId INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
       intErrorLogId INTEGER NOT NULL,
       strVariable TEXT NOT NULL,
       strValue TEXT NULL,
       FOREIGN KEY(intErrorLogId) REFERENCES ErrorLog(intErrorLogId)
               ON DELETE CASCADE
               ON UPDATE CASCADE );
CREATE INDEX IF NOT EXISTS idxEEErrorLogId ON ErrorEnvironment(intErrorLogId);
ERROR_TABLE_DEFINITION
   rm -f $ERROR_INIT_SQLITE
fi
# Helper functions to "escape" strings tohexadecimal strings
function Error_Hexit # StringToHex
        {
         echo -n "$1" | hexdump -v -e '1 1 "%02X"'
        }

function Error_Hexit_File # FileToHex
        {
         if [ -e $1 -a -s $1 ] ;then
            hexdump -v -e '1 1 "%02X"' < $1
         else
            Error_Hexit '(No message)'
         fi
        }
# The error handling function:
# Enable with: trap Error_Handler ERR
# Disable with: trap '' ERR
function Error_Handler
        {
         local EXITCODE=$?
         trap '' ERR # switch off error handling to prevent wild recursion.
         local ARRAY=( `caller 0` )
         Error_Create_Init_File
         # Write the error message (from STDERR) and read the generated autonumber:
         local INSERT_ID=`$SQLITE3_EXECUTABLE -batch -init $ERROR_INIT_SQLITE $ERRORDB "INSERT INTO ErrorLog(strMessage) VALUES(CAST(x'$(Error_Hexit_File $ERROROUTPUT)' AS TEXT));SELECT last_insert_rowid();"`
         # Write the stack trace:
         local STACKLEVEL=0
         local STACK_ENTRY=`caller $STACKLEVEL`
         until [ -z "$STACK_ENTRY" ];do
               local STACK_ARRAY=( $STACK_ENTRY )
               $SQLITE3_EXECUTABLE -batch -init $ERROR_INIT_SQLITE $ERRORDB "INSERT INTO ErrorStackTrace(intErrorLogId,strSourceFile,strFunction,intLine) VALUES($INSERT_ID, CAST(x'$(Error_Hexit ${STACK_ARRAY[2]})' AS TEXT), '${STACK_ARRAY[1]}', ${STACK_ARRAY[0]})"
               let STACKLEVEL+=1
               STACK_ENTRY=`caller $STACKLEVEL`
         done
         # Write the error environment:
         for VAR in EXITCODE BASH_COMMAND BASH_LINENO BASH_ARGV $ERROR_ENVIRONMENT_VARIABLES ;do
             local CONTENT=$(Error_Hexit "${!VAR}")
             $SQLITE3_EXECUTABLE -batch -init $ERROR_INIT_SQLITE $ERRORDB "INSERT INTO ErrorEnvironment(intErrorLogId,strVariable,strValue) VALUES($INSERT_ID, '$VAR', CAST(x'$CONTENT' AS TEXT));"
         done
         # Clean up and provide feedback:
         if [ -e $ERROROUTPUT ] ;then
            cat $ERROROUTPUT 1>&2
         fi
         rm -f $ERROR_INIT_SQLITE
         rm -f $ERROROUTPUT
         if [ -n "$ERROR_CLEANUP_ACTION" ] ;then
            $ERROR_CLEANUP_ACTION
         fi
         exit $EXITCODE
        }

Usage example

#!/bin/bash

# Find out where I am to load the library from the same directory:
HERE=`dirname $0`

# Error handling settings:
ERRORDB=~/test.sqlite
ERROR_ENVIRONMENT_VARIABLES='USER TERM PATH HOSTNAME LANG DISPLAY NOTEXIST'
ERROR_CLEANUP_ACTION="echo I'm cleaning up!"

source $HERE/liberrorhandler.bash
trap Error_Handler ERR
# The above trap statement will do nothing in this example,
# unless you comment out the other trap statement.

function InnerFunction
        {
         trap Error_Handler ERR
         # The above trap statement will cause the error handler to be called.
         cat $1 2> $ERROROUTPUT
         # This will fail, because the file passed in $1 does not exist.
        }

function OuterFunction
        {
         InnerFunction /doesnot.exist
        }

OuterFunction 2>$ERROROUTPUT

References

Advanced Bash Scripting Guide: http://www.tldp.org/LDP/abs/html/index.html
trap: http://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html#index-trap
caller: http://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html#index-caller
SQLite3 SQL Syntax: http://sqlite.org/lang.html
SQLite3 Pragma commands: http://sqlite.org/pragma.html
SQLite3 command line client: http://www.linuxcommand.org/man_pages/sqlite31.html
SQLite Manager (Firefox plugin): http://sqlite-manager.googlecode.com/