Sunday, October 19, 2014

How to create and use Python CGI scripts

http://xmodulo.com/create-use-python-cgi-scripts.html

Have you ever wanted to create a webpage or process user input from a web-based form using Python? These tasks can be accomplished through the use of Python CGI (Common Gateway Interface) scripts with an Apache web server. CGI scripts are called by a web server when a user requests a particular URL or interacts with the webpage (such as clicking a "Submit" button). After the CGI script is called and finishes executing, the output is used by the web server to create a webpage displayed to the user.

Configuring the Apache web server to run CGI scripts

In this tutorial we assume that an Apache web server is already set up and running. This tutorial uses an Apache web server (version 2.2.15 on CentOS release 6.5) that is hosted at the localhost (127.0.0.1) and is listening on port 80, as specified by the following Apache directives:
1
2
ServerName 127.0.0.1:80
Listen 80
HTML files used in the upcoming examples are located in /var/www/html on the web server. This is specified via the DocumentRoot directive (specifies the directory that webpages are located in):
1
DocumentRoot "/var/www/html"
Consider a request for the URL: http://localhost/page1.html
This will return the contents of the following file on the web server:
/var/www/html/page1.html
To enable use of CGI scripts, we must specify where CGI scripts are located on the web server. To do this, we use the ScriptAlias directive:
1
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
The above directive indicates that CGI scripts are contained in the /var/www/cgi-bin directory on the web server and that inclusion of /cgi-bin/ in the requested URL will search this directory for the CGI script of interest.
We must also explicitly permit the execution of CGI scripts in the /var/www/cgi-bin directory and specify the file extensions of CGI scripts. To do this, we use the following directives:
1
2
3
4
    Options +ExecCGI
    AddHandler cgi-script .py
Consider a request for the URL: http://localhost/cgi-bin/myscript-1.py
This will call the following script on the web server:
/var/www/cgi-bin/myscript-1.py

Creating a CGI script

Before creating a Python CGI script, you will need to confirm that you have Python installed (this is generally installed by default, however the installed version may vary). Scripts in this tutorial are created using Python version 2.6.6. You can check your version of Python from the command line by entering either of the following commands (the -V and --version options display the version of Python that is installed):
$ python -V
$ python --version
If your Python CGI script will be used to process user-entered data (from a web-based input form), then you will need to import the Python cgi module. This module provides functionality for accessing data that users have entered into web-based input forms. You can import this module via the following statement in your script:
1
import cgi
You must also change the execute permissions for the Python CGI script so that it can be called by the web server. Add execute permissions for others via the following command:
# chmod o+x myscript-1.py

Python CGI Examples

Two scenarios involving Python CGI scripts will be considered in this tutorial:
  1. Create a webpage using a Python script
  2. Read and display user-entered data and display results in a webpage
Note that the Python cgi module is required for Scenario 2 because this involves accessing user-entered data from web-based input forms.

Example 1: Create a webpage using a Python script

For this scenario, we will start by creating a webpage /var/www/html/page1.html with a single submit button:
1
2
3
4
5
6
<html>
<h1>Test Page 1</h1>
<form name="input" action="/cgi-bin/myscript-1.py" method="get">
<input type="submit" value="Submit">
</form>
</html>
When the "Submit" button is clicked, the /var/www/cgi-bin/myscript-1.py script is called (specified by the action parameter). A "GET" request is specified by setting the method parameter equal to "get". This requests that the web server return the specified webpage. An image of /var/www/html/page1.html as viewed from within a web browser is shown below:

The contents of /var/www/cgi-bin/myscript-1.py are:
1
2
3
4
5
6
7
#!/usr/bin/python
print "Content-Type: text/html"
print ""
print ""
print "

CGI Script Output

"
print "This page was generated by a Python CGI script.
"
print "
" The first statement indicates that this is a Python script to be run with the /usr/bin/python command. The print "Content-Type: text/html" statement is required so that the web server knows what type of output it is receiving from the CGI script. The remaining statements are used to print the text of the webpage in HTML format.
When the "Submit" button is clicked in the above webpage, the following webpage is returned:

The take-home point with this example is that you have the freedom to decide what information is returned by the CGI script. This could include the contents of log files, a list of users currently logged on, or today's date. The possibilities are endless given that you have the entire Python library at your disposal.

Example 2: Read and display user-entered data and display results in a webpage

For this scenario, we will start by creating a webpage /var/www/html/page2.html with three input fields and a submit button:
1
2
3
4
5
6
7
8
9
<html>
<h1>Test Page 2</h1>
<form name="input" action="/cgi-bin/myscript-2.py" method="get">
First Name: <input type="text" name="firstName"><br>
Last Name: <input type="text" name="lastName"><br>
Position: <input type="text" name="position"><br>
<input type="submit" value="Submit">
</form>
</html>
When the "Submit" button is clicked, the /var/www/cgi-bin/myscript-2.py script is called (specified by the action parameter). An image of /var/www/html/page2.html as viewed from within a web browser is shown below (note that the three input fields have already been filled in):

The contents of /var/www/cgi-bin/myscript-2.py are:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/usr/bin/python
import cgi
form = cgi.FieldStorage()
print "Content-Type: text/html"
print ""
print ""
print "

CGI Script Output

"
print ""
print "The user entered data are:
"
print "First Name: " + form["firstName"].value + "
"
print "Last Name: " + form["lastName"].value + "
"
print "Position: " + form["position"].value + "
"
print "
"
print "
" As mentioned previously, the import cgi statement is needed to enable functionality for accessing user-entered data from web-based input forms. The web-based input form is encapsulated in the form object, which is a cgi.FieldStorage object. Once again, the "Content-Type: text/html" line is required so that the web server knows what type of output it is receiving from the CGI script. The data entered by the user are accessed in the statements that contain form["firstName"].value, form["lastName"].value, and form["position"].value. The names in the square brackets correspond to the values of the name parameters defined in the text input fields in /var/www/html/page2.html.
When the "Submit" button is clicked in the above webpage, the following webpage is returned:

The take-home point with this example is that you can easily read and display user-entered data from web-based input forms. In addition to processing data as strings, you can also use Python to convert user-entered data to numbers that can be used in numerical calculations.

Summary

This tutorial demonstrates how Python CGI scripts are useful for creating webpages and for processing user-entered data from web-based input forms. More information about Apache CGI scripts can be found here and more information about the Python cgi module can be found here.

How to verify the authenticity and integrity of a downloaded file on Linux

http://xmodulo.com/verify-authenticity-integrity-downloaded-file.html

When you download a file (e.g., an installer, an ISO image, or a compressed archive) from the web, the file can be corrupted under a variety of error conditions, e.g., due to transmission errors on the wire, interrupted download, faulty storage hardware, file system errors, etc. Such failure cases aside, a file can also be deliberately tampered with by determined attackers during or before download. For example, an attacker with a compromised certificate authority could mount a man-in-the-middle (MITM) attack, tricking you into downloading a malware-ridden file from a bogus HTTPS website.
To protect yourself against these kinds of problems, it is often recommended that you verify the authenticity and integrity of a file when you download it from the web. Especially when you downloaded rather sensitive files (e.g., OS images, application binaries, executable installers, etc), blindly trusting downloaded files is not a good habit.
One quick and easy way to verify the integrity of a downloaded file is to use various checksum tools (e.g., md5sum, sha256sum, cksum) to compute and compare checksums (e.g., MD5, SHA or CRC). However, checksums are vulnerable to collision attacks, and also cannot be used to verify the authenticity (i.e., owner) of a file.
If you would like to verify both authenticity (owner) and integrity (content) of a downloaded file, you need to rely on cryptographic signatures instead. In this tutorial, I am going to describe how to check file authenticity and integrity by using GnuPG (GNU Privacy Guard).
In this example, I am going to verify a disk image file available for download from https://onionshare.org. In this website, the publisher offers their official public key, as well as its fingerprint for key verification purpose.

As for a file to download, the publisher offers its corresponding PGP signature as well.

Install GnuPG and Generate a Key Pair

Let's start by installing GnuPG on your Linux system.
On Debian, Ubuntu, and other Debian-derivatives:
$ sudo apt-get install gnupg
On Fedora, CentOS or RHEL:
$ sudo yum install gnupg
After installation, generate a key pair which you will be using in this tutorial.
$ gpg --gen-key

During key generation, you will be asked to provide your name and email address, as well as a passphrase to protect your private key. You can also choose when the key pair will expire (no expiration by default). Depending on keysize you choose (between 1024 to 4096 bits), the key generation process can take a couple of minutes or more, as it requires collecting a sufficient amount of random data, which come from your desktop activities (e.g., keyboard typing, mouse movement, disk access).
After key generation is finished, a public and a private key will be stored in ~/.gnupg directory for use.

Establish Trust with a File Owner

The first step in verifying a downloaded file is to establish trust with whoever is offering the file for download. For this purpose, we download the public key of a file owner, and verify that the owner of the public key is who he or she claims to be.
After downloading the public key of a file owner:
$ wget https://onionshare.org/signing-key.asc
go ahead and import the public key into your keyring with gpg command:
$ gpg --import signing-key.asc

Once the public key of the owner is imported, it will print out a key ID (e.g., "EBA34B1C") as shown above. Make a note of this key ID.
Now, check the fingerprint of the imported public key by running:
$ gpg --fingerprint EBA34B1C

You will see the fingerprint string of the key. Compare this string with the fingerprint displayed in the website. If they match, you may choose to trust the file owner's public key.
Once you decided to trust the public key, you can mark that so explicitly, by editing the key:
$ gpg --edit-key EBA34B1C
This command will show you GPG prompt:

Type "trust" at GPG prompt, which will allow you to choose the trust level of this key from 1 to 5.

In this case, I decided to assign trust "4". After that, sign it with your own private key by typing "sign", and then finalize by typing "save" at GPG prompt:

Note that this way of explicitly assigning a trust to a public key is not required, and implicit trust by simply importing the key is often sufficient.
The implication of assigning a "full" trust to the key is that if another key X is signed with this fully trusted key, the key X will be also considered valid by you. In general, key validation relies on a sophisticated mechanism known as "web of trust".
Coming back to the tutorial, now let's check a list of imported keys.
$ gpg --list-keys

You should see at least two keys: one key with depth 0 and ultimate trust ("1u"), which is your own key, and the other key with depth 1 and full trust ("1f"), which is the key signed by yourself earlier.

Verify the Authenticity/Integrity of a File

Once you have established a trust relationship with a file owner using his/her public key, we are now ready to verify the authenticity and integrity of a file that you downloaded from the owner.
In our example, the file owner publishes a file and a corresponding PGP signature (*.asc) separately. The role of the signature is to certify and put a timestamp on the file.
A typical signature (*.asc) looks like the following.
-----BEGIN PGP SIGNATURE-----

iQIcBAABCgAGBQJUJGhsAAoJEP1yCtnro0sc1jUP/ixNY/lKdrcMIAUoqlWKNE8f
sj4SFiwREMew76w66GASDF03fa5zPX6EsS2kucgx8ZsfEiSmN5T0y2P/aSaXwZqF
kywZVEzirKtca5AJ4DBzu6qrt9GgSw6JBJVv1oBJCMNyO+eAj341paR3MudvnyQz
H/N5tc4Qcilzy6M184opGIzy4ipEmMXfLHsd7WJpAyn+tO/z3uhh9NkNuygZpaFr
olpSWPE8revdDJyfMfSmb3ZrFmhLn7FCEltOi+a7SluvrMclizfnbec9rgLJtjo0
CPDZY7tsWmmL0DA3VvpMVqGvkg/Dyhpn2IIDrNaLAlvGQ5aovf+4tjad5IHvyaWx
4Gds93G6Hqvv5RwGx7OR3hgt2o0Y+qFsVDxVnPxerGhXeJXHzSDwLQMpdj9IoSU
Ae/53XXnxqSN6POZcwHiHvbsv0pdlg0Ea0dDAAN0ZeINNyZf1R0tLjWkcgpvGCtv
qkJuYFF9W9cWHraPY2ov5Hs/JZzPcG0eVpnDdzfOOH1gDKADq9A5D2X5QJCulsh9
WwU3X+E43OqIsoRzBucItD9HhZbEH7t8Q0xAqnAkgU3hriZp3dN4cnMfhM6I9hli
EmpSpLKCceMexu2o9QgzGXVm+AGZJe4QkuwAhRIccp5JDMVny61UlKTasjy6co8h
5GBhhYybPEFM+G1BODMd
=c9wo
-----END PGP SIGNATURE-----
Let's download both the file and its signature:
$ wget https://onionshare.org/files/0.6/OnionShare.dmg
$ wget https://onionshare.org/files/0.6/OnionShare.dmg.asc
Now verify the PGP signature of the downloaded file.
$ gpg --verify OnionShare.dmg.asc OnionShare.dmg

If the output of the command contains "Good signature from ", the downloaded .dmg file has been successfully authenticated and verified. If the downloaded file were tampered with in any way after the signature has been generated, the verification would fail.
At this point you can be rest assured and trust the downloaded file.

Thursday, October 16, 2014

What is good reference management software on Linux

http://xmodulo.com/reference-management-software-linux.html

Have you ever written a paper so long that you thought you would never see the end of it? If so, you know that the worst part is not dedicating hours on it, but rather that once you are done, you still have to order and format your references into a structured convention-following bibliography. Hopefully for you, Linux has the solution: bibliography/reference management tools. Using the power of BibTex, these programs can help you import your citation sources, and spit out a structured bibliography. Here is a non-exhaustive list of open-source reference management software on Linux.

1. Zotero


Surely the most famous tool for collecting references, Zotero is known for being a browser extension. However, there also exists a convenient Linux stand alone program. Among its biggest advantages, Zotero is easy to use, and can be coupled with LibreOffice or other text editors to manage the bibliography of documents. I personally appreciate the interface and the plugin manager. However, Zotero is quickly limited if you have a lot of needs about your bibliography.

2. JabRef


JabRef is one of the most advanced tools out there for citation management. You can import from a plethora of format, lookup entries from external databases (like Google Scholar), and export straight to your favorite editor. JabRef integrates your environment nicely, and can even support plugins. And as a final touch, JabRef can connect to your own SQL database. The only downside to all of this is of course the learning curve.

3. KBibTex


For KDE adepts, the desktop environment has its own dedicated bibliography manager called KBibTex. And as you might expect from a program of this caliber, the promised quality is delivered. The software is highly customizable, from the shortcuts to the behavior and appearance. It is easy to find duplicates, to preview the results, and to export directly to a LaTeX editor. But the best feature in my opinion is the integration of Bibsonomy, Google Scholar, and even your Zotero account. The only downside is that the interface seems a bit cluttered at first. Hopefully spending enough time in the settings should fix that.

4. Bibfilex


Capable of running in both Gtk and Qt environment, Bibfilex is a user friendly bibliography management tool based on Biblatex. Less advanced than JabRef or KBibTex, it is fast and lightweight. Definitely a smart choice for making a bibliography quickly without thinking too much. The interface is slick and reflects just the necessary functions. I give it extra credits for the complete manual that you can get from the official download page

5. Pybliographer


As indicated by its name, Pybliographer is a non-graphical tool for bibliography management written in Python. I personally like to use Pybliographic as the graphical front-end. The interface is extremely clear and minimalist. If you just have a few references to export and don't really have time to learn how to use an extensive piece of software, Pybliographer is the place to go. A bit like Bibfilex, the intent is on user-friendliness and quick use.

6. Referencer


Probably my biggest surprise when doing this list, Referencer is really appealing to the eye. Capable of integrating itself perfectly with Gnome, it can find and import your documents, look up their reference on the web, and export to LyX, while being sexy and really well designed. The few shortcuts and plugins are a good bonus along with the library style interface.
To conclude, thanks to these tools, you will not have to worry about long papers anymore, or at least not about the reference section. What did we miss? Is there a bibliography management tool that you prefer? Let us know in the comments.