How to remove trailing whitespaces in a file on Linux

http://ask.xmodulo.com/remove-trailing-whitespaces-linux.html

Question: I have a text file in which I need to remove all trailing whitespsaces (e.g., spaces and tabs) in each line for formatting purpose. Is there a quick and easy Linux command line tool I can use for this?

When you are writing code for your program, you must understand that there are standard coding styles to follow. For example, "trailing whitespaces" are typically considered evil because when they get into a code repository for revision control, they can cause a lot of problems and confusion (e.g., "false diffs"). Many IDEs and text editors are capable of highlighting and automatically trimming trailing whitepsaces at the end of each line.
Here are a few ways to remove trailing whitespaces in Linux command-line environment.

Method One

A simple command line approach to remove unwanted whitespaces is via sed.
The following command deletes all spaces and tabs at the end of each line in input.java.

$ sed -i 's/[[:space:]]*$//' input.java

If there are multiple files that need trailing whitespaces removed, you can use a combination of find and sed. For example, the following command deletes trailing whitespaces in all *.java files recursively found in the current directory as well as all its sub-directories.

$ find . -name "*.java" -type f -print0 | xargs -0 sed -i 's/[[:space:]]*$//'

Method Two

Vim text editor is able to highlight and trim whitespaces in a file as well.
To highlight all trailing whitespaces in a file, open the file with Vim editor and enable text highlighting by typing the following in Vim command line mode.

:set hlsearch

Then search for trailing whitespaces by typing:

/\s\+$

This will show all trailing spaces and tabs found throughout the file.

Then to clean up all trailing whitespaces in a file with Vim, type the following Vim command.

:%s/\s\+$//

This command means substituting all whitespace characters found at the end of the line (\s\+$) with no character.

HowTo: Linux Check Password Strength With Cracklib-check Command

http://www.cyberciti.biz/security/linux-password-strength-checker

Using the same password on different servers allows attackers to access your accounts if cracker manage to steal your password from a less secure server. This is true for online website accounts too. So solution is to create unique passwords for server accounts like your email, sftp and ssh accounts. General guideline to create a strong and unique password is as follows:

Creating a strong and unique password for Linux or Unix-like systems

1. Create a password with mix of numbers, special symbols, and alphabets.
2. Make sure your password is hard to guess. You can use tool such as makepasswd to create hard to guess password.
3. Do not use simple words like "password", "123456", "123abc" or "qwerty".
4. Use a unique password for all your server accounts.
5. A minimum password length of 12 to 14 characters should be used. See how to configure CentOS / RHEL / Fedora Linux based server password quality requirements.
6. Generating passwords randomly where feasible. You can do this with a simple shell script function.
7. If possible use two-factor authentication.
8. Use pam_crack to ensure strong passwords and to check passwords against a dictionary attack.

But, how do you test the effectiveness of a password in resisting guessing and brute-force attacks under Linux? The answer is simple use cracklib-check command.

Install cracklib on a Linux based system

Type the following yum command to install on RHEL and friends:
# yum install cracklib
Type the following apt-get command to install on Debian/Ubuntu and friends:
# apt-get install libcrack2

Say hello to cracklib-check

This command takes a list of passwords from keyboard (stdin) and checks them using libcrack2. The idea is simple: try to prevent users from choosing passwords that could be guessed by "crack" by filtering them out, at source.

Examples

Test a simple password like "password", enter:
$ echo "password" | cracklib-check
Sample outputs:

password: it is based on a dictionary word

Try sequential patterns such as "abc123456":
$ echo "abc123456" | cracklib-check
Sample outputs:

abc123456: it is too simplistic/systematic

Try a password with a mix of letters, numbers, and symbols:
$ echo 'i1oVe|DiZza' | cracklib-check
Sample outputs:

i1oVe|DiZza: OK

The above password increases the difficulty of guessing or cracking your password. I used a random phrase (easy to remember) "I Love Pizza" and inserted random characters to create a strong hard to guess password - "i1oVe|DiZza".

Fig.01: Linux cracklib-check command examples

Putting it all together

 
#!/bin/bash
# A sample shell script to add user to the system
# Check password for strength 
# Written by Vivek Gite under GPL v2.x+
# ----------------------------------------------
read -p "Enter username : " user
read -sp "Enter password : " password
echo
echo "Tesing password strength..."
echo
result="$(cracklib-check <<<"$password")"
# okay awk is  bad choice but this is a demo 
okay="$(awk -F': ' '{ print $2}' <<<"$result")"
if [[ "$okay" == "OK" ]]
then
 echo "Adding a user account please wait..."
 /sbin/useradd -m -s /bin/bash $user
 echo "$user:$password" | /sbin/chpasswd
else
 echo "Your password was rejected - $result"
        echo "Try again."
fi
 

A note about password manager

A reasonable compromise for using large numbers of passwords is to record them in a password manager, which include stand-alone applications, web browser extensions, or a manager built into the operating system. See how to install gpass - an easy to use and secure password manager for GNOME2 under RHEL / CentOS / Fedora Linux desktop. gpass stores all your password in an encrypted (Blowfish) file, protected by a master-password.

Check out related media

(Video:01 - How to create a strong password)

Recommended readings:

• OpenSSH server best security practices
• HowTo set password quality requirements under Linux.
• Linux upgrade password hashing algorithm to SHA-512.
• man pages - cracklib-check, useradd, chpasswd, and passwd commands.

Regular Expressions In grep

http://www.cyberciti.biz/faq/grep-regular-expressions

How do I use the Grep command with regular expressions on a Linux and Unix-like operating systems?

Linux comes with GNU grep, which supports extended regular expressions. GNU grep is the default on all Linux systems. The grep command is used to locate information stored anywhere on your server or workstation.

Regular Expressions

Regular Expressions is nothing but a pattern to match for each input line. A pattern is a sequence of characters. Following all are examples of pattern:
^w1
w1|w2
[^ ]

grep Regular Expressions Examples

Search for 'vivek' in /etc/passswd
grep vivek /etc/passwd
Sample outputs:

vivek:x:1000:1000:Vivek Gite,,,:/home/vivek:/bin/bash
vivekgite:x:1001:1001::/home/vivekgite:/bin/sh
gitevivek:x:1002:1002::/home/gitevivek:/bin/sh

Search vivek in any case (i.e. case insensitive search)
grep -i -w vivek /etc/passwd
Search vivek or raj in any case
grep -E -i -w 'vivek|raj' /etc/passwd
The PATTERN in last example, used as an extended regular expression.

Anchors

You can use ^ and $ to force a regex to match only at the start or end of a line, respectively. The following example displays lines starting with the vivek only:
grep ^vivek /etc/passwd
Sample outputs:

vivek:x:1000:1000:Vivek Gite,,,:/home/vivek:/bin/bash
vivekgite:x:1001:1001::/home/vivekgite:/bin/sh

You can display only lines starting with the word vivek only i.e. do not display vivekgite, vivekg etc:
grep -w ^vivek /etc/passwd
Find lines ending with word foo:
grep 'foo$' filename
Match line only containing foo:
grep '^foo$' filename
You can search for blank lines with the following examples:
grep '^$' filename

Character Class

Match Vivek or vivek:
grep '[vV]ivek' filename
OR
grep '[vV][iI][Vv][Ee][kK]' filename
You can also match digits (i.e match vivek1 or Vivek2 etc):
grep -w '[vV]ivek[0-9]' filename
You can match two numeric digits (i.e. match foo11, foo12 etc):
grep 'foo[0-9][0-9]' filename
You are not limited to digits, you can match at least one letter:
grep '[A-Za-z]' filename
Display all the lines containing either a "w" or "n" character:
grep [wn] filename
Within a bracket expression, the name of a character class enclosed in "[:" and ":]" stands for the list of all characters belonging to that class. Standard character class names are:

• [:alnum:] - Alphanumeric characters.
• [:alpha:] - Alphabetic characters
• [:blank:] - Blank characters: space and tab.
• [:digit:] - Digits: '0 1 2 3 4 5 6 7 8 9'.
• [:lower:] - Lower-case letters: 'a b c d e f g h i j k l m n o p q r s t u v w x y z'.
• [:space:] - Space characters: tab, newline, vertical tab, form feed, carriage return, and space.
• [:upper:] - Upper-case letters: 'A B C D E F G H I J K L M N O P Q R S T U V W X Y Z'.

In this example match all upper case letters:
grep '[:upper:]' filename

Wildcards

You can use the "." for a single character match. In this example match all 3 character word starting with "b" and ending in "t":

grep '\<b.t\>' filename

Where,

• \< Match the empty string at the beginning of word
• \> Match the empty string at the end of word.

Print all lines with exactly two characters:
grep '^..$' filename
Display any lines starting with a dot and digit:
grep '^\.[0-9]' filename

Escaping the dot

The following regex to find an IP address 192.168.1.254 will not work:
grep '192.168.1.254' /etc/hosts
All three dots need to be escaped:
grep '192\.168\.1\.254' /etc/hosts
The following example will only match an IP address:

egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' filename

The following will match word Linux or UNIX in any case:
egrep -i '^(linux|unix)' filename

How Do I Search a Pattern Which Has a Leading - Symbol?

Searches for all lines matching '--test--' using -e option Without -e, grep would attempt to parse '--test--' as a list of options:
grep -e '--test--' filename

How Do I do OR with grep?

Use the following syntax:
grep 'word1|word2' filename
OR
grep 'word1\|word2' filename

How Do I do AND with grep?

Use the following syntax to display all lines that contain both 'word1' and 'word2'
grep 'word1' filename | grep 'word2'

How Do I Test Sequence?

You can test how often a character must be repeated in sequence using the following syntax:

{N}
{N,}
{min,max}

Match a character "v" two times:
egrep "v{2}" filename
The following will match both "col" and "cool":
egrep 'co{1,2}l' filename
The following will match any row of at least three letters 'c'.
egrep 'c{3,}' filename
The following example will match mobile number which is in the following format 91-1234567890 (i.e twodigit-tendigit)

grep "[[:digit:]]\{2\}[ -]\?[[:digit:]]\{10\}" filename

How Do I Hightlight with grep?

Use the following syntax:
grep --color regex filename

How Do I Show Only The Matches, Not The Lines?

Use the following syntax:
grep -o regex filename

Regular Expression Operator

Regex operator	Meaning
.	Matches any single character.
?	The preceding item is optional and will be matched, at most, once.
*	The preceding item will be matched zero or more times.
+	The preceding item will be matched one or more times.
{N}	The preceding item is matched exactly N times.
{N,}	The preceding item is matched N or more times.
{N,M}	The preceding item is matched at least N times, but not more than M times.
-	Represents the range if it's not first or last in a list or the ending point of a range in a list.
^	Matches the empty string at the beginning of a line; also represents the characters not in the range of a list.
$	Matches the empty string at the end of a line.
\b	Matches the empty string at the edge of a word.
\B	Matches the empty string provided it's not at the edge of a word.
\<	Match the empty string at the beginning of word.
\>	Match the empty string at the end of word.

grep vs egrep

egrep is the same as grep -E. It interpret PATTERN as an extended regular expression. From the grep man page:

       In basic regular expressions the meta-characters ?, +, {, |, (, and ) lose their special meaning; instead use the backslashed versions \?, \+, \{,
       \|, \(, and \).
       Traditional egrep did not support the { meta-character, and some egrep implementations support \{ instead, so portable scripts should avoid  {  in
       grep -E patterns and should use [{] to match a literal {.
       GNU grep -E attempts to support traditional usage by assuming that { is not special if it would be the start of an invalid interval specification.
       For example, the command grep -E '{1' searches for the two-character string {1 instead of reporting a syntax  error  in  the  regular  expression.
       POSIX.2 allows this behavior as an extension, but portable scripts should avoid it.

References:

• man page grep and regex(7)
• info page grep

Debian / Ubuntu Linux Delete Old Kernel Images Command

http://www.cyberciti.biz/faq/debian-ubuntu-linux-delete-old-kernel-images-command

I'm a new Ubuntu Linux user and noticed that old kernel still exists in my system. Why doesn't Ubuntu remove old kernels automatically? How do I delete old unused kernel images to free disk space. How to remove unused old kernel images on Ubuntu Linux safely?

You need to delete and/or remove old kernels from system manually. Ubuntu and Debian based system keeps old kernel images so that system can be booted if newer kernel failed. The safest way to purge and remove old kernel is as follows. In this tutorial you will learn how to delete unused old kernel images on Ubuntu or Debian Linux to free disk space as well as the various state of linux-image package.

Step #1: Boot into new kernel

First, boot into newly installed kernel. Verify this with the following command:
$ uname -mrs
$ uname -a
Sample outputs:

Linux server1 3.13.0-68-generic #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

To list all installed Linux kernel images, enter:
# dpkg --list | egrep -i --color 'linux-image|linux-headers'
Sample outputs:

Fig.01: Check what kernel image(s) are installed on your system (click to enlarge)

Step #2: Delete unwanted and unused kernel images

You can remove kernel images one by one using the following syntax:
# apt-get --purge remove linux-image-3.13.0-67-generic
OR
$ sudo apt-get --purge remove linux-image-3.13.0-67-generic

A note about newer Ubuntu and Debian system

On newer system all obsolete kernels and headers should automatically be flagged as no more needed, and thus can be purged with the following single command:
$ sudo apt-get autoremove

Understanding package states in Ubuntu and Debian Linux

Consider the following example:
# dpkg --list | grep linux-image
Sample outputs:

rc  linux-image-3.13.0-62-generic        3.13.0-62.102                         amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-63-generic        3.13.0-63.103                         amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-65-generic        3.13.0-65.106                         amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-66-generic        3.13.0-66.108                         amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-67-generic        3.13.0-67.110                         amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
ii  linux-image-3.13.0-68-generic        3.13.0-68.111                         amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-62-generic  3.13.0-62.102                         amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-63-generic  3.13.0-63.103                         amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-65-generic  3.13.0-65.106                         amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-66-generic  3.13.0-66.108                         amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-67-generic  3.13.0-67.110                         amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
ii  linux-image-extra-3.13.0-68-generic  3.13.0-68.111                         amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
ii  linux-image-generic                  3.13.0.68.74                          amd64        Generic Linux kernel image

The first column indicates package flags like rc, ii. So, what do the various dpkg flags like 'ii' 'rc' mean?

• rc: It means package is in remove/deinstall state and only config file exists.
• ii: It means package is in install state and it is 100% installed on the system.

You can remove all linux-image packages in rc state using the following command:
# x=$(dpkg --list | grep -i linux-image | grep ^rc| awk '{ print $2}')
# echo "$x"
# apt-get --purge remove $x
Sample outputs:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  linux-image-3.13.0-62-generic* linux-image-3.13.0-63-generic*
  linux-image-3.13.0-65-generic* linux-image-3.13.0-66-generic*
  linux-image-3.13.0-67-generic* linux-image-extra-3.13.0-62-generic*
  linux-image-extra-3.13.0-63-generic* linux-image-extra-3.13.0-65-generic*
  linux-image-extra-3.13.0-66-generic* linux-image-extra-3.13.0-67-generic*
0 upgraded, 0 newly installed, 10 to remove and 0 not upgraded.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 65623 files and directories currently installed.)
Removing linux-image-3.13.0-62-generic (3.13.0-62.102) ...
Purging configuration files for linux-image-3.13.0-62-generic (3.13.0-62.102) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-62-generic /boot/vmlinuz-3.13.0-62-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-62-generic /boot/vmlinuz-3.13.0-62-generic
Removing linux-image-3.13.0-63-generic (3.13.0-63.103) ...
Purging configuration files for linux-image-3.13.0-63-generic (3.13.0-63.103) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-63-generic /boot/vmlinuz-3.13.0-63-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-63-generic /boot/vmlinuz-3.13.0-63-generic
Removing linux-image-3.13.0-65-generic (3.13.0-65.106) ...
Purging configuration files for linux-image-3.13.0-65-generic (3.13.0-65.106) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-65-generic /boot/vmlinuz-3.13.0-65-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-65-generic /boot/vmlinuz-3.13.0-65-generic
Removing linux-image-3.13.0-66-generic (3.13.0-66.108) ...
Purging configuration files for linux-image-3.13.0-66-generic (3.13.0-66.108) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-66-generic /boot/vmlinuz-3.13.0-66-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-66-generic /boot/vmlinuz-3.13.0-66-generic
Removing linux-image-3.13.0-67-generic (3.13.0-67.110) ...
Purging configuration files for linux-image-3.13.0-67-generic (3.13.0-67.110) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-67-generic /boot/vmlinuz-3.13.0-67-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-67-generic /boot/vmlinuz-3.13.0-67-generic
Removing linux-image-extra-3.13.0-62-generic (3.13.0-62.102) ...
Purging configuration files for linux-image-extra-3.13.0-62-generic (3.13.0-62.102) ...
Removing linux-image-extra-3.13.0-63-generic (3.13.0-63.103) ...
Purging configuration files for linux-image-extra-3.13.0-63-generic (3.13.0-63.103) ...
Removing linux-image-extra-3.13.0-65-generic (3.13.0-65.106) ...
Purging configuration files for linux-image-extra-3.13.0-65-generic (3.13.0-65.106) ...
Removing linux-image-extra-3.13.0-66-generic (3.13.0-66.108) ...
Purging configuration files for linux-image-extra-3.13.0-66-generic (3.13.0-66.108) ...
Removing linux-image-extra-3.13.0-67-generic (3.13.0-67.110) ...
Purging configuration files for linux-image-extra-3.13.0-67-generic (3.13.0-67.110) ...

Type the following command again to see the results:
# dpkg --list | egrep -i --color 'linux-image|linux-headers'
Sample outputs:

ii  linux-headers-3.13.0-68              3.13.0-68.111                         all          Header files related to Linux kernel version 3.13.0
ii  linux-headers-3.13.0-68-generic      3.13.0-68.111                         amd64        Linux kernel headers for version 3.13.0 on 64 bit x86 SMP
ii  linux-headers-generic                3.13.0.68.74                          amd64        Generic Linux kernel headers
ii  linux-image-3.13.0-68-generic        3.13.0-68.111                         amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
ii  linux-image-extra-3.13.0-68-generic  3.13.0-68.111                         amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
ii  linux-image-generic                  3.13.0.68.74                          amd64        Generic Linux kernel image

Web Walker: How to Bypass Internet Surveillance or Even a Total Web Shut Down

https://proparanoid.wordpress.com/2012/08/25/web-walker-how-to-bypass-internet-surveillance-or-even-a-total-web-shut-down

A simple solution: establish a Web Walker, your own Internet emulation to totally bypass the existing system — and its snoops

By H. Michael Sweeney, Copyright REMOVED. Permissions to duplicate granted provided it is reproduced in full with all links in tact, and referenced to the original article here at proparanoid.wordpress.com.

Web Walker: How to Bypass Internet Surveillance or Defeat a Total Web Shut Down

Updated Nov 26, 2012: Removed copyright and some text changes which do not impact conceptually, other than it is in preparation for creating an Open Source project to make a real operational Web Walker network available World wide. Additions introduced with RED TEXT.

Web Walker: How to Bypass Internet Surveillance or Defeat a Total Web Shut Down

This is  a serious discussion of modest technical complexity which will take about fifteen minutes to read. If you bore easily with technospeak and are just curious, don’t bother yourself. Read this short, fun piece, instead, as it has more entertainment value. And then come back when you are more serious about preserving your freedoms in the face of the NWO.

The FBI, CIA, DIA, DHS, NSA, and dozens of other agencies including the Federal Reserve spy on internet users, and it is going to get much worse, SOON — perhaps to include martial law and suspension of the Constitution in the major next terrorism wave.

For example, CISPA and other legislation continues to threaten to remake the Internet into a completely transparent spying mechanism, or worse, it seems logical to expect government to shut these systems down altogether in a Martial Law or other declared emergency, or if there is a 99% Occupy on Steroids event, or worse. Since the attacks of Sept. 11, the government has created more than 260 new Agencies, the bulk of which are geared up for or in support of spying on you and me, what we do, say, and think.  King George-style Sedition laws cannot be far behind, and have already been proposed by some Congressionals, though defeated if actually reaching the Floor. Currently, offering this advice is legal, but you can bet government snoops don’t want you to know about it.
Survival in the future, more than ever, will depend on access to key information, and when government actions shut down or seek to have unlimited access and control of communications… you don’t want to be just another person without a clue while serious risks to your safety unfold nearer and nearer to you; if they shut down the Web, you can bet TV, phones, and radio will also be curtailed except for ‘official’ propaganda. And when it comes back on line, it will be a totally new Internet in terms of the threat of surveillance. Read this review of what to expect under Martial Law and you will better understand the concern. I WILL NOT GO PEACFULLY  INTO THAT DARK NIGHT, for it is born of pure evil, that yet more evil may be undertaken, and I am not willing to live within an evil construct with no way out. How say you?

Hillary Clinton: “We are loosing the (dis)information war (with our citizens, thanks to alternative media outlets).”

These Hitlerite Police State tactics are being undertaken by a paranoid government in the name of combating terrorism, but the simple truth is, they do not fear terrorists, but the actual truth itself, and those who seek and share it. Terrorism is an invented enemy of political-control convenience that does less harm than unrighteous violence dished out by out-of-control Police. As clue this is true, very few of the hundreds of Federal Agencies and new weapon/surveillance systems/laws are aimed directly at Terrorists. No, the focus is ALL on citizens.
What better way to combat truth and free exchange of ideas and information than through the Internet, and by it, back doors into our computers, phones, and game systems by that same digital interface? You can no longer depend on the Military-Industrial-Intelligence-Media Complex to maintain such systems, provide you with protections of your privacy, or even uninterrupted access. Its time to become your own Island of communications safety. Become a Web Walker ‘Node’ (WW node). Not just because of privacy concerns, but also in case the government moves to shut down or replace the Web with a snoop friendly version… or even a natural disaster which disables whole portions of the current WWW.
There’s nothing particularly novel about the idea (but by all means, feel free to credit me with accolades).  It is not patentable, but even if it were, I would freely put it in the public domain. Also feel free to come up with better solutions; I claim no superior skills or knowledge which cannot be surpassed by the likes of an Anonymous hacker or a software or hardware engineer, or every day super geek. It is merely a nuts and bolts solution to insuring privacy and continuity in the face of an overbearing Big Brother mentality.
To effect solution which protects privacy, truth, and unbroken access requires three things. Some hardware you probably do not already have, but may easily obtain; some like-minded people willing to work with you within a defined geographical boundary; and a willingness to take a little more trouble with your communications than you may be currently used to dealing with, though much of that can be addressed by those who know how to create automated scripting. Web Walker: How to Bypass Internet Surveillance or Defeat a Total Web Shut Down

Web Walker Equipment:

Most of us employ a router in our home to service more than one computer, game system, or a laptop that roams throughout the house and perhaps even the yard, etc. A typical router can service four or five such devices to a distance of a couple of hundred feet. When the signal goes through walls and furnishings, the maximum distance drops notably, but some of us may have purchased booster antenna systems to compensate the signal loss. What we need to do is think bigger, and more powerful, and to establish cooperative communications ‘nodes’ to extend range beyond our individual capabilities.

As it happens, Apple Computer has our back; visit the Apple Store for more information and technical description. Their Airport Extreme router is reasonably priced (can be purchased refurbished from Apple for as little as $130, and new for about $120 more — I’ve bought used ones for as low as $50). Take care not to confuse with the Airport Express, a lesser unit.

The Extreme does not care if you are using a Macintosh or a PC, a game system, a smart phone, or anything else you might want to hook up to it, including Apple TV, a shared printer, or a hard drive for automated backup for all connected users. Unlike most systems which operate in just one frequency (2.4 Ghz), it also uses 5 GHz, where there is less signal clutter in the radio environment. That improves both range and prevents data errors that slow things down.
But what makes it especially useful is its power and capacity, and it offers a built-in range extending feature.  As it sits, it can handle up to five times the data throughput of most routers, and does so for twice the distance of the best of them (802.11a/b/g standards), and is less subject to interference from other RF sources. As an example of range, I’ve enjoyed a useful signal at nearly 500 feet away, in my car, with several homes and lots of trees and power poles between us. Best of all for our purposes, it can accommodate up to 50 users simultaneously.
And, like Macintosh computers, it has superlative security features to prevent unauthorized access, which will be critical. It is indeed extendable with the ability to add up to four ‘bridges’ (more routers) in daisy-chain fashion (or a radial layout) to essentially quadruple what is already an impressive operating envelope, or to resolve difficult signal penetration issues within a building. Bridges become range extension nodes in your WW network; a Bridge Node (B node) to allow roughly 50 more users. Lesser Apple routers also could be used, or perhaps competing units, but with reduced performance/capacity. Here is a YouTube video on setting up wireless bridging.
Web Walker: How to Bypass Internet Surveillance or Defeat a Total Web 

Booster antenna systems

When you couple any node to a booster antenna, you can enjoy phenomenal range. Apple has designed these units with an internal, concealed omnidirectional (all directions) antenna and provides software that controls the gain and other aspects which can improve nominal performance. Some Extremes feature a jack for adding an Apple made booster antenna, but it is also possible to modify any model using instructions found here, and a $20 adapting cable that works with any standard booster system. By itself, the unit has a 20 db rating, though apparently offering better performance than competitor units with similar ratings. Were you to double or triple that with a booster, you theoretically double or triple the distance. Only the surrounding or intervening environment dictates resulting performance (e.g., trees, structures, hills).
Boosters come in two flavors: omnidirectional to boost coverage area around you in all directions, and directional, to extend range through an arc of about 15-30 degrees in a given direction.  I am told you can actually use both omnidirectional and directional boosters simultaneously with a simple signal splitter. They are available for both indoor and outdoor placement (outdoor increases reach), but an outdoor unit can be concealed inside and aimed through a window or attic vent, which is highly suggested to avoid visual detection by Men in Black.
Because networks can be ‘named’ and selected by name for use, you can have any number of WW nodes, each with or without bridge nodes, each with or without their own booster for a given directional or omnidirectional need. Any one of these can be simultaneously connected to an existing ISP for ‘normal’ use, or can be operated as de facto Web Walker ‘ISP’ in a closed network, and easily be reconfigured (plug and play, or in this case, plug and share) at will.
Directional systems usually rely upon what is called a Yagi antenna design, which may or may not be enclosed in a can or tube, or small triangular housing. Where the signals going into or coming from a Yagi are additionally passed through an amplifier, phenomenal range can be achieved, though if building your own ‘black box’ solution, you must be aware of FCC guidelines and limitations on signal strength. But by way of example, an amplified Yagi system can be so powerful and precise that it is best used on a tripod with a rifle scope to center it on target destinations miles away. That is how the WW can be made to reach users well into the countryside, where line-of-sight access is available. By such a method, it would even be possible to link nearby communities in daisy-chain fashion.
Booster systems start in the 9 db range but you can find them up to 49 db, which can mean up to 1,500 feet between bridges, give or take. That’s well over a mile in total distance if employing four bridges, and about 3 million square feet of user coverage. Claims of distances for Yagi systems of up to 4,800 feet are out there, but don’t get sucked in by anyone who sells a product specifying distance — it’s a disreputable tactic that usually involves a sham. Web Walker: How to Bypass Internet Surveillance or Defeat a Total Web Shut Down
Like-minded people to connect with:
That’s the whole point, right? Such a system is not intended to replace the Internet for everyone who is currently subscribed to an ISP. That would cause bandwidth issues beyond the current state of the art’s ability to inexpensively cope. It is to allow people of common need and support to stay in truly confidential touch and securely share critical information which would otherwise be deprived of them, or obtained from them surreptitiously without permission. So you pick and choose friends, family, coworkers, compatriots, and patriots whom you think best meet your needs. That does not mean you cannot also have others join the network to enhance functionality. Here’s how you reach them…
First, map out the locations of people who should be logical participants and who are interested in joining your WW.  Some may already be ‘in range.’ For those further away, consider directional antenna or bridge solutions, or a combination. For bridges used to fill in the distance gaps with useful signals, seek out the most logical locations for placement of bridges, and see if you know someone there, or could make friends with someone in the area. Explain your goals and see where the dialog goes. Hopefully you would find them kindred Constitutionalists or Activists at heart, or otherwise willing to house your bridge unit and keep it powered up in exchange for benefits, and perhaps join your WW node or even contribute toward costs.
But bridges merely render you and up to 50 other persons within range into a relatively small private local-area WW network, and may still fall short of enough range to reach intended persons at greater distance. But there is even more we can do. Because each individual node operator/user can have multiple named routers and select which is in use, any user can establish their own WW node and serve as a relay station from one such node to another… and if needed, to another, and another, ad nauseum.
By this means, a whole city can be encompassed. Multiple routes or paths from point A to D via B and C are also advised so that system integrity is maintained if a node is lost. There are additional security-related route redundancy considerations discussed shortly.
I’d suggest an ideal WW operator’s site would look like this: three Web Walkers with directional boosters just for distance in three differing directions, and one omnidirectional for those around you, perhaps a system you make available to all neighbors who apply, over time (a move considered a security risk, at best). I think of it as being triads of node triads in a pattern which blossoms outward like some kind of Dandelion seed puff covering the whole of its core being. But there are many environmental reasons an actual geometric pattern is not likely to result.
And at the end of those routes, that is where you want your most loyal of kindred spirits, because they should do the same, and likewise at the end of their otherwise completely separate routes.  Keep in mind, too, that a bridge operator can likewise add additional units and establish their own WW node while remaining a bridge element in yours. It is completely free form to need for each individual node operator/user. Web Walker: How to Bypass Internet Surveillance or Defeat a Total Web Shut Down
Changes in how you communicate:
Each WW node operator/user would be key to facilitating communications beyond their own node’s connected users (other nodes). Everyone on their own node (also true of B nodes), which functions with the ease of a local network, would become a peer-to-peer communicant party with others on that node; it is a local Intranet. They would be able to easily send information and files to anyone else in that same network, or even make portions of their computer files directly available to select or even all other communicants, with or without password requirements.  That is, in fact, how they could mount a personal Web site for access by others, essentially serving as their own Web host system. Nothing new in that.

Google Earth Image: An excellent resource for planning WW layouts as it can show terrain and building heights to the planners advantage.

Martial law, Police State, revolution, natural disaster, Internet shut down, CISP, PIPA
Add some Intranet email software clients or peer-to-peer clients and you get some ease of communications which would seem rather normalized to the way one uses the Web, currently. But as for Web Walker-based user Web sites, the actual Internet would be out of bounds, or should be, to prevent assault from spyware or worms designed to discover and track Web Walker activity.
One simple solution is to buy a used Macintosh, which is generally impervious to externalized attacks of this sort, and which would be safer for use on any compromised Internet government might control. Go ahead and mount Windows and your PC software on it so it behaves like a PC if that’s what you are more comfortable with. In a way, a Mac is often more ‘PC compatible’ than many PCs, anyway. Protect Web Walker best by complete dissociation from the WWW by use of a dedicated router for the actual Internet. You can easily bounce back and forth between WW and WWW sources with a few mouse clicks, and copy and past may prove useful.
But where an elaborate WW node (e.g., multiple routers in multiple directions) is employed, the potential exists to relay information (email or Web site data) between any two adjacent WW locations. And there is no end of relays possible. All that is required is a localized addressing scheme, which is easy to come by because each server has a name, and each user an intranet IP address unique to the router. So whomever is the central-most party of a Triad (the designer, if you will), would determine the name of the WW router for addressing purposes, and track THAT node’s user IPs. So to send data or request data to/ from an outside Web Walker, would only require that you knew the address;  IPnumber@router name.  Thus whenever a new router (node) is to be added to the greater WW network, a proposed router name should be submitted to the network to allow people to indicate if it was perhaps already in use at their end.
As in the real Internet, you should never share addresses, as that could compromise user privacy expectations. That means no one knows exactly where on the greater WW network a given user is located. So to send a request for a ‘Web page’ on the WW network, or an email, one uses email (I suggest using PGP – Pretty Good Privacy encryption because it employs a sender/recipient decryption key scheme so that only the recipient can access) to send to the WW node operator. When the operator gets it, he compares the address to those listed on his own node(s), and sends it on if the recipient is one of his ‘flock.’ This could be automated with minimal software.
Otherwise, he switches his computer over to each of the other WW nodes which link to other, distant WW nodes, and forwards the file (also subject to automation). In this way the message goes everywhere until it finds the right operator, and is forwarded to the specific recipient. If a Web page request, the recipient sends an email back containing the Web markup language code for the file, along with all embedded images, videos, etc.  This may require an email client which has no file size limitation, such as a virtual private network email client, which is a good choice, anyway, for management of email by a node operator.
The flaw in this plan is that the designer would need to spend a considerable amount of time looking at data flows and switching routers and forwarding as needed from one node to another. That could prove a severe issue in a major City environment well connected to large numbers of individuals. Therefore, two additional bits of advice are offered. Web Walker: How to Bypass Internet Surveillance or Defeat a Total Web Shut Down

1) to some degree, restrict membership to those persons of value to the localized network. That means people who truly are kindred spirits, or who have useful resources or skills, such as weapons, food, or whatever is appropriate for assuring mutual survival. This is not our Children’s Internet;

2) Users should limit communications to serious needs, only. You don’t go ‘surfing,’ and you don’t send spam, or watch movies, porn, play games, etc.

3) Employ scripting, utility software, or write custom software to handle as much of the work load as possible. Macintosh has system-level scripting built in, as do many kinds of Web utilities, and Linux/Unix systems are also quite flexible in this respect.Web Walker: How to Bypass Internet Surveillance or Defeat a Total Web Shut Down

Men in Black
Naturally, the government will not like this idea very much. Who knows? They may label me a terrorist any moment for having published this idea. Welcome to do so if they want a bit of bad publicity.  I’m sure I’m already on their list, anyway. But their concern means that you would likely want to hide your units and shield your rooms to prevent so many strong signals from screaming, ‘Here I am!’ Better they should see only one signal at a time, if possible, and not easily ascertain its point of origin. Ask me about Primus Pick Proof Locks for your home, if truly worried about Men in Black snoops.
Again, it would be wise to establish multiple routes between critical or important users or resources in case one route is compromised or suffers technical failure. Any two adjacent WW Node operators would know each other and would become aware of any such loss of service (again, to play safe, shut down any signal to it) and investigate carefully before deciding what to do about it (e.g., replace the route, help repair, etc.)
Unless they specifically passed a law or edict against, even if discovered, you should not likely be in too great a danger except from confiscation of equipment and questioning. Likely, the files on your computer would be a greater risk of getting you into trouble if they felt like pressing a given matter. Spare equipment, secret locations, and escape tunnels may be called for in the worst of times.
And should revolution come, I have heard it said regarding one such revolution, “It was the best of times, and the worst of times…” But in such a case, being able to communicate can make a bad time the best possible.

Update: Due to popular response and interest, I’m going to attempt to launch an Open Source Project to make Web Walker real. The first order of business will be to come up with a new name: at the time of first posting just a few months back, there was only one Web Walker out there, a Web authoring service, and one product  (a one word trade name). Now, however, there are nearly a half dozen Web Walkers of all manner since then, for some reason. Why, if I was paranoid, I’d think it on purpose~ :) Ideas anyone?

Click the FOLLOW button at very page top to learn more about the Open Source Project, or better yet, especially if wanting to learn about the may options or ways in which you can participate, use the CONTACT link at page top to express your interest. There will be more posts on the Project, to include many enhancements such as an inherent mechanism to simulate social networking and search engines free of advertising, spying mechanisms, and useless distracting fluff. THANK YOU for taking the time to learn about Web Walker!

PLEASE: Comment (page bottom, or contact me), Rate (at page top), Share/Tweet this article. Visit my Reader Forum to promote your own cause or URL, even if unrelated to this article.

Related articles

• Would you give the government remote control over your router? (arstechnica.com)
• Ron, Rand Paul: Get Government Away From Internet (personalliberty.com)

Mission Impossible: Hardening Android for Security and Privacy

https://blog.torproject.org/blog/mission-impossible-hardening-android-security-and-privacy

Executive Summary

The future is here, and ahead of schedule. Come join us, the weather's nice.
This blog post describes the installation and configuration of a prototype of a secure, full-featured, Android telecommunications device with full Tor support, individual application firewalling, true cell network baseband isolation, and optional ZRTP encrypted voice and video support. ZRTP does run over UDP which is not yet possible to send over Tor, but we are able to send SIP account login and call setup over Tor independently.
The SIP client we recommend also supports dialing normal telephone numbers if you have a SIP gateway that provides trunking service.
Aside from a handful of binary blobs to manage the device firmware and graphics acceleration, the entire system can be assembled (and recompiled) using only FOSS components. However, as an added bonus, we will describe how to handle the Google Play store as well, to mitigate the two infamous Google Play Backdoors.

Introduction

Android is the most popular mobile platform in the world, with a wide variety of applications, including many applications that aid in communications security, censorship circumvention, and activist organization. Moreover, the core of the Android platform is Open Source, auditable, and modifiable by anyone.
Unfortunately though, mobile devices in general and Android devices in particular have not been designed with privacy in mind. In fact, they've seemingly been designed with nearly the opposite goal: to make it easy for third parties, telecommunications companies, sophisticated state-sized adversaries, and even random hackers to extract all manner of personal information from the user. This includes the full content of personal communications with business partners and loved ones. Worse still, by default, the user is given very little in the way of control or even informed consent about what information is being collected and how.
This post aims to address this, but we must first admit we stand on the shoulders of giants. Organizations like Cyanogen, F-Droid, the Guardian Project, and many others have done a great deal of work to try to improve this situation by restoring control of Android devices to the user, and to ensure the integrity of our personal communications. However, all of these projects have shortcomings and often leave gaps in what they provide and protect. Even in cases where proper security and privacy features exist, they typically require extensive configuration to use safely, securely, and correctly.
This blog post enumerates and documents these gaps, describes workarounds for serious shortcomings, and provides suggestions for future work.
It is also meant to serve as a HOWTO to walk interested, technically capable people through the end-to-end installation and configuration of a prototype of a secure and private Android device, where access to the network is restricted to an approved list of applications, and all traffic is routed through the Tor network.
It is our hope that this work can be replicated and eventually fully automated, given a good UI, and rolled into a single ROM or ROM addon package for ease of use. Ultimately, there is no reason why this system could not become a full fledged off the shelf product, given proper hardware support and good UI for the more technical bits.
The remainder of this document is divided into the following sections:

1. Hardware Selection
2. Installation and Setup
3. Google Apps Setup
4. Recommended Software
5. Device Backup Procedure
6. Removing the Built-in Microphone
7. Removing Baseband Remnants
8. Future Work
9. Changes Since Initial Posting

Hardware Selection

If you truly wish to secure your mobile device from remote compromise, it is necessary to carefully select your hardware. First and foremost, it is absolutely essential that the carrier's baseband firmware is completely isolated from the rest of the platform. Because your cell phone baseband does not authenticate the network (in part to allow roaming), any random hacker with their own cell network can exploit these backdoors and use them to install malware on your device.
While there are projects underway to determine which handsets actually provide true hardware baseband isolation, at the time of this writing there is very little public information available on this topic. Hence, the only safe option remains a device with no cell network support at all (though cell network connectivity can still be provided by a separate device). For the purposes of this post, the reference device is the WiFi-only version of the 2013 Google Nexus 7 tablet.
For users who wish to retain full mobile access, we recommend obtaining a cell modem device that provides a WiFi access point for data services only. These devices do not have microphones and in some cases do not even have fine-grained GPS units (because they are not able to make emergency calls). They are also available with prepaid plans, for rates around $20-30 USD per month, for about 2GB/month of 4G data. If coverage and reliability is important to you though, you may want to go with a slightly more expensive carrier. In the US, T-Mobile isn't bad, but Verizon is superb.
To increase battery life of your cell connection, you can connect this access point to an external mobile USB battery pack, which typically will provide 36-48 hours of continuous use with a 6000mAh battery.
The total cost of a Wifi-only tablet with cell modem and battery pack is only roughly USD $50 more than the 4G LTE version of the same device.
In this way, you achieve true baseband isolation, with no risk of audio or network surveillance, baseband exploits, or provider backdoors. Effectively, this cell modem is just another untrusted router in a long, long chain of untrustworthy Internet infrastructure.
However, do note though that even if the cell unit does not contain a fine-grained GPS, you still sacrifice location privacy while using it. Over an extended period of time, it will be possible to make inferences about your physical activity, behavior and personal preferences, and your identity, based on cell tower use alone.

Installation and Setup

We will focus on the installation of Cyanogenmod 11 using Team Win Recovery Project, both to give this HOWTO some shelf life, and because Cyanogenmod 11 features full SELinux support (Dear NSA: What happened to you guys? You used to be cool. Well, some of you. Some of the time. Maybe. Or maybe not).
The use of Google Apps and Google Play services is not recommended due to security issues with Google Play. However, we do provide workarounds for mitigating those issues, if Google Play is required for your use case.

Installation and Setup: ROM and Core App Installation

With the 2013 Google Nexus 7 tablet, installation is fairly straight-forward. In fact, it is actually possible to install and use the device before associating it with a Google Account in any way. This is a desirable property, because by default, the otherwise mandatory initial setup process of the stock Google ROM sends your device MAC address directly to Google and links it to your Google account (all without using Tor, of course).
The official Cyanogenmod installation instructions are available online, but with a fresh out of the box device, here are the key steps for installation without activating the default ROM code at all (using Team Win Recovery Project instead of ClockWorkMod).
First, on your desktop/laptop computer (preferably Linux), perform the following:

1. Download the latest CyanogenMod 11 release (we used cm-11-20140504-SNAPSHOT-M6)
2. Download the latest Team Win Recovery Project image (we used 2.7.0.0)
3. Download the F-Droid package (we used 0.66)
4. Download the Orbot package from F-Droid (we used 13.0.7)
5. Download the Droidwall package from F-Droid (we used 1.5.7)
6. Download the Droidwall Firewall Scripts attached to this blogpost
7. Download the Google Apps for Cyanogenmod 11 (optional)

Because the download integrity for all of these packages is abysmal, here is a signed set of SHA256 hashes I've observed for those packages.
Once you have all of those packages, boot your tablet into fastboot mode by holding the Power button and the Volume Down button during a cold boot. Then, attach it to your desktop/laptop machine with a USB cable and run the following commands from a Linux/UNIX shell:

 apt-get install android-tools-adb android-tools-fastboot
 fastboot devices
 fastboot oem unlock
 fastboot flash recovery openrecovery-twrp-2.7.0.0-flo.img

After the recovery firmware is flashed successfully, use the volume keys to select Recovery and hit the power button to reboot the device (or power it off, and then boot holding Power and Volume Up).
Once Team Win boots, go into Wipe and select Advanced Wipe. Select all checkboxes except for USB-OTG, and slide to wipe. Once the wipe is done, click Format Data. After the format completes, issue these commands from your Linux shell:

 adb server start
 adb push cm-11-20140504-SNAPSHOT-M6-flo.zip /sdcard/
 adb push gapps-kk-20140105-signed.zip /sdcard/ # Optional

After this push process completes, go to the Install menu, and select the Cyanogen zip, and optionally the gapps zip for installation. Then click Reboot, and select System.
After rebooting into your new installation, skip all CyanogenMod and Google setup, disable location reporting, and immediately disable WiFi and turn on Airplane mode.
Then, go into Settings -> About Tablet and scroll to the bottom and click the greyed out Build number 5 times until developer mode is enabled. Then go into Settings -> Developer Options and turn on USB Debugging.
After that, run the following commands from your Linux shell:

 adb install FDroid.apk
 adb install org.torproject.android_86.apk
 adb install com.googlecode.droidwall_157.apk

You will need to approve the ADB connection for the first package, and then they should install normally.
VERY IMPORTANT: Whenever you finish using adb, always remember to disable USB Debugging and restore Root Access to Apps only. While Android 4.2+ ROMs now prompt you to authorize an RSA key fingerprint before allowing a debugging connection (thus mitigating adb exploit tools that bypass screen lock and can install root apps), you still risk additional vulnerability surface by leaving debugging enabled.

Installation and Setup: Initial Configuration

After the base packages are installed, go into the Settings app, and make the following changes:

1. Wireless & Networks More... =>
• Temporarily Disable Airplane Mode
• NFC -> Disable
• Re-enable Airplane Mode
2. Location Access -> Off
3. Security =>
• PIN screen Lock
• Allow Unknown Sources (For F-Droid)
4. Language & Input =>
• Spell Checker -> Android Spell Checker -> Disable Contact Names
• Disable Google Voice Typing/Hotword detection
• Android Keyboard (AOSP) =>
  • Disable AOSP next-word suggestion (do this first!)
  • Auto-correction -> Off
5. Backup & reset =>
• Enable Back up my data (just temporarily, for the next step)
• Uncheck Automatic restore
• Disable Backup my data
6. About Tablet -> Cyanogenmod Statistics -> Disable reporting
7. Developer Options -> Device Hostname -> localhost
8. SuperUser -> Settings (three dots) -> Notifications -> Notification (not toast)
9. Privacy -> Privacy Guard =>
   • Enabled by default
   • Settings (three dots) -> Show Built In Apps
   • Enable Privacy Guard for every app with the following exceptions:
     • Calendar
     • Config Updater
     • Google Account Manager (long press)
     • Modify Settings -> Off
     • Wifi Change -> Off
     • Data Change -> Off
     • Google Play Services (long press)
     • Location -> Off
     • Modify Settings -> Off
     • Draw on top -> Off
     • Record Audio -> Off
     • Wifi Change -> Off
     • Google Play Store (long press)
     • Location -> Off
     • Send SMS -> Off
     • Modify Settings -> Off
     • Data change -> Off
     • Google Services Framework (long press)
     • Modify Settings -> Off
     • Wifi Change -> Off
     • Data Change -> Off
     • Trebuchet

Now, it is time to encrypt your tablet. It is important to do this step early, as I have noticed additional apps and configuration tweaks can make this process fail later on.
We will also do this from the shell, in order to set a different password than your screen unlock pin. This is done to mitigate the risk of compromise of this password from shoulder surfers, and to allow the use of a much longer (and non-numeric) password that you would prefer not to type every time you unlock the screen.
To do this, open the Terminal app, and type the following commands:

su
vdc cryptfs enablecrypto inplace NewMoreSecurePassword

Watch for typos! That command does not ask you to re-type that password for confirmation.

Installation and Setup: Disabling Invasive Apps and Services

Before you configure the Firewall or enable the network, you likely want to disable at least a subset of the following built-in apps and services, by using Settings -> Apps -> All, and then clicking on each app and hitting the Disable button:

• com.android.smspush
• com.google.android.voicesearch
• Face Unlock
• Google Backup Transport
• Google Calendar Sync
• Google One Time Init
• Google Partner Setup
• Google Contacts Sync
• Google Search
• Hangouts
• Market Feedback Agent
• News & Weather
• One Time Init
• Picasa Updater
• Sound Search for Google Play
• TalkBack

Installation and Setup: Tor and Firewall configuration

Ok, now let's install the firewall and tor support scripts. Go back into Settings -> Developer Options and enable USB Debugging and change Root Access to Apps and ADB. Then, unzip the android-firewall.zip on your laptop, and run the installation script:

./install-firewall.sh

That firewall installation provides several key scripts that provide functionality that is currently impossible to achieve with any app (including Orbot):

1. It installs a userinit script to block all network access during boot.
2. It disables "Google Captive Portal Detection", which involves connection attempts to Google servers upon Wifi assocation (these requests are made by the Android Settings UID, which should normally be blocked from the network, unless you are first registering for Google Play).
3. It contains a Droidwall script that configures Tor transproxy rules to send all of your traffic through Tor. These rules include a fix for a Linux kernel Tor transproxy packet leak issue.
4. The main firewall-torify-all.sh Droidwall script also includes an input firewall, to block all inbound connections to the device. It also fixes a Droidwall permissions vulnerability
5. It installs an optional script to allow the Browser app to bypass Tor for logging into WiFi captive portals.
6. It installs an optional script to temporarily allow network adb access when you need it (if you are paranoid about USB exploits, which you should be).
7. It provides an optional script to allow the UDP activity of LinPhone to bypass Tor, to allow ZRTP-encrypted Voice and Video SIP/VoIP calls. SIP account login/registration and call setup/signaling can be done over TCP, and Linphone's TCP activity is still sent through Tor with this script.

Note that with the exception of the userinit network blocking script, installing these scripts does not activate them. You still need to configure Droidwall to use them.
We use Droidwall instead of Orbot or AFWall+ for five reasons:

1. Droidwall's app-based firewall and Orbot's transproxy are known to conflict and reset one another.
2. Droidwall does not randomly drop transproxy rules when switching networks (Orbot has had several of these types of bugs).
3. Unlike AFWall+, Droidwall is able to auto-launch at "boot" (though still not before the network and Android Services come online and make connections).
4. AFWall+'s "fix" for this startup data leak problem does not work on Cyanogenmod (hence our userinit script instead).
5. Aside from the permissions issue fixed by our firewall-torify-all.sh script, AFWall+ provides no additional security fixes over the stock Droidwall.

To make use of the firewall scripts, open up Droidwall and hit the config button (the vertical three dots), go to More -> Set Custom Script. Enter the following:

. /data/local/firewall-torify-all.sh
#. /data/local/firewall-allow-adb.sh
#. /data/local/firewall-allow-linphone-udp.sh
#. /data/local/firewall-allow-nontor-browser.sh

NOTE: You must not make any typos in the above. If you mistype any of those files, things may break. Because the userinit.sh script blocks all network at boot, if you make a typo in the torify script, you will be unable to use the Internet at all!
Also notice that these scripts have been installed into a readonly root directory. Because they are run as root, installing them to a world-writable location like /sdcard/ is extremely unwise.
Later, if you want to enable one of network adb, LinPhone UDP, or captive portal login, go back into this window and remove the leading comment ('#') from the appropriate lines (this is obviously one of the many aspects of this prototype that could benefit from real UI).
Then, configure the apps you want to allow to access the network. Note that the only Android system apps that must access the network are:

• CM Updater
• Downloads, Media Storage, Download Manager
• F-Droid

Orbot's network access is handled via the main firewall-torify-all.sh script. You do not need to enable full network access to Orbot in Droidwall.
The rest of the apps you can enable at your discretion. They will all be routed through Tor automatically.
Once Droidwall is configured, you can click on the Menu (three dots) and click the "Firewall Disabled" button to enable the firewall. Then, you can enable Orbot. Do not grant Orbot superuser access. It still opens the transproxy ports you need without root, and Droidwall is managing installation of the transproxy rules, not Orbot.
You are now ready to enable Wifi and network access on your device. For vulnerability surface reduction, you may want to use the Advanced Options -> Static IP to manually enter an IP address for your device to avoid using dhclient. You do not need a DNS server, and can safely set it to 127.0.0.1.

Google Apps Setup

If you installed the Google Apps zip, you need to do a few things now to set it up, and to further harden your device. If you opted out of Google Apps, you can skip to the next section.

Google Apps Setup: Initializing Google Play

The first time you use Google Play, you will need to enable four apps in Droidwall: "Google Account Manager, Google Play Services...", "Settings, Dev Tools, Fused Location...", "Gmail", and "Google Play" itself.
If you do not have a Google account, your best bet is to find open wifi to create one, as Google will often block accounts created through Tor, even if you use an Android device.
After you log in for the first time, you should be able to disable the "Google Account Manager, Google Play Services...", "Gmail", and the "Settings..." apps in Droidwall, but your authentication tokens in Google Play may expire periodically. If this happens, you should only need to temporarily enable the "Google Account Manager, Google Play Services..." app in Droidwall to obtain new ones.

Google Apps Setup: Mitigating the Google Play Backdoors

If you do choose to use Google Play, you need to be very careful about how you allow it to access the network. In addition to the risks associated with using a proprietary App Store that can send you targeted malware-infected packages based on your Google Account, it has at least two major user experience flaws:

1. Anyone who is able to gain access to your Google account can silently install root or full permission apps without any user interaction what-so-ever. Once installed, these apps can retroactively clear what little installation notification and UI-based evidence of their existence there was in the first place.
2. The Android Update Process does not inform the user of changes in permissions of pending update apps that happen to get installed after an Android upgrade.

The first issue can be mitigated by ensuring that Google Play does not have access to the network when not in use, by disabling it in Droidwall. If you do not do this, apps can be installed silently behind your back. Welcome to the Google Experience.
For the second issue, you can install the SecCheck utility, to monitor your apps for changes in permissions during a device upgrade.

Google Apps Setup: Disabling Google Cloud Messaging

If you have installed the Google Apps zip, you have also enabled a feature called Google Cloud Messaging.
The Google Cloud Messaging Service allows apps to register for asynchronous remote push notifications from Google, as well as send outbound messages through Google.
Notification registration and outbound messages are sent via the app's own UID, so using Droidwall to disable network access by an app is enough to prevent outbound data, and notification registration. However, if you ever allow network access to an app, and it does successfully register for notifications, these notifications can be delivered even when the app is once again blocked from accessing the network by Droidwall.
These inbound notifications can be blocked by disabling network access to the "Google Account Manager, Google Play Services, Google Services Framework, Google Contacts Sync" in Droidwall. In fact, the only reason you should ever need to enable network access by this service is if you need to log in to Google Play again if your authentication tokens ever expire.
If you would like to test your ability to control Google Cloud Messaging, there are two apps in the Google Play store than can help with this. GCM Test allows for simple send and receive pings through GCM. Push Notification Tester will allow you to test registration and asynchronous GCM notification.

Recommended Privacy and Auditing Software

Ok, so now that we have locked down our Android device, now for the fun bit: secure communications!
We recommend the following apps from F-Droid:

1. Xabber
Xabber is a full Java implementation of XMPP, and supports both OTR and Tor. Its UI is a bit more streamlined than Guardian Project's ChatSecure, and it does not make use of any native code components (which are more vulnerable to code execution exploits than pure Java code). Unfortunately, this means it lacks some of ChatSecure's nicer features, such as push-to-talk voice and file transfer.
Despite better protection against code execution, it does have several insecure default settings. In particular, you want to make the following changes:

• Notifications -> Message text in Notifications -> Off (notifications can be read by other apps!)
• Accounts -> Integration into system accounts -> Off
• Accounts -> Store message history -> Don't Store
• Security -> Store History -> Off
• Security -> Check Server Certificate
• Chat -> Show Typing Notifications -> Off
• Connection Settings -> Auto-away -> disabled
• Connection Settings -> Extended away when idle -> Disabled
• Keep Wifi Awake -> On
• Prevent sleep Mode -> On
2. Offline Calendar
Offline Calendar is a hack to allow you to create a fake local Google account that does not sync to Google. This allows you to use the Calendar App without risk of leaking your activities to Google. Note that you must exempt both this app and Calendar from Privacy Guard for it to function properly.

3. LinPhone
LinPhone is a FOSS SIP client that supports TCP TLS signaling and ZRTP. Note that neither TLS nor ZRTP are enabled by default. You must manually enable them in Settings -> Network -> Transport and Settings -> Network -> Media Encryption.
ostel.co is a free SIP service run by the Guardian Project that supports only TLS and ZRTP, but does not allow outdialing to normal PSTN telephone numbers. While Bitcoin has many privacy issues of its own, the Bitcoin community maintains a couple lists of "trunking" providers that allow you to obtain a PSTN phone number in exchange for Bitcoin payment.

4. Plumble Plumble is a Mumble client that will route voice traffic over Tor, which is useful if you would like to communicate with someone over voice without revealing your IP to them, or your activity to a local network observer. However, unlike Linphone, voice traffic is not end-to-end encrypted, so the Mumble server can listen to your conversations.
   
5. K-9 Mail and APG K-9 Mail is a POP/IMAP client that supports TLS and integrates well with APG, which will allow you to send and receive GPG-encrypted mail easily. Before using it, you should be aware of two things: It identifies itself in your mail headers, which opens you up to targeted attacks specifically tailored for K-9 Mail and/or Android, and by default it includes the subject of messages in mail notifications (which is bad, because other apps can read notifications). There is a privacy option to disable subject text in notifications, but there is no option to disable the user agent in the mail headers.
   
6. OSMAnd~
A free offline mapping tool. While the UI is a little clunky, it does support voice navigation and driving directions, and is a handy, private alternative to Google Maps.

7. VLC The VLC port in F-Droid is a fully capable media player. It can play mp3s and most video formats in use today. It is a handy, private alternative to Google Music and other closed-source players that often report your activity to third party advertisers. VLC does not need network access to function.
   
8. Firefox
We do not yet have a port of Tor Browser for Android (though one is underway -- see the Future Work section). Unless you want to use Google Play to get Chrome, Firefox is your best bet for a web browser that receives regular updates (the built in Browser app does not). HTTPS-Everywhere and NoScript are available, at least.

9. Bitcoin
Bitcoin might not be the most private currency in the world. In fact, you might even say it's the least private currency in the world. But, it is a neat toy.

10. Launch App Ops The Launch App Ops app is a simple shortcut into the hidden application permissions editor in Android. A similar interface is available through Settings -> Privacy -> Privacy Guard, but a direct shortcut to edit permissions is handy. It also displays some additional system apps that Privacy Guard omits.
    
11. Permissions The Permissions app gives you a view of all Android permissions, and shows you which apps have requested a given permission. This is particularly useful to disable the record audio permission for apps that you don't want to suddenly decide to listen to you. (Interestingly, the Record Audio permission disable feature was broken in all Android ROMs I tested, aside from Cyanogenmod 11. You can test this yourself by revoking the permission from the Sound Recorder app, and verifying that it cannot record.)
    
12. CatLog
In addition to being supercute, CatLog is an excellent Android monitoring and debugging tool. It allows you to monitor and record the full set of Android log events, which can be helpful in diagnosing issues with apps.

13. OS Monitor
OS Monitor is an excellent Android process and connection monitoring app, that can help you watch for CPU usage and connection attempts by your apps.

14. Intent Intercept
Intent Intercept allows you to inspect and extract Android Intent content without allowing it to get forwarded to an actual app. This is useful for monitoring how apps attempt to communicate with eachother, though be aware it only covers one of the mechanisms of inter-app communication in Android.

Backing up Your Device Without Google

Now that your device is fully configured and installed, you probably want to know how to back it up without sending all of your private information directly to Google. While the Team Win Recovery Project will back up all of your system settings and apps (even if your device is encrypted), it currently does not back up the contents of your virtualized /sdcard. Remembering to do a couple adb pulls of key directories can save you a lot of heartache should you suffer some kind of data loss or hardware failure (or simply drop your tablet on a bridge while in a rush to catch a train).
The backup.sh script uses adb to pull your Download and Pictures directories from the /sdcard, as well as pulls the entire TWRP backup directory.
Before you use that script, you probably want to delete old TWRP backup folders so as to only pull one backup, to reduce pull time. These live in /sdcard/TWRP/BACKUPS/, which is also known as /storage/emulated/0/TWRP/BACKUPS in the File Manager app.
To use this script over the network without a usb cable, enable both USB Debugging and ADB Over Network in your developer settings. The script does not require you to enable root access from adb, and you should not enable root because it takes quite a while to run a backup, especially if you are using network adb.
Prior to using network adb, you must edit your Droidwall custom scripts to allow it (by removing the '#' in the #. /data/local/firewall-allow-adb.sh line you entered earlier), and then run the following commands from a non-root Linux shell on your desktop/laptop (the ADB Over Network setting will tell you the IP and port):

killall adb
adb connect ip:5555

VERY IMPORTANT: Don't forget to disable USB Debugging, as well as the Droidwall adb exemption when you are done with the backup!

Removing the Built-in Microphone

If you would really like to ensure that your device cannot listen to you even if it is exploited, it turns out it is very straight-forward to remove the built-in microphone in the Nexus 7. There is only one mic on the 2013 model, and it is located just below the volume buttons (the tiny hole).
To remove it, all you need to do is pop off the the back panel (this can be done with your fingernails, or a tiny screwdriver), and then you can shave the microphone right off that circuit board, and reattach the panel. I have done this to one of my devices, and it was subsequently unable to record audio at all, without otherwise affecting functionality.
You can still use apps that require a microphone by plugging in headphone headset that contains a mic built in (these cost around $20 and you can get them from nearly any consumer electronics store). I have also tested this, and was still able to make a Linphone call from a device with the built in microphone removed, but with an external headset. Note that the 2012 Nexus 7 does not support these combination microphone+headphone jacks (and it has a secondary microphone as well). You must have the 2013 model.
The 2013 Nexus 7 Teardown video can give you an idea of what this looks like before you try it. Again you do not need to fully disassemble the device - you only need to remove the back cover.
Pro-Tip: Before you go too crazy and start ripping out the cameras too, remember that you can cover the cameras with a sticker or tape when not in use. I have found that regular old black electrical tape applies seamlessly, is non-obvious to casual onlookers, and is easy to remove without smudging or gunking up the lenses. Better still, it can be removed and reapplied many times without losing its adhesive.

Removing the Remnants of the Baseband

There is one more semi-hardware mod you may want to make, though.
It turns out that the 2013 Wifi Nexus 7 does actually have a partition that contains a cell network baseband firmware on it, located on the filesystem as the block device /dev/block/platform/msm_sdcc.1/by-name/radio. If you run strings on that block device from the shell, you can see that all manner of CDMA and GSM log messages, comments, and symbols are present in that partition.
According to ADB logs, Cyanogenmod 11 actually does try to bring up a cell network radio at boot on my Wifi-only Nexus 7, but fails due to it being disabled. There is also a strong economic incentive for Asus and Google to make it extremely difficult to activate the baseband even if the hardware is otherwise identical for manufacturing reasons, since they sell the WiFi-only version for $100 less. If it were easy to re-enable the baseband, HOWTOs would exist (which they do not seem to, at least not yet), and they would cut into their LTE device sales.
Even so, since we lack public schematics for the Nexus 7 to verify that cell components are actually missing or hardware-disabled, it may be wise to wipe this radio firmware as well, as defense in depth.
To do this, open the Terminal app, and run:

su
cd /dev/block/platform/msm_sdcc.1/by-name
dd if=/dev/zero of=./radio

I have wiped that partition while the device was running without any issue, or any additional errors from ADB logs.
Note that an anonymous commenter also suggested it is possible to disable the baseband of a cell-enabled device using a series of Android service disable commands, and by wiping that radio block device. I have not tested this on a device other than the WiFI-only Nexus 7, though, so proceed with caution. If you try those steps on a cell-enabled device, you should archive a copy of your radio firmware first by doing something like the following from that dev directory that contains the radio firmware block device.

dd if=./radio of=/sdcard/radio.img

If anything goes wrong, you can restore that image with:

dd if=/sdcard/radio.img of=./radio

Future Work

In addition to streamlining the contents of this post into a single additional Cyanogenmod installation zip or alternative ROM, the following problems remain unsolved.

Future Work: Better Usability

While arguably very secure, this system is obviously nowhere near usable. Here are some potential improvements to the user interface, based on a brainstorming session I had with another interested developer.
First of all, the AFWall+/Droidwall UI should be changed to be a tri-state: It should allow you to send app traffic over Tor, over your normal internet connection, or block it entirely.
Next, during app installation from either F-Droid or Google Play (this is an Intent another addon app can actually listen for), the user should be given the chance to decide if they would like that app's traffic to be routed over Tor, use the normal Internet connection, or be blocked entirely from accessing the network. Currently, the Droidwall default for new apps is "no network", which is a great default, but it would be nice to ask users what they would like to do during actual app installation.
Moreover, users should also be given a chance to edit the app's permissions upon installation as well, should they desire to do so.
The Google Play situation could also be vastly improved, should Google itself still prove unwilling to improve the situation. Google Play could be wrapped in a launcher app that automatically grants it network access prior to launch, and then disables it upon leaving the window.
A similar UI could be added to LinPhone. Because the actual voice and video transport for LinPhone does not use Tor, it is possible for an adversary to learn your SIP ID or phone number, and then call you just for the purposes of learning your IP. Because we handle call setup over Tor, we can prevent LinPhone from performing any UDP activity, or divulging your IP to the calling party prior to user approval of the call. Ideally, we would also want to inform the user of the fact that incoming calls can be used to obtain information about them, at least prior to accepting their first call from an unknown party.

Future Work: Find Hardware with Actual Isolated Basebands

Related to usability, it would be nice if we could have a serious community effort to audit the baseband isolation properties of existing cell phones, so we all don't have to carry around these ridiculous battery packs and sketch-ass wifi bridges. There is no engineering reason why this prototype could not be just as secure if it were a single piece of hardware. We just need to find the right hardware.
A random commenter claimed that the Galaxy Nexus might actually have exactly the type of baseband isolation we want, but the comment was from memory, and based on software reverse engineering efforts that were not publicly documented. We need to do better than this.

Future Work: Bug Bounty Program

If there is sufficient interest in this prototype, and/or if it gets transformed into a usable addon package or ROM, we may consider running a bug bounty program where we accept donations to a dedicated Bitcoin address, and award the contents of that wallet to anyone who discovers a Tor proxy bypass issue or remote code execution vulnerability in any of the network-enabled apps mentioned in this post (except for the Browser app, which does not receive security updates).

Future Work: Port Tor Browser to Android

The Guardian Project is undertaking a port of Tor Browser to Android as part of their OrFox project. This will greatly improve the privacy of your web browsing experience on the Android device over both Firefox and Chrome. We look forward to helping them in any way we can with this effort.

Future Work: WiFi MAC Address Randomization

It is actually possible to randomize the WiFi MAC address on the Google Nexus 7. The closed-source root app Mac Spoofer is able to modify the device MAC address using Qualcomm-specific methods in such a way that the entire Android OS becomes convinced that this is your actual MAC.
However, doing this requires installation of a root-enabled, closed-source application from the Google Play Store, which we believe is extremely unwise on a device you need to be able to trust. Moreover, this app cannot be autorun on boot, and your MAC address will also reset every time you disable the WiFi interface (which is easy to do accidentally). It also supports using only a single, manually entered MAC address.
Hardware-independent techniques (such as a the Terminal command busybox ifconfig wlan0 hw ether ) appear to interfere with the WiFi management system and prevent it from associating. Moreover, they do not cause the Android system to report the new MAC address, either (visible under Settings -> About Tablet -> Status).
Obviously, an Open Source F-Droid app that properly resets (and automatically randomizes) the MAC every time the WiFi interface is brought up is badly needed.

Future Work: Disable Probes for Configured Wifi Networks

The Android OS currently probes for all of your configured WiFi networks while looking for open wifi to connect to. Configured networks should not be probed for explictly unless activity for their BSSID is seen. The xda-developers forum has a limited fix to change scanning behavior, but users report that it does not disable the active probing behavior for any "hidden" networks that you have configured.

Future Work: Recovery ROM Password Protection

An unlocked recovery ROM is a huge vulnerability surface for Android. While disk encryption protects your applications and data, it does not protect many key system binaries and boot programs. With physical access, it is possible to modify these binaries through your recovery ROM.
The ability to set a password for the Team Win recovery ROM in such a way that a simple "fastboot flash recovery" would overwrite would go a long way to improving device security. At least it would become evident to you if your recovery ROM has been replaced, in this case (due to the absence of the password).
It may also be possible to restore your bootloader lock as an alternative, but then you lose the ability to make backups of your system using Team Win.

Future Work: Disk Encryption via TPM or Clever Hacks

Unfortunately, even disk encryption and a secure recovery firmware is not enough to fully defend against an adversary with an extended period of physical access to your device.
Cold Boot Attacks are still very much a reality against any form of disk encryption, and the best way to eliminate them is through hardware-assisted secure key storage, such as through a TPM chip on the device itself.
It may also be possible to mitigate these attacks by placing key material in SRAM memory locations that will be overwritten as part of the ARM boot process. If these physical memory locations are stable (and for ARM systems that use the SoC SRAM to boot, they will be), rebooting the device to extract key material will always end up overwriting it. Similar ARM CPU-based encryption defenses have also been explored in the research literature.

Future Work: Download and Build Process Integrity

Beyond the download integrity issues mentioned above, better build security is also deeply needed by all of these projects. A Gitian descriptor that is capable of building Cyanogenmod and arbitrary F-Droid packages in a reproducible fashion is one way to go about achieving this property.

Future Work: Removing Binary Blobs

If you read the Cyanogenmod build instructions closely, you can see that it requires extracting the binary blobs from some random phone, and shipping them out. This is the case with most ROMs. In fact, only the Replicant Project seems concerned with this practice, but regrettably they do not support any wifi-only devices. This is rather unfortunate, because no matter what they do with the Android OS on existing cell-enabled devices, they will always be stuck with a closed source, backdoored baseband that has direct access to the microphone, if not the RAM and the entire Android OS.
Kudos to them for finding one of the backdoors though, at least.

Changes Since Initial Posting

1. Updated firewall scripts to fix Droidwall permissions vulnerability.
2. Updated Applications List to recommend VLC as a free media player.
3. Mention the Guardian Project's planned Tor Browser port (called OrFox) as Future Work.
4. Mention disabling configured WiFi network auto-probing as Future Work
5. Updated the firewall install script (and the android-firewall.zip that contains it) to disable "Captive Portal detection" connections to Google upon WiFi association. These connections are made by the Settings service user, which should normally be blocked unless you are Activating Google Play for the first time.
6. Updated the Executive Summary section to make it clear that our SIP client can actually also make normal phone calls, too.
7. Document removing the built-in microphone, for the truly paranoid folk out there.
8. Document removing the remnants of the baseband, or disabling an existing baseband.
9. Update SHA256SUM of FDroid.apk for 0.63
10. Remove multiport usage from firewall-torify-all.sh script (and update android-firewall.zip).
11. Add pro-tip to the microphone removal section: Don't remove your cameras. Black electrical tape works just fine, and can be removed and reapplied many times without smudges.
12. Update android-firewall.zip installation and documentation to use /data/local instead of /etc. CM updates will wipe /etc, of course. Woops. If this happened to you while updating to CM-11-M5, download that new android-firewall.zip and run install-firewall.sh again as per the instructions above, and update your Droidwall custom script locations to use /data/local.
13. Update the Future work section to describe some specific UI improvements.
14. Update the Future work section to mention that we need to find hardware with actual isolated basebands. Duh. This should have been in there much earlier.
15. Update the versions for everything
16. Suggest enabling disk crypto directly from the shell, to avoid SSD leaks of the originally PIN-encrypted device key material.
17. GMail network access seems to be required for App Store initialization now. Mention this in Google Apps section.
18. Mention K-9 Mail, APG, and Plumble in the Recommended Apps section.
19. Update the Firewall instructions to clarify that you need to ensure there are no typos in the scripts, and actually click the Droidwall UI button to enable the Droidwall firewall (otherwise networking will not work at all due to userinit.sh).
20. Disable NFC in Settings config

• mikeperry's blog