Sameh Attia

Sunday, June 12, 2011

How To Set Up Software RAID1 On A Running System (Incl. GRUB2 Configuration) (Debian Squeeze)

This guide explains how to set up software RAID1 on an already running Debian Squeeze system. The GRUB2 bootloader will be configured in such a way that the system will still be able to boot if one of the hard drives fails (no matter which one).
I do not issue any guarantee that this will work for you!

1 Preliminary Note

In this tutorial I'm using a Debian Squeeze system with two hard drives, /dev/sda and /dev/sdb which are identical in size. /dev/sdb is currently unused, and /dev/sda has the following partitions:

/dev/sda1: /boot partition, ext4;
/dev/sda2: swap;
/dev/sda3: / partition, ext4

In the end I want to have the following situation:

/dev/md0 (made up of /dev/sda1 and /dev/sdb1): /boot partition, ext4;
/dev/md1 (made up of /dev/sda2 and /dev/sdb2): swap;
/dev/md2 (made up of /dev/sda3 and /dev/sdb3): / partition, ext4

This is the current situation:

df -h

root@server1:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             4.0G  712M  3.1G  19% /
tmpfs                 249M     0  249M   0% /lib/init/rw
udev                  244M  100K  244M   1% /dev
tmpfs                 249M     0  249M   0% /dev/shm
/dev/sda1             472M   25M  423M   6% /boot
root@server1:~#

fdisk -l

2 Installing mdadm

The most important tool for setting up RAID is mdadm. Let's install it like this:

apt-get install initramfs-tools mdadm

MD arrays needed for the root file system: <-- all
Afterwards, we load a few kernel modules (to avoid a reboot):

modprobe linear
modprobe multipath
modprobe raid0
modprobe raid1
modprobe raid5
modprobe raid6
modprobe raid10

Now run

cat /proc/mdstat

The output should look as follows:

root@server1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices:
root@server1:~#

3 Preparing /dev/sdb

To create a RAID1 array on our already running system, we must prepare the /dev/sdb hard drive for RAID1, then copy the contents of our /dev/sda hard drive to it, and finally add /dev/sda to the RAID1 array.
First, we copy the partition table from /dev/sda to /dev/sdb so that both disks have exactly the same layout:

sfdisk -d /dev/sda | sfdisk --force /dev/sdb

The output should be as follows:

root@server1:~# sfdisk -d /dev/sda | sfdisk --force /dev/sdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdb: 652 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
/dev/sdb: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *      2048    999423     997376  83  Linux
/dev/sdb2        999424   1998847     999424  82  Linux swap / Solaris
/dev/sdb3       1998848  10483711    8484864  83  Linux
/dev/sdb4             0         -          0   0  Empty
Warning: partition 1 does not end at a cylinder boundary
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
root@server1:~#

The command

fdisk -l

should now show that both HDDs have the same layout:

Next we must change the partition type of our three partitions on /dev/sdb to Linux raid autodetect:

fdisk /dev/sdb

root@server1:~# fdisk /dev/sdb

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): <-- m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): <-- t
Partition number (1-4): <-- 1
Hex code (type L to list codes): <-- L

0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris
1  FAT12           39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
2  XENIX root      3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
3  XENIX usr       40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
4  FAT16 <32M      41  PPC PReP Boot   85  Linux extended  c7  Syrinx
5  Extended        42  SFS             86  NTFS volume set da  Non-FS data
6  FAT16           4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
7  HPFS/NTFS       4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility
8  AIX             4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt
9  AIX bootable    50  OnTrack DM      93  Amoeba          e1  DOS access
a  OS/2 Boot Manag 51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O
b  W95 FAT32       52  CP/M            9f  BSD/OS          e4  SpeedStor
c  W95 FAT32 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs
e  W95 FAT16 (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  GPT
f  W95 Ext'd (LBA) 55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor
12  Compaq diagnost 61  SpeedStor       a9  NetBSD          f4  SpeedStor
14  Hidden FAT16 <3 63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary
16  Hidden FAT16    64  Novell Netware  af  HFS / HFS+      fb  VMware VMFS
17  Hidden HPFS/NTF 65  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  70  DiskSecure Mult b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 75  PC/IX           bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 80  Old Minix       be  Solaris boot    ff  BBT
1e  Hidden W95 FAT1
Hex code (type L to list codes): <-- fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): <-- t
Partition number (1-4): <-- 2
Hex code (type L to list codes): <-- fd
Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): <-- t
Partition number (1-4): <-- 3
Hex code (type L to list codes): <-- fd
Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help): <-- w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
root@server1:~#
To make sure that there are no remains from previous RAID installations on /dev/sdb, we run the following commands:

mdadm --zero-superblock /dev/sdb1
mdadm --zero-superblock /dev/sdb2
mdadm --zero-superblock /dev/sdb3

If there are no remains from previous RAID installations, each of the above commands will throw an error like this one (which is nothing to worry about):

root@server1:~# mdadm --zero-superblock /dev/sdb1
mdadm: Unrecognised md component device - /dev/sdb1
root@server1:~#

Otherwise the commands will not display anything at all.

4 Creating Our RAID Arrays

Now let's create our RAID arrays /dev/md0, /dev/md1, and /dev/md2. /dev/sdb1 will be added to /dev/md0, /dev/sdb2 to /dev/md1, and /dev/sdb3 to /dev/md2. /dev/sda1, /dev/sda2, and /dev/sda3 can't be added right now (because the system is currently running on them), therefore we use the placeholder missing in the following three commands:

mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1

mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdb2

mdadm --create /dev/md2 --level=1 --raid-disks=2 missing /dev/sdb3

You might see the following message for each command - just press y to continue:
root@server1:~# mdadm --create /dev/md2 --level=1 --raid-disks=2 missing /dev/sdb3
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? <-- y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
root@server1:~#
The command

cat /proc/mdstat

should now show that you have three degraded RAID arrays ([_U] or [U_] means that an array is degraded while [UU] means that the array is ok):

root@server1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sdb3[1]
      4241396 blocks super 1.2 [2/1] [_U]

md1 : active raid1 sdb2[1]
      499700 blocks super 1.2 [2/1] [_U]

md0 : active raid1 sdb1[1]
      498676 blocks super 1.2 [2/1] [_U]

unused devices:
root@server1:~#

Next we create filesystems on our RAID arrays (ext4 on /dev/md0 and /dev/md2 and swap on /dev/md1):

mkfs.ext4 /dev/md0
mkswap /dev/md1
mkfs.ext4 /dev/md2

Next we must adjust /etc/mdadm/mdadm.conf (which doesn't contain any information about our new RAID arrays yet) to the new situation:

cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_orig
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Display the contents of the file:

cat /etc/mdadm/mdadm.conf

At the bottom of the file you should now see details about our three (degraded) RAID arrays:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST 

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Tue, 24 May 2011 14:09:09 +0200
# by mkconf 3.1.4-1+8efb9d1
ARRAY /dev/md/0 metadata=1.2 UUID=b40c3165:17089af7:5d5ee79b:8783491b name=server1.example.com:0
ARRAY /dev/md/1 metadata=1.2 UUID=62e4a606:878092a0:212209c5:c91b8fef name=server1.example.com:1
ARRAY /dev/md/2 metadata=1.2 UUID=94e51099:d8475c57:4ff1c60f:9488a09a name=server1.example.com:2

5 Adjusting The System To RAID1

Now let's mount /dev/md0 and /dev/md2 (we don't need to mount the swap array /dev/md1):

mkdir /mnt/md0
mkdir /mnt/md2

mount /dev/md0 /mnt/md0
mount /dev/md2 /mnt/md2

You should now find both arrays in the output of

mount

root@server1:~# mount
/dev/sda3 on / type ext4 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/sda1 on /boot type ext4 (rw)
/dev/md0 on /mnt/md0 type ext4 (rw)
/dev/md2 on /mnt/md2 type ext4 (rw)
root@server1:~#

Next we modify /etc/fstab. Comment out the current /, /boot, and swap partitions and add new lines for them where you replace the UUIDs with /dev/md0 (for the /boot partition), /dev/md1 (for the swap partition) and /dev/md2 (for the / partition) so that the file looks as follows:

vi /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#                
proc            /proc           proc    defaults        0       0
# / was on /dev/sda3 during installation
#UUID=e4e38871-0115-477d-94f9-34b079d26248 /               ext4    errors=remount-ro 0       1
/dev/md2 /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
#UUID=7e2fb013-073e-4312-a669-f34b35069bfb /boot           ext4    defaults        0       2
/dev/md0 /boot           ext4    defaults        0       2
# swap was on /dev/sda2 during installation
#UUID=1a5951f8-d0ab-4e0e-b42a-871f81b6fd82 none            swap    sw              0       0
/dev/md1 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

Next replace /dev/sda1 with /dev/md0 and /dev/sda3 with /dev/md2 in /etc/mtab:

vi /etc/mtab

/dev/md2 / ext4 rw,errors=remount-ro 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
udev /dev tmpfs rw,mode=0755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
/dev/md0 /boot ext4 rw 0 0
/dev/md0 /mnt/md0 ext4 rw 0 0
/dev/md2 /mnt/md2 ext4 rw 0 0

Now up to the GRUB2 boot loader. Create the file /etc/grub.d/09_swraid1_setup as follows:

cp /etc/grub.d/40_custom /etc/grub.d/09_swraid1_setup
vi /etc/grub.d/09_swraid1_setup

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64' --class debian --class gnu-linux --class gnu --class os {
        insmod raid
        insmod mdraid
        insmod part_msdos
        insmod ext2
        set root='(md/0)'
        echo    'Loading Linux 2.6.32-5-amd64 ...'
        linux   /vmlinuz-2.6.32-5-amd64 root=/dev/md2 ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /initrd.img-2.6.32-5-amd64
}

Make sure you use the correct kernel version in the menuentry stanza (in the linux and initrd lines). You can find it out by running

uname -r

or by taking a look at the current menuentry stanzas in the ### BEGIN /etc/grub.d/10_linux ### section in /boot/grub/grub.cfg. Also make sure that you use root=/dev/md2 in the linux line.
The important part in our new menuentry stanza is the line set root='(md/0)' - it makes sure that we boot from our RAID1 array /dev/md0 (which will hold the /boot partition) instead of /dev/sda or /dev/sdb which is important if one of our hard drives fails - the system will still be able to boot.
Because we don't use UUIDs anymore for our block devices, open /etc/default/grub...

vi /etc/default/grub

... and uncomment the line GRUB_DISABLE_LINUX_UUID=true:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_LINUX_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Run

update-grub

to write our new kernel stanza from /etc/grub.d/09_swraid1_setup to /boot/grub/grub.cfg.
Next we adjust our ramdisk to the new situation:

update-initramfs -u

Now we copy the contents of /dev/sda1 and /dev/sda3 to /dev/md0 and /dev/md2 (which are mounted on /mnt/md0 and /mnt/md2):

cp -dpRx / /mnt/md2

cd /boot
cp -dpRx . /mnt/md0

6 Preparing GRUB2 (Part 1)

Afterwards we must make sure that the GRUB2 bootloader is installed on both hard drives, /dev/sda and /dev/sdb:

grub-install /dev/sda
grub-install /dev/sdb

Now we reboot the system and hope that it boots ok from our RAID arrays:

reboot

7 Preparing /dev/sda

If all goes well, you should now find /dev/md0 and /dev/md2 in the output of

df -h

root@server1:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              4.0G  714M  3.1G  19% /
tmpfs                 249M     0  249M   0% /lib/init/rw
udev                  244M  132K  244M   1% /dev
tmpfs                 249M     0  249M   0% /dev/shm
/dev/md0              472M   25M  423M   6% /boot
root@server1:~#

The output of

cat /proc/mdstat

should be as follows:

root@server1:~# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb3[1]
      4241396 blocks super 1.2 [2/1] [_U]

md1 : active (auto-read-only) raid1 sdb2[1]
      499700 blocks super 1.2 [2/1] [_U]

md0 : active raid1 sdb1[1]
      498676 blocks super 1.2 [2/1] [_U]

unused devices:
root@server1:~#

Now we must change the partition types of our three partitions on /dev/sda to Linux raid autodetect as well:

fdisk /dev/sda

root@server1:~# fdisk /dev/sda

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help): <-- t
Partition number (1-4): <-- 1
Hex code (type L to list codes): <-- fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): <-- t
Partition number (1-4): <-- 2
Hex code (type L to list codes): <-- fd
Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): <-- t
Partition number (1-4): <-- 3
Hex code (type L to list codes): <-- fd
Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help): <-- w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
root@server1:~#
Now we can add /dev/sda1, /dev/sda2, and /dev/sda3 to the respective RAID arrays:

mdadm --add /dev/md0 /dev/sda1
mdadm --add /dev/md1 /dev/sda2
mdadm --add /dev/md2 /dev/sda3

Now take a look at

cat /proc/mdstat

... and you should see that the RAID arrays are being synchronized:

root@server1:~# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda3[2] sdb3[1]
      4241396 blocks super 1.2 [2/1] [_U]
      [==========>..........]  recovery = 54.6% (2319808/4241396) finish=0.7min speed=45058K/sec

md1 : active raid1 sda2[2] sdb2[1]
      499700 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[2] sdb1[1]
      498676 blocks super 1.2 [2/2] [UU]

unused devices:
root@server1:~#

(You can run

watch cat /proc/mdstat

to get an ongoing output of the process. To leave watch, press CTRL+C.)

Wait until the synchronization has finished (the output should then look like this:

root@server1:~# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda3[2] sdb3[1]
      4241396 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[2] sdb2[1]
      499700 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[2] sdb1[1]
      498676 blocks super 1.2 [2/2] [UU]

unused devices:
root@server1:~#

).
Then adjust /etc/mdadm/mdadm.conf to the new situation:

cp /etc/mdadm/mdadm.conf_orig /etc/mdadm/mdadm.conf
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

/etc/mdadm/mdadm.conf should now look something like this:

cat /etc/mdadm/mdadm.conf

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST 

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Tue, 24 May 2011 14:09:09 +0200
# by mkconf 3.1.4-1+8efb9d1
ARRAY /dev/md/0 metadata=1.2 UUID=b40c3165:17089af7:5d5ee79b:8783491b name=server1.example.com:0
ARRAY /dev/md/1 metadata=1.2 UUID=62e4a606:878092a0:212209c5:c91b8fef name=server1.example.com:1
ARRAY /dev/md/2 metadata=1.2 UUID=94e51099:d8475c57:4ff1c60f:9488a09a name=server1.example.com:2

8 Preparing GRUB2 (Part 2)

Now we delete /etc/grub.d/09_swraid1_setup...

rm -f /etc/grub.d/09_swraid1_setup

... and update our GRUB2 bootloader configuration:

update-grub
update-initramfs -u

Now if you take a look at /boot/grub/grub.cfg, you should find that the menuentry stanzas in the ### BEGIN /etc/grub.d/10_linux ### section look pretty much the same as what we had in /etc/grub.d/09_swraid1_setup (they should now also be set to boot from /dev/md0 instead of (hd0) or (hd1)), that's why we don't need /etc/grub.d/09_swraid1_setup anymore.
Afterwards we must make sure that the GRUB2 bootloader is installed on both hard drives, /dev/sda and /dev/sdb:

grub-install /dev/sda
grub-install /dev/sdb

Reboot the system:

reboot

It should boot without problems.
That's it - you've successfully set up software RAID1 on your running Debian Squeeze system!

9 Testing

Now let's simulate a hard drive failure. It doesn't matter if you select /dev/sda or /dev/sdb here. In this example I assume that /dev/sdb has failed.
To simulate the hard drive failure, you can either shut down the system and remove /dev/sdb from the system, or you (soft-)remove it like this:

mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm --manage /dev/md1 --fail /dev/sdb2
mdadm --manage /dev/md2 --fail /dev/sdb3

mdadm --manage /dev/md0 --remove /dev/sdb1
mdadm --manage /dev/md1 --remove /dev/sdb2
mdadm --manage /dev/md2 --remove /dev/sdb3

Shut down the system:

shutdown -h now

Then put in a new /dev/sdb drive (if you simulate a failure of /dev/sda, you should now put /dev/sdb in /dev/sda's place and connect the new HDD as /dev/sdb!) and boot the system. It should still start without problems.
Now run

cat /proc/mdstat

and you should see that we have a degraded array:

root@server1:~# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda3[2]
      4241396 blocks super 1.2 [2/1] [U_]

md1 : active (auto-read-only) raid1 sda2[2]
      499700 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sda1[2]
      498676 blocks super 1.2 [2/1] [U_]

unused devices:
root@server1:~#

The output of

fdisk -l

should look as follows:

root@server1:~# fdisk -l

Disk /dev/sda: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e0f78

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          63      498688   fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2              63         125      499712   fd  Linux raid autodetect
Partition 2 does not end on cylinder boundary.
/dev/sda3             125         653     4242432   fd  Linux raid autodetect
Partition 3 does not end on cylinder boundary.

Disk /dev/sdb: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/md0: 510 MB, 510644224 bytes
2 heads, 4 sectors/track, 124669 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 511 MB, 511692800 bytes
2 heads, 4 sectors/track, 124925 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md2: 4343 MB, 4343189504 bytes
2 heads, 4 sectors/track, 1060349 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md2 doesn't contain a valid partition table
root@server1:~#

Now we copy the partition table of /dev/sda to /dev/sdb:

sfdisk -d /dev/sda | sfdisk --force /dev/sdb

root@server1:~# sfdisk -d /dev/sda | sfdisk --force /dev/sdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdb: 652 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
/dev/sdb: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *      2048    999423     997376  fd  Linux raid autodetect
/dev/sdb2        999424   1998847     999424  fd  Linux raid autodetect
/dev/sdb3       1998848  10483711    8484864  fd  Linux raid autodetect
/dev/sdb4             0         -          0   0  Empty
Warning: partition 1 does not end at a cylinder boundary
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
root@server1:~#

Afterwards we remove any remains of a previous RAID array from /dev/sdb...

mdadm --zero-superblock /dev/sdb1
mdadm --zero-superblock /dev/sdb2
mdadm --zero-superblock /dev/sdb3

... and add /dev/sdb to the RAID array:

mdadm -a /dev/md0 /dev/sdb1
mdadm -a /dev/md1 /dev/sdb2
mdadm -a /dev/md2 /dev/sdb3

Now take a look at

cat /proc/mdstat

root@server1:~# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb3[3] sda3[2]
      4241396 blocks super 1.2 [2/1] [U_]
      [======>..............]  recovery = 32.2% (1367168/4241396) finish=1.0min speed=44102K/sec

md1 : active raid1 sdb2[3] sda2[2]
      499700 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdb1[3] sda1[2]
      498676 blocks super 1.2 [2/2] [UU]

unused devices:
root@server1:~#

Wait until the synchronization has finished:

root@server1:~# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb3[3] sda3[2]
      4241396 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb2[3] sda2[2]
      499700 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdb1[3] sda1[2]
      498676 blocks super 1.2 [2/2] [UU]

unused devices:
root@server1:~#

Then install the bootloader on both HDDs:

grub-install /dev/sda
grub-install /dev/sdb

That's it. You've just replaced a failed hard drive in your RAID1 array.

10 Links

The Software-RAID Howto: http://tldp.org/HOWTO/Software-RAID-HOWTO.html
Debian: http://www.debian.org

Thursday, June 9, 2011

Using and Understanding IPv6 on openSUSE 11

Desktop - OpenSUSE

ipv6

So IPv6 has been around a while now but do we know any benefits? We may know that it is 128bits long and too difficult to type but if that is all then we do ourselves a little injustice in not knowing how easy IPv6 can make our lives. So let us take a look at some of the automation features of IPv6 including auto-address assignment, router discovery and why we do not need a broadcast address.

IPv6 Addresses

There are different address spaces, or types, within the new protocol. These types define their scope (or where they are effective). But first we look at the addressing scheme. Yes, the address is 128 bits but is written in hexadecimal (hex).

Each hex character represents 4 binary bits. The address is then written in blocks of 4 hex numbers (4 * 4 bits = 16 bits per block) delimited by a colon. This will mean that or address is written as 32 hex characters within the following range:

0000:0000:0000:0000:0000:0000:0000:00000

FFFF: FFFF FFFF: FFFF: FFFF: FFFF: FFFF: FFFF

Main Points

128 bits
8 words which are colon separated
Each word is 4 hex characters or 16 bits

Why use IPv6

We are still going to have IPv4 for the foreseeable future, the next 10 years or so. But IANA the governing body of the internet are running out of IPv4 Addresses. The central body are down to their last /8(1000 addresses) as is the Asia Pacific Local Authority. This means that in the APAC region IPv4 will only be issued to customers who can prove they are implementing IPv6 but have a proven need for the addresses. Public addresses are a limited resource that is running out.

IPv6 addressed being so much longer have vastly more addresses, trillions per person on earth so it is difficult to see how we could run out of IPv6 addresses. Technology improvements in the header of IPv6 will allow for better access and sharing of resources though built in Quality of Service (QoS), for example Remote Desktop Protocol (RDP) will require a constant bandwidth stream whereas as http, the main web browsing protocol works well with less consistent bandwidth . Major factors in adoption will also the automation features built-in to IPv6 where services such as gateways and DNS can be detected on the network without configuration.

Global Addresses

These addresses are public addresses that are issued on the internet and often begin with 2001:. This could give us a possible host with the ip-address of:

2001:0000:0000:0000:0000:0000:0000:0001

Do not fear - we have a form of shorthand that will allow us to omit leading zeros in each 4 block section so the address will become shortened to the following:

2001::1

NOTE: 2001:: is the first release of IPv6 Global Address. A real address would start 2001: the second word would then be an identifier for your Region in the World. Your ISP would then issue you a range using the third word. The prefix would then be /48 (3 * 16). The fourth bit you could then subnet yourself within your own enterprise to give you a /64 bit prefix. Below we see the network portion or first 64 bits in the four words of a typical global address:

2001

Regional ID

Your assigned prefix

Enterprise Subnet

Local Host

The local host (as in IPv4 127.0.0.1) will be addressed as:

0000:0000:0000:0000:0000:0000:0000:0001, or more simply:

::1

The output of “netstat –ant –inet6” from my openSUSE 11 box is shown below:

Anything listed as listening on the Local Address ::1 is listening on the internal or loopback adaptor
Anything listed as listening on the Local Address :: is listening on all IPv6 Interfaces
Ignore the extra colon. This is normal and separates the IPv6 address from the port.

Link-Local Addresses

These start FE80:000o:0000:0000 and end with the MAC address of the host ensuring the uniqueness of the address on the network. The scope is restricted to the local network, and all hosts on it. This network scope cannot be routed and has no access to the internet. However, this does mean we have auto-configuration of the local hosts on our network. We do not need to set anything for communication to work locally. You may compare the Link-Local address with the 169.254 addresses in IPv4.

The network is written as FE80::/64. Indicating the network portion is 64 bits and the last 64 bits make up the host. In IPv4 this would be called the subnet-mask, in IPv6 this is called the prefix.

The previous image shows the Link-local address from a Windows 7 machine. The %20 on the end is just used to identify the connection, or in other words, distinguishing between eth0 and eth1 if it were Linux.

Below is the Link-local address shown from openSUSE 11.As you can see the network address is the same as in Windows 7, FE80::/64

Unique Local Addresses

These addresses may compare with the private addresses in IPv4 such as 10.0.0.0, 172.16.0.0 and 192.168.0.0. The IPv6 range is FC00::/7 The first 7 bits must start 1111110 in binary. For the first 8 bits (2 hex characters) this would give a range starting 11111100 to 11111101 or in hex: FC to FD

When using a ULA it is normal to generate a unique address with a /48 prefix and then create your own subnets within the 4^th hex word. The first 48 bits, or three words,being your ULA and bits 49-64, the fourth word, used for your subnet. This allows for a /64 bit prefix and use of MAC addresses to represent the 64 bits of the host.

To generate your own ULA use one of you MAC address from a network you own, (Network Card Serial Number), and use the online calculator to generate your /48 bit prefix: http://www.sixxs.net/tools/grh/ula/

It does not matter which MAC address you choose to use in the calculator, it is just a method of ensuring a unique input. Even though ULAs do not connect directly to the internet, there is just no need to use the same addresses on every network in the way we have become used to with the 192.168 network in IPv4.

You can see from the output of the calculator above the prefix FD81:2043:94E6::/48 has been created for my use. With the complete use of the 4^thword (bit 49 – 64), this would give me 65,000 subnets I could use if I opted for the /64 prefix.

FD81:2043:94E6::/64

All Hosts

There is no broadcast address for the network but we can access “all hosts” though the Multicast Address ff02::1

On linux if we use the command:

Ping6 –I eth0 ff02::1 we will see a response from each host

On Windows if we use the command:

Ping -6 ff02::1 we will see the response from the multicast group.

All Routers

Especially in smaller offices or home networks there is no need to configure a gateway for your clients as there is a multicast group for all routers: ff02::2. If the group exists on your network then you have an IPv6 Router. Again we can check for the existence of such devices by pinging the multicast address.

In short IPv6 will save us time and effort and these are only a few of the new features that are incorporated in the new protocol. We will need to learn more but the sooner the start the sooner we will become familiar with the new features within IPv6.

Tuesday, June 7, 2011

Undelete Files on an ext3 or ext4 File System

So you’ve been futzing round in the file system, and been over vigorous with the rm command and deleted a crucial file that you or (more scarily) a significant other, can’t live without. What to do? After that initial hot flush has died down, you must be calm, and work fast. You have three options:

if the file is still open in a running application

To be honest, I can’t imagine this happening a lot, but it’s possible. The scenario is this: you’re editing/using a file in an application, and whilst the application is still open, you delete the file. If this happens to you, then recovery of the file is pretty simple using a tool that should already be installed on your system. So:

step1:

Do not close any applications!

step2:

Open a terminal, and type:

lsof | grep "/path/to/file"

If the file is being used by a running application, then you should get something like this:

progname 5559 user 22r REG 8,5 1282410 1294349 /path/to/file

If you get no output, then the file isn’t being used by a running application, and you’ll need to use one of the other methods.

step3:

Looking at the output from step2, you’ll need the numbers from the 2nd and 4th column (ie. 5559 and 22r). With the second number, drop the “r” (leaving “22″). Now type the following command:

cp /proc/5559/fd/22 /path/to/restored.file

And voila, your file should be back in the form of /path/to/restored.file.
Easy.

using extundelete

Extundelete is a very useful little program for restoring deleted files. It should be able to restore files from ext2, ext3 and ext4 partitions. Restoring ext3 and ext4 is a very difficult process, because the journalling part of the file system deletes the information that points to where a file’s data is on the disk when it’s deleted, unlike the ext2 system, which leaves it in place. But fortunately for us, very often, that pointer information is still in the file system’s journal, and therefore this information can be used to find the data on the disk and restore it. So, with that said, on with the show.
First you’ll need to install extundelete. If your distro doesn’t have a pre-built package, then you’ll have to compile it from source. You can get the source from here. If you need help compiling a tarball, check out this page.

step1:

As soon as possible, you need to mount the relevant partition as read-only. The sooner you do this the better, so that there is no risk of another program writing data over the top of your recently deleted file. To do this, open a terminal, and type:

mount -o remount,ro /dev/partition

The partition will be something like /dev/sda2 or /dev/sdb1. You can find the relevant device by typing mount in a terminal. You’ll get an output like:

matt@mailserver:~$ mount
/dev/sdb1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
none on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
none on /var/lib/ureadahead/debugfs type debugfs (rw,relatime)
/dev/sdc1 on /media/disk type vfat (rw,noexec,nosuid,nodev)
/dev/sdb2 on /home type ext3 (rw)
/dev/sda1 on /home/matt/storage type ext3 (rw)
/dev/sdd1 on /media/temp type vfat (rw)
/dev/sde1 on /media/disk1 type vfat (rw)
/dev/sdf1 on /media/disk2 type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096)

So, if I want to make my /home partition read-only, it would be /dev/sdb2. Once you’ve mounted it read-only, you can relax a bit. At least you won’t be risking making the situation worse.

step2:

Now you can use the extundelete command to restore your file. extundelete can be used to restore all deleted files from a partition, or a specific file or directory. In this example we’ll just be restoring a specific file, but for more options, look here. To restore the deleted file, type the following into a terminal:

extundelete --restore-file /path/to/deleted/file

step3:

extundelete (if it works) should restore the file to a subdirectory in the current directory called RECOVERED_FILES. Voila – problem solved.

using photorec

If extundelete doesn’t work, then you can try photorec. Photorec works in a different way to extundelete. Instead of trying to find the information that points to where the deleted file data is on the disk, it tries to find the data by parsing the data itself to identify files. This method is less targeted than extundelete, but may still work if the information pointing to the file has been deleted from the journal. Photorec can find deleted files of a particular type. So if you’ve deleted a .mp3 file, you would get Photorec to find all deleted .mp3 files and hope that it finds the file you’re looking for. Photorec seems to be well supported on the main distros, so you should be able to find a package for easy install. So:

step1:

Rather than list the process in detail here, there is a very good explanation of the steps on the photorec wiki.
That’s it. Happy undeleting and remember prevention is always better than the cure. Backup.

Sunday, June 12, 2011

How To Set Up Software RAID1 On A Running System (Incl. GRUB2 Configuration) (Debian Squeeze)

1 Preliminary Note

2 Installing mdadm

3 Preparing /dev/sdb

4 Creating Our RAID Arrays

5 Adjusting The System To RAID1

6 Preparing GRUB2 (Part 1)

7 Preparing /dev/sda

8 Preparing GRUB2 (Part 2)

9 Testing

10 Links

Thursday, June 9, 2011

Using and Understanding IPv6 on openSUSE 11

IPv6 Addresses

Main Points

Why use IPv6

Global Addresses

Local Host

Link-Local Addresses

Unique Local Addresses

All Hosts

All Routers

Tuesday, June 7, 2011

Undelete Files on an ext3 or ext4 File System

if the file is still open in a running application

step1:

step2:

step3:

using extundelete

step1:

step2:

step3:

using photorec

step1:

Sameh Attia

Followers

About Me

Blog Archive