Tuesday, June 21, 2016

From MySQL To NoSQL! How To Migrate Your MySQL Data To MongoDB Using Mongify Utility


Welcome again. Big data is here and therefore there needs to be a solution to store such kind of data in a database which is independent of the boundaries of normalization and relationships. RDBMS is no longer a great solution for storing big data. And that is why noSQL databases are now needed everywhere. Today, I am going to explain how the mongify utility can be used to migrate a database from MySQL to MongoDB. But before we jump into it, let me share with you little background information:

Introduction to MySQL

MySQL is an open source relational database management system (RDBMS) which uses the Structured Query Language (SQL) as a mechanism for dealing and interacting with the data. Although MySQL is one of the widely used and well known database management systems and is considered as reliable, scalable and efficient database management system, It is NOT well suited for handling big data and especially with HUGE insertion rates.

Introduction to MongoDB

MongoDB server is an opensource document database which stores data in JSON (which is a key:value) format. It has no db schemas filled with joins and relationships and is highly recommended as backend for web applications where huge volume of data is inserted and processed in real time.

When to Use MongoDB and When Not?

If you need a flexible database solution with no strict schema and expect a very high insert rate into your database also if reliability and security is of less concern for you then you can go for MongoDB. While on the other hand when security and reliability is of prime concern and you do not expect very huge write transactions into your database then you may use MySQL or any other RDBMS.

Introduction to Mongify

Mongify is a utility (or a ruby gem ) written in the ruby language and is used to migrate databases from SQL to mongodb. Further detailed information about ruby language and ruby gems can be found on their corresponding websites. Mongify utility migrates databases while not caring about primary keys and foreign keys as in case of an RDBMS. It supports data migration from MySQL, SQLite and other relational databases however this article only focuses on migrating data from MySQL to MongoDB.

Install Ruby if not already installed

As mentioned earlier, the mongify utility is based on ruby language therefore we need to install ruby if it is not already present on the sytem.
The following command can be used to install ruby on Ubuntu systems:
 apt-get install ruby
Below screen displays a typical output of this command:

Install ‘gem’ Package

Once ruby has been installed successfully, the next step is to install the ‘gem’ package which itself is the ruby gem manager. We will use the below command to achieve this:
apt-get install gem
The output for this command should be something list below:

Install Other Dependencies If Not Already Installed

Once these packages are installed, we need to complete a few more prerequisite packages to install and run mongify. These package dependencies are mentioned as below:
  1. ruby-dev
  2. mongodb
  3. libmysqlclient-dev
Besides these packages there are a few ‘gems’ needed as run time dependencies. These runtime dependencies include (at least):
  1. activerecord
  2. activesupport
  3. bson
  4. bson_ext
  5. highline
  6. mongo
Once all these dependencies are met, we are good to go for installing the mongify gem.

Install ‘mongify’ gem

The below command can be used to install the mongify utility:
sudo gem install mongify
The output for this command may look like something below:

Create a database.config file

Next, we need to create a database configuration file. This configuration file will contain the details and credentials for MySQL database and the MongoDB. Here we need to make sure that the correct database name, username and password are used for the MySQL database that we need to migrate.
The contents of the database.config may look similar to as shown in the following screenshot:

Check if Database Config is Correct

Next, we can check if the newly created database.config file is correct. We can use below command:
 mongify check database.config
If everything is alright, the output for this command can be something like this:

Create a Database Translation File

Now if the configuration file is correct, we can proceed to the next step which is to create a translation file.
We will use the below command to create a translation file:
mongify translation database.config >> translation.rb
The output for this command should be something like below:
We are almost done! But wait, one more step is needed and that is the actual step which will migrate the database for us.

Process the Translation File

This will be the step which will process the translation file and will create a new database in Mongodb for us. We will use below command :
mongify process database.config translation.rb
And the output should be something like below:
Congratulations! We have successfully migrated our database named ‘cloud’ from MySQL to Mongodb. This can be confirmed within the mongo shell by running below command:
$ mongo
>> db.stats()
The output for this command should be something like this:
In the above screenshot the details about our newly migrated database are displayed. It contains the database name, total number of tables (collections) and other details.


In this article we demonstrated how can we use the mongify utility to migrate an existing MySQL database to MongoDB. If you like this article or if you have any queries regarding the procedure, you are most welcome to share your comments and feedback here. We will come back with a new topic soon. Happy reading!

How to select the fastest apt mirror on Ubuntu Linux


The following guide will provide you with some information on how to improve Ubuntu's repository download speed by selecting the closest, that is, possibly fastest mirror relative to your geographical location.

1. Country Code

The simplest approach is to make sure that your Ubuntu mirror defined within /etc/apt/sources.list includes a relevant country code appropriate to your location. For example, below you can find a official United States Ubuntu mirror as found in /etc/apt/sources.list:
deb http://us.archive.ubuntu.com/ubuntu/ xenial main restricted
If you are not located in United States simply overwrite the us country code with appropriate code of your country. That is, if your are located for example in Australia update your /etc/apt/sources.list file for all entries as:
deb http://au.archive.ubuntu.com/ubuntu/ xenial main restricted

2. Use mirror protocol

Using mirror protocol as part of your /etc/apt/sources.list entry will instruct apt command to fetch mirrors located within your country only. In order to use mirror protocol update all lines within /etc/apt/sources.list file from the usual eg.:
deb http://us.archive.ubuntu.com/ubuntu/ xenial main restricted
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted
Repeat the above for all relevant lines where appropriate. Alternatively, use sed command to automatically edit your /etc/apt/sources.list file. Update the below sed command where appropriate to fit your environment:
$ sudo sed -i -e 's/http:\/\/us.archive/mirror:\/\/mirrors/' -e 's/\/ubuntu\//\/mirrors.txt/' /etc/apt/sources.list

3. Manual apt mirror selection

The above solutions look easy and they might just work for you. However, the mirror selected by apt may not be the fastest as it can be burdened by high latency. In this case you may try to choose your mirror manually from the list of mirrors located within your country. Use wget command to retrieve the list. The below wget command will retrieve apt ubuntu mirrors related to your country. Example:
$ wget -qO - mirrors.ubuntu.com/mirrors.txt
Based on your experience select the best mirror and alter your /etc/apt/sources.list apt configuration file appropriately.

4. Choosing the fastest mirror with netselect

This solution is preferred, as it guarantees the fastest mirror selection. For this we are going to use netselect command. The netselect package is not available within Ubuntu's standard repository by default, so we will need to borrow it from Debian stable repository:
$ sudo apt-get install wget
$ wget http://ftp.au.debian.org/debian/pool/main/n/netselect/netselect_0.3.ds1-26_amd64.deb
$ sudo dpkg -i netselect_0.3.ds1-26_amd64.deb
Once you have the netselect command available on your Ubuntu system use it to locate the fastest mirror based on the lowest icmp latency. The netselect output will be relative to your location. The below example output will show top 20 apt Ubuntu mirrors ( if available ):
$ sudo netselect -s 20 -t 40 $(wget -qO - mirrors.ubuntu.com/mirrors.txt)
   12 http://ubuntu.uberglobalmirror.com/archive/
   20 http://ubuntu.mirror.serversaustralia.com.au/ubuntu/
   21 http://ubuntu.mirror.digitalpacific.com.au/archive/
   38 http://mirror.aarnet.edu.au/pub/ubuntu/archive/
   39 http://mirror.overthewire.com.au/ubuntu/
   45 http://mirror.internode.on.net/pub/ubuntu/ubuntu/
  121 http://mirror.netspace.net.au/pub/ubuntu/
  148 http://mirror.waia.asn.au/ubuntu/
  152 http://mirror.as24220.net/pub/ubuntu-archive/
  162 http://mirror.tcc.wa.edu.au/ubuntu/
  664 http://archive.ubuntu.com/ubuntu/
  664 http://archive.ubuntu.com/ubuntu/
 3825 http://archive.ubuntu.com/ubuntu/
Only found 13 hosts out of 20 requested.
Alter manually your /etc/apt/sources.list file to reflect the above netselect results or use sed command, where the lower score number on the left represents a higher mirror transfer rate. Example:
$ sudo sed -i 's/http:\/\/us.archive.ubuntu.com\/ubuntu\//http:\/\/ubuntu.uberglobalmirror.com\/archive\//' /etc/apt/sources.list

5. Comparing results

The following are my apt-get update command results, while located within Australia:
US MIRROR ( http://us.archive.ubuntu.com/ubuntu ):
Fetched 23.1 MB in 20s (1148 kB/s) 

MIRROR protocol( mirror://mirrors.ubuntu.com/mirrors.txt):
Fetched 23.1 MB in 4min 45s (81.0 kB/s)

AU MIRROR ( http://au.archive.ubuntu.com/ubuntu ):
Fetched 23.1 MB in 12s (1788 kB/s)

NETSTAT Auto-Selected ( http://ubuntu.uberglobalmirror.com/archive ):
Fetched 23.1 MB in 6s (3544 kB/s)

Active Directory Alternative For Linux : How To Install And Setup Resara Server On Linux


Resara Server is an Active Directory compatible open source Linux server for small businesses and simple networks. The management console lets you manage users, share files, and configure DHCP and DNS. Resara Server utilizes a technology called Samba, which is an open source implementation of the Active Directory framework. Although Samba is not actually Active Directory, it is designed to provide the same services and is compatible with almost all Active Directory components which provide network management services, such as
user authentication and computer management.
It is as a designed simple and easy to use system, here are the main features of Resara Server.
• Active Directory Compatible Domain with Samba 4
• User Management
• Computer Management
• DNS and DHCP Management
• Admin Console
• Backup System

Installing Resara Server:

To install and setup Resra Server you will be required an IP for the server, its FQDN, a default gateway, subnet mask and DNS server. Then download the installation media by following the Resara server Download Link . After downloading the installation media, boot your system from the downloaded ISO image and click on the Forward key to proceed to the Resara Installation setup.
Select the language and click on the ‘Forward’ button to move to the next step.
Resara installation
Choose your region and select the time zone from the available options and then click on the ‘forward’ key.
Time zone
Select your keyboard layout, if its other than your default.
keyboard layout
Here you need to select the hard disk to be used for installation. This will erase all the data on the disk, so make sure that no data is present on the disk. then select the ‘Forward’ button to move to the next option.
prepare disk space
Create your user name and password and move forward.
user settings
Review the installation summary before doing a click on the ‘Install’ button. Once you are OK with your selected options then click on the ‘Install’ key to start the installation process.
installation summary
Your installation process will be completing soon, just relax for a while and wait for completion.
installation progress
Once your installation complete, you will be asked to restart your computer. Disconnect your CD or ISO image and reboot your system .
restart your system
After system reboot you will be able to login to your Resara server by providing your user credentials that you have created earlier.
User Login

Resara Server Configurations:

As we have successfully installed Resara server, now we are going to start its configuration. This will guide you through the process of configuring and provisioning your Resara Server.
Resara Configurations

Network Configurations:

Set a permanent IP for your server including gateway and DNS settings. You can change the servers IP in the future via the Admin Console if necessary.
Network Configurations

Date and Time:

Set the time, date, and time zone for your server and make sure that the time between the server and client computers must be within 5 minutes of each other, Otherwise, they will not be able to join to the domain.
Date and time

Domain setup:

Configure the name of your server and domain to whatever is most appropriate for your network. The full domain
name will autofill based on what you have typed for your short domain name. But, your domain name must be unique to your organization.
domain settings

Admin Password:

Enter the admin password, for the administrator account that must contain one capital letter and a number. Once typed click on the Next button.
Admin Password

DHCP Server:

Resara Server can act as a DHCP server for your network. If you enabled this feature, then make sure you set an IP range that can communicate with the server, and also does not interfere with any other clients on your network.
dhcp setup

Server Provisioning:

Once your configurations are complete, it will go through the provisioning process that may take few minutes. You can check the Show Log box to watch what the server is doing.
Server Provisioning
Once the server has finished provisioning you can click the finished button, which will then launch the
Admin Console for further configuration of your server. Or, you can start joining computers to your domain
Finish configurations

Resara Server Admin console:

Welcome to the Resara server admin console. You can also launch it by clicking on the Admin Console icon on your Desktop, or in the Resara folder in the list of applications in your start menu.
Resara Admin Console
There are 7 sections available here such as Users, Computers, Shares, Storage, DHCP, DNS and Server. Administration of Resara Server is seperated into management tabs and each tab is responsible for a different administrative task.


Resara Server has been adopted by many types of organizations around the world. The open source Community Edition is popular among non-profits because it provides essential domain controller functionality at no cost. Larger non-profits and corporations choose the commercial version for support and scalability features, like server replication and load-balancing. This is one of the best tool that every Linux system administrator must learn and setup. Let’s give it a try and do share your comments and thoughts on its working and your experience about Resara Server. Thank you for reading.