Thursday, June 9, 2016

How to effectively clear your bash history

http://www.techrepublic.com/article/how-to-effectively-clear-your-bash-history

If you're serious about security on your Linux machines, you might want to clear the bash history. Learn how to do this more effectively than with just a single command.


Image: Jack Wallen
On your Linux machines, a history of your bash commands is retained. This is great when you need to repeat a command or can't remember exactly how you executed a command in a previous session. However, this can also be seen as a security issue. What if someone gains access to your machine, opens a terminal window, and checks through your bash history to see what commands you've run?
Bash has a handy way to clear the history: issue the command history -c. There's a slight problem with that approach. Let me explain.

First off, your bash history is retained in the file ~/.bash_history. When you have a terminal open, and you issue a command, it writes the command to the history file. So issuing history -c will clear the history from that file. The problem comes about when you have multiple terminal windows open. Say you have two terminal windows open and you issue history -c from the first one and close that window. You then move to the second terminal window, and you type the exit command to close that window. Because you had a second bash window open, even after running the history -c command in the first, that history will be retained. In other words, the history -c command only works when it is issued from the last remaining terminal window.
How do you get around that? You empty the .bash_history file either on a per-instance basis or by using a crontab job to do it regularly. If security is a serious matter for you, consider setting up the crontab job. Here's how.
SEE: Linux Foundation launches badge program to boost open source security (ZDNet)

Clearing bash history on a regular basis

Before I show how to set up the crontab job for this, know that the ~/.bash_history file can be cleared with the command:
cat /dev/null > ~/.bash_history
That will empty out the contents of the file, but keep the file in place.
Let's say you want to clear the .bash_history file for user olivia (who administers your Linux server) at 11:00 p.m. every day. You would create a cron job under the olivia account. To do that, log in as the user olivia, open a terminal window, and issue the command crontab -e. When the crontab editor opens, enter the following:
00 23 * * * cat /dev/null > ~/.bash_history
Save that file and cron will start clearing out olivia's history at 11:00 p.m. every day.

A surefire method

This is a surefire method of clearing out your bash history. Don't always rely on the history -c command, because you never know when a second (or a third) terminal is still open, ready to keep that history retained.

Gathering insights from data: An overview of the Elastic stack

https://opensource.com/life/16/6/overview-elastic-stack

Gathering insights from data: An overview of the Elastic stack
Image credits : 
Tony Smith via Flickr (CC BY 2.0)
The Elastic stack is a versatile collection of open source software tools that make gathering insights from data easier. Formerly referred to as the ELK stack (in reference to Elasticsearch, Logstash, and Kibana), the growing list of tools that integrate with the platform (such as Beats) have outgrown the acronym but provide ever-growing capability for users and developers alike.
At the upcoming Southeast Linuxfest 2016, I'll be covering some of the steps to get started using each of these parts of the stack. In this article, we'll look at each in turn to summarize the capabilities, requirements, and interesting use cases that apply to each.

Beats & Logstash

Collecting data is where it all starts. Logstash and Beats both serve this need, though each is finely tuned to suit different needs and uses. Whether your priority is light resource usage or extensive features, either solution has been designed to meet those requirements.

Beats

Beats are lightweight, fast data shippers and collectors that typically do one thing and do it well. For example, Packetbeat collects and ships data about packet activity on a network interface, while Filebeat can tail a log file and send logs to be processed downstream. Beats are designed to be fast, portable, and easy to deploy on individual hosts for specific needs. There are even community-created Beats for uses like monitoring HTTP endpoints and NGINX health.
Written in Go and based upon the shared foundation of libbeat, Beats are meant to be easily implemented so that even resource-constrained environments can be measured easily and without much overhead.

Logstash

Logstash is similarly capable of collecting machine data, but where it shines is the plethora of open source plugins available to enrich data. For example, while collecting webserver logs is useful, deeply parsing the user-agent data to extract traffic statistics can be beneficial, which the useragent filter can do. Or, if using the Twitter plugin, you may want to perform sentiment analysis on user tweets.
Custom plugins are simple Ruby libraries, which enable users to extend functionality and prototype new features quickly. Performance isn't an afterthought, however: Logstash ships with JRuby by default, which opens up possibilities for concurrency and real threads.

Elasticsearch

Once data has been collected and enriched, storing it in Elasticsearch opens up a range of possibilities.
Elasticsearch is a search engine at its heart, with a myrid of use cases borne of its flexibility and ease of use. Based on Apache Lucene, Elasticsearch strives to make both the operational challenges (such as scalability and reliability) and application-based needs (like freetext search and autocomplete) easier for end users.
Operationally, Elasticsearch's elasticity stems from splitting indices into shards, which can be spread across multiple hosts to balance load and boost performance. With planning, this means that datasets can grow well beyond the capabilities of one machine to handle.
Some examples of the analytics that can be performed on Elasticsearch include:
  • Geo searching. When documents are inserted with geo metadata, results can be overlaid on a map to visualize how documents relate to real-world longitudes and latitudes.
  • Graphing. Recent plugins have added graph search to Elasticsearch, which can answer interesting questions such as relationships between data in the Panama Papers.
  • Aggregations. Answering questions such as which pages returned the most 500 errors become a matter of forming the right query for the fields in a log file.
While all these capabilities are driven by Elasticsearch's core, exposing them in a user-friendly interface is left to the next layer in the stack.

Kibana

Kibana is a browser-based visualization frontend for Elasticsearch. It enables users to easily consume data in aggregate that would otherwise be difficult to process; making logs, metrics, and unstructured data searchable and more usable for humans. Additional plugins can be used for specialized cases, such as Timelion for timeseries data.
Because Kibana persists most of its data within Elasticsearch, managing Kibana dashboards and visualizations is a similar exercise as managing other indices in Elasticsearch. Charts, graphs, and other visualizations sit atop Elasticsearch APIs which can be easily inspected for closer analysis or use in other systems.

An open ecosystem

Like other multi-faceted systems, the Elastic stack is supported by tools to help manage deployment and configuration, such as with Ansible, Puppet, and Chef. Standard distribution repositories are similarly available.
It's worth noting that all of these open source projects are spread across a wide array of languages. While Beats are written in Go for portable, efficient distribution of compiled binaries, Kibana uses Javascript for unified development of frontend and backend components. With many languages and an open codebase, users should feel free to get involved with the feature development and bugfixing efforts that matter to them.
Now is a great time to play with the stack and see what you can accomplish with it, with a wide variety of solutions and global community standing by to support users and developers alike.

3 Emerging Open Source Data Analytics Tools Beyond Apache Spark

https://www.linux.com/news/3-emerging-open-source-data-analytics-tools-beyond-apache-spark

Projects

These data analytics projects are on the rise: Apache Grappa, Apache Drill, and Apache Kafka.
On the data analytics front, profound change is in the air, and open source tools are leading many of the changes. Sure, you are probably familiar with some of the open source stars in this space, such as Hadoop and Apache Spark, but there is now a strong need for new tools that can holistically round out the data analytics ecosystem. Notably, many of these tools are customized to process streaming data.
The Internet of Things (IoT), which is giving rise to sensors and other devices that produce continuous streams of data, is just one of the big trends driving the need for new analytics tools. Streaming data analytics are needed for improved drug discovery, and NASA and the SETI Institute are even collaborating to analyze terabytes of complex, streaming deep space radio signals.
While Apache Spark grabs many of the headlines in the data analytics space, given billions of development dollars thrown at it by IBM and other companies, several unsung open source projects are also on the rise. Here are three emerging data analytics tools worth exploring:

Grappa

Big organizations and small ones are working on new ways to cull meaningful insights from streaming data, and many of them are working with data generated on clusters and, increasingly, on commodity hardware. That puts a premium on affordable data-centric approaches that can improve on the performance and functionality of tools such as MapReduce and even Spark. Enter the open source Grappa project, which scales data-intensive applications on commodity clusters and offers a new type of abstraction that can beat classic distributed shared memory (DSM) systems.
You can get the source code for Grappa and find more about it here. Grappa began when a group of engineers with experience running Big Data jobs on Cray systems wondered if they could challenge the analytics that Cray systems were capable of on off-the-shelf commodity hardware.
As the developers note: “Grappa provides abstraction at a level high enough to subsume many performance optimizations common to data-intensive platforms. However, its relatively low-level interface provides a convenient abstraction for building data-intensive frameworks on top of. Prototype implementations of (simplified) MapReduce, GraphLab, and a relational query engine have been built on Grappa that out-perform the original systems.”
Grappa is freely available on GitHub under a BSD license. If you are interested in seeing Grappa at work, you can follow easy quick-start directions in the application’s README file to build and run it on a cluster. To learn how to write your own Grappa applications, check out the tutorial.

Apache Drill

The Apache Drill project is making such a difference in the Big Data space that companies such as MapR have even wrapped it into their Hadoop distributions. It is a Top-Level project at Apache and is being leveraged along with Apache Spark in many streaming data scenarios.
For example, at the New York Apache Drill meeting back in January of this year, MapR system engineers showed how Apache Spark and Drill could be used in tandem in a use case involving packet capture and near-real-time query and search.
Drill is notable in streaming data applications because it is a distributed, schema-free SQL engine. DevOps and IT staff can use Drill to interactively explore data in Hadoop and other NoSQL databases, such as HBase and MongoDB. There is no need to explicitly define and maintain schemas, as Drill can automatically leverage the structure that's embedded in the data. It is able to stream data in memory between operators, and minimizes the use of disks unless needed to complete a query.

Apache Kafka

The Apache Kafka project has emerged as a star for real-time data tracking capabilities. It provides unified, high-throughput, low-latency processing for real-time data. Confluent and other organizations have also produced custom tools for using Kafka with data streams.
Apache Kafka was originally developed by LinkedIn, and was subsequently open sourced in early 2011. It is a hardened, tested tool and many organizations require workers with Kafka knowledge. Cisco, Netflix, PayPal, Uber, and Spotify are among well-known companies using Kafka.
The engineers who created Kafka at LinkedIn also created Confluent, which focuses on Kafka. Confluent University offers training courses for Kafka developers, and for operators/administrators. Both onsite and public courses are available.
Are you interested in more unsung open source data analytics projects on the rise? If so, you can find more in my recent post on the topic.