If you're serious about security on your Linux machines, you might
want to clear the bash history. Learn how to do this more effectively
than with just a single command.
On your Linux machines, a history of your bash commands is retained.
This is great when you need to repeat a command or can't remember
exactly how you executed a command in a previous session. However, this
can also be seen as a security issue. What if someone gains access to
your machine, opens a terminal window, and checks through your bash
history to see what commands you've run?
Bash has a handy way to clear the history: issue the command history -c. There's a slight problem with that approach. Let me explain.
First off, your bash history is retained in the file ~/.bash_history. When you have a terminal open, and you issue a command, it writes the command to the history file. So issuing history -c will clear the history from that file. The problem comes about when you have multiple terminal windows open. Say you have two terminal windows open and you issue history -c
from the first one and close that window. You then move to the second
terminal window, and you type the exit command to close that window.
Because you had a second bash window open, even after running the history -c command in the first, that history will be retained. In other words, the history -c command only works when it is issued from the last remaining terminal window.
How do you get around that? You empty the .bash_history
file either on a per-instance basis or by using a crontab job to do it
regularly. If security is a serious matter for you, consider setting up
the crontab job. Here's how. SEE:Linux Foundation launches badge program to boost open source security (ZDNet)
Clearing bash history on a regular basis
Before I show how to set up the crontab job for this, know that the ~/.bash_history file can be cleared with the command: cat /dev/null > ~/.bash_history
That will empty out the contents of the file, but keep the file in place.
Let's say you want to clear the .bash_history
file for user olivia (who administers your Linux server) at 11:00 p.m.
every day. You would create a cron job under the olivia account. To do
that, log in as the user olivia, open a terminal window, and issue the
command crontab -e. When the crontab editor opens, enter the following: 00 23 * * * cat /dev/null > ~/.bash_history
Save that file and cron will start clearing out olivia's history at 11:00 p.m. every day.
A surefire method
This is a surefire method of clearing out your bash history. Don't always rely on the history -c command, because you never know when a second (or a third) terminal is still open, ready to keep that history retained.
The
Elastic stack is a versatile collection of open source software tools
that make gathering insights from data easier. Formerly referred to as
the ELK stack (in reference to Elasticsearch, Logstash, and Kibana), the growing list of tools that integrate with the platform (such as Beats) have outgrown the acronym but provide ever-growing capability for users and developers alike.
At the upcoming Southeast Linuxfest 2016,
I'll be covering some of the steps to get started using each of these
parts of the stack. In this article, we'll look at each in turn to
summarize the capabilities, requirements, and interesting use cases that
apply to each.
Beats & Logstash
Collecting data is where it all starts. Logstash and Beats both serve
this need, though each is finely tuned to suit different needs and
uses. Whether your priority is light resource usage or extensive
features, either solution has been designed to meet those requirements.
Beats
Beats are lightweight, fast data shippers and collectors that
typically do one thing and do it well. For example, Packetbeat collects
and ships data about packet activity on a network interface, while
Filebeat can tail a log file and send logs to be processed downstream.
Beats are designed to be fast, portable, and easy to deploy on
individual hosts for specific needs. There are even community-created
Beats for uses like monitoring HTTP endpoints and NGINX health.
Written in Go and based upon the shared foundation of libbeat,
Beats are meant to be easily implemented so that even
resource-constrained environments can be measured easily and without
much overhead.
Logstash
Logstash is similarly capable of collecting machine data, but where
it shines is the plethora of open source plugins available to enrich
data. For example, while collecting webserver logs is useful, deeply
parsing the user-agent data to extract traffic statistics can be
beneficial, which the useragent filter can do. Or, if using the Twitter plugin, you may want to perform sentiment analysis on user tweets.
Custom plugins are simple Ruby libraries, which enable users to
extend functionality and prototype new features quickly. Performance
isn't an afterthought, however: Logstash ships with JRuby by default,
which opens up possibilities for concurrency and real threads.
Elasticsearch
Once data has been collected and enriched, storing it in Elasticsearch opens up a range of possibilities.
Elasticsearch is a search engine at its heart, with a myrid of use cases borne of its flexibility and ease of use. Based on Apache Lucene,
Elasticsearch strives to make both the operational challenges (such as
scalability and reliability) and application-based needs (like freetext
search and autocomplete) easier for end users.
Operationally, Elasticsearch's elasticity stems from splitting
indices into shards, which can be spread across multiple hosts to
balance load and boost performance. With planning, this means that
datasets can grow well beyond the capabilities of one machine to handle.
Some examples of the analytics that can be performed on Elasticsearch include:
Geo searching. When documents are inserted with geo
metadata, results can be overlaid on a map to visualize how documents
relate to real-world longitudes and latitudes.
Graphing. Recent plugins have added graph search to
Elasticsearch, which can answer interesting questions such as
relationships between data in the Panama Papers.
Aggregations. Answering questions such as which
pages returned the most 500 errors become a matter of forming the right
query for the fields in a log file.
While all these capabilities are driven by Elasticsearch's core,
exposing them in a user-friendly interface is left to the next layer in
the stack.
Kibana
Kibana is a browser-based visualization frontend for Elasticsearch.
It enables users to easily consume data in aggregate that would
otherwise be difficult to process; making logs, metrics, and
unstructured data searchable and more usable for humans. Additional
plugins can be used for specialized cases, such as Timelion for timeseries data.
Because Kibana persists most of its data within Elasticsearch,
managing Kibana dashboards and visualizations is a similar exercise as
managing other indices in Elasticsearch. Charts, graphs, and other
visualizations sit atop Elasticsearch APIs which can be easily inspected
for closer analysis or use in other systems.
An open ecosystem
Like other multi-faceted systems, the Elastic stack is supported by
tools to help manage deployment and configuration, such as with Ansible, Puppet, and Chef. Standard distribution repositories are similarly available.
It's worth noting that all of these open source projects are spread
across a wide array of languages. While Beats are written in Go for
portable, efficient distribution of compiled binaries, Kibana uses
Javascript for unified development of frontend and backend components.
With many languages and an open codebase, users should feel free to get
involved with the feature development and bugfixing efforts that matter
to them.
Now is a great time to play with the stack and see what you can
accomplish with it, with a wide variety of solutions and global
community standing by to support users and developers alike.
On
the data analytics front, profound change is in the air, and open
source tools are leading many of the changes. Sure, you are probably
familiar with some of the open source stars in this space, such as
Hadoop and Apache Spark, but there is now a strong need for new tools
that can holistically round out the data analytics ecosystem. Notably,
many of these tools are customized to process streaming data.
The Internet of Things (IoT), which is giving rise to sensors and
other devices that produce continuous streams of data, is just one of
the big trends driving the need for new analytics tools. Streaming data
analytics are needed for improved drug discovery, and NASA and the SETI
Institute are even collaborating to analyze terabytes of complex,
streaming deep space radio signals.
While Apache Spark grabs many of the headlines in the data analytics
space, given billions of development dollars thrown at it by IBM and
other companies, several unsung open source projects are also on the
rise. Here are three emerging data analytics tools worth exploring:
Grappa
Big organizations and small ones are working on new ways to cull
meaningful insights from streaming data, and many of them are working
with data generated on clusters and, increasingly, on commodity
hardware. That puts a premium on affordable data-centric approaches that
can improve on the performance and functionality of tools such as
MapReduce and even Spark. Enter the open source Grappa project,
which scales data-intensive applications on commodity clusters and
offers a new type of abstraction that can beat classic distributed
shared memory (DSM) systems.
You can get the source code for Grappa and find more about it here.
Grappa began when a group of engineers with experience running Big Data
jobs on Cray systems wondered if they could challenge the analytics
that Cray systems were capable of on off-the-shelf commodity hardware.
As the developers note: “Grappa provides abstraction at a level high
enough to subsume many performance optimizations common to
data-intensive platforms. However, its relatively low-level interface
provides a convenient abstraction for building data-intensive frameworks
on top of. Prototype implementations of (simplified) MapReduce,
GraphLab, and a relational query engine have been built on Grappa that
out-perform the original systems.”
Grappa is freely available on GitHub
under a BSD license. If you are interested in seeing Grappa at work,
you can follow easy quick-start directions in the application’s README
file to build and run it on a cluster. To learn how to write your own
Grappa applications, check out the tutorial.
Apache Drill
The Apache Drill project is making such a difference in the Big Data
space that companies such as MapR have even wrapped it into their Hadoop
distributions. It is a Top-Level project at Apache and is being
leveraged along with Apache Spark in many streaming data scenarios.
For example, at the New York Apache Drill meeting back in January of
this year, MapR system engineers showed how Apache Spark and Drill could
be used in tandem in a use case involving packet capture and
near-real-time query and search.
Drill is notable in streaming data applications because it is a
distributed, schema-free SQL engine. DevOps and IT staff can use Drill
to interactively explore data in Hadoop and other NoSQL databases, such
as HBase and MongoDB. There is no need to explicitly define and maintain
schemas, as Drill can automatically leverage the structure that's
embedded in the data. It is able to stream data in memory between
operators, and minimizes the use of disks unless needed to complete a
query.
Apache Kafka
The Apache Kafka project has emerged as a star for real-time data
tracking capabilities. It provides unified, high-throughput, low-latency
processing for real-time data. Confluent and other organizations have also produced custom tools for using Kafka with data streams.
Apache Kafka was originally developed by LinkedIn, and was
subsequently open sourced in early 2011. It is a hardened, tested tool
and many organizations require workers with Kafka knowledge. Cisco,
Netflix, PayPal, Uber, and Spotify are among well-known companies using
Kafka.
The engineers who created Kafka at LinkedIn also created Confluent,
which focuses on Kafka. Confluent University offers training courses for
Kafka developers, and for operators/administrators. Both onsite and public courses are available.
Are you interested in more unsung open source data analytics projects on the rise? If so, you can find more in my recent post on the topic.