The Phoronix Test Suite has more than 120 test profiles and 50 test suites with new suites and tests continuing to come in through its extensible architecture, but for organizations with multiple test systems or entire testing farm(s) devoted to performance monitoring and regression tracking, they have had to strap the Phoronix Test Suite atop their own management systems or hack away at simple scripts to deploy our testing software across an array of systems.
Today though we are announcing the public beta launch of Phoromatic. Phoromatic is a remote test management system that allows controlling any number of PTS-powered systems through a single web-based interface, which also allows all of the test results to be viewed from a central source.
This article provides a first-look and guide for some of the possibilities of Phoromatic from those looking to build a benchmarking test farm or for individuals simply wanting to benchmark computers across the world. We also share in this article what may be coming next to our Linux-focused benchmarking empire.
Phoromatic was originally mentioned during the Phoronix Test Suite 2.0 development cycle, but the web-based service has not gone public in beta until this morning.
The Phoromatic module in the Phoronix Test Suite 2.2 client provides a few updates to the Phoromatic support and interested users are encouraged to use the just-released Bardu Beta 2 or newer. This was pushed back from 2.0 Sandtorg due to strapped development resources, waiting for a few internal pts-core API changes in 2.2 Bardu, and then awaiting feedback from various PTS-using companies.
The initial public beta of Phoromatic at Phoromatic.com is free and available to anyone to use. This version provides all of the core functionality needed for automating timed, recurring tests across any number of test systems, while many more features will be flipped on and implemented over the coming weeks and months as we receive feedback from its users.
Like all other Phoronix Test Suite components, Phoromatic was designed from the ground-up to be able to handle automated performance testing/profiling in a clean, reproducible, and easy-to-use manner with turnkey deployment capabilities.
Whether you are just an individual with a single system or a couple of Linux/BSD/OpenSolaris/MacOSX systems that you want to benchmark or are an IHV/ISV with an entire test farm, setting up Phoromatic is extremely quick and easy.
Once creating a Phoromatic account, two key codes are shown from the web-based control panel. With the Phoronix Test Suite (2.2 Beta or later is recommended, ideally the Git package) installed on the test systems, simply run phoronix-test-suite module-setup phoromatic on each of the test nodes and enter these two codes, which associates that test system with the account.
Following that, phoronix-test-suite phoromatic.start just needs to be run (or added to any boot scripts, if you wish) on the test node(s) for it to begin communicating with the Phoromatic server and to receive installation commands, test run commands, etc. It is that easy, and after that point, it is all handled from our web-based interface.
The beta version at Phoromatic.com currently supports setting up recurring test schedules with a schedule consisting of a time at which the testing should begin, what day(s) of the week the schedule should run, and what test system(s) to target (and an option for all test systems) with this schedule.
Once a schedule has been created, any number of tests and/or suites can be added to that schedule. These tests/suites with any specified arguments will then be run at the specified time on the given systems, assuming the phoronix-test-suite phoromatic.start command is running on the test systems.
It is as easy as that. Multiple test schedules are supported and any combination of tests. Once tests being run through Phoromatic are completed, the results are immediately available from the Phoromatic web interface.
There is also settings within Phoromatic for being notified via email when new test results are available, to also upload the results to Phoronix Global, etc.
The Phoromatic web interface also supports comparing multiple test results all from your web browser, whether it is to compare the results from a same schedule across multiple systems, or to look at how the performance has changed of a test system over the course of time.
To compare multiple test results, simply click on the tag icon in the front of each results row and then click on the header above that table (i.e. "Latest Test Results").
Features like being able to analyze batch runs, line graphs showing how the performance of test system(s) with a given test schedule have changed over the course of time, an option to issue a test call when a new revision control commit is made to a particular project (i.e. new kernel or X Server commit in Git) rather than on a timed basis (event-driven testing), and other statistical/analytical features will also be turned on within Phoromatic in the near future.
For now we are ensuring all of the core functionality is solid and to see what features the Phoromatic users are most interested in utilizing.
Various other features are also on the way such as the ability to clone results from a private Phoromatic repository, advanced search / sorting capabilities, and an option for anonymous uploads to a public Phoromatic repository.
The enterprise version of Phoromatic will also allow the Phoromatic server to be installed within a corporate network, to support multiple user accounts for managing the web-interface, and features to better integrate Phoromatic within an already existing in-house testing environment.
From the web-interface, Phoromatic also shows all linked-up test systems along with their installed software/hardware, which is maintained in real-time and shown are any test errors/warnings generated on that test system along with the current tasks of that test system.
The core of Phoromatic is also going to be used to power the next-generation version of Phoronix Global, which will launch in 2010 with a plethora of new features and improvements, and picking up some of the Phoromatic features like web-based results comparison, AJAX-driven menus, sort-able tables, email notifications, better search capabilities, and other analytical capabilities.
Whether you are just an enthusiast using the Phoronix Test Suite on a couple of systems or are a company looking to automate the scheduled execution of Phoronix Test Suite tests across a multitude of systems, we invite you to try out this public beta of Phoromatic.com and to supply feedback to us as we have a platter of new features to still implement.
With Phoromatic, it should be incredibly easy to assemble your own "benchmarking test farm" and to enjoy its robust set of features.
As another step in our global benchmarking crusade, another Phoronix Test Suite product currently in brainstorming is codenamed "Phorotrack".
Phorotrack is a plug-in for Phoromatic to provide additional features and options for tracking the performance of a particular software component over time.
As part of Phorotrack, we would also be looking to offer public performance tracking figures for key software projects and distributions.
This would be using our own in-house test farm at Phoronix.com both on real hardware and in a virtualized environment -- to effectively reach a point of "cloud benchmarking" for regression monitoring.
Projects using Phorotrack could then connect in and request particular test runs from the available Phoronix Test Suite profiles, which then would begin running routinely from our farm and report the results back to them, in order to aide them with monitoring of important and relevant performance metrics.
Outside individuals could also connect in to donate computing power towards performance testing for a particular project, effectively like Folding@HOME but for community benchmarking of open-source software.
In fact, already later this month as an experiment we may begin tracking the performance of a few key Linux distributions with their nightly builds on a daily basis through Phoromatic and this add-on.
Phoronix Test Suite 2.2 with Phoromatic support and many other new features (improved statistical significance, network proxy support, test recovery support, autonomous Git bisecting support of performance regressions, an improved GTK GUI, etc) will be officially released in November.
PTS Desktop Live 2009.4 is also on the way, and of course there's even more exciting work coming down the pipe in 2010 -- Windows 7 benchmarking support could even come.
No comments:
Post a Comment