NAME¶

innotop - MySQL and InnoDB transaction/status monitor.

SYNOPSIS¶

To monitor servers normally:

 innotop

To monitor InnoDB status information from a file:

 innotop /var/log/mysql/mysqld.err

To run innotop non-interactively in a pipe-and-filter configuration:

 innotop --count 5 -d 1 -n

To monitor a database on another system using a particular username and password:

 innotop -u <username> -p <password> -h <hostname>

DESCRIPTION¶

innotop monitors MySQL servers. Each of its modes shows you a different aspectof what's happening in the server. For example, there's a mode for monitoringreplication, one for queries, and one for transactions. innotop refreshes itsdata periodically, so you see an updating view.

innotop has lots of features for power users, but you can start and run it withvirtually no configuration. If you're just getting started, see"QUICK-START". Press '?' at any time while running innotop forcontext-sensitive help.

QUICK-START¶

To start innotop, open a terminal or command prompt. If you have installedinnotop on your system, you should be able to just type "innotop" and pressEnter; otherwise, you will need to change to innotop's directory and type "perlinnotop".

With no options specified, innotop will attempt to connect to a MySQL server onlocalhost using mariadb_read_default_group=client for other connectionparameters. If you need to specify a different username and password, use the-u and -p options, respectively. To monitor a MySQL database on anotherhost, use the -h option.

After you've connected, innotop should show you something like the following:

 [RO] Query List (? for help) localhost, 01:11:19, 449.44 QPS, 14/7/163 con/run
 CXN        When   Load  QPS    Slow  QCacheHit  KCacheHit  BpsIn    BpsOut
 localhost  Total  0.00  1.07k   697      0.00%     98.17%  476.83k  242.83k
 CXN        Cmd    ID         User  Host      DB   Time   Query
 localhost  Query  766446598  test  10.0.0.1  foo  00:02  INSERT INTO table (

(This sample is truncated at the right so it will fit on a terminal when running'man innotop')

If your server is busy, you'll see more output. Notice the first line on thescreen, which tells you that readonly is set to true ([RO]), what mode you'rein and what server you're connected to. You can change to other modes withkeystrokes; press 'T' to switch to a list of InnoDB transactions, for example.

Press the '?' key to see what keys are active in the current mode. You canpress any of these keys and innotop will either take the requested action orprompt you for more input. If your system has Term::ReadLine support, you canuse TAB and other keys to auto-complete and edit input.

To quit innotop, press the 'q' key.

OPTIONS¶

innotop is mostly configured via its configuration file, but some of theconfiguration options can come from the command line. You can also specify afile to monitor for InnoDB status output; see "MONITORING A FILE" for moredetails.

You can negate some options by prefixing the option name with --no. Forexample, --noinc (or --no-inc) negates "--inc".

--color

Enable or disable terminal coloring. Corresponds to the "color" config filesetting.

--config

Specifies a configuration file to read. This option is non-sticky, that is tosay it does not persist to the configuration file itself.

--count

Refresh only the specified number of times (ticks) before exiting. Each refreshis a pause for "interval" seconds, followed by requesting data from MySQLconnections and printing it to the terminal.

--delay

Specifies the amount of time to pause between ticks (refreshes). Corresponds tothe configuration option "interval".

--help

Print a summary of command-line usage and exit.

--host

Host to connect to.

--inc

Specifies whether innotop should display absolute numbers or relative numbers(offsets from their previous values). Corresponds to the configuration option"status_inc".

--mode

Specifies the mode in which innotop should start. Corresponds to theconfiguration option "mode".

--nonint

Enable non-interactive operation. See "NON-INTERACTIVE OPERATION" for more.

--password

Password to use for connection.

--port

Port to use for connection.

--skipcentral

Don't read the central configuration file.

--timestamp

In -n mode, write a timestamp either before every screenful of output, or ifthe option is given twice, at the start of every line. The format is controlledby the timeformat config variable.

--user

User to use for connection.

--version

Output version information and exit.

--write

Sets the configuration option "readonly" to 0, making innotop write therunning configuration to ~/.innotop/innotop.conf on exit, if no configurationfile was loaded at start-up.

HOTKEYS¶

innotop is interactive, and you control it with key-presses.

• Uppercase keys switch between modes.
• Lowercase keys initiate some action within the current mode.
• Other keys do something special like change configuration or show theinnotop license.

Press '?' at any time to see the currently active keys and what they do.

MODES¶

Each of innotop's modes retrieves and displays a particular type of data fromthe servers you're monitoring. You switch between modes with uppercase keys.The following is a brief description of each mode, in alphabetical order. Toswitch to the mode, press the key listed in front of its heading in thefollowing list:

A: Health Dashboard

This mode displays a single table with one row per monitored server. Thecolumns show essential overview information about the server's health, andcoloration rules show whether replication is running or if there are any verylong-running queries or excessive replication delay.

B: InnoDB Buffers

This mode displays information about the InnoDB buffer pool, page statistics,insert buffer, and adaptive hash index. The data comes from SHOW INNODB STATUS.

This mode contains the "buffer_pool", "page_statistics","insert_buffers", and "adaptive_hash_index" tables by default.

C: Command Summary

This mode is similar to mytop's Command Summary mode. It shows the"cmd_summary" table, which looks something like the following:

 Command Summary (? for help) localhost, 25+07:16:43, 2.45 QPS, 3 thd, 5.0.40
 _____________________ Command Summary _____________________
 Name                    Value    Pct     Last Incr  Pct
 Select_scan             3244858  69.89%          2  100.00%
 Select_range            1354177  29.17%          0    0.00%
 Select_full_join          39479   0.85%          0    0.00%
 Select_full_range_join     4097   0.09%          0    0.00%
 Select_range_check            0   0.00%          0    0.00%
    

The command summary table is built by extracting variables from"STATUS_VARIABLES". The variables must be numeric and must match the prefixgiven by the "cmd_filter" configuration variable. The variables are thensorted by value descending and compared to the last variable, as shown above.The percentage columns are percentage of the total of all variables in thetable, so you can see the relative weight of the variables.

The example shows what you see if the prefix is "Select_". The defaultprefix is "Com_". You can choose a prefix with the 's' key.

It's rather like running SHOW VARIABLES LIKE "prefix%" with memory andnice formatting.

Values are aggregated across all servers. The Pct columns are not correctlyaggregated across multiple servers. This is a known limitation of the groupingalgorithm that may be fixed in the future.

D: InnoDB Deadlocks

This mode shows the transactions involved in the last InnoDB deadlock. A secondtable shows the locks each transaction held and waited for. A deadlock iscaused by a cycle in the waits-for graph, so there should be two locks held andone waited for unless the deadlock information is truncated.

InnoDB puts deadlock information before some other information in the SHOWINNODB STATUS output. If there are a lot of locks, the deadlock information cangrow very large, and there is a limit on the size of the SHOW INNODBSTATUS output. A large deadlock can fill the entire output, or even betruncated, and prevent you from seeing other information at all. If you arerunning innotop in another mode, for example T mode, and suddenly you don't seeanything, you might want to check and see if a deadlock has wiped out the datayou need.

If it has, you can create a small deadlock to replace the large one. Use the'w' key to 'wipe' the large deadlock with a small one. This will not workunless you have defined a deadlock table for the connection (see "SERVERCONNECTIONS").

You can also configure innotop to automatically detect when a large deadlockneeds to be replaced with a small one (see "auto_wipe_dl").

This mode displays the "deadlock_transactions" and "deadlock_locks" tablesby default.

F: InnoDB Foreign Key Errors

This mode shows the last InnoDB foreign key error information, such as thetable where it happened, when and who and what query caused it, and so on.

InnoDB has a huge variety of foreign key error messages, and many of them arejust hard to parse. innotop doesn't always do the best job here, but there'sso much code devoted to parsing this messy, unparseable output that innotop islikely never to be perfect in this regard. If innotop doesn't show you whatyou need to see, just look at the status text directly.

This mode displays the "fk_error" table by default.

I: InnoDB I/O Info

This mode shows InnoDB's I/O statistics, including the I/O threads, pending I/O,file I/O miscellaneous, and log statistics. It displays the "io_threads","pending_io", "file_io_misc", and "log_statistics" tables by default.

K: InnoDB Lock Waits

This mode shows information from InnoDB plugin's transaction and locking tables.You can use it to find when a transaction is waiting for another, and kill theblocking transaction. It displays the "innodb_blocked_blocker" table.

L: Locks

This mode shows information about current locks. At the moment only InnoDBlocks are supported, and by default you'll only see locks for which transactionsare waiting. This information comes from the TRANSACTIONS section of the InnoDBstatus text. If you have a very busy server, you may have frequent lock waits;it helps to be able to see which tables and indexes are the "hot spot" forlocks. If your server is running pretty well, this mode should show nothing.

You can configure MySQL and innotop to monitor not only locks for which atransaction is waiting, but those currently held, too. You can do this with theInnoDB Lock Monitor (<http://dev.mysql.com/doc/en/innodb-monitor.html>). It'snot documented in the MySQL manual, but creating the lock monitor with thefollowing statement also affects the output of SHOW INNODB STATUS, which innotopuses:

  CREATE TABLE innodb_lock_monitor(a int) ENGINE=INNODB;
    

This causes InnoDB to print its output to the MySQL file every 16 seconds or so,as stated in the manual, but it also makes the normal SHOW INNODB STATUS outputinclude lock information, which innotop can parse and display (that's theundocumented feature).

This means you can do what may have seemed impossible: to a limited extent(InnoDB truncates some information in the output), you can see which transactionholds the locks something else is waiting for. You can also enable and disablethe InnoDB Lock Monitor with the key mappings in this mode.

This mode displays the "innodb_locks" table by default. Here's a sample ofthe screen when one connection is waiting for locks another connection holds:

 _________________________________ InnoDB Locks __________________________
 CXN        ID  Type    Waiting  Wait   Active  Mode  DB    Table  Index
 localhost  12  RECORD        1  00:10   00:10  X     test  t1     PRIMARY
 localhost  12  TABLE         0  00:10   00:10  IX    test  t1
 localhost  12  RECORD        1  00:10   00:10  X     test  t1     PRIMARY
 localhost  11  TABLE         0  00:00   00:25  IX    test  t1
 localhost  11  RECORD        0  00:00   00:25  X     test  t1     PRIMARY
    

You can see the first connection, ID 12, is waiting for a lock on the PRIMARYkey on test.t1, and has been waiting for 10 seconds. The second connectionisn't waiting, because the Waiting column is 0, but it holds locks on the sameindex. That tells you connection 11 is blocking connection 12.

M: Master/Slave Replication Status

This mode shows the output of SHOW SLAVE STATUS and SHOW MASTER STATUS in threetables. The first two divide the slave's status into SQL and I/O thread status,and the last shows master status. Filters are applied to eliminate non-slaveservers from the slave tables, and non-master servers from the master table.

This mode displays the "slave_sql_status", "slave_io_status", and"master_status" tables by default.

O: Open Tables

This section comes from MySQL's SHOW OPEN TABLES command. By default it isfiltered to show tables which are in use by one or more queries, so you canget a quick look at which tables are 'hot'. You can use this to guess whichtables might be locked implicitly.

This mode displays the "open_tables" mode by default.

U: User Statistics

This mode displays data that's available in Percona's enhanced version of MySQL(also known as Percona Server with XtraDB). Specifically, it makes it easy toenable and disable the so-called "user statistics." This feature gathers statson clients, threads, users, tables, and indexes and makes them available asINFORMATION_SCHEMA tables. These are invaluable for understanding what yourserver is doing. They are also available in MariaDB.

The statistics supported so far are only from the TABLE_STATISTICS andINDEX_STATISTICS tables added by Percona. There are three views: one of table stats,one of index stats (which can be aggregated with the = key), and one of both.

The server doesn't gather these stats by default. You have to set the variableuserstat_running to turn it on. You can do this easily with innotop from U mode,with the 's' key.

Q: Query List

This mode displays the output from SHOW FULL PROCESSLIST, much like mytop's query list mode. This mode does not show InnoDB-related information. This is probably one of the most useful modes for general usage.

There is an informative header that shows general status information aboutyour server. You can toggle it on and off with the 'h' key. By default,innotop hides inactive processes and its own process. You can toggle these onand off with the 'i' and 'a' keys.

You can EXPLAIN a query from this mode with the 'e' key. This displays thequery's full text, the results of EXPLAIN, and in newer MySQL versions, eventhe optimized query resulting from EXPLAIN EXTENDED. innotop also tries torewrite certain queries to make them EXPLAIN-able. For example, INSERT/SELECTstatements are rewritable.

This mode displays the "q_header" and "processlist" tables by default.

R: InnoDB Row Operations and Semaphores

This mode shows InnoDB row operations, row operation miscellaneous, semaphores,and information from the wait array. It displays the "row_operations","row_operation_misc", "semaphores", and "wait_array" tables by default.

S: Variables & Status

This mode calculates statistics, such as queries per second, and prints them outin several different styles. You can show absolute values, or incremental valuesbetween ticks.

You can switch between the views by pressing a key. The 's' key prints asingle line each time the screen updates, in the style of vmstat. The 'g' key changes the view to a graph of the same numbers, sort of like tload. The 'v' key changes the view to a pivoted table of variable names on the left, with successive updates scrolling across the screen from left to right. You can choose how many updates to put on the screen with the "num_status_sets" configuration variable.

Headers may be abbreviated to fit on the screen in interactive operation. Youchoose which variables to display with the 'c' key, which selects frompredefined sets, or lets you create your own sets. You can edit the current setwith the 'e' key.

This mode doesn't really display any tables like other modes. Instead, it usesa table definition to extract and format the data, but it then transforms theresult in special ways before outputting it. It uses the "var_status" tabledefinition for this.

T: InnoDB Transactions

This mode shows transactions from the InnoDB monitor's output, in top-like format. This mode is the reason I wrote innotop.

You can kill queries or processes with the 'k' and 'x' keys, and EXPLAIN a querywith the 'e' or 'f' keys. InnoDB doesn't print the full query in transactions,so explaining may not work right if the query is truncated.

The informational header can be toggled on and off with the 'h' key. Bydefault, innotop hides inactive transactions and its own transaction. You cantoggle this on and off with the 'i' and 'a' keys.

This mode displays the "t_header" and "innodb_transactions" tables bydefault.

INNOTOP STATUS¶

The first line innotop displays is a "status bar" of sorts. What it containsdepends on the mode you're in, and what servers you're monitoring. The firstfew words are always [RO] (if readonly is set to 1), the innotop mode, such as"InnoDB Txns" for T mode, followed by a reminder to press '?' for help at anytime.

SERVER ADMINISTRATION¶

While innotop is primarily a monitor that lets you watch and analyze yourservers, it can also send commands to servers. The most frequently usefulcommands are killing queries and stopping or starting slaves.

You can kill a connection, or in newer versions of MySQL kill a query but not aconnection, from "Q: Query List" and "T: InnoDB Transactions" modes.Press 'k' to issue a KILL command, or 'x' to issue a KILL QUERY command.innotop will prompt you for the server and/or connection ID to kill (innotopdoes not prompt you if there is only one possible choice for any input).innotop pre-selects the longest-running query, or the oldest connection.Confirm the command with 'y'.

In "Slave Replication Status"" in "M: Master mode, you can start and stop slaveswith the 'a' and 'o' keys, respectively. You can send these commands to manyslaves at once. innotop fills in a default command of START SLAVE or STOP SLAVEfor you, but you can actually edit the command and send anything you wish, suchas SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1 to make the slave skip one binlog eventwhen it starts.

You can also ask innotop to calculate the earliest binlog in use by any slaveand issue a PURGE MASTER LOGS on the master. Use the 'b' key for this. innotopwill prompt you for a master to run the command on, then prompt you for theconnection names of that master's slaves (there is no way for innotop todetermine this reliably itself). innotop will find the minimum binlog in use bythese slave connections and suggest it as the argument to PURGE MASTER LOGS.

in "U: User Statistics" mode, you can use the 's' key to start and stopthe collection of the statistics data for TABLE_STATISTICS and similar.

SERVER CONNECTIONS¶

When you create a server connection using '@', innotop asks you for a series ofinputs, as follows:

DSN

A DSN is a Data Source Name, which is the initial argument passed to the DBImodule for connecting to a server. It is usually of the form

 DBI:MariaDB:;mariadb_read_default_group=mysql;host=HOSTNAME
    

Since this DSN is passed to the DBD::MariaDB driver, you should read the driver'sdocumentation at <https://metacpan.org/pod/DBD::MariaDB> forthe exact details on all the options you can pass the driver in the DSN. Youcan read more about DBI at <http://dbi.perl.org/docs/>, and especially at<http://search.cpan.org/~timb/DBI/DBI.pm>.

The mariadb_read_default_group=mysql option lets the DBD driver read your MySQLoptions files, such as ~/.my.cnf on UNIX-ish systems. You can use this to avoidspecifying a username or password for the connection.

InnoDB Deadlock Table

This optional item tells innotop a table name it can use to deliberately createa small deadlock (see "D: InnoDB Deadlocks"). If you specify this option,you just need to be sure the table doesn't exist, and that innotop can createand drop the table with the InnoDB storage engine. You can safely omit or justaccept the default if you don't intend to use this.

Username

innotop will ask you if you want to specify a username. If you say 'y', it willthen prompt you for a user name. If you have a MySQL option file that specifiesyour username, you don't have to specify a username.

The username defaults to your login name on the system you're running innotop on.

Password

innotop will ask you if you want to specify a password. Like the username, thepassword is optional, but there's an additional prompt that asks if you want tosave the password in the innotop configuration file. If you don't save it inthe configuration file, innotop will prompt you for a password each time itstarts. Passwords in the innotop configuration file are saved in plain text,not encrypted in any way.

Once you finish answering these questions, you should be connected to a server.But innotop isn't limited to monitoring a single server; you can define manyserver connections and switch between them by pressing the '@' key. See"SWITCHING BETWEEN CONNECTIONS".

SERVER GROUPS¶

If you have multiple MySQL instances, you can put them into named groups, suchas 'all', 'masters', and 'slaves', which innotop can monitor all together.

You can choose which group to monitor with the '#' key, and you can press theTAB key to switch to the next group. If you're not currently monitoring agroup, pressing TAB selects the first group.

To create a group, press the '#' key and type the name of your new group, thentype the names of the connections you want the group to contain.

SWITCHING BETWEEN CONNECTIONS¶

innotop lets you quickly switch which servers you're monitoring. The most basicway is by pressing the '@' key and typing the name(s) of the connection(s) youwant to use. This setting is per-mode, so you can monitor different connectionsin each mode, and innotop remembers which connections you choose.

You can quickly switch to the 'next' connection in alphabetical order with the'n' key. If you're monitoring a server group (see "SERVER GROUPS") this willswitch to the first connection.

You can also type many connection names, and innotop will fetch and display datafrom them all. Just separate the connection names with spaces, for example"server1 server2." Again, if you type the name of a connection that doesn'texist, innotop will prompt you for connection information and create theconnection.

Another way to monitor multiple connections at once is with server groups. Youcan use the TAB key to switch to the 'next' group in alphabetical order, or ifyou're not monitoring any groups, TAB will switch to the first group.

innotop does not fetch data in parallel from connections, so if you aremonitoring a large group or many connections, you may notice increased delaybetween ticks.

When you monitor more than one connection, innotop's status bar changes. See"INNOTOP STATUS".

ERROR HANDLING¶

Error handling is not that important when monitoring a single connection, but iscrucial when you have many active connections. A crashed server or lostconnection should not crash innotop. As a result, innotop will continue to runeven when there is an error; it just won't display any information from theconnection that had an error. Because of this, innotop's behavior might confuseyou. It's a feature, not a bug!

innotop does not continue to query connections that have errors, because theymay slow innotop and make it hard to use, especially if the error is a problemconnecting and causes a long time-out. Instead, innotop retries the connectionoccasionally to see if the error still exists. If so, it will wait until somepoint in the future. The wait time increases in ticks as the Fibonacci series,so it tries less frequently as time passes.

Since errors might only happen in certain modes because of the SQL commandsissued in those modes, innotop keeps track of which mode caused the error. Ifyou switch to a different mode, innotop will retry the connection instead ofwaiting.

By default innotop will display the problem in red text at the bottom of thefirst table on the screen. You can disable this behavior with the"show_cxn_errors_in_tbl" configuration option, which is enabled by default.If the "debug" option is enabled, innotop will display the error at thebottom of every table, not just the first. And if "show_cxn_errors" isenabled, innotop will print the error text to STDOUT as well. Error messagesmight only display in the mode that caused the error, depending on the mode andwhether innotop is avoiding querying that connection.

NON-INTERACTIVE OPERATION¶

You can run innotop in non-interactive mode, in which case it is entirelycontrolled from the configuration file and command-line options. To startinnotop in non-interactive mode, give the L"<--nonint"> command-line option.This changes innotop's behavior in the following ways:

• Certain Perl modules are not loaded. Term::Readline is not loaded, sinceinnotop doesn't prompt interactively. Term::ANSIColor and Win32::Console::ANSImodules are not loaded. Term::ReadKey is still used, since innotop may have toprompt for connection passwords when starting up.
• innotop does not clear the screen after each tick.
• innotop does not persist any changes to the configuration file.
• If "--count" is given and innotop is in incremental mode (see "status_inc"and "--inc"), innotop actually refreshes one more time than specified so itcan print incremental statistics. This suppresses output during the firsttick, so innotop may appear to hang.
• innotop only displays the first table in each mode. This is so the output canbe easily processed with other command-line utilities such as awk and sed. Tochange which tables display in each mode, see "TABLES". Since "Q: QueryList" mode is so important, innotop automatically disables the "q_header"table. This ensures you'll see the "processlist" table, even if you haveinnotop configured to show the q_header table during interactive operation.Similarly, in "T: InnoDB Transactions" mode, the "t_header" table issuppressed so you see only the "innodb_transactions" table.
• All output is tab-separated instead of being column-aligned with whitespace, andinnotop prints the full contents of each table instead of only printing onescreenful at a time.
• innotop only prints column headers once instead of every tick (see"hide_hdr"). innotop does not print table captions (see"display_table_captions"). innotop ensures there are no empty lines in theoutput.
• innotop does not honor the "shorten" transformation, which normally shortenssome numbers to human-readable formats.
• innotop does not print a status line (see "INNOTOP STATUS").

CONFIGURING¶

Nearly everything about innotop is configurable. Most things are possible tochange with built-in commands, but you can also edit the configuration file.

While running innotop, press the '$' key to bring up the configuration editingdialog. Press another key to select the type of data you want to edit:

S: Statement Sleep Times

Edits SQL statement sleep delays, which make innotop pause for the specifiedamount of time after executing a statement. See "SQL STATEMENTS" for adefinition of each statement and what it does. By default innotop does notdelay after any statements.

This feature is included so you can customize the side-effects caused bymonitoring your server. You may not see any effects, but some innotop usershave noticed that certain MySQL versions under very high load with InnoDBenabled take longer than usual to execute SHOW GLOBAL STATUS. If innotop callsSHOW FULL PROCESSLIST immediately afterward, the processlist contains morequeries than the machine actually averages at any given moment. Configuringinnotop to pause briefly after calling SHOW GLOBAL STATUS alleviates thiseffect.

Sleep times are stored in the "stmt_sleep_times" section of the configurationfile. Fractional-second sleeps are supported, subject to your hardware'slimitations.

c: Edit Columns

Starts the table editor on one of the displayed tables. See "TABLE EDITOR".An alternative way to start the table editor without entering the configurationdialog is with the '^' key.

g: General Configuration

Starts the configuration editor to edit global and mode-specific configurationvariables (see "MODES"). innotop prompts you to choose a variable from amongthe global and mode-specific ones depending on the current mode.

k: Row-Coloring Rules

Starts the row-coloring rules editor on one of the displayed table(s). See"COLORS" for details.

p: Manage Plugins

Starts the plugin configuration editor. See "PLUGINS" for details.

s: Server Groups

Lets you create and edit server groups. See "SERVER GROUPS".

t: Choose Displayed Tables

Lets you choose which tables to display in this mode. See "MODES" and"TABLES".

CONFIGURATION FILE¶

innotop's default configuration file locations are $HOME/.innotop and /etc/innotop/innotop.conf, and they are looked for in that order. If the first configuration file exists, the second will not be processed. Those can be overridden with the "--config" command-line option. You can edit it by hand safely, however innotop reads the configuration file when it starts, and, if readonly is set to 0, writes it out again when it exits. Thus, if readonly is set to 0, any changes you make by hand while innotop is running will be lost.

innotop doesn't store its entire configuration in the configuration file. Ithas a huge set of default configuration values that it holds only in memory,and the configuration file only overrides these defaults. When you customize adefault setting, innotop notices, and then stores the customizations into thefile. This keeps the file size down, makes it easier to edit, and makesupgrades easier.

A configuration file is read-only be default. You can override that with"--write". See "readonly".

The configuration file is arranged into sections like an INI file. Eachsection begins with [section-name] and ends with [/section-name]. Eachsection's entries have a different syntax depending on the data they need tostore. You can put comments in the file; any line that begins with a #character is a comment. innotop will not read the comments, so it won't writethem back out to the file when it exits. Comments in read-only configurationfiles are still useful, though.

The first line in the file is innotop's version number. This lets innotopnotice when the file format is not backwards-compatible, and upgrade smoothlywithout destroying your customized configuration.

The following list describes each section of the configuration file and the datait contains:

general

The 'general' section contains global configuration variables and variables thatmay be mode-specific, but don't belong in any other section. The syntax is asimple key=value list. innotop writes a comment above each value to help youedit the file by hand.

plugins

This section holds a list of package names of active plugins. If the pluginexists, innotop will activate it. See "PLUGINS" for more information.

filters

This section holds user-defined filters (see "FILTERS"). Each line is in theformat filter_name=text='filter text' tbls='table list'.

The filter text is the text of the subroutine's code. The table list is a listof tables to which the filter can apply. By default, user-defined filters applyto the table for which they were created, but you can manually override that byediting the definition in the configuration file.

active_filters

This section stores which filters are active on each table. Each line is in theformat table_name=filter_list.

tbl_meta

This section stores user-defined or user-customized columns (see "COLUMNS").Each line is in the format col_name=properties, where the properties are aname=quoted-value list.

connections

This section holds the server connections you have defined. Each line is inthe format name=properties, where the properties are a name=value list. Theproperties are self-explanatory, and the only one that is treated specially is'pass' which is only present if 'savepass' is set. This section of theconfiguration file will be skipped if any DSN, username, or passwordcommand-line options are used. See "SERVER CONNECTIONS".

active_connections

This section holds a list of which connections are active in each mode. Eachline is in the format mode_name=connection_list.

server_groups

This section holds server groups. Each line is in the formatname=connection_list. See "SERVER GROUPS".

active_server_groups

This section holds a list of which server group is active in each mode. Eachline is in the format mode_name=server_group.

max_values_seen

This section holds the maximum values seen for variables. This is used to scalethe graphs in "S: Variables & Status" mode. Each line is in the formatname=value.

active_columns

This section holds table column lists. Each line is in the formattbl_name=column_list. See "COLUMNS".

sort_cols

This section holds the sort definition. Each line is in the formattbl_name=column_list. If a column is prefixed with '-', that column sortsdescending. See "SORTING".

visible_tables

This section defines which tables are visible in each mode. Each line is in theformat mode_name=table_list. See "TABLES".

varsets

This section defines variable sets for use in "S: Status & Variables" mode.Each line is in the format name=variable_list. See "VARIABLE SETS".

colors

This section defines colorization rules. Each line is in the formattbl_name=property_list. See "COLORS".

stmt_sleep_times

This section contains statement sleep times. Each line is in the formatstatement_name=sleep_time. See "S: Statement Sleep Times".

group_by

This section contains column lists for table group_by expressions. Each line isin the format tbl_name=column_list. See "GROUPING".

CUSTOMIZING¶

You can customize innotop a great deal. For example, you can:

• Choose which tables to display, and in what order.
• Choose which columns are in those tables, and create new columns.
• Filter which rows display with built-in filters, user-defined filters, andquick-filters.
• Sort the rows to put important data first or group together related rows.
• Highlight rows with color.
• Customize the alignment, width, and formatting of columns, and applytransformations to columns to extract parts of their values or format the valuesas you wish (for example, shortening large numbers to familiar units).
• Design your own expressions to extract and combine data as you need. This givesyou unlimited flexibility.

All these and more are explained in the following sections.

 sub filter {
    my ( $set ) = @_;
    # YOUR CODE HERE
 }

 sub filter {
    my ( $set ) = @_;
    $set->{info} =~ m/product/;
 }

 1. /
 2. cmd ^(?!Gandalf)

 Query List (? for help) localhost, 32:33, 0.11 QPS, 1 thd, 5.0.38-log
 CXN        Cmd        Cnt  ID      User   Host           Time   Query
 localhost  Query      49    12933  webusr localhost      19:38  SELECT * FROM
 localhost  Sending Da 23     2383  webusr localhost      12:43  SELECT col1,
 localhost  Sleep      120     140  webusr localhost    5:18:12
 localhost  Statistics 12    19213  webusr localhost      01:19  SELECT * FROM

 foo bar
 === ===
 1   3
 2   4

 name set0 set1
 ==== ==== ====
 foo  1    2
 bar  3    4

 state  eq  Locked       black on_red
 cmd    eq  Sleep        white
 user   eq  system user  white
 cmd    eq  Connect      white
 cmd    eq  Binlog Dump  white
 time   >   600          red
 time   >   120          yellow
 time   >   60           green
 time   >   30           cyan

 sub compute_column_value {
    my ( $set, $cur, $pre ) = @_;
    my $val = # EXPANDED STRING GOES HERE
    return $val;
 }

 Questions/Uptime_hires

 $set->{Questions}/$set->{Uptime_hires}

 Editing table definition for Buffer Pool.  Press ? for help, q to quit.
 name               hdr          label                  src
 cxn                CXN          Connection from which  cxn
 buf_pool_size      Size         Buffer pool size       IB_bp_buf_poo
 buf_free           Free Bufs    Buffers free in the b  IB_bp_buf_fre
 pages_total        Pages        Pages total            IB_bp_pages_t
 pages_modified     Dirty Pages  Pages modified (dirty  IB_bp_pages_m
 buf_pool_hit_rate  Hit Rate     Buffer pool hit rate   IB_bp_buf_poo
 total_mem_alloc    Memory       Total memory allocate  IB_bp_total_m
 add_pool_alloc     Add'l Pool   Additional pool alloca  IB_bp_add_poo

VARIABLE SETS¶

Variable sets are used in "S: Variables & Status" mode to define more easilywhat variables you want to monitor. Behind the scenes they are compiled to alist of expressions, and then into a column list so they can be treated justlike columns in any other table, in terms of data extraction andtransformations. However, you're protected from the tedious details by a syntaxthat ought to feel very natural to you: a SQL SELECT list.

The data source for variable sets, and indeed the entire S mode, is thecombination of SHOW STATUS, SHOW VARIABLES, and SHOW INNODB STATUS. Imaginethat you had a huge table with one column per variable returned from thosestatements. That's the data source for variable sets. You can now query thisdata source just like you'd expect. For example:

 Questions, Uptime, Questions/Uptime as QPS

Behind the scenes innotop will split that variable set into three expressions,compile them and turn them into a table definition, then extract as usual. Thisbecomes a "variable set," or a "list of variables you want to monitor."

innotop lets you name and save your variable sets, and writes them to theconfiguration file. You can choose which variable set you want to see with the'c' key, or activate the next and previous sets with the '>' and '<' keys.There are many built-in variable sets as well, which should give you a goodstart for creating your own. Press 'e' to edit the current variable set, orjust to see how it's defined. To create a new one, just press 'c' and type itsname.

You may want to use some of the functions listed in "TRANSFORMATIONS" to helpformat the results. In particular, "set_precision" is often useful to limitthe number of digits you see. Extending the above example, here's how:

 Questions, Uptime, set_precision(Questions/Uptime) as QPS

Actually, this still needs a little more work. If your "interval" is lessthan one second, you might be dividing by zero because Uptime is incremental inthis mode by default. Instead, use Uptime_hires:

 Questions, Uptime, set_precision(Questions/Uptime_hires) as QPS

This example is simple, but it shows how easy it is to choose which variablesyou want to monitor.

PLUGINS¶

innotop has a simple but powerful plugin mechanism by which you can extendor modify its existing functionality, and add new functionality. innotop'splugin functionality is event-based: plugins register themselves to be calledwhen events happen. They then have a chance to influence the event.

An innotop plugin is a Perl module (.pm) file placed in innotop's "plugin_dir"directory. On UNIX systems, you can place a symbolic link to the module insteadof putting the actual file there. innotop automatically discovers files named "*.pm". If there is a corresponding entry in the "plugins" configuration file section, innotop loads and activates the plugin.

The module must conform to innotop's plugin interface. Additionally, the sourcecode of the module must be written in such a way that innotop can inspect thefile and determine the package name and description.

 use strict;
 use warnings FATAL => 'all';
 package Innotop::Plugin::Example;
 # description: Adds an 'example' column to every table
 sub new {
    my ( $class, %vars ) = @_;
    # Store reference to innotop's variables in $self
    my $self = bless { %vars }, $class;
    # Design the example column
    my $col = {
       hdr   => 'Example',
       just  => '',
       dec   => 0,
       num   => 1,
       label => 'Example',
       src   => 'example', # Get data from this column in the data source
       tbl   => '',
       trans => [],
    };
    # Add the column to every table.
    my $tbl_meta = $vars{tbl_meta};
    foreach my $tbl ( values %$tbl_meta ) {
       # Add the column to the list of defined columns
       $tbl->{cols}->{example} = $col;
       # Add the column to the list of visible columns
       unshift @{$tbl->{visible}}, 'example';
    }
    # Be sure to return a reference to the object.
    return $self;
 }
 # I'd like to be called when a data set is being rendered into a table, please.
 sub register_for_events {
    my ( $self ) = @_;
    return qw(set_to_tbl_pre_filter);
 }
 # This method will be called when the event fires.
 sub set_to_tbl_pre_filter {
    my ( $self, $rows, $tbl ) = @_;
    # Set the example column's data source to the value 1.
    foreach my $row ( @$rows ) {
       $row->{example} = 1;
    }
 }
 1;

SQL STATEMENTS¶

innotop uses a limited set of SQL statements to retrieve data from MySQL fordisplay. The statements are customized depending on the server version againstwhich they are executed; for example, on MySQL 5 and newer, INNODB_STATUSexecutes "SHOW ENGINE INNODB STATUS", while on earlier versions it executes"SHOW INNODB STATUS". The statements are as follows:

 Statement           SQL executed
 =================== ===============================
 INDEX_STATISTICS    SELECT * FROM INFORMATION_SCHEMA.INDEX_STATISTICS
 INNODB_STATUS       SHOW [ENGINE] INNODB STATUS
 KILL_CONNECTION     KILL
 KILL_QUERY          KILL QUERY
 OPEN_TABLES         SHOW OPEN TABLES
 PROCESSLIST         SHOW FULL PROCESSLIST
 SHOW_MASTER_LOGS    SHOW MASTER LOGS
 SHOW_MASTER_STATUS  SHOW MASTER STATUS
 SHOW_SLAVE_STATUS   SHOW SLAVE STATUS
 SHOW_STATUS         SHOW [GLOBAL] STATUS
 SHOW_VARIABLES      SHOW [GLOBAL] VARIABLES
 TABLE_STATISTICS    SELECT * FROM INFORMATION_SCHEMA.TABLE_STATISTICS

DATA SOURCES¶

Each time innotop extracts values to create a table (see "EXPRESSIONS" and"TABLES"), it does so from a particular data source. Largely because of thecomplex data extracted from SHOW INNODB STATUS, this is slightly messy. SHOWINNODB STATUS contains a mixture of single values and repeated values that formnested data sets.

Whenever innotop fetches data from MySQL, it adds two extra bits to each set:cxn and Uptime_hires. cxn is the name of the connection from which the datacame. Uptime_hires is a high-resolution version of the server's Uptime statusvariable, which is important if your "interval" setting is sub-second.

Here are the kinds of data sources from which data is extracted:

STATUS_VARIABLES

This is the broadest category, into which the most kinds of data fall. Itbegins with the combination of SHOW STATUS and SHOW VARIABLES, but other sourcesmay be included as needed, for example, SHOW MASTER STATUS and SHOW SLAVESTATUS, as well as many of the non-repeated values from SHOW INNODB STATUS.

DEADLOCK_LOCKS

This data is extracted from the transaction list in the LATEST DETECTED DEADLOCKsection of SHOW INNODB STATUS. It is nested two levels deep: transactions, thenlocks.

DEADLOCK_TRANSACTIONS

This data is from the transaction list in the LATEST DETECTED DEADLOCKsection of SHOW INNODB STATUS. It is nested one level deep.

EXPLAIN

This data is from the result set returned by EXPLAIN.

INNODB_BLOCKED_BLOCKER

This data is from the INFORMATION_SCHEMA tables related to InnoDB locks andthe processlist.

INNODB_TRANSACTIONS

This data is from the TRANSACTIONS section of SHOW INNODB STATUS.

IO_THREADS

This data is from the list of threads in the FILE I/O section of SHOW INNODBSTATUS.

INNODB_LOCKS

This data is from the TRANSACTIONS section of SHOW INNODB STATUS and is nestedtwo levels deep.

MASTER_SLAVE

This data is from the combination of SHOW MASTER STATUS and SHOW SLAVE STATUS.

OPEN_TABLES

This data is from SHOW OPEN TABLES.

PROCESSLIST

This data is from SHOW FULL PROCESSLIST.

PROCESSLIST_STATS

This data is from SHOW FULL PROCESSLIST and computes stats such as the maximum timea user query has been running, and how many user queries are running. A "userquery" excludes replication threads.

OS_WAIT_ARRAY

This data is from the SEMAPHORES section of SHOW INNODB STATUS and is nested onelevel deep. It comes from the lines that look like this:

 --Thread 1568861104 has waited at btr0cur.c line 424 ....
    

MYSQL PRIVILEGES¶

• You must connect to MySQL as a user who has the SUPER privilege for many of thefunctions.
• If you don't have the SUPER privilege, you can still run some functions, but youwon't necessarily see all the same data.
• You need the PROCESS privilege to see the list of currently running queries in Qmode.
• You need special privileges to start and stop slave servers.
• You need appropriate privileges to create and drop the deadlock tables if needed(see "SERVER CONNECTIONS").

SYSTEM REQUIREMENTS¶

You need Perl to run innotop, of course. You also need a few Perl modules: DBI,DBD::MariaDB, Term::ReadKey, and Time::HiRes. These should be included with mostPerl distributions, but in case they are not, I recommend using versionsdistributed with your operating system or Perl distribution, not from CPAN.Term::ReadKey in particular has been known to cause problems if installed fromCPAN.

If you have Term::ANSIColor, innotop will use it to format headers more readablyand compactly. (Under Microsoft Windows, you also need Win32::Console::ANSI forterminal formatting codes to be honored). If you install Term::ReadLine,preferably Term::ReadLine::Gnu, you'll get nice auto-completion support.

I run innotop on Gentoo GNU/Linux, Debian and Ubuntu, and I've had feedback frompeople successfully running it on Red Hat, CentOS, Solaris, and Mac OSX. Idon't see any reason why it won't work on other UNIX-ish operating systems, butI don't know for sure. It also runs on Windows under ActivePerl withoutproblem.

innotop has been used on MySQL versions 3.23.58, 4.0.27, 4.1.0, 4.1.22, 5.0.26,5.1.15, and 5.2.3. If it doesn't run correctly for you, that is a bug thatshould be reported.

FILES¶

$HOMEDIR/.innotop and/or /etc/innotop are used to store configuration information. Files include the configuration file innotop.conf, the core_dump file which contains verbose error messages if "debug" is enabled, and the plugins/ subdirectory.

GLOSSARY OF TERMS¶

tick

A tick is a refresh event, when innotop re-fetches data from connections anddisplays it.

ACKNOWLEDGEMENTS¶

The following people and organizations are acknowledged for various reasons.Hopefully no one has been forgotten.

Aaron Racine,Allen K. Smith,Aurimas Mikalauskas,Bartosz Fenski,Brian Miezejewski,Christian Hammers,Cyril Scetbon,Dane Miller,David Multer,Dr. Frank Ullrich,Giuseppe Maxia,Google.com Site Reliability Engineers,Google Code,Jan Pieter Kunst,Jari Aalto,Jay Pipes,Jeremy Zawodny,Johan Idren,Kristian Kohntopp,Lenz Grimmer,Maciej Dobrzanski,Michiel Betel,MySQL AB,Paul McCullagh,Sebastien Estienne,Sourceforge.net,Steven Kreuzer,The Gentoo MySQL Team,Trevor Price,Yaar Schnitman,and probably more people that have not been included.

(If your name has been misspelled, it's probably out of fear of puttinginternational characters into this documentation; earlier versions of Perl mightnot be able to compile it then).

COPYRIGHT, LICENSE AND WARRANTY¶

This program is copyright (c) 2006 Baron Schwartz.Feedback and improvements are welcome.

THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIEDWARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OFMERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE.

This program is free software; you can redistribute it and/or modify it underthe terms of the GNU General Public License as published by the Free SoftwareFoundation, version 2; OR the Perl Artistic License. On UNIX and similarsystems, you can issue `man perlgpl' or `man perlartistic' to read theselicenses.

You should have received a copy of the GNU General Public License along withthis program; if not, write to the Free Software Foundation, Inc., 51 FranklinStreet, Fifth Floor, Boston, MA 02110-1335 USA.

Execute innotop and press '!' to see this information at any time.

AUTHOR¶

Originally written by Baron Schwartz; currently maintained by Aaron Racine.

BUGS¶

You can report bugs, ask for improvements, and get other help and support at<https://github.com/innotop/innotop>. There are mailing lists, a source codebrowser, a bug tracker, etc. Please use these instead of contacting themaintainer or author directly, as it makes our job easier and benefits others if thediscussions are permanent and public. Of course, if you need to contact us inprivate, please do.