MINI SHELL

Overview
--------
diskdump provides dump feature that can dump memory with dump partitions.
diskdump consists of kernel portion and diskdumputils package.
The kernel portion inhibits interrupts, freezes all other CPUs, and then for
each page of data, issues the I/O command to the host adapter driver, followed
by calling the interrupt handler of the adapter driver iteratively until
the I/O has completed.
The diskdumputils package is installed on the machine that you wish to capture
dumps on, in the event of a system panic.  It loads and configures the diskdump
kernel modules so that if the machine crashes, the memory dump will be dumped
to disk.
When diskdump is executed, of course, the system is in a serious trouble.
Therefore, it is possible that user resources on the disk device containing
the dump partition are corrupted.  To avoid this danger, signatures to identify
are written to the complete dump partition.  When a system panic occurs,
the diskdump module reads the dump partition and checks on whether
the signatures written are correct.  If all of the signatures are verified,
the diskdump presumes that it will be writing to the correct device, and that
it is highly possible that it will be able to write memory dump correctly.

Features
--------
There are several features in diskdump service.  Each feature that is marked
with an X in the following table is available in RHEL3 and/or RHEL4.  Basic Dump
Facility is a feature that dumps memory without specifying any module parameters
and any options.  Partial Dump and Compressed Dump are enabled by specifying
module parameters.  Swap Partiton Support is enabled by specifying swap
partition that works with the supported driver as a dump device.  The other
features are enabled by specifying options for their features.  See the later
descriptions in this README if you need further information about each feature.

	Feature			RHEL3	RHEL4	File to Configure
	---------------------------------------------------------------
	Basic Dump Facility	 X	 X	/etc/sysconfig/diskdump
	Partial Dump		 	 X	/etc/modprobe.conf
	Swap Partiton Support		 X	/etc/sysconfig/diskdump
	Compressed Dump			 X	/etc/modprobe.conf
	Preserved Dump		 X	 X	/etc/sysconfig/diskdump
	Deferred Savecore		 X	/etc/sysconfig/diskdump
	Message Complement		 X	/etc/sysconfig/diskdump


Supported Drivers
-----------------
Diskdump is only supported with the following storage adapters:

	RHEL3			RHEL4
	------------------------------------
	aic7xxx			aic7xxx
	aic79xx			aic79xx
	dpt_i2o
				ipr
	megaraid2		megaraid
	mptfusion		mptfusion
	sym53c8xx		sym53c8xx
	sata_promise		sata_promise
	ata_piix		ata_piix
	CCISS

Supported Kernels
-----------------
diskdump is supported in the following Red Hat kernels, where <kernel-version>
is the version containing this diskdumputils package:

	RHEL3
		kernel*-<kernel-version>.i686.rpm
		kernel*-<kernel-version>.athlon.rpm
		kernel*-<kernel-version>.ia64.rpm
		kernel*-<kernel-version>.x86_64.rpm
		kernel*-<kernel-version>.ia32e.rpm
	RHEL4
		kernel*-<kernel-version>.i686.rpm
		kernel*-<kernel-version>.ia64.rpm
		kernel*-<kernel-version>.x86_64.rpm
		kernel*-<kernel-version>.ppc64.rpm

Setup
-----
1. Dump Device Selection

The first step in the configuration process is to designate a disk device
to dump memory to in the event of a system crash.  The dump device may be
any of the following:
 - a full disk device - RHEL3 only - (e.g. /dev/sda)
 - a partition of a disk device (e.g. /dev/sda4)
 - a swap partition - RHEL4 only - (e.g. /dev/sda2)

As for usual operations, the size of dump device should be large enough to save
the whole dump.  However, it is possible that the size can be reduced with
certain diskdump features that are described later.  The dump size to be written
consists of the size of whole physical memory plus a header field.  To determine
the exact size required, refer to the output of /proc/diskdump after
the diskdump module is loaded:

 # modprobe diskdump

 # cat /proc/diskdump

 # sample_rate: 8
 # block_order: 2
 # fallback_on_err: 1
 # allow_risky_dumps: 1
 # total_blocks: 262042
 #

The total block size is shown by page-size units, so in this
example, the selected device must contain at least (262042 * 4096) bytes
on an i386 machine.

Note: during a diskdump operation, memory contents residing on the swap
partition are not preserved.  Therefore the dump partition size corresponds
to physical memory; rather than physical memory plus the size of the swap
partition.

Next, based on the information above, consider which devices you select
as a dump device.  To do that, follow the instructions below.

Edit /etc/sysconfig/diskdump appropriately in the following format to register
a dump device:

 ----------------
 DEVICE=/dev/sde1
 ----------------

Multiple dump devices can be registered in a colon-separated format like:

 -------------------------
 DEVICE=/dev/sda2:/dev/sdb
 -------------------------

The benefit of designating more than one dump device is redundancy.
For example, if each dump device was controlled by a different driver, even
if a system panic occurred in a driver that controls one of the registered
devices, the memory could be dumped out using the other registered device.  In
this case it is required that each dump device be sufficiently large to store
the full dump.  The registered dump devices are used in order of the left.

If you configure dumping to the swap partition, it is required that
both /usr and /var must be mounted locally; for reasons described in the
remainder of this paragraph.
In the event of a system crash, the memory contents are saved to /var/crash.
When the system reboots, the diskdumputils commands are run to preserve the
saved memory, which it must read off /var/crash.  The diskdump commands
themselves are mounted under /usr.  This memory saving operation is run in the
boot sequence prior to both enabling swap and mounting remote filesystems.
If /usr and /var were mounted remotely, the diskdump service would fail
because remote file systems are usually mounted later than the swap
initialization in rc.sysinit.


2. Dump Device Formatting

The second step in the configuration process involves formatting the dump
device.
Any dump device needs to be specially formatted for diskdump before being
used.  Accordingly, the designated dump partition cannot be used to create
a conventional filesystem on it.

The dump device formatting needs to be done once by the system administrator.

 # service diskdump initialformat

The above command cannot format any swap partitions or any mounted devices.  If
you format them in order to use them as dump-dedicated devices, you first
swapoff or unmount those devices.  Then, run the command below.

 # diskdumpfmt -f <device>


3. Enable Diskdump Service

Lastly, start the diskdump service:

 # chkconfig diskdump on
 # service diskdump start

If diskdump startup succeeds, either [OK] or [WARNING] is shown.  Otherwise,
[FAIL] is shown.  [WARNING] means some of the registered devices are not
appropriate, but at least one device is available.  [FAIL] means diskdump
startup fails because the registered devices are not appropriate.  See
/var/log/messages if you need further information.

The registered device/partition can be referred through /proc/diskdump
interface.

 # cat /proc/diskdump
 /dev/sde1 514080 1012095

The first value means start sector of the registered device.  The second value
means the number of sectors.
If the registered dump device needs to be replaced, edit
/etc/sysconfig/diskdump.  Format the new dump device as described above.
Then restart the diskdump service.  To restart the service, run the command
below.

 # service diskdump restart

To check on whether diskdump service is enabled, run the command below.

 # service diskdump status

Just in case, run the command above after start/restart.

To check the status of the registered device(s), run the command below.

 # service diskdump devicestatus

The command above is helpful to check on whether there are any inappropriate
devices when multiple devices are registered in /etc/sysconfig/diskdump.

4. Additional Options

Additional options to enable features such as Preserved Dump, Deferred Savecore,
and Message Complement are available in diskdump service.  The following options
can be set in /etc/sysconfig/diskdump.
 - PRESERVEDUMP		(may be used with other options)
 - SKIPSAVECORE		(may be used with other options)
 - EXPIRATION		(an auxiliary option for only SKIPSAVECORE)
 - MAILTO		(an auxiliary option)
 - FROM			(an auxiliary option)
 - SALVAGEMESSAGE	(an independent option for only Message Complement)

To enable Preserved Dump feature, edit /etc/sysconfig/diskdump appropriately in
the following format if you want to preserve dump data:

 ----------------
 PRESERVEDUMP=yes
 ----------------

If it is specified as above, then diskdump service won't format the device
only when saving panic dump failed due to some error.  Consequently, you need
to investigate the cause which it failed and resolve the problem, and then must
run:

 # service diskdump restart

If saving panic dump fails again with the command above, then you must run:

 # service diskdump enabledevice

so that diskdump service can restart.  The command above enables the registered
device again (i.e. the device can be used as the diskdump-dedicated device).
Note that the device would be formatted by the command above, enabledevice,
even if saving panic dump fails (i.e. the dump data would be lost).  If a single
device is registered, then diskdump is not ready to dump memory until
the command above is finished.
It is highly recommended that two or more devices are registered if this
parameter is effective.

To enable Deferred Savecore feature, edit /etc/sysconfig/diskdump appropriately
in the following format if you want to skip saving panic dump so that you can
save the dump whenever you want:

 ----------------
 SKIPSAVECORE=yes
 ----------------

If it is specified as above, then diskdump service won't save panic dump and
format the device even if the device has panic dump.  Consequently, you must
run:

 # service diskdump enabledevice

as well as the case of PRESERVEDUMP.
It is highly recommended that two or more devices are registered if this
parameter is effective.

You can set the expiration to dump data as the following format in
/etc/sysconfig/diskdump.

 ------------
 EXPIRATION=7
 ------------

If it is specified as the above, then the dump device is formatted by diskdump
service when 7 days have passed since the memory was dumped.  If it is specified
as 0 or none, then the dump data has no expiration.  Consequently, you must
run:

 # service diskdump enabledevice

as well as the case of SKIPSAVECORE.
Of course, a negative value is invalid for EXPIRATION, so diskdump service
won't start.
Note that EXPIRATION is valid only when SKIPSAVECORE is effective.

You can email someone to notify something as the following format in
/etc/sysconfig/diskdump.

 ---------------------------------------
 MAILTO="admin@foo.com, tech@ml.foo.com"
 FROM="root@serv1.foo.com"
 ---------------------------------------

If you want to notify someone of such that the memory dump still remains
in the registered device, set both MAILTO and FROM in /etc/sysconfig/diskdump
appropriately.  Both parameters should be filled with appropriate email address.
MAILTO should be specified at least one email address if you enable it.  If you
specify two or more email addresses to MAILTO, you need to separate them by
commas.
With all parameters specified appropriately, an email that is created by
a template would be sent by diskdump service automatically when saving panic
dump was skipped because of PRESERVEDUMP and/or SKIPSAVECORE.  The template is
located in /etc/diskdump/mail_template.us.  You can edit the template as you
wish.
Note that both parameters are valid only when PRESERVEDUMP and/or SKIPSAVECORE
are/is effective.

To disable Message Complement feature edit /etc/sysconfig/diskdump appropriately
in the following format:

 -----------------
 SALVAGEMESSAGE=no
 -----------------

This feature retrieves kernel dump messages from log_buf area of the saved
crash dump file and appends these messages to the end of /var/log/messages.
In almost case, the messages are the Oops, but the last kernel messages which
were not written in /var/log/messages also may be appended.  This feature only
works if just after rebooting from system panic and before syslogd and klogd
are invoked.  So the appended message will not disturb the time sequence of
messages.


Testing diskdump
----------------
To test the diskdump functionality, press [Alt] + [SysRq] + [C] or run
the command below.

 # echo c > /proc/sysrq-trigger

After completing the dump, a vmcore file will be created during the next
reboot sequence, and saved in a directory with a name of the name format:

  /var/crash/127.0.0.1-<date>

The vmcore file's format is same as that created by the netdump
facility, so you can use the crash(8) command to analyze it

Note
----
Once you set up, it is not necessary to do anything after that.
After the initial configuration process there are no additional steps
required.  Be sure to keep the designated dump partition to be sufficiently
large.  If there is not enough space, the dump file will be partially saved;
resulting in an incomplete dump file named vmcore-incomplete.

Diskdump currently contains one customizable script file called
diskdump-nospace.  The diskdump-nospace script is called prior
to the creation of the vmcore file if /var/crash does not have
enough space to hold the complete dumpfile.  The script may be
customized to clean up enough space for the dump in question
to proceed.

Tunable Parameters
------------------
Tunable parameters can be set up with /etc/module.conf as for RHEL3, or with
/etc/modprobe.conf as for RHEL4.  The diskdump module has following module
parameters:

block_order:	Specifies the dump-time I/O block size.  Default value is 2,
		which sets the I/O block size equal to "page-size << 2", or
		16 kbytes on an i386 machine.  Larger values may make for
		better performance, but occupies more module memory.

sample_rate:	Determine how many blocks in the dump partition are verified
		before actual memory dumping begins.  Default value is 8,
		which means one of every "1<<8" (256) blocks are verified.
		Specifying zero means all blocks in the partition are verified,
		and a negative value disables verification.

dump_level:	A memory collection level that specifies which memory pages
		will be dumped.  Default value of 0 dumps all pages of
		physical RAM into the vmcore file.  To avoid excessively
		large vmcore files, page cache pages, zero-filled pages,
		free pages, and user application pages may be eliminated
		from the file.  Specifying one of the dump_level values
		will skip one or more memory page type(s) if that page type
		is marked with an X in the following table:

		dump	cache	cache	zero	free	user	description
		level	page	private	page	page	page
		---------------------------------------------------------
		  0		 				default
		  1	 X	 X
		  2		 	 X
		  3	 X	 X	 X
		  4		 		 X
		  5	 X	 X		 X
		  6	 	 	 X	 X
		  7	 X	 X	 X	 X
		  8		 			 X
		  9	 X	 X			 X
		 10		 	 X		 X
		 11	 X	 X	 X		 X
		 12		 		 X	 X
		 13	 X	 X		 X	 X
		 14		 	 X	 X	 X
		 15	 X	 X	 X	 X	 X	minimum dump
		 17	 X
		 19	 X	 	 X			recommended
		 21	 X	 		 X
		 23	 X	 	 X	 X
		 25	 X	 			 X
		 27	 X	 	 X		 X
		 29	 X	 		 X	 X
		 31	 X	 	 X	 X	 X

compress:	Specify whether compression for dump data is enabled or not.
		The default value is 0, which means that memory pages will be
		dumped without compression.  If the value is 1, memory pages
		will be compressed using the deflation algorithm, GZIP.

Partial Dump feature, RHEL4 only, provides a memory collection level that can
select the amount of physical memory that is dumped.  All of physical memory is
usually not required to investigate a kernel issue.  Most of physical memory
typically contains user application data, page cache memory (file data), free
memory pages, and zero-filled pages.  By skipping one of more of those page
types when creating the vmcore file, the crash dump will be significantly
smaller, and the dump procedure less time-consuming.  While the actual vmcore
file size may vary because of the status of system and the dump_level
specified, the minimum amount of data required to analyze the dump will always
be captured.  However, since there may be circumstances where it will be
necessary to capture all of physical memory, it is not recommended that a dump
partition size be less than the actual amount of physical memory.
Note that the partial dump feature has some risks.  There are memory management
lists which are scanned for a page's memory attribute, so if the list has been
corrupted, the scanning process may fail.  For example, when specifying a
dump_level from 4-7 or from 12-15, the kernel's free page linked lists are
scanned; if the list is corrupt, diskdump may hang.  Furthermore, it is
possible that a page type that has been skipped may be necessary to fully
investigate the cause of some issues.  Therefore, a memory collection level
should be selected to suit each situation.  The recommended level is 19,
because it is easiest to determine whether a page is zero-filled or if it is a
page cache page, and because no page lists need to be traversed.

The benefit of Compressed Dump feature, RHEL4 only, is that the size of dump
device can be reduced.  The dump device to be used has to be larger than memory
size unless compression is enabled.  If there are no devices enough to dump,
then the device that its size is smaller than the system memory size can be used
by enabling compression.  Memory pages are compressed and dumped to the dump
device, and a dump file is created without decompression.  Therefore,
consumption of /var/crash can be also saved.  As another benefit, time both to
dump memory and to create a dump file could be reduced.  Note that compression
rate cannot be known in advance and that small dump device may become
insufficient even if compression is enabled.

Note: As for ppc64 machine, both Partial Dump feature and Compressed Dump
feature are available, but cannot be used together.  This will be supported
in the near future.

Example:

The following option sets I/O block size to 32 kbytes, and verification is
done on every block in the partition.  Also, cache page and zero page are
skipped by partial dump feature.

	options diskdump block_order=3 sample_rate=0 dump_level=19 compress=1

Note that the diskdump service always needs to restart after setting up
as above.  Then, run the command below in order to check on whether those
parameters were set up correctly.

 # cat /proc/diskdump

 # sample_rate: 0
 # block_order: 3
 # fallback_on_err: 1
 # allow_risky_dumps: 1
 # dump_level: 19
 # compress: 1
 # total_blocks: 262042
 #
 sde1 514080 1012095