Thursday, May 1, 2008
Configure Jumbo Frames to Boost Network Performance / Throughput
First, make sure your network driver supports custom MTU. Second you need to have a compatible gigabit NIC and switch (such as Cisco Catalyst 4000/4500 Switches with Supervisor III or Supervisor IV ) that is jumbo frame clean.
Jumbo frames can reduce server overhead such as a big decrease in CPU usage when transferring larger file. Also you should see some increase in network throughput.
Configure eth0 interface for Jumbo Frames. Warning! These examples depend upon a compatible network gear such gigabit Ethernet switches and gigabit Ethernet network interface cards.
# ifconfig eth0 mtu 9000
To make changes permanent, append the directive into the network configuration file for eth0.
# vi /etc/sysconfig/network-script/ifcfg-eth0 # Redhat/CentOS/Fedora
MTU=9000
# vi /etc/network/interfaces # Debian/Ubuntu
MTU=9000
# /etc/init.d/networking restart
To confirm the MTU used between two specific devices:
# ip route get {IP-address}
# ip route get 192.168.1.1
192.168.1.1 dev eth0 src 192.168.1.100
cache mtu 9000 advmss 1460 hoplimit 64
Application Protocol Tunning. You may need to tune the application/network protocol such as NFS and SMB to take advantage of Jumbo Frames.
Monitor and debug system calls using strace
strace is useful for diagnostic, instructional, and debugging. System admins, diagnosticians and troubleshooters will find it invaluable for solving problems with programs for which the source is not readily available.
strace for Linux, added many of the features of Solaris/FreeBSD's truss command from SVR4, and produced an strace that worked on both platforms.
$ strace -o /tmp/output.txt /bin/bash
$ grep '^open' /tmp/output.txt
To see only a trace of the open, close, read, and write system calls, enter:$ strace -e trace=open,close,read,write df > output.txt
Another good option is ltrace - its use is very similar to strace command.
Wednesday, April 30, 2008
Find out Linux Disk utilization
You can use iostat to find out disk utilization and for monitoring system input/output device loading by observing the time the physical disks are active in relation to their average transfer rates.
iostat -d -x interval count
- -d : Display the device utilization report (d == disk)
- -x : Display extended statistics including disk utilization
- interval : It is time period in seconds between two samples . iostat 2 will give data at each 2 seconds interval.
- count : It is the number of times the data is needed . iostat 2 5 will give data at 2 seconds interval 5 times
Display 3 reports of extended statistics at 5 second intervals for disk:
$iostat -d -x 5 3
Linux 2.6.18-53.1.4.el5 (moon.nixcraft.in) 12/17/2007
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 1.10 39.82 3.41 13.59 309.50 427.48 43.36 0.17 10.03 1.03 1.75
sdb 0.20 18.32 1.15 6.08 117.36 195.25 43.22 0.51 71.14 1.26 0.91
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 108.40 1.40 64.40 49.60 1382.40 21.76 0.04 0.67 0.44 2.92
sdb 0.00 37.80 0.00 245.20 0.00 2254.40 9.19 28.91 108.49 1.08 26.36
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 97.01 1.00 57.29 39.92 1234.33 21.86 0.03 0.58 0.50 2.89
sdb 0.00 38.32 0.00 288.42 0.00 2623.55 9.10 32.97 122.30 1.15 33.27 Where,
- rrqm/s : The number of read requests merged per second that were queued to the hard disk
- wrqm/s : The number of write requests merged per second that were queued to the hard disk
- r/s : The number of read requests per second
- w/s : The number of write requests per second
- rsec/s : The number of sectors read from the hard disk per second
- wsec/s : The number of sectors written to the hard disk per second
- avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
- avgqu-sz : The average queue length of the requests that were issued to the device
- await : The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
- svctm : The average service time (in milliseconds) for I/O requests that were issued to the device
- %util : Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.
To interpret the output result for optimization, first note down following values from the iostat output:
- The average service time (svctm)
- Percentage of CPU time during which I/O requests were issued (%util)
- See if a hard disk reports consistently high reads/writes (r/s and w/s)
If any one of these are high, take one of the following action:
- Get high speed disk and controller for file system (for example move from SATA I to SAS 15k disk)
- Tune software or application or kernel or file system for better disk utilization
- Use RAID array to spread the file system
For example, from about iostat report it appears that /dev/sdb under load.
Configure Linux to track and log failed login attempt recoreds
You can use faillog command to display faillog records or to .set login failure limits. Without arguments it display only list of user faillog records who have ever had a login failure.
PAM Settings: Under RHEL/CentOS 5.x, you need to modify /etc/pam.d/system-auth. You need to configure PAM module pam_tally.so. Otherwise faillog will never display failed login attempts.
PAM Configuration to recored failed login attempts: pam_tally.so maintains a count of attempted accesses, can reset count on success, can deny access if too many attempts fail.
# vi /etc/pam.d/system-auth # Appendauth required pam_tally.so no_magic_root
account required pam_tally.so deny=3 no_magic_root lock_time=180
Where,
- deny=3 : Deny access if tally for this user exceeds 3 times.
- lock_time=180 : Always deny for 180 seconds after failed attempt. There is also unlock_time=n option. It allow access after n seconds after failed attempt. If this option is used the user will be locked out for the specified amount of time after he exceeded his maximum allowed attempts. Otherwise the account is locked until the lock is removed by a manual intervention of the system administrator.
- magic_root : If the module is invoked by a user with uid=0 the counter is not incremented. The sys-admin should use this for user launched services, like su, otherwise this argument should be omitted.
- no_magic_root : Avoid root account locking, if the module is invoked by a user with uid=0
Display all failed login attempts for a user
# faillog -u didi
Display faillog records for all users
# faillog -a
-r can reset the counters of login failures or one record if used with the -u USERNAME option:
# faillog -r
# faillog -r -u vivek <-- only reset counter for vivek user
On large Linux login server, it might be useful to clear all counts every midnight or week from a cron job# crontab -e@weekly /usr/bin/faillog -r
Clonezilla Ghost - The Linux answer to Norton Ghost
Norton Ghost is slow. There is Norton Ghost Corporate Edition with multicasting and fast cloning system.
Clonezilla is a free, open source, and multitasking. With DRBL (Diskless Remote Boot in Linux [DRBL] provides a diskless or systemless environment for client machines) and network boot enabled client computers, the only thing you have to prepare is a Clonezilla server. The best thing is ~ you do not even have to prepare a bootable CD or floppy with Partition Image for every client computer.
Download Clonezilla
A simple howto on setting up Clonezilla/DRBL on Ubuntu Linux
A simple documentation to quickly setup up Clonezilla on a PXE server
Copy hard disk or partition image to another system using a network and netcat
netcat is a swiss army knife in networking. It is designed to be a reliable "back-end" tool that can be used directly or easily driven by other programs/scripts, as well as a feature-rich network debugging and exploration tool, since it can create almost any kind of connection you would need and has several interesting built-in capabilities.
One handy and trusted use is to migrating data between two server hard drives. You can also use ssh for the same purpose, but encryption adds its own overheads.
A sample setup:
-----------------------
HostA // 192.168.1.1
------------------------
sda
NETWORK
sdb
------------------------
HostB // 192.168.1.2
-------------------------
To copy /dev/sda on HostA to /dev/sdb on HostB, first login as root.
On hostB (receiving end ~ write image mode), open port:
# netcat -p 2222 -l |bzip2 -d | dd of=/dev/sdb
Where,
- -p 2222 : the source port, subject to privilege restrictions and availability. Make sure port 2222 is not used by another process.
- -l : listen for an incoming connection rather than initiate a connection to a remote host.
- bzip2 -d : Compresses image using the Burrows-Wheeler block sorting text compression algorithm, and Huffman coding. This will speed up network transfer ( -d : force decompression mode)
- dd of=/dev/sda : The hard disk. You can also specify partition such as /dev/sda1
On hostA (send data over a network ~ read image mode), login as root:
# bzip2 -c /dev/sda | netcat hostA 2222 # OR use IP:# bzip2 -c /dev/sda | netcat 192.168.1.1 2222
Apparently, this process takes its own time.
A note about latest netcat 1.84-10 and above: Above syntax will generate an error. It is an error to use -l in conjunction with the -p, -s, or -z. Additionally, any timeouts specified with -w are ignored. So use nc:
On hostA:# nc -l 2222 > /dev/sdb
On hostB:# nc hostA 2222< /dev/sda # Or# nc 192.168.1.1 2222< /dev/sda
Using hostB, connect to the listening nc process at 2222 (hostA), feeding it the file (/dev/sda) which is to be transferred.
On hostA:# nc -l 2222 | bzip2 -d > /dev/sdb
On hostB:# bzip2 -c /dev/sda | nc 192.168.1.1 2222
Restrict Users to SCP and SFTP and Block SSH Shell Access with rssh
SCP/SFTP -> sshd -> Call sftpd subsystem -> Requires a shell -> User login
rssh is a restricted shell for providing limited access to a host via ssh, it allows following operations only:
- scp - Secure file copy
- sftp - Secure FTP
- cvs - Easily retrieve old versions to see exactly which change caused the bug
- rsync - Backup and sync file system
- rdist - Backup / RDist program maintains identical copies of files on multiple hosts.
- Default configuration file is located at /etc/rssh.conf (FreeBSD - /usr/local/etc/rssh.conf)
- Default rssh binary location /usr/bin/rssh (FreeBSD - /usr/local/bin/rssh)
- Default port none - ( openssh 22 port used - rssh is shell with security features)
Login as the root user, create a new user# useradd -m -d /home/didi -s /usr/bin/rssh didi
Change user's shell:# usermod -s /usr/bin/rssh didi # or
# chsh -s /usr/bin/rssh didivivek
Now try login via ssh or sftp using username didi:$ sftp didi@my.backup.server.com # or$ ssh didi@my.backup.server.com
This account is restricted by rssh.
This user is locked out.
Connection to my.backup.server.com closed.
By default rssh locks down everything including any sort of access. Now grant access to sftp and scp for all users by appending:# vi /etc/rssh.confallowscp
allowsftp
rssh reads configuration on fly (there is no rssh daemon). Now user can run scp and sftp, without shell access:$ scp /path/to/file didi@my.backup.server.com:/. # or$ sftp didi@my.backup.server.com:/.
Connecting to lmy.backup.server.com...
didi@my.backup.server.com's password:
sftp> pwd
Remote working directory: /home/didi
sftp> Understanding command configuration options
- allowscp : Tells the shell that scp is allowed.
- allowsftp : Tells the shell that sftp is allowed.
- allowcvs : Tells the shell that cvs is allowed.
- allowrdist : Tells the shell that rdist is allowed.
- allowrsync : Tells the shell that rsync is allowed.
Tip: Create a group for rssh users, and limit executable access to the binaries to users in that group to improve security. Use standard file permissions carefully and appropriately.
Chroot jail to lock users to their home only
rssh's chrootpath option is used to set the root of the chroot jail. A chroot affects only the current process and its children. If the default home directory is /home/didi, normal user can access files in /etc, /sbin or /bin. This allows an attacker to install sneaky things via your web server in /tmp.
=> Chroot directory: /users.
Tip: If possible mount /users filesystem with the noexec/nosuid option to improve security.
=> Required directories in jail:
- /users/dev - Device file
- /users/etc - Configuration file such as passwd
- /users/lib - Shared libs
- /users/usr - rssh and other binaries
- /users/bin - Copy default shell such as /bin/csh or /bin/bash
=> Required files in jail at /users (default for RHEL / CentOS / Debian Linux):
- /etc/ld.so.cache
- /etc/ld.so.cache.d/*
- /etc/ld.so.conf
- /etc/nsswitch.conf
- /etc/passwd
- /etc/group
- /etc/hosts
- /etc/resolv.conf
- /usr/bin/scp
- /usr/bin/rssh
- /usr/bin/sftp
- /usr/libexec/openssh/sftp-server OR /usr/lib/openssh/sftp-server
- /usr/libexec/rssh_chroot_helper OR /usr/lib/rssh/rssh_chroot_helper (suid must be set on this binary)
- /bin/sh or /bin/bash (default shell)
Tip: Limit the binaries which live in the jail to the absolute minimum required to improve security. Usually /bin/bash and /bin/sh is not required but some system may give out error.
The files need to be placed in the jail directory mimic their placement in the root file system. So you need to copy all required files. For example, copy /usr/bin/rssh to /users/usr/bin/rssh.
Building the Chrooted Jail
# mkdir -p /users/{dev,etc,lib,usr,bin}
# mkdir -p /users/usr/bin
# mkdir -p /users/libexec/openssh# mknod -m 666 /users/dev/null c 1 3
# cd /users/etc
# cp /etc/ld.so.cache .
# cp -avr /etc/ld.so.cache.d/ .
# cp /etc/ld.so.conf .
# cp /etc/nsswitch.conf .
# cp /etc/passwd .
# cp /etc/group .
# cp /etc/hosts .
# cp /etc/resolv.conf .
Open /usres/group and /users/passwd file and remove root and all other accounts.
# cd /users/usr/bin
# cp /usr/bin/scp .
# cp /usr/bin/rssh .
# cp /usr/bin/sftp .
# cd /users/usr/libexec/openssh/
# cp /usr/libexec/openssh/sftp-server . # Or# cp /usr/lib/openssh/sftp-server .
# cd /users/usr/libexec/
# cp /usr/libexec/rssh_chroot_helper # Or
# cp /usr/lib/rssh/rssh_chroot_helper
# cd /users/bin/
# cp /bin/sh . # Or
# cp /bin/bash .
The library files that these binaries need can be found by using ldd / strace. For example:ldd /usr/bin/sftp
Copy all these libraries to /users/lib and other appropriate location. Or use l2chroot script:# cd /sbin
# wget -O l2chroot http://www.cyberciti.biz/files/lighttpd/l2chroot.txt
# chmod +x l2chroot
Open l2chroot and set BASE variable to point to chroot directory (jail) location:BASE="/users"# l2chroot /usr/bin/scp
# l2chroot /usr/bin/rssh
# l2chroot /usr/bin/sftp
# l2chroot /usr/libexec/openssh/sftp-server # or
# l2chroot /usr/lib/openssh/sftp-server
# l2chroot /usr/libexec/rssh_chroot_helper # or
# l2chroot /usr/lib/rssh/rssh_chroot_helper
# l2chroot /bin/sh # or# l2chroot /bin/bash # or
The syslog library function works by writing messages into a FIFO file such as /dev/log. You need to pass -a /path/to/chroot/dev/log option. This argument specify additional sockets to which syslogd has to listen. This is needed if some daemons will be running within a chroot() environment. Up to 19 additional sockets can be used. To have more, increase the symbol MAXFUNIX within the syslogd.c source file, find:# vi /etc/sysconfig/syslogSYSLOGD_OPTIONS="-m 0"
Append -a /users/dev/logSYSLOGD_OPTIONS="-m 0 -a /users/dev/log"
# /etc/init.d/syslog restart
(For Debian/Ubuntu, apply changes to /etc/default/syslogd)
Set chrootpath to /users# vi /etc/rssh.confchrootpath=/users# /etc/init.d/sshd start
Add user to the jail:# useradd -m -d /users/did -s /usr/bin/rssh didi
sftp> pwd
Remote working directory: /vivek
sftp> cd /tmp
Couldn't canonicalise: No such file or directoryPer User Configuration Options for chroot JailFrom the man page: The user configuration directive allows for the configuration of options on a per-user basis. THIS KEYWORD OVERRIDES ALL OTHER KEYWORDS FOR THE SPECIFIED USER. That is, if you use a user keyword for user foo, then foo will use only the settings in that user line, and not any of the settings set with the keywords above. The user keyword’s argument consists of a group of fields separated by a colon (:), as shown below. The fields are, in order:
- username : The username of the user for whom the entry provides options
- umask : The umask for this user, in octal, just as it would be specified to the shell access bits. Five binary digits, which indicate whether the user is allowed to use rsync, rdist, cvs, sftp, and scp, in that order. One means the command is allowed, zero means it is not.
- path : The directory to which this user should be chrooted (this is not a command, it is a directory name).
rssh examples of configuring per-user options:
# vi /etc/rssh.confuser=tom:077:00010 # All user tom to bypass our chroot jailuser=jerry:011:00100 # Provide jerry cvs access with no chrootuser=spike:011:10000 # Provide spike rsync access with no chrootuser="tyke:011:00001:/users" # Provide tyke access with chroot jail located at /usersuser=nibbles:011:00001:"/usr/local/tv/shows/tom and jerry" # Provide nibbles scp access with chroot directory, space must be quoted.