Sébastien Han

Stacker! Cepher! What's next?


Some ops commands reminder.

Put a password on your private key:

$ ssh-keygen -p -f .ssh/id_rsa
Key has comment '.ssh/id_rsa'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

Bash colored prompt (Gentoo style) and history with dates:

# History with date and red prompt
export HISTTIMEFORMAT="%Y/%m/%d - %H:%M:%S "
export PS1='\[\033[01;31m\]\u@\h\[\033[01;34m\] \w #\[\033[00m\] '

Then edit /etc/bash.bashrc with the following (red prompt root and green prompt for normal users):

if [[ ${EUID} == 0 ]] ; then
PS1='\[\033[01;31m\]\u@\h\[\033[01;34m\] \w #\[\033[00m\] '
PS1='\[\033[01;32m\]\u@\h\[\033[01;34m\] \w \$\[\033[00m\] '

MOTD OpenStack (/etc/update-motd.d/92-nova)


echo "OpenStack services state is:"
nova-manage service list|grep $(hostname) | awk '{ print $1 " " $2 " " $3 " " $4 " " "\033[1;31m" $5 "\033[0m" " " $6 " " $7 }'
echo ""

MOTD Ceph (/etc/update-motd.d/92-ceph)


echo "Ceph state is:"
/usr/bin/ceph health | awk '{print "\033[1;31m" $1-$10 "\033[0m" " " "\033[1;31m" $2 "\033[0m" " " "\033[1;31m" $3 "\033[0m" " " "\033[1;31m" $4 "\033[0m" " " "\033[1;31m" $5 "\033[0m" " " "\033[1;31m" $6 "\033[0m"}'
echo ""

Repair FS on boot, same behavior as e2fsck -p

$ sudo sed -i "s/^FSCKFIX=no$/FSCKFIX=yes/" /etc/default/rcS

e2fsck -p man page:

-p     Automatically  repair ("preen") the file system.  This option will cause e2fsck to automatically fix any filesystem problems that can be safely fixed without human intervention.  If e2fsck discovers a problem which may
             require the system administrator to take additional corrective action, e2fsck will print a description of the problem and then exit with the value 4 logically or'ed into the exit code.  (See  the  EXIT  CODE  section.)
             This option is normally used by the system's boot scripts.  It may not be specified at the same time as the -n or -y options.

Add a VLAN manually:

$ sudo vconfig add blue 3020
Added VLAN with VID == 3020 to IF -:blue:-
$ sudo ifconfig blue.3020 up

Make this persistant in /etc/network/interfaces, if you don’t want to put an ip on the device because pacemaker does it for you use the setting below:

auto blue.3020
iface blue.3020 inet static
        vlan_raw_device blue
        pre-up ifconfig $IFACE up
        post-down ifconfig $IFACE down

SMNPD logging:

# snmpd options (use syslog, close stdin/out/err).
SNMPDOPTS='-LS0-5d -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid'

Delete an IPtable rules based on row number:

$ sudo iptables -L -vnx --line-numbers
$ sudo iptables -t nat -D <chain-name> <number>


$ sudo iptables -I POSTROUTING -t nat -j MASQUERADE -s <ip-range-source> -d <ip-range-dest> -o <nic>

Quickly check which directory uses most of the space + total:

root@misc01 ~ # du -csh /var/*
8.7M /var/backups
350M /var/cache
4.0K /var/crash
5.6G /var/lib
4.0K /var/local
0 /var/lock
1.2G /var/log
196K /var/mail
4.0K /var/opt
0 /var/run
52K /var/spool
4.0K /var/tmp
12K /var/www
7.1G total

Verifying that a Private Key Matches a Certificate.

Fix a broken MySQL replication (bad transaction):

STOP SLAVE; SET GLOBAL sql_slave_skip_counter = 1; START SLAVE;

Get a core dump:

$ sudo ulimit -c unlimited
$ echo /my/wanted/path/ > /proc/sys/kernel/core_pattern
$ sudo kill -SEGV `ps faux | grep [c]eph-osd | awk '{print $2}'`

Show header HTTP with tcpdump

$ tcpdump -s 1024 -l -A src or dst

Bringing up and down a bond without restart the server, interfaces conf:

auto eth0
allow-lebond eth0
iface eth0 inet manual
        bond-master     lebond

auto eth1
allow-lebond eth1
iface eth1 inet manual
        bond-master     lebond

auto lebond
iface lebond inet manual
        bond-slaves     none
        bond-mode       active-backup
        bond-miimon     100

Then issue:

$ sudo ifdown eth0 eth1 lebond && sudo ifup eth0 eth1 lebond

Stupid MySQL error:

Unable to use slave's temporary directory /tmp - Can't create/write to file '/tmp/SQL_LOAD-' (Errcode: 17)

Solved by:

$ sudo rm /tmp/SQL_LOAD-


mysql> FLUSH LOGS;

LVS status:

$ sudo ipvsadm -Ln -t <Public IP>:80
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP <Public IP>:80 wlc
-> Masq 100 1 4
-> Masq 100 0 0

ext4 without reservation for root:

$ sudo mkfs.ext4 -m0

Online extend an LV:

$ sudo lvextend -L +50G -r /dev/mapper/rootvg-seb

MySQL must not swap. This can be managed via /proc/sys/vm/swappiness, this value determines how aggressive is the system in term of swapping. Default is 60, for MySQL server 0 is recommended. Setting 0 does not mean that you never swap, the system will only swap to prevent out of memory.

$ sudo sysctl -w vm.swappiness=0
$ sudo echo "vm.swappiness=0" >> /etc/sysctl.conf
$ sudo swapoff -a && swapon -a

Nice output of device structure, filesystem, device mapper:

$ sudo lsblk -f
└─vda1 ext4 cloudimg-rootfs /

$ sudo lsblk -t
vda 0 512 0 512 512 1 deadline 128
└─vda1 0 512 0 512 512 1 deadline 128

Bandwidth limiting pipe with throttle and tc.