Assuming that /var/lib/ceph/mds/mds is the mds data point.

Edit ceph.conf and add a MDS section like so:

[mds]
  mds data = /var/lib/ceph/mds/mds.$id
  keyring = /var/lib/ceph/mds/mds.$id/mds.$id.keyring

[mds.0]
  host = {hostname}

Create the authentication key (only if you use cephX):

1

$ sudo ceph auth get-or-create mds.0 mds 'allow ' osd 'allow *' mon 'allow rwx' > /var/lib/ceph/mds/mds0/mds.0.keyring

Eventually start the service

1
2
3
4

$ sudo service ceph start mds.0
=== mds.0 ===
Starting Ceph mds.0 on ceph...
starting mds.0 at :/0

Check the status of the cluster:

1
2
3
4
5
6

$ ceph -s
   health HEALTH_OK
   monmap e3: 1 mons at {1=192.168.251.100:6790/0}, election epoch 1, quorum 0 1
   osdmap e318: 2 osds: 2 up, 2 in
    pgmap v8214: 280 pgs: 280 active+clean; 3818 MB data, 9545 MB used, 10432 MB / 19978 MB avail
   mdsmap e70: 1/1/1 up {0=0=up:active}

Note if you want to add more MDSs, they will appear like this:

1
2
3
4
5
6

$ ceph -s
   health HEALTH_OK
   monmap e3: 1 mons at {1=192.168.251.100:6790/0}, election epoch 1, quorum 0 1
   osdmap e318: 2 osds: 2 up, 2 in
    pgmap v8214: 280 pgs: 280 active+clean; 3818 MB data, 9545 MB used, 10432 MB / 19978 MB avail
   mdsmap e70: 1/1/1 up {0=0=up:active}, 1 up:standby

Easy, isn’t it? FYI filesystem metadata live in RADOS cluster. So MDS servers are quite ephemeral daemons. Don’t be surprised if you don’t find anything (expect the MDS key) inside the mds data directory.

The title of the article is not that explicit, actually I had trouble to find a proper one. Thus let me clarify a bit. Here is the context I was wondering if Glance was capable of converting images within its store. The quick answer is no, but I think such feature is worth to be implemented. Glance could be able to convert a QCOW2 image to a RAW format. Usually if you already have an image within let’s say a Ceph cluster (RBD), you have to download the image (since you probably don’t have the source image file anymore), then manually convert it with qemu-img (QCOW2 –> RAW) and eventually import it into Glance. Enough talk about this, I’ll address this in a future article. For now let’s stick to the first matter. Imagine that you have a KVM cluster backed by a Ceph Cluster and your CTO wants you to migrate the whole environment to OpenStack because it’s trendy (joking, OpenStack just rocks!). You’re not going to backup all your images and then build a new cluster or something like that, you might want OpenStack (Glance) to be aware of your Ceph cluster. Generally speaking you just have to connect Glance to one of your image pool. After this, the only thing to do is to create (it’s more registering the images ID and metadata than creating a new image) into Glance. No worries here’s the explanation. Longest introduction ever.

In this article, I’m assuming that Glance is already connected to Ceph and to the proper RBD pool. Before starting anything, please understand that within the current Grizzly stable branch, the RBD backend is not implemented. That’s funny because we don’t need that much to implement it. The bug report is on launchpad and the proposed feature is under review on Gerrit.

However if you want to enable the fix now:

• Go to the line 278 of /opt/stack/glance/glance/api/v1/images.py
• Then simply edit the line like so:

1

for scheme in ['s3', 'swift', 'http', 'rbd']:

Let’s test this!

Get the image size from the rbd client:

1
2
3
4
5
6

$ rbd -p images info ubuntu-raw
rbd image 'ubuntu-raw':
size 2048 MB in 512 objects
order 22 (4096 KB objects)
block_name_prefix: rb.0.3ded.2eb141f2
format: 1

Eventually create/register the new image:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

$ glance image-create --size 2147483648 --name ubuntu-rbd --store rbd --disk-format raw --container-format ovf --location rbd://ubuntu-raw
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | None                                 |
| container_format | ovf                                  |
| created_at       | 2013-05-06T15:29:26                  |
| deleted          | False                                |
| deleted_at       | None                                 |
| disk_format      | raw                                  |
| id               | 0d47c421-b079-44ff-bcc5-ee711d500512 |
| is_public        | False                                |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | ubuntu-rbd-hack                      |
| owner            | 19292b3b597b4ecc9a41103cc312a42f     |
| protected        | False                                |
| size             | 2147483648                           |
| status           | active                               |
| updated_at       | 2013-05-06T15:29:26                  |
+------------------+--------------------------------------+

R Note about the URI from the --location option, there are 2 way to build it, it can be:

• rbd://<fsid>/<pool>/<image>/<snapshot>
• rbd://<image-name> ; Glance will figured out the pool since you put it into the Glance configuration.

It either 1 or 4 field(s).

Of course the example was only with one image but the method will definitely work for a whole Ceph cluster with tons of images!

Slides are available on Slideshare

See you in Hong Kong! Cheers!

Grizzly brought the multi-backend functionality to cinder and tons of new drivers. The main purpose of this article is to demonstrate how we can take advantage of the tiering capability of Ceph.

I. Ceph

To configure Ceph to use different storage devices see my previous article: Ceph 2 speed storage with CRUSH.

II. Cinder

Assuming your 2 pools are called:

• rbd-sata points to the SATA rack
• rbd-ssd points to the SSD rack

II.1 Configuration

Cinder configuration file:

# Multi backend options

# Define the names of the groups for multiple volume backends
enabled_backends=rbd-sata,rbd-ssd

# Define the groups as above
[rbd-sata]
volume_driver=cinder.volume.driver.RBDDriver
rbd_pool=cinder-sata
volume_backend_name=RBD_SATA
# if cephX is enable
#rbd_user=cinder
#rbd_secret_uuid=<None>
[rbd-ssd]
volume_driver=cinder.volume.driver.RBDDriver
rbd_pool=cinder-ssd
volume_backend_name=RBD_SSD
# if cephX is enable
#rbd_user=cinder
#rbd_secret_uuid=<None>

Unfortunately the rbd driver doesn’t support this variable yet. This feature has been submitted here: https://review.openstack.org/#/c/27535/.

Then create the pointers:

1
2
3
4
5
6
7
8
9

$ cinder type-key ssd set volume_backend_name=RBD_SSD
$ cinder type-key sata set volume_backend_name=RBD_SATA
$ cinder extra-specs-list
+--------------------------------------+------+---------------------------------------+
|                  ID                  | Name |              extra_specs              |
+--------------------------------------+------+---------------------------------------+
| b1522968-e4fa-4372-8ac4-3925b7c79ee1 | ssd  |  {u'volume_backend_name': u'RBD_SSD'} |
| b50bf5a3-6044-4392-beeb-432302f6421c | sata | {u'volume_backend_name': u'RBD_SATA'} |
+--------------------------------------+------+---------------------------------------+

Then restart cinder services:

1

$ sudo restart cinder-api ; sudo restart cinder-scheduler ; sudo restart cinder-volume

Eventually create 2 volume type, one for each backend:

1
2
3
4
5
6
7
8
9
10
11
12
13

$ cinder type-create ssd
+--------------------------------------+------+
|                  ID                  | Name |
+--------------------------------------+------+
| b1522968-e4fa-4372-8ac4-3925b7c79ee1 | ssd  |
+--------------------------------------+------+

$ cinder type-create sata
+--------------------------------------+------+
|                  ID                  | Name |
+--------------------------------------+------+
| b50bf5a3-6044-4392-beeb-432302f6421c | sata |
+--------------------------------------+------+

II.2. Play with it

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

$ cinder create --volume_type ssd --display_name vol-ssd 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2013-04-22T14:54:53.917580      |
| display_description |                 None                 |
|     display_name    |               vol-ssd                |
|          id         | 4c777d96-66e4-4f85-815c-92d4503c5c8c |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 ssd                  |
+---------------------+--------------------------------------+

$ cinder create --volume_type ssd --display_name vol-sata 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2013-04-22T14:54:58.831327      |
| display_description |                 None                 |
|     display_name    |               vol-sata               |
|          id         | 8e347bd1-2044-40a2-ae87-ee9a23cddd71 |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 ssd                  |
+---------------------+--------------------------------------+

Does it work?

1
2
3
4
5

$ rbd -p cinder-ssd ls
volume-8e347bd1-2044-40a2-ae87-ee9a23cddd71

$ rbd -p cinder-sata ls
volume-4c777d96-66e4-4f85-815c-92d4503c5c8c

It’s nice that the multi-backend came with Cinder, we are gradually getting to enjoy the full power of Ceph!

Materials to start playing with Ceph. This Vagrant box contains a all-in-one Ceph installation.

I. Setup

First Download and Install Vagrant.

Download the Ceph box: here. This box contains one virtual machine:

• Ceph VM contains 2 OSDs (1 disk each), 1 MDS, 1 MON, 1 RGW. A modified CRUSH Map, it simply represents a full datacenter and applies a replica per OSD
• VagrantFile for both VM client and ceph
• Other include files

Download an extra VM for the client here, note that Debian and Red Hat based system work perfectly, thus it’s up to you:

• Client: just an Ubuntu installation

Initialize the Ceph box:

1
2
3
4
5
6
7
8
9

$ wget https://www.dropbox.com/s/hn28qgjn59nud6h/ceph-all-in-one.box
...
...
$ vagrant box add ceph-all-in-one ceph-all-in-one.box
[vagrant] Downloading with Vagrant::Downloaders::File...
[vagrant] Copying box to temporary location...
[vagrant] Extracting box...
[vagrant] Verifying box...
[vagrant] Cleaning up downloaded box...

Initialize the Client box:

1
2
3
4
5
6
7

$ wget http://dl.dropbox.com/u/1537815/precise64.box
$ vagrant box add ubuntu-12.04.1 precise64.box
[vagrant] Downloading with Vagrant::Downloaders::File...
[vagrant] Copying box to temporary location...
[vagrant] Extracting box...
[vagrant] Verifying box...
[vagrant] Cleaning up downloaded box...

Check your boxes:

1
2
3

$ vagrant box list
ceph-all-in-one
ubuntu-12.04.1

Import all the files from the box:

1
2
3

$ mkdir setup
$ cp /Users/leseb/.vagrant.d/boxes/ceph-all-in-one/include/* setup/
$ mv setup/_Vagrantfile Vagrantfile

In order to make the setup easy, I assume that your working directory is $HOME/ceph. At the end, your tree directory looks like this:

.
├── Vagrantfile
├── ceph-all-in-one.box
├── precise64.box
└── setup
    ├── ceph.conf
    ├── ceph.sh
    └── keyring

II. Start it!

Check the state of your virtual machines:

1
2
3
4
5
6
7
8
9

$ vagrant status
Current VM states:

client                   poweroff
ceph                     poweroff

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

Eventually run them:

1
2
3

$ vagrant up ceph && vagrant up client
...
...

The next time, you’ll run the client, run it this way to don’t re-provision the machine:

1

$ vagrant up --no-provision client

Eventually SSH on your client:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

$ vagrant ssh client
...

vagrant@ceph:~$ sudo ceph -s
   health HEALTH_OK
   monmap e3: 1 mons at {1=192.168.251.100:6790/0}, election epoch 1, quorum 0 1
   osdmap e179: 2 osds: 2 up, 2 in
    pgmap v724: 96 pgs: 96 active+clean; 9199 bytes data, 2071 MB used, 17906 MB / 19978 MB avail; 232B/s wr, 0op/s
   mdsmap e54: 1/1/1 up {0=0=up:active}

vagrant@ceph:~$ sudo ceph osd tree
# id  weight  type name up/down reweight
-1  2 root default
-4  2   datacenter dc
-5  2     room laroom
-6  2       row larow
-3  2         rack lerack
-2  2           host ceph
0 1             osd.0 up  1
1 1             osd.1 up  1

III. Bonus upgrade to Cuttlefish

It’s fairly easy to upgrade the box to last stable version Cuttlefish. For this simply edit /etc/apt/sources.list.d/ceph.list/ceph.list with the following:

deb http://ceph.com/debian-cuttlefish/ precise main

Then run:

1
2
3
4

$ sudo apt-get update && apt-get install ceph
$ sudo service ceph restart
$ sudo ceph -v
ceph version 0.61 (237f3f1e8d8c3b85666529860285dcdffdeda4c5)

R Note: if for some reasons you get a status were only 1/2 OSDs are up, just restart the mon. This should do the trick :-).

I use this box everyday for all my test, it’s quite handy to destroy and rebuild it within a minute. Build, destroy, build destroy, I think you got it! Hope it ;-)

Sometimes it’s just funny to experiment the theory, just to notice “oh well it works as expected”. This is why today I’d like to share some experiments with 2 really specific flags: noout and nodown. Behaviors describe in the article are well known because of the design of Ceph, so don’t yell at me: ‘Tell us something we don’t know!’, simply see this article a set of exercises that demonstrate some Ceph internal functions :-).

I. What do they do?

Flags definitions:

• noout: an OSD marked as out means that it might be running but doesn’t actually receive any data since it’s not part of the CRUSH Map (opposite of being marked in). Thus the option noout prevents OSDs from being marked out of the cluster.
• nodown: an OSD marked as down means that it’s unresponsive to the health check of its peers, thus a weight of 0 is put and the OSD won’t receive any data, this prevents clients from writing to it. However, note that the OSD is still part of the CRUSH map. At the end using the nodown option forces all the OSD to always remain with a weight of 1 (something else, but Ceph won’t change the set value).

II. Experiments

II.1. noout on a running cluster

It’s interesting to look at the PG behavior. Example sample with the PG number 3.4::

1
2

$ ceph pg dump | egrep ^3.4
3.4 0 0 0 0 0 0 0 active+clean  2013-03-27 18:20:33.847308  0'0 57'10 [1,4] [1,4] 0'0 2013-03-27 18:20:33.847246  0'0 2013-03-27 18:20:33.847246

We can see that the primary OSD of the pg number 3.4 is the osd.1 thanks to the field ‘active’ with [1,4], where the first active OSD represents the primary OSD.

Then apply the flag:

1
2

$ ceph osd set noout
set noout

You get notified from the cli:

1
2
3
4
5
6

$ ceph -s
   health HEALTH_WARN noout flag(s) set
   monmap e3: 3 mons at {0=192.168.252.10:6789/0,1=192.168.252.11:6789/0,2=192.168.252.12:6789/0}, election epoch 14, quorum 0,1,2 0,1,2
   osdmap e63: 6 osds: 6 up, 6 in
   pgmap v285: 200 pgs: 200 active+clean; 20480 KB data, 391 MB used, 23452 MB / 23844 MB avail
   mdsmap e1: 0/0/1 up

Just recall that OSD can have different states depending on the object:

• primary: it manages

  • replication to secondary OSDs
  • data re-balancing
  • recovery from failure
  • data consistency (scrubbing operations)

• secondary:

  • acts as a slave from the primary and receives order from it

Then stop the primary OSD:

1
2
3
4

$ sudo service ceph stop osd.1

$ ceph pg dump | egrep ^3.4
3.4 0 0 0 0 20971520  595 595 active+degraded 2013-03-27 18:29:52.215491  61'5  60'20 [4] [4] 0'0 2013-03-27 18:20:33.847246  0'0 2013-03-27 18:20:33.847246

Now only the OSD 4 is active and switch to primary for this PG, this one will receive all the IO operations. Under normal circonstances (and because the init script does it), the OSD should be marked as out automatically. Then a new secondary OSD will be elected.

Create an object and it into RADOS:

1
2
3
4
5
6
7
8

$ dd if=/dev/zero of=seb bs=10M count=2
2+0 records in
2+0 records out
20971520 bytes (21 MB) copied, 0.0386746 s, 542 MB/s

$ sync

$ rados put seb seb

Yes it’s there inside osd.4 (as expected):

1
2

$ sudo ls /var/lib/ceph/osd/osd.4/current/3.4_head/
seb__head_3E715054__3

Obviously you won’t find anything in the old primary OSD (1). Of course a pg dump confirms that we have one object (second field):

1
2

$ ceph pg dump | egrep ^3.4
3.4 1 0 1 0 20971520  595 595 active+degraded 2013-03-27 18:29:52.215491  61'5  60'20 [4] [4] 0'0 2013-03-27 18:20:33.847246  0'0 2013-03-27 18:20:33.847246

Now restart the OSD process and unset the noout value:

1
2
3
4

$ sudo service ceph start osd.1

$ ceph osd unset noout
unset noout

OSD 1 will get re-promoted as primary, OSD 4 as secondary and the object will be replicated from osd.4 to osd.1. A new pg dump can attest this:

1
2

$ ceph pg dump | egrep ^3.4
3.4 1 0 0 0 20971520  595 595 active+clean  2013-03-27 18:41:50.970358  61'5  62'20 [1,4] [1,4] 0'0 2013-03-27 18:20:33.847246  0'0 2013-03-27 18:20:33.847246

Stop and think, ok, what did we learn from this exercise? Well you already might have guess, this flag is ideal to perform maintenance operations. Ok you can’t satisfy the wished number of replica but this is a temporary procedure and really like the flexibility that Ceph brings here. Now let’s switch to the nodownoption which is a complete different story.

II.2. nodown on a running cluster

Apply the flag:

1
2
3
4
5
6
7
8
9

$ ceph osd set nodown
set nodown

$ ceph -s
   health HEALTH_WARN nodown flag(s) set
   monmap e3: 3 mons at {0=192.168.252.10:6789/0,1=192.168.252.11:6789/0,2=192.168.252.12:6789/0}, election epoch 14, quorum 0,1,2 0,1,2
   osdmap e66: 6 osds: 6 up, 6 in
   pgmap v294: 200 pgs: 200 active+clean; 20480 KB data, 395 MB used, 23448 MB / 23844 MB avail
   mdsmap e1: 0/0/1 up

Create an object and it into RADOS:

1
2
3
4
5
6
7
8

$ dd if=/dev/zero of=baba bs=10M count=2
2+0 records in
2+0 records out
20971520 bytes (21 MB) copied, 0.0428323 s, 490 MB/s

$ sync

$ rados put baba baba

Annnnnd the cluster HANG,hanging hanging hanging… !!!!!!

Simply because a way or another the synchronous and the atomicity of the write request can’t be satisfied. Either the primary or secondary OSD remain with a weight that makes it available to receive data from clients.

Eventually you end up with the following WARNINGS logs:

osd.4 [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.211296 secs
osd.4 [WRN] slow request 30.211296 seconds old, received at 2013-03-27 19:01:58.127010: osd_op(client.4757.0:1 baba [writefull 0~4194304] 3.f9c3dd2e) v4 currently waiting for subops from [1]
osd.4 [WRN] 1 slow requests, 1 included below; oldest blocked for > 60.235452 secs
osd.4 [WRN] slow request 60.235452 seconds old, received at 2013-03-27 19:01:58.127010: osd_op(client.4757.0:1 baba [writefull 0~4194304] 3.f9c3dd2e) v4 currently waiting for subops from [1]

That’s normal since the OSD is down but appears as up, so the client tries to write the object to it…

However it’s interesting to note that stripping model can be easily catch. Here the primary was osd.4 so ok the first 4M was written:

1

-rw-r--r--  1 root root 4.0M Mar 27 19:10 baba__head_F9C3DD2E__3

From this, it’s fairly easy to determine how writes objects. Ceph is writing 4M per 4M blocks and wait, see the following process:

--> first 4M osd.primary journal --> osd.primary --> osd.secondary journal --> osd.secondary --> second 4M osd.primary journal --> osd.primary --> and so on...

This command will change the behavior to 8M:

1

$ rados -b 8388608 put baba baba

Stop and think, ok, what did we learn from this exercise? Well I might have miss something, thus if one Inktank’s fellow is around, I’ll be happy to learn the idea behind this option, because I simply can’t think about a proper usage of it.

Conclusion:

I hope you enjoyed (maybe learn?) from those exercises. The main point of this article was to show that you can easily operate in degraded mode with the noout option. For a more technical depth read the Ceph documentation.

Next week is the OpenStack Summit conference. eNovance’s team will be present at the event. The convention is going to be awesome, I saw plenty of amazing sessions and I really look forward to attending those talks. Speaking about session, Emilien Macchi and I will give a talk about High Availability in OpenStack. You can check the talk page here. Hope to see you there!

Just started with Grizzly and already been through some minor issues :).

After a fresh installation, while trying to start all the Nova services, I came across the following error from the logs. See below the scheduler logs:

TRACE nova AMQPChannelException: (406, u"PRECONDITION_FAILED - inequivalent arg 'x-ha-policy'for queue 'scheduler' in vhost '/':
received the value 'all' of type 'longstr' but current is none", (50, 10), 'Channel.queue_declare')

This error is quite explicit and not related to OpenStack. Somehow a queue with the same name was already living in this vhost, for which the x-ha-policy was set to none thus while I restarted the nova-scheduler, it reclared the queue with a different policy like so x-ha-policy: all. At the end, we need to purge all the queues to avoid any conflicts.

Quick fix:

1
2
3

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl reset
$ sudo rabbitmqctl start_app

Bonus, this how to enable the mirrored queues in your nova.conf:

rabbit_hosts = <ip-rabbit-server1>:5672,<ip-rabbit-server2>:5672
rabbit_ha_queues = True

RabbitMQ needs to be confirmed in clustering mode, see RabbitMQ official documentation.

Enjoy your mirrored queues :D

• Testing environment is Vagrant ready.
• Bobtail Debian latest stable version will be installed
• The module only supports CephX, at least for now
• Generic deployment for 3 monitors based on a template file examples/common.sh which respectively includes mon.sh, osd.sh, mds.sh.
• Generic deployment for N OSDs. OSD disks need to be set from the examples/site.pp file (line 71). Puppet will format specified disks in XFS (only filesystem implemented) using these options: -f -d agcount=<cpu-core-number> -l size=1024m -n size=64k and finally mounted with: rw,noatime,inode64. Then it will mount all of them and append the appropriate lines in the fstab file of each storage node. Finally the OSDs will be added into Ceph.

All the necessary materials (sources and how-to) are publicly available (and for free) under AGPL license on eNovance’s Github. Those manifests do the job quite nicely, although we still need to work on MDS (90% done, just need a validation), RGW (0% done) and a more flexible implementation (authentication and filesystem support). Obviously comments, constructive critics and feedback are more then welcome. Thus don’t hesitate to drop an email to either François (f.charlier@enovance.com) or I (sebastien@enovance.com) if you have further questions.

A Placement Group (PG) aggregates a series of objects into a group, and maps the group to a series of OSDs. A common mistake while creating a pool is to use the rados command which by default creates a pool of 8 PGs. Sometime you don’t properly know how to set this value thus you use the ceph command but put an extremely high value for it. Both case are bad and could lead to some unfortunate situations. In this article, I will explore some methods to work around this major problem.

Short and experimental solution, you should not use, please just note that this command has just been implemented so running it could result in data loss. Currently some patches are set in review so this command is not stable yet. If you really want to try it, I suggest to play with only against a test cluster, so once again DON’T TRY THIS ON A PRODUCTION CLUSTER, merci :-).

1

$ ceph osd pool set <poolname> pg_num <numpgs> --allow-experimental-feature

See the example below:

1
2

$ ceph osd pool set monpool pg_num 512 --allow-experimental-feature
set pool 512 pg_num to 512

Clean and perfectly safe work around:

1
2
3
4

$ ceph osd pool create <my-new-pool> <pg_num>
$ rados cppool <my-old-pool> <my-new-pool>
$ ceph osd pool delete <my-old-pool>
$ ceph osd pool rename <my-new-pool> <my-old-pool>

It’s one of the good feature that must be implemented since Ceph is designed to scale under the infinite, the pg_num could grow as the cluster does.

Somehow one of my MySQL server had the variable general_log set to 1, which is pretty anoying because the file keeps growing and growing. See the symptom below:

1
2
3
4
5
6

$ sudo lsof /var/log/mysql
COMMAND  PID  USER   FD   TYPE DEVICE    SIZE/OFF NODE NAME
mysqld  6849 mysql    3u   REG  252,7         160   12 /var/log/mysql/mysql-bin.index
mysqld  6849 mysql   11w   REG  252,7 19042985830   14 /var/log/mysql/mysql.log.1 (deleted)
mysqld  6849 mysql   40w   REG  252,7    52876402   17 /var/log/mysql/mysql-bin.000029
mysqld  6849 mysql   72u   REG  252,7    52876402   17 /var/log/mysql/mysql-bin.000029

Don’t forget to edit your my.cnf and then change the value of the global variable:

1
2

mysql> SET GLOBAL general_log=0;
Query OK, 0 rows affected (13.17 sec)

This might take a couple of seconds to remove the file entirely, you can still watch your lsof command and see the size dicreasing.

As always, I hope it helps :)

One of the most annoying thing is when the resolv.conf of your VM keeps changing because of the information sent by the DNSMASQ process. In this article, I assume that your setup has some conventions such as:

• One network per customer, with fixed_ips range with something like 10.100.$ID.0/24
• One ID (number) per tenant

First create a template file for your initial dhclient configuration file. Using a template file makes things easier to build base images.

1

$ sudo cp /etc/dhcp/dhclient.conf /etc/dhcp/dhclient.conf.template

Then append the following lines in /etc/dhcp/dhclient.conf.template:

supersede domain-name-servers 10.100.X.254;
supersede domain-name "template.your-super-cloud.domain";

The above example shows my fixed_ips range. You might have to edit it with your own range.

Eventually edit your /etc/rc.local with the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Grab the tenant name
TENANT=$(curl -f http://169.254.169.254/latest/meta-data/hostname | cut -f 1 -d '-')

# Grab the tenant ID
ID=$(echo $TENANT | cut -c 1-2)

# Update the DHCP conf
\cp /etc/dhcp/dhclient.conf.template /etc/dhcp/dhclient.conf

sed -i s/template/$TENANT/ /etc/dhcp/dhclient.conf
sed -i s/X/$ID/ /etc/dhcp/dhclient.conf

/usr/bin/killall dhclient3
/sbin/dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -1 eth0

Enjoy!

First check your slave status:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.20.1.51
                  Master_User: replication
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000029
          Read_Master_Log_Pos: 52089474
               Relay_Log_File: mysqld-relay-bin.000036
                Relay_Log_Pos: 52089620
        Relay_Master_Log_File: mysql-bin.000029
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 52089474
              Relay_Log_Space: 52089820
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1

As we can see the current Master_Log_File is mysql-bin.000029, so we have to purge all binaries prior to this one. Then check the master binary logs status

1
2
3
4
5
6
7
8
9
10
11
12
13

mysql> SHOW BINARY LOGS;
+------------------+-----------+
| Log_name         | File_size |
+------------------+-----------+
| mysql-bin.000022 | 104858289 |
| mysql-bin.000023 | 104857671 |
| mysql-bin.000024 | 104857898 |
| mysql-bin.000025 | 104857690 |
| mysql-bin.000026 | 104860121 |
| mysql-bin.000027 | 104858120 |
| mysql-bin.000028 | 104858421 |
| mysql-bin.000029 |  52091311 |
+------------------+-----------+

In my case, I want a retention of 5 days so I can start by deleting from mysql-bin.000024 to mysql-bin.000022.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

mysql> PURGE BINARY LOGS TO 'mysql-bin.000025';
Query OK, 0 rows affected (0.45 sec)

mysql> SHOW BINARY LOGS;
+------------------+-----------+
| Log_name         | File_size |
+------------------+-----------+
| mysql-bin.000025 | 104857690 |
| mysql-bin.000026 | 104860121 |
| mysql-bin.000027 | 104858120 |
| mysql-bin.000028 | 104858421 |
| mysql-bin.000029 |  52313463 |
+------------------+-----------+
5 rows in set (0.00 sec)

Eventually set the new value for expire_logs_days and don’t forget to edit your my.cnf:

1
2
3
4
5
6
7
8
9
10
11

mysql> SET GLOBAL expire_logs_days=5;
Query OK, 0 rows affected (0.00 sec)


mysql> SHOW VARIABLES LIKE 'expire_logs_days';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| expire_logs_days | 5     |
+------------------+-------+
1 row in set (0.00 sec)

• http://dev.mysql.com/doc/refman/5.5/en/dynamic-system-variables.html

The title of the article is a bit wrong, but it’s certainly the easiest to understand :-).

First let’s create a new pool, and call it webdata. Ideally this pool will store web content.

1
2

$ ceph osd create webdata 500
successfully created pool webdata

Then grab the pool id:

1
2

$ ceph osd dump | grep webdata
pool 5 'webdata' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 500 pgp_num 500 last_change 12 owner 0

Eventually assign

1
2

$ ceph mds add_data_pool 5
added data pool 5 to mdsmap

Mount the Ceph Filesystem:

1

$ sudo mount -t ceph 172.17.1.7:/ /srv/mds/pools/webdata

Check the layout of the directory, as we can see the pool with the id 0 has been assigned by default to. This pool corresponds to the default pool called data. By setting a new layout, we will change the default pool by the one we created before.

1
2
3
4
5
6
7
8
9
10
11
12
13

$ cephfs /srv/mds/pools/webdata/ show_layout
layout.data_pool:     0
layout.object_size:   4194304
layout.stripe_unit:   4194304
layout.stripe_count:  1

$ cephfs /srv/mds/pools/webdata/lol/ set_layout -p 5 -u 4194304 -c 1 -s 4194304

$ cephfs /srv/mds/pools/webdata/lol/ show_layout
layout.data_pool:     5
layout.object_size:   4194304
layout.stripe_unit:   4194304
layout.stripe_count:  1

Unmount and mount the Ceph Filesystem:

1
2
3

$ sudo umount /srv/mds/pools/webdata
$ sudo mount -t ceph 172.17.1.7:/ /srv/mds/pools/webdata
$ sudo touch /srv/mds/pools/webdata/marche

Oh! There are objects in there!

1
2
3
4
5
6
7
8
9
10
11
12

$ rados --pool=webdata ls
10000000008.00000000
10000000008.00000001
10000000008.00000002
10000000008.00000003
10000000008.00000004

$ ll /srv/mds/pools/webdata
total 18436
drwxr-xr-x 1 root root 18874368 Jan 11 13:51 ./
drwxr-xr-x 3 root root     4096 Jan 11 12:19 ../
-rw-r--r-- 1 root root 18874368 Jan 11 13:51 marche

References:

• http://ceph.com/docs/master/man/8/mount.ceph/
• http://ceph.com/docs/master/man/8/cephfs/
• http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/6148

It’s fair to say that the geo-replication is one of the most requested feature by the community. This article is draft, a PoC about Ceph geo-replication.

Disclaimer: yes this setup is tricky and I don’t guarantee that this will work for you.

I. The idea

The original idea came out from a discussion with a friend of mine Tomáš Šafranko. The problem was that wanted to deploy acccross two (really) close datacenters with very low latencies, but we got only 2 datacenters… Ceph monitors number has to be uneven in order to properly manage the membership. That’s make the setup even harder, since there is no asynchronous mode at the moment, we just have to deal with this design. The ideal sceanario is to have 3 datacenters and each of them hosts one monitor, which means that your data are spread accross 3 datacenters as well. An another solution could be to store your data on 2 datacenters and use a VPS (close from your 2 DCs) as a monitor. Both scenarios were not possible, just because the VPS would have brought way more latencies. So we thought, thought and dug deeper and all of the sudden the Pacemaker idea came out. As far I’m concerned, I don’t use Pacemaker to manage Ceph daemons, to be honest I’m a bit reluctant to use it. It’s not that I don’t trust it, it’s just that I try to keep as much control as possible on my daemons. Moreover I’d liked to keep away the Pacemakerized everything syndrom. Beside of this, I have to admit that Pacemaker can be useful here as an automatic restart solution after a daemon crash for instance (this is the main purpose of current RAs). But let’s be honest daemons don’t crash for nothing, they just run and if they don’t (crash) they probably have a good reason to behave like this (disk full or whatever). Basically if something goes wrong I prefer to act manually in this kind of situation, investigate and understand what happened. Anyway this can be a long debat and this is not the purpose of this introduction. Once again it’s my own opinion, I like Pacemaker, I use it everyday, simply not for everything ;-).

The following drawing describes the Pacemaker idea, 2 monitors are fully actives on one location and a third one runs on one side of the cluster and then move back to the other DC if the latest fails. Sounds easy right?

                                                            ,-----------.
                                                            |  clients  |
                                                            `-----------'
                                                 ,---------.     |     ,---------.
                                                 | clients |     |     | clients |
                                                 `---------'     |     `---------'
                                                            \    |    /
                                                             \ ,---. /
                                                             ('     `)
                                                            (   WAN   )
                                                             (.     ,)
                                                             / `---' \
                                                            /         \
                                                 ,---------.           ,---------.
                                                 |  mon.0  |           |  mon.1  |
                                                 `---------'           `---------'
                                                      |                     |
                                                 ,---------.           ,---------.
                                                 |  osd.0  |           |  osd.1  |
                                                 `---------'           `---------'
                                                 |  osd.2  |           |  osd.3  |
                                                 `---------'           `---------'
                                                 |  osd.4  |           |  osd.5  |
                                                 `---------'           `---------'
                                                 |  .....  |           |  .....  |
                                                 `---------'           `---------'
                                                Data Center 1         Data Center 2
                                                           \             /
                                                            \           /
                                                     ,-----------------------.
                                                     |     Floating mon.2    |
                                                     |     Active/Passive    |
                                                     |  Managed by Pacemaker |
                                                     `-----------------------'

II. How-to

A bit of a technical overview:

• DRBD manages the Monitor data directory
• Pacemaker manages the Monitor daemon and his IP address

II.1. DRBD setup

First install DRBD:

1

$ sudo apt-get install drbd-utils -y

1
2
3
4
5
6

$ sudo lvcreate vg00 -L 5G -n ceph-mon
Logical volume "ceph-mon" created

$ sudo mkfs.ext4 /dev/mapper/vg00-ceph--mon
...
...

DRBD resource configuration, create a new file in /etc/drbd.d/mon.res and append the following content:

resource mon {
  device    /dev/drbd0;
  disk      /dev/mapper/vg00-ceph--mon;
  meta-disk internal;
  on floating-mon-01 {
    address   10.20.1.41:7790;
  }
  on floating-mon-02 {
    address   10.20.1.42:7790;
  }
}

Check your configuration:

1
2
3
4
5

$ sudo drbdadm dump all
...
...
$ echo $?
0

Wipe off the content of the logical volume:

1
2

$ sudo dd if=/dev/zero of=/dev/mapper/vg00-ceph--mon bs=1M count=128
...

Bring up your resource:

1
2
3
4
5
6

$ sudo drbdadm -- --ignore-sanity-checks create-md mon
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success

Activate the resource:

1
2

$ sudo modprobe drbd
$ sudo drbdadm up mon

Put one node as the primary:

1

$ sudo drbdadm -- --overwrite-data-of-peer primary mon

Now DRBD is syncing blocks, so simply wait until the sync is complete:

1
2
3
4
5
6

$ sudo cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: 71955441799F513ACA6DA60
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:69504 nr:0 dw:0 dr:70168 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:5173180
    [>....................] sync'ed:  1.5% (5048/5116)Mfinish: 1:04:20 speed: 1,336 (1,284) K/sec

At the end you should have something like this:

version: 8.3.11 (api:88/proto:86-96)
srcversion: 71955441799F513ACA6DA60 
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:5242684 nr:0 dw:0 dr:5243348 al:0 bm:320 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Format your new device and perform a quick test:

1
2
3
4
5

$ sudo mkfs.ext4 /dev/drbd0
$ sudo mount /dev/drbd0 /mnt/
$ sudo drbd-overview
  0:mon  Connected Primary/Secondary UpToDate/UpToDate C r----- /mnt ext4 5.0G 204M 4.6G 5%
$ sudo umount /mnt

II.2. Build the third monitor

Yes, for this we are about to build the third monitor from scratch:

1
2

$ sudo mkdir -p /srv/ceph/mon0
$ sudo mount /dev/drbd0 /srv/ceph/mon0

Retrieve the current monitor key from the cluster:

1
2

$ sudo ceph auth get mon. -o /tmp/monkey
exported keyring for mon.

Grab the current monitor map:

1
2

$ sudo ceph mon getmap -o /tmp/lamap
got latest monmap

Examine it (if you’re curious):

1
2
3
4
5
6
7
8

$ sudo monmaptool --print /tmp/lamap
monmaptool: monmap file /tmp/lamap
epoch 6
fsid eb2efd30-64c7-4e8f-b2fd-81c2923e96cd
last_changed 2013-01-22 14:37:00.654689
created 2013-01-11 11:34:03.779220
0: 172.17.1.11:6789/0 mon.1
1: 172.17.1.12:6789/0 mon.2

Initialize the mon data directory:

1
2
3

$ sudo ceph-mon -i 0 --mkfs --monmap /tmp/lamap --keyring /tmp/monkey
...
...

Edit your ceph.conf on your floating-mon-01 and add the third monitor:

[mon.0]
    host = floating-mon-01
    mon addr = 172.17.1.100:6789

Eventually on floating-mon-01:

[mon.0]
    host = floating-mon-02
    mon addr = 172.17.1.100:6789

For the other configuration files, the host flag doesn’t really matter, it only matters for the node hosting the resource because this what the init script from ceph will read in the first place to manage the services. For the client, the only thing that matters is the IP address:

[mon]
    mon data = /srv/ceph/mon$id
    mon osd down out interval = 60
[mon.0]
    mon addr = 172.17.1.100:6789
[mon.1]
    host = mon-02
    mon addr = 172.17.1.11:6789
[mon.2]
    host = mon-03
    mon addr = 172.17.1.12:6789

II.3. Pacemaker setup

First install Pacemaker:

1

$ sudo apt-get install pacemaker -y

Now log into the crm shell by typing crm within your current shell. Edit your cluster properties:

1

$ sudo crm configure edit

Put the following:

    stonith-enabled="false" \
    no-quorum-policy="ignore" \
    pe-warn-series-max="1000" \
    pe-input-series-max="1000" \
    pe-error-series-max="1000" \
    cluster-recheck-interval="5min"

rsc_defaults $id=”rsc-options” \

    resource-stickiness="500"

II.3.1. Cluster resources

DRBD resource:

1
2
3
4
5
6
7
8
9
10
11
12

primitive drbd_mon ocf:linbit:drbd \
        params drbd_resource="mon" \
        op start interval="0" timeout="90s" \
        op stop interval="0" timeout="180s" \
        op promote interval="0" timeout="180s" \
        op demote interval="0" timeout="180s" \
        op monitor interval="29s" role="Master" \
        op monitor interval="31s" role="Slave"
ms ms_drbd_mon drbd_mon \
        meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Started"
colocation col_mon_on_drbd inf: g_ceph_mon ms_drbd_mon:Master
order ord_mon_after_drbd inf: ms_drbd_mon:promote g_ceph_mon:start

Filesystem resource:

1
2
3
4
5

primitive p_fs_mon ocf:heartbeat:Filesystem \
        params device="/dev/drbd/by-res/mon" directory="/srv/ceph/mon0" fstype="ext4" options="noatime,nodiratime" \
        op start interval="0" timeout="60s" \
        op stop interval="0" timeout="180s" \
        op monitor interval="60s" timeout="60s"

The monitor resource, note that the Ceph resource agent are a dependancie of the ceph package. You can easily find them in /usr/lib/ocf/resource.d/ceph:

1
2
3
4

primitive p_ceph_mon ocf:ceph:mon
    op start interval="0" timeout="60s" \
    op stop interval="0" timeout="180s" \
    op monitor interval="10s" timeout="30s"

Group all the resources together:

1

group g_ceph_mon p_fs_mon p_sym_mon_var p_sym_mon_etc p_vip_mon

II.3.2. Final design

At the end, you should see:

============
Last updated: Wed Jan 23 00:15:48 2013
Last change: Wed Jan 23 00:15:48 2013 via cibadmin on floating-mon-01
Stack: openais
Current DC: floating-mon-01 - partition with quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
2 Nodes configured, 2 expected votes
5 Resources configured.
============

Online: [ floating-mon-01 floating-mon-02 ]

 Master/Slave Set: ms_drbd_mon [drbd_mon]
     Masters: [ floating-mon-01 ]
     Slaves: [ floating-mon-02 ]
 Resource Group: g_ceph_mon
     p_fs_mon   (ocf::heartbeat:Filesystem):    Started floating-mon-01
     p_vip_mon  (ocf::heartbeat:IPaddr2):       Started floating-mon-01
     p_ceph_mon (ocf::ceph:mon):        Started floating-mon-01

II.4. CRUSH Map

The setup is the following:

• 2 Datacenter
• N OSDs, preferably an even number of OSDs on each location

Let say we want to end up with the following topolgy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ ceph osd tree
dumped osdmap tree epoch 621
# id    weight  type name   up/down reweight
-1  12  root default
-3  12      datacenter dc-1
-2  3           host ceph-01
0   1               osd.0   up  1
1   1               osd.1   up  1
-4  3           host ceph-02
2   1               osd.2   up  1
3   1               osd.3   up  1
-5            datacenter dc-2
-6  3           host ceph-03
4   1               osd.4   up  1
5   1               osd.5   up  1
-9  3           host ceph-04
5   1               osd.6   up  1
7   1               osd.7   up  1

Retrieve your CRUSH Map and fulfil it with all your hosts and locations:

1
2

$ ceph osd getcrushmap -o ma-crush-map
$ crushtool -d ma-crush-map -o ma-crush-map.txt

Your CRUSH Map:

# begin crush map

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 3 osd.4
device 3 osd.5
device 3 osd.6
device 3 osd.7

# types
type 0 osd
type 1 host
type 2 rack
type 3 row
type 4 room
type 5 datacenter
type 6 root

# buckets
host ceph-01 {
    id -2       # do not change unnecessarily
    # weight 2.000
    alg straw
    hash 0  # rjenkins1
    item osd.0 weight 1.000
    item osd.1 weight 1.000
}
host ceph-02 {
    id -4       # do not change unnecessarily
    # weight 2.000
    alg straw
    hash 0  # rjenkins1
    item osd.2 weight 1.000
    item osd.3 weight 1.000
}
host ceph-03 {
    id -6       # do not change unnecessarily
    # weight 2.000
    alg straw
    hash 0  # rjenkins1
    item osd.4 weight 1.000
    item osd.5 weight 1.000
}   
host ceph-04 {
    id -9       # do not change unnecessarily
    # weight 2.000
    alg straw
    hash 0  # rjenkins1
    item osd.6 weight 1.000
    item osd.7 weight 1.000
} 
datacenter dc-1 {
    id -3          # do not change unnecessarily
    # weight 2.000
    alg straw
    hash 0  # rjenkins1
    item ceph-01 weight 2.000
    item ceph-02 weight 2.000
}
datacenter dc-2 {
    id -5          # do not change unnecessarily
    # weight 2.000
    alg straw
    hash 0  # rjenkins1
    item ceph-03 weight 2.000
    item ceph-04 weight 2.000
}

# end crush map

II.4.1 Add a bucket

Add a bucket for the DC:

root default {
    id -1           # do not change unnecessarily
    # weight 4.000
    alg straw
    hash 0  # rjenkins1
    item dc-1 weight 2.000
    item dc-2 weight 2.000
}

II.4.2. Add a rule

Add a rule for the bucket nearly created:

# rules
rule dc {
    ruleset 0
    type replicated
    min_size 1
    max_size 10
    step take default
    step chooseleaf firstn 0 type datacenter
    step emit
}

Eventually recompile and inject the new CRUSH map:

1
2

$ crushtool -c ma-crush-map.txt -o ma-nouvelle-crush-map
$ ceph osd setcrushmap -i ma-nouvelle-crush-map

Since we set the rule dc to 0, every pool will by default use this one. Thus we don’t to specify a crush_ruleset each time we create a pool :-).

III. Break it!

In order to simulate a crash from one DC, the machines hosting the monitors on the active side (DC-1) have been shutdown, the resource migrated to DC-2. In the meantime a loop of ceph -s has been performed to check how long the interruption was.

health HEALTH_OK
monmap e7: 3 mons at {0=172.17.1.100:6789/0,1=172.17.1.11:6789/0,2=172.17.1.12:6789/0}, election epoch 16, quorum 0,1,2 0,1,2
osdmap e142: 4 osds: 8 up, 8 in
pgmap v20300: 1576 pgs: 1576 active+clean; 1300 MB data, 2776 MB used, 394 GB / 396 GB avail
mdsmap e61: 1/1/1 up {0=0=up:active}

real    0m0.011s
user    0m0.004s
sys 0m0.004s

health HEALTH_WARN 1 mons down, quorum 0,2 0,1
monmap e7: 3 mons at {0=172.17.1.100:6789/0,1=172.17.1.11:6789/0,2=172.17.1.12:6789/0}, election epoch 18, quorum 0,2 0,1
osdmap e142: 8 osds: 8 up, 8 in
pgmap v20300: 1576 pgs: 1576 active+clean; 1300 MB data, 2776 MB used, 394 GB / 396 GB avail
mdsmap e61: 1/1/1 up {0=0=up:active}

real    0m5.336s
user    0m0.004s
sys 0m0.004s

health HEALTH_WARN 1 mons down, quorum 0,2 0,1
monmap e7: 3 mons at {0=172.17.1.100:6789/0,1=172.17.1.11:6789/0,2=172.17.1.12:6789/0}, election epoch 18, quorum 0,2 0,1
osdmap e142: 8 osds: 8 up, 8 in
pgmap v20300: 1576 pgs: 1576 active+clean; 1300 MB data, 2776 MB used, 394 GB / 396 GB avail
mdsmap e61: 1/1/1 up {0=0=up:active}

real    0m0.011s
user    0m0.000s
sys 0m0.008s

health HEALTH_WARN 1 mons down, quorum 0,2 0,1
monmap e7: 3 mons at {0=172.17.1.100:6789/0,1=172.17.1.11:6789/0,2=172.17.1.12:6789/0}, election epoch 18, quorum 0,2 0,1
osdmap e142: 8 osds: 8 up, 8 in
pgmap v20300: 1576 pgs: 1576 active+clean; 1300 MB data, 2776 MB used, 394 GB / 396 GB avail
mdsmap e61: 1/1/1 up {0=0=up:active}

real    0m0.011s
user    0m0.004s
sys 0m0.004s

As you can see results are quite encouraging since they showed around 5 seconds of downtime. Abiously this wasn’t a real life scenario since no writes were running, even so I assume that these would have been delayed anyway. As a reminder please keep in mind that this setup was experimental, some of you might consider it. However it strongly recommend you to perform way more tests than I did. It was a first shot with a pretty encouraging outcome I believe. As always feel free to critic, comment and bring interesting discussions on the comments section. Of course such setup as a downside, notably on the directories that store monitor data, some files such the ones in $mon_root/logm/ and $mon_root/pgmap/ are actively changing, the failover might might lead to some weird issues… So once again this needs to be heavily tested ;-).

Availability zone in OpenStack. The main purpose of this article is to play a bit with availability zones.

The good thing with availability zones is that you can manage and isolate different entities in your infrastructure. For instance, if some customers need really fast VMs you can host them on your super expensive compute rack full of SSDs.Then what you can do is boot all the instances of your customer that way:

1

$ nova boot bla bla bla --availability-zone <zone>:<compute-node>

Everything is managed by this nova flag, and the only thing that you have to do is to define a name for your zone (obviously put something relevant):

# Availability zone
node_availability_zone=le_rack_du_seb

You could end up with a similar output:

1
2
3
4
5
6
7
8
9
10
11
12
13

$ sudo nova-manage service list
Binary           Host                                 Zone                    Status     State Updated_At
nova-cert        c2-controller-01                     le_rack_du_seb          enabled    :-)   2012-12-11 16:22:56
nova-scheduler   c2-controller-01                     le_rack_du_seb          enabled    :-)   2012-12-11 16:23:32
nova-consoleauth c2-controller-01                     le_rack_du_seb          enabled    :-)   2012-12-11 16:23:32
nova-network     c2-compute-01                        le_rack_du_seb-01       enabled    :-)   2012-12-11 16:23:32
nova-compute     c2-compute-01                        le_rack_du_seb-01       enabled    :-)   2012-12-11 16:23:32
nova-compute     c2-compute-02                        le_rack_du_seb-01       enabled    :-)   2012-12-11 16:23:32
nova-network     c2-compute-02                        le_rack_du_seb-01       enabled    :-)   2012-12-11 16:23:32
nova-compute     c2-compute-03                        le_rack_du_seb-02       enabled    :-)   2012-12-11 16:23:32
nova-network     c2-compute-03                        le_rack_du_seb-02       enabled    :-)   2012-12-11 16:23:32
nova-compute     c2-compute-04                        le_rack_du_seb-02       enabled    :-)   2012-12-11 16:23:32
nova-network     c2-compute-04                        le_rack_du_seb-02       enabled    :-)   2012-12-11 16:23:32

That’s all! As always I hope it helps ;-)

How to use a memory profiler to track memory usage of Ceph daemons!

To start tracking right away during daemon’ startup simply put the following variable in the /etc/init.d/ceph script and start your OSD daemon:

export CEPH_HEAP_PROFILER_INIT=1

The position doesn’t really matter ;-).

1
2
3
4
5

$ sudo service ceph start osd.0
=== osd.0 ===
Starting Ceph osd.0 on ceph-01...
Starting tracking the heap
starting osd.0 at :/0 osd_data /srv/ceph/osd0 /journal/journal

Start the profiler:

1
2

$ ceph osd tell 0 heap start_profiler
ok

Ceph log shows:

osd.0 [INF] osd.0 started profiler 

Let the profiler running and after some hours, dump the results into a file:

1
2

$ ceph osd tell 0 heap dump
ok

Ceph log shows:

osd.0 [INF] osd.0dumping heap profile now.
osd.0 [INF] ------------------------------------------------
osd.0 [INF] MALLOC:       10810792 (   10.3 MB) Bytes in use by application
osd.0 [INF] MALLOC: +       438272 (    0.4 MB) Bytes in page heap freelist
osd.0 [INF] MALLOC: +       172656 (    0.2 MB) Bytes in central cache freelist
osd.0 [INF] MALLOC: +       165632 (    0.2 MB) Bytes in transfer cache freelist
osd.0 [INF] MALLOC: +      2044136 (    1.9 MB) Bytes in thread cache freelists
osd.0 [INF] MALLOC: +       786432 (    0.8 MB) Bytes in malloc metadata
osd.0 [INF] MALLOC:   ------------
osd.0 [INF] MALLOC: =     14417920 (   13.8 MB) Actual memory used (physical + swap)
osd.0 [INF] MALLOC: +            0 (    0.0 MB) Bytes released to OS (aka unmapped)
osd.0 [INF] MALLOC:   ------------
osd.0 [INF] MALLOC: =     14417920 (   13.8 MB) Virtual address space used
osd.0 [INF] MALLOC:
osd.0 [INF] MALLOC:           2669              Spans in use
osd.0 [INF] MALLOC:             36              Thread heaps in use
osd.0 [INF] MALLOC:           4096              Tcmalloc page size
osd.0 [INF] ------------------------------------------------
osd.0 [INF] Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
osd.0 [INF] Bytes released to the OS take 

Stop the profiler:

1
2

$ ceph osd tell 0 heap stop_profiler
ok

Log shows:

osd.0 [INF] osd.0 stopped profiler

Read the .heap file with Google heap tool:

1
2

$ sudo apt-get install google-perftools -y
$ sudo google-pprof /usr/bin/ceph-osd -gv osd-0001.heap

Hint, if you don’t have any virtual interface you can look for a text content:

1

$ sudo google-pprof /usr/bin/ceph-osd --text osd-0001.heap

I had to use a memory profiler because I recently noticed some memory leaks from Ceph OSDs. This has been discussed on the Ceph Mailing List.

From time to time I notice that switching from my audio local source to my Airplay station (and the other way around) somehow muted the sound on Chrome. After some googling I noticed that the problem came from a flash player plugin. You just need to disable it.

For this go to chrome://plugins, Show details an then disable the PPAPI plugin located in:

/Applications/Google Chrome.app/Contents/Versions/24.0.1312.52/Google Chrome Framework.framework/Internet Plug-Ins/PepperFlash/PepperFlashPlayer.plugin

Hope it helps! `

A lot of new features came with the version 0.55 of Ceph, one of them is that CephX authentication is enable by default. If you run v0.48 Argonaut without CephX and want to update to the latest Bobtail, you might run through some problems if you don’t edit your configuration file.

In previous versions stable branch, you could simply use the following setting:

 auth supported = [cephx | none]

This option is now deprecated. Bobtail now integrates a new finer-grained authentication, it supports 3 new authentication methods:

• cluster:
• service: internal daemons communication, for instance OSD to OSD connections
• client: client side, machine that tries to connect to the cluster

By default daemons require CephX authentication, which means that OSD, MON and MDS will now use CephX to connect to each others. ON the other side, clients will continue to connect with disabled authentication.

[global]
...
auth cluster required = none    
auth service required = none
...

To disable client authentication as well:

[global]
...
auth client required = none
...

W Important: If your cluster does not currently have an auth supported line that enables authentication, you must explicitly turn it off in Bobtail using the settings below.::

Et voilà !