General:Ceph

Extracting Data off CephFS

List inodes in directory, starting with the "root" inode

rados -p metadata listomapkeys 10000000000.00000000 | less

Get the omap value for the next path

rados -p metadata getomapval 10000000000.00000000 some_directory_head

value (412 bytes) :
00000000  02 00 00 00 00 00 00 00  49 0b 06 71 01 00 00 85  |........I..q....|
00000010  14 00 00 00 01 00 00 00  00 00 00 7e 80 48 57 fc  |...........~.HW.|
00000020  86 82 1e ed 41 00 00 08  09 00 00 09 09 00 00 01  |....A...........|
00000030  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000060  01 00 00 00 ff ff ff ff  ff ff ff ff 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  7e 80 48 57 fc 86 82 1e  |........~.HW....|
00000080  f4 41 c8 55 af 61 a0 01  0e 00 00 00 00 00 00 00  |.A.U.a..........|
00000090  02 02 20 00 00 00 00 00  00 00 00 00 00 00 7e 80  |.. ...........~.|
000000a0  48 57 fc 86 82 1e 07 00  00 00 00 00 00 00 15 00  |HW..............|
000000b0  00 00 00 00 00 00 03 02  38 00 00 00 a4 c3 14 00  |........8.......|
000000c0  00 00 00 00 22 e6 88 da  9d 00 00 00 74 19 00 00  |....".......t...|
000000d0  00 00 00 00 89 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 3d 10 4d 57  |............=.MW|
000000f0  87 e8 e8 29 03 02 38 00  00 00 a4 c3 14 00 00 00  |...)..8.........|
00000100  00 00 22 e6 88 da 9d 00  00 00 74 19 00 00 00 00  |..".......t.....|
00000110  00 00 89 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 00 3d 10 4d 57 87 e8  |..........=.MW..|
00000130  e8 29 1d 0b a2 1e 00 00  00 00 00 00 00 00 00 00  |.)..............|
00000140  00 00 01 00 00 00 00 00  00 00 cf 4d 00 00 00 00  |...........M....|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ff ff  |................|
00000160  ff ff ff ff ff ff 00 00  00 00 01 01 10 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000190  fe ff ff ff ff ff ff ff  00 00 00 00              |............|
0000019c

The inode for the next lookup is in red. In the correct order it is 10000001485.00000000. Perform more lookups until you have the inode of the file you want.

[root@hobbit01 ~]# rados -p metadata getomapval 1000b96f842.00000000 some_file_head
value (412 bytes) :
00000000  02 00 00 00 00 00 00 00  49 0b 06 71 01 00 00 12  |........I..q....|
00000010  56 16 0c 00 01 00 00 00  00 00 00 60 f2 0f 57 1f  |V..........`..W.|
00000020  ee 48 2a a4 81 00 00 08  09 00 00 09 09 00 00 01  |.H*.............|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 40 00  |..............@.|
00000040  01 00 00 00 00 00 40 00  00 00 00 00 00 00 00 00  |......@.........|
00000050  ff ff ff ff 22 00 00 00  6b 58 c1 4c 00 00 00 00  |...."...kX.L....|
00000060  01 00 00 00 ff ff ff ff  ff ff ff ff 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  95 e6 13 57 f8 21 27 19  |...........W.!'.|
00000080  60 f2 0f 57 1f ee 48 2a  00 00 00 00 00 00 00 00  |`..W..H*........|
00000090  02 02 20 00 00 00 00 00  00 00 00 00 00 00 00 00  |.. .............|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  00 00 00 00 00 00 03 02  38 00 00 00 00 00 00 00  |........8.......|
000000c0  00 00 00 00 6b 58 c1 4c  00 00 00 00 01 00 00 00  |....kX.L........|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000f0  00 00 00 00 03 02 38 00  00 00 00 00 00 00 00 00  |......8.........|
00000100  00 00 6b 58 c1 4c 00 00  00 00 01 00 00 00 00 00  |..kX.L..........|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000130  00 00 05 f2 01 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  00 00 01 00 00 00 00 00  00 00 a5 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ff ff  |................|
00000160  ff ff ff ff ff ff 00 00  00 00 01 01 10 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000190  fe ff ff ff ff ff ff ff  00 00 00 00              |............|
0000019c

In blue, is the number of bytes for each segment (4MB). In red is the pool ID. In Green is the number of bytes (0x4cc1586b). Divide 0x4cc1586b by 4MB. Then we have to extract that many segments. The above inode is 0x133 segments long. We then need to get all those segments.

for x in {0..307}; do
  segment=$(printf %8x'\n' ${x} | sed -e 's/ /0/g');
  rados -p cachepool get 1000c165612.$segment some_file.$segment
done

Once the segments have been extracted, you must put them together.

cat some_file.* > some_file

Then you can delete the segments

rm -f some_file.*

Congratulations and godspeed.