15 Nov 14 Dump and restore block device data on the fly by the network

Sometimes you may need to copy data from a block device (or LVM logical volume or snapshot) from one server to another., but you don’t want to dump the image to disk, move to the other server, then import. You may need (or just want) to copy on the fly, transfering data on the net.

To do this, and have ETA on the operation you need the pv executable. The command nc is used to stream data on the network, while pigz is used to compress data (gzip uses just one CPU, while pigz uses all available CPU, and it’s much faster).

On the origin server (server1) you have a block device (lvm logical volume in this case) called /dev/vg0/vm-111-disk-1, while on the destination server (server2) you want to overwrite a LVM logical volume called /dev/vg0/vm-112-disk-1 with data coming from the origin server.
To do this, assuming the device is big 20GB, you may run those commands:

Server side (destination server, server2, ip 192.168.0.2):

nc -l -n -p 2102 -q 2 | pigz -d | pv -pre –size=20G | dd iflag=fullblock bs=512k of=/dev/vg0/vm-112-disk-1

Client side (origin server, server1, 192.168.0.1):

dd if=/dev/vg0/vm-111-disk-1 bs=512k | pv -pre –size=20G | pigz | nc -q 2 192.168.0.2 2102

Data will be read, compressed, transfered on the network on (port TCP 2102 on our case, from 192.168.0.1 to 192.168.0.2), uncompressed on the destination server and restored on disk, and you’ll have ETA and progress indication:

Output server side (destination server, server2):

root@server2 ~ # nc -l -n -p 2102 -q 2 | pigz -d | pv -pre –size=20G | dd iflag=fullblock bs=512k of=/dev/vg0/vm-112-disk-1
[71.2MB/s] [=========================================================================================================================================>] 100%
40960+0 records in
40960+0 records out
21474836480 bytes (21 GB) copied, 296.436 s, 72.4 MB/s

Output client side (origin server, server1):

root@server1 ~ # dd if=/dev/vg0/vm-111-disk-1 bs=512k | pv -pre –size=20G | pigz | nc -q 2  192.168.0.2 2102
[72.2MB/s]
[=========================================================================================================================================>]
100%
40960+0 records
40960+0 records out
21474836480 bytes (21 GB) copied, 283.531 s, 75.7 MB/s