DMclone

Posted onby admin

Introduction¶

DMclone for Windows is a reliable disk clone software. It can help Windows users securely and easily clone HDD, SSD, USB flash drive, memory card, RAID, Server from one location to another. It offers sector-by-sector disk cloning solution to copy hard drive without any data loss. Efficient Disk Cloning Software DMclone for Windows is one of the best disk cloning software. It can help you securely and easily clone HDD, SSD, memory card, USB flash drive, etc from one location to another.

dm-clone is a device mapper target which produces a one-to-one copy of anexisting, read-only source device into a writable destination device: Itpresents a virtual block device which makes all data appear immediately, andredirects reads and writes accordingly.

The main use case of dm-clone is to clone a potentially remote, high-latency,read-only, archival-type block device into a writable, fast, primary-type devicefor fast, low-latency I/O. The cloned device is visible/mountable immediatelyand the copy of the source device to the destination device happens in thebackground, in parallel with user I/O.

For example, one could restore an application backup from a read-only copy,accessible through a network storage protocol (NBD, Fibre Channel, iSCSI, AoE,etc.), into a local SSD or NVMe device, and start using the device immediately,without waiting for the restore to complete.

When the cloning completes, the dm-clone table can be removed altogether and bereplaced, e.g., by a linear table, mapping directly to the destination device.

The dm-clone target reuses the metadata library used by the thin-provisioningtarget.

Glossary¶

Hydration

The process of filling a region of the destination device with data fromthe same region of the source device, i.e., copying the region from thesource to the destination device.

Once a region gets hydrated we redirect all I/O regarding it to the destinationdevice.

Design¶

Sub-devices¶

The target is constructed by passing three devices to it (along with otherparameters detailed later):

  1. A source device - the read-only device that gets cloned and source of thehydration.

  2. A destination device - the destination of the hydration, which will become aclone of the source device.

  3. A small metadata device - it records which regions are already valid in thedestination device, i.e., which regions have already been hydrated, or havebeen written to directly, via user I/O.

The size of the destination device must be at least equal to the size of thesource device.

Regions¶

dm-clone divides the source and destination devices in fixed sized regions.Regions are the unit of hydration, i.e., the minimum amount of data copied fromthe source to the destination device.

The region size is configurable when you first create the dm-clone device. Therecommended region size is the same as the file system block size, which usuallyis 4KB. The region size must be between 8 sectors (4KB) and 2097152 sectors(1GB) and a power of two.

Dmclone Review

Reads and writes from/to hydrated regions are serviced from the destinationdevice.

A read to a not yet hydrated region is serviced directly from the source device.

A write to a not yet hydrated region will be delayed until the correspondingregion has been hydrated and the hydration of the region starts immediately.

Note that a write request with size equal to region size will skip copying ofthe corresponding region from the source device and overwrite the region of thedestination device directly.

Discards¶

dm-clone interprets a discard request to a range that hasn’t been hydrated yetas a hint to skip hydration of the regions covered by the request, i.e., itskips copying the region’s data from the source to the destination device, andonly updates its metadata.

Dmclone review

If the destination device supports discards, then by default dm-clone will passdown discard requests to it.

Background Hydration¶

dm-clone copies continuously from the source to the destination device, untilall of the device has been copied.

Copying data from the source to the destination device uses bandwidth. The usercan set a throttle to prevent more than a certain amount of copying occurring atany one time. Moreover, dm-clone takes into account user I/O traffic going tothe devices and pauses the background hydration when there is I/O in-flight.

A message hydration_threshold <#regions> can be used to set the maximum numberof regions being copied, the default being 1 region.

dm-clone employs dm-kcopyd for copying portions of the source device to thedestination device. By default, we issue copy requests of size equal to theregion size. A message hydration_batch_size <#regions> can be used to tune thesize of these copy requests. Increasing the hydration batch size results indm-clone trying to batch together contiguous regions, so we copy the data inbatches of this many regions.

When the hydration of the destination device finishes, a dm event will be sentto user space.

Updating on-disk metadata¶

On-disk metadata is committed every time a FLUSH or FUA bio is written. If nosuch requests are made then commits will occur every second. This means thedm-clone device behaves like a physical disk that has a volatile write cache. Ifpower is lost you may lose some recent writes. The metadata should always beconsistent in spite of any crash.

Target Interface¶

Constructor¶

metadata dev

Fast device holding the persistent metadata

destination dev

The destination device, where the source will be cloned

source dev

Read only device containing the data that gets cloned

region size

The size of a region in sectors

#feature args

Number of feature arguments passed

feature args

no_hydration or no_discard_passdown

#core args

An even number of arguments corresponding to key/value pairspassed to dm-clone

core args

Key/value pairs passed to dm-clone, e.g. hydration_threshold256

Optional feature arguments are:

no_hydration

Create a dm-clone instance with background hydrationdisabled

no_discard_passdown

Disable passing down discards to the destination device

Optional core arguments are:

hydration_threshold <#regions>

Maximum number of regions being copied fromthe source to the destination device at anyone time, during background hydration.

hydration_batch_size <#regions>

During background hydration, try to batchtogether contiguous regions, so we copy datafrom the source to the destination device inbatches of this many regions.

Status¶

metadata block size

Fixed block size for each metadata block in sectors

#used metadata blocks

Number of metadata blocks used

#total metadata blocks

Total number of metadata blocks

region size

Configurable region size for the device in sectors

#hydrated regions

Number of regions that have finished hydrating

#total regions

Total number of regions to hydrate

#hydrating regions

Number of regions currently hydrating

#feature args

Number of feature arguments to follow

feature args

Feature arguments, e.g. no_hydration

#core args

Even number of core arguments to follow

core args

Key/value pairs for tuning the core, e.g.hydration_threshold 256

clone metadata mode

ro if read-only, rw if read-write

In serious cases where even a read-only mode is deemedunsafe no further I/O will be permitted and the statuswill just contain the string ‘Fail’. If the metadatamode changes, a dm event will be sent to user space.

Messages¶

disable_hydration

Disable the background hydration of the destination device.

enable_hydration

Enable the background hydration of the destination device.

hydration_threshold <#regions>

Set background hydration threshold.

Dmclone

hydration_batch_size <#regions>

Set background hydration batch size.

Examples¶

Clone a device containing a file system¶

  1. Create the dm-clone device.

  2. Mount the device and trim the file system. dm-clone interprets the discardssent by the file system and it will not hydrate the unused space.

  3. Enable background hydration of the destination device.

  4. When the hydration finishes, we can replace the dm-clone table with a lineartable.

    The metadata device is no longer needed and can be safely discarded or reusedfor other purposes.

Known issues¶

Hdclone Download

  1. We redirect reads, to not-yet-hydrated regions, to the source device. Ifreading the source device has high latency and the user repeatedly reads fromthe same regions, this behaviour could degrade performance. We should usethese reads as hints to hydrate the relevant regions sooner. Currently, werely on the page cache to cache these regions, so we hopefully don’t end upreading them multiple times from the source device.

  2. Release in-core resources, i.e., the bitmaps tracking which regions arehydrated, after the hydration has finished.

  3. During background hydration, if we fail to read the source or write to thedestination device, we print an error message, but the hydration processcontinues indefinitely, until it succeeds. We should stop the backgroundhydration after a number of failures and emit a dm event for user space tonotice.

Why not…?¶

Dmclone 1.2

We explored the following alternatives before implementing dm-clone:

Dmclone 1.0 (win Mac)

  1. Use dm-cache with cache size equal to the source device and implement a newcloning policy:

    • The resulting cache device is not a one-to-one mirror of the source deviceand thus we cannot remove the cache device once cloning completes.

    • dm-cache writes to the source device, which violates our requirement thatthe source device must be treated as read-only.

    • Caching is semantically different from cloning.

  2. Use dm-snapshot with a COW device equal to the source device:

    • dm-snapshot stores its metadata in the COW device, so the resulting deviceis not a one-to-one mirror of the source device.

    • No background copying mechanism.

    • dm-snapshot needs to commit its metadata whenever a pending exceptioncompletes, to ensure snapshot consistency. In the case of cloning, we don’tneed to be so strict and can rely on committing metadata every time a FLUSHor FUA bio is written, or periodically, like dm-thin and dm-cache do. Thisimproves the performance significantly.

  3. Use dm-mirror: The mirror target has a background copying/mirroringmechanism, but it writes to all mirrors, thus violating our requirement thatthe source device must be treated as read-only.

  4. Use dm-thin’s external snapshot functionality. This approach is the mostpromising among all alternatives, as the thinly-provisioned volume is aone-to-one mirror of the source device and handles reads and writes toun-provisioned/not-yet-cloned areas the same way as dm-clone does.

    Still:

    • There is no background copying mechanism, though one could be implemented.

    • Most importantly, we want to support arbitrary block devices as thedestination of the cloning process and not restrict ourselves tothinly-provisioned volumes. Thin-provisioning has an inherent metadataoverhead, for maintaining the thin volume mappings, which significantlydegrades performance.

    Moreover, cloning a device shouldn’t force the use of thin-provisioning. Onthe other hand, if we wish to use thin provisioning, we can just use a thinLV as dm-clone’s destination device.