“dt” Tools Synchronize Capabilities

Vertriebspartner finden

The “dt sync” tool allows synchronizing a source directory tree to a destination tree so that the destination is an exact copy of the source thereafter. It works similarly to a “dt cp” call but here only changed files will be synchronized which is significantly faster for subsequent syncs. Unless otherwise specified, files and folders (objects) not existing in the source are removed from the destination. The sync can also copy entire HELIOS volumes. It automatically updates the desktop database which allows active source and destination volumes while “dt sync” is running.

Options

-f (force overwrite existing file)
If an existing destination file cannot be opened, remove it and try again.
This option is normally not required if “dt sync” is running as root (as recommended).

-i interactive: ask before overwrite.

-k (try to preserve desktop ID)
Try to preserve all desktop file IDs. However this works only if the destination has no files stored so far. For subsequent syncs the -k option makes no sense.

-n (ignore resource and streams)
Do not sync SRC file resource forks and streams to the destination.

-o Print warning if files are write open during sync process.

-v (verbose)
Show warnings and imply the -V version to show statistics (see below)

-V (show statistics)

Show a summary statistics line at the end, e.g.:
updated (files/size): 14122/17.3G - skip: 0/0B - removed: 0/0B - directories: 733 - elapsed: 3 min 31 s 790 ms

-z (force zero desktop ID for destination)
This option will set all destination IDs to zero. However the main benefit of “dt sync” is that it can sync between two active volumes, therefore this option is not needed.

-c (disable charset conversion)
“dt” supports the conversion of character sets between differently encoded HELIOS volumes, e.g. UTF-8 and old MacRoman. However, these days all supported volumes are UTF-8 encoded only. Therefore this option is not needed.

-y (force scanning for streams)
This option will collect file streams even if the resource file is not marked for existing streams. This option is not needed unless file streams have been generated by non-HELIOS software modules.

-X (suppress close delay for desktop)
By default, the “desksrv” process keeps the volume desktop database open for another 10 seconds after “dt” has quit, to start up quicker for subsequent “dt” runs. However, for “dt sync” this option is not of any benefit.

-E (send notify events)
If specified, the Spotlight index and other notification based services will receive update events.

-K (do not preserve symbolic links)

This option copies the link target instead of creating a symbolic link when copying files containing symbolic links.

-D (do not clean up)
Do not remove files in DEST that no longer exist in SRC. This option is switched off by default because it is best to remove files from the destination if they do not exist in the source. Only this way can the synchronization destination size be calculated because then the destination requires the same amount of space as the source.

-S (scan only)
Only scan what needs to be done and print the summary. Together with the -L option, it prints the individual files being scheduled for a sync.

-L (list one line per processed file to identify what would be changed, e.g.

cp ovr : /Volumes/temp/.DS_Store

cp new : /Volumes/temp/Altona/altona_visual_1v2a_x3.pdf

cp new : /Volumes/temp/Altona/altona_visual_1v2a_x3.pdf.annotation

rmdir : /Volumes/temp/Altona_Old/

remove : /Volumes/temp/Altona/altona_visual_1v2a_x3.pdf

chown : /Volumes/temp/Altona/altona_visual_1v2a_x4.pdf

chmod : /Volumes/temp/Altona/altona_visual_1v2a_x5.pdf

-J Specifies the job name.

Examples

The following are common cases how “dt sync” may be used. “dt sync” should be used with root permissions to sync all objects, different owners, and permissions.

1) Synchronize a source folder with a destination folder

# dt sync /SRC /DEST

A proper crontab entry to sync every night at 3am would be:

0 3 * * * /usr/local/helios/bin/dt sync /SRC /DEST

An enhanced cron entry which also sends notifications by e-mail would be:

0 3 * * * /usr/local/helios/bin/dt sync /SRC /DEST 2>&1 | mail -s "Daily Sync" admin@yourdomain.com

By default, “dt sync” prints nothing but critical errors. The exit code 0 means that the job was successfully done. The -V option shows a summary statistics line after the sync is completed. The -v option shows warnings and statistics. The -E option sends file notification events to keep the Spotlight search index up to date.

2) Synchronize a single folder within a path

# dt sync /SRC/FolderA /DEST/FolderA

3) Scan only – print a report of what would be synchronized

# dt sync -S /SRC /DEST

or alternatively:
# dt sync -S -L /SRC /DEST

Notes

When syncing folders containing symbolic links, these symbolic links will always be re-created in the destination because the modification dates of symbolic links cannot be set, and therefore “dt sync” just re-creates all symbolic links.

“dt sync” supports UTF-8 volumes and file name encodings, other encodings have not been tested.

FAQ

Q: How does it compare to “dt cp”?
A: It works similarly to a “dt cp” call but here only changed files will be synced which is significantly faster for subsequent syncs.

Q: How does it compare to “rsync”?
A: “rsync” can sync files without requiring that the target file system be mounted on the HELIOS server. However when “rsync” syncs HELIOS volumes, the destination volume cannot be active because “rsync” copies changed files but does not update the HELIOS volume desktop and Spotlight databases. Hence “rsync” requires a rebuild. “dt sync” has the advantage that it updates the HELIOS desktop database and Spotlight index files. This allows active source and destination volumes while “dt sync” is running.

Q: Is “dt sync” a backup solution?
A: In the general sense, no. But in some respects, yes, as in syncing to a cold or hot spare server or storage system. “dt sync” copies data to a second location which can be published as a volume. The benefit is that in case of server failure or data loss, the customer can switch to the sync copy to continue working. This would supplement a standard backup or archive solution which provides long term tape storage and is required to archive data going back many years.

Q: Is this a “live” continuous sync process? Or does it sync at specified times?
A: “dt sync” is a command line program that launches when specified; it is not a continuous ongoing process. It can easily be automated as a cron job, example cron jobs are included in the manual.

Q: Which HELIOS products include “dt sync”?
A: The “dt” tools belong to HELIOS Base which is included in every HELIOS file server product: EtherShare, PCShare and WebShare.

Q: Must the source or destination files be within HELIOS volumes?
A: No.

Q: How must file systems be accessible to “dt sync”?
A: “dt sync” is a local synchronization tool, but remote file systems can be locally mounted via NFS to make them available for syncing.

Q: How are the file differences between servers determined?
A: “dt sync” scans the source and destination files and folders and compares modification and size information to check for differences. We don't use the desktop database in case files were added or modified (e.g via FTP) without updating the desktop database. This also enables support of non-HELIOS volumes. The new Spotlight database does index file dates (and hence files changed since the last sync), but this is optional and Spotlight may not be turned on.

Q: When changed files are written to the target server, are the complete files transferred, or just the delta of each file?
A: At present we copy the entire file because source and destination are usually local. Check-summing the source and destination would be as/more expensive as just copying the data.

Q: Is any compression used for file transfers?
A: No, no compression is used/needed because files on disk are always uncompressed.

Q: What optimization is used for better performance?
A: “dt sync” is much more than just a find with execution of a copy for each file. First it doesn't spawn a process per file, a single “dt” command does the entire task without additional helper processes. It also updates the destination Desktop and Spotlight databases. By default (can be turned off) it also removes obsolete files/folders on the destination in case they do not exist in the source volume/folder.

Q: If a site fails over to a synced backup server, and then later the source server is brought back up, would it be straightforward to use “dt sync” to sync in the opposite direction, to bring the primary server back up to date?
A: Yes, just do a “dt sync” from the old backup server disk to the main disk.

Q: What are the options regarding files on the destination that are no longer on the source?
A: Unless otherwise specified, files and folders (objects) not existing in the source are removed from the destination.

Q: Are file and record locking of files used to detect busy files?
A: No, “dt sync” works like all other host disk copying tools, without checking locks. However a snapshot of the source volumes, such as supported by AIX, Solaris, and the new HELIOS Virtual Server Appliance, will ensure that data gets synced consistently as of the moment the snapshot was created.

Q: Is it possible to get a file list of the files being updated or removed without doing the sync?
A: Yes, “dt sync” scan -L option allows getting the file list only of affected files.

Q: My server has plenty of CPU and IO performance. Can I run multiple instances of “dt sync” simultaneously, to reduce the total copy time?
A: Yes. You can run multiple copies of “dt sync”, e.g. for different volumes or for different folders, to work in parallel.

Q: Can I run parallel syncs to two different destination storage raids or servers?
A: Yes, “dt sync” can be called multiple times with different destinations.

Q: What about the “dt sync” performance?
A: “dt sync” is performance optimized and runs directly on the server, offering the highest performance possible on the connected disk system. Other client/server solutions will be significantly slower because they suffer from the latency of the communication between the different processes/systems.