|
HELIOS Base addendum |
1.1 Command descriptions1.3 Using "dt" for backup/restore
This chapter describes the "dt tools", which mimic the functionality of most major UNIX commands for handling files, while maintaining the integrity of the desktop database. "dt tools" also provide access to Mac specific file information and streams from a UNIX prompt.
Important: The "dt" program does not consider any file locking and can therefore also copy files in use. In order to avoid damage, consider this and make sure nobody is working concurrently on these particular files before manipulating them with "dt".
If you need to access files which are stored in a HELIOS volume via host scripts or commands, there are a lot of reasons for using the "dt tools".
Note: Host commands means any access under UNIX, e.g. any script or program (like a backup program), which accesses files stored in a HELIOS volume.
Any manipulation of a file or folder inside a HELIOS volume using ordinary UNIX commands like "cp", "mv", "rm" or other programs, will cause inconsistencies between volume information and the related desktop database, and you may lose addtional data, e.g. file streams. Especially restoring files from a backup will cause such an inconsistency.
In the following sections, you will find more information on this problem, and on how the "dt tools" help avoid it.
Files and directories in the Windows/NTFS environment may have a certain number of file streams. File streams contain meta data such as document title, subject, author, etc., similar to the resource fork of a Mac file.
The "dt tools" can handle Windows NTFS file streams. While accessing a file on the network that has been created in the Windows/NTFS environment, e.g. renaming or moving it on a HELIOS server, all pertinent file streams are considered as well.
Also, file attributes such as Archive, Hidden, System, Creation date, label settings, etc. are also preserved when using the "dt tools". In addition, a fast file search can be performed via the desktop database, with file ID support.
Use the utilities whenever you would (usually) use any of the following UNIX commands (if either source and/or destination files/folders are located in a HELIOS volume):
You may use the "dt tools" even if the source or destination is not located in a HELIOS volume, or if you are not sure about this. The "dt tools" automatically select the proper mode of operation, e.g. if source and destination are no volumes "dt" acts exactly like the corresponding UNIX program.
It is required to use the "dt tools" together with EtherShare, PCShare and/or ImageServer. These HELIOS products have improved verification mechanisms to recognize potentially harmful configurations, and will disable access to volumes for which consistency between volume and desktop information cannot be assured. In addition, warning or error messages are logged to the system error log file, which may help you locate the cause of the problem.
Just to help you understand the possible error messages and the specific behavior of the "dt tools", here are the main differences to standard UNIX commands:
- Each Mac file consists of two parts, the so-called "data fork" which is stored as a standard UNIX file, and a corresponding "resource fork" which is stored in an ".rsrc" subdirectory. Additional information for each file is stored in a corresponding entry in a volume based "desktop" database. Therefore, when using UNIX commands, each operation like move, copy, or delete, must consider both the "data fork" and "resource fork" as well as the database entry instead of just one single file.
- Windows files and directories may have additional streams as well.
- When looking at a volume as a UNIX directory you may not notice the differences, because the resource fork subdirectory and the database files are stored as hidden UNIX files.
- The "dt tools" simulate the behavior of standard UNIX commands as close as possible, but there are slight differences in the number and functionality of the available options, and due to the extended functionality you may notice a different behavior and different error messages. For example, you may see error messages about the resource fork of a file, which you would not see when using standard UNIX commands (the UNIX commands - in fact - ignore this part of the file).
- You may see different errors on "dt chmod", "dt chown", and "dt chgrp" because in volumes, you need permissions to change not only the specified file but also the associated resource directory, the resource file and all streams.
- For more information about resource and stream handling errors, you may use the -v (verbose) option.
Before you try and use any of the "dt" commands, you should be well familiar with the corresponding UNIX commands. If you need more information about how the "dt tools" work and how problems arise without the utilities, please read the "Technical notes" following each command description, and 1.2 "Additional information".
We do not recommend to sym-link regular "cp", "mv", "rm" commands to the "dt tools". Although this is possible, you should verify your current and future system environment very carefully, to assure proper operation of your programs, scripts and established workflow.
Especially, you should check whether your site uses special HSM (Hierarchical Storage Management), tape robot, or RAID (Redundant Array of Independent Disks) software which may also use their own versions of these UNIX commands. Take into account that different users and script based programs may make use of different shells and environment settings.
We suggest to check for each HELIOS volume the consistency between the volume information and the volume's desktop database. HELIOS Base offers a rebuild option, namely -s (scan only), which can be used for this purpose (see chapter 7.10 "rebuild" in the Base manual).
Simply issue rebuild -s <volume mount point> for every defined public or private HELIOS volume. Any output of this "scan" indicates a potentially harmful inconsistency between volume and desktop information. You should only continue after having re-synchronized the desktop database by means of an ordinary rebuild.
Take some time to make yourself familiar with the utilities and the way they behave inside a HELIOS volume, between different HELIOS volumes, and between non-HELIOS and HELIOS volumes.
The "dt tools" use standard error messages starting with the program name (including the command argument), followed by the file or directory currently processed, and the error message, e.g.:
The error messages used are similar to the errors issued by the standard UNIX commands. Please note that you may encounter additional error messages regarding resource fork, streams, or desktop database.
In some cases the "dt tools" may issue error messages displaying the full path of a file name instead of the passed relative one. This is due to the fact that the desktop database always needs absolute file names instead of relative ones. In this case, you may see the completely resolved (including symbolic links) absolute path name in the error message.
For the following command descriptions, the knowledge of the functionality of the corresponding UNIX commands is assumed. Please refer to your UNIX manual if you are not familiar with the commands.
If you do not need or want to know, how the "dt tools" operate, you only need to read the main command description for each UNIX command which is emulated by the "dt tools". The technical description following each command is meant for advanced users or system administrators, and explains some details or the special behavior of the respective command.
With very few exceptions, "dt" behaves identically on all platforms supported by HELIOS. Please note that, for this reason, some platform-specific options could not be implemented.
For all commands, the standard UNIX permissions apply when accessing, removing, or overwriting a file.
These files are handled implicitly by "dt" and there is no need to specify them directly. You should never move, copy, or remove these files with other commands, as e.g. the standard UNIX tools. Do not manipulate them at all.
Note: In the following, the term "volume" is used for a directory that is defined as a HELIOS volume which contains a desktop database (in contrast to a simple UNIX "directory"). Likewise, the term "volume root" referes to the top level folder of a HELIOS volume. The "HELIOS Applications" volume, for example, is by default defined to reside in "/usr/local/helios/public", which is its "volume root" directory under UNIX.
"dt" will recognize access to HELIOS volumes from another server only by means of available ".DeskServer" files.
"dt" will process data, resource and file stream parts of files/folders in HELIOS volumes. If no resource information exists, "dt" will be able to create at least minimum information. Especially copy or move operations may require more free space on destination file systems.
Usage: dt { rm | rmdir | mv | cp | set | ls | mkdir | touch | upd | chmod | chown | chgrp | idinfo | iddump }
Calling "dt" with the command argument but no further arguments, prints the usage for the specific command.
(-X option) Usually, the desktop of a volume is kept open for about 30 sec. when being used by "dt" in order to avoid overhead (desktop database completely open/closed). This is because further access is to be expected. However, for certain applications it is not advisable to block the desktop for a volume that long. Hence, with the -X option set, the desktop is closed immediately.
(-y option) Force scanning for streams that are not specified in the resource information of a file. If additional streams exist for a file but the file information is not marked for streams, this option checks the entire ".rsrc" directory if streams are detected for the current file. To avoid unnecessary search operations, the standard resource is marked if additional streams are available.
(-E option) If this option is specified, file events for all registered services are notified to the notification server.
-i Prompt for confirmation for each file. If the
answer begins with y (yes), the file is deleted,
otherwise the file remains on the server.
Note: Missing resource forks are not reported unless the -v verbose option is set.
The volume root directory is automatically touched to announce the changes to any Mac that has mounted the volume.
When removing a directory using the -r option, "dt" automatically tries to remove any orphan resource forks or file streams. The volume root directory may only be removed if this volume is not in use by any other client, either EtherShare, PCShare, WebShare or the "dt tools", otherwise a "directory not empty" error is issued.The "dt rmdir" command will remove the directory entry specified by each dirname operand, provided that this operand refers to an empty directory.
-p Allow users to remove the directory dirname and
its parent directories which become empty. A
message is printed on the standard error about
whether the whole path is removed or part of the
path remains for some reason.
- Rename a file "f1" to the destination file "f2"
- Move the passed files "f1- fn" into the destination directory "d1"
- Move the passed directory into the destination directory "d2". If "d2" does not exist, rename "d1" to "d2".
-f Move the file(s) without prompting even if it is
overwriting an existing target. Note that this is
the default if the standard input is not a terminal.
-i Prompt for confirmation for each file. If the
answer begins with y (yes), the file is moved,
otherwise it remains where it is.
-k Keep ID, try to allocate the source ID and use it
for the destination as well (see
1.3 "Using "dt" for backup/restore"). In case
this is impossible, you will only get a warning
if -v is set. You can use this parameter to
preserve proper working of Mac aliases.
-c Move a file from one HELIOS volume to another
HELIOS volume with a different character set
without converting to the target volume
character set or reporting error messages.
Important: If you do not use the -c option with the "dt mv" and "dt cp" commands, an error message only occurs if you copy/move files from a HELIOS volume to another. Copying/moving from a UNIX directory will not provoke any warning at all.
The behavior of the different UNIX implementations differs if both the -f option and the -i option are set. "dt mv" uses the following rule: if both the -f option and the -i option are set, this is not considered an error; here, the -f option will override the -i option.
Important: If you move a directory between HELIOS volumes, "dt" cannot simply rename just the directory entry. Here, the Mac Finder would have to perform two steps for this operation, namely copying a folder from one volume to the other and then deleting the folder on the source volume. "dt" has a similar task to accomplish. For every file and folder within the directory to move, first the object must be copied to the destination volume and registered in the destination desktop. Then, the moved objects must be deleted from the source volume and unregistered from its desktop database.
This is similar to moving a UNIX directory to a different device, where also all affected files must be copied and then deleted, with the difference that here all resources and streams must be handled, too.Please note that any Mac file/folder to which aliases are pointing will have lost its connection after successfully been moved between volumes. This would be the same if you used the Mac Finder for this operation. If source and target directory are on different file systems, "dt mv" copies the file and deletes the original; any hard links to other files are lost.
Volume root for both source and destination, is automatically touched to announce the changes to any Mac that has mounted the volume.
Usually, if a file is copied to a directory without a desktop (pure UNIX directory), the file ID is set to zero. When using the -k option, the file ID is maintained (if possible).
dt cp [-fikpnvzcyRXE] f1 f2
dt cp [-fikpnvzcyRXE] f1 ... fn d1
dt cp -r [-fikpnvzcyRXE] d1 ... dn-1 dn
- Copy a file "f1" to the destination file "f2"
- Copy the passed files "f1- fn" into the destination directory "d1"
- Copy the passed directories into the destination directory "dn"
-f Copy the file without prompting even if it is
overwriting an existing target. Note that this is
the default if the standard input is not a terminal.
-i Prompt for confirmation for each file. If the
answer begins with y (yes), the file is copied,
otherwise it is not.
-c Copy a file from one HELIOS volume to another
HELIOS volume with a different character set
without converting to the target volume
character set or reporting error messages.
The volume root directory for both source and destination, is automatically touched to announce the changes to any Mac that has mounted the volume.
If your source does not have a resource fork, and your destination is a volume, a default resource fork is created. Please do also take into account that every default resource will require 64 bytes, thus using the minimum disk block size on the destination file system. If there is only little free disk space on the destination volume, first check whether the destination file system can store all of the files. You can roughly calculate 1K (fragment size) per file/folder. Consult your UNIX system administrator or the appropriate pages in your UNIX manual for information on how to retrieve more exact information about your file system settings.
"dt set" modifies the additional file information stored in the file's resource fork. For more information about the flags and fields used here refer to your Mac documentation.
You can use HELIOS Admin to extract the creator/type information an application sets for certain icons. For a description see "Extension Mappings" in the EtherShare manual and the content of "HELIOSDIR/var/conf/suffixes".
-L Set the file to number of allowed concurrent
program launches, where "integer" is the
number of launches.
-C Clear the file ID numbers in the resource
fork (set to zero) and force the "afpsrv" to
allocate new ID numbers.
Please note that you may list these information using
"dt ls". If you need to set type or creator info with non-printable values, you may as well enter these values as octal values with a preceding backslash ("\"), and use the following escape sequences:
Please note that for both type and creator, all characters including 0 are valid. Each passed creator or type must be exactly 4 bytes long.
When you call "dt set" for a file without a resource fork, a default resource fork is created that is updated with the passed parameters, even if the destination file is not located in a HELIOS volume.
Please note that "dt set" does not create a missing ".rsrc" directory; use "dt touch" first on the file name in order to force creation of a related ".rsrc" subdirectory where the file's resource information can be stored.
For each file that is a directory, the directory content is listed; for each file that is an ordinary file, the name and (optional) resource related information are listed. When no argument is given, the current directory is listed.
-a List all entries, including those beginning with a
dot character ("."), which are normally not listed.
Note: Mac aliases cannot be created with the "dt tools". The internal structure of aliases is not documented by Apple.
This command prints the file mode (read-write access for owner, group, and others), the number of links (1), Finder attributes (read-only, protected), Desktop ID (5), type and creator (MooV, TVOD), user (Jon), group (admin), size (21988 bytes), date and file name (abc.mov).
Please note that the output format (e.g. time format) depends on the current settings for the local environment.
"dt ls" uses user and group information from the authentication server (if running), otherwise only UNIX users/ groups are displayed.
The "dt mkdir" command creates the passed directories including the corresponding resource fork and the desktop database entry.
-p This option creates "dir" by creating
all the non-existing parent directories first. The
mode given to intermediate directories will be
the difference between 777 and the bits set in
"umask".
Note: If no mode parameter is set, the final directory is created in octal mode 0777 considering the file mode creation mask "umask".
The command "touch" sets the access and modification times for the passed files. A file is created if it does not already exist.
The "upd" command touches the volume directory for the passed file or directory, so that all Macs that have mounted this volume can display the changes. This may be necessary e.g. if a file is created under UNIX and the Mac would display the new icon only after closing and re-opening the corresponding window.
Please note that all "dt" commands automatically update any needed volumes; so this command is only required, if files are created or changed by other, non-"dt" commands. Check if you can adjust your workflow to "dt" commands and thus avoid potentially harmful non-"dt" commands.
Note: Mac clients only scan for changes every 10 seconds. Usually, folder information are updated within this 10 seconds period. However, depending on folder sizes, number of folders open, foreground applications, etc., this may take longer. Thus, it may take some time after issuing an "upd" command, before the changes become visible on the client. You can either click into the folder window, or close and re-open it in case the displayed information do not seem to be up-to-date. In rare situations, it may be necessary to unmount and remount the volume in question in order to let the Mac flush its cache entries. Make sure that you are using the latest Mac OS and up-to-date AppleTalk/Network drivers.
An absolute mode is specified using octal numbers (0-7), please see the UNIX documentation for a full description.
A symbolic mode specification is a comma-separated list (without any intervening spaces) of symbolic mode expressions of the form:
In order to adjust permissions for files/directories and their related resource parts, you must own the directory in question. In case the resource directory does not match the enclosing directory's permissions, "dt chmod" cannot adjust the ".rsrc" directory's permissions. You have to log in as "root" in order to synchronize permissions of the two directories. Please refer to the UNIX documentation for a full description.
dt chmod a=rwx,g+s file
sets read ("r"), write ("w"), and execution ("x") permission for all ("a") users, and adds group set-ID mode ("s") for groups' permissions.
The "chown" command sets the user ID of the passed file to the new user ID specified by owner, and, optionally, the group ID to the new group.
-h If the file is a symbolic link, change the owner of
the symbolic link (not the owner of the file that is
referenced by the symbolic link). This option is
not supported on all platforms.
owner[:group] (use "." or ":")
A user ID and an optional group ID to be assigned. Both IDs may either be specified by the numeric ID or by the corresponding user or group name.
Please note that you need the necessary access rights to change the user or group ID of a file - according to UNIX semantics, only the user id "root" is allowed to change ownership settings.
Important: If the user specified with the "dt chown" command has never been validated by the HELIOS authentication server, it issues the "User not validated" error message. However, all regular UNIX users, e.g. "nobody", "root" are automatically validated.
"dt chown" uses user and group information from the authentication server (if running), otherwise oonly UNIX users/ groups are displayed.
-h If the file is a symbolic link, change the group of
the symbolic link (not the group of the file that is
referenced by the symbolic link). This option is
not supported on all platforms.
Please note that you need the necessary access rights to change the group ID of a file - according to UNIX semantics, only the user id "root" is allowed to change ownership settings.
"dt chgrp" uses user and group information from the authentication server (if running), otherwise oonly UNIX users/ groups are displayed.
The main purpose of this command is problem detecting. The Mac file, or respectively directory ID, is stored in both the file's resource fork and the desktop database. The ID stored in the resource fork may be listed by using the "dt ls" command.
The "idinfo" command may be used to display database information for the passed ID. If there is no difference between the resource data and the corresponding database entry, you should get the same information, otherwise the system administrator should issue a "rebuild" for the specific volume. You can use rebuild -s <volume root> to verify that the volume information is properly reflected in the related desktop database.
Note: Each volume has its own ID scope. Therefore, you must specify the correct volume for each ID you request. If you do not specify a volume with the -v option, the current working directory is assumed.
volume: '/support' id=3159 pid=2 name: 'temp'
specifying the volume detected, the ID, the parent ID for this file, and the corresponding name stored in the database.
Note: Please note that an ID detected in the database that does not have any corresponding file in that volume does not cause any problems or data loss.
Each Mac file on a HELIOS volume has an ID number that is stored in both its resource fork and the desktop database. The "dt iddump" command outputs all file IDs of the volume in question (specify volume path) from the desktop database to "stdout".
The information in this section are mainly meant for system administrators who want to integrate the "dt tools" into custom solutions, e.g. backup scripts. You do not need these information for the simple use of the "dt tools" (however, it does not hurt to read on).
Each HELIOS volume is accompanied by a corresponding desktop database, which holds additional information about each file or directory stored in this volume.
Each file, directory or stream in a HELIOS volume consists of two parts: the "data fork" and the "resource fork".
Though invisible to the Mac user, both parts of a file and the corresponding information in the desktop database are handled transparently and automatically by EtherShare. On Windows, a file consists of a long number of additional streams (but no desktop database info).
If you also use ImageServer and/or PCShare these HELIOS products will also coordinate access to shared files/folders with the HELIOS volume desktop databases, and handle the resources and streams.
Looking at a volume from the UNIX point of view, you will find two files: the data fork as the "normal" UNIX file, and the resource fork in a subdirectory ".rsrc" with the same name, and all corresponding streams.
Since UNIX files consist of only one part, but Mac files of two (see Mac files under UNIX), each pure UNIX directory ("UNIX-dir") should only have entries without a corresponding resource fork, and in a HELIOS volume ("volume-dir") each entry should have a corresponding resource fork and streams.
If you now copy or move an entry from a "UNIX-dir" into a "volume-dir", "dt" always creates a default resource for this file. This is also the default behavior, if you move or copy a file from one "volume-dir" to the other.
If you copy an entry from a "volume-dir" into a "UNIX-dir", "dt" copies or moves the resource fork and streams as well, even if the resource fork is not needed here. This is done in order to avoid any unintentional data loss. If you do not want to consider the resource forks here, you may use the -n option for the "mv" or "cp" command.
The "dt tools" do not require the desktop server to run on the UNIX server where you issue the commands, but by default the "dt tools" try to connect to the local desktop server on the server where you issue the commands first.
If this volume is already in use (e.g. mounted as a volume on a Mac) the "dt tools" try to connect to the current server which serves this volume. This could be a different server, e.g. if this volume is mounted via NFS. Source and destination are handled completely independently, e.g. a source file for a copy command may be served by a local desktop server, while the destination directory which is located in a NFS-mounted directory is served by another HELIOS desktop server.
The "dt tools" are unable to connect to a desktop server, when the target directory is not mounted by any Mac and no local desktop server is running. In this case start a local desktop server or mount this volume on a Mac.
The "dt tools" will do an extensive verification in order to connect to the proper desktop server. Nevertheless, please verify your environment so that no single HELIOS volume can be accessed at the same time from different HELIOS servers.
Note: In case of different desktop servers being involved in a "move directory" operation, it may be necessary to traverse through directories instead of just moving the directory entry - see dt mv for details.
Please note that you may unintentionally cause this problem by accessing an already mounted volume via auto-mounter:
On any mounted volume, the HELIOS file servers automatically detect any file which has an accompanying resource fork with a zero ID (the ID set to numerical zero) when the parent directory is opened from a Mac.
A file without a valid ID does not have any database entry, and therefore, a new desktop database entry is created, and the resource fork is updated using the new ID.
The "dt tools" never create zero ID resource forks in volumes unless you force this behavior using the -z flag for the "cp" and "mv" commands.
Important: This option should only be used by system administrators - it is not intended for normal use.
One main purpose of the "dt tools" is to avoid problems when restoring data from a backup device. Here we describe how you should use the "dt tools" for backup/restore purposes:
You may backup your data as usual directly from your volume. Please make sure that the hidden ".rsrc" directories are included. To prevent backup of incomplete data in files, make sure that the files/folders to be backed up will not be manipulated from other Mac or UNIX applications. Only if you intend to do a complete backup and restore of a HELIOS volume you might include the ".Desktop" file. For the period of time you need to backup and restore this volume, HELIOS must be stopped with "stop-helios".
Important: Do never restore your data directly into the destination volume!
Please create a restore directory on the same device (partition) where the destination volume resides, e.g. if you intend to restore files into e.g "/data/mypics", create a directory "/data/restore". This "restore" directory must not be defined as a volume, or even being mounted. Now restore your data using your usual restore procedure into this newly created directory "restore". Make sure that you do not restore the ".Desktop" and ".DeskServer" files.
Important: Do never remove these files (".Desktop" and ".DeskServer") from a mounted volume or manipulate them directly. This may cause not only data loss for this volume, but may cause problems for all defined volumes on the server. In case you want to backup/restore these files, make sure that EtherShare and PCShare (if you are sharing volumes) are stopped completely prior to your backup/restore operation. Only if you stop HELIOS by means of "stop-helios" and stop access from other HELIOS servers, intermediate access to these files from HELIOS processes can be prevented.
Finally move the restored files into their final destination. This procedure avoids any data loss and may be used in a mounted destination volume.
Usually, new (unused) file IDs are assigned when you copy files to a backup destination. It may, however, be sensible to keep the file ID (using the -k option). Thus, the file can be re-used with its original ID. This is especially useful if you are using Mac aliases pointing to these files/folders. You should use this option sensibly. In case a complete desktop rebuild has been conducted after your backup operation, aliases may point to the wrong destination.
To avoid any problems, it may be a good idea to run an integrity check for the destination volume by entering rebuild -s on the command line, before restoring any files.
© 2006 HELIOS Software GmbH