Introduction & Overview

Data storage in corporate LANs and WANs is growing at an explosive and sometimes alarming pace. Between 1991 and 1995, it is estimated that a typical computer network experienced a three-fold increase in data storage. The dramatic decrease in prices of hard disk storage, and the unprecedented growth of networks and new applications such as multimedia, have fueled this explosion even more.

This proliferation of data storage poses challenging problems for network and system administrators. Initially, system administrators "threw" more hardware at the problem to satisfy user requirements for more disk space. In the last few years, IS and IT personnel - especially those who have been exposed to data center/mainframe storage management concepts - have started to explore and deploy hierarchical storage management (HSM) solutions for client/server distributed computing environment. HSM is a policy-based storage management strategy whereby data is automatically "migrated" from one storage medium to another based on a set of policies. Migrated files appear "local" to the system and when accessed, they are automatically recalled from the migration store.

It is estimated that there are only a few hundred sites worldwide that are currently deploying HSM to manage storage in mission-critical, client/server environments. Most who have deployed HSM for their client/ server environment have been disappointed with the results. The reasons for this low satisfaction level vary, but two important factors emerge:

• HSM was thought of as a panacea to all storage management problems and vendors promised dramatic reductions in total cost of ownership: "just install this software and all your storage management problems go away". After deployment, IS personnel found that they traded in one set of problems for another, and mission critical data was not being protected or managed effectively.
• Another important factor was that after deployment, IS personnel realized that the dynamics of the mainframe/data center environment are completely different from the distributed client/server environment. Issues such as heterogeneous platforms and operating systems, network bandwidth and varying tape speeds were not considered before deployment. As a result, users found that the HSM solution that was touted as a "corporate standard" in reality managed storage on only one of the dozen or so different platforms that typically exist in the distributed environment.

This white paper is a primer on Hierarchical Storage Management (HSM) for the distributed client/server environment. It provides the reader with information on basic HSM definitions, its benefits, the role HSM plays in overall storage management, and factors to consider before deploying HSM. It demonstrates how HSM, when properly deployed, is an effective tool to manage storage of distributed data in a client/server environment. The final section of this paper contains information on Legato Systems' HSM Application, including an implementation overview and a Statement of Direction on future plans.

HSM - Definitions and Concepts

The operative words in the phrase Hierarchical Storage Management (HSM) are Hierarchy and Storage. HSM is a policy-driven data management strategy whereby data is moved from one storage medium to another based on a set of policies.

The hierarchy of storage media is often a function of cost of media (in $/MB) and access speed of data from the specific media. Typically, magnetic disk is the first "stage" of hierarchy (i.e., most expensive media, but fastest access time), and tape is the last stage (i.e., slowest access time but least expensive storage media). See Figure 1.

Figure 1.

Migration client. A migration client is any system on the network which contains data that needs to be migrated - either now or in the future. A migration client consists of multiple file systems or volumes; one or all volumes can be under migration control.

Migration server. A migration server is a system on the network which, with the migration store, provides migration services to clients on the network.

Migration store. The migration store resides on the migration server. It contains data that originally resided on the migration clients before being automatically migrated over to the server. The migration store can be any type of storage media - disks, tapes, optical etc. Typically, the migration store consists of a hierarchy of storage media and the data "moves" across this hierarchy based on certain staging policies.

File migration. This in an operation where files are "migrated" from one physical location to another based on migration policies set by administrators. A migration operation frees up space on the client's hard disk and makes room for new files.

File migration can be automatic or on-demand. "Automatic file migration" is initiated when a threshold as defined in the migration policy is reached. This operation is completely transparent to the user and the administrator. "On-demand migration" is initiated by the system administrator who needs to migrate certain files immediately and cannot afford to wait until a threshold condition is reached.

Migration policies define the criteria for automatically migrating files, i.e., when files should be migrated from a system and what specific selection criteria should be used to select files for migration.

Water marks are a type of migration policy based on disk capacity. High water mark defines the critical phase when a file system or disk reaches its critical level of capacity. Once that threshold is reached, the selection criteria can be based on file usage (e.g., migrate a file if it has not been read from or written to for over 90 days), file size (e.g., migrate files that are over 30MB) and file owners (e.g., migrate Mary's files). Migration occurs until the file system reaches the low water mark

File Recall or de-migration. File migration, whether automatic or on-demand, should be completely transparent to the user. Even after the file is migrated, it should appear to be "local" to the user. When a user needs to access a migrated file for use, it is recalled (de-migrated) automatically from the migration store. In some cases, users might notice a pronounced delay when trying to access the file since the migrated store can be a tape which has latent delay in access rates.

Figure 2.

Data staging is defined as the process whereby the migrated data is moved from one level of storage hierarchy to the next within the migrated store. Staging can be one or many levels. As before, the operation of staging is transparent to the user and is governed by staging policies. Staging policies dictate when the data is staged from one storage medium to the next within the hierarchy. Typically, staging is based on time, e.g., "within the migrated store, move data from optical to tape if it has resided on optical for more than 30 days".

Figure 3.

A complete Hierarchical Storage Management solution is a balanced combination of migration and staging established within the larger context of distributed data management.

Myths and Misconceptions

Given the plethora of vendors selling HSM solutions, there are many myths and misconceptions why an organization should implement HSM for its distributed environment. HSM should not be deployed for any of the following reasons:

As a substitute for backup and archiving. This is the most common trap that IS personnel fall into. HSM is not a replacement for backup and archiving. It is complementary to both backup and archive and must be viewed as such. While the distinction between backup and HSM is clear, the difference between archiving and HSM is often nebulous.

Data archiving is the process of taking a "snapshot" of a file or a series of related files as it resides on primary media (disk) at a given point in time. The image of the snapshot typically resides on removable media - usually tapes or optical disk. Archiving provides an extra level of protection for company's mission critical data. An important side benefit of archiving is that once the snapshot is safely stored on removable media, the necessary files can optionally be deleted from the primary storage. This approach can be used to conserve hard disk capacity. Archiving is normally performed on related files or data associated with specific projects.

To reduce hardware storage costs. Some IS personnel have deployed HSM with the notion that they would never have to purchase incremental hard disk storage capacity. Justifying an HSM solution based on a media cost optimization scheme is a moving target, given the rate at which street prices of storage hardware have been decreasing.

Why deploy HSM ?

As a complementary solution to backup and archiving. By strategically deploying HSM, organizations not only can protect the network against accidental data loss, but also manage network storage resources more effectively.

To "house clean" old files and data. HSM should be viewed as a storage management policy that allows organization to effectively manage data on the network by moving older, unused data to the migrated store. The primary benefit is migrating old files so newer files can be managed better and without compromising the availability of the migrated file. Any cost savings from storage hardware should be viewed as a secondary benefit.

Prerequisites for Deploying HSM:

Since distributed computing is extremely heterogeneous in nature, organizations should ensure that certain basic storage management elements are in place before HSM is deployed enterprise-wide. These include:

A sound backup and archiving strategy. As mentioned earlier, HSM is not a replacement for backup or archiving. Before HSM is deployed, it is recommended that organizations have a sound backup and archive strategy in place for data protection across the heterogeneous network.

Proper analysis of size of networks and age of data. Typically large networks with an inordinate amount of old data are ideal candidates for deploying HSM. While it is difficult to quantify old data, it is recommended that organizations deploy HSM if data older than one year takes up a sizable chunk of the network storage resources. There are several tools available in the market that analyze the age of data.

Commitment from end users. It is imperative that IS personnel gain full support from their end-user community before deploying HSM across the organization. The good and the bad news about the file migration component of HSM is that it is transparent to the end user. End users are often irritated to find that their data has been migrated to a different place without their knowledge. This is exacerbated when they try to access a migrated file and it takes longer than usual to access it, due to the built in latency in demigrating the file. Users should also be participating in the management of their data by using archiving tools as provided.

Factors & Considerations

To ensure a successful, integrated storage management strategy, organizations should take into consideration various issues before selecting a HSM solution. Key factors include:

Integrated solution. Where possible, organizations should implement backup, archive and HSM from the same software vendor. Deploying an integrated solution from the same vendor has many advantages. These include:

• Consistent approach to managing corporate data
• Effective sharing of storage resources, e.g., the same autochanger can be used for backup as well as migrated store
• Common user interface
• Single point of contact for technical support and other support services
• Simplified disaster recovery procedure

Peaceful coexistence between backup and HSM. One key advantage of deploying the solution from one vendor is satisfying the requirement for "peaceful" coexistence between backup/archiving and HSM. If the solution is not integrated, users might find that backing up a file causes recall or de-migration, or that the migrated store cannot be backed up using the backup product. Conversely, a valid candidate for migration might be ignored if accessed by the backup software.

An enterprise problem requires an enterprise solution. Heterogeneity and distributed client/server computing go hand-in-hand. It is critical that the HSM solution being deployed supports (or the vendor plans to support) all the key platforms in the network.

Independence from storage hardware. The best software for HSM should work with any storage media including tape, optical, or magnetic disk. It should not force the user to purchase a specific hardware device (e.g., "works only with optical jukebox X from vendor Y").

Legato NetWorker's HSM Implementation

Overview:

In January 1996, Legato announced and shipped the NetWorker HSM Application and NetWorker StorSuite.

The NetWorker HSM Application is sold as an add-on to the NetWorker for UNIX Release 4.2 backup server software. The HSM Application leverages Legato's universal client connections pricing and packaging model, meaning that as soon as the server is enabled to provide HSM services (through the addition of a simple alphanumeric enabler code), existing NetWorker clients can immediately take advantage of the HSM services. Legato has announced HSM for Solaris and SunOS clients. Support for other platforms will be forthcoming.

NetWorker StorSuite is a centrally controlled, integrated suite of storage management applications that consolidates backup, archive and HSM under a consistent framework. StorSuite fully automates the entire storage management process, allowing network managers to create a centralized storage center for the distributed network enterprise with reduced operational and administrative intervention.

Implementation:

File migration. File migration on a NetWorker server is typically controlled by the system administrator. NetWorker's intuitive graphical user interface allows the administrator to set up migration policies.

The migration window (Figure 4) contains all the parameters for setting up migration policies for NetWorker clients. The administrator can set up a "global" migration policy for all the file systems within a given client, or can optionally configure a unique migration policy for each file system within a client.

Figure 4. NetWorker migration window.

High water mark defines the threshold condition when automatic migration should commence, thereby preventing "out of disk space" situations. Migration continues until all eligible files are migrated or until the low water mark is reached. The "%" for high and low-water mark indicate the percentage of disk space to be kept free or full.

Before the actual migration begins, NetWorker "selects" files that are candidates for migration. The selection criteria can also be unique for each file system or client. Selection criteria include:

• last access time - files that have not been read or written to since this time are eligible for migration
• minimum file size - files smaller than the prescribed value are not migrated
• file owner
• file group

NetWorker also gives the administrator to the capability to "preserve" certain files. Preserved files will never be candidates for migration. Administrators can specify entire directories or sub directories to be preserved.

Pre-migration "sweep". Similar to the nightly (or daily) network-wide backup operation, NetWorker conducts a network-wide pre-migration process. During this sweep, files that are candidates for migration are migrated, but the original files continue to remain on the client system's hard disk.

When the high water mark is reached, NetWorker examines the candidate list, verifies that pre-migrated files have not changed since the migration sweep, and replaces the contents of the file with a stub. If a file has changed between the pre-migration phase and the actual migration, it is no longer considered to be a candidate for migration.

Once the migrated data is sent to the server, the file can be managed by NetWorker's powerful server side features including media pools, cloning and verification. Media pools ensure that migrated data is not mixed with backup and archive data on the same tape. This ensures that migrated data is never automatically recycled (unless explicitly specified by the administrator). Cloning allows the administrator to make multiple copies of the migrated data and store them in different locations. Verification ensures that the migrated data on a media is readable before it gets replaced by a stub on the client system.

HSM consistency checker: The NetWorker HSM Application includes a file consistency checker program (nsrhsmck) that is run to ensure that file system integrity is always maintained. The consistency checker is designed to fix renamed files, invalid stubs and incomplete transactions (e.g., if a system crashes during the middle of a migration). nsrhsmck ensures that there is no data loss in the storage management environment.

Figure 5.

Super-fulls. NetWorker HSM also includes a powerful feature called super-full. Super-full for a NetWorker client includes a full level save for the client as well as the migrated data for the client. So NetWorker not only backs up the file stubs, it also backs up the data that the stubs point to. This powerful feature is vital during disaster recovery since a particular client's data is self-contained in the media.

Recall or de-migration. When a user or an application tries to open a migrated file, the stub triggers a recall. When the recall is successfully complete, the stub is replaced by the original file. If the local hard disk doesn't contain enough free space to de-migrate the file, appropriate notifications are issued.

Integration. One of the key advantages of NetWorker StorSuite integration is that the backup and HSM applications are "aware" of each other. For example, when NetWorker backs up a stub, it recognizes it as a stub and does not initiate a recall. Also, the migrated store is backed up by NetWorker. This cohesive and consistent approach ensures that data integrity is always maintained.

Migration reports. NetWorker's migration control window provides up-to-the-minute status reports on migration activities. This centralized administration feature displays the migration history for all migration clients - including statistics on pre-migration, migration and recall for every file system under migration control. These reports can also be "dispatched" by NetWorker's powerful notification engine.

Command Line Options: Users can utilize NetWorker's powerful command line capabilities to generate ad-hoc reports and functions. For instance, the command nsrmig -n produces a list of files eligible for migration without actually migrating them. Similarly, the command nsrpmig -n produces a report of files eligible for pre-migration. In addition to scheduled or automatic migration, users can also perform on-demand or manual migration. This feature is especially useful when the file system is full or is in the process of becoming full.

Figure 6. NetWorker migration control window.

Benefits to Legato's Approach

The main benefits to Legato's NetWorker HSM Application and StorSuite are:

Heterogeneous platform support. In addition to Solaris and SunOS platform support, Legato intends to broaden HSM and StorSuite to other platforms including HP-UX, AIX, NetWare and Windows NT.

Software-only solution. NetWorker is designed as a solution that is independent of the underlying storage hardware. This gives the administrator the freedom of choice to integrate NetWorker HSM and StorSuite with their favorite storage hardware.

Centralized administration. All NetWorker HSM and StorSuite environments can be managed from any client on the network, as well as using leading system and network management frameworks.

Easy to install and use. NetWorker for UNIX Release 4.2 is "HSM Ready". By keying in a simple enabler code, NetWorker backup server can be transformed into a StorSuite server.

Statement of Direction:

In addition to the platforms currently supported by Legato, leading system vendors such as Digital Equipment, Siemens Nixdorf and Silicon Graphics have licensed the NetWorker HSM technology.

As part of our on-going commitment to provide high quality products to our customers, Legato will enhance the functionality of NetWorker HSM during the next 12-18 months. The emphasis of these new developments will be focused on the following areas:

• Extended platform support
• Multi-level staging at the server

Support for standards and HSM-ready file systems from OS vendors, including a DMIG-based file system for the UNIX platforms.