File Space Accounting

Introduction

File Space Accounting allows recursive monitoring of file usage and provides usage reports that can be grouped by user ID, file type, project, hosts, and more.

The StorageAnalyzer collects file space data locally from every client where the StorageAnalyzer is enabled. A central configuration file on the Core Server is used to specify which file spaces to account. Each file space is accounted locally since this accounting process is I/O intensive. Subdirectories that are mounted over the network are not accounted by default due to I/O intensity, and because networked file systems are often mounted on many hosts. But network-mounted file systems are accounted if file space is configured to do so in the file systems configuration.

The first configuration that affects file space accounting is the file systems configuration file. File systems excluded in this file are also excluded from file space accounting. As mentioned previously, all non-local file systems are excluded by default.

This provides a clear understanding of how file space is being utilized. This will produce the following aggregated data types used for historical reporting:

• (29) Filespace

The following section will guide you on configuring file space accounting.

Requirements

• An Open iT Core Client connected to an Open iT Core Server or a coexistent Open iT Core setup
• Activated StorageAnalyzer and file space collector

Configuring File Space Accounting

Open iT collects file space data from all clients. A central configuration file on the Core Server specifies which file spaces to account for. The process is I/O intensive; therefore, each file space is accounted for locally. By default, subdirectories mounted over the network are not included in the accounting. However, network-mounted file systems can be included if configured to do so.

Windows

The filespace_win.conf configuration file contains a list of mappings to filter the components considered in storage monitoring.

NOTE

The server distributes the filespace_win.conf configuration to the clients automatically. Any modification on the client will be lost.

The file has three main configuration parts:

Name	Description
site	This includes configuration for various sites, domains, or workgroups that will be included in storage monitoring.
exclude	This includes configuration for various files that will be excluded in storage monitoring.
map	This includes mapping configuration for specific hosts and files to a specific account.

Parts of the File Space Configuration File

Adding Site to File Space Storage Monitoring

1. In the Core Server, open filespace_win.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration. Read through the written instructions for configuration.

2. Locate the line:

   filespace_win.conf

   all
   {
   # Pattern that must match for this site/domain/workgroup to be considered.
   # Use * to match all possible sites/domains/workgroups.
   # Example of using a workgroup:
   # site=WORKGROUP
   site=*

3. Add new entries as needed, following the format:

   site=<site_name>
   Example

   site=*
   site=SVG.OPENIT.LOCAL
   site=LOCAL

   The example means to include storage monitoring for the domain SVG.OPENIT.LOCAL and the workgroup LOCAL.

4. Save the changes.

Excluding Files to File Space Storage Monitoring

Excluded file systems indicated in the configuration file are also excluded in file space accounting. All non-local file systems are also excluded by default.

1. In the Core Server, open filespace_win.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration. Read through the written instructions for configuration.

2. Locate the line:

   filespace_win.conf

   # Files to be excluded from accounting.
   # Default is to exclude names enclosed in curly braces and names ending in
   # .sys (system files).
   # NOTE: Some of the files excluded by default may not be accessible, and
   # warnings will be reported and incomplete reports generated if they are
   # included.
   exclude={*}
   exclude=*.sys

3. Add new entries as needed, following the format:

   exclude=<file_name>
   Example

   exclude=*tmp
   exclude=*temp

   The example will exclude temporary files (files ending with .tmp and .temp) in file space storage monitoring.

4. Save the changes.

Adding Maps to File Space Storage Monitoring

1. In the Core Server, open filespace_win.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration. Read through the written instructions for configuration.

2. Locate the line:

   filespace_win.conf

   # Mappings.
   map
   {
       # Global match. Account all systems on "My Computer" if no other config is available.
       global
       {
           host=*
           match
           {
               all
               {
                   dir=*
                   account=UNDEF
               }
           }
       }

3. Add new blocks under the map block, following the format:

   Example

   map
   {
   ...
       temp
       {
           ost=desktop123win
           match
           {
               temps
               {
                   dir=?:\temp*
                   account=TEMP
               }
           }
       }

   The example will include temporary files in the host desktop123win for storage monitoring.

4. Save the changes.

Unix

The file_space configuration file contains a list of mappings to filter the components considered in storage monitoring.

NOTE

The server distributes file_space configuration to the clients automatically. Any modification on the client will be lost.

The file has three main configuration parts:

Name	Description
DOMAIN	This includes configuration for various domains or workgroups that will be included in storage monitoring.
EXCLUDE	This includes configuration for various files that will be excluded in storage monitoring.
map	This includes mapping configuration for specific hosts and files to a specific account.

Parts of the File Space Configuration File

Adding Domain to File Space Storage Monitoring

1. In the Core Server, open file_space in the Configuration directory, which is by default in /var/opt/openit/etc. Read through the written instructions in the file.

2. Add a new line for the DOMAIN using the following syntax:

   DOMAIN Syntax

   DOMAIN:         <domain_name>
   DOMAIN Example

   DOMAIN:         samplecompany.com

   This example indicates that the hosts to be accounted for are within the samplecompany.com domain.

   NOTE

   Lines starting with hash signs (#) are considered comments and are ignored in the processing.

3. Save the changes.

Excluding File to File Space Storage Monitoring

1. In the Core Server, open file_space in the Configuration directory, which is by default in /var/opt/openit/etc. Read through the written instructions in the file.

2. Add a new line for the EXCLUDE using the following syntax:

   EXCLUDE Syntax

   EXCLUDE:      <files_or_directories_to_exclude_from_accounting>
   EXCLUDE Example

   EXCLUDE:      \.snapshot

   This example will exclude .snapshot files from the accounting.

   Add new entries as needed. Consider using regular expressions to better match the files or directories to be excluded. Make sure that all instructions and rules are followed to avoid malfunctions.

   Entries are processed from top to bottom, and the last pattern that matches is prioritized.

   NOTE

   Lines starting with hash signs (#) are considered comments and are ignored in the processing.

3. Save the changes.

Adding Maps to File Space Storage Monitoring

1. In the Core Server, open file_space in the Configuration directory, which is by default in /var/opt/openit/etc. Read through the written instructions in the file.

2. Add a new mapping line using the following syntax:

   Maps Syntax

   <hostname>:<directory>      <account_name_or_group>
   Maps Example

   fs1:/testfs    TEST

   The example above enables file space accounting on host fs1 and the directory /testfs is accounted to TEST.

   Add new entries as needed. Consider using regular expressions to better match the directories and accounts. Make sure that all instructions and rules are followed to avoid malfunctions.

   Entries are processed from top to bottom, and the last pattern that matches is prioritized.

   NOTE

   Lines starting with hash signs (#) are considered comments and are ignored in the processing.

3. Save the changes.

Configuring File Type Classification

File space accounting collects all file data included in file systems by default. Three possible methods (levels) classify the files:

Level	Description
Level 0 Accounting	Select this as a faster alternative if the file type is of no interest. The file type will be regarded as undefined and set to undef.
Level 1 Accounting	This is the default method that uses filenames to decide on the file type.
Level 2 Accounting	This is the most advanced and slowest method that uses the file contents to decide on the file type.

File Type Levels

1. Open filespace.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration.

2. Locate the filespace block.

3. Change the value of the filetype attribute to the desired value from 0 to 2. By default, value is set to 1.

   filespace.conf

   filename(FileName)="${OpeniT.directories.conf}/filename_win.conf"{}
   filetype(Integer)="1"{}
   temperatures(FileName)="${OpeniT.directories.conf}/file_temperature"{}

4. Save the changes.

For Unix systems, you may refer to the Configuring File Type Classification in Unix page.

Configuring File Names to File Type Mapping

Windows

Configure the filename_win.conf file to map file extensions to file types using the default file type Level 1.

NOTE

The server distributes filename_win.conf configuration to the clients automatically. Any modification on the client will be lost.

Before mapping, verify the level and file used:

1. Open filespace.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration.

2. Locate the filespace block.

3. Verify the values of the filename and filetype attributes.

   filespace.conf

   filespace(FileName)="${OpeniT.directories.conf}/filespace_win.conf"{}
   filename(FileName)="${OpeniT.directories.conf}/filename_win.conf"{}
   filetype(Integer)="1"{}
   temperatures(FileName)="${OpeniT.directories.conf}/file_temperature"{}

After verification:

1. In the Core Server, open filename_win.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration.

2. Read through and follow the written instructions for configuration. Use any of the following syntaxes:

   Using Comma

   <file_type>
   {
       <file_extension1>,<file_extension2>,<file_extensionN>
   }
   Using New Line

   <file_type>
   {
       <file_extension1>
       <file_extension2>
       <file_extensionN>
   }
   Using Both Comma and New Line

   <file_type>
   {
       <file_extension1>,<file_extension2>,<file_extension3>
       <file_extension4>
       <file_extensionN>
   }
   Example

   binary
   {
       exe,com,bin
       ocx,sys,scr
   }

   This example will map exe, com, bin, ocx, sys, and scr to file type binary.

   By default, the file already has defined mappings. Add new entries as needed. Globbing is allowed in extension mapping. Use only lower case for file extensions. Make sure that all instructions and rules are followed to avoid malfunctions.

3. Save the changes.

Unix

Configure the filename_type file to map file extensions to file types using the default file type Level 1.

NOTE

The server distributes filename_type configuration to the clients automatically. Any modification on the client will be lost.

Before mapping, verify the level and file used:

1. Open openit.cfg in the openit directory, which is by default in /etc/opt/openit/.

2. Locate the STORAGE_FILE_SPACE_LEVEL and STORAGE_FILENAME_TYPE_MAP.

3. Verify that the values are 1 and filename_type, respectively.

   openit.cfg

   STORAGE_FILE_SPACE_LEVEL    1
   STORAGE_FILENAME_TYPE_MAP   filename_type

After verification:

1. In the Core Server, open filename_type in the etc directory, which is by default in /var/opt/openit/etc.

2. The mapping file is line oriented and contains two types of lines: regular expression mappings and comments.

   The regular expression mappings begin with match: followed by a Perl type regular match expression and a file type.

   filename_type Syntax

   match:      <Perl_type_regular_expressions>       <file_type>

   The Perl type regular expression may match files depending on directory or file name.

   NOTE

   The regular expression uses the entire file path of the file.

   Example

   match:      \/bin\/[^/]+$       binary

   This example will map any string that ends with /bin/<string> to file type binary. <string> is a single file or name that does not include another /.

   By default, the file already has defined mappings. Add new entries as needed. Make sure that all instructions and rules are followed to avoid malfunctions.

3. Save the changes.

Configuring File Temperature

The file_temperature configuration file contains various temperature levels for the files accounted for. There are five default temperature levels, each with its own default label and upper boundary (in days).

file_temperature

# Temperature settings for file_space accounting.

# This file has the default settings for five
# temperature levels.
# Temperature labels  Upper boundary(days)
# ------------------  --------------------
hot                   30
warm                  90
tepid                 360
cold                  720
frozen                infinity

In this case, the lower boundary is considered as the upper boundary of the previous level. The lower boundary of the first level is assumed to be 0. The last level should have no number but infinity. This means that there will be no upper limit for this temperature level.

NOTE

The server distributes file_temperature configuration to the clients automatically. Any modification on the client will be lost.

Windows

1. In the Core Server, open file_temperature in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration.

2. Modify the configuration file as needed.

   NOTE

   Temperature levels must be in ascending order of the upper boundary values.

3. Save the changes.

Unix

1. In the Core Server, open file_temperature in the Configuration directory, which is by default in /var/opt/openit/etc.

2. Modify the configuration file as needed.

   NOTE

   Temperature levels must be in ascending order of the upper boundary values.

3. Save the changes.

NOTE

There may be any number of temperature levels, but there is no way of changing the levels for old data. The temperature levels of data collected earlier remain unchanged, even if the default settings are changed.

Configuring File Space Period and Offset

1. Go to the scheduler directory, which is by default in C:\Program Files\OpeniT\Core\Configuration\scheduler, and open collect_storage_filespace-win.oconf.

2. Locate and set the value of root.scheduler.jobs.collect_filespace_win.scheduling.start-triggers.trigger#1.period. The default value is P7D.

   collect_storage_filespace-win.oconf

   period
   {
       description=Run weekly
       type=timespan
       value=P7D
   }

3. Locate and set the value of root.scheduler.jobs.collect_filespace_win.scheduling.start-triggers.trigger#1.offset. The default value is P-5DT2H.

   collect_storage_filespace-win.oconf

   offset
   {
       description=Rather than running at midnight Monday, start 2 hours before midnight on Saturday
       type=timespan
       value=P-5DT2H
   }
   NOTE

   Specify the period and offset values to your desired timespan based on the ISO-8601 duration format PnDTnHnMn.nS.

4. Save the changes.

For Unix systems, you may refer to the Configuring File Space Period and Offset in Unix page.

Sample Historical Report

File Storage Report