Technical Description for Managing Virtuals

Introduction

Each Open iT data type has a limited number of classifications and measurements. These are strictly limited to the values defined when the data types were created. Though this will cover most of the needs for reporting, there are occasions when more is needed. This is where the Virtuals come in. There are two general types of virtuals:

• Virtual Classifications
• Virtual Measurements

Common to all these values is that they are not present in the actual database and are calculated or inserted when reports are created. This is a very flexible way of adding additional values, but at the cost of slower report creation.

Virtual Classifications

Virtual classifications are mostly useful for adding extra information to a data type by lookup. I.e., if there is an external source of information about which user belongs to which department, this can be added as a virtual classification and made available for selection in reports.

note

All virtual classification data is specified down to the day level. This means that even if reports are for a greater resolution (like month or year), the backend needs to read data at the day level in the database. This can potentially slow down the report generation by the relevant scale (e.g., 365 for yearly reports).

Adding a Virtual Classification to a Data Type

Virtual classifications must be added in the acc_types file, which is located by default in:

• C:\Program Files\OpeniT\Core\Configuration for Windows
• /var/opt/openit/etc/ for Unix

The syntax for a virtual classification is:

Virtual Classification Syntax

(Virtual^Name, Description,
    ((Source^Value, mapping_file)))

Name	Description
Virtual Name	Classification name appearing in the report creation GUI
Description	Longer description file shown in "help" text
Source Value	Which data type classification value map is based on
mapping_file	Name of mapping file used to create the virtual classification

Virtual Classification Syntax Parameters

Make sure you use the parenthesis in exactly the same places. Also, note that the ^ character is a substitute for the space character in the acc_types configuration file.

To add a map from User name to Department in the Individual license usage v2.0, you can add the following lines to the data type definition:

Example

(Department,Mapping^from^User^name^to^Department,
    ((User^name,user_department_mapping)))

The updated data type would then look like this:

Individual license usage v2.0

(Individual^license^use^v2.0,
    (
     (Count,Record^Count,x,SUM),
     (Elapsed^time,Accumulated^time^for^user^in^hours,Hour,SUM),
     (Max^used,Maximum^concurrent^licenses^used^by^user,x,MAX),
     (Distinct,Always^one^for^each^extracted^unique^classification,x,IDENT)
    ),
     (Vendor^License,
     Package,
     Feature,
     Feature^version,
     Prime^time,
     User^group,
     User^name,
     (Department,Mapping^from^User^name^to^Department,
        ((User^name,user_department_mapping)))),
)

The new mapping file must be named user_department_mapping and placed in:

• C:\Program Files\OpeniT\Core\Configuration\mappings (Windows)
• /var/opt/openit/etc/mappings (Unix)

on the Open iT Core Server. The format of the mapping file is described below.

Mapping Sequence

The mapping sequence for virtual classifications is as follows:

1. Exact Map
2. Field Map
3. Regular Expression Map
4. Sub Map

The order in which the sections are defined in the mapping file is irrelevant, and the first match from these sequences will be used, i.e., a value matched in Exact Map will block any possible matches in, e.g., Field Map or Regular Expression Map.

Exact Mapping

The Exact Mapping section is the easiest to maintain correctly since there is never any ambiguity about which line to match. However, all source values must be given and can grow to be huge. It is primarily recommended that this mapping be maintained automatically by scripts or programs that extract data from external sources (like Active Directory, SAP, etc.).

Mapping Syntax

Mapping Syntax

[EXACT-MAPPINGS]:

mosaic:   1993-01-01:Web-browser:1
netscape: 1995-01-01:Web-browser:2
explorer: 1997-01-01:Web-browser:5

Name	Description
source	Source value from the classifications in the data type (like Command, User name, etc.)
from-date	Start date from which the mapping line is valid
destination	Result value in the report for the match
weight	Weight value that is multiplied to all measurements in data in a report

Mapping Syntax Fields

Matching Rules

1. All configuration lines in the exact mapping section are read and ordered before mapping is done, i.e., the sequence of the mapping lines in this section is insignificant.
2. All measurements for matched records are multiplied by the configured weight.
3. If there are multiple lines with the same source and from-date, mapping, and weight multiplication will be applied to each of the multiple lines (records are duplicated).

Regular Expression Mapping

The Regular Expression Mapping can allow a mapping file with significantly fewer mapping lines. There are, however, more difficult restrictions when setting this up, and it is harder to maintain correctly. This is most useful when the source values contain values that are easy to map through regular expressions.

Mapping Syntax

Mapping Syntax

[REGEXP-MAPPINGS]:

ora.*:  2000-01-01:Oracle:1
^.*sh$: 2005-01-01:Shells:1

The syntax for the fields is the same as the exact mappings, but the source values can include Perl regular expressions.

Matching Rules

1. Regular expressions are checked line by line in the order they appear in the REGEXP-MAPPINGS block.
   Note that only the source field is used for initial matching.

2. If one regular expression matches, no further lines will be checked. For example:

   Example

   ^bash:2008-01-00:Shells:1
   ^.*sh:2007-01-01:OldShells:1

   A bash command occurring before 2008-01-01 will not be mapped to OldShells (it has already matched the line before but has been rejected because the date is not covered by the configured date range, from 2008-01-01 and onwards).

3. If multiple identical regular expressions occur adjacently, these will be handled as a group, and dates can be unsorted.

   Example

   ^.*sh:2008-01-01:Shells:1
   ^.*sh:2007-01-01:OldShells:1

   This works because the backend joins the expressions into a single match (i.e., it matches /^.*sh/ once), and then, after determining a match, it checks if there are multiple date ranges in the group.

   This means that lines with identical regular expressions MUST be adjacent in the mapping file, but the map data does not need to be sorted.

4. If there are multiple identical regular expressions and start dates, data will be split on multiple destinations based on weight.

   Example 1

   ^app:2008-01-01:Production:0.5
   ^app:2008-01-01:Exploration:0.5

   This will share cost among Production and Exploration, while:

   Example 2

   ^app:2008-01-01:Production:0.5
   ^app:2008-01-02:Exploration:0.5

   would not, since it would give 0.5 cost to Production on 2008-01-01 and 0.5 cost on Exploration from 2008-01-02 and onwards.

General Recommendations

Here are some general recommendations when setting up regular expression mapping:

1. Match the most specific expression first.
   A more generic expression will otherwise cause a match even if the date is invalid, giving Unmatched classification values in the reports.

   Example

   ^bash:2005-01-01:Shells:1
   ^.*sh:2005-01-01:OldShells:1

2. If a specific expression has different from-date or destination values, they must be explicitly specified.
   If you want to map all shell commands (commands ending in sh) from 2007-01-01 onwards to value OldShells, but bash should move to Shells from 2008-01-01, you can use:

   Example

   ^bash:2008-01-01:Shells:1
   ^.*sh:2007-01-01:OldShells:1

   As described above, bash will always match the first line even when the date is before 2008-01-01. You need to explicitly give all values from this regular expression.

   Example

   ^bash:2007-01-01:OldShells:1
   ^bash:2008-01-01:Shells:1
   ^.*sh:2007-01-01:OldShells:1

Field Mapping

Field mapping allows a classification to be viewed as consisting of several fields separated by some delimiter, e.g., /. Each field can be extracted by the field mapping and inserted into a virtual classification. It requires one configuration file for each field to be used. The input classification must somehow be ordered as a string consisting of all the fields. Because of this, it is not suitable for all data types and classifications.

This was originally intended to be used with the Filespace data type. There is a classification in this data type called Account, which allows for custom values listed in the file_space configuration file, which is located by default in:

• C:\Program Files\OpeniT\Core\Configuration (Windows)
• /var/opt/openit/etc (Unix)

Disk folders can be accounted to accounts organized as <department>/<user>. Each disk folder must be handled by the file_space configuration file. In addition, the (two) mapping files must contain a specification of which field should be selected for the virtual classifications.

Mapping Syntax

Mapping Syntax

[FIELD-MAPPINGS]:

FIELD: 0
DELIMITER: "/"
DEPT_A: 2005-01-01:Sales:1
DEPT_B: 2005-01-01:Accounting:1

Matching Rules

The mapping works as an exact mapping on one field (FIELD number when split by DELIMITER) of the source classification.

Sub Mapping

Subroutine mapping actually means calling a subroutine on the source classification and letting the return value be used as the target classification.

It has been used for mapping the Windows full user name format \\x\username to the last part of the string.

Mapping Syntax

Mapping Syntax

[SUB-MAPPINGS]:

sub {
    my( $value ) = @_;
    my $sep = '\\\\';
    my @fields = split( $sep, $value );
    return pop @fields;
}

Subroutine Rules

1. The subroutine must be anonymous.
2. The source value is given as the only parameter.
3. The target value must be returned.
4. Strings may need to contain escape characters (backslash) to compile.

Virtual Measurements

Virtual measurements are calculated at the time the report is generated. Currently, only one type of virtual measurement is supported. Distinct is used for counting distinct classifications.

Distinct

The Distinct measurement is strongly connected to the GUI concepts of selection and presentation. In the selection phase, all classifications to be extracted from the database are selected. During the presentation, the classifications to be displayed in the reports are selected. Usually, the two sets of classifications are the same, i.e., the same values taken from the database are displayed.

However, it is allowed not to show all the classifications selected from the database, and in some cases, this can be used in a very particular way. The Distinct measurement counts the number of classifications from the database that were selected in the selection phase but not presented in the report. For each classification displayed in the report, Distinct counts the underlying classifications that were selected from the database.

To be able to use Distinct for a data type, a measurement line needs to be inserted in acc_types for the data type.

Example

(Distinct,Always^one^for^each^extracted^classification,x,IDENT)

It should be located at the bottom position of the measurement list.

Individual license use v2.0

(Individual^license^use^v2.0,
 (
   (Count,Record^Count,x,SUM),
   (Elapsed^time,Accumulated^time^for^user^in^hours,Hour,SUM),
   (Max^used,Maximum^concurrent^licenses^used^by^user,x,MAX),
   (Distinct,Always^one^for^each^extracted^unique^classification,x,IDENT)
 ),
   (Vendor^License,
   Package,
   Feature,
   Feature^version,
   Prime^time,
   User^group, 
   User^name,
   (Department,Mapping^from^User^name^to^Department,
     ((User^name,user_department_mapping)))),
)

Many data types already have the Distinct type included, in particular, most of the license data types.

• Windows Periodic Summary App Usage
• Windows Inventory
• Total License Use v2.0
• Individual License Use v2.0
• Host License Use v2.0
• Host User License Use
• Host User License Use Logfile
• Total License Use Logfile
• Windows Module Usage
• Total License Queued
• Individual License Queued
• License Optimizer Use
• Server Connections
• Total Queued Logfile
• Individual Queued Logfile
• Internet Explorer URL Use
• Host License Use Logfile
• Individual License Use Logfile
• Total License Use Licenseevents
• Individual License Use Licenseevents
• OLAP User Concurrency

Examples

Using Distinct to Count Distinct Features Used by Each User

In Individual license use v2.0, it is possible to count the different features of each vendor license used by each individual user. This is done by selecting Vendor License, Feature, and User name in the selection, and Vendor License and User name in the presentation. The Distinct measurement must be used. The report will show the number of different features each user has used of each vendor license.

Vendor License	Feature	User name	Distinct
VL	F1	A	1
VL	F1	B	1
VL	F1	C	1
VL	F2	B	1
VL	F3	B	1

Distinct Vendor License-Feature-Username entries

Vendor License	User name	Distinct
VL	A	1
VL	B	3
VL	C	1

Counted Distinct Features per Vendor License-Username

The table shows that user A and user C have used 1 feature of vendor license VL. User B has used 3 different features.

Using Feature Groups to Count Distinct Users in Vendor License

This example describes how to create virtual classifications to get around one of the limits in the distinct measurement classification.

Introduction

The distinct values in the current report backend count all unique combinations of selected classifications. This means it is impossible to use a classification solely to limit the set of matched records if you also want to report on distinct values.

Let's say you have a vendor license VL with many features (F1 + ... + Fn). You want to count the number of users of VL but only for a subset (F1+F2) of the features. Now, you need to select classifications:

• Vendor License
• Feature (and sub-selecting F1 and F2)
• User name

and combined with the Distinct measurement, you will get one record for each unique combination of all the classifications.

If you remove the Feature name from the report, you will get an implicit sub-total calculation for this classification. However, it will still be included when counting the Distinct values, giving you more values than you want. Depending on what classification you show, you will get, e.g.,

Vendor License	Feature	User	Distinct
VL	F1	A	1
VL	F1	B	1
VL	F2	A	1
VL	F2	C	1

Distinct Vendor License-Feature-User entries

Vendor License	Feature	Distinct
VL	F1	2
VL	F2	2

Counted Distinct Users per Vendor License-Feature

Vendor License	Distinct
VL	4

Counted Unique Feature-User combination per Vendor License

All reports give the same number, which is not the number of unique users across the selected features from the vendor license (which is 3), because the number of unique values is still the same whether they are displayed or not.

Using Virtuals to Circumvent Problem

In this particular case, there is a way of avoiding this problem by using virtuals, but it's a bit cumbersome because the administrator of Open iT must set it up and tailor it to each feature set.

We need to select the desired features without getting different values for the desired features. This can be done by creating a virtual classification that maps the desired features into a single virtual Feature group.

Here are the steps:

1. Add virtual classification (called Feature group) to data type.
2. Add virtual classification to classification mapping file.
3. Create a virtual mapping file.
4. Regenerate the classification map.
5. Restart or refresh the report GUI.

The steps will permanently add a new Feature group classification to the data type.

Below is a detailed example of where we want to do this for the Individual license use v2.0 data type.

Add Virtual Classification to Data Type

Here, we add a virtual classification called Feature group, which is based on the Feature classifications and uses a mapping file called feature_group_mapping. The virtual definition looks like this:

feature_group_mapping

(Feature^group,Mapping^from_feature^to^group,
 ((Feature,feature_group_mapping)))

It should be inserted into the data type definition in acc_types. It may look like this afterward:

Individual license use v2.0

(Individual^license^use^v2.0,
 (
   (Count,Record^Count,x,SUM),
   (Elapsed^time,Accumulated^time^for^user^in^hours,Hour,SUM),
   (Max^used,Maximum^concurrent^licenses^used^by^user,x,MAX),
   (Distinct,Always^one^for^each^extracted^unique^classification,x,IDENT)
 ),
   (Vendor^License,
   Package,
   Feature,
   Feature^version,
   Prime^time,
   User^group,
   User^name,
   (Feature^group,Mapping^from_feature^to^group,
    ((Feature,feature_group_mapping)))),
)

Add Virtual Classification to Classification Mapping File

We also need to add this virtual classification in the classification mapping loaded by the GUI so we can select the correct value.

Since the Feature group is virtual, it needs to be added in the singleton list for the data type in classvar_mapping_config, which is located by default in:

• C:\Program Files\OpeniT\Core\Configuration for Windows
• /var/opt/openit/etc/ for Unix

The file may look like this after the update:

classvar_mapping_config

(Individual^license^use^v2.0,
 (Prime^time, Feature^group),
 ((Package, Feature), (Vendor^License, Feature), (Feature, Feature^version),
    (User^group, User^name))),

Create Virtual Mapping File

Now, we need to create a mapping file. This should be named feature_group_mapping, which should be located in:

• C:\Program Files\OpeniT\Core\Configuration\mappings for Windows
• /var/opt/openit/etc/mappings for Unix

Here, we need to add the maps from the individual features to the group. In this example, we want to map feature F1 and F2 and all features starting with SUB to our DISTINCT_GROUP.

feature_group_mapping

[EXACT-MAPPINGS]:

F1:2000-01-01:DISTINCT_GROUP:1
F2:2000-01-01:DISTINCT_GROUP:1

[REGEXP-MAPPING]:

SUB.*:2000-01-01:DISTINCT_GROUP:1

Next Steps

Generating Classification Variable Mapping

Updating Database

Creating a Report

Now you can create a report for data type Individual license use v2.0 by selecting:

• Vendor License
• Feature group (and sub-selecting DISTINCT_GROUP)
• User name

Removing Feature group and User name from the presentation should yield the desired values.