This page is intentionally hidden from navigation but remains accessible via direct link. It is retained for internal reference purposes only and may contain outdated or transitional content.
Data Flow
The client converts the raw data collected from the license manager or data source to a readable format by Open iT. It periodically sends the converted data to the server, stored in its incoming directory. The server then processes this data into a reportable format at predefined intervals and stores it in its database.
Once the process is complete, the data is ready for report generation. The report generation sequence interacts with the server to retrieve the data for generating reports. At this stage, based on the type of report selected from the user interface, report generation programs are run and send the output to the specified directories. View these reports using a standard web browser.
Open iT supports (and is continuously adding support for) a variety of license managers and data sources. Some applications and processes for supporting these are similar, but it also depends on how data are provided (e.g., log files, sample-based data via license status utility, API, etc.).
Data Collectors
Agents installed in the Open iT clients collect data and send it to the Core Server for processing and storage in a suitable format for reporting. The data collected by the client may be from multiple license servers and data sources.
Open iT LicenseAnalyzer has two separate data collection sets: one based on polling at regular intervals and the other based on collecting debug log data with samples to augment the debug log source.
Sample-based Collection
Open iT Licpoll
Open iT LicPoll (License Poller), named openit_licpoll, is a binary data collectors use to poll license servers regularly to get the license use status and availability.
The collection runs hourly using a 5-minute sample interval by default. However, both the sample interval and collection period are configurable. The primary data is transferred to the server every night for processing according to the client's timezone. You can also configure the data transfer interval.
In addition, the client sends license status snapshots (unless explicitly turned off) to the server when sampling. For example, the server will get a recent snapshot of the usage to display in the LicenseMonitor tool.
The data file contains one complete snapshot of usage and is signed to prevent tampering.
The snapshot frequency will increase or decrease if the sampling interval is modified.
Open iT Poll Collector
Open iT uses a binary called openit_pollcollector to collect license usage data by polling through another utility from the license server with a configurable polling interval of 5 minutes by default. The collected data is converted into Open iT data format and is parsed by dedicated parsers per license manager.
Compared to the License Poller (LicPoll), Poll Collector is used when the data source is in SQL, database, or CSV format. Currently, the data collected through Poll Collector does not support real-time reporting.
Open iT Log File Collector
The openit_logfilecollector binary reads new lines in the debug logs and collects all existing data in the first scan. After the initial collection, the collector uses a bookmark to identify log file rotation and avoid rereading older data.
Debug log data is collected raw (all the read lines kept as is) and wrapped with meta information. The collection interval is 6 hours by default but is configurable.
The same is true for sample-based collection. The primary data is transferred to the server every night according to the client's timezone, and the data transfer interval is configurable.
Data Processing
Data collected from sample-based and log file-based undergoes preprocessing on the client side using the Open iT parser binaries. These binaries convert raw data into Open iT data, which arrives in the Core Server's staging directory called incoming. The server determines where the data is targeted based on their file names. The data files come with the following formats:
archiver-*.indata-*.instatus-*.in
The archiver files contain the collected license data and other data from the client that will be the input to the archiver binary for data archiving. Data files are the output of the filtrator binary and input to the check_and_split_data program for database creation. The License Monitor application uses the status files.
Open iT database, including the archive, is a file-based repository listing the compressed data by source and date. The archiver binary reads the file with the archiver staging directory and stores the data in the appropriate archive location.
After moving the data to the archive, the archiver creates a job list for the jobcreator, which creates jobs to produce the appropriate reporting data types. One job is created per day per data type, and the data types created are configurable.
The loadmanager program is a simple batch scheduler responsible for running the processing jobs with the filtrator binary. It controls the jobs created by the jobcreator. The filtrator is the central processing program that converts data from archive formats to reportable data. Then, the output from the filtrator is moved to the incoming staging directory for database loading.
Database Loading
The first step in the loading process, check_and_split_data, involves validating and moving data to the correct database directories. The Open iT database is a file-based repository organized into a year/month/day directory structure.
After the validation and insertion, the add and summasummarum binaries aggregate the data into the following time chunks:
- Day
- Week
- Month
- Year
The Open iT Core Reporter uses the aggregated data, while the Analysis Server requires additional steps for loading.
Analysis Server Data Loading
The Analysis Server connects to the Core Server database directory to process configured data types and load them to the SQL Server. After loading, a command runs to process and normalize the SQL data warehouse. The Analysis Server then runs a ProcessOlap command to load data from the SQL Server data warehouse to the cubes.
After these, you can generate reports from the Analysis Server's Analysis tab, SQL Server Reporting Server (SSRS), or Excel dashboards.
Raw Archive Regeneration
On the client's side, the collector programs parse the collected raw data, convert them to Open iT data, and send it to the server for further processing. A problem arises when faulty parsing occurs.
The Raw Archive Regenerator is a program that converts raw data into Open iT data on the server's side. It allows fixing the corresponding parser and regenerating raw data to the correct Open iT data.