Appendix

Installing Internet Information Services on Microsoft Windows Server 2016 / 2019 / 2022

1. Install the role “Internet Information Services (IIS)” using the Server Manager and click on “Add Roles”.
2. Choose “Web Server (IIS)” and click on “Next “
   
3. Enable all flags for Security (for details see prerequisites listed in our SPI) and set the flag for “Dynamic Content Compression”.
   
   Click “Next” and finish the installation.

Debugging and Logging

Microsoft Windows

On Microsoft Windows systems the Fabasoft app.telemetry agent and server components and the Software-Telemetry library will log important or critical events to the Microsoft Windows event log “Application and Services Logs\Fabasoft app.telemetry”.

Linux

On Linux systems the Fabasoft app.telemetry agent and server components and the Software-Telemetry library will log important or critical events to the Linux syslog daemon.

The messages are logged with the application name (“app.telemetry”) and daemon process name (apptelemetryagent or apptelemetryserver).

The default logging location is /var/log/messages or the system journal depending on your system logging setup.

The used log level of Fabasoft app.telemetry daemons can be changed in the configuration file of the service: /etc/app.telemetry/<service>.conf

The syntax for the configuration of the used log level is the following:

The default log level is LOG_WARNING which means that all events of this and higher levels (ERR, CRIT, ALERT, EMERG) are logged.

The available log levels are:

• LOG_EMERG (0)
• LOG_ALERT (1)
• LOG_CRIT (2)
• LOG_ERR (3)
• LOG_WARNING (4)
• LOG_NOTICE (5)
• LOG_INFO (6)
• LOG_DEBUG (7)

Additional to the logging of important process events to the syslog daemon the app.telemetry Linux processes can optionally be configured to write much more process trace output for debugging purpose. This ProcessOutFile can be configured in the configuration file of the desired process:

Fabasoft app.telemetry Client - Logging

When detecting any troubles using the Fabasoft app.telemetry web browser client with your desired web browser, first check if that web browser is supported by Fabasoft app.telemetry.

One step further for problem solving is using debugging tools for your web browser and watch the debugging console output or the network requests.

Special Configuration Parameters

Configuration of Listening Ports for Fabasoft app.telemetry Agent/Server

If special environmental conditions require to use different listening ports for your Fabasoft app.telemetry agents (default = 10001) and server (default = 10000) change the default port numbers to your desired port numbers.

After changing the value for the Fabasoft app.telemetry agent restart the agent service (and update the agent configuration in the app.telemetry infrastructure by means of using the web client) and after changing the value for the Fabasoft app.telemetry server restart the server and the web server services.

Configuration of Listening Ports on Linux

On Linux systems change the ports in the configuration files mentioned below and restart the agent/server daemons.

For the Fabasoft app.telemetry server port the Apache web server also has to be restarted after the changes have been saved.

Configuration of Listening Ports on Microsoft Windows

On Microsoft Windows systems change the ports in the Microsoft Windows registry in the keys mentioned below and restart the agent/server services.

Start the Microsoft Windows Registry Editor and create the following registry keys:

Create new DWORD-values with the name “ListenPort” and the desired port number (entered as decimal value!):

Configuration of Software-Telemetry Data Directory (Server)

The Fabasoft app.telemetry server stores the Software-Telemetry data files required for deep analysis of Software-Telemetry requests (detailed data view) on the server’s hard disk.

These data files require (based on the infrastructure size and level of permanent Software-Telemetry) a medium or big amount of space on the disk. The files are stored in separate folders for each day, so that they can be deleted after some time. A later use of these files by means of import or restore from a backup is not supported by the Fabasoft app.telemetry server.

Configuration of Software-Telemetry Data Directory on Linux

On Linux systems change the target path for storing the Software-Telemetry data in the server configuration file mentioned below and restart the server daemon.

The default location of these files/directories on Linux systems is /var/opt/app.telemetry/server/telemetry.

Configuration of Software-Telemetry Data Directory on Microsoft Windows

On Microsoft Windows systems change the target path for storing the Software-Telemetry data in the Microsoft Windows registry in the key mentioned below and restart the server service afterwards.

Create new “String”-value below the “Server” registry key with the name “SoftwareTelemetryDataPath” and the desired directory path:

The default location of these files/directories on Microsoft Windows systems is located in the “Application Data” directory in the sub path “Fabasoft app.telemetry\server\telemetry”.

For example on Microsoft Windows Server: C:\ProgramData\Fabasoft app.telemetry\server\telemetry.

Configuration of Software-Telemetry Session Export Directory (Server)

Feedback sessions or reported Software-Telemetry sessions are normally stored within the Software-Telemetry raw data therefore the session/feedback details will not be available any longer if the raw data directories of the corresponding day was deleted (e.g. if raw data cleanup after x days is configured).

The Fabasoft app.telemetry server can be configured to export Software-Telemetry sessions automatically to a file system directory on the server’s hard disk. The responsible server setting is called “SoftwareTelemetrySessionPath” and can be configured depending on the target system as follows:

On Linux systems set the configuration parameter in the server configuration file as mentioned below and restart the server daemon.

On Microsoft Windows systems add the configuration parameter as new Microsoft Windows registry key (“String”-value) with the desired directory path:

This feature is enabled by default using a default path beside the rawdata directory on the server (e.g. /var/opt/app.telemetry/server/sessions). You can disable this automatic session export by setting an empty value for the configuration key mentioned above.

Note: Software-Telemetry sessions will only exported once. Any old available sessions will be exported after the first restart of the app.telemetry server after the export directory was configured. Any new sessions will be exported after they are fully processed by the server to the directory configured at that time.

Configuration of Disk Cache and Memory Buffer for the Agent

The Fabasoft app.telemetry agent stores the Software-Telemetry data sent by any instrumented application temporary in memory or on the hard disk (as configured) until the app.telemetry server fetches that data to process and persist it on the server’s hard disk.

When the server is not able to fetch the data (e.g. is too slow, or the network connection is down) the agent’s memory consumption will increase until a defined limit is reached. After that limit is reached the telemetry data will be written to the hard disk in a temporary folder which can also be limited with a maximum size limit.

With Fabasoft app.telemetry 2011 Spring Release an additional 3rd level cache has been implemented to provide a method to store request data on a remote medium in case the Fabasoft app.telemetry Server is not able to collect telemetry data for a longer period of time.

The following configuration parameters exist and can be used to change the default values:

• TelemetryDiskCacheDirectory: defines the data directory where the agent stores temporary data.
• TelemetryDiskCacheSize: defines the maximum disk cache size limit in megabytes (MB)
  • Minimum: 100 MB
  • Maximum: 100 GB (102400)
  • Default: 1 GB (1024)
• TelemetryBufferSize: defines the maximum agent memory buffer size limit in megabytes (MB)
  • Minimum: 20 MB
  • Maximum: 1 GB (1024)
  • Default: 100 MB
  • Note: this value is only an approximate value limiting the telemetry buffer in the agent process, the real memory consumption of the agent process may be higher (up to twice the defined limit)
• TelemetryExtendedDiskCacheDirectory: The directory path where the extended 3rd level cache should be stored. Default this value is not set and therefore not used.

The configuration of these parameters can be set on Microsoft Windows systems in the Windows registry and on Linux systems in the agent configuration file. A change of any parameter requires the agent process to be restarted.

How caches are written:

• Software-Telemetry data blocks received from individual processes are copied from the shared memory to the agent process memory and stored in a "per process" queue.
• The queuing thread fetches the data block from "per process" queues and apply compression. As compression is CPU bound this thread may utilize one CPU. On heavy data load compression may limit data throughput and agent memory consumption may raise.
• If the TelemetryBufferSize limit is reached, data blocks will be written to the disk cache (in TelemetryDiskCacheDirectory up to TelemetryDiskCacheSize MB).
• If the TelemetryBufferSize limit is reached and the disk cache is disabled or full, data blocks will be written to the extended disk cache in TelemetryExtendedDiskCacheDirectory. Each file is written continuously from memory cache, so the cache file size is at least TelemetryBufferSize MB but maximum 100MB in size.
• If the TelemetryBufferSize limit is reached and the disk cache is disabled or full and no extended disk cache is configured or the extended disk cache is not accessible, data will be dropped.

How caches are read:

• If data is stored in the standard disk cache, this data will be transferred to the server first.
• If the disk cache is empty, the content of the extended disk cache will be passed to the server file by file.
• If the disk cache is empty and no extended disk cache files are present the data will be fetched from the memory cache.

Configuration of Disk Cache and Memory Buffer for the Agent on Linux

On Linux systems the temporary data settings can be modified with the agent configuration file located at /etc/app.telemetry/agent.conf

The default agent temporary data directory is /var/opt/app.telemetry/agent/telemetry.

Specify the size limits with integer numbers in megabytes (MB).

Changes of any parameter will not take effect until the agent process is restarted.

Configuration of Disk Cache and Memory Buffer for the Agent on Microsoft Windows

On Microsoft Windows systems the temporary data settings can be modified in the Microsoft Windows registry in the key mentioned below.

Navigate to or create registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent

Create a new “String”-value for every of the following entries.

Specify the size limits with integer numbers in megabytes (MB) - (registry value of type String).

The default agent temporary data directory is <ProgramData>/Fabasoft app.telemetry/agent/telemetry.

Changes of any parameter will not take effect until the agent process is restarted.

Configuration of Database Rollforward Logs on the app.telemetry Server

To reduce the possibility of a data loss when the database storing request data is not available or when the Fabasoft app.telemetry Server has been stopped, a log containing all processed requests will be persisted on the file system. When the Fabasoft app.telemetry server restarts, all files will be processed to persist pending requests. Requests that cannot be written to the database will be copied into additional files, which can be reprocessed when the failure situation has been resolved.

To turn on the “Database Rollforward Log” you have to specify a folder, where the log files will be written to. For each log pool a subdirectory "LogFile" + database table prefix will be created. Each log file is named "logfile" + <timestamp> + ".dat" and may contain up to 10000 Requests. A log file will be deleted, when each requests has been either successfully committed to the database or has been copied to a "skipfile". A "skipfile" is named "skipfile" + <timestamp> + ".dat" and contains records that could not be written to the database. To retry writing records from a skipfile to the database, the "skipfile" must be renamed to "logfile" and the file will be reprocessed when the app.telemetry server will be restarted.

When “Database Rollforward Logs” are active, it is not necessary to keep all pending requests in memory, so the memory consumption of the app.telemetry server process in case of a slow or unavailable database will not increase so fast.

Configuration of Database Rollforward Logs on Linux Systems

All configuration values can be defined in the app.telemetry server configuration file: /etc/app.telemetry/server.conf.

Define the properties and values as key-value pairs.

SoftwareTelemetryLogfilePath: The directory path where the directories for the log files will be created.

Configuration of Database Rollforward Logs on Microsoft Windows Systems

All configuration values can be defined as Microsoft Windows registry keys.

Create the desired registry value as “String”-value below the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server

SoftwareTelemetryLogfilePath: The directory path where the directories for the log files will be created.

Fabasoft app.telemetry with SELinux

The basic support for SELinux on RHEL/CentOS systems includes the following:

• The Fabasoft app.telemetry RPMs include the basic SELinux policy files.
• The files installed by Fabasoft app.telemetry RPMs are installed to /opt/app.telemetry/selinux and include the following file types:
  • <process>.fc: SELinux file context settings for <process>.
  • <process>.te: SELinux policy rules for <process>.
  • <process>.if: SELinux interface rules for <process>.
  • <process>.pp: precompiled SELinux policy module for <process> compiled for latest supported CentOS 7.x version (supported with Fabasoft app.telemetry server).
  • <process>.sh: Shell script helping to adapt SELinux policies for your environment.

The Fabasoft app.telemetry RPMs already include the basic SELinux configuration for running in a simple environment.

In some situations you may need to modify the app.telemetry SELinux policy for your systems. The following shell script may help you.

Syntax: <process>.sh [ --update | --rebuild | --uninstall | --skip-build ]

• --skip-build: update the SELinux policy module without rebuilding using the current policy file /opt/app.telemetry/selinux/<process>.pp and restore SELinux file contexts (FCs).
• --rebuild: rebuilding policy module based on local sources (.fc, .te, .if) and then reload new policy module and restore FCs.
• --update: check SELinux audit.log for any denies and ask to extend any missing rules. Then also rebuild, reload and restore FCs.
• --uninstall: uninstall the SELinux policy module for this <process>.

SELinux audit logs are normally logged to /var/log/audit/audit.log.

The Fabasoft app.telemetry server SELinux policy provides tunable SELinux Booleans for enabling or disabling security critical server features (default they are disabled):

• allow_apptelemetryserver_cmdlinenotifications: with this Boolean you can enable command line notifications for the server (running any shell script). Based on your command line notification script some more SELinux permissions may be required.

The current state of the SELinux Booleans can be checked with the command getsebool <name>.

To enable or disable these optional permissions use the following command: setsebool [-P] <name>=true|false. The optional –P flag makes the setting permanent (for reboots).

Warning: Be careful on changing any app.telemetry default settings (file system paths, network ports, etc.) to custom settings. This may break the default SELinux support and may require manual changing of the policy!

Using PostgreSQL Database for app.telemetry Server on Microsoft Windows

Since version 2013 Summer Release the app.telemetry Server also supports the PostgreSQL database on Microsoft Windows (to be able to use a free and unlimited database).

Installation:

1. Just install the PostgreSQL package for Microsoft Windows (Win x86-64) available from the website http://www.postgresql.org/: new window on your Fabasoft app.telemetry server.
2. Add the path of the libpq.dll (e.g. “C:\Program Files\PostgreSQL\9.6\bin”) to the system PATH variable.
3. Restart the app.telemetry server service.

Configuration:

1. Create a database and a database login role in PostgreSQL (e.g. using pgAdmin tool).
2. Add a “PostgreSQL Database Connection” below “Database Connections” using the edit view of the Fabasoft app.telemetry client. (The menu entry will only be available, if the Fabasoft app.telemetry server is able to successfully load the libpq library.)

The new database support has been tested with PostgreSQL version 9.6 but should work with all newer versions of PostgreSQL for Microsoft Windows (64 bit).

Configuration of PostgreSQL Database for app.telemetry Server on Linux

On Linux systems a PostgreSQL database can be used to persist data (like requests, Top-X reports, SLA data, etc.) of the app.telemetry server.

The Fabasoft app.telemetry server connects to the PostgreSQL database via network connection (TCP/IP), so the following PostgreSQL authentication settings are responsible for login checks:

• PostgreSQL database on Localhost: IPv4 local connections ... 127.0.0.1/32
• PostgreSQL database on remote host (remote Network IPv4)

The PostgreSQL authentication settings are managed by the pg_hba.conf configuration file from the PostgreSQL data directory (normally located at: /var/lib/pgsql/data).

There are two possibilities how to set up the PostgreSQL database authentication settings in order to use with the Fabasoft app.telemetry server:

• Trusted authentication (trust)
  • should only be set up for local connections - consider security issues
  • you do not have to specify a password within the app.telemetry client database connection properties
  • Test connection from the Linux shell by means of: psql -d testdb -U testuser
• Password authentication (password)
  • requires to set a password for the PostgreSQL database user
    • ALTER ROLE testuser WITH PASSWORD 'mypwd';
  • requires to specifiy the password within the app.telemetry client database connection properties
  • Test connection from the Linux shell by means of: psql -d testdb -U testuser -W and enter the defined password when prompted

Possible Configuration Problems:

• Password Authentication - but no password configured for database user
  • when using password authentication, ensure that you have set a password for your database user within the database management console (psql)
• Password Authentication - but no password specified for database user within app.telemetry client
• Authentication set for the wrong target
  • ensure that your authentication settings are specified for the correct network address (IPv4) - for local connections via the local network address host 127.0.0.1/32

Sample Configuration

• Configure PostgreSQL authentication settings by means of modifying pg_hba.conf:

• Setup the database: su postgres; psql
  • Create a new database user/role: CREATE ROLE testuser LOGIN;
  • Create a new database instance: CREATE DATABASE testdb OWNER testuser;
  • Set the password for the database user: ALTER ROLE testuser WITH PASSWORD 'mypwd';
• Test database connection (you should exit from the previous psql session (\q) and from previous su postgres environment (exit))
  • psql -d testdb -U testuser -W
• Start the Fabasoft app.telemetry web browser client and switch to edit view
  • create a new database connection and enter the database properties
    • Database Server = localhost
    • Database Port = 5432
    • Database Login User Name = testuser
    • Password for Login = mypwd
    • Database = testdb
  • Click the "Test" button to check the configuration

Database Cleanup using Partitioned Tables

Using PostgreSQL Version 11 and later, with Fabasoft app.telemetry 2020 a cleanup feature was introduced that allows automatic cleanup based on “Partitioned Tables”. That feature can be activated per “Database Connection”.

When selected, all Log Pool, Statistics and Counter Value tables are created as “Partitioned Tables” with one partition per day. When the selected data retention period is reached, the respective data partition is dropped. As this operation is quite fast in comparison to deleting all records using DELETE statements is enables automatic data cleanup for high data rates, where DELETE statements are too slow to perform the cleanup.

There is no migration process supported for old tables. Therefore, it is not possible to automatically migrate log pool and counter data to partitioned tables. It is recommended to create a new database for using partitioned tables, create a dedicated “Database Connection” for using “Partitioned Tables” and migrate the log pools and the default database to the new database connection.

If needed, you can manually migrate existing data to the partitioned tables by creating the respective partitions and by inserting data using database export/import tools.

Installing Fabasoft app.telemetry on RHEL/CentOS 8 and 9

With Fabasoft app.telemetry 2021 we have introduced full support for RHEL/CentOS 8, since Fabasoft app.telemetry 2024 also RHEL 9 is supported so these platforms can be used as app.telemetry agent platform (incl. WebAPI or Apache module) or to install the whole app.telemetry server on this platform or even to collect syslog messages by means of using the app.telemetry syslog forwarder.

Note: RHEL/CentOS 8 and 9 is using the systemd tools to manage running services therefore other commands have to be used to start, stop and manage the app.telemetry daemons.

Start/Stop/Status of Daemons

On RHEL/CentOS 8 and 9 you have to use the systemctl command to start or stop a daemon or the get the current status.

Enable Autostart of Daemons

The systemctl command can also be used to enable or disable automatic startup of a service.

Note: The app.telemetry services will be enabled by default, but not started, during the RPM installation.

In order to get a full list of supported systemctl commands see the man page (man systemctl).

Customize Startup Parameters (Core Dump Settings)

The default startup settings (overwritten on upgrades) for the app.telemetry daemons are located in /usr/lib/systemd/system/<name>.service and define the basic startup parameters like automatic restart on failure.

Custom startup parameters can be defined by adding your own configuration file to the directory /etc/systemd/system/<name>.service.d/. These custom files will not be overwritten on RPM upgrades.

For example to turn on core dumps for a crashing app.telemetry agent process add following file:

The Location where those core dumps are written can be customized in the configuration file /etc/sysctl.conf:

Note: all app.telemetry services use a systemd feature called PrivateTmp which means that /tmp as seen by the service is not the same as the /tmp seen by any other process thus /tmp can’t be used.

Running Fabasoft app.telemetry services without root context

In order to provide better security, it is possible to run the Fabasoft app.telemetry services in a different user context than the root user. Note: the Fabasoft app.telemetry server does require different steps than the agent.

Fabasoft app.telemetry server settings

In order to run the Fabasoft app.telemetry server services in a different user context, following steps are necessary.

Create a service user and group using groupadd and useradd.

Create a configuration file to tell systemd to use the specific user. Place this file either directly under the particular configuration directory in /etc/systemd/system/<name>.service.d or create a configurationfile elsewhere and place a symbolic link to this file into the /etc/systemd/system/<name>.service.d directory.

The default location of the local socket used for the telemetry transport is /var/run/apptelemetryagent.sock, which is restricted to root users. Since Version 2020 (Hotfix Version) you can change this port name in the /etc/app.telemetry/agent.conf.

For the Softwaretelemetry library to use this port, you have to set the APM_TRANSPORT environment variable in each instrumented process. All app.telemetry services are instrumented, so you have to set APM_TRANSPORT accordingly.

Create links from the systemd configuration directories to the service user configuration.

On the app.telemetry Server you have to repeat that for each service.

• apptelemetryconfiguration.service
• apptelemetryecomm.service
• apptelemetryserver.service
• apptelemetrysync.service
• apptelemetryworker.service

Finally, the rights for the Fabasoft app.telemetry directory structure has to be changed to the newly added user rights. See the following settings:

On the app.telemetry Server you have to restore the access to some files for the webservice:

After all configuration steps run systemctl daemon-reload and restart all Fabasoft app.telemetry services.

Fabasoft Folio Web Services

When changing the service use of the app.telemetry agent service on a Fabasoft Folio Webserver instance, the local environment settings of all webserver instances have to be modified to specify the telemetry socket or port. For every webserver instance a directory exists at /var/opt/fabasoft/instances/WebServer_10*/env/. Create the environment variable APM_TRANSPORT as a file there and set the content according to the app.telemetry agent configuration (e.g. tcp://localhost:10002 or socket://@apptelemetryagent.sock). Afterwards the rights for this environment variable have to be set to fscsrv:fsc and on active running SELinux run restorecon on the new environment file as well.
Restart all Fabasoft Folio Webservices using fscmgmt restart <ID Webservices>.

More Information on the TCP socket can be found at the Fabasoft app.telemetry knowledge base: https://help.apptelemetry.fabasoft.com/index.php?topic=doc/Fabasoft-apptelemetry-Knowledge-Base/feature-details.htm#tcp-transport-for-agent-library-communications: new window

SELinux

When working with SELinux the basic SELinux policy files need to be modified. The changed TelemetrySocket location has to be included into the agents context settings. Like described in , the SELinux configuration files can be found at /opt/app.telemetry/selinux. In the apptelemetryagent.fc file change the preconfigured path /var/run/apptelemetryagent\.sock to your preferred location, for example:

Now rebuild with the included apptelemetryagent.sh file:

TelemetrySocket location as directory in /var/run

The example above demonstrates that it is possible to use a subfolder inside /var/run as your new TelemetrySocket location. Due to cleanup inside this directory when rebooting, it is necessary to configure this subfolder as RuntimeDirectory in the agents service configuration. Modify your conf-file inside /etc/systemd/system/apptelemetryagent.service.d/:

When working with httpd also change the configuration inside /usr/lib/systemd/system/httpd.service.d by setting APM_TRANSPORT to support the new socket location.