2018 Update Rollup 2

AppendixPermanent link for this heading

Installing Internet Information Services on Microsoft Windows Server 2008 / 2012 / 2016Permanent link for this heading

  1. Install the role “Internet Information Services (IIS)” using the Server Manager and click on “Add Roles”.
  2. Choose “Web Server (IIS)” and click on “Next
  3. Enable all flags for Security (for details see prerequisites listed in our SPI) and set the flag for “Dynamic Content Compression”.

    Click “Next” and finish the installation.

Debugging and LoggingPermanent link for this heading

Microsoft WindowsPermanent link for this heading

On Microsoft Windows systems the Fabasoft app.telemetry agent and server components and the Software-Telemetry library will log important or critical events to the Microsoft Windows event log “Application and Services Logs\Fabasoft app.telemetry”.

LinuxPermanent link for this heading

On Linux systems the Fabasoft app.telemetry agent and server components and the Software-Telemetry library will log important or critical events to the Linux syslog daemon.

The messages are logged with the application name (“app.telemetry”) and daemon process name (apptelemetryagent or apptelemetryserver).

The default logging location is /var/log/messages or the system journal depending on your system logging setup.

The used log level of Fabasoft app.telemetry daemons can be changed in the configuration file of the service: /etc/app.telemetry/<service>.conf

The syntax for the configuration of the used log level is the following:

Log Level Configuration:

LogLevel <level>

#   Default setting is:

LogLevel LOG_WARNING

The default log level is LOG_WARNING which means that all events of this and higher levels (ERR, CRIT, ALERT, EMERG) are logged.

The available log levels are:

  • LOG_EMERG (0)
  • LOG_ALERT (1)
  • LOG_CRIT (2)
  • LOG_ERR (3)
  • LOG_WARNING (4)
  • LOG_NOTICE (5)
  • LOG_INFO (6)
  • LOG_DEBUG (7)

Additional to the logging of important process events to the syslog daemon the app.telemetry Linux processes can optionally be configured to write much more process trace output for debugging purpose. This ProcessOutFile can be configured in the configuration file of the desired process:

Example configuration for apptelemetryserver: /etc/app.telemetry/server.conf

# The ProcessOutFile property defines a file where the daemons debugging output should be written.

# This property is disabled by default.

# The default installation contains a logrotate config for:
#    /v
ar/log/app.telemetry/apptelemetryserverd.log

ProcessOutFile /var/log/app.telemetry/apptelemetryserverd.log

Fabasoft app.telemetry Client - LoggingPermanent link for this heading

When detecting any troubles using the Fabasoft app.telemetry web browser client with your desired web browser, first check if that web browser is supported by Fabasoft app.telemetry.

One step further for problem solving is using debugging tools for your web browser and watch the debugging console output or the network requests.

Special Configuration ParametersPermanent link for this heading

Configuration of Listening Ports for Fabasoft app.telemetry Agent/ServerPermanent link for this heading

If special environmental conditions require to use different listening ports for your Fabasoft app.telemetry agents (default = 10001) and server (default = 10000) change the default port numbers to your desired port numbers.

After changing the value for the Fabasoft app.telemetry agent restart the agent service (and update the agent configuration in the app.telemetry infrastructure by means of using the web client) and after changing the value for the Fabasoft app.telemetry server restart the server and the web server services.

Configuration of Listening Ports on LinuxPermanent link for this heading

On Linux systems change the ports in the configuration files mentioned below and restart the agent/server daemons.

app.telemetry Server Port Configuration (/etc/app.telemetry/server.conf)

ListenPort 10000

app.telemetry Agent Port Configuration (/etc/app.telemetry/agent.conf)

ListenPort 10001

For the Fabasoft app.telemetry server port the Apache web server also has to be restarted after the changes have been saved.

Configuration of Listening Ports on Microsoft WindowsPermanent link for this heading

On Microsoft Windows systems change the ports in the Microsoft Windows registry in the keys mentioned below and restart the agent/server services.

Start the Microsoft Windows Registry Editor and create the following registry keys:

app.telemetry Registry Keys

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Configuration

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Ecomm

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Sync

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Worker

Create new DWORD-values with the name “ListenPort” and the desired port number (entered as decimal value!):

app.telemetry Registry Keys for Listening Ports

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\ListenPort = 10001

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Configuration\ListenPort = 10006

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Ecomm\ListenPort = 10003

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server\ListenPort = 10000

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Sync\ListenPort = 10007

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Worker\ListenPort = 10004

Configuration of Software-Telemetry Data Directory (Server)Permanent link for this heading

The Fabasoft app.telemetry server stores the Software-Telemetry data files required for deep analysis of Software-Telemetry requests (detailed data view) on the server’s hard disk.

These data files require (based on the infrastructure size and level of permanent Software-Telemetry) a medium or big amount of space on the disk. The files are stored in separate folders for each day, so that they can be deleted after some time. A later use of these files by means of import or restore from a backup is not supported by the Fabasoft app.telemetry server.

Configuration of Software-Telemetry Data Directory on LinuxPermanent link for this heading

On Linux systems change the target path for storing the Software-Telemetry data in the server configuration file mentioned below and restart the server daemon.

Software-Telemetry Data Path Configuration (/etc/app.telemetry/server.conf)

SoftwareTelemetryDataPath /var/opt/app.telemetry/server/telemetry

The default location of these files/directories on Linux systems is /var/opt/app.telemetry/server/telemetry.

Configuration of Software-Telemetry Data Directory on Microsoft WindowsPermanent link for this heading

On Microsoft Windows systems change the target path for storing the Software-Telemetry data in the Microsoft Windows registry in the key mentioned below and restart the server service afterwards.

Create new “String”-value below the “Server” registry key with the name “SoftwareTelemetryDataPath” and the desired directory path:

Software-Telemetry Data Path Registry Key

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server\SoftwareTelemetryDataPath = “<your Path>”

The default location of these files/directories on Microsoft Windows systems is located in the “Application Data” directory in the sub path “Fabasoft app.telemetry\server\telemetry”.

For example on Microsoft Windows Server 2016: C:\ProgramData\Fabasoft app.telemetry\server\telemetry.

Configuration of Software-Telemetry Session Export Directory (Server)Permanent link for this heading

Feedback sessions or reported Software-Telemetry sessions are normally stored within the Software-Telemetry raw data therefore the session/feedback details will not be available any longer if the raw data directories of the corresponding day was deleted (e.g. if raw data cleanup after x days is configured).

The Fabasoft app.telemetry server can be configured to export Software-Telemetry sessions automatically to a file system directory on the server’s hard disk. The responsible server setting is called “SoftwareTelemetrySessionPath” and can be configured depending on the target system as follows:

On Linux systems set the configuration parameter in the server configuration file as mentioned below and restart the server daemon.

Software-Telemetry Session Path Configuration (/etc/app.telemetry/server.conf)

SoftwareTelemetrySessionPath /var/opt/app.telemetry/server/session-export

On Microsoft Windows systems add the configuration parameter as new Microsoft Windows registry key (“String”-value) with the desired directory path:

Software-Telemetry Session Path Registry Key

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server\SoftwareTelemetrySessionPath = “<your Path>”

This feature is enabled by default using a default path beside the rawdata directory on the server (e.g. /var/opt/app.telemetry/server/sessions). You can disable this automatic session export by setting an empty value for the configuration key mentioned above.

Note: Software-Telemetry sessions will only exported once. Any old available sessions will be exported after the first restart of the app.telemetry server after the export directory was configured. Any new sessions will be exported after they are fully processed by the server to the directory configured at that time.

Configuration of Disk Cache and Memory Buffer for the AgentPermanent link for this heading

The Fabasoft app.telemetry agent stores the Software-Telemetry data sent by any instrumented application temporary in memory or on the hard disk (as configured) until the app.telemetry server fetches that data to process and persist it on the server’s hard disk.

When the server is not able to fetch the data (e.g. is too slow, or the network connection is down) the agent’s memory consumption will increase until a defined limit is reached. After that limit is reached the telemetry data will be written to the hard disk in a temporary folder which can also be limited with a maximum size limit.

With Fabasoft app.telemetry 2011 Spring Release an additional 3rd level cache has been implemented to provide a method to store request data on a remote medium in case the Fabasoft app.telemetry Server is not able to collect telemetry data for a longer period of time.

The following configuration parameters exist and can be used to change the default values:

  • TelemetryDiskCacheDirectory: defines the data directory where the agent stores temporary data.
  • TelemetryDiskCacheSize: defines the maximum disk cache size limit in megabytes (MB)
    • Minimum: 100 MB
    • Maximum: 100 GB (102400)
    • Default: 1 GB (1024)
  • TelemetryBufferSize: defines the maximum agent memory buffer size limit in megabytes (MB)
    • Minimum: 20 MB
    • Maximum: 1 GB (1024)
    • Default: 100 MB
    • Note: this value is only an approximate value limiting the telemetry buffer in the agent process, the real memory consumption of the agent process may be higher (up to twice the defined limit)
  • TelemetryExtendedDiskCacheDirectory: The directory path where the extended 3rd level cache should be stored. Default this value is not set and therefore not used.

The configuration of these parameters can be set on Microsoft Windows systems in the Windows registry and on Linux systems in the agent configuration file. A change of any parameter requires the agent process to be restarted.

How caches are written:

  • Software-Telemetry data blocks received from individual processes are copied from the shared memory to the agent process memory and stored in a "per process" queue.
  • The queuing thread fetches the data block from "per process" queues and apply compression. As compression is CPU bound this thread may utilize one CPU. On heavy data load compression may limit data throughput and agent memory consumption may raise.
  • If the TelemetryBufferSize limit is reached, data blocks will be written to the disk cache (in TelemetryDiskCacheDirectory up to TelemetryDiskCacheSize MB).
  • If the TelemetryBufferSize limit is reached and the disk cache is disabled or full, data blocks will be written to the extended disk cache in TelemetryExtendedDiskCacheDirectory. Each file is written continuously from memory cache, so the cache file size is at least TelemetryBufferSize MB but maximum 100MB in size.
  • If the TelemetryBufferSize limit is reached and the disk cache is disabled or full and no extended disk cache is configured or the extended disk cache is not accessible, data will be dropped.

How caches are read:

  • If data is stored in the standard disk cache, this data will be transferred to the server first.
  • If the disk cache is empty, the content of the extended disk cache will be passed to the server file by file.
  • If the disk cache is empty and no extended disk cache files are present the data will be fetched from the memory cache.

Configuration of Disk Cache and Memory Buffer for the Agent on LinuxPermanent link for this heading

On Linux systems the temporary data settings can be modified with the agent configuration file located at /etc/app.telemetry/agent.conf

Example: Disk Cache and Memory Buffer Configuration (/etc/app.telemetry/agent.conf)

TelemetryDiskCacheDirectory /var/opt/app.telemetry/agent/telemetry

TelemetryDiskCacheSize 1024

TelemetryBufferSize 100

TelemetryExtendedDiskCacheDirectory /remote-NAS/temp-storage/agent-01

The default agent temporary data directory is /var/opt/app.telemetry/agent/telemetry.

Specify the size limits with integer numbers in megabytes (MB).

Changes of any parameter will not take effect until the agent process is restarted.

Configuration of Disk Cache and Memory Buffer for the Agent on Microsoft WindowsPermanent link for this heading

On Microsoft Windows systems the temporary data settings can be modified in the Microsoft Windows registry in the key mentioned below.

Navigate to or create registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent

Create a new “String”-value for every of the following entries.

Specify the size limits with integer numbers in megabytes (MB) - (registry value of type String).

Example: Disk Cache and Memory Buffer Configuration Registry Keys

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\ TelemetryDiskCacheDirectory = “C:\...”

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\TelemetryDiskCacheSize = “1024”

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\TelemetryBufferSize = “100”

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\TelemetryExtendedDiskCacheDirectory = “X:\remote-NAS\temp-storage\agent-01”

The default agent temporary data directory is <ProgramData>/Fabasoft app.telemetry/agent/telemetry.

Changes of any parameter will not take effect until the agent process is restarted.

Configuration of Database Rollforward Logs on the app.telemetry ServerPermanent link for this heading

To reduce the possibility of a data loss when the database storing request data is not available or when the Fabasoft app.telemetry Server has been stopped, a log containing all processed requests will be persisted on the file system. When the Fabasoft app.telemetry server restarts, all files will be processed to persist pending requests. Requests that cannot be written to the database will be copied into additional files, which can be reprocessed when the failure situation has been resolved.

To turn on the “Database Rollforward Log” you have to specify a folder, where the log files will be written to. For each log pool a subdirectory "LogFile" + database table prefix will be created. Each log file is named "logfile" + <timestamp> + ".dat" and may contain up to 10000 Requests. A log file will be deleted, when each requests has been either successfully committed to the database or has been copied to a "skipfile". A "skipfile" is named "skipfile" + <timestamp> + ".dat" and contains records that could not be written to the database. To retry writing records from a skipfile to the database, the "skipfile" must be renamed to "logfile" and the file will be reprocessed when the app.telemetry server will be restarted.

When “Database Rollforward Logs” are active, it is not necessary to keep all pending requests in memory, so the memory consumption of the app.telemetry server process in case of a slow or unavailable database will not increase so fast.

Configuration of Database Rollforward Logs on Linux SystemsPermanent link for this heading

All configuration values can be defined in the app.telemetry server configuration file: /etc/app.telemetry/server.conf.

Define the properties and values as key-value pairs.

SoftwareTelemetryLogfilePath: The directory path where the directories for the log files will be created.

Example: Database Rollforward Log Configuration (/etc/app.telemetry/server.conf)

SoftwareTelemetryLogfilePath /var/logfiles...

Configuration of Database Rollforward Logs on Microsoft Windows SystemsPermanent link for this heading

All configuration values can be defined as Microsoft Windows registry keys.

Create the desired registry value as “String”-value below the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server

SoftwareTelemetryLogfilePath: The directory path where the directories for the log files will be created.

Example: Database Rollforward Log Configuration Registry Key

HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server\SoftwareTelemetryLogfilePath = “C:\data\logs\...”

Fabasoft app.telemetry with SELinuxPermanent link for this heading

The basic support for SELinux on RHEL/CentOS 7 systems includes the following:

  • The Fabasoft app.telemetry RPMs include the basic SELinux policy files.
  • The files installed by Fabasoft app.telemetry RPMs are installed to /opt/app.telemetry/selinux and include the following file types:
    • <process>.fc: SELinux file context settings for <process>.
    • <process>.te: SELinux policy rules for <process>.
    • <process>.if: SELinux interface rules for <process>.
    • <process>.pp: precompiled SELinux policy module for <process> compiled for latest supported CentOS 7.x version (supported with Fabasoft app.telemetry server).
    • <process>.sh: Shell script helping to adapt SELinux policies for your environment.

The Fabasoft app.telemetry RHEL/CentOS 7 RPMs already include the basic SELinux configuration for running in a simple environment.

In some situations you may need to modify the app.telemetry SELinux policy for your systems. The following shell script may help you.

Syntax: <process>.sh [ --update | --rebuild | --uninstall | --skip-build ]

  • --skip-build: update the SELinux policy module without rebuilding using the current policy file /opt/app.telemetry/selinux/<process>.pp and restore SELinux file contexts (FCs).
  • --rebuild: rebuilding policy module based on local sources (.fc, .te, .if) and then reload new policy module and restore FCs.
  • --update: check SELinux audit.log for any denies and ask to extend any missing rules. Then also rebuild, reload and restore FCs.
  • --uninstall: uninstall the SELinux policy module for this <process>.

SELinux audit logs are normally logged to /var/log/audit/audit.log.

The Fabasoft app.telemetry server SELinux policy provides tunable SELinux Booleans for enabling or disabling security critical server features (default they are disabled):

  • allow_apptelemetryserver_cmdlinenotifications: with this Boolean you can enable command line notifications for the server (running any shell script). Based on your command line notification script some more SELinux permissions may be required.

The current state of the SELinux Booleans can be checked with the command getsebool <name>.

To enable or disable these optional permissions use the following command: setsebool [-P] <name>=true|false. The optional –P flag makes the setting permanent (for reboots).

Linux Shell Commands:

> setsebool -P allow_apptelemetryserver_cmdlinenotifications=true

> getsebool allow_apptelemetryserver_cmdlinenotifications

    allow_apptelemetryserver_cmdlinenotifications --> on

Warning: Be careful on changing any app.telemetry default settings (file system paths, network ports, etc.) to custom settings. This may break the default SELinux support and may require manual changing of the policy!

Using PostgreSQL Database for app.telemetry Server on Microsoft WindowsPermanent link for this heading

Since version 2013 Summer Release the app.telemetry Server also supports the PostgreSQL database on Microsoft Windows (to be able to use a free and unlimited database).

Installation:

  1. Just install the PostgreSQL package for Microsoft Windows (Win x86-64) available from the website http://www.postgresql.org/ on your Fabasoft app.telemetry server.
  2. Add the path of the libpq.dll (e.g. “C:\Program Files\PostgreSQL\9.6\bin”) to the system PATH variable.
  3. Restart the app.telemetry server service.

Configuration:

  1. Create a database and a database login role in PostgreSQL (e.g. using pgAdmin tool).
  2. Add a “PostgreSQL Database Connection” below “Database Connections” using the edit view of the Fabasoft app.telemetry client. (The menu entry will only be available, if the Fabasoft app.telemetry server is able to successfully load the libpq library.)

The new database support has been tested with PostgreSQL version 9.6 but should work with all recent versions of PostgreSQL for Microsoft Windows (64 bit).

Configuration of PostgreSQL Database for app.telemetry Server on LinuxPermanent link for this heading

On Linux systems a PostgreSQL database can be used to persist data (like requests, Top-X reports, SLA data, etc.) of the app.telemetry server.

The Fabasoft app.telemetry server connects to the PostgreSQL database via network connection (TCP/IP), so the following PostgreSQL authentication settings are responsible for login checks:

  • PostgreSQL database on Localhost: IPv4 local connections ... 127.0.0.1/32
  • PostgreSQL database on remote host (remote Network IPv4)

The PostgreSQL authentication settings are managed by the pg_hba.conf configuration file from the PostgreSQL data directory (normally located at: /var/lib/pgsql/data).

There are two possibilities how to set up the PostgreSQL database authentication settings in order to use with the Fabasoft app.telemetry server:

  • Trusted authentication (trust)
    • should only be set up for local connections - consider security issues
    • you do not have to specify a password within the app.telemetry client database connection properties
    • Test connection from the Linux shell by means of: psql -d testdb -U testuser
  • Password authentication (password)
    • requires to set a password for the PostgreSQL database user
      • ALTER ROLE testuser WITH PASSWORD 'mypwd';
    • requires to specifiy the password within the app.telemetry client database connection properties
    • Test connection from the Linux shell by means of: psql -d testdb -U testuser -W and enter the defined password when prompted

Possible Configuration Problems:

  • Password Authentication - but no password configured for database user
    • when using password authentication, ensure that you have set a password for your database user within the database management console (psql)
  • Password Authentication - but no password specified for database user within app.telemetry client
  • Authentication set for the wrong target
    • ensure that your authentication settings are specified for the correct network address (IPv4) - for local connections via the local network address host 127.0.0.1/32

Sample Configuration

  • Configure PostgreSQL authentication settings by means of modifying pg_hba.conf:

Example Configuraion: /var/lib/pgsql/data/pg_hba.conf

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only (not used with Fabasoft app.telemetry Server)
local   all         all                               trust

# IPv4 local connections:
host    all         all         127.0.0.1/32          password

  • Setup the database: su postgres; psql
    • Create a new database user/role: CREATE ROLE testuser LOGIN;
    • Create a new database instance: CREATE DATABASE testdb OWNER testuser;
    • Set the password for the database user: ALTER ROLE testuser WITH PASSWORD 'mypwd';
  • Test database connection (you should exit from the previous psql session (\q) and from previous su postgres environment (exit))
    • psql -d testdb -U testuser -W
  • Start the Fabasoft app.telemetry web browser client and switch to edit view
    • create a new database connection and enter the database properties
      • Database Server = localhost
      • Database Port = 5432
      • Database Login User Name = testuser
      • Password for Login = mypwd
      • Database = testdb
    • Click the "Test" button to check the configuration

Installing Fabasoft app.telemetry on RHEL/CentOS 7Permanent link for this heading

With Fabasoft app.telemetry 2015 we have introduced full support for RHEL/CentOS 7 so this platform can be used as app.telemetry agent platform (incl. WebAPI or Apache module) or to install the whole app.telemetry server on this platform or even to collect syslog messages by means of using the app.telemetry syslog forwarder.

Note: RHEL/CentOS 7 is using the systemd tools to manage running services therefor other commands have to be used to start, stop and manage the app.telemetry daemons.

Start/Stop/Status of DaemonsPermanent link for this heading

On RHEL/CentOS 7 you have to use the systemctl command to start or stop a daemon or the get the current status.

Example daemon service commands using systemctl

systemctl start|status|stop <name>.service

systemctl start apptelemetryagent.service
systemctl status apptelemetryserver.service

systemctl stop apptelemetryworker.service

Enable Autostart of DaemonsPermanent link for this heading

The systemctl command can also be used to enable or disable automatic startup of a service.

Examples for enabling autostart using systemctl

systemctl enable|disable <name>.service

systemctl disable apptelemetryagent.service
systemctl enable apptelemetryserver.service

Note: The app.telemetry services will be enabled by default, but not started, during the RPM installation.

In order to get a full list of supported systemctl commands see the man page (man systemctl).

Customize Startup Parameters (Core Dump Settings)Permanent link for this heading

Similar to the daemon init scripts and their configuration files located in /etc/sysconfig/<daemonname> that gave you the possibility to customize startup parameters on RHEL/CentOS until version 6.x the systemd tools also provide similar configuration hooks.

The default startup settings (overwritten on upgrades) for the app.telemetry daemons are located in /usr/lib/systemd/system/<name>.service and define the basic startup parameters like automatic restart on failure.

Custom startup parameters can be defined by adding your own configuration file to the directory /etc/systemd/system/<name>.service.d/. These custom files will not be overwritten on RPM upgrades.

For example to turn on core dumps for a crashing app.telemetry agent process add following file:

/etc/systemd/system/apptelemetryagent.service.d/dumps.conf

[Service]

LimitCORE=infinity

The Location where those core dumps are written can be customized in the configuration file /etc/sysctl.conf:

/etc/sysctl.conf

# own core file pattern...

kernel.core_pattern=/var/dumps/core.%e.%p.%h.%t

Note: all app.telemetry services use a systemd feature called PrivateTmp which means that /tmp as seen by the service is not the same as the /tmp seen by any other process thus /tmp can’t be used.