On Microsoft Windows systems the Fabasoft app.telemetry agent and server components and the Software-Telemetry library will log important or critical events to the Microsoft Windows event log “Application and Services Logs\Fabasoft app.telemetry”.
On Linux systems the Fabasoft app.telemetry agent and server components and the Software-Telemetry library will log important or critical events to the Linux syslog daemon.
The messages are logged with the application name (“app.telemetry”) and daemon process name (apptelemetryagent or apptelemetryserver).
The default logging location is /var/log/messages or the system journal depending on your system logging setup.
The used log level of Fabasoft app.telemetry daemons can be changed in the configuration file of the service: /etc/app.telemetry/<service>.conf
The syntax for the configuration of the used log level is the following:
Log Level Configuration:
# Default setting is:
The default log level is LOG_WARNING which means that all events of this and higher levels (ERR, CRIT, ALERT, EMERG) are logged.
The available log levels are:
Additional to the logging of important process events to the syslog daemon the app.telemetry Linux processes can optionally be configured to write much more process trace output for debugging purpose. This ProcessOutFile can be configured in the configuration file of the desired process:
Example configuration for apptelemetryserver: /etc/app.telemetry/server.conf
# The ProcessOutFile property defines a file where the daemons debugging output should be written.
# This property is disabled by default.
# The default installation contains a logrotate config for:
When detecting any troubles using the Fabasoft app.telemetry web browser client with your desired web browser, first check if that web browser is supported by Fabasoft app.telemetry.
One step further for problem solving is using debugging tools for your web browser and watch the debugging console output or the network requests.
If special environmental conditions require to use different listening ports for your Fabasoft app.telemetry agents (default = 10001) and server (default = 10000) change the default port numbers to your desired port numbers.
After changing the value for the Fabasoft app.telemetry agent restart the agent service (and update the agent configuration in the app.telemetry infrastructure by means of using the web client) and after changing the value for the Fabasoft app.telemetry server restart the server and the web server services.
On Linux systems change the ports in the configuration files mentioned below and restart the agent/server daemons.
app.telemetry Server Port Configuration (/etc/app.telemetry/server.conf)
app.telemetry Agent Port Configuration (/etc/app.telemetry/agent.conf)
For the Fabasoft app.telemetry server port the Apache web server also has to be restarted after the changes have been saved.
On Microsoft Windows systems change the ports in the Microsoft Windows registry in the keys mentioned below and restart the agent/server services.
Start the Microsoft Windows Registry Editor and create the following registry keys:
app.telemetry Registry Keys
Create new DWORD-values with the name “ListenPort” and the desired port number (entered as decimal value!):
app.telemetry Registry Keys for Listening Ports
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\ListenPort = 10001
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Configuration\ListenPort = 10006
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Ecomm\ListenPort = 10003
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server\ListenPort = 10000
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Sync\ListenPort = 10007
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Worker\ListenPort = 10004
The Fabasoft app.telemetry server stores the Software-Telemetry data files required for deep analysis of Software-Telemetry requests (detailed data view) on the server’s hard disk.
These data files require (based on the infrastructure size and level of permanent Software-Telemetry) a medium or big amount of space on the disk. The files are stored in separate folders for each day, so that they can be deleted after some time. A later use of these files by means of import or restore from a backup is not supported by the Fabasoft app.telemetry server.
On Linux systems change the target path for storing the Software-Telemetry data in the server configuration file mentioned below and restart the server daemon.
Software-Telemetry Data Path Configuration (/etc/app.telemetry/server.conf)
The default location of these files/directories on Linux systems is /var/opt/app.telemetry/server/telemetry.
On Microsoft Windows systems change the target path for storing the Software-Telemetry data in the Microsoft Windows registry in the key mentioned below and restart the server service afterwards.
Create new “String”-value below the “Server” registry key with the name “SoftwareTelemetryDataPath” and the desired directory path:
Software-Telemetry Data Path Registry Key
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server\SoftwareTelemetryDataPath = “<your Path>”
The default location of these files/directories on Microsoft Windows systems is located in the “Application Data” directory in the sub path “Fabasoft app.telemetry\server\telemetry”.
For example on Microsoft Windows Server 2016: C:\ProgramData\Fabasoft app.telemetry\server\telemetry.
Feedback sessions or reported Software-Telemetry sessions are normally stored within the Software-Telemetry raw data therefore the session/feedback details will not be available any longer if the raw data directories of the corresponding day was deleted (e.g. if raw data cleanup after x days is configured).
The Fabasoft app.telemetry server can be configured to export Software-Telemetry sessions automatically to a file system directory on the server’s hard disk. The responsible server setting is called “SoftwareTelemetrySessionPath” and can be configured depending on the target system as follows:
On Linux systems set the configuration parameter in the server configuration file as mentioned below and restart the server daemon.
Software-Telemetry Session Path Configuration (/etc/app.telemetry/server.conf)
On Microsoft Windows systems add the configuration parameter as new Microsoft Windows registry key (“String”-value) with the desired directory path:
Software-Telemetry Session Path Registry Key
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server\SoftwareTelemetrySessionPath = “<your Path>”
This feature is enabled by default using a default path beside the rawdata directory on the server (e.g. /var/opt/app.telemetry/server/sessions). You can disable this automatic session export by setting an empty value for the configuration key mentioned above.
Note: Software-Telemetry sessions will only exported once. Any old available sessions will be exported after the first restart of the app.telemetry server after the export directory was configured. Any new sessions will be exported after they are fully processed by the server to the directory configured at that time.
The Fabasoft app.telemetry agent stores the Software-Telemetry data sent by any instrumented application temporary in memory or on the hard disk (as configured) until the app.telemetry server fetches that data to process and persist it on the server’s hard disk.
When the server is not able to fetch the data (e.g. is too slow, or the network connection is down) the agent’s memory consumption will increase until a defined limit is reached. After that limit is reached the telemetry data will be written to the hard disk in a temporary folder which can also be limited with a maximum size limit.
With Fabasoft app.telemetry 2011 Spring Release an additional 3rd level cache has been implemented to provide a method to store request data on a remote medium in case the Fabasoft app.telemetry Server is not able to collect telemetry data for a longer period of time.
The following configuration parameters exist and can be used to change the default values:
The configuration of these parameters can be set on Microsoft Windows systems in the Windows registry and on Linux systems in the agent configuration file. A change of any parameter requires the agent process to be restarted.
How caches are written:
How caches are read:
On Linux systems the temporary data settings can be modified with the agent configuration file located at /etc/app.telemetry/agent.conf
Example: Disk Cache and Memory Buffer Configuration (/etc/app.telemetry/agent.conf)
The default agent temporary data directory is /var/opt/app.telemetry/agent/telemetry.
Specify the size limits with integer numbers in megabytes (MB).
Changes of any parameter will not take effect until the agent process is restarted.
On Microsoft Windows systems the temporary data settings can be modified in the Microsoft Windows registry in the key mentioned below.
Navigate to or create registry key:
Create a new “String”-value for every of the following entries.
Specify the size limits with integer numbers in megabytes (MB) - (registry value of type String).
Example: Disk Cache and Memory Buffer Configuration Registry Keys
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\ TelemetryDiskCacheDirectory = “C:\...”
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\TelemetryDiskCacheSize = “1024”
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\TelemetryBufferSize = “100”
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Agent\TelemetryExtendedDiskCacheDirectory = “X:\remote-NAS\temp-storage\agent-01”
The default agent temporary data directory is <ProgramData>/Fabasoft app.telemetry/agent/telemetry.
Changes of any parameter will not take effect until the agent process is restarted.
To reduce the possibility of a data loss when the database storing request data is not available or when the Fabasoft app.telemetry Server has been stopped, a log containing all processed requests will be persisted on the file system. When the Fabasoft app.telemetry server restarts, all files will be processed to persist pending requests. Requests that cannot be written to the database will be copied into additional files, which can be reprocessed when the failure situation has been resolved.
To turn on the “Database Rollforward Log” you have to specify a folder, where the log files will be written to. For each log pool a subdirectory "LogFile" + database table prefix will be created. Each log file is named "logfile" + <timestamp> + ".dat" and may contain up to 10000 Requests. A log file will be deleted, when each requests has been either successfully committed to the database or has been copied to a "skipfile". A "skipfile" is named "skipfile" + <timestamp> + ".dat" and contains records that could not be written to the database. To retry writing records from a skipfile to the database, the "skipfile" must be renamed to "logfile" and the file will be reprocessed when the app.telemetry server will be restarted.
When “Database Rollforward Logs” are active, it is not necessary to keep all pending requests in memory, so the memory consumption of the app.telemetry server process in case of a slow or unavailable database will not increase so fast.
All configuration values can be defined in the app.telemetry server configuration file: /etc/app.telemetry/server.conf.
Define the properties and values as key-value pairs.
SoftwareTelemetryLogfilePath: The directory path where the directories for the log files will be created.
Example: Database Rollforward Log Configuration (/etc/app.telemetry/server.conf)
All configuration values can be defined as Microsoft Windows registry keys.
Create the desired registry value as “String”-value below the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server
SoftwareTelemetryLogfilePath: The directory path where the directories for the log files will be created.
Example: Database Rollforward Log Configuration Registry Key
HKEY_LOCAL_MACHINE\SOFTWARE\Fabasoft app.telemetry\Server\SoftwareTelemetryLogfilePath = “C:\data\logs\...”
The basic support for SELinux on RHEL/CentOS 7 systems includes the following:
The Fabasoft app.telemetry RHEL/CentOS 7 RPMs already include the basic SELinux configuration for running in a simple environment.
In some situations you may need to modify the app.telemetry SELinux policy for your systems. The following shell script may help you.
Syntax: <process>.sh [ --update | --rebuild | --uninstall | --skip-build ]
SELinux audit logs are normally logged to /var/log/audit/audit.log.
The Fabasoft app.telemetry server SELinux policy provides tunable SELinux Booleans for enabling or disabling security critical server features (default they are disabled):
The current state of the SELinux Booleans can be checked with the command getsebool <name>.
To enable or disable these optional permissions use the following command: setsebool [-P] <name>=true|false. The optional –P flag makes the setting permanent (for reboots).
Linux Shell Commands:
> setsebool -P allow_apptelemetryserver_cmdlinenotifications=true
> getsebool allow_apptelemetryserver_cmdlinenotifications
allow_apptelemetryserver_cmdlinenotifications --> on
Warning: Be careful on changing any app.telemetry default settings (file system paths, network ports, etc.) to custom settings. This may break the default SELinux support and may require manual changing of the policy!
Since version 2013 Summer Release the app.telemetry Server also supports the PostgreSQL database on Microsoft Windows (to be able to use a free and unlimited database).
The new database support has been tested with PostgreSQL version 9.6 but should work with all recent versions of PostgreSQL for Microsoft Windows (64 bit).
On Linux systems a PostgreSQL database can be used to persist data (like requests, Top-X reports, SLA data, etc.) of the app.telemetry server.
The Fabasoft app.telemetry server connects to the PostgreSQL database via network connection (TCP/IP), so the following PostgreSQL authentication settings are responsible for login checks:
The PostgreSQL authentication settings are managed by the pg_hba.conf configuration file from the PostgreSQL data directory (normally located at: /var/lib/pgsql/data).
There are two possibilities how to set up the PostgreSQL database authentication settings in order to use with the Fabasoft app.telemetry server:
Possible Configuration Problems:
Example Configuraion: /var/lib/pgsql/data/pg_hba.conf
# TYPE DATABASE USER CIDR-ADDRESS METHOD
# "local" is for Unix domain socket connections only (not used with Fabasoft app.telemetry Server)
# IPv4 local connections:
Using PostgreSQL Version 11 an later, Fabasoft app.telemetry 2020 implements a cleanup feature that allows automatic cleanup based on “Partitioned Tables”. That feature can be activated per “Database Connection”.
When selected, all Log Pool, Statistics and Counter Value tables are created as “Partitioned Tables” with one partition per day. When the selected data retention period is reached, the respective data partition is dropped. As this operation is quite fast in comparison to deleting all records using DELETE statements is enables automatic data cleanup for high data rates, where DELETE statements are too slow to perform the cleanup.
There is no migration process supported for old tables. Therefore, it is not possible to automatically migrate log pool and counter data to partitioned tables. It is recommended to create a new database for using partitioned tables, create a dedicated “Database Connection” for using “Partitioned Tables” and migrate the log pools and the default database to the new database connection.
If needed, you can manually migrate existing data to the partitioned tables by creating the respective partitions and by inserting data using database export/import tools.
With Fabasoft app.telemetry 2015 we have introduced full support for RHEL/CentOS 7, since Fabasoft app.telemetry 2021 also RHEL/CentOS 8 is supported so these platforms can be used as app.telemetry agent platform (incl. WebAPI or Apache module) or to install the whole app.telemetry server on this platform or even to collect syslog messages by means of using the app.telemetry syslog forwarder.
Note: RHEL/CentOS 7 and 8 is using the systemd tools to manage running services therefor other commands have to be used to start, stop and manage the app.telemetry daemons.
On RHEL/CentOS 7 you have to use the systemctl command to start or stop a daemon or the get the current status.
Example daemon service commands using systemctl
systemctl start|status|stop <name>.service
systemctl start apptelemetryagent.service
The systemctl command can also be used to enable or disable automatic startup of a service.
systemctl enable|disable <name>.service
systemctl disable apptelemetryagent.service
Note: The app.telemetry services will be enabled by default, but not started, during the RPM installation.
In order to get a full list of supported systemctl commands see the man page (man systemctl).
Similar to the daemon init scripts and their configuration files located in /etc/sysconfig/<daemonname> that gave you the possibility to customize startup parameters on RHEL/CentOS until version 6.x the systemd tools also provide similar configuration hooks.
The default startup settings (overwritten on upgrades) for the app.telemetry daemons are located in /usr/lib/systemd/system/<name>.service and define the basic startup parameters like automatic restart on failure.
Custom startup parameters can be defined by adding your own configuration file to the directory /etc/systemd/system/<name>.service.d/. These custom files will not be overwritten on RPM upgrades.
For example to turn on core dumps for a crashing app.telemetry agent process add following file:
The Location where those core dumps are written can be customized in the configuration file /etc/sysctl.conf:
# own core file pattern...
Note: all app.telemetry services use a systemd feature called PrivateTmp which means that /tmp as seen by the service is not the same as the /tmp seen by any other process thus /tmp can’t be used.
In order to provide better security, it is possible to run the Fabasoft app.telemetry services in a different user context than the root user. Note: the Fabasoft app.telemetry server does require different steps than the agent.
In order to run the Fabasoft app.telemetry server services in a different user context, following steps are necessary.
Create a service user and group using groupadd and useradd.
Examples for creating a service group and user
useradd apmsrv -g apmsrv
Create a configuration file to tell systemd to use the specific user. Place this file either directly uner the particular configuration directory in /etc/systemd/system/<name>.service.d or create a configurationfile elsewhere and place a symbolic link to this file into the /etc/systemd/system/<name>.service.d directory.
The default location of the local socket used for the telemetry transport is /var/run/apptelemetryagent.sock, which is restricted to root users. Since Version 2020 (Hotfix Version) you can change this port name in the /etc/app.telemetry/agent.conf.
Sample socket configuration in /etc/app.telemetry/agent.conf
# TelemetrySocket: The app.telemetry library (native-C) can communicate with the agent via UNIX domain socket.
# The default location is /var/run/apptelemetryagent.sock which the app.telemetry library connects to by default.
# Make sure to configure APM_TRANSPORT=socket://<TelemetrySocket> in all instrumented applications.
# For sockets in the abstract namespace use a leading @ (e.g. TelemetrySocket @apptelemetryagent.sock).
For the Softwaretelemetry library to use this port, you have to set the APM_TRANSPORT environment variable in each instrumented process. All app.telemetry services are instrumented, so you have to set APM_TRANSPORT accordingly.
Sample service user configuration in /etc/app.telemetry/apptelemetryserviceuser.conf
Create links from the systemd configuration directories to the service user configuration.
Examples for creating a link to the configuration file
ln -s /etc/app.telemetry/apptelemetryserviceuser.conf /etc/systemd/system/apptelemetryagent.service.d/apptelemetryserviceuser.conf
On the app.telemetry Server you have to repeat that for each service.
Finally, the rights for the Fabasoft app.telemetry directory structure has to be changed to the newly added user rights. See the following settings:
Change the access rights to the service user
chown -R ampsrv:apmsrv /etc/app.telemetry /var/opt/app.telemetry
On the app.telemetry Server you have to restore the access to some files for the webservice:
Revert the access rights of some files to the webservice user
chown apache:apache /etc/app.telemetry/webserver/cli_certificate.pem
chown apache:apache /etc/app.telemetry/htgroup
chown apache:apache /etc/app.telemetry/htpasswd
After all configuration steps run systemctl daemon-reload and restart all Fabasoft app.telemetry services.
When changing the service use of the app.telemetry agent service on a Fabasoft Folio Webserver instance, the local environment settings of all webserver instances have to be modified to specify the telemetry socket or port. For every webserver instance a directory exists at /var/opt/Fabasoft/instances/WebServer_10*/env/. Create the environment variable APM_TRANSPORT as a file there and set the content according to the app.telemetry agent configuration (e.g. tcp://localhost:10002 or socket://@apptelemetryagent.sock). Afterwards the rights for this environment variable have to be set to fscsrv:fsc and on active running SELinux run restorecon on the new environment file as well.
Restart all Fabasoft Folio Webservices using fscmgmt restart <ID Webservices>.