Introduction

Introduction

Fabasoft app.telemetry provides a Software-Telemetry SDK that enables applications to integrate into Fabasoft app.telemetry. Applications can integrate by registering themselves into Software-Telemetry and by providing information (events) used within Software-Telemetry log pools and sessions. These events help you to track the control flow through a distributed environment of your application.

Application Registration

To register your application and the modules in an accurate way you have four standard layers of registration information:

• appName: the application name
• appId: an application ID (an instance or major version of the base application)
• appTierName: an application tier or module (e.g. to separate backend from frontend logic)
• appTierId: the application tier ID (e.g. the service instance # 2)

Use the four standard layers of registration information to structure your services in an adequate way, so your application is represented meaningful in the infrastructure view of the Fabasoft app.telemetry client. The appName and appId represent the id of a top level service group. Inside you may group services (appTierName) by service-type (e.g. web-service vs. database-service) or service-context (e.g. put services for sales and accounting in different groups) and use the appTierId to distinguish between the particular services.

As of Fabasoft app.telemetry version 18.3 it is possible to register additional properties to allow greater flexibility of application registrations.

You can see each of these registered applications in a separate process column in the Software-Telemetry analysis view. Put the registration on a position in the source code of your application that runs once at startup of the application. The combination of all four parameters should be unique per Server/virtual Server (Cluster).

Note: Do not use random information or unpredictable information like process id or start time for registration parameters as this information is used to register services in the infrastructure and random information will result in multiple registered services.

Module Registration

For each module inside your application you may register a separate module. Each of these modules is drawn in a separate column inside the Software-Telemetry analysis view. For these modules you can register event descriptions. Put the registration on a position in the source code of your module that runs once at creation of the module.

This figure shows the overview of a selected request of the Fabasoft app.telemetry Client where you can navigate into the event details.

This figure shows the event details of a selected request of the Fabasoft app.telemetry Client where you can see time usage of any single event, parameter values and context handling. To navigate deeper into the details use the "plus"-icon. You can see the current position in your request in the call stack view on the right side where you can also navigate to other positions.

Counter Registration

As of Fabasoft app.telemetry Version 2018 UR 3 (18.3) applications can also register counters of various types to provide additional performance and status information that is independent of requests.

Standard Modules

With SDK version 2009 Summer Release two standard module definitions and some general predefined events have been introduced. These modules define common instrumentation points for areas that are common to many projects.
Note: Do not use those predefined event-IDs listed below for your own events!

Operating System Module

The Operating System Module defines standard events for file system access, process execution and wait operations. These events should be used to instrument applications in a consistent way.
To use this module, call the RegisterModule-function or create an object of class APMModule with a module name equal to “Operating System” using the constant APM_MODULE_NAME_OS. The event descriptions are statically defined in the app.telemetry server process and may therefore not be registered within the application.

The constant names are prefixed with “APM_” for the C/C++ API (APM_<constant>) and in the other object-oriented languages they are members of the APM base class (APM.<constant>).

XMLHttpRequest

The XMLHttpRequest is defined to be used in an http client application to trace the synchronous or asynchronous events of a request.
The module has been used in the implementation of the SendXMLHttpRequest function in the JavaScript version of the SDK but may as well be used in your own application instrumentation.

The constant names are prefixed with “APM_” for the C/C++ API (APM_<constant>) and in the other object-oriented languages they are members of the APM base class (APM.<constant>).

HttpRequest

The HttpRequest Module is used to trace the parameters of http requests such as URL, method, status, header fields and message bodies.

The constant names are prefixed with “APM_” for the C/C++ API (APM_<constant>) and in the other object-oriented languages they are members of the APM base class (APM.<constant>).

Predefined Events

The SDK already provides a set of predefined events used for general purpose.

Note: Do not use any of these predefined event-IDs for your own event declarations!

Error/Trace Events

The constant names are prefixed with “APM_” for the C/C++ API (APM_<constant>) and in the other object-oriented languages they are members of the APM base class (APM.<constant>).

Use these standard events to mark error, warning or info conditions in your application like in the following example:

Events for Defining Sequence Hierarchies

The Sequence functionality has been deprecated and has been removed in Version 2015. Register your own events and log columns instead.

Predefined Basic Constants

Flags

Flags are used to specify the type of an event and are passed as argument to every event call.

The constant names are prefixed with “APM_” for the C/C++ API (APM_<constant>) and in the other object-oriented languages they are members of the APM base class (APM.<constant>).

Log Level

An event will only be logged if the event-level is lower or equal to the value selected in the session or log-definition.

The constant names are prefixed with “APM_” for the C/C++ API (APM_<constant>) and in the other object-oriented languages they are members of the APM base class (APM.<constant>).

Application Properties

Application Properties are static and unchanging properties used in addition to the four standard registration parameters when creating service objects in the app.telemetry infrastructure.

The constant names are prefixed with “APM_” for the C/C++ API (APM_<constant>) and in the other object-oriented languages they are members of the APM base class (APM.<constant>).

Application Values

Application values can provide additional, changeable (very low change frequency) information about an application or it’s runtime environment.

The constant names are prefixed with “APM_” for the C/C++ API (APM_<constant>) and in the other object-oriented languages they are members of the APM base class (APM.<constant>).

Request and Context Handling

After your application and all defined modules are registered you can start with the instrumentation of your program sequence.

When the request enters your application you have to create a Software-Telemetry context with a specific filter value. This filter value should be one of those you have registered for any module of this process with the RegisterFilterValue-method. By calling the CreateContext function with a filter value matching the filter value of a started Software-Telemetry session recording is enabled.

This context information is used by the analysis components to track the control flow through the distributed environment. Then you can subsequently fire events which are tracked by the Software-Telemetry. After firing the last event you have to call the method ReleaseContext to finish recording events and to tell app.telemetry that processing has finished.

Passing Context

If processing in your application involves other services (thread/process/module) you may track control and data flow between services by passing the Software-Telemetry context between the services.

With the GetContext-method you acquire a context token. Then you pass the token with your request data to the other service being invoked (for example by transmitting the context token over the network stream).

In the second service you use the token to attach to this context using the AttachContext-method. By doing this Software-Telemetry recording is being enabled in the called service and data will be displayed accordingly in the analysis view.

To pass the context token to an HTTP endpoint it must be set as a header with the name “x-apm-telemetry-context” and the value must be encoded using base64, this allows intermediaries such as load-balancers or reverse proxies to insert themselves in between the communicating parties to provide a more complete picture of the request flow. Instrumented HTTP endpoints must respond with a sync mark token as an http header with the name “x-apm-telemetry-syncmark” that is also base64 encoded for the same reason.

Synchronizing Timeline with SyncMarks

To get a correct timeline if the first service depends on the output of the second you have to use synchronization markers. Inside the second service you have to acquire a synchronization marker using the function GetSyncMark. After doing this you have to pass the synchronization marker together with the response back to the first service. Then you call the method SetSyncMark inside the first service. In the following figure you can see an illustration of the program flow.

Log Definitions

Each event that is recorded by any Fabasoft app.telemetry agent (when the instrumented application fires these events) is sent to the Fabasoft app.telemetry server, which is the responsible service to process and analyze these events.

The Fabasoft app.telemetry server requires a definition (analyzer structure) for interpreting and analyzing the Software-Telemetry events.

The developer who instruments the application is responsible to provide a log definition for his instrumentation points. A log definition is an XML file containing a description of events, their parameters with the data type and settings how to persist these events in a database for statistical analysis.

Application Specific Log Definitions (Analyzer Structure)

The app.telemetry Software-Telemetry analyzer module on the Fabasoft app.telemetry server requires at least one valid log definition to be defined. Otherwise no telemetry data will be processed.

The log definitions are normally delivered by the application vendor (by the developer who instrumented the application by means of Software-Telemetry SDK). These log definitions must be imported into the infrastructure model via the Fabasoft app.telemetry client by means of importing an XML file in the client interface in the edit mode.

Log Definition Syntax

Log definitions must have the following XML format:

The attributes of an entry are defined as follows:

• eventid: as defined in module implementation
• module: name of module as defined in module implementation
• paramid: ID of parameter <1..n> - starting at index 1 and the max value is the number of available parameters for this event. For block entries the paramid is set to 0, because block entries are virtual calculated values.
• name: the desired name for this log entry (will be used to display parameter column header and to persist in DB). This name must not contain any special characters and should not contain spaces – must be a valid database column name
• type: <0..6> … data type of entry definition – nearly the same as columndatatype
  • 0 … block start event type (has no data type)
  • 1..6: same as columndatatype (both type columns have to be equal for normal entries)
• tablename: Database table name where to store this entry. This name must not contain any special characters and spaces - must be a valid database table name.
• tabletype: <1|2> ... type of database table where to store entry
  • 1 ... defines the base table storing all relevant meta information for the request. (There should only be one table of tabletype 1 in a single APMLogAnalyzer definition, because all base table columns are stored for this tabletype)
  • 2 ... additional table for data that may occur more than once within a request (for example: database SQL query).
• columndatatype: <1..6> ...data type how to persist this entry in the DB
  • 1 ... Int32: 32 bit integer number
  • 2 ... Int64: 64 bit integer number
  • 3 ... String: textual value – if this data-type is set, the attribute columnlength must be set too.
  • 4 ... Duration: used for calculated duration between two events (when block is set) – unit in database = [100ns] (that means value 10000 in DB = 1ms); the Fabasoft app.telemetry Client displays the duration already formatted in milliseconds.
  • 5 ... Timestamp: 64 bit long value used for storing date/time information
  • 6 ... Double: 64 bit floating point number
• columnlength: length for database column definition (number of characters) – only required for data type 3 (string)
• format: optional application specific format definition
  for example: fsc:address, fsc:reference, fsc:name
• flags: bit-mask of additional flags:
  Specify the sum of the flags as a decimal value in the flags attribute.
  • 0x0004 ... start of block
  • 0x0008 ... end of block
  • calculation type (only one can be selected)
    • 0x0000 ... unique entry (first occurrence)
    • 0x0010 ... entry could occur multiple times – each event occurrence will be persisted (if used in combination with tabletype 2)
    • 0x0020 ... unique entry (last occurrence) - event will be overwritten each time a new instance of the same event is fired
    • 0x0030 … count (counts number of event occurrences)
    • 0x0040 … add (multiple values are added)
  • 0x0100 … the column is a dimension (valid only for integer and string type columns)
  • 0x0200 … the column is a measure (valid only for integer and duration type columns)
  • 0x0400 … the measure column is null-able, so a count column of not null values is calculated (valid only for measure columns)
  • 0x0800 … calculate minimum (valid only for measure columns)
  • 0x1000 … calculate maximum (valid only for measure columns)
  • generic column (based on calculated other column) ... all these generic column definitions require the parent-attribute to define the base column ("name") containing the base values for the new calculated column:
    • 0x10000 ... categorize value of other column: requires calculation and format attributes:
    •   calculation ... defines value split points
    •   format ... to define a textual representation of the classified values
    •     example to split the request duration into classes: parent="duration"   calculation="10000000;50000000;150000000" format="enum:1:0-1s;2:1-5s;3:5-15s;4:&gt;15s"
    • 0x20000 ... split value of other column (with regular expression): requires calculation attribute
    •   calculation ... defines the regular expression for splitting the base value into a new resulting value
    •     example to split up the only the URL path from a full URL containing additional URL arguments: parent="Page URL (referer)" calculation="[\w]+://[^/]*([^\?]+)" paramid="1" ... enclose the desired text part in the regex with grouping braces - define the desired group match number with the paramid-attribute (first/only 1 matching group ... paramid="1").
    • 0x40000 ... application property (since Version 21.1)
    •   parent ... the name of the application property or registration parameter (apm:appName, apm:appId, apm:appTierName, apm:appTierId, apm:hostname)
    •     example to get a column filled with the application tier id: parent="apm:appTierId".
    • 0x50000 ... calculate column based on expression: requires calculation attribute (since Version 24.1)
    •   calculation ... defines the SQL like expression for calculating the column
    •     example to get a processing time as request duration minus transfer time: calculation="duration – readtime - writetime".
  • Format options mask 0xF00000
    • 0x100000 ... value is shown as IP address
    • 0x200000 ... value is shown as URL
    • 0x300000 … text values of agent or service column are stored on database to provide texts when agent objects are deleted
  • 0x1000000 … the column is anonymized (valid only for integer and string type columns)
  • 0x2000000 … the column is used as quick filter option in the telemetry and research view
• block: optional definition of block of events – used to block two events to one together (to measure for example the time between)
  • use flag 4 as start event and flag 8 as end event and data type 4 (duration)

Explanation of example (above): this definition defines several different events of different types.

• Query (ID: 2003):
  • The SQL database query is recorded as event with ID “2003” in the application in the module “Fabasoft Components Kernel”.
  • The parameter will be named as “query” in the Software-Telemetry logs and in the database table “query”.
  • The database table type “2” defines to use an extra table which does only contain the query and a reference to the basic entry in the common table.
  • The data type of this entry is 3, which defines that the query is recorded as a string. The reserved field size in the database table is set to 4000 characters.
  • The flags value 16 means that each occurrence of this event is persisted in the extra database table “query”.
• Readtime (ID: 104):
  • The event with ID “104” of the module “Virtual Application” is handled as specific block, which means that this event is recorded as duration between two corresponding events.
  • For block entries the “paramid” is set to 0, because these entries are calculated entries with no direct value passed from the event.
  • The first entry defines the start of the block with type=”0” (irrelevant for start entry – defined in end entry) flags=”4” and the block was named “readvalues”.
  • The end entry (flags=”8”) also requires to have the same block name, so that the server knows that these two block definitions belong together. The end entry defines the final entry name “readtime” and the data type “4” (duration). This entry will be persisted in the base table “request” (type 1).

Defining a Sequence Hierarchy

The Sequence column functionality has been deprecated and has been removed in Version 2015. Register your own events and log columns instead.

Custom Telemetry Module Registration via Package

Since Fabasoft app.telemetry 2014 Summer Release it is possible to extend the module and event registration information done during instrumenting an application in the code by means of providing a custom “wizard” package containing special registration information:

• <app>cfg.zip
  • package.xml: only containing a packageid for your application and a version of the current package file

• module_registration.xml: this file provides custom color definitions for any of your application modules (if not defined a fallback color is used) and custom improved display names for special telemetry events and format definitions event parameters.

The module name must match with the registered application module in the telemetry module registration and you can define a specific color that should be used for that module everywhere in the app.telemetry client.

The rule set can be applied for multiple application instances running with different application names or ids just by specifying a list of valid application registration mapping (e.g. the application can register as “Fabasoft Folio” or “Fabasoft Cloud” and the above example will apply for requests from both applications.

You can now assign improved display names for any of your telemetry events identified inside the defined module with the event id. Note: every listed event in this package file will overwrite the original event registration. The following properties can be defined/overwritten with an improved value:

• name: the original event name or an improved one.
• displayname: can be a simple text string extended with dynamic values fetched from the event parameters referenced by the parameter {name} defined in the format attribute.
• format: for naming the parameter values and for special formatting of raw values (like fsc:reference for COO-addresses) - see “Log Definition Syntax”

The new displayname for telemetry events is on the one hand available as additional table column in the request details table and on the other hand it is used as default name in the request call stack (the tooltip will show the name and the displayname property as well as some additional information).

Additional to the event definitions the modules can also define a specific ordering of the modules in the request overview chart by means of listing all possible followers of every module.