Generic REST API

The REST API source is a generic receiver that can pull JSON data from any REST API endpoint. It supports both logs and metrics collection, with configurable authentication, pagination, and time-based offset tracking.

Note: Due to the wide range of possible use cases for this source, it offers a best-effort integration with common API patterns, but may not completely align with every REST API.

Supported Platforms

Platform	Metrics	Logs	Traces
Linux	✓	✓
Windows	✓	✓
macOS	✓	✓

Prerequisites

• A REST API endpoint that returns JSON data

• Appropriate authentication credentials (if required)

Configuration Table

Field	Type	Default	Required	Description
url	string		true	The base URL for the REST API endpoint.
response_format	string	json	false	Response body format: json (standard JSON array/object) or ndjson (newline-delimited JSON). In NDJSON mode, each line is a separate JSON object.
response_field	string		false	The name of the field in the response that contains the array of items. If empty, the response is assumed to be a top-level array. For nested fields, use dot notation (e.g., response.data). Not used when response_format is ndjson.
metrics	object		false	Metrics configuration
auth_mode	string	none	false	Authentication mode: none, apikey, bearer, basic, oauth2, or akamai_edgegrid.
apikey	object		false	API Key configuration
bearer	object		false	Bearer Token configuration
basic	object		false	Basic Auth configuration
oauth2	object		false	OAuth2.0 Client Credentials configuration
akamai_edgegrid	object		false	Akamai EdgeGrid configuration
headers	map		false	A map of custom headers to send with each request. Header values will appear in plaintext unless marked as Sensitive.
pagination	object		false	Pagination configuration
Time-bounding fields	—		false	Top-level start_time_param_name, start_time_value, end_time_param_name, end_time_value, and timestamp_format fields. See Time-Bounding Configuration.
min_poll_interval	duration	10s	false	Minimum interval between API polls. The receiver resets to this interval when data is received. Increase this to prevent hitting API rate limits.
max_poll_interval	duration	5m	false	Maximum interval between API polls. The receiver uses adaptive polling that starts at min_poll_interval and backs off when no data is returned, up to this maximum.
backoff_multiplier	float	2.0	false	Multiplier for increasing the poll interval when no data or a partial page is returned. Must be greater than 1.0.
storage	component		false	The component ID of a storage extension for checkpointing.
timeout	duration	10s	false	HTTP client timeout.

Auth Mode Configuration

None (No Authentication)

Use auth_mode: none for public APIs that don't require authentication. No additional configuration is needed.

API Key

Field	Type	Default	Required	Description
header_name	string		true	Header name for API key.
value	string		true	API key value.

Bearer Token

Field	Type	Default	Required	Description
token	string		true	Bearer token value.

Basic Auth

Field	Type	Default	Required	Description
username	string		true	Username for basic auth.
password	string		true	Password for basic auth.

OAuth 2.0 Client Credentials

Field	Type	Default	Required	Description
client_id	string		true	OAuth2.0 client ID.
client_secret	string		true	OAuth2.0 client secret.
token_url	string		true	OAuth2.0 token endpoint URL.
scopes	[]string		false	OAuth2.0 scopes to request.
endpoint_params	map[string]string		false	Additional parameters to send to the token endpoint.

Akamai EdgeGrid

The Akamai API requires an enterprise license. This authentication method has not been tested against an Akamai API.

Field	Type	Default	Required	Description
akamai_access_token	string		true	Akamai EdgeGrid access token.
akamai_client_token	string		true	Akamai EdgeGrid client token.
akamai_client_secret	string		true	Akamai EdgeGrid client secret.

Time-Bounding Configuration

These top-level fields add start and/or end time query parameters to every request. They work with any pagination mode (or none). When used with timestamp pagination, the start time advances automatically through response data.

Field	Type	Default	Required	Description
start_time_param_name	string		false	Query parameter name for start time (e.g., "since", "from", "start_time"). Required when pagination.mode is timestamp.
start_time_value	string		false	Value for the start time: "now" resolves to the current time at receiver start (not dynamically updated), or a fixed timestamp in the configured format (e.g., "2025-01-01T00:00:00Z" or "1704067200" for epoch). For timestamp pagination, this is the initial start value.
end_time_param_name	string		false	Query parameter name for end time (e.g., "until", "to", "end_time"). If set, sends an end time on every request.
end_time_value	string	now	false	Value for the end time: "now" (default) resolves to the current time at receiver start (not dynamically updated), or a fixed timestamp in the configured format.
timestamp_format	string	RFC3339	false	Format for both start and end time query parameters. Accepts Go time format strings or epoch formats (see Timestamp Formats).

Pagination Configuration

Field	Type	Default	Required	Description
pagination.mode	string	none	false	Pagination mode: none, offset_limit, page_size, or timestamp.
pagination.response_source	string	body	false	Where to extract pagination response attributes from: body (response body or NDJSON metadata line) or header (HTTP response headers). Applies to total_record_count_field, next_offset_field_name, and total_pages_field_name.
pagination.total_record_count_field	string		false	Name of the field or header containing total record count.
pagination.page_limit	int	0	false	Maximum number of pages to fetch (0 = no limit).
pagination.zero_based_index	bool	false	false	Indicates that the requested data starts at index 0.

Offset/Limit Pagination

Field	Type	Default	Required	Description
pagination.offset_limit.offset_field_name	string		false	Query parameter name for offset.
pagination.offset_limit.limit_field_name	string		false	Query parameter name for limit.
pagination.offset_limit.starting_offset	int	0	false	Starting offset value.
pagination.offset_limit.next_offset_field_name	string		false	Name of the field or header containing the next offset token. When set, the receiver uses token-based (cursor) pagination instead of numeric offsets. For body sources, supports nested fields with dot notation (e.g., pagination.next_cursor).

Page/Size Pagination

Field	Type	Default	Required	Description
pagination.page_size.page_num_field_name	string		false	Query parameter name for page number.
pagination.page_size.page_size_field_name	string		false	Query parameter name for page size.
pagination.page_size.starting_page	int	1	false	Starting page number.
pagination.page_size.total_pages_field_name	string		false	Name of the field or header containing total page count.

Timestamp-Based Pagination

Timestamp pagination uses the top-level start_time_param_name and timestamp_format fields to send an advancing start time. The start time is automatically updated from response data as pages are consumed.

Field	Type	Default	Required	Description
pagination.timestamp.timestamp_field_name	string		true	Field name in each response item containing the timestamp (e.g., "ts", "timestamp").
pagination.timestamp.page_size_field_name	string		false	Query parameter name for page size (e.g., "perPage", "limit").
pagination.timestamp.page_size	int	100	false	Page size to use.

Common timestamp formats:

• 2006-01-02T15:04:05Z07:00 - RFC3339 (default)

• 20060102150405 - YYYYMMDDHHMMSS

• 2006-01-02 15:04:05 - Date and time with space separator

• 2006-01-02 - Date only

Epoch timestamp formats (sends numeric values instead of formatted strings):

• epoch_s - Unix epoch seconds (e.g., 1704067200)

• epoch_ms - Unix epoch milliseconds (e.g., 1704067200000)

• epoch_us - Unix epoch microseconds (e.g., 1704067200000000)

• epoch_ns - Unix epoch nanoseconds (e.g., 1704067200000000000)

• epoch_s_frac - Fractional epoch seconds (e.g., 1704067200.123456). The integer part is seconds and the fractional digits represent sub-second precision (.123 = milliseconds, .123456 = microseconds, .123456789 = nanoseconds). Used by APIs that expect seconds.fraction format in both responses and query parameters.

Metrics Configuration

The metrics configuration allows you to customize how metrics are extracted from API responses.

Field	Type	Default	Required	Description
metrics.name_field	string		true	Field name in each response item containing the metric name. If not found, the metric will be dropped and a warning will be logged.
metrics.description_field	string		false	Field name in each response item containing the metric description. If not specified or not found, defaults to Metric from REST API.
metrics.type_field	string		false	Field name in each response item containing the metric type (gauge, sum, histogram, summary). If not specified or not found, defaults to gauge.
metrics.unit_field	string		false	Field name in each response item containing the metric unit. If not specified or not found, no unit is set.
metrics.monotonic_field	string		false	Field name in each response item indicating if a sum metric is monotonic (boolean). Only applies to sum metrics. If not specified or not found, defaults to false for safety.
metrics.aggregation_temporality_field	string		false	Field name in each response item containing the aggregation temporality (cumulative or delta). If not specified or not found, defaults to cumulative.

Note: When field names are configured, those fields are automatically excluded from metric attributes to avoid duplication. If the name_field isn't found, the metric will be dropped.

Response Format

The receiver expects JSON responses in one of two formats:

1. Top-level array:

[
  { "id": "1", "message": "log entry 1" },
  { "id": "2", "message": "log entry 2" }
]

1. Object with data field:

{
  "data": [
    { "id": "1", "message": "log entry 1" },
    { "id": "2", "message": "log entry 2" }
  ],
  "total": 2
}

When using the second format, specify the field name in response_field (e.g., "data").

Checkpointing

When a storage extension is configured via the storage field, the receiver saves its pagination state to storage. This allows the receiver to resume from where it left off after a restart, preventing duplicate data collection.

The checkpoint includes:

• Current pagination state (offset/page number/timestamp)

• Number of pages fetched

For timestamp-based pagination, the advancing start time is persisted so the receiver continues from the most recently observed timestamp after a restart.

Config Fingerprint

Each checkpoint is tagged with a fingerprint derived from the config fields that define what data the receiver fetches: url, pagination.mode, start_time_value, end_time_value, pagination.offset_limit.starting_offset, and pagination.page_size.starting_page. If any of these change between runs, the stored checkpoint is considered invalid and discarded, because it tracks pagination state for a different query. Other config changes (such as credentials or polling intervals) do not invalidate the checkpoint.