Log Drains

You can use log drains to send your application’s logs to an external server with the following command.

clever drain create [--alias <alias>] <DRAIN-TYPE> <DRAIN-URL> [--username <username>] [--password <password>]

Where DRAIN-TYPE is one of:

• syslog-tcp: for TCP syslog endpoint;
• syslog-udp: for UDP syslog endpoint;
• raw-http: for HTTP endpoint (note that this endpoint has optional username/password parameters as HTTP Basic Authentication);
• elasticsearch: for Elasticsearch endpoint (note that this endpoint requires username/password parameters as HTTP Basic Authentication);
• datadog: for Datadog endpoint (note that this endpoint needs your Datadog API Key);
• newrelic: for NewRelic endpoint (note that this endpoint needs your NewRelic API Key);
• ovh-tcp: for OVH TCP syslog endpoint (note that this endpoint has an optional sd-params parameter).

You can list the currently activated drains with this command.

clever drain [--alias <alias>]

And remove them if needed

clever drain remove [--alias <alias>] <DRAIN-ID>

If the status of your drain appears as DISABLED without you disabling it, it may be because it haven’t been able to send your application logs to your drain endpoint or because the requests timed out after 25 seconds.

Use the logs drain to send your add-on’s logs by using --addon flag, the value must be the add-on ID starting by addon_.

Elasticsearch

Elasticsearch drains use the Elastic bulk API. To match this endpoint, specify /_bulk at the end of your Elasticsearch endpoint.

clever drain create elasticsearch https://xxx-elasticsearch.services.clever-cloud.com/_bulk --username USERNAME --password PASSWORD

Each day, we will create an index logstash-<yyyy-MM-dd> and push logs to it.

Index Lifecycle Management

Depending on the amount of logs generated by your application, you might want to manage the lifecycle of your log indexes to prevent your Elasticsearch instance from running out of storage space.

To do so, Elasticsearch provides a feature called Index Lifecycle management that allows you to create a policy to delete indexes based on their creation date.

With our Elasticsearch add-on, you can choose to create a Kibana application in which you can create the policy and apply it to your indexes with an index template, but you can create them manually through API requests.

Here is an example that will create a policy to delete indexes older than 30 days:

curl -X PUT "https://username:password@xxx-elasticsearch.services.clever-cloud.com/_ilm/policy/logs_drain?pretty" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
'

An index template example to apply the policy based on an index pattern:

curl -X PUT "https://username:password@xxx-elasticsearch.services.clever-cloud.com/_index_template/logs_drain?pretty" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logstash-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs_drain"
    }
  }
}
'

For more information, please refer to the official documentation.

Datadog

To create a Datadog drain, you just need to use:

clever drain create datadog "https://http-intake.logs.datadoghq.com/v1/input/<API_KEY>?ddsource=clevercloud&service=<SERVICE>&hostname=<HOST>"

NewRelic

To create a NewRelic drain, use:

clever drain create newrelic "https://log-api.eu.newrelic.com/log/v1" --api-key "<API_KEY>"

OVHcloud Logs Data Platform

To export logs from an application or an add-on to OVHcloud Logs Data Platform, use the following setup:

• A TCP drain log with clever drain create ovh-tcp
• Your Logs Data Platform host with port 514 (SSL ports aren’t supported for TCP drains)
• The write token for your stream (provided on your Logs Data Platform console)

On your terminal, use the following command:

clever drain create ovh-tcp tcp://<host>:514 -app <application-id-or-name> --sd-params="X-OVH-TOKEN=\"<token>\""

clever drain create ovh-tcp tcp://<host>:514 -addon <addon_id> --sd-params="X-OVH-TOKEN=\"<token>\""

Community software

HTTPS-based solution

Some tools available on GitHub enable to create a drain to collect logs through an HTTPS endpoint. This project, for example, is fully compatible with Clever Cloud.

You could host it as an app and an add-on on Clever Cloud. A complete README explains all the features.