How to aggregate logs from several Jenkins Jobs\Pipelines in one place?

Ilya Khadykin

Our project is responsible for migrating data from one system to another. We are going to run transformation, validation and migration scripts using Jenkins.

It's unclear for me how to aggregate logs from several Jobs or Pipelines in Jenkins. How it can be done?

We'll rely on logs heavily to identify any issues found during validation etc.

In terms of our planned setup we'll have AWS EC2 instances + we can use Datadog (our company uses it). Can we use Datadog for this purpose?

stephenlechner

You can reference this doc to find where the default logging path is for Jenkins depending on your OS. (For linux, it's /var/log/jenkins/jenkins.log if you don't configure it to be something else.

Then as long as your Datadog agent is v6+ you can use the Datadog agent to tail your jenkins.log file by following this doc.

Specifically, you'd add this line to your dadatod.yaml:

logs_enabled: true

and add this content to any old conf.yaml file nested in your conf.d/ directory, such as conf.d/jenkins.d/conf.yaml:

logs:
  - type: file
    path: /var/log/jenkins/jenkins.log
    service: jenkins
    source: jenkins

Then the agent will tail your log file as it's written to, and will forward it to your Datadog account so you can query, graph, and monitor on your log data there.

Once you have the logs coming in, you may want to write a processing pipeline to get the critical attributes parsed out, but that would be material for a new question :) .

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

How to manage several kvm from one place?

Collect files from several different directories and put them in one place

How to select several aggregate values in one query in Azure Cosmos DB

How to place several elements in one row when using a list?

How to use several hot strings with special characters at one place?

Mongo aggregate several documents into one

View AWS ALB access logs in one place

How to aggregate ngMessage of several field

aggregate several computations into one in Ember.computed

Python - put several ticks in one place

How SQL Server iterates over rows when computing several aggregate functions in one select query

How to move Jenkins from one PC to another

Use aggregate functions with group by from several tables

How to aggregate multiple columns from a list of column names one at a time?

How do I aggregate data from many files into one file?

How to aggregate and simultaneously replace values from one dataset for the second in R

How to collapse/aggregate values into several columns of dicts

CMake: How to create one object from several source files?

how to import several express middleware from one module?

How to cut a segment or several segment from one line?

How to merge several detached signatures from different people into one?

How to get one particular atribute from a class with several atributes

How to get value from one of several possible URL parameters?

How to transfer several rows from one table to other table (SQL)

How to get value of one of <td> from several <td> using jQuery?

How to get one value from several identical ones?

How to get data from multiple topics in one place for processing?

How to log all axios calls from one place in code

How can one drop/delete columns from a KDB table in place?