The project is hosted on GitHub: y0ngb1n/spring-boot-samples, feel free to Star, Fork 😘
Importance of Logs#
Why It Matters#
- Operations and Maintenance: A doctor examines a patient, and logs are the patient's statements about their condition.
- Malicious attacks, malicious registrations, fake orders, malicious password guessing, etc.
Challenges Faced#
- Many points of concern, any one of which could cause problems.
- Logs are scattered across many machines, and only when issues arise do we find that logs have been deleted.
- Many operations personnel are like firefighters, going wherever there is a problem.
Centralized Log Management#
Log Search > Formatted Analysis > Retrieval and Visualization > Risk Alerts
Quickly Set Up ELK Integrated Environment#
Technology Selection#
So, what exactly is ELK? "ELK" is an acronym for three open-source projects, which are:
E(Elasticsearch) is a search and analytics engine.L(Logstash) is a server-side data processing pipeline that can simultaneously collect data from multiple sources, transform it, and then send it to "repositories" like Elasticsearch.K(Kibana) allows users to visualize data in Elasticsearch using graphs and charts.
Quick Deployment#
Use Docker to deploy the basic environment above. Refer to the configuration file docker-compose.yml and enter the following commands for one-click deployment:
# Check configuration
docker-compose config
# Start services (use -d for background)
docker-compose up -d
# Stop and remove services
docker-compose down
Configure Logstash#
Refer to the configuration file logstash-config.conf, an example is as follows:
input {
tcp {
mode => "server"
host => "0.0.0.0"
port => 8080
codec => json_lines
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
# The index name should refer to the index templates configuration, e.g.: logs-*-*
# index => "app-logs-%{app_name}-%{+YYYY.MM.dd}"
index => "app-logs-%{+YYYY.MM.dd}"
}
}
Spring Boot Integration with ELK for Log Management#
Add Dependency 
#
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>${logstash-logback-encoder.version}</version>
</dependency>
Add Logback Configuration#
Method 1: Configure via logback-spring.xml#
Refer to the configuration file logback-spring.xml, an example is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml" />
<!-- This will be serialized into the log document by default -->
<springProperty scope="context" name="APP_NAME" source="spring.application.name"/>
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>192.168.50.88:8880</destination>
<encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" >
<!-- customFields is an optional parameter specified in the Logstash configuration for the index name, this field will be added to the log document -->
<customFields>{"app_name":"${APP_NAME}"}</customFields>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="LOGSTASH" />
<appender-ref ref="CONSOLE" />
</root>
</configuration>
Method 2: Configure via Java Config (Customizable Starter)#
@Slf4j
@Configuration
@ConditionalOnProperty(prefix = "elk.logstash", name = "enabled", havingValue = "true")
public class LogstashLogbackConfig {
private static final String LOGSTASH_APPENDER_NAME = "LOGSTASH";
private static final String LOGSTASH_ASYNC_APPENDER_NAME = "ASYNC_LOGSTASH";
@Value("${spring.application.name}")
private String appName;
@Autowired
private LogstashProperties logstash;
@Bean
@ConfigurationProperties(prefix = "elk.logstash")
public LogstashProperties logstash() {
return new LogstashProperties();
}
@PostConstruct
private void addLogstashAppender() {
LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
log.info("Initializing LogstashAppender");
final LogstashTcpSocketAppender logstashAppender = new LogstashTcpSocketAppender();
logstashAppender.setName(LOGSTASH_APPENDER_NAME);
logstashAppender.setContext(loggerContext);
logstashAppender.addDestinations(
new InetSocketAddress(this.logstash.getHost(), this.logstash.getPort())
);
// https://github.com/logstash/logstash-logback-encoder
final LogstashEncoder logstashEncoder = new LogstashEncoder();
logstashEncoder.setIncludeContext(false);
String customFields = "{\"app_name\":\"" + this.appName +"\",\"idol\":\"yangbin\"}";
logstashEncoder.setCustomFields(customFields);
final ShortenedThrowableConverter throwableConverter = new ShortenedThrowableConverter();
throwableConverter.setRootCauseFirst(true);
logstashEncoder.setThrowableConverter(throwableConverter);
logstashAppender.setEncoder(logstashEncoder);
logstashAppender.start();
// Wrap the appender in an Async appender for performance
final AsyncAppender asyncLogstashAppender = new AsyncAppender();
asyncLogstashAppender.setContext(loggerContext);
asyncLogstashAppender.setName(LOGSTASH_ASYNC_APPENDER_NAME);
asyncLogstashAppender.setQueueSize(this.logstash.getQueueSize());
asyncLogstashAppender.addAppender(logstashAppender);
asyncLogstashAppender.start();
loggerContext.getLogger("ROOT").addAppender(asyncLogstashAppender);
}
}
Simulate Random Logs with a Timer#
...
2021-07-25 23:00:36.552 INFO 15928 --- [ main] i.g.y.s.e.config.LogstashLogbackConfig : Initializing LogstashAppender
2021-07-25 23:00:36.813 INFO 15928 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2021-07-25 23:00:37.017 INFO 15928 --- [ main] o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'taskScheduler'
2021-07-25 23:00:37.084 INFO 15928 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2021-07-25 23:00:37.095 INFO 15928 --- [ scheduling-1] i.g.y.s.elk.scheduler.MockLogScheduler : [4a2fa762-9390-48f3-8478-4fcdbf6ba017] mock log event, log something...
2021-07-25 23:00:37.097 WARN 15928 --- [ scheduling-1] i.g.y.s.elk.scheduler.MockLogScheduler : [c859978e-5150-4fe4-979f-9acd7957a55a] mock log event, log something...
2021-07-25 23:00:37.097 ERROR 15928 --- [ scheduling-1] i.g.y.s.elk.scheduler.MockLogScheduler : [7625e2e6-62e9-4bbf-a998-1e741723f824] mock log event, log something...
2021-07-25 23:00:37.599 INFO 15928 --- [ scheduling-1] i.g.y.s.elk.scheduler.MockLogScheduler : [54ba65f2-05d7-4f27-ab63-bd19a0165b6b] mock log event, log something...
2021-07-25 23:00:37.599 WARN 15928 --- [ scheduling-1] i.g.y.s.elk.scheduler.MockLogScheduler : [901ca6a0-fc7b-4749-8661-4c81f7c4701d] mock log event, log something...
2021-07-25 23:00:37.599 ERROR 15928 --- [ scheduling-1] i.g.y.s.elk.scheduler.MockLogScheduler : [e2010eda-3f63-49ec-bb97-1157a2b11f01] mock log event, log something...
...