Logging and Log Management

Logging and Log Management, 1st Edition

The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management

Logging and Log Management, 1st Edition,Anton Chuvakin,Kevin Schmidt,Chris Phillips,ISBN9781597496353

  &      &      





235 X 191

A comprehensive guide to log management written by the experts who wrote the rules.

Print Book + eBook

USD 59.94
USD 99.90

Buy both together and save 40%

Print Book


In Stock

Estimated Delivery Time
USD 49.95

eBook Overview

VST format:

DRM Free included formats: EPub, Mobi, PDF, EPUB, MOBI

USD 49.95
Add to Cart

Key Features

  • Comprehensive coverage of log management including analysis, visualization, reporting and more
  • Includes information on different uses for logs -- from system operations to regulatory compliance
  • Features case Studies on syslog-ng and actual real-world situations where logs came in handy in incident response
  • Provides practical guidance in the areas of report, log analysis system selection, planning a log analysis system and log data normalization and correlation


Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management introduces information technology professionals to the basic concepts of logging and log management. It provides tools and techniques to analyze log data and detect malicious activity. The book consists of 22 chapters that cover the basics of log data; log data sources; log storage technologies; a case study on how syslog-ng is deployed in a real environment for log collection; covert logging; planning and preparing for the analysis log data; simple analysis techniques; and tools and techniques for reviewing logs for potential problems. The book also discusses statistical analysis; log data mining; visualizing log data; logging laws and logging mistakes; open source and commercial toolsets for log data collection and analysis; log management procedures; and attacks against logging systems. In addition, the book addresses logging for programmers; logging and compliance with regulations and policies; planning for log analysis system deployment; cloud logging; and the future of log standards, logging, and log analysis. This book was written for anyone interested in learning more about logging and log management. These include systems administrators, junior security engineers, application developers, and managers.


Computer Security staff and program managers; system, network, and application administrators; computer security incident response teams; and others who are responsible for performing duties related to computer security log management.

Anton Chuvakin

Ph.D., Stony Brook University, Stony Brook, NY.

Dr. Anton Chuvakin is a recognized security expert in the field of log management and PCI DSS compliance. He is an author of the books "Security Warrior" and "PCI Compliance" and has contributed to many others, while also publishing dozens of papers on log management, correlation, data analysis, PCI DSS, and security management. His blog (http://www.securitywarrior.org) is one of the most popular in the industry. Additionaly, Anton teaches classes and presents at many security conferences across the world and he works on emerging security standards and serves on the advisory boards of several security start-ups. Currently, Anton is developing his security consulting practice, focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations. Anton earned his Ph.D. from Stony Brook University.

Affiliations and Expertise

is a recognized security expert in the field of log management and PCI DSS compliance.

View additional works by Anton Chuvakin

Kevin Schmidt

Kevin J. Schmidt is a senior manager at Dell SecureWorks, Inc., an industry leading MSSP, which is part of Dell. He is responsible for the design and development of a major part of the company’s SIEM platform. This includes data acquisition, correlation and analysis of log data. Prior to SecureWorks, Kevin worked for Reflex Security where he worked on an IPS engine and anti-virus software. And prior to this he was a lead developer and architect at GuardedNet, Inc.,which built one of the industry’s first SIEM platforms. Kevin is also a commissioned officer in the United States Navy Reserve (USNR). Kevin has over 19 years of experience in software development and design, 11 of which have been in the network security space. He holds a B.Sc. in computer science.

Affiliations and Expertise

is a team lead and senior software developer at SecureWorks, Inc.

Chris Phillips

Christopher Phillips is a manager and senior software developer at Dell SecureWorks, Inc. He is responsible for the design and development of the company's Threat Intelligence service platform. He also has responsibility for a team involved in integrating log and event information from many third party providers for customers to have their information analyzed by the Dell SecureWorks systems and security professionals. Prior to Dell SecureWorks, Christopher has worked for McKesson and Allscripts where he worked with clients on HIPAA compliance and security and integrating healthcare systems. Christopher has over 18 years of experience in software development and design. He holds a Bachelors of Science in Computer Science and an MBA.

Affiliations and Expertise

Christopher Phillips is a manager and senior software developer at Dell SecureWorks, Inc.

Logging and Log Management, 1st Edition

Acknowledgments About the Authors About the Technical Editor Foreword Preface Chapter 1 Logs, Trees, Forest: The Big Picture Introduction Log Data Basics What Is Log Data? How is Log Data Transmitted and Collected? What is a Log Message? The Logging Ecosystem A Look at Things to Come Logs Are Underrated Logs Can Be Useful Resource Management Intrusion Detection Troubleshooting Forensics Boring Audit, Fun Discovery People, Process, Technology Security Information and Event Management (SIEM) Summary Chapter 2 What is a Log? Introduction Definitions Logs? What logs? Log Formats and Types Log Syntax Log Content Criteria of Good Logging Ideal Logging Scenario Summary Chapter 3 Log Data Sources Introduction Logging Sources Syslog SNMP The Windows Event Log Log Source Classification Security-Related Host Logs Security-Related Network Logs Security Host Logs Summary Chapter 4 Log Storage Technologies Introduction Log Retention Policy Log Storage Formats Text-Based Log Files Binary Files Compressed Files Database Storage of Log Data Advantages Disadvantages Defining Database Storage Goals Hadoop Log Storage Advantages Disadvantages The Cloud and Hadoop Getting Started with Amazon Elastic MapReduce Navigating the Amazon Uploading Logs to Amazon Simple Storage Services (S3) Create a Pig Script to Analyze an Apache Access Log Processing Log Data in Amazon Elastic MapReduce (EMR) Log Data Retrieval and Archiving Online ?Near-line Offline Summary Chapter 5 syslog-ng Case Study Introduction Obtaining syslog-ng What Is syslog-ngsyslog-ng? Example Deployment Configurations Troubleshooting syslog-ng Summary Chapter 6 Covert Logging Introduction Complete Stealthy Log Setup Stealthy Log Generation Stealthy Pickup of Logs IDS Log Source Log Collection Server “Fake” Server or Honeypot Logging in Honeypots Honeynet’s Shell Covert Keystroke Logger Honeynet’s Sebek2 Case Study Covert Channels for Logging Brief Summary Chapter 7 Analysis Goals, Planning, and Preparation: What Are We Looking For? Introduction Goals Past Bad Things Future Bad Things, Never Before Seen Things, and All But the Known Good Things Planning Accuracy Integrity Confidence Preservation Sanitization Normalization Challenges with Time ?Preparation Separating Log Messages Parsing Data Reduction Summary Chapter 8 Simple Analysis Techniques Introduction Line by Line: Road to Despair Simple Log Viewers Real-Time Review Historical Log Review Simple Log Manipulation Limitations of Manual Log Review Responding to the Results of Analysis Acting on Critical Logs Acting on Summaries of Non-Critical Logs Developing an Action Plan Automated Actions Examples Incident Response Scenario Routine Log Review Summary Chapter 9 Filtering, Normalization, and Correlation Introduction Filtering Artificial Ignorance Normalization IP Address Validation Snort Windows Snare Generic Cisco IOS Messages Regular Expression Performance Concerns Correlation Micro-Level Correlation Macro-Level Correlation Using Data in Your Environment Simple Event Correlator (SEC) Stateful Rule Example Building Your Own Rules Engine Common Patterns to Look For The Future Summary Chapter 10 Statistical Analysis Introduction Frequency Baseline Thresholds Anomaly Detection Windowing Machine Learning k-Nearest Neighbor (kNN) Applying the k-NN Algorithm to Logs Combining Statistical Analysis with Rules-Based Correlation Summary Chapter 11 Log Data Mining Introduction Data Mining Intro Log Mining Intro Log Mining Requirements What We Mine For? Deeper into Interesting Summary Chapter 12 Reporting and Summarization Introduction Defining the Best Reports Authentication and Authorization Reports Network Activity Reports Why They Are Important Specifics Reports Who Can Use These Reports Resource Access Reports Why They Are Important Specifics Reports Who Can Use These Reports Malware Activity Reports Why They Are Important Specific Reports Who Can Use These Reports Critical Errors and Failures Reports Why They Are Important Specifics Reports Who Can Use These Reports Summary Chapter 13 Visualizing Log Data Introduction Visual Correlation Real-Time Visualization Treemaps Log Data Constellations Traditional Log Data Graphing Summary Chapter 14 Logging Laws and Logging Mistakes Introduction Logging Laws Law 1-Law of Collection Law 2-Law of Retention Law 3-Law of Monitoring Law 3-Law of Availability Law 4-Law of Security Law 5-Law of Constant Changes Logging Mistakes Not Logging at All Not Looking at Log Data Storing for Too Short a Time Prioritizing Before Collection Ignoring Application Logs Only Looking for Known Bad Entries Summary Chapter 15 Tools for Log Analysis and Collection Introduction Outsource, Build, or Buy Building a Solution Buy Outsource Questions for You, Your Organization, and Vendors Basic Tools for Log Analysis Grep Awk Microsoft Log Parser Other Basic Tools to Consider The Role of the Basic Tools in Log Analysis Utilities for Centralizing Log Information Syslog Rsyslog Snare Log Analysis Tools-Beyond the Basics OSSEC OSSIM Other Analysis Tools to Consider Commercial Vendors Splunk NetIQ Sentinel IBM q1Labs Loggly Summary Chapter 16 Log Management Procedures: Log Review, Response, and Escalation Introduction Assumptions, Requirements, and Precautions Requirements Precautions Common Roles and Responsibilities PCI and Log Data Key Requirement 10 Other Requirements Related to Logging Logging Policy Review, Response, and Escalation Procedures and Workflows Periodic Log Review Practices and Patterns Building an Initial Baseline Using a Log Management Tool Building an Initial Baseline Manually Main Workflow: Daily Log Review Exception Investigation and Analysis Incident Response and Escalation Validation of Log Review Proof of Logging Proof of Log Review Proof of Exception Handling Logbook-Evidence of Exception of Investigations Recommended Logbook Format Example Logbook Entry PCI Compliance Evidence Package Management Reporting Periodic Operational Tasks Daily Tasks Weekly Tasks Monthly Tasks Quarterly Tasks Annual Tasks Additional Resources Summary Chapter 17 Attacks Against Logging Systems Introduction Attacks What to Attack? Attacks on Confidentiality Attacks on Integrity Attacks on Availability Summary Chapter 18 Logging for Programmers Introduction Roles and Responsibilities Logging for Programmers What Should Be Logged? Logging APIs for Programmers Log Rotation Bad Log Messages Log Message Formatting Security Considerations Performance Considerations Summary Chapter 19 Logs and Compliance Introduction PCI DSS Key Requirement 10 ISO2700x Series HIPAA FISMA NIST 800-53 Logging Guidance Summary Chapter 20 Planning Your Own Log Analysis System Introduction Planning Roles and Responsibilities Resources Goals Selecting Systems and Devices for Logging Software Selection Open Source Commercial Policy Definition Logging Policy Log File Rotation Log Data Collection Retention/Storage Response Architecture Basic Log Server and Log Collector Log Server and Log Collector with Long-Term Storage Distributed Scaling Summary Chapter 21 Cloud Logging Introduction Cloud Computing Service Delivery Models Cloud Deployment Models Characteristics of a Cloud Infrastructure Standards? We Don’t Need No Stinking Standards! Cloud Logging A Quick Example: Loggly Regulatory, Compliance, and Security Issues Big Data in the Cloud A Quick Example: Hadoop SIEM in the Cloud Pros and Cons of Cloud Logging Cloud Logging Provider Inventory Additional Resources Summary Chapter 22 Log Standards and Future Trends Introduction Extrapolations of Today to the Future More Log Data More Motivations More Analysis Log Future and Standards Adoption Trends Desired Future Summary Index

Quotes and reviews

"The authors provide a way to simplify the complex process of analyzing large quantities of varied logs. The log management and log analysis approaches they recommend are addressed in detail."--Reference and Research Book News, August 2013
"…Anton Chuvakin and his co-authors Kevin Schmidt and Christopher Phillips bring significant real-world experience to the reader and an important book on the topic....For those that want to find the gold in their logs…[it] is a great resource that shows how to maximize the gold that often lays hidden in your large stores of log data."--RSA Conference, December 2012


Shop with Confidence

Free Shipping around the world
▪ Broad range of products
▪ 30 days return policy

Contact Us