Forensic Discovery (Addison-Wesley Professional Computing Series) / Edition 1

by Dan Farmer, Wietse Venema

Overview

"Don't look now, but your fingerprints are all over the cover of this book. Simply picking it up off the shelf to read the cover has left a trail of evidence that you were here.

"If you think book covers are bad, computers are worse. Every time you use a computer, you leave elephant-sized tracks all over it. As Dan and Wietse show, even people trying to be sneaky leave evidence all over, sometimes in surprising places.

"This book is about computer archeology. It's about finding out what might have been based on what is left behind. So pick up a tool and dig in. There's plenty to learn from these masters of computer security."
—Gary McGraw, Ph.D., CTO, Cigital, coauthor of Exploiting Software and Building Secure Software

"A wonderful book. Beyond its obvious uses, it also teaches a great deal about operating system internals."
—Steve Bellovin, coauthor of Firewalls and Internet Security, Second Edition, and Columbia University professor

"A must-have reference book for anyone doing computer forensics. Dan and Wietse have done an excellent job of taking the guesswork out of a difficult topic."
—Brad Powell, chief security architect, Sun Microsystems, Inc.

"Farmer and Venema provide the essential guide to 'fossil' data. Not only do they clearly describe what you can find during a forensic investigation, they also provide research found nowhere else about how long data remains on disk and in memory. If you ever expect to look at an exploited system, I highly recommend reading this book."
—Rik Farrow, Consultant, author of Internet Security for Home and Office

"Farmer and Venema do for digital archaeology what Indiana Jones did for historical archaeology. Forensic Discovery unearths hidden treasures in enlightening and entertaining ways, showing how a time-centric approach to computer forensics reveals even the cleverest intruder."
—Richard Bejtlich, technical director, ManTech CFIA, and author of The Tao of Network Security Monitoring

"Farmer and Venema are 'hackers' of the old school: They delight in understanding computers at every level and finding new ways to apply existing information and tools to the solution of complex problems."
—Muffy Barkocy, Senior Web Developer, Shopping.com

"This book presents digital forensics from a unique perspective because it examines the systems that create digital evidence in addition to the techniques used to find it. I would recommend this book to anyone interested in learning more about digital evidence from UNIX systems."
—Brian Carrier, digital forensics researcher, and author of File System Forensic Analysis

The Definitive Guide to Computer Forensics: Theory and Hands-On Practice

Computer forensics—the art and science of gathering and analyzing digital evidence, reconstructing data and attacks, and tracking perpetrators—is becoming ever more important as IT and law enforcement professionals face an epidemic in computer crime. In Forensic Discovery, two internationally recognized experts present a thorough and realistic guide to the subject.

Dan Farmer and Wietse Venema cover both theory and hands-on practice, introducing a powerful approach that can often recover evidence considered lost forever.

The authors draw on their extensive firsthand experience to cover everything from file systems, to memory and kernel hacks, to malware. They expose a wide variety of computer forensics myths that often stand in the way of success. Readers will find extensive examples from Solaris, FreeBSD, Linux, and Microsoft Windows, as well as practical guidance for writing one's own forensic tools. The authors are singularly well-qualified to write this book: They personally created some of the most popular security tools ever written, from the legendary SATAN network scanner to the powerful Coroner's Toolkit for analyzing UNIX break-ins.

After reading this book you will be able to

• Understand essential forensics concepts: volatility, layering, and trust
• Gather the maximum amount of reliable evidence from a running system
• Recover partially destroyed information—and make sense of it
• Timeline your system: understand what really happened when
• Uncover secret changes to everything from system utilities to kernel modules
• Avoid cover-ups and evidence traps set by intruders
• Identify the digital footprints associated with suspicious activity
• Understand file systems from a forensic analyst's point of view
• Analyze malware—without giving it a chance to escape
• Capture and examine the contents of main memory on running systems
• Walk through the unraveling of an intrusion, one step at a time

The book's companion Web site contains complete source and binary code for open source software discussed in the book, plus additional computer forensics case studies and resource links.

Product Details

ISBN-13:	9780201634976
Publisher:	Addison-Wesley
Publication date:	12/30/2004
Series:	Addison-Wesley Professional Computing Series
Edition description:	First Edition
Pages:	217
Product dimensions:	7.20(w) x 9.58(h) x 0.77(d)

About the Author

Dan Farmer is author of a variety of security programs and papers. He is currently chief technical officer of Elemental Security, a computer security software company. Together he and Wietse Venema, have written many of the world's leading information security and forensics packages, including the SATAN network security scanner and the Coroner's Toolkit.

Wietse Venema has written some of the world's most widely used software, including TCP Wrapper and the Postfix mail system. He is currently a research staff member at IBM Research. Together, he and Dan Farmer have written many of the world's leading information security and forensics packages, including the SATAN network security scanner and the Coroner's Toolkit.

Read an Excerpt

Today, only minutes pass between plugging in to the Internet and being attacked by some other machine—and that's only the background noise level of nontargeted attacks. There was a time when a computer could tick away year after year without coming under attack. For examples of Internet background radiation studies, see CAIDA 2003, Cymru 2004, or IMS 2004.

With this book, we summarize experiences in post-mortem intrusion analysis that we accumulated over a decade. During this period, the Internet grew explosively, from less than a hundred thousand connected hosts to more than a hundred million (ISC 2004). This increase in the number of connected hosts led to an even more dramatic—if less surprising—increase in the frequency of computer and network intrusions. As the network changed character and scope, so did the character and scope of the intrusions that we faced. We're pleased to share some of these learning opportunities with our readers.

In that same decade, however, little changed in the way that computer systems handle information. In fact, we feel that it is safe to claim that computer systems haven't changed fundamentally in the last 35 years—the entire lifetime of the Internet and of many operating systems that are in use today, including Linux, Windows, and many others. Although our observations are derived from today's systems, we optimistically expect that at least some of our insights will remain valid for another decade.What You Can Expect to Learn from This Book

The premise of the book is that forensic information can be found everywhere you look. With this guiding principle in mind, we develop tools to collect information from obviousand not-so-obvious sources, we walk through analyses of real intrusions in detail, and we discuss the limitations of our approach.

Although we illustrate our approach with particular forensic tools in specific system environments, we do not provide cookbooks for how to use those tools, nor do we offer checklists for step-by-step investigation. Instead, we present a background on how information persists, how information about past events may be recovered, and how the trustworthiness of that information may be affected by deliberate or accidental processes.

In our case studies and examples, we deviate from traditional computer forensics and head toward the study of system dynamics. Volatility and the persistence of file systems and memory are pervasive topics in our book. And while the majority of our examples are from Solaris, FreeBSD, and Linux systems, Microsoft's Windows shows up on occasion as well. Our emphasis is on the underlying principles that these systems have in common: we look for inherent properties of computer systems, rather than accidental differences or superficial features.

Our global themes are problem solving, analysis, and discovery, with a focus on reconstruction of past events. This approach may help you to discover why events transpired, but that is generally outside the scope of this work. Knowing what happened will leave you better prepared the next time something bad is about to occur, even when that knowledge is not sufficient to prevent future problems. We should note up front, however, that we do not cover the detection or prevention of intrusions. We do show that traces from one intrusion can lead to the discovery of other intrusions, and we point out how forensic information may be affected by system-protection mechanisms, and by the failures of those mechanisms.Our Intended Audience

We wrote this book for readers who want to deepen their understanding of how computer systems work, as well as for those who are likely to become involved with the technical aspects of computer intrusion or system analysis. System administrators, incident responders, other computer security professionals, and forensic analysts will benefit from reading this book, but so will anyone who is concerned about the impact of computer forensics on privacy.

Although we have worked hard to make the material accessible to nonexpert readers, we definitely do not target the novice computer user. As a minimal requirement, we assume strong familiarity with the basic concepts of UNIX or Windows file systems, networking, and processes.Organization of This Book

The book has three parts: we present foundations first, proceed with analysis of processes, systems, and files, and end the book with discovery. We do not expect you to read everything in the order presented. Nevertheless, we suggest that you start with the first chapter, as it introduces all the major themes that return throughout the book.

In Part I, "Basic Concepts," we introduce general high-level ideas, as well as basic techniques that we rely on in later chapters.

• Chapter 1, "The Spirit of Forensic Discovery," shows how general properties of computer architecture can impact post-mortem analysis. Many of the limitations and surprises that we encounter later in the book can already be anticipated by reading this chapter.
• Chapter 2, "Time Machines," introduces the concept of timelining, using examples of host-based and network-based information, including information from the domain name system. We look at an intrusion that stretches out over an entire year, and we show examples of finding time information in non-obvious places.

In Part II, "Exploring System Abstractions," we delve into the abstractions of file systems, processes, and operating systems. The focus of these chapters is on analysis: making sense of information found on a computer system and judging the trustworthiness of our findings.

• Chapter 3, "File System Basics," introduces fundamental file system concepts, as well as forensic tools and techniques that we will use in subsequent chapters.
• Chapter 4, "File System Analysis," unravels an intrusion by examining the file system of a compromised machine in detail. We look at both existing files and deleted information. As in Chapter 2, we use correlation to connect different observations, and to determine their consistency.
• Chapter 5, "Systems and Subversion," is about the environment in which user processes and operating systems execute. We look at subversion of observations, ranging from straightforward changes to system utilities to almost undetectable malicious kernel modules, and detection of such subversion.
• Chapter 6, "Malware Analysis Basics," presents techniques to discover the purpose of a process or a program file that was left behind after an intrusion. We also discuss safeguards to prevent malware from escaping, and their limitations.

In Part III, "Beyond the Abstractions," we look beyond the constraints of the file, process, and operating system abstractions. The focus of this part is on discovery, as we study the effects of system architecture on the decay of information.

• Chapter 7, "The Persistence of Deleted File Information," shows that large amounts of deleted file information can survive intact for extended periods. We find half-lives on the order of two to four weeks on actively used file systems.
• Chapter 8, "Beyond Processes," shows examples of persistence of information in main memory, including the decrypted contents of encrypted files. We find large variations in persistence, and we correlate these variations to operating system architecture properties.

The appendices present background material: Appendix A is an introduction to the Coroner's Toolkit and related software. Appendix B presents our current insights with respect to the order of volatility and its ramifications when capturing forensic information from a computer system.Conventions Used in This Book

In the examples, we use constant-width font for program code, command names, and command input/output. User input is shown in bold constant-width font. We use $ as the shell command prompt for unprivileged users, and we reserve # for super-user shells. Capitalized names, such as Argus, are used when we write about a system instead of individual commands.

Whenever we write "UNIX," we implicitly refer to Solaris, FreeBSD, and Linux. In some examples we include the operating system name in the command prompt. For example, we use solaris$ to indicate that an example is specific to Solaris systems.

As hinted at earlier, many examples in this book are taken from real-life intrusions. To protect privacy, we anonymize information about systems that are not our own. For example, we replace real network addresses with private network addresses such as 10.0.0.1 or 192.168.0.1, and we replace host names or user names. Where appropriate, we even replace the time and time zone.Web Sites

The examples in this book feature several small programs that were written for the purpose of discovery and analysis. Often we were unable to include the entire code listing because the additional detail would only detract from the purpose of the book. The complete source code for these and other programs is made available online at these Web sites:http://www.fish.com/forensics/
http://www.porcupine.org/forensics/

On the same Web sites, you will also find bonus material, such as case studies that were not included in the book and pointers to other resources.

Table of Contents

	Preface	xiii
	About the Authors	xix
Part I	Basic Concepts	1
Chapter 1	The Spirit of Forensic Discovery	3
1.1	Introduction	3
1.2	Unusual Activity Stands Out	4
1.3	The Order of Volatility (OOV)	5
1.4	Layers and Illusions	8
1.5	The Trustworthiness of Information	10
1.6	The Fossilization of Deleted Information	12
1.7	Archaeology vs. Geology	13
Chapter 2	Time Machines	17
2.1	Introduction	17
2.2	The First Signs of Trouble	17
2.3	What's Up, MAC? An Introduction to MACtimes	18
2.4	Limitations of MACtimes	20
2.5	Argus: Shedding Additional Light on the Situation	21
2.6	Panning for Gold: Looking for Time in Unusual Places	25
2.7	DNS and Time	28
2.8	Journaling File Systems and MACtimes	31
2.9	The Foibles of Time	34
2.10	Conclusion	35
Part II	Exploring System Abstractions	37
Chapter 3	File System Basics	39
3.1	Introduction	39
3.2	An Alphabet Soup of File Systems	40
3.3	UNIX File Organization	40
3.4	UNIX File Names	44
3.5	UNIX Pathnames	44
3.6	UNIX File Types	45
	Regular Files	45
	Directories	45
	Symbolic Links	46
	IPC Endpoints	46
	Device Files	47
3.7	A First Look Under the Hood: File System Internals	48
3.8	UNIX File System Layout	54
3.9	I've Got You Under My Skin: Delving into the File System	55
3.10	The Twilight Zone, or Dangers Below the File System Interface	56
3.11	Conclusion	57
Chapter 4	File System Analysis	59
4.1	Introduction	59
4.2	First Contact	59
4.3	Preparing the Victim's File System for Analysis	60
4.4	Capturing the Victim's File System Information	61
4.5	Sending a Disk Image Across the Network	63
4.6	Mounting Disk Images on an Analysis Machine	65
4.7	Existing File MACtimes	68
4.8	Detailed Analysis of Existing Files	70
4.9	Wrapping Up the Existing File Analysis	72
4.10	Intermezzo: What Happens When a File Is Deleted?	73
	Parent Directory Entry	75
	Parent Directory Attributes	75
	Inode Blocks	75
	Data Blocks	76
4.11	Deleted File MACtimes	76
4.12	Detailed Analysis of Deleted Files	77
4.13	Exposing Out-of-Place Files by Their Inode Number	78
4.14	Tracing a Deleted File Back to Its Original Location	80
4.15	Tracing a Deleted File Back by Its Inode Number	81
4.16	Another Lost Son Comes Back Home	82
4.17	Loss of Innocence	82
4.18	Conclusion	85
Chapter 5	Systems and Subversion	87
5.1	Introduction	87
5.2	The Standard Computer System Architecture	88
5.3	The UNIX System Life Cycle, from Start-up to Shutdown	89
5.4	Case Study: System Start-up Complexity	90
5.5	Kernel Configuration Mechanisms	92
5.6	Protecting Forensic Information with Kernel Security Levels	95
5.7	Typical Process and System Status Tools	96
5.8	How Process and System Status Tools Work	99
5.9	Limitations of Process and System Status Tools	100
5.10	Subversion with Rootkit Software	101
5.11	Command-Level Subversion	102
5.12	Command-Level Evasion and Detection	102
5.13	Library-Level Subversion	106
5.14	Kernel-Level Subversion	107
5.15	Kernel Rootkit Installation	107
5.16	Kernel Rootkit Operation	108
5.17	Kernel Rootkit Detection and Evasion	111
5.18	Conclusion	115
Chapter 6	Malware Analysis Basics	117
6.1	Introduction	117
6.2	The Dangers of Dynamic Program Analysis	118
6.3	Program Confinement with Hard Virtual Machines	119
6.4	Program Confinement with Soft Virtual Machines	119
6.5	The Dangers of Confinement with Soft Virtual Machines	121
6.6	Program Confinement with Jails and chroot ()	122
6.7	Dynamic Analysis with System-Call Monitors	123
6.8	Program Confinement with System-Call Censors	126
6.9	Program Confinement with System-Call Spoofing	129
6.10	The Dangers of Confinement with System Calls	131
6.11	Dynamic Analysis with Library-Call Monitors	132
6.12	Program Confinement with Library Calls	133
6.13	The Dangers of Confinement with Library Calls	135
6.14	Dynamic Analysis at the Machine-Instruction Level	136
6.15	Static Analysis and Reverse Engineering	136
6.16	Small Programs Can Have Many Problems	140
6.17	Malware Analysis Countermeasures	141
6.18	Conclusion	141
Part III	Beyond the Abstractions	143
Chapter 7	The Persistence of Deleted File Information	145
7.1	Introduction	145
7.2	Examples of Deleted Information Persistence	146
7.3	Measuring the Persistence of Deleted File Contents	147
7.4	Measuring the Persistence of Deleted File MACtimes	149
7.5	The Brute-Force Persistence of Deleted File MACtimes	149
7.6	The Long-Term Persistence of Deleted File MACtimes	153
7.7	The Impact of User Activity on Deleted File MACtimes	154
7.8	The Trustworthiness of Deleted File Information	156
7.9	Why Deleted File Information Can Survive Intact	157
7.10	Conclusion	159
Chapter 8	Beyond Processes	161
8.1	Introduction	161
8.2	The Basics of Virtual Memory	162
8.3	The Basics of Memory Pages	164
8.4	Files and Memory Pages	164
8.5	Anonymous Memory Pages	165
8.6	Capturing Memory	165
8.7	The savecore Command	167
	Memory Device Files: /dev/mem and/dev/kmem	168
	Swap Space	169
	Other Memory Locations	169
8.8	Static Analysis: Recognizing Memory from Files	171
8.9	Recovering Encrypted File Contents Without Keys	172
	Creating an Encrypted File	172
	Recovering the Encrypted File from Main Memory	172
8.10	File System Blocks vs. Memory Page Technique	173
8.11	Recognizing Files in Memory	175
8.12	Dynamic Analysis: The Persistence of Data in Memory	177
8.13	File Persistence in Memory	179
8.14	The Persistence of Nonfile, or Anonymous, Data	180
8.15	Swap Persistence	182
8.16	The Persistence of Memory Through the Boot Process	182
8.17	The Trustworthiness and Tenacity of Memory Data	182
8.18	Conclusion	185
Appendix A	The Coroner's Toolkit and Related Software	187
A.1	Introduction	187
A.2	Data Gathering with grave-robber	187
A.3	Time Analysis with mactime	188
A.4	File Reconstruction with lazarus	189
A.5	Low-Level File System Utilities	191
A.6	Low-Level Memory Utilities	192
Appendix B	Data Gathering and the Order of Volatility	193
B.1	Introduction	193
B.2	The Basics of Volatility	193
B.3	The State of the Art	194
B.4	How to Freeze a Computer	195
	Before You Start	196
	Actually Collecting Data	197
B.5	Conclusion	198
	References	199
	Index	207

Editorial Reviews

The Barnes & Noble Review
Think of it as CSI: Unix, Linux, and Windows. The creators of the legendary SATAN and Coroner’s Toolkit security packages have written the definitive guide to forensics problem solving, analysis, and discovery.

Forensic data can be found everywhere you look -- if you know how. So Dan Farmer and Wietse Venema systematically explain how information and traces of past events persist, and how to recover and assess them.

You’ll learn how computer architecture impacts your analysis; then master the crucial concept of timelining: drawing on host- and network-based information to understand what happened when. (The authors present a case study intrusion that lasted a full year.)

Farmer and Venema systematically demonstrate how to examine file systems and evaluate the trustworthiness of the data you capture. You’ll learn how to identify subversions of user processes and operating systems -- from simple changes to malicious kernel modules. There’s a chapter on uncovering the hidden purpose of malware. You’ll even learn how to discover clues in RAM (for example, decrypted contents of encrypted files).

This book is about the spirit and thinking involved in successful forensic analysis. It’s not a “cookbook.” But there are more than enough specific techniques and tools coverage to make you effective in real-world investigations. There are plenty of examples, too: from Solaris, FreeBSD, Linux, occasionally Windows. Strikingly, however, the underlying principles remain constant regardless of environment. And whether you’re a sysadmin or law enforcement professional, you need to know them. Bill Camarda, from the February 2005 Read Only

bn.com