DTrace: Anatomy of a Rock Star Application

The makers of the mega-hit dynamic tracing framework discuss changes in development

By Alex Handy

October 15, 2006 —

Adam Leventhal, Mike Shapiro and Bryan Cantrill are rock stars at Sun Microsystems. The three have spent almost the entire new millennium working on DTrace, the ground-breaking dynamic tracing framework in Solaris 10 that has won a dedicated following and much praise in the industry and press.

DTrace gives developers a view from the top of an application to the bottom. Using either prewritten probes or those written in the tool’s D language, DTrace follows a process or thread from its initiation to its termination. That gives developers a view into what resources are being used, where slowdowns occur, and where serious errors enter the data stream.

We sat down with staff engineer Leventhal, distinguished engineer Shapiro, and senior staff engineer Cantrill to discuss the genesis of the DTrace project, and to talk about how software has changed over the past 10 years.

Adam Leventhal: DTrace…was designed to be able to answer arbitrary questions with concision. There are two aspects to the power of DTrace. One is the dynamic nature of it: being able to ask arbitrary questions from any place in the system, from the lowest levels like the I/O subsystems and scheduling, through the kernel and system calls, all the way up to areas of Java and userland and things like that—aspects of any piece of the system. The other is that you can gather arbitrary data that is not just a predetermined payload, but rather anything you can think of. It was designed for production systems, meaning that it needs to always be safe. There can never be some way you could instrument the system in a way that could crash it or cause data corruption. Which is very different than any other tracing framework that's been designed in the past, or any sort of analysis tool that was designed in the past. They've either been these static tools which don't let you have the sort of breadth or depth or observability. Or they've been designed for developers in development, not for production systems.

Bryan Cantrill: The key observation that we had is the nature of software has changed. It used to be that you would generate a binary and ship it, and if there was a problem with the binary, you could just go back to development and fix it.

Mike Shapiro: Also that binary was your whole application. One binary, one processor, one thread, and most likely written in one language. And probably only using the base system, so it might access three libraries.

Cantrill: It was entirely static. So it was in C, or C++, or Pascal or Fortran

Leventhal: And the libraries were all from one vendor.

Cantrill: Life was simple. And the tools for understanding software reflected that simplicity. Then in the ’90s, the promise of componentization that had been fringe became mainstream. And that happened, arguably first with Java, but really it was the advent of the Web, where you could suddenly start really realizing distributed computing goals. You had multiple applications now, or multiple processes that form a single application. So an application became not just a binary. But then you had a Web server and an app server and a database server, and now you don't have one language—you've got three or four or five. You've got these different environments, different protected domains, and the problem is that the problems in software didn't really show up in development. They started showing up increasingly in production. So it was only when you deployed these systems, and you had hundreds of thousands or millions of users on them, where you start seeing these problems. And the existing tools were kind of barely keeping up in the simple world, but when the world became more complicated, the existing tools were not going to work.

Shapiro: I think the other consequence of what Bryan is talking about that's important to also see is that it's not just the same people with different tools, but that the people roles and what the roles are have totally changed as well. A lot of the people you talk to probably…run the gamut from someone who might be writing one of those components to someone who might be more like an integrator. And so the componentization drives productivity because you can sort of have people who focus on assembly activities, or the business logic, or the glue. And that changes roles as well, because it means that there's no notion of, “Well, if I have a problem, only the guy who wrote this had this great tool to figure out what was going on,” because there's not just one guy anymore. There's this huge stack of stuff that was assembled, and you don't even have the source code for all of it, right? Even if you had the world's greatest source level debugger, it doesn't really apply to a lot of these new roles. And that's why the tools not only have to be different but also have to look more at who is actually using these things. That's also where the focus on production comes from, because that's where the actual stuff gets assembled into something, right?

Leventhal: We are taking a very different tack than previous tool makers. Previous tool makers have been focusing on developers in development, which puts very different constraints on the problem, and they've been focusing on one particular aspect of the system, like debugging Java, or debugging C, or even analyzing kernel statistics. We were trying to make a tool that had a systemic scope, that would let you gather information from all of these different sources.

Shapiro: And I think that part of this is a consequence of where these things are coming from. The reason why the old tools were very much focused in that way, like around a language, is because they were written by the same people that wrote the compiler. Usually long after they wrote the compiler. The debugger goes with the compiler, and you go back and you add your -g flag and you get your debugging, and you run your thing.

Cantrill: It's the people who wrote the compiler writing the developer tools. So what is their experience with writing software? It's writing a compiler. And a compiler, in the grand scheme of things, is actually a pretty straightforward piece of software. I wouldn't say it's simple. I wouldn't say that it’s easy. But I would say that it's straightforward.

Leventhal: Because you're going from one place to another, not just this amorphous realm of real applications.

Cantrill: It's like you don't have to deal with this varying throughput, or worry about latency, or all these technologies. With a compiler, it's like, “This input needs to produce that output.” And if it doesn't, it’s busted, and if it does, it works. And so the problem is the development tools were aimed at understanding applications that were that simple. And the world is just a lot more complicated. We're not a development tools organization. We're operating systems guys. I mean, we're all kernel developers. That's our background. So we were coming at this problem from a systemic production perspective. In terms of whether it was hard...it was damn hard. It still is damn hard. Because there are still unsolved problems that we want to solve that have all of us scratching our heads. And I think that, if I can speak on behalf of all three of us, each of us was taken to the very brink of our ability. And there were moments, at least speaking for me personally, where I was thinking, “How am I going to do this? We're not going to be able to pull this off! There are just too many balls in the air, and this is too complicated.” When you are being too clever at the microprocessor level, which we are in several cases, you're always worried that the other shoe is going to drop. You're going to get all the way to the point of being able to ship it, and you're going to discover that there's some edge condition that completely invalidates what you're doing.

Leventhal, Shapiro and Cantrill explain just what DTrace does, by providing an understanding as to why the program was designed in the first place.

Cantrill: The reason this [traceability into software performance] had not been solved actually goes back to a property of software: You can't see when the instructions are executed. If you want to actually know what the hell your software is doing, historically you had to add constructs to your code that say, “If debugging, if logging, if whatever, do this.” The problem is that that construct slows the system down, because it's a load, it's a compare, it's a branch. So even in the operating system, even something in the I/O stack, which is an extremely important instrument, you can't slow down I/O 4 percent. Or even 2 percent. Because then, your competitor, who is not adding that kind of observability, starts beating you up because their I/O is 2 percent faster. Then you've got to go to the customer and explain to the customer why they want to give up that 2 percent. And lots of customers don't want to give up that performance. Of course, when they're having a problem, they're willing to give up tons of performance to solve the problem.

The constraint on the problem is: When I'm not asking a question—when I don't want to know what the system is doing—[the system] needs to run at its fastest possible speed. There needs to be 0.0 percent performance hit for having this infrastructure involved. The people who did this before concentrated too much on minimizing that performance hit when it's running overall.

But it turns out, that doesn't matter as much. The truth is, when I want to know what my system is doing, I've got some other problem. I'm already having the worst day of my life, and you're saying I can't do this because it's going to make I/O 2 percent slower? Well, make I/O 2 percent slower! The CEO's on the phone with me because they've seen this huge spike in I/O, and 2 percent doesn't matter! It's not the enabled probe effect that matters, it's the disabled probe effect that matters. And before, fundamentally, that was not the focus. We were the first team to really focus on the disabled probe effect and the disabled probe effect almost exclusively.

Shapiro: DTrace can go anywhere and instrument anything. There's a collection of tens of thousands of probes—and you can create more—and the probes are like little programmable sensors that don't really exist. They're places where DTrace can dynamically turn on instrumentation. And you write a little program in a scripting language we have called D, which looks just like C with a few extensions to make it more appropriate for the task of tracing, and you write these little things. Really, it looks like a sort of Awk or Perl script. So you're phrasing your question in this language.

By virtue of having a programming language, you can raise arbitrary questions. You can ask anything you want. What your cost is, is proportional to your question. So, you type in a little script, and say, “What's going on with this?” and you see it and hit control-C and the system goes back to running as it was before, and you ask another question.

Cantrill: In the past, you could just use a little tool in Solaris called Pstack, and if you Pstack the JVM, you'd get all these question marks in these Java frames. So you got C++ frames, and you could see that you're in the interpreter getting question marks after question marks. The bottom of the system’s like, “I dunno!” [shrugs]

Shapiro: And the JVM is particularly nasty at this problem because, traditionally, if you look at the interpreter for something like Perl, it's not doing anything like the JVM is doing. Basically, its interpreting like traditional interpreting: It's interpreting the bytecode. It's building a data structure that represents the stack, right? The JVM is mixing interpreting with actually converting bytecode into assembly language on the fly. And doing hot-spot compiling. So you have this mishmash of stuff.

So what we did to solve the Java programming problem, is we made the guys who wrote the JVM write a chunk of this code in our D language. This basically implements the “How do I take a C stack frame, and go to the corresponding Java thing, and get the method name, and get the things that you want and return that?” They compile that into our bytecode, and then, that gets glued into the JVM.

The JVM is not just a program, right? It's also embeddable. So, it's like a shared library, really that implements it wherever. Inside of that shared library are basically two things. It's the compiled bytecode of how to call DTrace—how to do this. And then there's a little piece of other glue code. Which beams this information up to the kernel when the JVM starts.

Cantrill: Adam and I were playing frisbee during lunch, and we were like...“This is as simple as it's gonna get. I mean, look at the problem! We've got multiple JVMs, and that knowledge exists only in the JVM!”

And we were faced with another very challenging problem, which is: How are you going to instrument these very dynamic environments in ways that semantically make sense.

Leventhal: And it was quite natural that the next step was extending it into these dynamic languages.

And thus, DTrace has extended beyond monitoring in Solaris and inside of Java Virtual Machines. As an open-source tool, the code is being massaged into Mac OS X 10.5, the Leopard. And the three-man team has since added JavaScript to the wide list of dynamic language environments already supported. Others are bringing the tool to the Linux and BSD environments. So the DTrace team rocks on. Maybe they’ll raise the roof at the next JavaOne!

Share this link: http://www.sdtimes.com/link/29691

Five changes for application development in 2010 “Lean and mean” defines how development organizations can become more efficient. Forrester offers five steps on getting there

AccuRev and Rally partner on agile solution AccuRev and Rally worked to integrate AccuRev's change and configuration management tool with Rally's agile life-cycle management solution. The result is a beta product called AccuBridge.

News Briefs: July 1, 2008 Cynergy's relocation triples the size of its headquarters, Microsoft releases version 1 of its Open XML SDK, NelsonHall predicts increased outsourcing, and AccuRev and Rally team up on agile CRM.

Add comment

Name*
Email*
Country

Comment
Preview

News on Monday
more>>
SharePoint Tech Report
more>>

Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?

Google Code turns 5
Google Code Turns 5, and adds a Paxos Algorithm to make the system more stable and reliable.
03/17/2010 11:16 AM EST

Test your Visual Studio 2010 know-how
Microsoft is offering free beta certification exams for Visual Studio 2010.
03/17/2010 11:08 AM EST

Microsoft lifts the hood on IE9
Microsoft is previewing IE9.
03/16/2010 01:10 PM EST