Blog Posts

How Slow Programs are Like Christmas

Method R Corporation makes systems faster. And we make people who make systems faster, faster. We train people to become performance optimization heroes.

Here is my story about why you should be interested in our training.

Slow programs remind me of Christmas when I was a kid. In early December, my parents would put a present for me under our Christmas tree. It would be wrapped up so I couldn’t see what was inside it. The unwrapping would not happen until Christmas morning, December 25. (That was the best-case scenario. If my Dad couldn’t be home because of work, then Christmas morning would come a day or two late.)

So, every day, for nearly a month, I would see that present under the tree and wonder what was in it. 

I’d take any clue I could find. What shape is it? What does it weigh? What does it sound like when I shake it? (Sometimes, my Mom and Dad would prohibit me from shaking it.) No matter how desperate the curiosity, all I could do was guess.

When Christmas came, I’d finally get to tear the paper off, and I would now see, plain as day, what had been in that box the whole time. All the clues and possibilities would collapse into a single reality. Finally, there was no more mystery, no more guessing.

Slow programs are like that. The clues aren’t enough. You guess a lot. But with slow programs, there’s no specially designated morning when your programs reveal their mysteries to you. They just keep irritating you, with no end in sight. You need somebody who knows how to tear the wrapping paper off those programs so you can see what they’re doing wrong. 

That’s the somebody I like being. The role scares a lot of people, but it doesn’t scare me. That’s because I trust my preparation. I know that I have three particular assets that tilt the game in my favor. 

Those assets are knowledgetools, and community. With the knowledge and tools I have, I don’t get stumped very often. But when I do, I have a network of friends who’ll help me out. My friends and I can solve just about anything. These three assets are huge for both my effectiveness and my confidence.

These three assets aren’t just for me. They’re for you, too. 

That’s the aim of my online course called “Mastering Oracle Trace Data.” In this course, I’ve bundled everything you need to claim those three assets for your own:

  1. You’ll learn the details about Oracle traces and the stories they’re trying to tell you. This is your knowledge asset.
  2. You’ll have, for the duration of your choosing, full-feature access to Method R Workbench, the most comprehensive software in the world for mining, managing, and manipulating Oracle traces. This is your tools asset.
  3. You’ll have access to our Slack channel, a global community of Oracle trace enthusiasts that can help you whenever you get stumped. You won’t be alone. You’ll have people who are there for you. This is your community asset. 

You can also fortify all three of your new assets by purchasing office hours for the course. Office hours are Zoom sessions, where you can spend time with my friends and me, discussing your questions about the material.

If you’re interested in becoming a more effective and confident optimizer, you can get started now. Just visit our course page for details.

That’s my story. I hope you’ll contact us if you’re interested.

And if you like stories like this, you’ll find a lot more in my How To Make Things Faster book, available wherever books are sold.

Blog Posts

A Design Decision

This week, my team at Method R devoted some time to an enhancement request that required an interesting design decision. This post is about the analysis behind that decision.

The enhancement request was for our flagship product called Method R Workbench. It’s an application that people use to mine, manage, and manipulate Oracle trace files.

One of its features, called mrskew, is a tool that allows a Workbench user to filter, group, and sort the raw dbcall and syscall data that Oracle Database processes write to their trace files. You can use mrskew from within the Workbench application, or from a *nix command line.

Here’s an example of a mrskew command. It’s what you would use to find out how long your program spent reading Oracle blocks from secondary storage. It will show you which blocks it read, how long they took, and how many times each block was read:

mrskew --name='db.*read' \
   --group='sprintf("%8d %8d", $p1, $p2)' \

Here’s the output:

sprintf("%8d %8d", $p1, $p2) DURATION      % CALLS     MEAN ... 
---------------------------- -------- ------ ----- --------
                  2        2 0.072918   1.0%    26 0.002805 ...
                 33   698186 0.051940   0.7%     1 0.051940 ...
                 50   339841 0.049261   0.7%     1 0.049261 ...

The important thing in this report is the meaning of the $p1 and $p2 variables. The combination of these two variables happens to represent the data block address (the file number and block number) of an Oracle block that was read by some kind of an Oracle read call. It would be nice for the report to tell you that instead of just telling you that the first two columns of numbers are the output of an sprintf function call.

We have a command-line option for that. The ‑‑group-label option lets you assign your own title for the group column. So, with some careful character counting, you could use…

‑‑group-label='    FILE    BLOCK'

…to get exactly the heading you want:

    FILE    BLOCK DURATION      % CALLS     MEAN ... 
----------------- -------- ------ ----- -------- 
       2        2 0.072918   1.0%    26  0.002805 ...
      33   698186 0.051940   0.7%     1  0.051940 ...
      50   339841 0.049261   0.7%     1  0.049261 ... 

That makes sense. Now it’s easy to see that Oracle has read one block (file #2, block #2) 26 times, consuming a total of 0.072918 seconds reading it.

The group label fits the output, only because of the careful character counting. The enhancement request was to allow the ‑‑group-label option to take an expression, not just a string. Like this:

--group-label='sprintf("%8s %8s", "FILE", "BLOCK")'

That way, he could print out the header he wanted, perfectly aligned, by just syncing his ‑‑group‑label expression to his ‑‑group expression, without having to count space characters that are literally invisible.

It’s a smart idea. The group label option should have been designed that way from the beginning. We eagerly approved the enhancement request and began thinking about the design.

When we thought it through, we ended up with two different ideas about how we could implement this new idea:

  1. Redefine ‑‑group‑label to take an expression instead of a string. mrskew will calculate the value of the expression before printing the column label.
  2. Create a new option, say, ‑‑new‑group‑label, that takes an expression as its argument. And leave ‑‑group‑label as it is.

The first idea is how the enhancement request was worded. The second idea entered our minds because the first idea creates a compatibility problem: if we change the spec of the existing ‑‑group‑label option, it will break some existing mrskew scripts. For example, these will work in Workbench 9.2.5:

--group-label="FILE BLOCK"

But if we redefine ‑‑group‑label to take an expression instead of a string, then these won’t work anymore. People will need to quote their string expressions like this:

--group-label='"FILE BLOCK"'

In the end, we decided to redefine the existing option and live with the compatibility breach.

The way we make decisions like this is that we create strenuous arguments for each idea. Here are some of the arguments we considered en route to our decision.

First, the customer experience (cognitive expenditure).

Everyone who participated in the debate had the customer experience foremost in mind. But how can we objectively measure “customer experience”? How do you structure a scientific debate about the superiority of one experience over another?

One way to do it is to measure cognitive expenditure—the amount of mental effort that a user has to invest to get the desired outcome from our software. We want to minimize cognitive expenditure, to maximize a customer’s return on investment of effort.

We began by realizing that responding to this enhancement request with one of our two ideas would necessarily force the user into one of two new regimes:

  1. The syntax of ‑‑group-label has changed.
  2. There’s a new ‑‑new-group-label option.

In regime 1, our users would have to learn the new syntax. That’s a cognitive expenditure. But it’s a one-time expenditure, which is good. The new syntax would be consistent with the existing ‑‑group syntax, which is actually a cognitive savings for our users over what we have now. However, if a customer had saved any scripts that used the old syntax, then the customer would have to convert those scripts. That’s a cognitive expenditure in a loop (one for each script), which is bad.

In regime 2, our users would have to learn about ‑‑new-group‑label, which is a cognitive expenditure. They’d still have to remember (or relearn) about ‑‑group‑label, too, which is a similar cognitive expenditure as the one in regime 1. They wouldn’t have to modify any old scripts, but they would have to make the choice of whether to use ‑‑group‑label or ‑‑new-group‑label, every time they wrote a script in the future. That’s another cognitive expenditure in a loop (one for each script), which is bad.

Second, the developer experience (technical debt).

We also need to consider the developer’s experience. We don’t want to create code that increases technical debt that makes the product unnecessarily difficult to support.

If we redefine ‑‑group-label, there’s no long-term effect to worry about. But if we add ‑‑new‑group‑label to the story, I would expect for people to wonder, why are there two such similar group label options, when one (the one that takes an expression) is clearly superior? And why does the inferior one have the better name?

At some point in the future, I envision wanting to clean up the cruft and have just the one group label feature. Naturally, the right name for it would be ‑‑group‑label. But of course, changing the spec that way would introduce a compatibility problem. To make things worse, this would occur in the future when—one would hope, if our business is growing—such a decision would impact even more customers than it would today. So then, why create the cruft in the first place? It’ll be a worse problem later than it is now.

The question that really seals the deal, is who will the change really affect? It’s really a probability question about customer experiences.

Most users who use the Workbench application will never experience our group label option directly. It’s there for everybody to use, but our Workbench has so many predefined reports built into it, most users never need to touch the group label option for themselves. When they do need to modify it, they’re usually tweaking a report that we’ve predefined for them, which is a low–cognitive-expenditure experience.

In the end, the Method R bears almost the entire cost of the ‑‑group‑label redefinition. It required us to revise:

Most users will experience the benefit of the ‑‑group‑label change, without ever knowing that, once upon a time, it changed. And that’s the way we want it. We want the product to be as smart as possible so that our customers get the most out of their investment, both cognitive and financial.

Blog Posts Videos

Fill the Glass

Today, Cary Millsap hosted the inaugural episode of his new weekly online session, called “Fill the Glass.” Episode 1 was an ask-me-anything session, covering topics including how to access the Method R workspace in Slack, advice about being your own publisher, and our GitHub repository (available now) for Cary’s and Jeff’s new book, “Tracing Oracle” (available soon).

Visit our “Fill the Glass” page for access to past recordings and future live sessions.

Blog Posts Videos

Insum Insider: How to Optimize a System with Cary Millsap

Today, Michelle Skamene, Monty Latiolais, and Richard Soule of Insum chatted with me for an hour on their live stream. I hope you’ll enjoy it as much as I did.

Blog Posts

Newsletter 2022-08-31

Here are some of the things we’ve been been working on.

Click here to subscribe.

New Book: Faster

Recently, I published a new book called Faster: How to Optimize a System. This one’s different from anything I’ve ever written. Faster is a book about how to make things go faster—mostly computers, but actually just about anything.

Faster is the single most concise material on the topic of performance analysis and optimization out there.

Jonah H. Harris, director of AI & ML at The Meet Group

Faster is a how book and a why book, but mostly, it’s a story book. It’s a personal journey that connects the dots about pretty much everything I’ve ever learned since becoming a consultant in 1989.

Cary Millsap has a gift. Yes, he’s brilliant at making things run faster, but his true genius is translating complex problems into simple, powerful ideas.

Liz Wiseman, New York Times bestselling author of Multipliers and Impact Players

Faster is meant to appeal not just to techies, but to anyone who comes into contact with technology. I wrote it in a style that company leaders and project teams and all their users can follow. It contains dozens of short chapters that you can read serially, or that you can enjoy at random.

Cary is one of the best presenters—technology or not—that you will come across, and this clearly comes through in Faster. It’s both satisfying and refreshing to see him explain effortlessly, in normal English, these topics that so many people stumble over.

Guðmundur Jósepsson, director and performance specialist at Inaris

I hope you’ll give Faster a try. If you have a copy, I’d love to hear from you. If you feel good about doing it, I hope you’ll help me spread the word and post a review at Amazon.

I can’t believe I was in my forties the first time I saw how to optimize a system the way Cary and Jeff do it. Now it doesn’t even make sense to me that anyone would try it any other way.

Richard Russell, former 7-Eleven enterprise architect

P.S.: Let me know if you’re interested in an online or live Faster workshop. I can help your whole department think clearly about performance.

New Course: Mastering Oracle Trace Data

Do you have training budget, but you’re tired of the traditional stuff?

We have what you need: nearly 14 hours of material—packaged in a way that’s perfectly suited to our post-apocalyptic travel-prohibited metaverse—it’s our course at Thinkific called Mastering Oracle Trace Data, based on the book of the same name.

It’s got tons of helpful video material, with lots of worked examples, and even a guest speaker or two.

WARNING: You’ll get the most out of this course if you have access to our Method R Workbench software product.

UN-WARNING: The course actually includes a limited-time license for Method R Workbench (and Method R Trace!), so you can experiment and solve problems while you’re experiencing the course.

MORE-WARNING: For the ultimate training experience, consider adding in some online office hours or an on-site visit (not available everywhere).

New Partner: Insum

A few months ago, I reconnected with one of my favorite Method R course alumni, Richard Soule. His employer, Insum, is a world leader in everything Oracle APEX. A few great conversations later, I’m proud to announce a new partnership between Method R Corporation and Insum.

Method R will provide software and training for Insum employees; Insum will promote Method R and help us develop new trace data collection software for APEX developers! I’m really eager to see where this partnership takes us.

New Service: Coaching

Jeff and I are experimenting with a new coaching type of service offering.

For a fixed monthly fee, we can coach your staff and help you solve problems. Of course, we can work with you through Zoom without racking up travel costs. We’re also happy to come see you every once in a while if it makes sense.

Most of our clients get everything they need from us without even having us log into their system. In other situations, we’ll work independently from time to time as a player-coach. Regardless of how the work gets done, we’re happy to teach your team everything we know.

Jeff and I have a lot of experience. We know where a lot of the traps are, and the optimization methods and tools we use give us an unfair advantage over anything else you’ve probably ever seen. Also, we have a comprehensive professional network that spans the globe.

Have a look at if you’re interested in the details.

That’s All for Now

That’s all for now; thank you for letting me catch up with you.

—Cary Millsap, President of Method R Corporation

Blog Posts

Troubleshooting a Complex Network of Overnight Batch Jobs

On Oracle-L this week is a discussion about buying Dynatrace vs. Oracle Enterprise Manager. Both tools work harmoniously together, but there’s a third choice you should consider, too: Method R Workbench. Our Workbench helps you mine unbelievable detail from Oracle trace files, even if you’re getting thousands of trace files at a time. Workbench works harmoniously with both Dynatrace and OEM and fills a gap that no other tool can fill.

Blog Posts

Fetching Oracle Trace Data from, Say, Amazon RDS

Last week, a Method R Workbench customer was having a hard time downloading Oracle trace files from Amazon RDS. The code that Amazon recommends doesn’t quite work right; it fails to copy trace file lines that are longer than 400 characters. So I wrote a replacement fetcher for him. This method should work for any Oracle implementation that permits you to access your trace data.

Blog Posts

Quantifying Buffer Cache Pressure

A few months ago, we received two trace files representing two executions of the same program. One execution took almost 14 times longer than the other. The cause of the difference was dominantly the number of “db file sequential read” calls made by the two executions. Why would two executions of the same program, processing approximately the same amount of data, present radically different “db file sequential read” call counts? One guess is that it’s pressure on the database buffer cache that’s causing the problem, but how can you prove it?

Blog Posts Videos

Some things you probably didn’t know about Oracle tracing (Chicago edition)

Today, I had the honor of speaking online to the Chicago Oracle Users Group in their “20 in 2020” webinar series. In this presentation, I talk about why it’s OK to trace everything on your system and how to do it safely, and I explain why you’d want to do that in the first place.

Some things you probably didn’t know about Oracle tracing
Blog Posts

Example of resetting user session handle attributes in JDBC

Earlier, Mark Williams showed us how in Java to clear the session handle attributes to enable our code to be more easily traced. Because I had not seen the trace data produced when using his method, I decided to construct a test to see the trace data produced when the code does and does not execute the Connection.isValid method.