Ollydbg The road to success...

Today I wrote an answer on Quora about how OllyDbg was written. I thought it was interesting so I decided to cross-post it here as a blog entry:

Writing a tool like OllyDbg is not a trivial task. OllyDbg is in essence a debugger with code analysis features, however it has a big range of relatively small extra features that make it a nice thing to use. As of my qualifications, I’m the main developer of x64dbg, which is heavily influenced by OllyDbg and offers similar features but on the 64-bit platform.

The debugger part starts out quite simply. Check out Creating a Basic Debugger for a simple example of what a debugger looks like. In essence you start a process (debuggee) with a special flag so it sends information (debug events) to the debugger. The debugger then processes these events and shows information about the process to the user. Certain events (mainly single-step and breakpoint exceptions) are used to control the execution of the debuggee. If you are interested in a little more detail, check out GleeBug. It is a debug library that will be used in x64dbg soon.

The code analysis is a far more difficult problem to solve. The ‘easy’ part involves giving information about instructions pointing to memory/exported symbols and providing nice syntax highlighting. An all-white deadlisting of instructions is not very friendly to read and OllyDbg does various things to ease the experience. The hard part is actually doing control flow analysis (detecting functions and loops for instance). If you’re interested I wrote up the details of a very naive algorithm here. Some more involved algorithms are available here (by me) and here (by Nukem). Analysis is really, really hard to get working right. There are lots of corner cases and obfuscated binaries that ‘fool’ your analysis and things get especially fuzzy when control flow branches inside of other instructions or otherwise breaks the linear disassembly.

Even if you have a really good debugger and code analyser, you are still far from done. Once people start using your software, bug reports and feature request will come raining in and it can sometimes be soul-destroying to work on a project like this (especially with just intellectual motivation).

Everything needs to be very generic and getting things stable becomes a very complex problem very quickly. See the x64dbg bug tracker if you are interested in what kind of issues people have and report.

Another thing that makes OllyDbg great is that it’s so user-friendly. There are no complex commands involved and everything can be done from the graphical user interface. This is what I believe makes OllyDbg strong. Creating a GUI similar to the one of OllyDbg is rather complex (and boring). It involves custom painted components and a lot of menus.

Finally, OllyDbg offers a great plugin ecosystem, allowing people to extend/modify OllyDbg in lots of ways. People created hundreds of plugins, ranging from simple breakpoint managers to a total overhaul of the program. Creating a system like this is very challenging and pleasing everyone is really hard. It involves writing wrappers around internal functionality and make those available to the plugin writer. You also need to provide the plugins with an interface to hook in the debug loop and intercept debug events.

I hope this answers your question. Feel free to ask me if you need more information.

More content is coming soon to this blog, stay tuned!

Duncan