A Linux kernel authority and noted software engineer

This week, Stephen Ibaraki, I.S.P., has an exclusive interview with the internationally known Linux expert, speaker, software engineer, and author, Robert Love.<

Robert is contributing editor for Linux Journal and a software engineer for MontaVista Software. His kernel projects include the process scheduler, the preemptive kernel, the VM, and multiprocessing enhancements. He maintains schedutils and procps.

Amongst his most recent writings is the book, Linux Kernel Development, a practical guide to the design and implementation of the Linux kernel.

Discussion:

Q: Robert, your unique expertise will be of great value to our audience. Thank you for doing this interview.

A: It is a pleasure.

Q: How did you get into computing and into writing?

A: My interest in computers goes back nearly as far as I can remember. But while computers have always been an interest, it was not until I started programming that I could really call computing a passion. Interestingly, I found myself intrigued by operating systems very early on, even before I began programming. The concept of the inner internals fascinated me more than the outer applications. Growing involved with Linux, and then ultimately the kernel itself, has been extremely gratifying.

Similarly, I have always enjoyed writing. Awhile back, I had the opportunity to write an article for Linux Journal, which I really enjoyed. The entire writing process was very fun and it is quite rewarding to write for an audience similar to you and help teach them something new. Thankfully, Linux Journal has continued to provide an outlet for my writing.

Q: How have you resolved the roadblocks in your career? What assets and processes proved to be the most valuable?

A: I feel the most important approach to anything you do in life is to genuinely enjoy it. If you sincerely revere something, and it challenges you, a natural reaction is to challenge it back and be the best you can be at it.

Q: Describe the major challenges you face in your job and how you overcome them?

A: Some aspects of kernel development can be rather frustrating. I talk about some of them - and their solutions - in my book.

The kernel is not afforded many of the luxuries of user-space programs and, consequently, some facets of a kernel make working inside the kernel harder than working outside of it. Wrapping your head around some of the nastier bugs, for example, is sometimes particularly painful. Nonetheless, through eternal vigilance, all problems are surmountable.

Q: Describe your latest book? Who should read this book and what background should they have to get the greatest value from it? Why would someone want to carefully study this book—what makes this book particularly unique and special?

A: Linux Kernel Development is a book on the design and implementation of the latest Linux kernel. I approach the material in a hopefully fun and practical manner, with an eye toward helping developers either understand the kernel for the first time or gain a stronger grasp on kernel internals. The book is of interest to developers wishing to (better) understand the Linux kernel, either simply for fun or so they can work on kernel projects. The book is also of interest to anyone simply curious about how the kernel works.

To really get the most out of the book, the reader should be familiar with Unix and Linux systems, have at least a basic understanding of the concepts behind operating system design, and know C. Really none of these are hard requirements, however, as the book tries to explain as much as possible from the ground up. Where help is needed, I included a bibliography of some of my favorite books.

A few things make the book unique. First, its approach is rather special in that it tackles the subject in a pragmatic and fun manner. I hope that the reader can relate to the book and find the information accessible. Second, as a kernel developer both by hobby and by vocation, the book is ripe with insider views and practical tips.

Q: Which parts of the kernel appeal to you most and why?

A: My personal operating system interests are scheduling, process management, virtual memory management, and locking.

Q: What specific tips can you provide on the kernel that would be useful to kernel developers?

A: Play around. Study the source. Never give up. Have fun.

Q: What new features are in the current kernel, and what features do you see in future kernels and when do you expect them to appear?

A: The new 2.6 kernel is rich with neat new features. A lot of these are discussed in the book. Some of the most notable are a new process scheduler, two new I/O schedulers, a new block I/O layer, a device model, a reworked VM, improved scalability, a preemptive kernel, and improved threading support.

Anything is up for grabs in future kernels. We should see development start on the next kernel, 2.7, in the coming months.

Q: Provide tips from your work for Linux Journal?

A: When technical writing, the most important thing to keep in mind is your audience. You need to relate to them, understand them, and convey the information to them in terms they can understand. It does not hurt to have a sense of humor, either!

Q: Describe good kernel coding practices.

A: There is a lot behind good kernel coding practices. Sane, clean, KISS designs are important. The core kernel developers, and consequently the kernel itself, have a well-defined style. A lot of it is just an extension of the Unix style and approach taken to normal Unix programming and Unix kernels in particular. But there is certainly a bit of Linux flavor in there. Understanding that style requires understanding the source code, watching the kernel development process, and so on. Following the kernel style is definitely important.

Q: Share your more vexing problems working with the kernel.

A: Even though I am not a big user of debuggers, oftentimes the lack of an official or standard kernel debugger is a nuisance. Thankfully, in 2.5 and 2.6, we have had kgdb (a kernel debugger) patches kept very up to date and functional.

Another gripe is our release behavior. I would like it if we could switch to a release structure like GNOME's. They do time-based releases. For at least one release, I would love to try that out.

Q: What are the major problems and successes with Open Source?

A: Open Source has many positive attributes. Obviously, I believe in it. It has definitely fostered the creation of best-of-breed projects: the Apache web server, the Linux kernel, and the GNU C compiler, for example. Open source creates a meritocracy, free from political or marketing agenda, with which projects are designed and built entirely in the open. This provides the "many eyes" phenomenon, which is often described as "with enough eyes, all bugs are shallow.

But not all is perfect. Sometimes innovation needs to be pushed from somewhere other than within, and open source might not provide that. It is often argued that open source projects, which are usually run entirely by developers, lack accessibility required by or features for non-developers. While I think today that is largely an issue of the past - just look at the resounding success of GNOME, for example - it is still a cited concern.

These problems are largely issues of the past now that open source has received corporate backing and funding, which can fund work in the less-than-glamorous chores.

Q: Can you make future predictions about specific products and services coming from the Open Source movement?

A: I think we are going to see more and more markets outside of personal computing and serving moving to open source based solutions. When I say this, I think of markets such as traditional embedded, scientific computing, and consumer electronics. It is not that these markets are going to engineer their solutions in open source, although that undoubtedly would be great, but that they are going to find open source based systems the way to go due to cost, technical superiority, wide support, and vendor neutrality.

Q: What are some common problems and their solutions facing developers today?

A: These days, systems are growing more and more complex and the barrier to entry can only be greater than it ever was before. I think this happens in all aspects of society, stuff just gets more complicated and involved, but in computing the complexity is not augmented by better tools or better methodologies. The disparity between the current generation, who understand the systems because they built them, and the next generation, who face this high barrier of entry, is growing wider and wider. To counter this, it is very important that documentation and education improve to match.

Q: Do you have some stories about very challenging situations and their resolution? Any humorous stories?

A: I once spent a couple days debugging a kernel problem, trying new approaches, recompiling, rebooting my target system, and growing more and more frustrated. It turned out, for reasons still unknown, that my NFS (a networked file system) mount, which was how I was accessing the target, was not synchronizing before I rebooted the target, so I never actually tried the new code. I do not even remember what the bug I was trying to fix was or why NFS was misbehaving. But, needless to say, when I figured out the problem I put a brown paper bag over my head for a week or two. The moral of this story is to always check your build versions to make sure you are actually running what you think you are. And do not have freaky NFS bugs.

Q: What trends do you see in program development? Please make some predictions about the future, and future technologies that businesses and IT professionals should be considering.

A: Business professionals should be considering the potential of open source technology, of course!

Q: With regards to security, please provide your detailed recommendations in this area?

A: It is important that security be an initial design feature and built into the system from the very beginning. Security cannot be an afterthought. It cannot be tacked on to an existing system. The initial design needs to ask the critical questions and design the requisite answers into the system. The implementation then needs to carry through with the design and correctly implement the security.

Additionally, security through obscurity (the practice of "securing" systems by hiding how they work) is nonsense. A system that is provably correct and provably secure can have both its design and implementation handed to the attacker. It should not matter.

Finally, no matter what you do, if the implementation does not match the design (e.g., you have a bug), then you have problems. If your system is filled with buffer overflows and other errors, then no secure design in the world will help you. Many people blame these sorts of errors on the programming language or maintaining old systems. To some end, I certainly agree with them. But, at the end of the day, the errors are still occurring due to incorrect programming. Many systems are programmed in C and C++. Nearly all of Linux is. Nearly all of Windows is. It is inexcusable if either system were continually plagued with security flaws, despite not using "modern" languages. Thankfully, Linux has proven very secure over the years.

Q: Which ten resources do you find the most useful?

A: Ten? Hmm. My friends, family, and coworkers. My personal workstation and my Apple ThinkPad. I love my Apple. Three great books: Knuth's The Art of Computer Programming, Stevens' Advanced UNIX Programming, and K&R's The C Programming Language. I probably never look at these any more nowadays, except maybe the first, but they are still incredible books. LWN [http://www.lwn.net/] to keep me up-to-date with Linux news and CNN [http://www.cnn.com/] to keep me up-to-date with the rest of the world. That ought to be ten.

If you were doing this interview, what three questions would you ask of someone in your position and what would be your answers?

A: Q1: What operating system do you most respect, aside from Linux?
A1: Definitely Mac OS X. I am very impressed by it.

Q2: Linux excels in the server, workstation, and embedded markets. What is next?
A2: Definitely the desktop, as in the home or office desktop. I feel the GNOME desktop is incredibly mature and well-featured. GNOME is beyond the "catching up" phase; it is now an innovator in the desktop space. Nonetheless, I think there are still political and technical issues that remain. Politically, Linux needs to continue to win deployments in large corporations. Management needs to be aware of Linux as an option on the desktop. They need to realize it works and realize it is a great option. Companies such as Ximian, Red Hat, and SuSE are making good inroads here. Technically, Linux still lacks application support, although applications such as Open Office, Mozilla, and Evolution are certainly excellent, especially as of late. We also need better hardware management, with the kernel reporting hardware events (such as the user plugging in a digital camera) and the desktop environment handling the event. But we are getting there. I love my Linux desktop!

Q3: Vi or Emacs? Boxers or briefs?
A3: Vim and boxers.

Q: Robert, you provide some deep insights and an interesting perspective. It was a pleasure interviewing you. Thank you for sharing your wealth of knowledge with our audience.

A:  Thank you for the opportunity. Take care.