What’s new in GNU make 3.82

GNU make 3.82 hit the streets last week, the first new release of the workhouse build tool in over four years. Why so long between releases? To me the answer is obvious: the tool Just Works ™, so there’s no need to churn out new releases chasing the latest development fad. But as this release shows, there is still room to innovate, without compromising on the points that make the tool so great. The two improvements I find most interesting are .ONESHELL, and changes to pattern-search behavior:
Read the rest of this entry »

Posted in News. Tags: , . 2 Comments »

Designing for high performance

Here’s the thing about high performance: you can’t just bolt it on at the end. It’s got to be baked in from day one. No doubt those of you who are experienced developers are now invoking the venerable Donald Knuth, who once said, “Premature optimization is the root of all evil.” But look at it this way: with very rare exceptions, no amount of performance tuning will turn an average system into a world class competitor.

Of course, high performance is the entire raison d’ĂȘtre for ElectricAccelerator. We knew from the start that parallelism would be the primary means of achieving our performance goals (although it’s not the only trick we used). Thanks to Amdahl’s law, we know that in order to accelerate a build by 100x, the serialized portion cannot be more than 1% of the baseline time. Thus it’s critical that absolutely everything that can be parallelized, is parallelized. And I mean everything, even the stuff that you don’t normally think about, because anything that doesn’t get parallelized disproportionately saps our performance. Anything that isn’t parallelized is a bottleneck.
Read the rest of this entry »

Using Markov Chains to Generate Test Input

One challenge that we’ve faced at Electric Cloud is how to verify that our makefile parser correctly emulates GNU Make. We started by generating test cases based on a close reading of the gmake manual. Then we turned to real-world examples: makefiles from dozens of open source projects and from our customers. After several years of this we’ve accumulated nearly two thousand individual tests of our gmake emulation, and yet we still sometimes find incompatibilities. We’re always looking for new ways to test our parser.

One idea is to generate random text and use that as a “makefile”. Unfortunately, truly random text is almost useless in this regard, because it doesn’t look anything like a real makefile. Instead, we can use Markov chains to generate random text that is very much like a real makefile. When we first introduced this technique, we uncovered 13 previously unknown incompatibilities — at the time that represented 10% of the total defects reported against the parser! Read on to learn more about Markov chains and how we applied them in practice.
Read the rest of this entry »

Makefile performance: built-in rules

Like any system that has evolved over many years, GNU Make is rife with appendages of questionable utility. One area this is especially noticeable is the collection of built-in rules in gmake. These rules make it possible to do things like compile a C source file to an executable without even having a makefile, or compile and link several source files with a makefile that simply names the executable and each of the objects that go into it.

But this convience comes at a price. Although some of the built-in rules are still relevant in modern environments, many are obsolete or uncommonly used at best. When’s the last time you compiled Pascal code, or used SCCS or RCS as your version control system? And yet every time you run a build, gmake must check every source file against each of these rules, on the off chance that one of them might apply. A simple tweak to your GNU Make command-line is all it takes to get a performance improvement of up to 30% out of your makefiles. Don’t believe me? Read on.
Read the rest of this entry »

Friday Fun: Generating Fibonacci Numbers with GNU Make

Nobody would ever claim that GNU Make is a general purpose programming language, but with a little work, we can coerce it into generating Fibonacci numbers for us. Why bother? Because we can.
Read the rest of this entry »