Organizational Cartography: what kind of development shop are you?

Image courtesy Norman B. Leventhal Map Center at the BPL

If you’ve ever worked on a significant software development project you’ve probably looked at some part of your process and said ‘we have to change this!’ More often than not the topic is the software production process: that problematic build-test-release cycle. We can all identify places where our process is slow or broken; so why do some initiatives succeed while others fail miserably? If you’re going to make a change which affects six people: stop reading – just go do it! Sixty? You’ll have to get buy-in from colleagues. Six hundred? You’re going to have to talk to your Director of Engineering. Navigating those waters benefits from drawing a map: you should know who your change will impact, how it will benefit them, and where the low-hanging fruit are. Over my four years with Electric Cloud I’ve successfully (and sometimes not so successfully!) sailed the process-improvement trade route; in this article I’m going to give you a heuristic in order to profile your own team and develop a navigation map. Read the rest of this entry »

Andy Glew on the need for discovered dependencies

Andy Glew’s written a piece over on his blog opining about the need for automatic discovery of dependencies and some applications of having that data. Understanding dependencies is a requirement for any (reasonable) build tool, and all build tools allow the specification of dependencies – even shell script: specifying a serial order lets you define fully implicit dependencies! – so if you had a source for complete, accurate dependency information you could generate a makefile, or an ant build.xml, or a shell script with a serial order, from that information, thus allowing a build to use any tool.

One of the challenges to that approach in practice is that most developers have a build tool of choice (make, ant, SCons, Visual Studio, etc) and there’s a resistance to using a meta-language to describe all the other bits (compiler options, libraries, variables) which are the other necessary parts of a build system. Getting developers to write in your tool/language is the sticky wicket, since a developer is going to need to change all the little details at some point, and that developer is going to be more comfortable doing it in the tool she’s accustomed to. And some tools don’t easily map 1:1 – lots of things which work well in SCons are hard/impossible in make, and migrating make to SCons is non-trivial.

In the end, Andy’s right on, though. The dependencies are the hard, magic bit – and in large software systems with modular structures and recursive build processes, getting accurate depdendencies is *hard* – so if you can get good deps the tool platform problems can be solved in lots of ways.

Andy has a couple questions in his article I should address, since I work for Electric Cloud:

"Past conversations, either with Ousterhout or with EC sales folk,
lead me
(1) unsure as to whether they have done the truly automatic thing for dependencies
(2) sure that they are not "diversity tolerant"
- they are yet another tool that requires everyone, the whole project, to use their build tool."

1) Yep, we really have done the truly automatic thing for dependencies – ElectricAccelerator tracks at the file-system level what files are read and written, collecting the data required to build the dependency graph for a build (and also letting ElectricAccelerator know when a target was prematurely fired, in order to revert it and re-run once all implicit deps are satisfied). The approach Andy indicates earlier in his article – using strace to trap this info – is a great first pass; strace slows a build down 5x or more, though, and isn’t available on Windows. ElectricAccelerator is built for speed, so we’re more elegant about how we do the data collection.

2) ElectricAccelerator assumes as a commercial requirement that developers are tightly bound to their tools, whether by choice, edict, or (mis)fortune. A commercial product which requires code change will fail, so it’s important to us to support specific tools. So we don’t actually require everyone to use our tool – we emulate *your* tool in most cases – and compatibility with the original infrastructure is maintained.

It’s an interesting idea, writing a tool-indepdenent build language, or writing a tool which can transition from one implementation (ANT) to another (Make) – and there are commercial companies which have such products available. Electric Cloud has focused on what we see as the fundamental hard problem – seamlessly discovering dependencies – and uses it to get to what I think Andy’s end goal is:

such a meta-build might make recursive make correct.

Which is exactly where ElectricAccelerator ends up: we fix the problems identified by Miller in Recursive Make Considered Harmful by allowing all implicit dependencies to be respected (and some other neat stuff too, like merging sub-makes transparently.)

Automatic Dependency Management with ElectricAccelerator

One of the well known problems with make is that it’s a real nuisance to completely specify all the dependencies in your project. For example, if you have a file main.c in your project, you probably already have a dependency like this in your makefile:

main.o: main.c

However, if main.c includes logging.h, you technically need to have a dependency like the following too — but you probably don’t have it:

main.o: logging.h

The kicker is that without this additional dependency, make will fail to realize that it needs to rebuild main.o if there is a change in logging.h. Of course, this wreaks havoc on your ability to do reliable incremental builds.

Read the rest of this entry »

Untangling Parallel Build Logs

I spend most of my time with ElectricAccelerator working on the “big” features — performance, scalability, fault-tolerance. It’s easy to forget that there are a ton of “little” features that can themselves make a big difference in the value of the system. Case in point: the build log. If you have any experience with parallel build systems, you know what a mess the build log becomes because you have any number of parallel commands all dumping output to a single logfile simultaneously. The output from each command gets interleaved with the output from other commands. Worse, the error messages get jumbled up too, so it becomes difficult to tell which commands are producing the errors.

I was reminded of this issue when it popped up again on the GNU make-help mailing list. Take a look at this recent post:

Read the rest of this entry »