FreeBSD Release Engineering

1 Introduction

The development of FreeBSD is a very open process. FreeBSD is comprised of contributions from thousands of people around the world. The FreeBSD Project provides anonymous CVS[1] access to the general public so that others can have access to log messages, diffs (patches) between development branches, and other productivity enhancements that formal source code management provides. This has been a huge help in attracting more talented developers to FreeBSD. However, I think everyone would agree that chaos would soon manifest if write access was opened up to everyone on the Internet. Therefore only a “select” group of nearly 300 people are given write access to the CVS repository. These committers[5] are responsible for the bulk of FreeBSD development. An elected core-team[6] of very senior developers provides some level of direction over the project.

The rapid pace of FreeBSD development leaves little time for polishing the development system into a production quality release. To solve this dilemma, development continues on two parallel tracks. The main development branch is the HEAD or trunk of our CVS tree, known as “FreeBSD-CURRENT” or “-CURRENT” for short.

A more stable branch is maintained, known as “FreeBSD-STABLE” or “-STABLE” for short. Both branches live in a master CVS repository in California and are replicated via CVSup[2] to mirrors all over the world. FreeBSD-CURRENT[7] is the “bleeding-edge” of FreeBSD development where all new changes first enter the system. FreeBSD-STABLE is the development branch from which major releases are made. Changes go into this branch at a different pace, and with the general assumption that they have first gone into FreeBSD-CURRENT and have been thoroughly tested by our user community.

In the interim period between releases, monthly snapshots are built automatically by the FreeBSD Project build machines and made available for download from ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/. The widespread availability of binary release snapshots, and the tendency of our user community to keep up with -STABLE development with CVSup and “make world”[7] helps to keep FreeBSD-STABLE in a very reliable condition even before the quality assurance activities ramp up pending a major release.

Bug reports and feature requests are continuously submitted by users throughout the release cycle. Problems reports are entered into our GNATS[8] database through email, the send-pr(1) application, or via the web interface provided at http://www.FreeBSD.org/send-pr.html. In addition to the multitude of different technical mailing lists about FreeBSD, the FreeBSD Quality Assurance mailing list provides a forum for discussing the finer points of “release-polishing”.

To service our most conservative users, individual release branches were introduced with FreeBSD 4.3. These release branches are created shortly before a final release is made. After the release goes out, only the most critical security fixes and additions are merged onto the release branch. In addition to source updates via CVS, binary patchkits are available to keep systems on the RELENG_X_Y branches updated.

2 Release Process

New releases of FreeBSD are released from the -STABLE branch at approximately four month intervals. The FreeBSD release process begins to ramp up 45 days before the anticipated release date when the release engineer sends an email to the development mailing lists to remind developers that they only have 15 days to integrate new changes before the code freeze. During this time, many developers perform what have become known as “MFC sweeps”. MFC stands for “Merge From CURRENT” and it describes the process of merging a tested change from our -CURRENT development branch to our -STABLE branch.

/usr/src# cvs update -rRELENG_4 -P -d

/usr/src# cvs rtag -rRELENG_4 RELENG_4_8_BP src

/usr/src# cvs rtag -b -rRELENG_4_8_BP RELENG_4_8 src

/usr/src# cvs rtag -rRELENG_4_8 RELENG_4_8_0_RELEASE src

3 Release Building

FreeBSD “releases” can be built by anyone with a fast machine and access to a source repository. (That should be everyone, since we offer anonymous CVS! See The Handbook for details.) The only special requirement is that the vn(4) device must be available. (On -CURRENT, this device has been replaced by the new md(4) memory disk driver.) If the device is not loaded into your kernel, then the kernel module should be automatically loaded when vnconfig(8) is executed during the boot media creation phase. All of the tools necessary to build a release are available from the CVS repository in src/release. These tools aim to provide a consistent way to build FreeBSD releases. A complete release can actually be built with only a single command, including the creation of ISO images suitable for burning to CDROM, installation floppies, and an FTP install directory. This command is aptly named make release.

make release CHROOTDIR=/local3/release \
       BUILDNAME=4.7-RELEASE \
       CVSROOT=/host/cvs/usr/home/ncvs \
       RELEASETAG=RELENG_4_7_0_RELEASE
       
    

/stage/cdrom# find . -type f | sed -e 's/^\.\///' | sort > filename.txt

4 Distribution

5 Extensibility

Although FreeBSD forms a complete operating system, there is nothing that forces you to use the system exactly as we have packaged it up for distribution. We have tried to design the system to be as extensible as possible so that it can serve as a platform that other commercial products can be built on top of. The only “rule” we have about this is that if you are going to distribute FreeBSD with non-trivial changes, we encourage you to document your enhancements! The FreeBSD community can only help support users of the software we provide. We certainly encourage innovation in the form of advanced installation and administration tools, for example, but we cannot be expected to answer questions about it.

6 Lessons Learned from FreeBSD 4.4

The release engineering process for 4.4 formally began on August 1st, 2001. After that date all commits to the RELENG_4 branch of FreeBSD had to be explicitly approved by the Release Engineering Team <re@FreeBSD.org>. The first release candidate for the x86 architecture was released on August 16, followed by 4 more release candidates leading up to the final release on September 18th. The security officer was very involved in the last week of the process as several security issues were found in the earlier release candidates. A total of over 500 emails were sent to the Release Engineering Team <re@FreeBSD.org> in little over a month.

Our user community has made it very clear that the security and stability of a FreeBSD release should not be sacrificed for any self-imposed deadlines or target release dates. The FreeBSD Project has grown tremendously over its lifetime and the need for standardized release engineering procedures has never been more apparent. This will become even more important as FreeBSD is ported to new platforms.

7 Future Directions

It is imperative for our release engineering activities to scale with our growing userbase. Along these lines we are working very hard to document the procedures involved in producing FreeBSD releases.

• Parallelism - Certain portions of the release build are actually “embarrassingly parallel”. Most of the tasks are very I/O intensive, so having multiple high-speed disk drives is actually more important than using multiple processors in speeding up the make release process. If multiple disks are used for different hierarchies in the chroot(2) environment, then the CVS checkout of the ports and doc trees can be happening simultaneously as the make world on another disk. Using a RAID solution (hardware or software) can significantly decrease the overall build time.

• Cross-building releases - Building IA-64 or Alpha release on x86 hardware? make TARGET=ia64 release.

• Regression Testing - We need better automated correctness testing for FreeBSD.

• Installation Tools - Our installation program has long since outlived its intended life span. Several projects are under development to provide a more advanced installation mechanism. The libh project was one such project that aimed to provide an intelligent new package framework and GUI installation program.

8 Acknowledgements

I would like to thank Jordan Hubbard for giving me the opportunity to take on some of the release engineering responsibilities for FreeBSD 4.4 and also for all of his work throughout the years making FreeBSD what it is today. Of course the release would not have been possible without all of the release-related work done by Satoshi Asami <asami@FreeBSD.org>, Steve Price <steve@FreeBSD.org>, Bruce A. Mah <bmah@FreeBSD.org>, Nik Clayton <nik@FreeBSD.org>, David O'Brien <obrien@FreeBSD.org>, Kris Kennaway <kris@FreeBSD.org>, John Baldwin <jhb@FreeBSD.org> and the rest of the FreeBSD development community. I would also like to thank Rodney Grimes <rgrimes@FreeBSD.org>, Poul-Henning Kamp <phk@FreeBSD.org>, and others who worked on the release engineering tools in the very early days of FreeBSD. This article was influenced by release engineering documents from the CSRG[13], the NetBSD Project[10], and John Baldwin's proposed release engineering process notes[11].

9 References

[1] CVS - Concurrent Versions System http://www.cvshome.org

[2] CVSup - The CVS-Optimized General Purpose Network File Distribution System http://www.polstra.com/projects/freeware/CVSup

[3] http://pointyhat.FreeBSD.org

[4] FreeBSD Ports Collection http://www.FreeBSD.org/ports

[5] FreeBSD Committers http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/contributors/staff-committers.html

[6] FreeBSD Core Team http://www.FreeBSD.org/administration.html#t-core

[7] FreeBSD Handbook http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook

[8] GNATS: The GNU Bug Tracking System http://www.gnu.org/software/gnats

[9] FreeBSD PR Statistics http://www.FreeBSD.org/prstats/index.html

[10] NetBSD Developer Documentation: Release Engineering http://www.NetBSD.org/developers/releng/index.html

[11] John Baldwin's FreeBSD Release Engineering Proposal http://people.FreeBSD.org/~jhb/docs/releng.txt

[12] PXE Jumpstart Guide http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/pxe/index.html

[13] Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic: The Release Engineering of 4.3BSD