UPDATING revision 1.222
1$NetBSD: UPDATING,v 1.222 2011/04/10 08:29:31 plunky Exp $ 2 3This file (UPDATING) is intended to be a brief reference to recent 4changes that might cause problems in the build process, and a guide for 5what to do if something doesn't work. 6 7For a more detailed description of the recommended way to build NetBSD 8using build.sh, see the BUILDING file. 9 10Note that much of the advice in this UPDATING file was written before 11build.sh existed. Nevertheless, the advice here may be useful for 12working around specific problems with build.sh. 13 14See also: BUILDING, build.sh, Makefile. 15 16Recent changes: 17^^^^^^^^^^^^^^^ 1820110328: 19 Building the Xorg binary was moved into a subdirectory to fix 20 ordering issues with "make all". It may be necessary to remove 21 the OBJDIR for external/mit/xorg/server/xorg-server/hw/xfree86 22 if your update build fails, as the "Xorg" entry there is now a 23 directory. 24 2520110121: 26 Assembler files no longer use -traditional-cpp. This can break 27 the build of individual parts of the tree. This is handled 28 correctly by build.sh. Manual builds have to update /usr/share/mk 29 and re-run config(1) for any kernel configurations as needed. 30 3120101217: 32 The tcpdump(8) program was changed to drop privileges and chroot(2) 33 by default. It may be necessary to manually update passwd(5) and 34 group(5) in order to make the program work with existing setups. 35 3620101125: 37 The latest changes to setenv(3) dissallow setting environment 38 variables with names that contain '='. Revision 1.18 of env.c 39 assumed that this was allowed. Installing a new libc with an 40 old copy of /usr/bin/env causes env x=1 printenv | grep x= to 41 break which affects the autoconf tests for dependency finding, 42 so building gcc will end up printing: 43 checking dependency style of gcc... none 44 configure: error: no usable dependency style found 45 Fix it by rebuilding and re-installing env. 46 4720101119: 48 Recent Xorg updates in xsrc/external/mit/ may cause various build 49 or run-time problems. Delete your entire DESTDIR and OBJDIR if you 50 have any build problems with xsrc, or problems with mismatched 51 versions between xorg-server and drivers. 52 5320100604: 54 The update of ATF to 0.9 causes old tests written in shell to fail 55 unless they are rebuilt. If you are building with MKUPDATE=yes, 56 you need to clean the src/external/bsd/atf/tests/ and the src/tests/ 57 trees by hand. 58 5920100522: 60 Recent Xorg updates in xsrc/external/mit/ will cause various build 61 problems. Delete your entire DESTDIR and OBJDIR if you have any 62 build problems with xsrc. 63 6420100522: 65 private section of <ctype.h> was splitted, and now mklocale(1) 66 include ctype_local.h, so you have to make cleandir in tools/mklocale. 67 6820100520: 69 The location of the xkb compiled descriptions has changed. Please 70 remove usr/X11R7/lib/X11/xkb/compiled from your $DESTDIR. 71 7220100222: 73 The shared objects file extension has been changed from .so to 74 .pico, in order to avoid conflicts with shared libraries names 75 libXX.so. All now stale regular .so files can be removed from 76 your object directories. 77 7820100204: 79 The termcap database has been removed from the sources, 80 but has not been marked obsolete so it is not removed 81 from the system when upgrading. 82 As such, you will need to remove them from your object 83 and destination directories. 84 8520091101: 86 After updating, it may be necessary to make the 'cleandir' 87 target in src/tools/yacc/ and in src/usr.bin/yacc/ before a 88 'build.sh -u tools' or 'build.sh -u distribution'. Ditto 89 src/tools/lex/ and src/usr.bin/lex/. 90 9120091001: 92 On amd64 you must rebuild tools (to get the new binutils) 93 before building a kernel, or the build fails on cpufunc.S. 94 9520091001: 96 An error will create a ./usr/X11R7/lib/X11/xkb/compiled/xkb 97 symlink, failing the build. Delete the link, and the subdir 98 it is in, and retry your build. 99 10020090718: 101 libc build changed so that strchr() provides the extra entry 102 point for index(). Update build of libc.a (and libc_pic.a) 103 may fail because the archive contains the unwanted index.o. 104 (Similarly for strrchr() and rindex().) 105 10620090709: 107 Native Xorg was upgraded again. Builds will probably fail again 108 without a clean objdir, at least for src/external/mit/xorg. 109 11020090616: 111 Native Xorg was upgraded. Builds will need a clean objdir for 112 src/external/mit/xorg. Upgrading a system from sets will not 113 work properly yet as the /usr/X11R7/lib/X11/xkb/symbols/pc 114 subdirectory has been changed into a file, and this needs to 115 be manually "rm -r"'ed before installing xbase.tgz. 116 11720090501: 118 Several new functions were added to string.h/libc, and this 119 can cause autoconf problems during the tool build for people 120 who fail to clean out their tools objects properly. If you 121 note messages about stpcpy, stpncpy or strnlen accompanying a 122 failure during the tool build, clean out all your tools 123 objects and start again. 124 12520090325: 126 The i386 port was switched to i486 default toolchain. This requires 127 cleaning your src/tools directory and $TOOLDIR and rebuilding them. 128 12920090126: 130 The __posix_fadvise50 system call changed assembly stub type. You 131 need to 'rm -f __posix_fadvise50.*' in the libc build directory 132 to avoid using the old assembly stub. 133 13420090202: 135 pkg_install now depends on the pkgdb cache for automatic conflict 136 detection. It is recommented to rebuild the cache with 137 ``pkg_admin rebuild''. 138 13920090110: 140 time_t and dev_t have been bumped to 64 bit quantities. To upgrade: 141 1. Make sure your kernel has COMPAT_50 in it. Build and install. 142 This is needed even in the MODULAR kernel because there is 143 conditionally compiled code in rtsock.c. 144 2. make sure build.sh completes and the binaries in a chroot work 145 before installing. 146 3. If you don't use build.sh and you build directly to root, and 147 your build breaks in the middle, don't despair. Make sure headers 148 are installed properly, and start building libraries first libc 149 and libutil, install them and then continue building all the 150 libraries in src/lib and src/gnu/lib and install them. Once 151 the new libraries are installed, you can restart the build. 152 4. If you compile packages and you notice link time warnings, 153 rebuild the required packages to update their shared libraries. 154 Any package you rebuild will require rebuilding all the packages 155 that depend on it. 156 5. Next time you run pwd_mkdb with the new binary, the file 157 will be upgraded and it will not be backwards compatible. 158 6. The utmpx/wtmpx files (/var/run/utmpx and /var/log/wtmpx, see 159 lastlogx(5)) have been versioned, and there is a heuristic 160 for utmp. You are better off removing the old files after 161 upgrading. The automated clearing of /var/run during 162 boot, and the automated rotating of files in /var/log by 163 newsyslog(8), may mean that you do not have to remove the 164 files manually. 165 7. The optional accounting file (/var/account/acct, see 166 accton(8)) has not been versioned, and will need to be 167 removed. The automatic rotation of the accounting file by 168 /etc/daily limits the bad consequences of failure to remove 169 the file. 170 8. Application software that writes time_t to binary files on 171 disk will break or need attention. Most notably: if you are 172 using PostgreSQL < 8.4, you need to dump your databases, 173 rebuild PostgreSQL with the new time_t, then restore. 174 17520081219: 176 config(1) has been updated, and one of the files it creates - 177 swapnetbsd.o - has changed format. You need to rebuild config 178 (done automatically by build.sh) and then you need to rerun 179 config on all kernel configuration files before rebuilding those 180 kernels. 181 18220081205: 183 If you build with MKX11=no, you should remove /etc/rc.d/xdm and 184 /etc/rc.d/xfs from DESTDIR because those files were moved to the xetc 185 set and will appear as extra files for MKX11=no update builds. 186 18720081122: 188 On i386, various kernel options(4) in GENERIC including 189 file systems have been disabled and moved into kernel modules. 190 Before trying a new GENERIC kernel, you have to prepare the 191 following files as well as a new GENERIC kernel: 192 193 - build and install kernel modules from src/sys/modules 194 195 - install the latest bootloader, which will load a module 196 for the file system from which the kernel is loaded automatically 197 198 If you have to load your kernel from a file system which is not of 199 the same type as the root file system, you have to load the necessary 200 file system module manually on the boot prompt or in the boot.cfg file. 201 20220080827: 203 If you built and installed a libc from sources between 204 2008/08/20 and 2008/08/26 you got a broken strtouq(3) 205 which results in false errors reported by lint(1). 206 Since this breaks the libc build itself, manual help is 207 needed -- lint must be disabled temporarily, e.g.: 208 $ (cd lib/libc && make MKLINT=no dependall install) 209 21020080813: 211 MKDEBUG build was broken because the .depend files did not know 212 about .go files. You need to remove all .depend files and rebuild. 213 21420080802: 215 A regression in binary compatibility for pthread_mutex_t has 216 been fixed. Unfortunately, the price is breaking compatibility 217 for -current. 218 219 Threaded programs (using libpthread) and C++ programs (using 220 libstdc++) compiled after 20070907 and before 20080802 need to 221 be recompiled. 222 223 One way to find affected pkgsrc packages: 224 225 $ grep REQUIRES=/usr/lib/libpthread /var/db/pkg/*/+BUILD_INFO 226 $ grep REQUIRES=/usr/lib/libstdc++ /var/db/pkg/*/+BUILD_INFO 227 22820080731: 229 WAPBL (metadata journaling support) has been added, but at this 230 time isn't backwards compatible with pre-WAPBL aware kernels 231 and userland (fsck_ffs in particular). Please make sure you 232 don't use a journaled filesystem with an older kernel/userland, 233 especially an uncleanly mounted journaled filesystem. WAPBL 234 also requires the super block to be in the UFS2 format. You 235 can use fsck_ffs -c 4 to update the superblock format. 236 23720080721: 238 Assembler warnings are now fatal if $WARNS>0 and $NOGCCERROR 239 isn't defined. 240 24120080531: 242 The ioctl number of DRVSUSPENDDEV command on /dev/drvctl changed 243 from 125 (conflicted with DRVCTLCOMMAND) to 129. The drvctl(8) 244 utility needs to be rebuilt and reinstalled as older binaries 245 won't work correctly. The following sequence of commands: 246 247 $ (cd sys/sys/ && nbmake-$arch includes) 248 $ (cd sbin/drvctl/ && nbmake-$arch clean) 249 $ (cd sbin/drvctl/ && nbmake-$arch all) 250 251 leaves new drvctl utility in sbin/drvctl build directory. 252 25320080503: 254 The <bsd.lib.mk> variable MKPRIVATELIB was renamed to LIBISPRIVATE. 255 25620080521: 257 For a while, unprivileged UPDATE builds would fail to 258 succeed at the checkflist stage, complaining that 259 ${DESTDIR}/stand/<arch>/ did not exist. A fix for this 260 problem was committed to share/mk/bsd.kmodule.mk, revision 1.9. 261 If you already hit this problem, update the .mk file, 262 remove ${DESTDIR}/stand/<arch>, and re-run the build. 263 26420080303: 265 Linker warnings are now fatal if $WARNS>0. 266 26720080126: 268 The posix_fadvise system call has been changed from an assembly 269 stub, to a c file that calls an assembly stub. You need to 270 'rm -f posix_fadvise.* .depend' in the libc build directory to 271 avoid using the old assembly stub. 272 27320071209: 274 The acpiec(4) driver has been split into two attachments. If you 275 get ACPI errors before the attachment, please update your kernel 276 configuration file appropriately or see GENERIC for more details. 277 27820071115: 279 The it(4) driver has been renamed to itesio(4) and the old port 280 argument specified in the kernel configuration file is not valid 281 anymore. The itesio(4) driver now uses the Super I/O address port 282 rather than the EC address port. Please update your kernel 283 configuration file appropriately or see GENERIC for more details. 284 28520071028: 286 The pccons(4) driver has been removed from the NetBSD/shark port. 287 You need to update any custom kernel configuration file you have 288 to remove any references to pccons (which includes removing the 289 now useless XSERVER option) and replace them with the correct 290 entries for the wscons driver. See the GENERIC configuration file 291 for more details. 292 29320070913: 294 A latent bug in dhclient/dhcpd that caused it to be unable to 295 enumerate interfaces was fixed. The bug began to cause 296 problems after 20070911 when the kernel's SIOCGIFCONF 297 implementation was repaired. From 20070529 to 20070911 racoon 298 could not enumerate interfaces. (These are noted because 299 normal kernel/userspace version matching hygiene is not 300 sufficient to avoid this problem.) Ensure that both kernel 301 and userland are from after 20070913. 302 30320070703: 304 nbinstall has been renamed ${MACHINE_GNU_ARCH}-install. It 305 calls the target-specific strip program, and the logic to pass 306 down STRIP from make has been removed. This requires a 307 re-installation of tools. 308 30920070422: 310 The way OS emulations lookup filenames inside the emulation root 311 has been changed. Rather than modify the pathname (and copy back 312 to userspace) namei() and lookup() directly check the emulation 313 root. One side effect is that absolute symlinks inside the emulated 314 root file system will be relative to that file system - unless they 315 start /../ this is useful when the emulated root is a real install 316 that has such links. 317 This might affect symlinks that have been added to reference outside 318 the emulated root. 319 32020070412: 321 The pckbc driver on sgimips IP32 has been removed. Use macekbc 322 instead. See the GENERIC32_IP3x kernel configuration for an 323 example. 324 32520070319: 326 src/lib/libc/Makefile revision 1.129 broke libc and ld.elf_so 327 on many platforms due to incorrect flags settings. If you 328 updated and built after about 20070315, do "nbmake-$arch 329 cleandir" in src/lib/libc and src/libexec/ld.elf_so to force a 330 rebuild of object files that might have been built 331 incorrectly, and ensure that you have at least 332 src/lib/libc/Makefile 1.130. 333 33420070210: 335 src/sys/sys/{sa.h,savar.h} were removed. 336 find ${OBJDIR} \( -name .depend -o -name '*.d' \) -print \ 337 | xargs egrep -l '/sa.h|/savar.h' | xargs rm 338 will allow dependencies on those files to get get rebuilt 339 34020070209: 341 The threading model was changed when the newlock2 branch 342 was merged to NetBSD-current. If you boot with a new 343 kernel (version 4.99.10), then you also need a new pthread 344 library (/usr/lib/libpthread.so.0.7). If you boot with 345 an old kernel, then you need the old pthread library 346 (/usr/lib/libpthread.so.0.6). Provided you keep the kernel and 347 the pthread library in sync, old threaded applications should 348 continue to work with an old or new kernel. Note that named(8) 349 is the only threaded application in the base system. 350 35120061214: 352 Following the move of string_to_flags() and flags_to_string() 353 from the bin/ls/ sources to libutil, users doing UPDATE builds 354 will need to do a "make cleandir" in 355 tools/mtree/, tools/makefs/, tools/binstall/, tools/pax/, 356 bin/pax/, bin/ls/, usr.sbin/mtree/, usr.sbin/makefs/, 357 usr.bin/xinstall/, libexec/ftpd/, rescue/, as well 358 as the installation images in distrib/ 359 in order to excise stale references to the old stat_flags.h header 360 file in the ls sources -- stat_flags.h has been removed. 361 36220061108: 363 The configure script used in the src/tools/gcc compiler has been 364 changed to indicate that our libc has ssp support built-in and 365 does not depend on -lssp and -lssp-nonshared. You'll need to 366 make clean in src/tools/gcc first to rebuild the compiler. 367 36820061009: 369 The sysctl variables net.inet{,6}.tcp{,6}.newreno are no longer 370 available. Use net.inet{,6}.tcp{,6}.congctl.selected instead. 371 37220060814: 373 The vt, vidcconsole, kbd, and rpckbd drivers on acorn32 have been 374 withdrawn. Use vidcvideo and pckbd instead. See the GENERIC 375 kernel configuration for an example. X servers from the last 376 few years should cope. 377 37820060703: 379 MPACPI is no more. We always configure PCI interrupts using ACPI 380 if we have an ACPI kernel. The option MPACPI_SCANPCI has been renamed 381 to ACPI_SCANPCI. Thanks to work from fvdl. 382 38320060627: 384 socket(2) has changed, and its system call has been versioned. 385 For userlands with the old version of socket(2), make sure that 386 your kernel has 'options COMPAT_30' set, or else 'bad system call' 387 errors will result. 388 389Hints for a more successful build: 390^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 391 Use build.sh, but do not use its "expert mode": 392 This will automatically build the tools in the 393 correct order, and it will keep the tools and the 394 new build products from interfering with the running 395 system. This will allow you to ignore most of the 396 other advice in this file. 397 Build a new kernel first: 398 This makes sure that any new system calls or features 399 expected by the new userland will be present. This 400 helps to avoid critical errors when upgrading. 401 Use object directories: 402 This helps to keep stale object 403 files from polluting the build if a Makefile "forgets" 404 about one. It also makes it easier to clean up after 405 a build. It's also necessary if you want to use the 406 same source tree for multiple machines. 407 To use object directories with build.sh: 408 a) invoke build.sh with the "-M" or "-O" options. 409 To use object directories without using build.sh: 410 a) cd /usr/src ; make cleandir 411 b) Add "OBJMACHINE=yes" to /etc/mk.conf 412 c) Add "MKOBJDIRS=yes" to /etc/mk.conf 413 d) cd /usr/src ; make build 414 Note that running "make obj" in a directory will create 415 in obj.$MACHINE directory. 416 Build to a DESTDIR: 417 This helps to keep old installed files (especially libraries) 418 from interfering with the new build. 419 To build to a DESTDIR with build.sh, use the "-D" option. 420 To build to a DESTDIR without using build.sh, set the DESTDIR 421 environment variable before running make build. It should be 422 set to the pathname of an initially empty directory. 423 Problems: if you do not use build.sh, you might need to 424 update critical utilities without using DESTDIR since 425 nothing is executed from what is installed in DESTDIR. 426 (See critical utils, below.) 427 Build often: 428 This keeps critical utilities current enough to not choke 429 on any other part of the source tree that depends on up to 430 date functionality. If you use build.sh, you should not have 431 this problem. 432 433What to do if things don't work: 434^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 435When things don't work there is usually a few things that commonly 436should be done. 437 1) make includes 438 This should be done automatically by make build. 439 2) cd share/mk && make install 440 Again, automatically done by make build. 441 442Failsafe rebuild of a small part of the tree: 443^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 444To make sure you rebuild something correctly you want to do 445something like the following: 446 1) Make sure the includes and .mk files are up to date. 447 2) Make sure any program used to build the particular 448 utility is up to date. (yacc, lex, etc...) 449 3) cd ...path/to/util... 450 make cleandir 451 rm ...all obj directories... 452 make cleandir # yes, again 453 make obj 454 make depend && make 455 456Failsafe rebuild of the entire tree: 457^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 458If you really want to make sure the source tree is clean and 459ready for a build try the following. Note that sourcing /etc/mk.conf 460(a make(1) Makefile) in this manner is not right, and will not work 461for anyone who uses any make(1) features in /etc/mk.conf. 462 463---cut here--- 464#!/bin/sh 465. /etc/mk.conf 466 467if [ -z $NETBSDSRCDIR ] ; then 468 NETBSDSRCDIR=/usr/src 469fi 470if [ \! -d $NETBSDSRCDIR ] ; then 471 echo Unable to find sources 472 exit 1 473fi 474find $NETBSDSRCDIR -name \*.o -o -name obj.\* -o -name obj -exec rm \{\} \; 475 476if [ -z $BSDOBJDIR ] ; then 477 BSDOBJDIR=/usr/obj 478fi 479if [ -d $BSDOBJDIR ] ; then 480 rm -rf $BSDOBJDIR 481fi 482 483cd $NETBSDSRCDIR && make cleandir 484 485---cut here--- 486 487Critical utilities: 488^^^^^^^^^^^^^^^^^^^ 489 usr.bin/compile_et 490 usr.bin/make 491 usr.bin/yacc 492 usr.bin/lex 493 usr.bin/xlint 494 usr.bin/config 495 496Other problems and possible solutions: 497^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 498Symptom:Complaints involving a Makefile. 499Fix: Rebuild usr.bin/make: 500 cd usr.bin/make && make && make install 501 Or, a failsafe method if that doesn't work: 502 cd usr.bin/make && cc *.c */*.c -I . -o make && mv make /usr/bin 503 504Fix: Make sure .mk files are up to date. 505 cd share/mk && make install 506 507Symptom:Kernel `config' fails to configure any kernel, including GENERIC. 508Fix: Rebuild usr.bin/config 509 510Symptom: 511Fix: Rebuild usr.bin/yacc 512 513Symptom: 514Fix: Rebuild usr.bin/lex 515 516Symptom: 517Fix: rm /usr/lib/libbfd.a 518 519Symptom:Obsolete intermediate files are used during compilation 520Fix: Try the following sequence of commands in the directory in question. 521 make cleandir; rm `make print-objdir`; make cleandir; make obj 522 (If you built the tree without "make obj" in the past, obsolete files 523 may remain. The command tries to clean everything up) 524 525Symptom:.../sysinst/run.c:xx: warning: initialization from incompatible 526 pointer type 527Fix: Rebuild and install usr.bin/menuc 528 529Symptom:mklocale not found during build in share/locale/ctype 530Fix: Build and install usr.bin/mklocale 531 532Symptom:undefined reference to `__assert13' or `__unsetenv13' 533Fix: Rebuild and install lib/libc 534 535Symptom:usr.bin/config fails to build. 536Fix: Try building with -DMAKE_BOOTSTRAP added to CFLAGS in Makefile. 537 538Symptom:undefined reference to `getprogname' or `setprogname' 539Fix: Rebuild and install lib/libc 540 541Symptom:lint does not understand the '-X' option 542Fix: May need to build & install libs with NOLINT=1 before rebuilding lint 543