Chapter 5 Troubleshooting

The most likely reason is the difference between physical memory addresses and virtual addresses.

The convention for most PC hardware is to use the memory area between 3.5 GB and 4 GB for a special purpose (usually for PCI). This address space is used to access PCI hardware. As a result real, physical memory can not be accessed by that address space.

What happens to the memory that should appear in that location is dependent on your hardware. Unfortunately, some hardware does nothing and the ability to use that last 500 MB of RAM is entirely lost.

Luckily, most hardware remaps the memory to a higher location so that it can still be used. However, this can cause some confusion if you watch the boot messages.

On a 32-bit version of FreeBSD, the memory appears lost, since it will be remapped above 4 GB, which a 32-bit kernel is unable to access. In this case, the solution is to build a PAE enabled kernel. See the entry on memory limits and about different memory limits on different platforms for more information.

On a 64-bit version of FreeBSD, or when running a PAE-enabled kernel, FreeBSD will correctly detect and remap the memory so it is usable. During boot, however, it may seem as if FreeBSD is detecting more memory than the system really has, due to the described remapping. This is normal and the available memory will be corrected as the boot process completes.

With SCSI drives, the drive should be capable of re-mapping these automatically. However, many drives ship with this feature disabled.

To enable bad block remapping edit the first device page mode, which can be done by giving the command (as root)

# camcontrol modepage sd0 -m 1 -e -P 3

and changing the values of AWRE and ARRE from 0 to 1:

AWRE (Auto Write Reallocation Enbld):  1
ARRE (Auto Read Reallocation Enbld):  1

Modern IDE drives also have bad block remapping features in the controller, and they ship with this feature turned on.

If you see warnings about bad blocks (on either type of drive), it is time to consider replacing the drive. You might be able to use the drive manufacturer's diagnostic program to lock out those bad blocks, but at best this will buy you some time.

This is basically a known problem. The EISA on-board SCSI controller in the HP Netserver machines occupies EISA slot number 11, so all the “true” EISA slots are in front of it. Alas, the address space for EISA slots >= 10 collides with the address space assigned to PCI, and FreeBSD's auto-configuration currently cannot handle this situation very well.

So now, the best you can do is to pretend there is no address range clash :), by bumping the kernel option EISA_SLOTS to a value of 12. Configure and compile a kernel, as described in the Handbook entry on configuring the kernel.

Of course, this does present you with a chicken-and-egg problem when installing on such a machine. In order to work around this problem, a special hack is available inside UserConfig. Do not use the “visual” interface, but the plain command-line interface there. Simply type the following command at the prompt and install your system as usual:

eisa 12
quit

While it is recommended you compile and install a custom kernel anyway.

Hopefully, future versions will have a proper fix for this problem.

This is usually caused by an interrupt conflict (e.g., two boards using the same IRQ). Boot with the -c option and change the ed0/de0/... entry to match your board.

If you are using the BNC connector on your network card, you may also see device timeouts because of bad termination. To check this, attach a terminator directly to the NIC (with no cable) and see if the error messages go away.

Some NE2000 compatible cards will give this error if there is no link on the UTP port or if the cable is disconnected.

This card has a bad habit of losing its configuration information. Refresh your card's settings with the DOS utility 3c5x9.exe.

If the only problem is that the printer is terribly slow, try changing your printer port mode as discussed in the Printer Setup section of the Handbook.

Signal 11 errors are caused when your process has attempted to access memory which the operating system has not granted it access to. If something like this is happening at seemingly random intervals then you need to start investigating things very carefully.

These problems can usually be attributed to either:

1. If the problem is occurring only in a specific application that you are developing yourself it is probably a bug in your code.

2. If it is a problem with part of the base FreeBSD system, it may also be buggy code, but more often than not these problems are found and fixed long before us general FAQ readers get to use these bits of code (that is what -current is for).

In particular, a dead giveaway that this is not a FreeBSD bug is if you see the problem when you are compiling a program, but the activity that the compiler is carrying out changes each time.

For example, suppose you are running make buildworld, and the compile fails while trying to compile ls.c into ls.o. If you then run make buildworld again, and the compile fails in the same place then this is a broken build -- try updating your sources and try again. If the compile fails elsewhere then this is almost certainly hardware.

What you should do:

In the first case you can use a debugger e.g. gdb(1) to find the point in the program which is attempting to access a bogus address and then fix it.

In the second case you need to verify that it is not your hardware at fault.

Common causes of this include:

1. Your hard disks might be overheating: Check the fans in your case are still working, as your disk (and perhaps other hardware might be overheating).

2. The processor running is overheating: This might be because the processor has been overclocked, or the fan on the processor might have died. In either case you need to ensure that you have hardware running at what it is specified to run at, at least while trying to solve this problem. i.e. Clock it back to the default settings.

   If you are overclocking then note that it is far cheaper to have a slow system than a fried system that needs replacing! Also the wider community is not often sympathetic to problems on overclocked systems, whether you believe it is safe or not.

3. Dodgy memory: If you have multiple memory SIMMS/DIMMS installed then pull them all out and try running the machine with each SIMM or DIMM individually and narrow the problem down to either the problematic DIMM/SIMM or perhaps even a combination.

4. Over-optimistic Motherboard settings: In your BIOS settings, and some motherboard jumpers you have options to set various timings, mostly the defaults will be sufficient, but sometimes, setting the wait states on RAM too low, or setting the “RAM Speed: Turbo” option, or similar in the BIOS will cause strange behavior. A possible idea is to set to BIOS defaults, but it might be worth noting down your settings first!

5. Unclean or insufficient power to the motherboard. If you have any unused I/O boards, hard disks, or CD-ROMs in your system, try temporarily removing them or disconnecting the power cable from them, to see if your power supply can manage a smaller load. Or try another power supply, preferably one with a little more power (for instance, if your current power supply is rated at 250 Watts try one rated at 300 Watts).

You should also read the SIG11 FAQ (listed below) which has excellent explanations of all these problems, albeit from a Linux® viewpoint. It also discusses how memory testing software or hardware can still pass faulty memory.

Finally, if none of this has helped it is possible that you have just found a bug in FreeBSD, and you should follow the instructions to send a problem report.

There is an extensive FAQ on this at the SIG11 problem FAQ.

The FreeBSD developers are very interested in these errors, but need some more information than just the error you see. Copy your full crash message. Then consult the FAQ section on kernel panics, build a debugging kernel, and get a backtrace. This might sound difficult, but you do not need any programming skills; you just have to follow the instructions.

This is a known problem with the ATI Mach64 video card. The problem is that this card uses address 2e8, and the fourth serial port does too. Due to a bug (feature?) in the sio(4) driver it will touch this port even if you do not have the fourth serial port, and even if you disable sio3 (the fourth port) which normally uses this address.

Until the bug has been fixed, you can use this workaround:

1. Enter -c at the boot prompt. (This will put the kernel into configuration mode).

2. Disable sio0, sio1, sio2 and sio3 (all of them). This way the sio(4) driver does not get activated -- no problems.

3. Type exit to continue booting.

If you want to be able to use your serial ports, you will have to build a new kernel with the following modification: in /usr/src/sys/dev/sio/sio.c (or in /usr/src/sys/pc98/cbus/sio.c for pc98) find the one occurrence of the string 0x2e8 and remove that string and the preceding comma (keep the trailing comma). Now follow the normal procedure of building a new kernel.

Due to the manner in which FreeBSD gets the memory size from the BIOS, it can only detect 16 bits worth of Kbytes in size (65535 Kbytes = 64 MB) (or less... some BIOSes peg the memory size to 16 MB). If you have more than 64 MB, FreeBSD will attempt to detect it; however, the attempt may fail.

To work around this problem, you need to use the kernel option specified below. There is a way to get complete memory information from the BIOS, but we do not have room in the bootblocks to do it. Someday when lack of room in the bootblocks is fixed, we will use the extended BIOS functions to get the full memory information... but for now we are stuck with the kernel option.

options MAXMEM=n

Where n is your memory in Kilobytes. For a 128 MB machine, you would want to use 131072.

Normally, FreeBSD determines a number of kernel parameters, such as as the maximum number of files that can be open concurrently, from the amount of memory installed in the system. On systems with one gigabyte of RAM or more, this “auto sizing” mechanism may choose values that are too high: while starting up, the kernel allocates various tables and other structures that fill up most of the available kernel memory. Later on, while the system is running, the kernel has no more space left for dynamic memory allocations, and panics.

Compile your own kernel, and add the VM_KMEM_SIZE_MAX to your kernel configuration file, increasing the maximum size to 400 MB (options VM_KMEM_SIZE_MAX=419430400). 400 MB appears to be sufficient for machines with up to 6 GB of memory.

The panic indicates that the system ran out of virtual memory for network buffers (specifically, mbuf clusters). You can increase the amount of VM available for mbuf clusters by following the instructions in the Network Limits section of the Handbook.

The FreeBSD kernel will only allow a certain number of processes to exist at one time. The number is based on the kern.maxusers sysctl(8) variable. kern.maxusers also affects various other in-kernel limits, such as network buffers (see this earlier question). If your machine is heavily loaded, you probably want to increase kern.maxusers. This will increase these other system limits in addition to the maximum number of processes.

To adjust your kern.maxusers value, see the File/Process Limits section of the Handbook. (While that section refers to open files, the same limits apply to processes.)

If your machine is lightly loaded, and you are simply running a very large number of processes, you can adjust this with the kern.maxproc tunable. If this tunable needs adjustment it needs to be defined in /boot/loader.conf. The tunable will not get adjusted until the system is rebooted. For more information about tuning tunables, you should see the loader.conf(5) and sysctl.conf(5) manual pages. If these processes are being run by a single user, you will also need to adjust kern.maxprocperuid to be one less than your new kern.maxproc value. (It must be at least one less because one system program, init(8), must always be running.)

To make a sysctl change permanent place the proper value in /etc/sysctl.conf. More information about system tuning with sysctl(8) can be found at the Tuning with sysctl section of the Handbook.

The logic that attempts to detect an out of date /var/db/kvm_*.db files sometimes fails and using a mismatched file can sometimes lead to panics.

If this happens, reboot single-user and do:

# rm /var/db/kvm_*.db

This is a conflict with an Ultrastor SCSI Host Adapter.

During the boot process enter the kernel configuration menu and disable uha0, which is causing the problem.

Your motherboard lacks the external logic to support automatic termination. Switch your SCSI BIOS to specify the correct termination for your configuration rather than automatic termination. The ahc(4) driver cannot determine if the external logic for cable detection (and thus auto-termination) is available. The driver simply assumes that this support must exist if the configuration contained in the serial EEPROM is set to “automatic termination”. Without the external cable detection logic the driver will often configure termination incorrectly, which can compromise the reliability of the SCSI bus.

You can find a detailed answer for this question in the Handbook.

The remote machine may be setting your terminal type to something other than the cons25 terminal type required by the FreeBSD console.

There are a number of possible work-arounds for this problem:

• After logging on to the remote machine, set your TERM shell variable to ansi or sco if the remote machine knows about these terminal types.

• Use a VT100 emulator like screen at the FreeBSD console. screen offers you the ability to run multiple concurrent sessions from one terminal, and is a neat program in its own right. Each screen window behaves like a VT100 terminal, so the TERM variable at the remote end should be set to vt100.

• Install the cons25 terminal database entry on the remote machine. The way to do this depends on the operating system on the remote machine. The system administration manuals for the remote system should be able to help you here.

• Fire up an X server at the FreeBSD end and login to the remote machine using an X based terminal emulator such as xterm or rxvt. The TERM variable at the remote host should be set to xterm or vt100.

The reasons for this behavior are explained by the following email, posted to the FreeBSD general questions mailing list by Peter Wemm <peter@FreeBSD.org>, in answer to a question about an internal modem that was no longer found after an upgrade to FreeBSD 4.X (the comments in [] have been added to clarify the context).

The PNP bios preconfigured it [the modem] and left it laying around in port space, so [in 3.X] the old-style ISA probes “found” it there.

Under 4.0, the ISA code is much more PnP-centric. It was possible [in 3.X] for an ISA probe to find a “stray” device and then for the PNP device ID to match and then fail due to resource conflicts. So, it disables the programmable cards first so this double probing cannot happen. It also means that it needs to know the PnP IDs for supported PnP hardware. Making this more user tweakable is on the TODO list.

To get the device working again requires finding its PnP ID and adding it to the list that the ISA probes use to identify PnP devices. This is obtained using pnpinfo(8) to probe the device, for example this is the output from pnpinfo(8) for an internal modem:

# pnpinfo
Checking for Plug-n-Play devices...

Card assigned CSN #1
Vendor ID PMC2430 (0x3024a341), Serial Number 0xffffffff
PnP Version 1.0, Vendor Version 0
Device Description: Pace 56 Voice Internal Plug & Play Modem

Logical Device ID: PMC2430 0x3024a341 #0
        Device supports I/O Range Check
TAG Start DF
    I/O Range 0x3f8 .. 0x3f8, alignment 0x8, len 0x8
        [16-bit addr]
    IRQ: 4  - only one type (true/edge)

[more TAG lines elided]

TAG End DF
End Tag

Successfully got 31 resources, 1 logical fdevs
-- card select # 0x0001

CSN PMC2430 (0x3024a341), Serial Number 0xffffffff

Logical device #0
IO:  0x03e8 0x03e8 0x03e8 0x03e8 0x03e8 0x03e8 0x03e8 0x03e8
IRQ 5 0
DMA 4 0
IO range check 0x00 activate 0x01

The information you require is in the Vendor ID line at the start of the output. The hexadecimal number in parentheses (0x3024a341 in this example) is the PnP ID and the string immediately before this (PMC2430) is a unique ASCII ID.

Alternatively, if pnpinfo(8) does not list the card in question, pciconf(8) can be used instead. This is part of the output from pciconf -vl for an onboard sound chip:

# pciconf -vl
chip1@pci0:31:5:        class=0x040100 card=0x00931028 chip=0x24158086 rev=0x02 hdr=0x00
    vendor   = 'Intel Corporation'
    device   = '82801AA 8xx Chipset AC'97 Audio Controller'
    class    = multimedia
    subclass = audio

Here, you would use the chip value, 0x24158086.

This information (Vendor ID or chip value) needs adding to the file /usr/src/sys/dev/sio/sio_isa.c.

You should first make a backup of sio_isa.c just in case things go wrong. You will also need it to make the patch to submit with your PR (you are going to submit a PR, are you not?) then edit sio_isa.c and search for the line:

static struct isa_pnp_id sio_ids[] = {

Then scroll down to find the correct place to add the entry for your device. The entries look like this, and are sorted on the ASCII Vendor ID string which should be included in the comment to the right of the line of code along with all (if it will fit) or part of the Device Description from the output of pnpinfo(8):

{0x0f804f3f, NULL},     /* OZO800f - Zoom 2812 (56k Modem) */
{0x39804f3f, NULL},     /* OZO8039 - Zoom 56k flex */
{0x3024a341, NULL},     /* PMC2430 - Pace 56 Voice Internal Modem */
{0x1000eb49, NULL},     /* ROK0010 - Rockwell ? */
{0x5002734a, NULL},     /* RSS0250 - 5614Jx3(G) Internal Modem */

Add the hexadecimal Vendor ID for your device in the correct place, save the file, rebuild your kernel, and reboot. Your device should now be found as an sio device.

The problem is that the application you are trying to run is looking for a specific kernel symbol, but, for whatever reason, cannot find it; this error stems from one of two problems:

• Your kernel and userland are not synchronized (i.e., you built a new kernel but did not do an installworld, or vice versa), and thus the symbol table is different from what the user application thinks it is. If this is the case, simply complete the upgrade process (see /usr/src/UPDATING for the correct sequence).

• You are not using /boot/loader to load your kernel, but doing it directly from boot2 (see boot(8)). While there is nothing wrong with bypassing /boot/loader, it generally does a better job of making the kernel symbols available to user applications.

The symptom: there is a long delay between the time the TCP connection is established and the time when the client software asks for a password (or, in telnet(1)'s case, when a login prompt appears).

The problem: more likely than not, the delay is caused by the server software trying to resolve the client's IP address into a hostname. Many servers, including the Telnet and SSH servers that come with FreeBSD, do this in order to, among other things, store the hostname in a log file for future reference by the administrator.

The remedy: if the problem occurs whenever you connect from your computer (the client) to any server, the problem is with the client; likewise, if the problem only occurs when someone connects to your computer (the server) the problem is with the server.

If the problem is with the client, the only remedy is to fix the DNS so the server can resolve it. If this is on a local network, consider it a server problem and keep reading; conversely, if this is on the global Internet, you will most likely need to contact your ISP and ask them to fix it for you.

If the problem is with the server, and this is on a local network, you need to configure the server to be able to resolve address-to-hostname queries for your local address range. See the hosts(5) and named(8) manual pages for more information. If this is on the global Internet, the problem may be that your server's resolver is not functioning correctly. To check, try to look up another host -- say, www.yahoo.com. If it does not work, that is your problem.

Following a fresh install of FreeBSD, it is also possible that domain and name server information is missing from /etc/resolv.conf. This will often cause a delay in SSH, as the option UseDNS is set to yes by default in the sshd_config file in /etc/ssh. If this is causing the problem, you will either need to fill in the missing information in /etc/resolv.conf or set UseDNS to no in sshd_config as a temporary workaround.

Stray IRQs are indications of hardware IRQ glitches, mostly from hardware that removes its interrupt request in the middle of the interrupt request acknowledge cycle.

One has three options for dealing with this:

• Live with the warnings. All except the first 5 per irq are suppressed anyway.

• Break the warnings by changing the value of MAX_STRAY_LOG from 5 to 0 in your platform's (e.g. i386) intr_machdep.c file and rebuild the new kernel and all the warnings will be suppressed.

• Break the warnings by installing parallel port hardware that uses IRQ 7 and the PPP driver for it (this happens on most systems), and install an ide drive or other hardware that uses IRQ 15 and a suitable driver for it.

This error message indicates you have exhausted the number of available file descriptors on your system. Please see the kern.maxfiles section of the Tuning Kernel Limits section of the Handbook for a discussion and solution.

There is a known problem when enabling Intel® Enhanced SpeedStep from the BIOS: it causes the kernel to start printing “calcru” messages like this:

calcru: runtime went backwards from 6 usec to 3 usec for pid 37 (pagezero)
calcru: runtime went backwards from 6 usec to 3 usec for pid 36 (vmdaemon)
calcru: runtime went backwards from 170 usec to 138 usec for pid 35 (pagedaemon)
calcru: runtime went backwards from 553 usec to 291 usec for pid 15 (swi6: task queue)
calcru: runtime went backwards from 15521 usec to 10366 usec for pid 2 (g_event)
calcru: runtime went backwards from 25 usec to 12 usec for pid 11 (swi1: net)
calcru: runtime went backwards from 4417 usec to 3960 usec for pid 1 (init)
calcru: runtime went backwards from 2084385 usec to 1793542 usec for pid 1 (init)
calcru: runtime went backwards from 408 usec to 204 usec for pid 0 (swapper)

It is because Intel SpeedStep (EIST) is incompatible with some motherboards.

Workaround: Disable the EIST feature in the BIOS. You can still achieve ACPI-based processor frequency throttling by using powerd(8).

Your computer has two or more clocks, and FreeBSD has chosen to use the wrong one.

Run dmesg(8), and check for lines that contain Timecounter. The one with the highest quality value that FreeBSD chose.

# dmesg | grep Timecounter
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
Timecounter "TSC" frequency 2998570050 Hz quality 800
Timecounters tick every 1.000 msec

You can confirm this by checking the kern.timecounter.hardware sysctl(3).

# sysctl kern.timecounter.hardware
kern.timecounter.hardware: ACPI-fast

It may be a broken ACPI timer. The simplest solution is to disable the ACPI timer in /etc/loader.conf:

debug.acpi.disabled="timer"

Or the BIOS may modify the TSC clock--perhaps to change the speed of the processor when running from batteries, or going into a power saving mode, but FreeBSD is unaware of these adjustments, and appears to gain or lose time.

In this example, the i8254 clock is also available, and can be selected by writing its name to the kern.timecounter.hardware sysctl(3).

# sysctl -w kern.timecounter.hardware=i8254
kern.timecounter.hardware: TSC -> i8254

Your computer should now start keeping more accurate time.

To have this change automatically run at boot time, add the following line to /etc/sysctl.conf:

kern.timecounter.hardware=i8254

This problem is common on laptops that boot more than one operating system. Some non-BSD operating systems leave PC card hardware in an inconsistent state. pccardd(8) will detect the card as “"(null)""(null)"” instead of its actual model.

You must remove all power from the PC card slot to fully reset the hardware. Completely power off the laptop. (Do not suspend it, do not let it go into standby; the power needs to be completely off.) Wait a few moments, and reboot. Your PC card should work now.

Some laptop hardware lies when it claims to be off. If the above does not work shut down, remove the battery, wait a moment, replace the battery, and reboot.

FreeBSD's boot loader is incorrectly recognizing the hard drive's geometry. This must be manually set within fdisk(8) when creating or modifying FreeBSD's slice.

The correct drive geometry values can be found within the machine's BIOS. Look for the number of cylinders, heads and sectors for the particular drive.

Within sysinstall(8)'s fdisk, hit G to set the drive geometry.

A dialog will pop up requesting the number of cylinders, heads and sectors. Type the numbers found from the BIOS separated by forward slashes. For example, values of 5000 cylinders, 250 heads, and 60 sectors would be entered as 5000/250/60.

Press Enter to set the values, and hit W to write the new partition table to the drive.

Enter sysinstall(8) and choose Configure, then Fdisk. Select the disk the Boot Manager resided on with the Space key. Press W to write changes to the drive. A prompt will appear asking which boot loader to install. Select this, and it will be restored.

This means that a process is trying to page memory to disk, and the page attempt has hung trying to access the disk for more than 20 seconds. It might be caused by bad blocks on the disk drive, disk wiring, cables, or any other disk I/O-related hardware. If the drive itself is actually bad, you will also see disk errors in /var/log/messages and in the output of dmesg. Otherwise, check your cables and connections.

The ata(4) driver reports “UDMA ICRC” errors when a DMA transfer to or from a drive is corrupted. The driver will retry the operation a few times. Should the retries fail, it will switch from DMA to the slower PIO mode of communication with the device.

The problem can be caused by many factors, although perhaps the most common cause is faulty or incorrect cabling. Check that the ATA cables are undamaged and rated for the Ultra DMA mode in use. If you are using removable drive trays, they must also be compatible. Be sure that all connections are making good contact. Problems have also been noticed when an old drive is installed on the same ATA channel as an Ultra DMA 66 (or faster) drive. Lastly, these errors can indicate that the drive is failing. Most drive vendors provide testing software for their drives, so test your drive, and, if necessary, back up your data and replace it.

The atacontrol(8) utility can be used to show and select the DMA or PIO modes used for each ATA device. In particular, atacontrol mode channel will show the modes in use on a particular ATA channel, where the primary channel is numbered 0, and so on.

An answer for this question can be found in the FreeBSD Glosssary, see LOR.

This means that a function that may sleep was called while a mutex (or other unsleepable) lock was held.

The reason this is an error is because mutexes are not intended to be held for long periods of time; they are supposed to only be held to maintain short periods of synchronization. This programming contract allows device drivers to use mutexes to synchronize with the rest of the kernel during interrupts. Interrupts (under FreeBSD) may not sleep. Hence it is imperative that no subsystem in the kernel block for an extended period while holding a mutex.

To catch such errors, assertions may be added to the kernel that interact with the witness(4) subsystem to emit a warning or fatal error (depending on the system configuration) when a potentially blocking call is made while holding a mutex.

In summary, such warnings are non-fatal, however with unfortunate timing they could cause undesirable effects ranging from a minor blip in the system's responsiveness to a complete system lockup.

This error does not mean that the touch(1) utility is missing. The error is instead probably due to the dates of the files being set sometime in the future. If your CMOS-clock is set to local time you need to run the command adjkerntz -i to adjust the kernel clock when booting into single user mode.