5. mac adddress of sip0

now starts the second part of the guide. the one where we need to prepare another box, to netboot the soekris. we are going to start with the TFTP (trivial ftp) setting. but before we go further, we should grab the ethernet address of the first sip0 plug (the soekris network plugs are driven by the sip driver). so in the console, as the soekris to boot using the network :

> boot f0

NSC DP83815/DP83816 Fast Ethernet UNDI, v1.03
Copyright (C) 2002, 2003 National Semiconductor Corporation
All rights reserved.

Pre-boot eXecution Environment  PXE-2.0 (build 082)
Copyright (C) 1997-2000  Intel Corporation

CLIENT MAC ADDR: 00 00 24 C5 DC 00
PXE-E53: No boot filename received

PXE-M0F: Exiting MacPhyter PXE ROM.

No Boot device available, enter monitor.

so what we see here is pretty logical. we see the ethernet managing chip, followed with the start of the pxe code in the bios that handles the netbooting. it will, of course, fail since we have not configured DHCP to give the box an IP (if you have a dhcpd around, just use a cross cable from the soekris to the box you will use to netboot it). in the failure messages, the mac address of the sip0 does appear : write it down.

the pxe we are going to use was designed by intel. it will let you boot x86 hardware the same way we can netboot the sun sparc machines (their firmware is just wonderful, check the sparc platform if you dont know it). pxe requires bootp, tftp and nfs. it does mean "pre-boot execution environment" and is pronounced "pixie". at first it seems to be complicated but it's not at all.

to netboot the soekris, i am going to use my laptop. i just plug it using a cross cable to the sip0 plug. i do use netbsd on the laptop too but you can use anything else as long as you got dhcp and tftp available there (don't forget nfs).

a. tftp

first, we need to select the boot binary we're going to supply to the soekris. since it has no vga capability and display device, we need to use the pxeboot_ia32_com0.bin file which is found on /usr/mdec/

this is also why we went into the soekris to set the speed to 9600 baud : this is the speed set in this file. i create a new folder called /tftpboot where i copy this file :

mkdir /tftpboot
cp /usr/mdec/pxeboot_ia32_com0.bin /tftpboot

now, we turn on the trivial ftp server on the laptop. we edit the /etc/inetd.conf file to remove the comment at the beginning of the following line :

tftp dgram udp wait root [..]

you will find two lines there. one for ipv4 and one for ipv6. were going to use ipv4 for now. we save the file and tell inetd to reload its config :

/etc/rc.d/inetd reload

we can now do a test to see if the soekris will be able to download the file :

[root][21] tftp 192.168.1.3
tftp> get pxeboot_ia32_com0.bin
Received 41905 bytes in 0.0 seconds
tftp> quit

it works fine then.

b. dhcpd

now we prepare the dhcp server to give the soekris an ip address. here we are going to use the mac address we recovered from the console port monitor. we first create an empty file for the lease :

touch /var/db/dhcpd.leases

and put the following content inside of /etc/dhcpd.conf :

server-name "x30";
max-lease-time 7200;
default-lease-time 600;
ddns-update-style ad-hoc;
option subnet-mask 255.255.255.0;

subnet 192.168.1.0 netmask 255.255.255.0 {
          range 192.168.1.4 192.168.1.9;
}

host pxehost {
        filename "pxeboot_ia32_com0.bin";
        server-name "x30";
        next-server 192.168.1.3;
        hardware ethernet 00:00:24:C5:DC:00;
        fixed-address 192.168.1.4;
        option root-path "/usr/tmp/pxestuff/root";
}

i also create a folder inside of /tmp (/usr/tmp) where i will put the kernel loaded by the second stage loader (more on this later).

add a "dhcpd=YES" to your /etc/rc.conf and start the dhcpd server using rc.d

c. nfs

now the soekris will get an ip on next reboot, and will be able to download the pxe boot binary using trivial ftp. but once it is loaded, it needs to load a netbsd kernel to boot. this will be done using a nfs share.

we add a line to /etc/exports to export in read/write the /usr/tmp/pxestuff folder. we also need to add services required by nfs to start and work (named, nfsd, mountd, rpcbind, lockd, nfs_server)

once all set in your rc.conf just start the services and the nfs_server at the end. we now need to install a kernel for the soekris on the nfs exported folder.

6. kernel for soekris

since the soekris is a 586 we can use any kernel, like the GENERIC one. but we need to tell the kernel on boot that its root will be found on nfs and not hard-disk. since i will use sip0 i just need to build a custom GENERIC kernel with the proper setting. in fact, if you want to use your soekris as a diskless device, this is all you need. you put its kernel on the nfs partition and install its netbsd system there, and it will netboot and work perfectly from now on.

all i want is to boot the soekris and build from there. so i create a kernel with little changes. first, make a copy of the GENERIC kernel config file (you first move to the folder with confs, which is /usr/src/sys/arch/i386/conf). we edit the file and change this line :

config netbsd root on ? type nfs

to :

config netbsd root on sip0 type nfs

so the kernel will look for its root on nfs. then, we need to force the first console to the first serial port. we remove the # in front of the following line too :

options 	CONSDEVNAME="\"com\"",CONADDR=0x3f8,CONSPEED=9600
options 	CONS_OVERRIDE

dont forget that the soekris console is found at 0x3f8. now, we build the kernel and put it where the soekris will find it :

cd /usr/src
./build.sh kernel=SOEKRIS2
cp /usr/src/sys/arch/i386/compile/obj/SOEKRIS_INSTALL/netbsd /usr/tmp/pxestuff/root/

we also need to install in the nfs exported folder everything the kernel will need to boot and work. otherwise, the kernel will break into the debugguer and you wont like it. you just move to the exported folder and uncompress there the netbsd distribution tarballs :

cd /usr/tmp/pxestuff/root
tar zxpf /usr/INSTALL/base.tgz
tar zxpf /usr/INSTALL/etc.tgz

when you install netbsd, it moves those to /usr/INSTALL and you should find those there. if you didnt keep them, just download them from the netbsd ftp or copy them from your installation media.

we build the devices there too :

cd /usr/tmp/pxestuff/root/dev
./MAKEDEV all

and we power-cycle the soekris to start the boot.

7. first autonomous boot

the soekris will now boot. first, it loads pxe code. it gets an ip from the dhcp server and downloads the first stage loader. which in turn, grabs the netbsd kernel from the nfs exported folder and boots it. once the kernel is done it will ask for console type and default shell. use vt110 or vt220 for console type and sh as default shell.

now, if all you wanted is a netbootable soekris you will build from there. it's rather easy to have a lot of netbooting machines from another box where you install required files, and kernels (sharing files being a must if the netbooted components are the same like in a cluster).

the next steps are to prepare the hard-disk and install netbsd there.

a. hard-disk config

the hard-disk i use is 40 gb. i will keep it simple so im going to use 1 gb for /, 1 gb for /var, twice the memory available as swap space and everything else for /usr (i remove the /home folder which i install onto /usr/home with a symbolic link on / for the home).

first, we create a single fdisk entry that covers the whole disk for netbsd :

fdisk -u wd0

(Thanks to Eric Becquet for correcting the above line, there was a mistake - i owe you one !!)

40 gb gives us 21474880928 sectors (sectors are 512 byte). we dont need to change what the bios thinks, and we move on. we want to change entry 0. for the kind use 169 which is the default netbsd id for partitions.

here is the fdisk we get then :

Partition table:
0: NetBSD (sysid 169)
    start 63, size 78140097 (38154 MB, Cyls 0-4864), Active
    1: 
    2: 
    3: 
    Bootselector disabled.

the disks per default dont use the first track. this means that sectors 0 to 62 are reserved and we can only start at 63. the first track is reserved for first stage boot loader and dont use it or the bios wont like it and nothing will boot. it is interesting to note that 63 is a kind of magic value. modern hard-disks have so much sectors that they virtualize the tracks, heads and cylinders. so on most modern disks, the first track is often much more than 63 sectors. but when you create your first partition at sector 63 the first track gets skipped.

now we can prepare the disklabel for the disk.

under netbsd d is reserved for the whole disk while c is used for the netbsd partition.

we use the following command to prepare the disklabel :

disklabel -i -I wd0

type ? to see available commands in this mode.

so per default we have just one partition in the disklabel, that spans the whole disk from sector 63 to the end of it.

here is my set up :

a will be my /, 1 gb (2097152 sectors)
b will be my swap, 256 mb (524288 sectors)
e will be my /var, 100 mb (204800 sectors)
f will be my /tmp, 1 gb (2097152 sectors)
g will be /usr, everything else...

so with calculated values, 512 bytes/sector, i want this :

name   size    offset        kind     mount-point

 a    2097152  63            ffs      /
 b     524288  2097215       swap     (swap)
 c   78140097  63            -
 d   78140160  0             -
 e     204800  2621503       ffs      /var
 f    2097152  2826303       ffs      /tmp
 g   73216705  4923455       ffs      /usr

the disklabel is rather easy to use. you type the letter of the partition you want to edit. to remove "e" i then type e, keep the current fs-type but i enter 0 to the next two questions. this removes that big partition.

now we can create the partitions. we type "a" and select 4.2BSD as file system type. offset is 63 since we must avoid first track, and we enter the partition size. the creation of the other partitions will be even easier, since when it asks you the offset you just need to enter the previous partition name.

once all the partitions are created, you save and quit the tool (use ? to display its options if required). we then use newfs on each new partition but not the swap (which does not require to be formated).

b. copying files

now it gets more tricky. we want to put all required files on the hard-disk so we can boot. we already have a complete netbsd system on /usr/tmp/pxestuff/root and we need to copy those files to the hard-disk, partition by partition since each is a different mount point.

you can do it by exporting using nfs the hard-disk from the soekris to the laptop. there, you would mount it and use build.sh to build a release you would drop there.

i am going to be a little hardcore and directly use cp :

cd /
mount /dev/wd0a /mnt
cp -Rp .cshrc .profile boot altroot bin dev etc lib libexec \
rescue root sbin stand /mnt/.
umount /mnt
mount /dev/wd0e /mnt
cp -Rp /var/* /mnt/.
umount /mnt
mount /dev/wd0g /mnt
cp -Rp /usr/* /mnt/.
umount /mnt

we go into dev and use ./MAKEDEV all and we need to install the required boot code to the hard-disk. we also need to build and install a new kernel at the root of the soekris filesystem, since it's going to boot alone from now on.

just make a new copy of GENERIC and force the boot on wd0a using ffs. keep forcing the serial console to 0x3f8 with the override for console so it forces it. we remove com1 and com2 and replace com0 by com* so it does not conflict with the forced serial console on com0.

we put the following on the soekris hard-disk fstab :

/dev/wd0a /     ffs    rw
/dev/wd0b none  swap   sw
/dev/wd0e /var  ffs    rw
/dev/wd0f /tmp  ffs    rw
/dev/wd0g /usr  ffs    rw
kernfs    /kern kernfs rw
procfs    /proc procfs rw,noauto

we edit the rc.conf to have rc_configured set to YES and we umount everything. now we install the boot code which is found in /usr/mdec :

installboot -m i386 -oconsole=com0,ioaddr=0x3f8,speed=9600 /dev/wd0d /usr/mdec/bootxx_ffsv1

then you need to use the following command :

fdisk -i wd0

Answer twice "y" to fdisk.

Thanks to Eric Becquet for hunting and fixing this mistake in this guide. I knew it was there but I could not find it without doing a full reinstall of my soekris : thank you :D

last, we configure the ip address of sip0 by creating a /etc/ifconfig.sip0 and we turn on the telnet server in inetd.conf and/or the ssh service in rc.conf.

now we can reboot the soekris and we can use telnet or ssh to get inside of it.

the rest is configuration of the little box for your needs :)

soekris kernel config file

#
# kernel configuration file
#
# this config file is based on generic + console forced on com0/3f8
# with ntp frequency fixed manually to compensate non-usual frequency
# from elan 520. patchlevel up to 026
#
# last update : gilb@nerim.net, november 28th 2006
#
# http://gilb.nerim.net/soekris.html
#

include 	"arch/i386/conf/std.i386"
options 	INCLUDE_CONFIG_FILE
maxusers	32
options 	I586_CPU

options 	MATH_EMULATE
options 	VM86

options 	MTRR
options 	PERFCTRS

# the soekris does not use a default speed of 9600 on
# serial console. i did force mine to 9600 so if you're not
# doing the same, don't forget to update the CONSPEED below

options 	CONSDEVNAME="\"com\"",CONADDR=0x3f8,CONSPEED=9600
options 	CONS_OVERRIDE

options 	RTC_OFFSET=0
options 	NTP
options 	KTRACE
options 	SYSTRACE

options 	SYSVMSG
options 	SYSVSEM
options 	SYSVSHM
options 	P1003_1B_SEMAPHORE

options 	SYSCTL_INCLUDE_DESCR

# experimental buffer queue strategy
options 	NEW_BUFQ_STRATEGY

# i put that in order to have DDB turn-on and directly spit
# on com0 on the back of the soekris

options 	DDB
options 	DDB_ONPANIC=1
options 	DDB_HISTORY_SIZE=512
options 	KGDB
options 	KGDB_DEVNAME="\"com\"",KGDB_DEVADDR=0x3f8,KGDB_DEVRATE=9600

# i only keep recent compat options
# that's netbsd 2.x and the usual 4.3BSD

options 	COMPAT_20
options 	COMPAT_43
options 	COMPAT_386BSD_MBRPART

# linux and freebsd compat jut in case, and ptys

options 	COMPAT_LINUX
options 	COMPAT_FREEBSD
options 	COMPAT_BSDPTY

file-system 	FFS
file-system 	MFS
file-system 	NFS
file-system 	KERNFS
file-system 	NULLFS
file-system 	PROCFS

options 	QUOTA
options 	SOFTDEP
options 	NFSSERVER

options 	INET
options 	INET6
options 	ISO,TPIP
options 	PPP_BSDCOMP
options 	PPP_DEFLATE
options 	PPP_FILTER
options 	PFIL_HOOKS
options 	IPFILTER_LOG

# i use altq. mostly CBQ and sometimes RED
# (as in random early detection of congestion, and not
# random early drop as usually noted)

options 	ALTQ
options 	ALTQ_CBQ
options 	ALTQ_RED

options 	NFS_BOOT_DHCP,NFS_BOOT_BOOTPARAM

options 	WSEMUL_VT100
options 	WSDISPLAY_COMPAT_SYSCONS
options 	WSDISPLAY_COMPAT_USL
options 	WSDISPLAY_COMPAT_RAWKBD
options 	PCDISPLAY_SOFTCURSOR

# the following line fixes a drift problem with the soekris when
# used with NTP

options		TIMER_FREQ=1189200

config		netbsd	root on wd0a type ffs

mainbus0 at root
cpu* at mainbus?

# PCI bus support
pci*	at mainbus? bus ?
pci*	at pchb? bus ?
pci*	at ppb? bus ?

# System Controllers
elansc* at pci? dev ? function ?

pchb*	at pci? dev ? function ?
pceb*	at pci? dev ? function ?
pcib*	at pci? dev ? function ?
pcmb*	at pci? dev ? function ?
ppb*	at pci? dev ? function ?
puc*	at pci? dev ? function ?

# ISA bus support
isa0	at mainbus?
isa0	at pceb?
isa0	at pcib?

# Math Coprocessor support
npx0	at isa? port 0xf0 irq 13

pcppi0		at isa?
sysbeep0	at pcppi?

# com0 on back, com1 internal

com0	at isa? port 0x3f8 irq 4
com1	at isa? port 0x2f8 irq 3

# ide, hard-disk

geodeide* 	at pci? dev ? function ?
atabus* at ata?
wd*	at atabus? drive ? flags 0x0000

# network cards are sip

sip*	at pci? dev ? function ?
nsphyter* at mii? phy ?

# while there is an usb port at the back of the soekris
# i just dont use it for now

# Pull in optional local configuration
include	"arch/i386/conf/GENERIC.local"

# Pseudo-Devices

pseudo-device 	crypto
pseudo-device	md		1
pseudo-device	vnd		4
options 	VND_COMPRESSION

# network pseudo-devices

pseudo-device	bpfilter	8	# Berkeley packet filter
pseudo-device	ipfilter		# IP filter (firewall) and NAT
pseudo-device	loop			# network loopback
pseudo-device	ppp		2	# Point-to-Point Protocol
pseudo-device	pppoe			# PPP over Ethernet (RFC 2516)
pseudo-device	tap			# virtual Ethernet
pseudo-device	tun		2	# network tunneling over tty
pseudo-device	gre		2	# generic L3 over IP tunnel
pseudo-device	gif		4	# IPv[46] over IPv[46] tunnel (RFC1933)
pseudo-device	vlan			# IEEE 802.1q encapsulation
pseudo-device	bridge			# simple inter-network bridging

pseudo-device	pty			# pseudo-terminals
pseudo-device	rnd			# /dev/random and in-kernel generator
pseudo-device	clockctl		# user control of clock subsystem

# wscons pseudo-devices
pseudo-device	wsmux			# mouse & keyboard multiplexor
pseudo-device	wsfont
pseudo-device	ksyms			# /dev/ksyms

soekris fdisk

Disk: /dev/rwd0d
NetBSD disklabel disk geometry:
cylinders: 77520, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
total sectors: 78140160

BIOS disk geometry:
cylinders: 1024, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 793314819494372

Partition table:
0: NetBSD (sysid 169)
    start 63, size 78140097 (38154 MB, Cyls 0-4864), Active
1: 
2: 
3: 
Bootselector disabled.

soekris disklabel

# /dev/rwd0d:
type: unknown
disk: TOSHIBA MK4032G
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 77520
total sectors: 78140160
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# microseconds
track-to-track seek: 0	# microseconds
drivedata: 0 

16 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:   2097648        63     4.2BSD   2048 16384     0  # (Cyl.      0*-   2081*)
 b:    525168   2097711       swap                     # (Cyl.   2081*-   2602*)
 c:  78140097        63     unused      0     0        # (Cyl.      0*-  77519)
 d:  78140160         0     unused      0     0        # (Cyl.      0 -  77519)
 e:  73418688   2622879     4.2BSD   2048 16384     0  # (Cyl.   2602*-  75438*)
 f:   2098593  76041567     4.2BSD   2048 16384     0  # (Cyl.  75438*-  77519)

soekris dmesg

NetBSD 3.0 (SOEKRIS) #4: Fri May  5 19:19:40 UTC 2006
	root@soekris:/usr/src/sys/arch/i386/compile/SOEKRIS
total memory = 127 MB
avail memory = 122 MB
BIOS32 rev. 0 found at 0xf7840
mainbus0 (root)
cpu0 at mainbus0: (uniprocessor)
cpu0: National Semiconductor Geode GX1 (586-class), 266.67 MHz, id 0x540
cpu0: features 808131
cpu0: features 808131
cpu0: "Geode(TM) Integrated Processor by National Semi"
cpu0: I-cache 12K uOp cache 8-way
WARNING: broken TSC disabled
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: vendor 0x1078 product 0x0001 (rev. 0x00)
sip0 at pci0 dev 6 function 0: NatSemi DP83815 10/100 Ethernet, rev 00
sip0: interrupting at irq 10
sip0: Ethernet address 00:00:24:c5:dc:00
nsphyter0 at sip0 phy 0: DP83815 10/100 media interface, rev. 1
nsphyter0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sip1 at pci0 dev 7 function 0: NatSemi DP83815 10/100 Ethernet, rev 00
sip1: interrupting at irq 10
sip1: Ethernet address 00:00:24:c5:dc:01
nsphyter1 at sip1 phy 0: DP83815 10/100 media interface, rev. 1
nsphyter1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sip2 at pci0 dev 8 function 0: NatSemi DP83815 10/100 Ethernet, rev 00
sip2: interrupting at irq 10
sip2: Ethernet address 00:00:24:c5:dc:02
nsphyter2 at sip2 phy 0: DP83815 10/100 media interface, rev. 1
nsphyter2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pcib0 at pci0 dev 18 function 0
pcib0: vendor 0x100b product 0x0510 (rev. 0x00)
vendor 0x100b product 0x0511 (miscellaneous bridge) at pci0 dev 18 function 1 not configured
geodeide0 at pci0 dev 18 function 2
geodeide0: AMD Geode SC1100 IDE controller (rev. 0x01)
geodeide0: bus-master DMA support present
geodeide0: primary channel wired to compatibility mode
geodeide0: primary channel interrupting at irq 14
atabus0 at geodeide0 channel 0
geodeide0: secondary channel wired to compatibility mode
geodeide0: secondary channel interrupting at irq 15
atabus1 at geodeide0 channel 1
vendor 0x100b product 0x0515 (miscellaneous bridge) at pci0 dev 18 function 5 not configured
vendor 0x0e11 product 0xa0f8 (USB serial bus, interface 0x10, revision 0x08) at pci0 dev 19 function 0 not configured
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pcppi0 at isa0 port 0x61
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0-0xff: using exception 16
wd0 at atabus0 drive 0: 
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 38154 MB, 77520 cyl, 16 head, 63 sec, 512 bytes/sect x 78140160 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd0(geodeide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33) (using DMA)
boot device: wd0
root on wd0a dumps on wd0b

last update : Sun Dec 10 23:12:32 UTC 2006