A very valuable vulnerability

While I very firmly wear a white hat, it is useful to be able to consider things from the perspective of the bad guys, in order to assess the likelihood of a vulnerability being exploited and its potential impact. For the subset of bad guys who exploit security vulnerabilities for profit — as opposed to selling them to spy agencies, for example — I imagine that there are some criteria which would tend to make a vulnerability more valuable: Much to my surprise, a few weeks ago I stumbled across a vulnerability satisfying every one of these criteria.

The vulnerability — which has since been fixed, or else I would not be writing about it publicly — was in Stripe's bitcoin payment functionality. Some background for readers not familiar with this: Stripe provides payment processing services, originally for credit cards but now also supporting ACH, Apple Pay, Alipay, and Bitcoin, and was designed to be the payment platform which developers would want to use; in very much the way that Amazon fixed the computing infrastructure problem with S3 and EC2 by presenting storage and compute functionality via simple APIs, Stripe fixed the "getting money from customers online" problem. I use Stripe at my startup, Tarsnap, and was in fact the first user of Stripe's support for Bitcoin payments: Tarsnap has an unusually geeky and privacy-conscious user base, so this functionality was quite popular among Tarsnap users.

Despite being eager to accept Bitcoin payments, I don't want to actually handle bitcoins; Tarsnap's services are priced in US dollars, and that's what I ultimately want to receive. Stripe abstracts this away for me: I tell Stripe that I want $X, and it tells me how many bitcoins my customer should send and to what address; when the bitcoin turns up, I get the US dollars I asked for. Naturally, since the exchange rate between dollars and bitcoins fluctuates, Stripe can't guarantee the exchange rate forever; instead, they guarantee the rate for 10 minutes (presumably they figured out that the exchange rate volatility is low enough that they won't lose much money over the course of 10 minutes). If the "bitcoin receiver" isn't filled within 10 minutes, incoming coins are converted at the current exchange rate.

For a variety of reasons, it is sometimes necessary to refund bitcoin transactions: For example, a customer cancelling their order; accidentally sending in the wrong number of bitcoins; or even sending in the correct number of bitcoins, but not within the requisite time window, resulting in their value being lower than necessary. Consequently, Stripe allows for bitcoin transactions to be refunded — with the caveat that, for obvious reasons, Stripe refunds the same value of bitcoins, not the same number of bitcoins. (This is analogous to currency exchange issues with credit cards — if you use a Canadian dollar credit card to buy something in US dollars and then get a refund later, the equal USD amount will typically not translate to an equal number of CAD refunded to your credit card.)

The vulnerability lay in the exchange rate handling. As I mentioned above, Stripe guarantees an exchange rate for 10 minutes; if the requisite number of bitcoins arrive within that window, the exchange rate is locked in. So far so good; but what Stripe did not intend was that the exchange rate was locked in permanently — and applied to any future bitcoins sent to the same address.

This made a very simple attack possible:

  1. Pay for something using bitcoin.
  2. Wait until the price of bitcoin drops.
  3. Send more bitcoins to the address used for the initial payment.
  4. Ask for a refund of the excess bitcoin.
Because the exchange rate used in step 3 was the one fixed at step 1, this allowed for bitcoins to be multiplied by the difference in exchange rates; if step 1 took place on July 2nd and steps 3/4 on August 2nd, for example, an arbitrary number of bitcoins could be increased by 30% in a matter of minutes. Moreover, the attacker does not need an account with Stripe; they merely need to find a merchant which uses Stripe for bitcoin payments and is willing to click "refund payment" (or even better, is set up to automatically refund bitcoin overpayments).

Needless to say, I reported this to Stripe immediately. Fortunately, their website includes a GPG key and advertises a vulnerability disclosure reward (aka. bug bounty) program; these are two things I recommend that every company does, because they advertise that you take security seriously and help to ensure that when people stumble across vulnerabilities they'll let you know. (As it happens, I had Stripe security's public GPG key already and like them enough that I would have taken the time to report this even without a bounty; but it's important to maximize the odds of receiving vulnerability reports.) Since it was late on a Friday afternoon and I was concerned about how easily this could be exploited, I also hopped onto Stripe's IRC channel to ask one of the Stripe employees there to relay a message to their security team: "Check your email before you go home!"

Stripe's handling of this issue was exemplary. They responded promptly to confirm that they had received my report and reproduced the issue locally; and a few days later followed up to let me know that they had tracked down the code responsible for this misbehaviour and that it had been fixed. They also awarded me a bug bounty — one significantly in excess of the $500 they advertise, too.

As I remarked six years ago, Isaac Asimov's remark that in science "Eureka!" is less exciting than "That's funny..." applies equally to security vulnerabilities. I didn't notice this issue because I was looking for ways to exploit bitcoin exchange rates; I noticed it because a Tarsnap customer accidentally sent bitcoins to an old address and the number of coins he got back when I clicked "refund" was significantly less than what he had sent in. (Stripe has corrected this "anti-exploitation" of the vulnerability.) It's important to keep your eyes open; and it's important to encourage your customers to keep their eyes open, which is the largest advantage of bug bounty programs — and why Tarsnap's bug bounty program offers rewards for all bugs, not just those which turn out to be vulnerabilities.

And if you have code which handles fluctuating exchange rates... now might be a good time to double-check that you're always using the right exchange rates.

Posted at 2016-10-28 19:00 | Permanent link | Comments

EC2's most dangerous feature

As a FreeBSD developer — and someone who writes in C — I believe strongly in the idea of "tools, not policy". If you want to shoot yourself in the foot, I'll help you deliver the bullet to your foot as efficiently and reliably as possible. UNIX has always been built around the idea that systems administrators are better equipped to figure out what they want than the developers of the OS, and it's almost impossible to prevent foot-shooting without also limiting useful functionality. The most powerful tools are inevitably dangerous, and often the best solution is to simply ensure that they come with sufficient warning labels attached; but occasionally I see tools which not only lack important warning labels, but are also designed in a way which makes them far more dangerous than necessary. Such a case is IAM Roles for Amazon EC2.

A review for readers unfamiliar with this feature: Amazon IAM (Identity and Access Management) is a service which allows for the creation of access credentials which are limited in scope; for example, you can have keys which can read objects from Amazon S3 but cannot write any objects. IAM Roles for EC2 are a mechanism for automatically creating such credentials and distributing them to EC2 instances; you specify a policy and launch an EC2 instance with that Role attached, and magic happens making time-limited credentials available via the EC2 instance metadata. This simplifies the task of creating and distributing credentials and is very convenient; I use it in my FreeBSD AMI Builder AMI, for example. Despite being convenient, there are two rather scary problems with this feature which severely limit the situations where I'd recommend using it.

The first problem is one of configuration: The language used to specify IAM Policies is not sufficient to allow for EC2 instances to be properly limited in their powers. For example, suppose you want to allow EC2 instances to create, attach, detach, and delete Elastic Block Store volumes automatically — useful if you want to have filesystems automatically scaling up and down depending on the amount of data which they contain. The obvious way to do this is would be to "tag" the volumes belonging to an EC2 instance and provide a Role which can only act on volumes tagged to the instance where the Role was provided; while the second part of this (limiting actions to tagged volumes) seems to be possible, there is no way to require specific API call parameters on all permitted CreateVolume calls, as would be necessary to require that a tag is applied to any new volumes being created by the instance. (There also seems to be a gap in the CreateVolume API call, in that it is documented as returning a list of tags assigned to the newly created volume, but does not advertise support for assigning tags as part of the process of creating a volume; but that at least could be easily fixed, and I'm not even sure if this is a failing in the API call or in the documentation of the API call.) The difficulty, and sometimes impossibility, of writing appropriately fine-grained IAM Policies results in a proliferation of overbroad policies; for example, the pre-written Policy which Amazon provides for allowing the Simple Systems Manager agent to make the API calls it requires (AmazonEC2RoleforSSM) permits it to GET and PUT any object in S3 — a terrifying proposition if you store data in S3 which you don't want to see made available to all of your "managed" EC2 instances.

As problematic as the configuration is, a far larger problem with IAM Roles for Amazon EC2 is access control — or, to be more precise, the lack thereof. As I mentioned earlier, IAM Role credentials are exposed to EC2 instances via the EC2 instance metadata system: In other words, they're available from (I presume that the "EC2ws" HTTP server which responds is running in another Xen domain on the same physical hardware, but that implementation detail is unimportant.) This makes the credentials easy for programs to obtain... unfortunately, too easy for programs to obtain. UNIX is designed as a multi-user operating system, with multiple users and groups and permission flags and often even more sophisticated ACLs — but there are very few systems which control the ability to make outgoing HTTP requests. We write software which relies on privilege separation to reduce the likelihood that a bug will result in a full system compromise; but if a process which is running as user nobody and chrooted into /var/empty is still able to fetch AWS keys which can read every one of the objects you have stored in S3, do you really have any meaningful privilege separation? To borrow a phrase from Ted Unangst, the way that IAM Roles expose credentials to EC2 instances makes them a very effective exploit mitigation mitigation technique.

To make it worse, exposing credentials — and other metadata, for that matter — via HTTP is completely unnecessary. EC2 runs on Xen, which already has a perfectly good key-value data store for conveying metadata between the host and guest instances. It would be absolutely trivial for Amazon to place EC2 metadata, including IAM credentials, into XenStore; and almost as trivial for EC2 instances to expose XenStore as a filesystem to which standard UNIX permissions could be applied, providing IAM Role credentials with the full range of access control functionality which UNIX affords to files stored on disk. Of course, there is a lot of code out there which relies on fetching EC2 instance metadata over HTTP, and trivial or not it would still take time to write code for pushing EC2 metadata into XenStore and exposing it via a filesystem inside instances; so even if someone at AWS reads this blog post and immediately says "hey, we should fix this", I'm sure we'll be stuck with the problems in IAM Roles for years to come.

So consider this a warning label: IAM Roles for EC2 may seem like a gun which you can use to efficiently and reliably shoot yourself in the foot; but in fact it's more like a gun which is difficult to aim and might be fired by someone on the other side of the room snapping his fingers. Handle with care!

Posted at 2016-10-09 08:30 | Permanent link | Comments


FreeBSD 11.0-RELEASE is just around the corner, and it will be bringing a long list of new features and improvements — far too many for me to list here. But as part of the release process, the FreeBSD release engineering team has built images for Amazon EC2, and as semi-official maintainer of that platform (I've never been appointed to this role, but I've been doing it for years and nobody has told me to stop...) I think there are some improvements in FreeBSD 11.0 which are particularly noteworthy for EC2 users.

First, the EC2 Console Screenshot functionality now works with FreeBSD. This provides a "VGA" output as opposed to the traditional "serial port" which EC2 has exposed as "console output" for the past decade, and is useful largely because the "VGA" output becomes available immediately whereas the "serial port" output can lag by several minutes. This improvement is a simple configuration change — older releases didn't waste time writing to a non-serial console because it didn't go anywhere until Amazon added support on their side — and can be enabled on older FreeBSD releases by changing the line console="comconsole" to boot_multicons="YES" in /boot/loader.conf.

The second notable change is support for EC2 "Enhanced Networking" using Intel 82599 hardware; on the C3, C4, R3, I2, D2, and M4 (excluding m4.16xlarge) families, this provides increased network throughput and reduced latency and jitter, since it allows FreeBSD to talk directly to the networking hardware rather than via a Xen paravirtual interface. Getting this working took much longer than I had hoped, but the final problem turned out not to be in FreeBSD at all — we were tickling an interrupt-routing bug in a version of Xen used in EC2. Unfortunately FreeBSD does not yet have support for the new "Elastic Network Adapter" enhanced networking used in P2 and X1 instance families and the m4.16xlarge instance type; I'm hoping that we'll have a driver for that before FreeBSD 11.1 arrives.

The third notable change is an improvement in EC2 disk throughput. This comes thanks to enabling indirect segment I/Os in FreeBSD's blkfront driver; while the support was present in 10.3, I had it turned off by default due to performance anomalies on some EC2 instances. (Those EC2 performance problems have been resolved, and disk I/O performance in EC2 on FreeBSD 10.3 can now be safely improved by removing the line hw.xbd.xbd_enable_indirect="0" from /boot/loader.conf.)

Finally, FreeBSD now supports all 128 CPUs in the x1.32xlarge instance type. This improvement comes thanks to two changes: The FreeBSD default kernel was modified in 2014 to support up to 256 CPUs (up from 64), but that resulted in a (fixed-size) section of preallocated memory being exhausted early in the boot process on systems with 92 or more CPUs; a few months ago I changed that value to tune automatically so that FreeBSD can now boot and not immediately panic with an out-of-the-box setup on such large systems.

I think FreeBSD/EC2 users will be very happy with FreeBSD 11.0-RELEASE; but I'd like to end with an important reminder: No matter what you might see on FTP servers, in EC2, or available via freebsd-update, the new release has not been released until you see a GPG-signed email from the release engineer. This is not just a theoretical point: In my time as a FreeBSD developer I've seen multiple instances of last-minute release re-rolls happening due to problems being discovered very late, so the fact that you can see bits doesn't necessarily mean that they are ready to be downloaded. I hope you're looking forward to 11.0-RELEASE, but please be patient.

Posted at 2016-10-03 06:00 | Permanent link | Comments

Write opinionated workarounds

A few years ago, I decided that I should aim for my code to be as portable as possible. This generally meant targeting POSIX; in some cases I required slightly more, e.g., "POSIX with OpenSSL installed and cryptographic entropy available from /dev/urandom". This dedication made me rather unusual among software developers; grepping the source code for the software I have installed on my laptop, I cannot find any other examples of code with strictly POSIX compliant Makefiles, for example. (I did find one other Makefile which claimed to be POSIX-compatible; but in actual fact it used a GNU extension.) As far as I was concerned, strict POSIX compliance meant never having to say you're sorry for portability problems; if someone ran into problems with my standard-compliant code, well, they could fix their broken operating system.

And some people did. Unfortunately, despite the promise of open source, many users were unable to make such fixes themselves, and for a rather large number of operating systems the principle of standards compliance seems to be more aspirational than actual. Given the limits which would otherwise be imposed on the user base of my software, I eventually decided that it was necessary to add workarounds for some of the more common bugs. That said, I decided upon two policies:

  1. Workarounds should be disabled by default, and only enabled upon detecting an afflicted system.
  2. Users should be warned that a workaround is being applied.

The first policy is essential for preventing a scenario often found in older software: A workaround is added for one system, but then that workaround introduces a problem on a second system and so a workaround is added for the workaround, and then a problem is found with that second workaround... and ten years later there's a stack of workarounds to workarounds which nobody dares to remove, even though the original problem which was being worked around has long since been corrected. If a workaround is disabled by default, it's less likely to provoke such a stack of workarounds — and it's going to be much easier to remove them once they're no longer needed.

The second policy is important as a matter of education: Users deserve to know that they're running a broken operating system. And running broken operating systems they are doing. Here are some of the warnings people will see, along with explanations (more for the benefit of people who arrive here via google than for my regular readership):

But as passionate as I am about user education, there's a far more important reason for that second policy: Getting things fixed. All of these are problems we could have worked around silently; indeed, with the exception of the LLVM bug (which I don't think anyone else has noticed) all of them have been worked around silently. But while silent workarounds solve the immediate problem for one piece of software, they do nothing to help the next developer who trips over those bugs. Warnings, on the other hand, can help to get bugs fixed: Indeed, a few months ago I fixed a bug in FreeBSD for the sole reason that I was getting annoyed by one of my own warning messages! Even if the vast majority of people who see those warnings disregard them, any chance that the right developer will get the message and fix a bug is better than none.

My regular readers will know that I care deeply about producing correct code, offering bounties for issues as trivial as misplaced punctuation in comments. But it isn't just my own code I care about; I'm affected by bugs in all of the code I run, and even by bugs in code I don't run if I rely on someone else who does. So please, if you find a bug, don't just work around it; shout it from the rooftops in the hope that the right people will hear.

Because if we all stop accepting broken code, we might eventually end up with less broken code.

Posted at 2016-04-11 10:00 | Permanent link | Comments

FreeBSD on EdgeRouter Lite - no serial port required

I recently bought an EdgeRouter Lite to use as a network gateway; I had been using a cheap consumer wifi/NAT device, but I wanted the extra control I could get by running FreeBSD rather than whatever mangled version of Linux the device came with. Someone wrote instructions on installing FreeBSD onto the EdgeRouter Lite two years ago, but they rely on using the serial port to reconfigure the boot loader — perfectly straightforward if you have a serial cable and know what you're doing, but I decided to take the opportunity to provide a more streamlined process.



  1. Fetch a FreeBSD HEAD source tree. (FreeBSD 10-STABLE is not supported yet. I think this might change between now and 10.3-RELEASE.)
  2. Download the image building script.
  3. Run ./buildimg.sh /path/to/src/tree disk.img.
  4. Remove three small screws from the back of the EdgeRouter Lite. Open the case and remove the USB drive. (Mine was held very firmly in place. I found that wiggling it towards and away from the board allowed me to gradually ease it free.)
  5. Plug the USB disk into the system where you built the FreeBSD image.
  6. Run dd if=/dev/USBDISK of=ERL.img where USBDISK is the name of the USB disk device (probably da0), to make a backup of the EdgeRouter Lite software in case something breaks and you need to restore it later.
  7. Run dd if=disk.img of=/dev/USBDISK (where USBDISK is as before) to write the FreeBSD disk image onto the EdgeRouter Lite USB disk.
  8. Plug the USB disk back into the EdgeRouter Lite, close the box, and replace the three screws.

There are three gigabit ethernet ports on the EdgeRouter Lite, marked on the case as "eth0", "eth1", and "eth2"; in FreeBSD, they show up as "octe0", "octe1", and "octe2" in the same order. With the configuration on my image:

That's pretty much all you need to know to install and use FreeBSD on the EdgeRouter Lite; but there are some interesting tricks involved in the script which builds the disk image, so for the rest of this blog post I will provide a brief "walkthrough" of the script.

#!/bin/sh -e
Shell scripts are run by /bin/sh. (No matter what some misguided Linux users may think, /bin/bash is not a standard shell.) The -e option tells the shell interpreter to exit if any of the commands fail — if something goes wrong, we should stop and let the user see what happened rather than continuing and producing a broken disk image later.
if [ -z "$1" ] || [ -z "$2" ]; then
	echo "buildimg.sh srcdir disk.img"
	exit 1
This script takes two options: The location of the FreeBSD source tree, and the name of the file to use for the disk image.
# Set environment variables so make can use them.
export TARGET=mips
export TARGET_ARCH=mips64
The EdgeRouter Lite is a 64-bit MIPS system; FreeBSD HEAD has a kernel configuration already defined for it.
export WITHOUT_MODULES="cxgbe mwlfw netfpga10g otusfw ralfw usb rtwnfw"
Unfortunately that kernel configuration disables building kernel modules; we want those so that we can use pf later. Some modules fail to build — I didn't investigate exactly why, but it was something related to firmware blobs — so we turn those off explicitly.
# Create working space
WORKDIR=`env TMPDIR=\`pwd\` mktemp -d -t ERLBUILD`
We create some temporary working space under the current directory. On many systems /tmp isn't large enough to hold a complete installation of FreeBSD, so I overrode the default there.
# Build MIPS64 world and ERL kernel
JN=`sysctl -n hw.ncpu`
( cd $SRCDIR && make buildworld -j${JN} )
( cd $SRCDIR && make buildkernel -j${JN} )
Build the MIPS64 world and kernel. The -j flag tells make to run several commands in parallel; we consult sysctl to find out how many CPUs we have available for the build.
# Install into a temporary tree
mkdir ${WORKDIR}/tree
( cd $SRCDIR && make installworld distribution installkernel DESTDIR=${WORKDIR}/tree )
We create a tree and install FreeBSD into it. The installworld and installkernel targets install the userspace and kernel binaries respectively; the distribution target installs standard configuration files.
# Download packages
cp /etc/resolv.conf ${WORKDIR}/tree/etc/
pkg -c ${WORKDIR}/tree install -Fy pkg djbdns isc-dhcp43-server
rm ${WORKDIR}/tree/etc/resolv.conf
The FreeBSD project provides precompiled binary packages for the 64-bit MIPS architecture; this allows us to put packages into the image we're building while avoiding the headaches of cross-building them. However, we cannot cross-install packages either, since packages can run scripts when they are installed — scripts which (since we're not building this disk image on a MIPS64 system) we won't be able to run. Instead, we simply download the packages into the image; they will be installed when the system first boots.
# FreeBSD configuration
cat > ${WORKDIR}/tree/etc/rc.conf <<EOF
The /etc/rc.conf file is the "master configuration file" on FreeBSD; most enabling/disabling of services is done here, as well as some more specific configuration.
Every host needs a name. We'll call this "ERL", lacking any better inspiration.
We're building a disk image which we'll write onto the provided USB disk, but the image is smaller than the disk; when the system first boots, this tells it to expand the root partition to fill the available space.
This is probably unnecessary, but I like to have a memory disk mounted on /tmp; if for some reason temporary files get created here, this will avoid burning up the flash storage.
ifconfig_octe1=" netmask"
ifconfig_octe2=" netmask"
We run DHCP on the "upstream" connection, but provide static network parameters for the "LAN" connections.
We're going to use the PF firewall; and we're going to be forwarding packets (both via the network address translation and between the two LAN ports) so we need that option too.
dhcpd_ifaces="octe1 octe2"
We don't want to run sendmail; we do want to run sshd (we'll use PF to restrict access, however); we do want to run ntpd, and we want it to set its clock when it starts, no matter how far off it is (the EdgeRouter Lite doesn't have a battery-powered clock, so it boots with a wildly wrong time set); we want to run svscan so that it can launch dnscache for us; and we want to run a dhcp daemon for the two LAN interfaces.
cat > ${WORKDIR}/tree/etc/pf.conf <<EOF
# Allow anything on loopback
set skip on lo0

# Scrub all incoming traffic
scrub in

# NAT outgoing traffic
nat on octe0 inet from { octe1:network, octe2:network } to any -> (octe0:0)

# Reject anything with spoofed addresses
antispoof quick for { octe1, octe2, lo0 } inet

# Default to blocking incoming traffic but allowing outgoing traffic
block all
pass out all

# Allow LAN to access the rest of the world
pass in on { octe1, octe2 } from any to any
block in on { octe1, octe2 } from any to self

# Allow LAN to ping us                                       
pass in on { octe1, octe2 } inet proto icmp to self icmp-type echoreq

# Allow LAN to access DNS, DHCP, and NTP
pass in on { octe1, octe2 } proto udp to self port { 53, 67, 123 }
pass in on { octe1, octe2 } proto tcp to self port 53

# Allow octe2 to access SSH
pass in on octe2 proto tcp to self port 22
Fairly straightforward PF configuration: NAT outgoing traffic onto the "upstream" connection; allow the local network to access DNS, DHCP, and NTP; and allow octe2 to access SSH. I opted to only allow ICMP echo request packets from the LAN side — some people prefer to respond to pings from anywhere, but I decided that for a general purpose image it was better to err in the direction of being silent. Similarly I decided to simply drop bad packets rather than sending TCP RST or ICMP unreachable responses.
mkdir -p ${WORKDIR}/tree/usr/local/etc
cat > ${WORKDIR}/tree/usr/local/etc/dhcpd.conf <<EOF
option domain-name "localdomain";
subnet netmask {
        option routers;
        option domain-name-servers;
subnet netmask {
        option routers;
        option domain-name-servers;
This provides a basic configuration for ISC DHCPD. I have a feeling that this could be simplified to have a single configuration block covering both LAN ports.
# Script to complete setup once we're running on the right hardware
mkdir -p ${WORKDIR}/tree/usr/local/etc/rc.d
cat > ${WORKDIR}/tree/usr/local/etc/rc.d/ERL <<'EOF'
I mentioned earlier that we couldn't cross-install packages; we take care of that now, with a script which runs the first time FreeBSD boots. The quotes around EOF in the here-document syntax instruct the shell that variables should not be expanded — important since we're creating a shell script which uses several shell variables.
# KEYWORD: firstboot
The "firstboot" keyword tells /etc/rc that this script should only be run the first time that the system boots.

# This script completes the configuration of EdgeRouter Lite systems.  It
# is only included in those images, and so is enabled by default.

. /etc/rc.subr

: ${ERL_enable:="YES"}

This is fairly standard rc.d script boilerplate.

	# Packages
	env SIGNATURE_TYPE=NONE pkg add -f /var/cache/pkg/pkg-*.txz
	pkg install -Uy djbdns isc-dhcp43-server
We want to install the two packages we downloaded into the image earlier.
	# DNS setup
	pw user add dnscache -u 184 -d /nonexistent -s /usr/sbin/nologin
	pw user add dnslog -u 186 -d /nonexistent -s /usr/sbin/nologin
	mkdir /var/service
	/usr/local/bin/dnscache-conf dnscache dnslog /var/service/dnscache
	touch /var/service/dnscache/root/ip/192.168
We configure dnscache to be launched by svscan and respond to DNS requests from the LAN.
	# Create ubnt user
	echo ubnt | pw user add ubnt -m -G wheel -h 0
We could have created this user while creating the disk image, but since we needed to have a firstboot script anyway it was easier to do it here. The -h 0 option means "read the password from standard input", which is why we're echoing it in from there.
	# We need to reboot so that services will be started
	touch ${firstboot_sentinel}-reboot
Part of the rc.d "firstboot" mechanism is to allow scripts to ask for the system to be rebooted after the first boot (and all the associated system initialization) is complete. In this case, we need to reboot in order to have svscan and isc-dhcpd running (since they weren't installed yet when the boot process started).

load_rc_config $name
run_rc_command "$1"
chmod 755 ${WORKDIR}/tree/usr/local/etc/rc.d/ERL
More boilerplate. The rc.d script must be executable.
# We want to run firstboot scripts
touch ${WORKDIR}/tree/firstboot
The sentinel file /firstboot tells FreeBSD that the system is booting for the first time and "firstboot" scripts should be run. At the end of the first boot, /etc/rc deletes this file.
# Create FAT32 filesystem to hold the kernel
newfs_msdos -C 33M -F 32 -c 1 -S 512 ${WORKDIR}/FAT32.img
mddev=`mdconfig -f ${WORKDIR}/FAT32.img`
mkdir ${WORKDIR}/FAT32
mount -t msdosfs /dev/${mddev}  ${WORKDIR}/FAT32
cp ${WORKDIR}/tree/boot/kernel/kernel ${WORKDIR}/FAT32/vmlinux.64
umount /dev/${mddev}
rmdir ${WORKDIR}/FAT32
mdconfig -d -u ${mddev}
The EdgeRouter Lite boot loader expects to launch a Linux kernel which is found at /vmlinux.64 within a FAT32 filesystem. Fortunately, it doesn't check that the kernel it's launching is Linux... so we create a FAT32 filesystem and drop a FreeBSD kernel in, named "/vmlinux.64" so that the EdgeRouter Lite boot loader launches it for us. (Our kernel is only about 10 MB, but the minimum size for a FAT32 filesystem is 33 MB.)
# Create UFS filesystem
echo "/dev/da0s2a / ufs rw 1 1" > ${WORKDIR}/tree/etc/fstab
makefs -f 16384 -B big -s 920m ${WORKDIR}/UFS.img ${WORKDIR}/tree
We use the makefs tool to create a UFS filesystem from the installed FreeBSD tree. The MIPS64 hardware is big-endian, and UFS is not endian-agnostic, so we need to tell makefs to create a big-endian filesystem; this also means that (assuming we're using a little-endian system to build this disk image) we can't mount the filesystem on the system we're using to create it.
# Create complete disk image
mkimg -s mbr		\
    -p fat32:=${WORKDIR}/FAT32.img \
    -p "freebsd:-mkimg -s bsd -p freebsd-ufs:=${WORKDIR}/UFS.img" \
    -o ${IMGFILE}
The EdgeRouter Lite boot loader expects the kernel to be found on the first MBR slice; and the FreeBSD ERL kernel configuration expects the root filesystem to be found at da0s2a so we'd better put it there.
# Clean up
chflags -R noschg ${WORKDIR}
rm -r ${WORKDIR}
Once we finish building the disk image, we don't need our staging tree or the separate filesystem images any more.

Posted at 2016-01-10 04:20 | Permanent link | Comments

Recent posts

Monthly Archives

Yearly Archives