partitioning for optimum firewall device running

[daemonbak]

hi all!

I am trying to setup my OpenBSD PF Firewall/Router/DHCP/DNS Caching box but am getting stuck at the partitioning setup. I could use some guidance.

System Stats:

- Dual Nic micro-itx board with pico psu
- 16GB low profile lexar USB drive as sd0 (system)
- 4GB Ram
- Purpose: Embedded OpenBSD PF Firewall/Router/DHCP/DNS Caching

Questions:

• 1 - Do I really need that many partions that OpenBSD auto makes (see Default "Auto" OpenBSD partitioning) below, or will my setup (see My "Custom" OpenBSD partitioning) below work great?
• 2 - I am not running X so /usr/X11R6 can be deleted
• 3 - Is there any security/OpenBSD magic that needs those on seperate partitions that would be hindered in my below setup?
• 4 - Do I need to specify the size in fstab for the mfs memory mounts (-s=20M). I have seen with and without size options
• 5 - Should I allocate more/less space for any of these physical partitions?
• 6 - Should I allocate more/less space for any of the MFS partitions in memory 9now i have it at around 100M)?
• 7 - /usr will be part of the read only '/' partition and not it's own partition (is this wrong)
• 8 - Am I doing this wrong or does this seem right for a 16GB flash embedded firewall device?

Critical Info:

Default "Auto" OpenBSD partitioning:

Filesystem	Mounted on
 /dev/sd2a	/
 /dev/sd2l	/home
 /dev/sd2d	/tmp
 /dev/sd2f	/usr
 /dev/sd2g	/usr/X11R6
 /dev/sd2h	/usr/local
 /dev/sd2k	/usr/obj
 /dev/sd2j	/usr/src
 /dev/sd2e	/var

In my scheme:

- I have deleted /tmp, /usr, /usr/X11R6, /usr/local, /usr/obj, /usr/src and /var
- /, /mfs, /home and will still be on the flash drive
- /, /mfs, /home and will be read only to help usb drive last longer
- /dev, /etc, /root and /var will be run in memory to be written to
- /mfs will be rsync backup for /dev, /etc, /root and /var

My "Custom" OpenBSD partitioning:

 a: / = 6.5GB
 b: swap = 1 sector
 d: /mfs = ~2GB (remainder of flash drive),
 e: /home = 6.5GB of your CF diskspace.

fstab:

sd0a / ffs ro,noatime,softdep 1 1
sd0d /mfs ffs ro,noatime,softdep 1 2
sd0e /home ffs ro,noatime,softdep 1 2
swap /root mfs rw,nosuid,-P=/mfs/root,-s=20M 0 0
swap /var mfs rw,nosuid,-P=/mfs/var,-s=30M 0 0
swap /dev mfs rw,nosuid,-P=/mfs/dev,-s=30M,-i=256 0 0
swap /etc mfs rw,nosuid,-P=/mfs/etc,-s=20M 0 0

Any and all input is so greatly appreciated!
Thanks as always.

[jggimi]

Your applications listed will fit in a few hundred megabytes, so you may partition how you wish.

You've eliminated a separate /tmp filesystem, and plan to make the root filesystem read-only. That's a conflict, since /tmp needs to be writeable. You can either leave root read/write, or create a memory file system for /tmp. If you select the latter, you will want to be sure your memory filesystem uses chmod 1777 for /tmp. There are two choices: mount_mfs(8) and the newer and somewhat experimental (see CAVEATS and BUGS) mount_tmpfs(8). You may also be able to use a memory filesystem for /var/run, depending on your use of the filesystem. I do this on one or two machines, and I ensure the utmp(5) file is created there via rc.local(8).

I perceive three operational advantages to partitioning -- whatever schema you implement.

1. Ensuring the root partition is located within the the BIOS addressing limits for boot blocks
2. Mount options can be used to limit functionality on some filesystems, for defense-in-depth after compromise.
3. Run-away storage growth will only affect the application(s) using the full partition ... other applications should continue to run if they have no dependency on applications impacted by a full partition.

Edited to add: /etc, also should be read/write -- rc(8) generates server keys and sets permissions of hostname.if files, and some administrative tools like dump(8) update status files stored there.

[daemonbak]

Thank you for the quick response.

Originally Posted by jggimi

Your applications listed will fit in a few hundred megabytes, so you may partition how you wish.

You've eliminated a separate /tmp filesystem, and plan to make the root filesystem read-only. That's a conflict, since /tmp needs to be writeable. You can either leave root read/write, or create a memory file system for /tmp.

I was going to cp -pRv /tmp to /var/tmp, delete /tmp and then make a symlink from /tmp pointed to /var/tmp. Would this work correctly?

Originally Posted by jggimi

/etc, also should be read/write -- rc(8) generates server keys and sets permissions of hostname.if files, and some administrative tools like dump(8) update status files stored there.

/etc will be run in memory to allow r/w. I could rsync ramdisk's /etc/ to /mfs/etc/on shutdown.rc , then on boot I could rsync the /mfs/etc backup to the /etc in memory. Same applies to /dev, /root and /var. Unless you see an inherent issue with this.

Continuing on from OP:

• 4 - Do I need to specify the size in fstab for the mfs memory mounts (-s=20M). I have seen with and without size options. Do i want to limit it or leave it open. perhaps there is a chance it will take up all memory available and grind the firewall to a halt.
• 6 - Should I allocate more/less space for any of the MFS partitions in memory now I have it at around 100M)?
• 7 - /usr will be part of the read only '/' partition and not it's own partition (is this wrong)
• 9 - Additional question, should i use DUID or old fashioned partition labels?

Thanks again. You are very helpful.

I wish there was an openbsd embedded flash openbsd tutorial written in the last few years. Most of my idea has been lifted from here: http://techblagh.blogspot.com/2008/0...kris-5501.html

I also plan on running carp and a few others as well.

[hanzer]

jggimi mentioned [the new] tempfs which looks interesting. You might get better performance/efficiency with that over mfs (only speculation based on a cursory perusal of the docs).

Just in case you haven't seen this yet, hier(7) and "How should I partition my disk?" introduce the structure and some of the issues/concerns/factors.

I noticed that you have partitions for /root and /home. That seems a little odd assuming that you're not going to be storing much/any user data on this network appliance. I might try a layout like this:

Partitions
/
<swap>
/usr
/mfs

tmpfs
/etc
/tmp
/var

I'm not sure how much activity occurs in /dev - whether or not it needs to be on a tmpfs, I don't know (but I'm curious (now)).

Edit: Oh, and softdep on a read-only filesystem might not make much sense.

[jggimi]

Originally Posted by daemonbak

I was going to cp -pRv /tmp to /var/tmp, delete /tmp and then make a symlink from /tmp pointed to /var/tmp. Would this work correctly?

rc(8) clears /tmp at boot, so there is nothing to copy. But please be warned, starting with 5.7 /var/tmp is no longer a unique directory in /var where temporary files can be stored to survive a reboot. It is now just a symbolic link back to /tmp, remaining only as a legacy vestige for programs dependent on that directory by name. See this link to the -current heir(7) man page.

/etc will be run in memory to allow r/w. I could rsync ramdisk's /etc/ to /mfs/etc/on shutdown.rc , then on boot I could rsync the /mfs/etc backup to the /etc in memory. Same applies to /dev, /root and /var. Unless you see an inherent issue with this.

When I previously published live media systems based on OpenBSD, which used CD/DVD, I had to do this, and used MFS filesystems for /root, /etc, /var, /dev, and /home. Note that /dev needs to be read-write also, since devices must be writeable. My apologies for not mentioning this earlier. I had forgotten.

In order to mount these mfs filesystems, I needed to modify rc(8). And, since these were live systems, and therefore always read-only OSes, I never had to worry about saving any changes. Users could store /home files on external media if needed.

A lot of people modified their copy of the rc script when they were using early flash memories for storage, because they had relatively short lifetimes for write operations. That's no longer an operational issue. Flash memories now available use one of the wear-leveling technologies, so that constant writes to the same LBAs are spread across the flash media, extending their lifespans.

I have several systems that use compact flash as their storage, and I don't worry about storage failures at all. Those systems use carp(4) for redundancy, and I have good, tested backup and recovery procedures. I have not had any media failures in two years of operation, with the exception of one DOA flash card that was never deployed. Will I have failures in production? Sure. But I'm not at all worried about recovery from such a failure.

Do I need to specify the size in fstab for the mfs memory mounts (-s=20M). I have seen with and without size options.

With tmpfs, if you do not specify a size you get the total unallocated space (RAM + swap) at that moment. I have set sizes for tempfs when I've wanted to limit, such as with /tmp. When I didn't care, such as with /usr/ports/pobj, I've let it set the maximum. You can mount multiple tempfs file systems without specifying a size, each will be granted the unallocated size at the moment of creation.

With mfs, on the other hand, if you don't specify a size you get a minimum. Testing just now on i386, I get 8 MB.

Should I allocate more/less space for any of the MFS partitions in memory now I have it at around 100M)?

I don't have an opinion for this. "Allocate what you need now and for growth" is pretty trite, and if you don't yet know, pretty useless. But that's my recommendation.

7 - /usr will be part of the read only '/' partition and not it's own partition (is this wrong)

For unchanging operations, that's fine. But if /usr/local is read only, you won't be able to add packages. Some of what you plan to do will restrict you from making changes easily -- that's another reason why on my flash memory systems I made the choice to use read/write.

...should i use DUID or old fashioned partition labels?

Up to you. Again, trite. The benefits of DUID is not having to worry about device numbering. The drawback is during disaster recovery or infrastucture swaps: if you restore onto a new drive, you create a new disklabel. Which means a new DUID. You either modify the disklabel to have the old DUID, or you modify fstab(5) to use the new DUID.

I wish there was an openbsd embedded flash openbsd tutorial written in the last few years.

The biggest changes: elimination of diskette-based tiny OpenBSD systems, requiring lots of gyrations, and elimination of flash memories that don't wear-level. Today, there's no significant operational difference between one of these i386/amd64 "embedded" systems and a larger platform, except lack of fans, and lack of video.

[hanzer]

Also, there are 14 usable partitions per disk: a-p (b for swap and c is reserved). I like to use them (but I'm kind of weird). Just to give you an idea of disk usage on a running machine:

hanzer@lucidrine[/home/hanzer]$ df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd0a     1007M   67.8M    889M     7%    /
/dev/sd0p     13.3G    507M   12.1G     4%    /home
/dev/sd0d      3.9G   33.3M    3.7G     1%    /tmp
/dev/sd0h      2.0G    490M    1.4G    26%    /usr
/dev/sd0i      7.9G    2.1G    5.4G    28%    /usr/local
/dev/sd0n      2.0G    781M    1.1G    41%    /usr/obj
/dev/sd0l      3.9G    1.8G    2.0G    48%    /usr/ports
/dev/sd0m      7.9G    2.8G    4.7G    37%    /usr/ports/pobj
/dev/sd0j      2.0G    775M    1.1G    40%    /usr/src
/dev/sd0k      2.0G    596M    1.3G    31%    /usr/xenocara
/dev/sd0o      2.0G    451M    1.4G    24%    /usr/xobj
/dev/sd0e      7.9G   19.5M    7.5G     0%    /var
/dev/sd0f      7.9G    4.0K    7.5G     0%    /var/snort
/dev/sd0g      7.9G    342M    7.1G     4%    /var/squid
hanzer@lucidrine[/home/hanzer]$ cat /etc/fstab
ada3ffff20e05c89.b none swap sw
ada3ffff20e05c89.a / ffs rw 1 1
ada3ffff20e05c89.p /home ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.d /tmp ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.h /usr ffs rw,nodev 1 2
ada3ffff20e05c89.i /usr/local ffs rw,nodev 1 2
ada3ffff20e05c89.n /usr/obj ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.l /usr/ports ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.m /usr/ports/pobj ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.j /usr/src ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.k /usr/xenocara ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.o /usr/xobj ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.e /var ffs rw,nodev,nosuid 1 2
ada3ffff20e05c89.f /var/snort ffs rw,nodev,nosuid,noatime,softdep 1 2
ada3ffff20e05c89.g /var/squid ffs rw,nodev,nosuid,noatime,softdep 1 2

This is not ideal, just the latest build. When I upgrade to 5.7, the layout will be changed. And, of course, your system will most likely be very different.

[rocket357]

I haven't read through this entire thread, but given your starting point of 4 GB of RAM and 16 GB of storage, I'd say have fun with it. I'm running an ancient PIII with 384 MB RAM on an 8 GB CF card, and it has more room than I can really think up uses for. It's running pf (of course), dhcpd, unbound, dnscrypt-proxy, tor, bgpd, isakmpd, ifstated, ladvd, and a few other odds and ends with no swap and it rarely if ever goes over 125 MB of RAM usage. It also has a "quite possibly" insane number of vlan interfaces, an LACP trunk, and dual uplinks configured =\

Probably not the wisest configuration (it could certainly stand to have a "feature-set reduction" and a carp twin), but I've only experienced one unplanned outage (CF card failure) in the past ~8-10 years I've been running it.

[daemonbak]

I have a couple questions.

if I were to go this route:

Actual Partitions
/ (ro)
<swap> (1 sector only)
/usr (ro)
/mfs (rw)

Memory Filesystem Partitions (tmpfs/mfs)
/etc (rw)
/tmp (rw)
/var (rw)
/dev (rw)

Would this cause any issues with logging in? I would have root login off for ssh, but I have a user that would ssh in and then elevate to root. But if I have /root and /home in the / partition (which is read only) is this an issue?


Also what about /usr and /usr/local. Do these need to be writeable?

Besides these following partitions, do any other need to be rw or just these:
/etc
/mfs
/tmp
/var
/dev

Thanks

[jggimi]

Originally Posted by daemonbak

Would this cause any issues with logging in?

No, but I noted the following:

<swap> (1 sector only)

Why bother? This is the same as running without any swap space. If you run out of (or have no) swap when needed, your system will stop operating, or fail. Which will happen will depend on whether it is a user or kernel process which fails.

Swap space larger than physical RAM is required if you wish to capture kernel core dumps after a panic. See crash(8) and ddb(4).

/mfs (rw)

This is your own partition, and as I do not understand your intended purpose I have no opinion regarding it.

But if I have /root and /home in the / partition (which is read only) is this an issue?

My preferred shell, tcsh, writes to a $HOME/.history file. Your user's shell or applications might not permit a read-only /home directory structure.

Also what about /usr and /usr/local. Do these need to be writeable?

Only by pkg_add(1) and pkg_delete(1).

Besides these following partitions, do any other need to be rw...

I think I've made a good case for permitting /home to be writable.

TL;DR for all my posts in this thread: Yes, you can dance across this minefield if you want; you own the music. But I think it would be better for you if you don't. Your proposed OS architecture will add complications that make the system difficult to manage, maintain, administrate, and support. All for -- in my opinion -- what is an insignificant operational MTBF-related benefit for any flash memory device manufactured in the last dozen years.

[daemonbak]

Originally Posted by jggimi

No, but I noted the following:
Why bother? This is the same as running without any swap space. If you run out of (or have no) swap when needed, your system will stop operating, or fail. Which will happen will depend on whether it is a user or kernel process which fails.

Swap space larger than physical RAM is required if you wish to capture kernel core dumps after a panic. See crash(8) and ddb(4).
This is your own partition, and as I do not understand your intended purpose I have no opinion regarding it.
My preferred shell, tcsh, writes to a $HOME/.history file. Your user's shell or applications might not permit a read-only /home directory structure.
Only by pkg_add(1) and pkg_delete(1).
I think I've made a good case for permitting /home to be writable.

TL;DR for all my posts in this thread: Yes, you can dance across this minefield if you want; you own the music. But I think it would be better for you if you don't. Your proposed OS architecture will add complications that make the system difficult to manage, maintain, administrate, and support. All for -- in my opinion -- what is an insignificant operational MTBF-related benefit for any flash memory device manufactured in the last dozen years.

I obviosuly don't get it. I have read Absolute OpenBSD, the FAQ/OBSD Manual and numerous blog posts. So I should have a handle on it. But I don't.

So w/ 4GB memory on a 16GB flash drive, how would you partition it?

I don't need any of the X11 partitions as this will be ssh only.

From reading the book, it looks like I don't need /usr/src and /usr/obj since I am not compiling from source. Although this may be an issue on upgrades. The book didn't go into details.

Would you run var/tmp and /tmp in mfs or just take that out of the equation.

I am going for stability and security.

So how would you partition my disk knowing my setup.

Thanks and sorry for being the guy that doesn't get it.

[jggimi]

The OS without the X filesets will fit in less than 300 Megabytes of disk space. That includes the compiler fileset. More storage will be needed if you run applications that require X libraries -- but your list in your first post didn't mention any.

One of my firewalls, running i386, runs the usual daemons you might find on firewall/routers: dhcpd, ftpproxy, identd, ifstated, ntpd, smtpd, syslog, and unbound built-in services. It also runs nginx and ddclient daemons, and has a few client applications (such as lynx) for administrative use. Including dependencies, there are 14 packages installed. This system runs in a single "a" partition on a 1GB compact flash card, and that filesystem is currently using 309 Megabytes. The card has a "b" partition which is swap space.

I have another system which is its twin -- it is a backup firewall/router, and as it is a backup firewall/router it also is a server for some additional applications, such as PHP applications. As this system is an application server, it has a 4GB compact flash card for storage. That system includes many more applications, including those that require both X libraries and X fonts, so the xbase*.tgz and xfont*.tgz filesets are also installed. At this time, I am using a single partition once again, and 1.4 GB are in use.

While these two systems do not use multiple partitions, there are benefits to having them, which I am not able to exploit with my single partition use-case. They are 1) managed growth, and 2) security.

Managed growth: runaway storage allocation on one filesystem will not impact any other filesystems. This might permit a system to keep running any remaining unaffected applications. Your defined list of applications doesn't do a lot of file allocation, so I don't perceive this as being of immediate benefit.

Security: some filesystem mount options can restrict use: nodev, nosuid, noexec, and as you've already noted, rdonly. These can be used as a layer of administrative security for systems which are multiuser, or as a layer of security limiting the capabilities of any attack vector that might issue arbitrary file input/output instructions .
The installation script provides automatic defaults which don't fit every use case -- they can't. You've already noted that there are filesystems you will never use.

I recommend starting with a custom configuration, with the intent to add to it later. Start with two partitions:

a: 512 MB root partition
b: swap space. If you want to ever store a kernel core dump, make this larger than 4GB.

That's it. But, this is a starting point. If, in the future, you decide you want to add applications that have active databases, you can create a partition for the application, or move /var, or /var/db, into its own partition. If you decide you want to do development, you can later add /usr/ports or /usr/src. Or in the future, you can separate /home into its own partition. Easy to do so.

I recommend this because at this time, your intended application set uses very little storage, and your network applications are all privilege separated. But what we intend today very rapidly changes to something we did not initially intend. Give yourself room to grow, change, and add without encumbering yourself. Otherwise, you'll end up re-installing in order to reconfigure, because reinstallation will be easier.

If this sort of simple configuration is anathema, take the defaults, then delete the partitions you don't believe you'll need. Either way, your total storage use will start at about 300 MB. Less, if you elect not to install the compiler set.

[daemonbak]

What about this: (see image).

I would keep /tmp and /var as tmpfs or mfs and sync it to the mfs partition and back on halt and boot. There would be ~3GB unalocated space for future use.

[ocicat]

Originally Posted by daemonbak

What about this:

The rule of thumb I use (which is advocated by many...) is to set the size of swap as 2x RAM + 1MB. I also allocate the same for /var/crash which is discussed in crash(8). As previously mentioned by jggimi, this allows the state of the kernel to be written to disk following a crash in order to ascertain the cause.

There would be ~3GB unalocated space for future use.

While I understand the advantages of unallocated space, 16GB isn't a space-rich environment. I would highly suggest reconsidering your partition sizes & experiment. As you should be seeing in this thread, there are a lot of different opinions, some of which are grounded. Only you know what your requirements really are. Any deviations from commonly used practices puts the responsibility of any repercussions on you.

[ibara]

Without contributing anything useful to this thread, you can type simply:

> p m

and get the same output

[jggimi]

/var

Since you're not running any databases, the only volatile files in that branch are in /var/run. I've got a workstation running with /var/run as tmpfs. It's never backed up, because it only contains run-time files such as process identifiers. The only thing I do is add this line to /etc/rc.local:

touch /var/run/utmp

The rest of /var has more permanent files, and you'll likely find it easier to keep /var a normal FFS filesystem rather than try to gyrate it in and out of a ramdisk.

I mentioned that swap has to be larger than RAM if you ever want to capture a kernel panic, and I note you've got it a bit larger. savecore(8) is run by rc(8) to store any core dumps in /var/crash during boot. If that space isn't sufficiently large, you might have to run savecore manually, pointing to an alternate directory with sufficient space. On my primary firewall, there's not enough storage to save any core dump, and in the event of any panics (none, after several years) I plan to run savecore manually to store to either an NFS mount or a USB attached drive, depending upon personal convenience at that time.

Other than these last comments, your design looks fine to me. My concern wasn't so much how you slice/dice the drive so much as avoiding issues caused by having read/only partitions for data that was not intended by the Project to be read/only.