DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD Security

OpenBSD Security Functionally paranoid!

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 5th February 2016
notooth notooth is offline
Shell Scout
 
Join Date: Jul 2015
Posts: 125
Smile Solution for updating file safely?

Hi,

Can anyone let me know if OpenBSD has any solution to prevent file damaged if the system crashed while updating large files?
Reply With Quote
  #2   (View Single Post)  
Old 5th February 2016
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default

At this moment there is not a journalling filesystem for OpenBSD yet.

You could do something like this:
  1. Copy largefile.txt to largefile.txt.tmp
  2. Update largefile.txt.tmp
  3. Rename/move largefile.txt -> largefile.txt.old
  4. Rename/move largefile.txt.tmp -> largefile.txt
  5. Delete largefile.txt.old

From rename(2):
Code:
     int     rename(const char *from, const char *to);   

DESCRIPTION
     The rename() function causes the link named from to be renamed as to.  If
     to exists, it is first removed.  Both from and to must be of the same
     type (that is, both directories or both non-directories), and must reside
     on the same file system.

     rename() guarantees that if to already exists, an instance of to will
     always exist, even if the system should crash in the middle of the
     operation.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 5th February 2016 at 11:01 PM.
Reply With Quote
  #3   (View Single Post)  
Old 5th February 2016
albator albator is offline
Shell Scout
 
Join Date: Jul 2011
Posts: 98
Default

I think it's possible to only use the steps 1,2 and 4.
In case of a crash, based on the information you provided, we may end up with only the original file.
But within a same file system, a rename operation is very fast.

Last edited by albator; 5th February 2016 at 11:58 PM.
Reply With Quote
  #4   (View Single Post)  
Old 6th February 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

The OpenBSD GENERIC kernels include an install target for make(1). Here's an excerpt from a kernel Makefile.
Code:
cmp -s bsd /bsd || ln -f /bsd /obsd
        cp bsd /nbsd
        mv /nbsd /bsd
If the newly built kernel is different from the one in the root directory, it creates a linked directory entry under the new name /obsd. Then it copies the new kernel to the root directory with the temporary name /nbsd, and only after the copy has completed does it issue the move.

At every step, the old kernel is recoverable, though the admin may need to refer to it by the name /obsd. The copy command may fail, or the OS may fail. If so, the /bsd file is left unaltered.

The move command afterwards is on the same filesystem and as albator stated is relatively brief, since it only touches metadata. As the original kernel file /bsd was linked as /obsd, none of that file's sectors will be touched at all during the move process. The move command will merely unlink the name /bsd from that file, and apply it instead to the /nbsd file which was already safely copied. This is because the /bsd file and /obsd file were linked -- these directory entries point to the same file on disk.

--

There are other options to consider for increasing data integrity. Consider enabling sync and disabling softdep, if optionally used on the filesystem in question. See mount(8).

Last edited by jggimi; 6th February 2016 at 01:00 AM. Reason: clarity
Reply With Quote
  #5   (View Single Post)  
Old 6th February 2016
notooth notooth is offline
Shell Scout
 
Join Date: Jul 2015
Posts: 125
Default

Thank you all
Reply With Quote
  #6   (View Single Post)  
Old 6th February 2016
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default

There is a difference in behaviour if the target file already exists. rename(2) will overwrite an already existing file , while link(2) will refuse to create a new link (directory entry), if that filename/directory entry already exists.

From link(2):
Code:
NAME
     link - make hard link to a file

     int
          link(const char *name1, const char *name2);

DESCRIPTION

     The link() function atomically creates the specified directory entry
     (hard link) name2 with the attributes of the underlying object pointed at
     by name1.  If the link is successful: the link count of the underlying
     object is incremented; name1 and name2 share equal access and rights to
     the underlying object.

     If name1 is removed, the file name2 is not deleted and the link count of
     the underlying object is decremented.

     name1 must exist for the hard link to succeed and both name1 and name2
     must be in the same file system.  As mandated by POSIX.1 name1 may not be
     a directory.

ERRORS

     [snip]

     [EEXIST]           The link named by name2 does exist
Keep in mind that this is the C function. The ln(1) command line utility will also refuse to create a new directory entry, if one already exists. You will have to use the -f option to unlink an already existing file, allowing the link to be made:

Code:
$ echo Shere Khan >new/animal
$ echo Simba >temp/lion

$ ln temp/lion new/animal
ln: new/animal: File exists

$ ls -li new/animal temp/lion
571653 -rw-r--r--  1 adriaan  wheel  11 Feb  6 16:22 new/animal
571654 -rw-r--r--  1 adriaan  wheel   6 Feb  6 16:22 temp/lion

$ ln -f temp/lion new/animal

$ ls -li new/animal temp/lion
571654 -rw-r--r--  2 adriaan  wheel  6 Feb  6 16:22 new/animal
571654 -rw-r--r--  2 adriaan  wheel  6 Feb  6 16:22 temp/lion

$ stat -s new/animal
st_dev=0 st_ino=571654 st_mode=0100644 st_nlink=2 st_uid=1001 st_gid=0 
st_rdev=2292638 st_size=6 st_atime=1454772179 st_mtime=1454772179
st_ctime=1454772265 st_blksize=16384 st_blocks=4 st_flags=0

$ stat -s temp/lion
st_dev=0 st_ino=571654 st_mode=0100644 st_nlink=2 st_uid=1001 st_gid=0
st_rdev=2292638 st_size=6 st_atime=1454772179 st_mtime=1454772179
st_ctime=1454772265 st_blksize=16384 st_blocks=4 st_flags=0

$ cat new/animal
Simba
$ cat temp/lion
Simba

$ rm temp/lion

$ stat -s new/animal
st_dev=0 st_ino=571654 st_mode=0100644 st_nlink=1 st_uid=1001 st_gid=0
st_rdev=2292638 st_size=6 st_atime=1454772179 st_mtime=1454772179
st_ctime=1454772265 st_blksize=16384 st_blocks=4 st_flags=0
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #7   (View Single Post)  
Old 6th February 2016
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

This subject presents an opportune moment to again advocate the importance of backups. If the data is important and/or it cannot be easily recreated, back it up, and back it up often.
Reply With Quote
  #8   (View Single Post)  
Old 6th February 2016
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

Paranoid user may consider to add sync(8) command between steps 1 and 2, 4 and 5 described in post #2.

Sync can be redundant if sync is used as parameter for mounted filesystem, as jggimi suggested.
Keep in mind that sync synchronizes all file systems.

Last edited by e1-531g; 6th February 2016 at 03:58 PM. Reason: Added further informations
Reply With Quote
  #9   (View Single Post)  
Old 6th February 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

A sync(8) at the end of each of each command step will flush any pending write cache buffers to the filesystems. But, a filesystem mounted with the sync option will cause each writes to wait until the drive confirms a pending write operation has completed. That's a huge difference ... in process performance and I/O throughput particularly. Synchronous mounts are not used as a standard practice. Applications that need to ensure data integrity, such as database management systems, typically issue fsync(2) at the point where a transaction needs to be committed.

Of course, having data written to the drive does not ensure it is readable. The drive may not be able to read the sectors written. The drive may have confirmed the write while sectors remain in the drive's write cache. The data written may not be a discrete entity, such as adding sectors to the end of a file but then missing the metadata update that notes the extension. The drive may fail later.

RAID systems may also improve data integrity, by providing redundant write operations. They can also reduce integrity and add risk, and should be used with care. And they are never replacements for backup. A redundant write is written to at least two locations. So are redundant errors.
Reply With Quote
Old 7th February 2016
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

Quote:
Originally Posted by jggimi View Post
RAID systems [..]. And they are never replacements for backup. A redundant write is written to at least two locations. So are redundant errors.
It's good to hear that are some sane people on the Earth.
Reply With Quote
Old 11th February 2016
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default

Quote:
Originally Posted by notooth View Post
Hi,

Can anyone let me know if OpenBSD has any solution to prevent file damaged if the system crashed while updating large files?
It is difficult [for me] to interpret what this means.

...prevent file damaged if the system crashed while updating large files?

Part of the sentence (above) is past tense.
Did something already happen regarding "updating large files"?
BTW - What does "updating large files" mean?
And part of the sentence suggests a future tense.
Are you preparing for, or speculating about, a potential "crash"?
Reply With Quote
Old 22nd February 2016
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

I now think that "paranoid user" was not the best phrase.

Anyway inspired by that question and answers:
https://unix.stackexchange.com/quest...-one-partition

I have created a script to update file more safely, than with default filesystem options, but still not sacrifice performance for long period of time.

Code:
/sbin/mount -u -o sync,noatime e2687744d2198a2e.a  / && echo "filesystem in sync" && mount 2>&1 | grep ' / '
#redundancy for updating file, but if you are going to just copy, it is not required
cp /testSync/there_will_be_copied_file1 /testSync/there_will_be_copied_file1.tmp
#copying fresh data, but one should update file insted of copy to it
cp /testSync/file1 /testSync/there_will_be_copied_file1.tmp
mv /testSync/there_will_be_copied_file1 /testSync/there_will_be_copied_file1.old
mv /testSync/there_will_be_copied_file1.tmp /testSync/there_will_be_copied_file1
rm /testSync/there_will_be_copied_file1.old
/sbin/mount -u -o noatime e2687744d2198a2e.a / && echo "filesystem not in sync" && mount 2>&1 | grep ' / '
This gives me output:
Code:
./copy.sh                                                                    
filesystem in sync
/dev/sd1a on / type ffs (local, noatime, synchronous)
filesystem not in sync
/dev/sd1a on / type ffs (local, noatime)
I think that there should also be if statements to check for errors, but it is more like proof of concept.
Of course everybody should do backups, but... maybe this is backup script?

Last edited by e1-531g; 22nd February 2016 at 05:02 PM.
Reply With Quote
Old 22nd February 2016
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default

By defaults OpenBSD mounts are already mounted with the "sync" option. See 14.19 - Why aren't we using async mounts?
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Old 22nd February 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Not qute. sync and async are mount options. The definitive guidance is in mount(8).

async

Metadata and file data are written asynchronously.

sync

Metadata and file data are written synchronously.

Neither async nor sync

File data is written asynchronously, metatdata is written synchronously.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
doing without X .. what files to safely remove ? daemonfowl OpenBSD General 17 22nd August 2012 12:07 AM
Windows solution? dzoni Other OS 3 11th July 2010 07:29 AM
Any Solution for Xorg high CPU usage in Ubuntu 9.04? Mantazz Other BSD and UNIX/UNIX-like 11 14th July 2009 06:10 AM
Which is the best solution from start bsd.mp? aleunix OpenBSD General 18 4th May 2009 06:33 PM
consider OpenVPN production-grade solution? nimnod Off-Topic 1 26th March 2009 12:22 AM


All times are GMT. The time now is 09:46 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick