DaemonForums  

Go Back   DaemonForums > Miscellaneous > Programming

Programming C, bash, Python, Perl, PHP, Java, you name it.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 21st November 2015
alex_b83 alex_b83 is offline
Port Guard
 
Join Date: Oct 2015
Posts: 11
Default Test small C program in BSD OS

Hello.

I've written small program in C language.
This program copies specified part of source file and saves it to destination file.

I have GNU/Linux OS installed on my computer.
Program works fine in Linux.
It also works with big files (size >= 2 GiB) in 64-bit Linux.

Help me please to test if program works in BSD OS (FreeBSD, NetBSD, OpenBSD),
if you have this OS installed on your computer.

Instruction on program usage and how to build it from source:
https://gitlab.com/open_source/partcp

Download ZIP archive with source code:
https://gitlab.com/open_source/partc...zip?ref=master

View source code:
https://gitlab.com/open_source/partc...aster/partcp.c

Program usage example:

I want to copy 1234 bytes from file "source.bin" starting at offset 5678.
Result must be saved to file "destination.bin".
I will use this command:

partcp source.bin 5678 1234 destination.bin


Thank you.
Reply With Quote
  #2   (View Single Post)  
Old 21st November 2015
LeFrettchen's Avatar
LeFrettchen LeFrettchen is offline
Marveled user
 
Join Date: Aug 2012
Location: Bronvaux, France
Posts: 231
Default

Tested on OpenBSD 5.7, with small files.

Works fine.
__________________
The secret of wisdom is not wisdom itself, it's the road that leads us there.
Reply With Quote
  #3   (View Single Post)  
Old 22nd November 2015
bashrules bashrules is offline
Fdisk Soldier
 
Join Date: Mar 2010
Location: Here
Posts: 48
Default

Hello Alex.

From your readme file:

Code:
dd utility copies part of file slowly when bs < 512.
partcp performs this task faster by using bigger memory blocks.
I assume small block sizes are bad because the hd head is then always moving between source file and target file back and forth.

This problem is then solved by setting in dd bs to a high value. Then dd and partcp should perform equally. What do you think?

A suggestion. To complete your project,
  • add a Makefile or configure script
  • add a testcase (say, provide a sourcefile and target file and have a make target that is calling partcp to generate the target file and then "diff" the generated target file with the provided target file).
Reply With Quote
  #4   (View Single Post)  
Old 2nd December 2015
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Old man from scene 24
 
Join Date: Apr 2008
Location: Dutchman living in the UK
Posts: 2,188
Default

This is indeed exactly the same as dd bs=4M ...
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #5   (View Single Post)  
Old 2nd December 2015
alex_b83 alex_b83 is offline
Port Guard
 
Join Date: Oct 2015
Posts: 11
Default

Quote:
Originally Posted by bashrules View Post
This problem is then solved by setting in dd bs to a high value. Then dd and partcp should perform equally. What do you think?
Quote:
Originally Posted by Carpetsmoker View Post
This is indeed exactly the same as dd bs=4M ...
Is the size of dd output file always a multiple of bs?
Reply With Quote
  #6   (View Single Post)  
Old 2nd December 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 5,707
Default

Each of the Project's utilities may be slightly different, as each has a different provenance.

OpenBSD's dd(1) is based on NetBSD's, as OpenBSD was itself a fork of NetBSD, a little over 20 years ago. It has been continuously maintained, with the last revisions in support of OpenBSD's new pledge(2) security framework.

Here's a link to the source code:

http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/bin/dd/
Reply With Quote
  #7   (View Single Post)  
Old 3rd December 2015
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Old man from scene 24
 
Join Date: Apr 2008
Location: Dutchman living in the UK
Posts: 2,188
Default

Quote:
Originally Posted by alex_b83 View Post
Is the size of dd output file always a multiple of bs?
If you add conv=sync:

Quote:
Pad every input block to the size of the ibs= buffer, appending null bytes. (If either block or unblock is also specified, append <space> characters, rather than null bytes.)
This is in POSIX so it can be reasonably expected to work on most platforms.
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #8   (View Single Post)  
Old 3rd December 2015
alex_b83 alex_b83 is offline
Port Guard
 
Join Date: Oct 2015
Posts: 11
Default

Quote:
Originally Posted by Carpetsmoker View Post
This is indeed exactly the same as dd bs=4M ...
Show me please full command line with given source_file, slice_offset, slice_length, destination_file as input values.
Reply With Quote
  #9   (View Single Post)  
Old 4th December 2016
alex_b83 alex_b83 is offline
Port Guard
 
Join Date: Oct 2015
Posts: 11
Question

Quote:
Originally Posted by bashrules View Post
This problem is then solved by setting in dd bs to a high value. Then dd and partcp should perform equally. What do you think?
As you know, this command will work slowly because of low ibs value:
Code:
dd ibs=1 if=source_file of=destination_file skip=slice_offset count=slice_length
I don't understand how to correctly replace the following command
Code:
partcp source_file slice_offset slice_length destination_file
with dd, considering that bs>1:
Code:
dd bs=high_value if=source_file of=destination_file skip=? count=?
According to dd manual page, skip and count parameters are multiples of (i)bs.
So we can't simply set them to slice_offset and slice_length, because in such case actual values will be:
skip = slice_offset * high_value and count = slice_length * high_value, which is wrong.

How do I correctly apply slice_offset and slice_length values here?

Quote:
Originally Posted by Carpetsmoker View Post
This is indeed exactly the same as dd bs=4M ...
Quote:
Originally Posted by alex_b83 View Post
Show me please full command line with given source_file, slice_offset, slice_length, destination_file as input values.
My question is still open.

Can you show me full command line for "dd bs=4M ..." to exactly replace the following command?
Code:
partcp source_file slice_offset slice_length destination_file
What should be written instead of ellipsis you wrote?
Reply With Quote
Old 2nd January 2017
alex_b83 alex_b83 is offline
Port Guard
 
Join Date: Oct 2015
Posts: 11
Default

GNU coreutils' dd accepts the count_bytes and skip_bytes input flag to more easily allow processing portions of a file.
This feature is relatively new (added on Feb 2012).

It allows to achieve fast (bs=4M) copy of arbitrary portion of a binary file using single commnad:
Code:
dd if=source_file of=destination_file bs=4M iflag=count_bytes,skip_bytes skip=slice_offset count=slice_length
You may also need to use iflag=fullblock: one, two.

----------------------------------------

And what about BSD OSes? count_bytes and skip_bytes flags are not implemented in BSD variants of dd.

In FreeBSD and NetBSD we can pipe tail into head.
In OpenBSD we can use multiple calls to dd (see "Using a dd without the skip_bytes and count_bytes options").

It appears that there is no easy-to-use and fast byte-grabber under Unix.
That's why I've written my program.
Reply With Quote
Old 2nd January 2017
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 5,707
Default

Quote:
Originally Posted by alex_b83 View Post
GNU coreutils' dd accepts the count_bytes and skip_bytes input flag to more easily allow processing portions of a file....
And what about BSD OSes?
You can install gnu coreutils on OpenBSD, it is available as a package. The gnu version of dd is installed as /usr/local/bin/gdd to differentiate it from the BSD version of dd.
Reply With Quote
Old 21st January 2017
alex_b83 alex_b83 is offline
Port Guard
 
Join Date: Oct 2015
Posts: 11
Lightbulb

jggimi, thank you for your answer!

Updated project description on GitLab.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Small dmesg annoyance v10 ... jb_daefo FreeBSD Ports and Packages 0 3rd January 2015 04:16 PM
Small MPLS Test Network Built With OpenBSD J65nko News 1 8th September 2013 07:40 PM
firewalling with a small soekris appliance wesley OpenBSD Security 2 6th June 2011 05:29 PM
small guide on mutt & fdm qmemo Guides 0 17th May 2011 10:54 PM


All times are GMT. The time now is 08:57 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick