DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD General

FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 23rd July 2008
PatrickBaer PatrickBaer is offline
Fdisk Soldier
 
Join Date: May 2008
Posts: 81
Default Backup getting too big, how to estimate the size after thinnig out?

Good morning!

We have a backup script here, creating a full backup every month and the next 30 days it deletes the old incremental, if there is any, creates a new directory with hard links to each corresponding etc. I haven't gotten behind all this yet.

But, we definitely need to clean up as we are running out of storage space! So before I go berserk and remove files, I want to find out, which files I will really delete by removing a directory.

In other words and more general:

I have 3 directories 1 2 3

Directory 1 holds files A B C D E

Directory 2 holds hardlinks A B C E

Directory 3 holds hardlinks A C E and files F G

How could I find out, how much storage I can free by deleting directory 1?

TIA

Patrick
Reply With Quote
  #2   (View Single Post)  
Old 23rd July 2008
18Googol2's Avatar
18Googol2 18Googol2 is offline
Real Name: whoami
Spam Deminer
 
Join Date: Apr 2008
Location: pwd
Posts: 283
Default

%du -h /path/to/dir/1 | tail -n 1
Reply With Quote
  #3   (View Single Post)  
Old 23rd July 2008
DutchDaemon's Avatar
DutchDaemon DutchDaemon is offline
Real Name: Ben
Spam Refugee
 
Join Date: Jul 2008
Location: Rotterdam, The Netherlands
Posts: 336
Default

Or just du -sh /dir
Reply With Quote
  #4   (View Single Post)  
Old 23rd July 2008
PatrickBaer PatrickBaer is offline
Fdisk Soldier
 
Join Date: May 2008
Posts: 81
Default

Not really, watch the HARD links!
Reply With Quote
  #5   (View Single Post)  
Old 23rd July 2008
DutchDaemon's Avatar
DutchDaemon DutchDaemon is offline
Real Name: Ben
Spam Refugee
 
Join Date: Jul 2008
Location: Rotterdam, The Netherlands
Posts: 336
Default

The du utility counts the storage used by symbolic links and not the
files they reference unless the -H or -L option is specified. If either
the -H or -L options are specified, storage used by any symbolic links
which are followed is not counted or displayed.

Files having multiple hard links are counted (and displayed) a single
time per du execution.
Reply With Quote
  #6   (View Single Post)  
Old 23rd July 2008
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Tcpdump Spy
 
Join Date: Apr 2008
Location: Netherlands
Posts: 2,243
Default

For an example of DutchDaemon's point:

Code:
[~]% du -h /rescue/
3.6M    /rescue/
[~]% ls -s /rescue/
total 445292
3680 [*               3680 date*            3680 fsck*            3680 ipf*             3680 mount_msdosfs*   3680 rdump*           3680 spppcontrol*
3680 atacontrol*      3680 dd*              3680 fsck_4.2bsd*     3680 kenv*            3680 mount_nfs*       3680 realpath*        3680 startslip*
3680 atmconfig*       3680 devfs*           3680 fsck_ffs*        3680 kill*            3680 mount_ntfs*      3680 reboot*          3680 stty*
3680 badsect*         3680 df*              3680 fsck_msdosfs*    3680 kldconfig*       3680 mount_nullfs*    3680 red*             3680 swapon*
3680 bsdlabel*        3680 dhclient*        3680 fsck_ufs*        3680 kldload*         3680 mount_udf*       3680 rescue*          3680 sync*
3680 bunzip2*           10 dhclient-script* 3680 fsdb*            3680 kldstat*         3680 mount_unionfs*   3680 restore*         3680 sysctl*
3680 bzcat*           3680 disklabel*       3680 fsirand*         3680 kldunload*       3680 mv*              3680 rm*              3680 tar*
3680 bzip2*           3680 dmesg*           3680 gbde*            3680 ldconfig*        3680 newfs*           3680 rmdir*           3680 tcsh*
3680 camcontrol*      3680 dump*            3680 getfacl*         3680 link*            3680 newfs_msdos*     3680 route*           3680 test*
3680 cat*             3680 dumpfs*          3680 groups*          3680 ln*                 2 nextboot*        3680 routed*          3680 tunefs*
3680 ccdconfig*       3680 dumpon*          3680 gunzip*          3680 ls*              3680 nos-tun*         3680 rrestore*        3680 umount*
3680 chflags*         3680 echo*            3680 gzcat*           3680 md5*             3680 pax*             3680 rtquery*         3680 unlink*
3680 chio*            3680 ed*              3680 gzip*            3680 mdconfig*        3680 ping*            3680 rtsol*           3680 vi*
3680 chmod*           3680 ex*              3680 halt*            3680 mdmfs*           3680 ping6*           3680 savecore*        3680 whoami*
3680 chroot*          3680 expr*            3680 hostname*        3680 mkdir*           3680 ps*              3680 sconfig*         3680 zcat*
3680 clri*            3680 fastboot*        3680 id*              3680 mknod*           3680 pwd*             3680 setfacl*
3680 cp*              3680 fasthalt*        3680 ifconfig*        3680 mount*           3680 rcorder*         3680 sh*
3680 csh*             3680 fdisk*           3680 init*            3680 mount_cd9660*    3680 rcp*             3680 slattach*
Most files (except nextboot and dhclient-script) in /rescue/ are a hard link to /rescue/rescue.
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #7   (View Single Post)  
Old 23rd July 2008
PatrickBaer PatrickBaer is offline
Fdisk Soldier
 
Join Date: May 2008
Posts: 81
Default

Still you don't understand me

I have made an example:

Code:
[pba@voodoo /tmp/test]$ ls -lia *
2008-01-01:
total 4
49358 drwxr-xr-x  2 pba  wheel  512 Jul 23 21:25 .
49357 drwxr-xr-x  5 pba  wheel  512 Jul 23 21:26 ..
49359 -rw-r--r--  3 pba  wheel    0 Jul 23 21:24 1
49360 -rw-r--r--  3 pba  wheel    0 Jul 23 21:24 2
49362 -rw-r--r--  1 pba  wheel    0 Jul 23 21:24 3

2008-01-02:
total 4
49363 drwxr-xr-x  2 pba  wheel  512 Jul 23 21:25 .
49357 drwxr-xr-x  5 pba  wheel  512 Jul 23 21:26 ..
49359 -rw-r--r--  3 pba  wheel    0 Jul 23 21:24 1
49360 -rw-r--r--  3 pba  wheel    0 Jul 23 21:24 2

2008-01-03:
total 4
49364 drwxr-xr-x  2 pba  wheel  512 Jul 23 21:26 .
49357 drwxr-xr-x  5 pba  wheel  512 Jul 23 21:26 ..
49359 -rw-r--r--  3 pba  wheel    0 Jul 23 21:24 1
49360 -rw-r--r--  3 pba  wheel    0 Jul 23 21:26 2
49366 -rw-r--r--  1 pba  wheel    0 Jul 23 21:26 4
So this gives me a total of five files stored: 1 2 3 4 5

If I delete the first directory, it will only free the space occupied by file 3.
If I delete the second directory also, it will not change anything, as the files 1 and 2 are also represented by the hard links in the third directory.

So the two tricky questions:

1. How do I find all files deleted before the last backup (in other words: not present in the last directory) and delete the inode, not just one hard link? Which would be file 3 in our example here. This should be easy by just deleting the directories!

2. How do I find out about the total amount of space consumed by those orphaned files? And this one is the tricky part
Reply With Quote
  #8   (View Single Post)  
Old 23rd July 2008
ephemera's Avatar
ephemera ephemera is offline
Knuth's homeboy
 
Join Date: Apr 2008
Posts: 537
Default

> 2. How do I find out about the total amount of space consumed by those orphaned files? And this one is the tricky part

what do you mean by orphaned files? files that are present in or more backup directories except the last one?

if you give a more precise desription of the problem then maybe i could write a quick script for this.
Reply With Quote
  #9   (View Single Post)  
Old 23rd July 2008
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

Or you could come with an aging script that deletes the oldest directory after a certain amount of time. For example, if dir1 was created more than 30 days ago, delete it to make room for dir31. Depends how your backup script actually works though.

I don't see the point of deleting all occurances of file3 if it no longer exists in the last directory. Isn't that the point of backups? So that you can go back in time to before a file was deleted so that you can recover it?
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
Old 24th July 2008
PatrickBaer PatrickBaer is offline
Fdisk Soldier
 
Join Date: May 2008
Posts: 81
Default

Quote:
Originally Posted by phoenix View Post
I don't see the point of deleting all occurances of file3 if it no longer exists in the last directory. Isn't that the point of backups? So that you can go back in time to before a file was deleted so that you can recover it?
Imagine running out of space on a 14T raid

And no, the point of (our) backups is to restore data in case of a hardware or software failure. Faulty users should not be included
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
backup freeBSD 7.0 using Backup Exec ccc FreeBSD General 2 25th April 2009 09:23 PM
CLI font size idefix FreeBSD General 3 7th February 2009 11:19 AM
Negative partition size? giga FreeBSD General 1 2nd January 2009 09:02 PM
Change the Size of a Slice kienjakenobi FreeBSD General 9 27th June 2008 03:01 PM
apache fd size problem oxy FreeBSD Ports and Packages 0 10th May 2008 10:49 AM


All times are GMT. The time now is 12:07 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick