DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

 
 
Thread Tools Display Modes
Prev Previous Post   Next Post Next
  #1   (View Single Post)  
Old 12th December 2011
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,135
Thanked 182 Times in 149 Posts
Default Tools to assist in database import

In a database import of over 200,000 records, I noticed that a length of 30 chars was not long enough to hold all email addresses.

With assistance of a short awk script I could quickly determine the optimal length:

Code:
$ awk -f field_length.awk import_file.csv

111628 065031051@rrrrrr.com
322626 0594500515@wwwwww.co.za
[snip]
361682 huilin19
78@wrwrwrwr.c
;om
xxxxxxxxx@hotmail.com
146
This unbelievable longest mail address probably was caused by either an UTF-16 issue or data corruption.

The awk script:
Code:
BEGIN {
SIZE = 0 ;
FS = "," ;
}

# field 1 is the primary key
# field 2 is the email address
{
if (length($2) > SIZE ) {
   print $1, $2 ;
   SIZE = length($2) ;
   }
}

END {
print $2 ;
print SIZE ;
}
BTW sed(1) confirmed this:
Code:
$ sed -ne '/361682/p' masterlist_comma.csv 
361682 huilin19
78@wrwrwrwr.c
;om
After editing out the offending line the result was:
Code:
 $ awk -f field_length.awk import_file2.csv
[snip]
58
So a field width of 60 seemed safe
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 12th December 2011 at 06:17 AM.
Reply With Quote
 

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
MySQL database import not helpful J65nko General software and network 2 10th December 2011 02:45 AM
When is a database not so relational? J65nko News 1 23rd May 2011 05:38 AM
zfs import terryd FreeBSD General 1 23rd November 2008 09:26 PM
import physical freeBSD into VMWARE (ESX) server as a vServer ccc FreeBSD General 6 3rd October 2008 07:04 AM
PHP database interfaces TerryP Programming 6 11th September 2008 01:03 PM


All times are GMT. The time now is 05:56 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick