VESA capabilities are fairly limited, there is a select list of predefined modes in the video BIOS, X may default at lower resolution (800x600).. you can try choosing another by one listed in xrandr output, i.e: xrandr -s 1024x768.
Based on the Xorg.0.log output on the first page, replacing your Monitor section with the following MIGHT help if no other modes are listed in xrandr output.
Code:
Section "Monitor"
Identifier "FLATRON 775FT"
DisplaySize 310 230
VertRefresh 50-160
HorizSync 30-70
Modeline "1024x768" 78.80 1024 1040 1136 1312 768 769 772 800 +hsync +vsync
Modeline "1152x864" 108.00 1152 1216 1344 1600 864 865 868 900 +hsync +vsync
EndSection
As for fluxbox, configuration is done inside text files, they're in
~/.fluxbox/ and the website has some fairly good
documentation.
Good luck.