View Single Post
  #6   (View Single Post)  
Old 3rd August 2019
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 1,027
Default

That's interesting. It's conceivable that for any given safety factor, such as using 8 digits, there might be some narrow special case fraction inputs such that the rounded results are still different. (Just speculation on my part.) As for checking of my script's results, I just used a hand canculator, e.g.,

(5/9)* 2.54 = 1.41111111...

and observed it seemed to round to 4 digits properly in the few cases I checked.
Reply With Quote