System and method for rounding reciprocal square root results of input floating point numbers

A system to generate an accurately rounded result of a reciprocal square root calculation performed on an input floating point number, the system comprising:

• a reciprocal square root calculation module configured to generate a calculated result of the reciprocal square root calculation in a first floating point precision;
• a fixed function logic unit configured to determine whether an infinitely precise result of a reciprocal square root operation performed on an input floating point number is greater than, less than, or equal to a particular number in the first floating point precision, the fixed function logic unit comprising:
• one or more fused multiply add hardware logic components configured to:
• (a) calculate a square of the particular number in a second floating point precision that is less than the first floating point precision;
• (b) calculate a rounding error in the calculated square;
• (c) calculate a first delta value in the first floating point precision by calculating the square multiplied by the input floating point number less one;
• and
• (d) calculate a second delta value in the first floating point precision by calculating the rounding error multiplied by the input floating point number plus the first delta value;
• and
• an output hardware logic module configured to output an indication of whether the infinitely precise result of the reciprocal square root operation is greater than, less than or equal to the particular number based on a sign of the second delta value;
• and
• rounding logic configured to round the calculated result to the second floating point precision according to a particular rounding method based on the output indication.