Category "precision"

Strange warning when calling sprintf with .* width specifier

For the following code: https://godbolt.org/z/WcGf9hEs3 #include <stdio.h> int main() { char temp_buffer[8]; double val = 25.3; sprint

XGBoost Model performance

I am trying to use XGBoost for classification. I am pretty doubtful on its accuracy. I have applied it with default parameters and the precision is 100%. xg_c

How to control the display precision of a NumPy float64 scalar?

I'm writing a teaching document that uses lots of examples of Python code and includes the resulting numeric output. I'm working from inside IPython and a lot

Project Euler #1 in JavaScript on HackerRank

I'm having some trouble with the test cases for Project Euler #1 on HackerRank and was hoping someone with some JS experience on HackerRank could help out. B

Warp Matrix-Multiply functions - are single-precision multiplicands supported?

In the CUDA Programming guide, v11.7, section B.24.6. Element Types & Matrix Sizes, there's a table of supported type combinations, in which the multiplicat

Working with small probabilities, via logs

Source: Google Code Jam. https://code.google.com/codejam/contest/10224486/dashboard#s=a&a=1 We're asked to calculate Prob(K successes from N trials) where

How do you round a double in Dart to a given degree of precision AFTER the decimal point?

Given a double, I want to round it to a given number of points of precision after the decimal point, similar to PHP's round() function. The closest thing I can

Difference between double- precision data type and numeric data type

What is the difference between double-precision data type and numeric data type in R programming?

Get file modification time to nanosecond precision

I need to get the full nanosecond-precision modified timestamp for each file in a Python 2 program that walks the filesystem tree. I want to do this in Python i

Significant digits with IEEE 754 float

The Wiki Double-precision floating-point format says: This gives 15–17 significant decimal digits precision. If a decimal string with at most 15 signific

Is there a functional difference between "2.00" and "2.00f"?

I ask because I am using the Box2D library, which calls for mostly float arguments. Although I see a lot of example code that uses the 0.00f format, I am not qu

Is there a way to make a complex number in NumPy with the accuracy of a Decimal type?

I'm working in Python with NumPy arrays of complex numbers that extend well past the normal floating point limits of NumPy’s default Complex type (numbers