'Drastically results variation when change condition in _gradient_descent tSNE scikit-Learn

I am working with some noisy data to classify the spectrum of light curves using the tSNE instance in scikit-Learn. The problem comes when I try to understand how the _gradient_descent function in _t_sne.py works. More specifically, some implementations (i. e. here) change the inc variable from update*grad<0 (scikit use this) to upgrade*grad>=0. When I use the _gradient_descent variant with >=, I get better results than in the < 0 variant:

Left:upgrade*grad>=0 Rigth:update*grad<0

Can anyone please tell me why scikit-Learn uses the < 0 variant? What does it mean? Can I use the >= 0 variant without broke tSNE? Thanks

The _gradient_descent function is:

def _gradient_descent(objective, p0, it, n_iter, n_iter_check=1, n_iter_without_progress=300, momentum=0.8, learning_rate=200.0, min_gain=0.01, min_grad_norm=1e-7, verbose=0, args=None, kwargs=None):
if args is None:
    args = []
if kwargs is None:
    kwargs = {}

p = p0.copy().ravel()
update = np.zeros_like(p)
gains = np.ones_like(p)
error = np.finfo(float).max
best_error = np.finfo(float).max
best_iter = i = it

tic = time()
for i in range(it, n_iter):
    check_convergence = (i + 1) % n_iter_check == 0
    # only compute the error when needed
    kwargs["compute_error"] = check_convergence or i == n_iter - 1

    error, grad = objective(p, *args, **kwargs)
    grad_norm = linalg.norm(grad)

    inc = update * grad < 0.0  # THIS LINE IS THE PROBLEM
    dec = np.invert(inc)
    gains[inc] += 0.2
    gains[dec] *= 0.8
    np.clip(gains, min_gain, np.inf, out=gains)
    grad *= gains
    update = momentum * update - learning_rate * grad
    p += update

    if check_convergence:
        toc = time()
        duration = toc - tic
        tic = toc

        if verbose >= 2:
            print(
                "[t-SNE] Iteration %d: error = %.7f,"
                " gradient norm = %.7f"
                " (%s iterations in %0.3fs)"
                % (i + 1, error, grad_norm, n_iter_check, duration)
            )

        if error < best_error:
            best_error = error
            best_iter = i
        elif i - best_iter > n_iter_without_progress:
            if verbose >= 2:
                print(
                    "[t-SNE] Iteration %d: did not make any progress "
                    "during the last %d episodes. Finished."
                    % (i + 1, n_iter_without_progress)
                )
            break
        if grad_norm <= min_grad_norm:
            if verbose >= 2:
                print(
                    "[t-SNE] Iteration %d: gradient norm %f. Finished."
                    % (i + 1, grad_norm)
                )
            break

return p, error, i


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source