Gekko: Problem with obtaining the solution (offline, Solver=1, 2, or 3) or in a reasonable time (online, remote=True)

Question

The simple optimization model below (a support vector machine, see https://www.supplychaindataanalytics.com/creating-a-support-vector-machine-using-gekko-in-python/ for further information) is an NLP with T=86 and U=6 (dataset generated for this minimal working example).

import numpy as np
import gekko as op
import itertools as it

a = np.random.rand(86, 6)
b = np.random.randint(0,6, size=(86))

C = range(len(set(b))) #Set of classes
U = range(len(a[0]))  #Set of input features
T = range(len(b)) #Set of the training points

def model (C,U,T,a,b,solve="y"):
    save_b = tuple(b)
    alpha_c=[None for j in C]
    z_c=[None for j in C]
    for j in C:
        for t in T:
            if b[t] == j:
                b[t] = +1
            else:
                b[t] = -1
        print(b)
        m = op.GEKKO(remote=False, name='SupportVectorMachine') 
        alpha = {t: m.Var(lb=0, ub=None) for t in T}
        n_a = {(t,i): a[t][i] for t,i in it.product(T,U)}
        n_b = {t: b[t] for t in T}  
        objs = {0: m.sum([alpha[t] for t in T]) - 0.5*m.sum([alpha[t]*alpha[tt] * n_b[t]*n_b[tt] * m.sum([n_a[(t,i)]*n_a[(tt,i)] for i in U]) for t,tt in it.product(T,T)])}
        cons = {0: {0: ( m.sum([alpha[t]*n_b[t] for t in T]) == 0) for t in T}}
        m.Maximize(objs[0])
        for keys1 in cons:
            for keys2 in cons[keys1]: m.Equation(cons[keys1][keys2])   
        if solve == "y":
            m.options.SOLVER=1
            m.solve(disp=False)
            for keys in alpha: 
                alpha[keys] =  alpha[keys].value[0]
                print(f"alpha[{keys}]", alpha[keys])
        x = [None for i in U]
        for i in U:
            x[i]=sum(alpha[t]*b[t]*n_a[(t,i)] for t in T)
        for t in T:
            if alpha[t]>0: 
                z=b[t] - sum(x[i]*n_a[(t,i)] for i in U)
                break
        b = list(save_b)
        alpha_c[j]=alpha
        z_c[j]=z
    return m,z,alpha

m, z, alpha = model(C,U,T,a,b) #Model and solve the problem

With m.options.SOLVER=1, the code exits with:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\tmpj66p0g5qsupportvectormachine\\options.json'

With m.options.SOLVER=2, the code exits with:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\tmpgat29b25supportvectormachine\\options.json'

With m.options.SOLVER=3, the code exits with:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\tmpgat29b25supportvectormachine\\options.json'

With m = op.GEKKO(remote=True, name='SupportVectorMachine') the code seems to take too much time to run and no output is reported.

I wondered why such a case occurs and how I could troubleshoot the code? Do we need to feed the solver's algorithm with an initial guess every time? (I am using gekko (1.21.5).) Thanks in advance.

ٍEdit: On Google's Colaboratory, it exits with:

Exception: @error: Solution Not Found

Answer 1

Try IMODE=2 (regression mode) where the model is defined only once and automatically applied for each data point. This greatly improves the speed of the model compilation. Below is a minimal example of IMODE=2. There is also a comparison to Scipy Minimize for this same example and a nonlinear regression example.

import numpy as np
from gekko import GEKKO
# load data
xm = np.array([18.3447,79.86538,85.09788,10.5211,44.4556, \
               69.567,8.960,86.197,66.857,16.875, \
               52.2697,93.917,24.35,5.118,25.126, \
               34.037,61.4445,42.704,39.531,29.988])
ym = np.array([5.072,7.1588,7.263,4.255,6.282, \
               6.9118,4.044,7.2595,6.898,4.8744, \
               6.5179,7.3434,5.4316,3.38,5.464, \
               5.90,6.80,6.193,6.070,5.737])
# define model
m = GEKKO()
# one value across all data
a,b,c = m.Array(m.FV,3,value=0)
c.LOWER = -100; c.UPPER = 100
# load data
x = m.Param(value=xm)
ymeas = m.Param(value=ym)
# predicted value
ypred = m.Var()
a.STATUS = 1; b.STATUS = 1; c.STATUS = 1
m.Equation(ypred == a + b/x + c*m.log(x))
m.Minimize(((ypred-ymeas)/ymeas)**2)
m.options.IMODE = 2   # regression mode
m.solve() # remote=False for local solve
print('Final SSE Objective: ' + str(m.options.objfcnval))

There are also a couple things in the code that improve the performance.

1. The m.sum([n_a[(t,i)]*n_a[(tt,i)] for i in U]) in the objective function can be replaced with a regular summation as sum([n_a[(t,i)]*n_a[(tt,i)] for i in U]). When all of the arguments are floating point numbers, this allows evaluation of the value instead of the Gekko symbolic form of the summation.

2. Can the cons be defined once (dimension T) instead of the same constraint repeated (dimension T*T)?

        cons = {0: m.sum([alpha[t]*n_b[t] for t in T]) == 0}
        m.Maximize(objs[0])
        for keys1 in cons:
            m.Equation(cons[keys1])

Thanks for posting the SVM code. If it can be transitioned into IMODE=2 then the solution efficiency can be greatly improved. If it is not possible (stay with IMODE=3 default) then hopefully the above suggestions help.

We're working on a new module in Gekko to import sklearn (sklearn example), tensorflow, gpflow and other models into Gekko. The potential is to use those specially crafted regression packages and export models to the more general optimization gekko package. We've found that a general gradient-based solver may be doing too much work in the fitting by providing automatic differentiation of constraints and objective with sparse 1st and 2nd derivatives when only 1st derivatives are required. The specifically designed optimizers for each regression method may be more efficient, especially with large data sets that train on GPUs. More documentation and results are expected in the next release of Gekko v1.1+ around Sept 2022.