How to let Pix2Pix with p5.js run?

Question

I would like to try the ml5.js Pix2Pix example for p5.js. If I just copy the code, update the paths, and try to let it run on my local server, it doesn't work.

Same here:

// Copyright (c) 2019 ml5
//
// This software is released under the MIT License.
// https://opensource.org/licenses/MIT

/* ===
ml5 Example
Pix2pix Edges2Pikachu example with p5.js using callback functions
This uses a pre-trained model on Pikachu images
For more models see: https://github.com/ml5js/ml5-data-and-training/tree/master/models/pix2pix
=== */

// The pre-trained Edges2Pikachu model is trained on 256x256 images
// So the input images can only be 256x256 or 512x512, or multiple of 256
const SIZE = 256;
let inputImg, inputCanvas, outputContainer, statusMsg, pix2pix, clearBtn, transferBtn, modelReady = false,
  isTransfering = false;

function setup() {
  // Create a canvas
  inputCanvas = createCanvas(SIZE, SIZE);
  inputCanvas.class('border-box').parent('canvasContainer');

  // Display initial input image
  inputImg = loadImage('https://ml5js.github.io/ml5-examples/javascript/Pix2Pix/Pix2Pix_promise/images/input.png', drawImage);

  // Selcect output div container
  outputContainer = select('#output');
  statusMsg = select('#status');

  // Select 'transfer' button html element
  transferBtn = select('#transferBtn');

  // Select 'clear' button html element
  clearBtn = select('#clearBtn');
  // Attach a mousePressed event to the 'clear' button
  clearBtn.mousePressed(function() {
    clearCanvas();
  });

  // Set stroke to black
  stroke(0);
  pixelDensity(1);

  // Create a pix2pix method with a pre-trained model
  pix2pix = ml5.pix2pix('https://github.com/ml5js/ml5-library/blob/main/examples/p5js/Pix2Pix/Pix2Pix_callback/models/edges2pikachu.pict', modelLoaded);
}

// Draw on the canvas when mouse is pressed
function draw() {
  if (mouseIsPressed) {
    line(mouseX, mouseY, pmouseX, pmouseY);
  }
}

// Whenever mouse is released, transfer the current image if the model is loaded and it's not in the process of another transformation
function mouseReleased() {
  if (modelReady && !isTransfering) {
    transfer()
  }
}

// A function to be called when the models have loaded
function modelLoaded() {
  // Show 'Model Loaded!' message
  statusMsg.html('Model Loaded!');

  // Set modelReady to true
  modelReady = true;

  // Call transfer function after the model is loaded
  transfer();

  // Attach a mousePressed event to the transfer button
  transferBtn.mousePressed(function() {
    transfer();
  });
}

// Draw the input image to the canvas
function drawImage() {
  image(inputImg, 0, 0);
}

// Clear the canvas
function clearCanvas() {
  background(255);
}

function transfer() {
  // Set isTransfering to true
  isTransfering = true;

  // Update status message
  statusMsg.html('Applying Style Transfer...!');

  // Select canvas DOM element
  const canvasElement = select('canvas').elt;

  // Apply pix2pix transformation
  pix2pix.transfer(canvasElement, function(err, result) {
    if (err) {
      console.log(err);
    }
    if (result && result.src) {
      // Set isTransfering back to false
      isTransfering = false;
      // Clear output container
      outputContainer.html('');
      // Create an image based result
      createImg(result.src).class('border-box').parent('output');
      // Show 'Done!' message
      statusMsg.html('Done!');
    }
  });
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script>
<script src="https://unpkg.com/ml5@latest/dist/ml5.min.js" type="text/javascript"></script>

<h1>Pix2Pix Edges2Pichaku Example</h1>
<p>1. Wait until the model is loaded</p>
<p>2. Press your mouse to draw a Pikachu on the left side of the canvas.</p>
<p>3. A colored Pikachu image will automatically appear on the right side of the canvas in ~2 seconds. You could also click the "Transfer" button to generate an new image.</p>
<p>4. You could click the "Clear" button to clear the canvas and draw again.</p>
<p id="status">Loading Model... Please wait...</p>
<div class="flex">
  <div>
    <div id="canvasContainer"></div>
    <div id="btnContainer" class="flex flex-space-between">
      <button id="clearBtn">Clear</button><br />
      <button id="transferBtn" class="btn">Transfer</button>
    </div>
  </div>
  <div id="transferContainer">
  </div>
  <div id="output"></div>
</div>

Here would be also a jsFiddle: https://jsfiddle.net/L6oaydrm/

Has anyone an idea how to let it run? Would be very thankful.

Answer 1

I think I was able to get the 'callback' example to work locally with some tinkering:

1. Download files from the example: https://github.com/ml5js/ml5-library/tree/main/examples/p5js/Pix2Pix/Pix2Pix_callback
2. Adjust the index.html to load ml5.min.js from the unpkg.com URL in your code.
3. Create a new function:

function startTransfer(){
  // Create a pix2pix method with a pre-trained model
  pix2pix = ml5.pix2pix('./models/edges2pikachu.pict', modelLoaded);
}

4. Replace all calls to transfer() except the first one in modelLoaded() with startTransfer().
5. Start a simple local web server; for me: python -m http.server worked.

The example appeared to work. I could draw on the canvas, and the ML model would redraw the Pikachu image factoring in the new lines I added. Note, sometimes the initial transfer is run before the template image (input.png) is loaded, and the result is a garbled yellow / red pixels; clicking 'Transfer' fixes this.

Basically, it always will reload the model into the ml5 library; I don't know of the performance implications of this, but it was redrawing relatively quickly in my browser. The file will be cached in the browser, so that isn't a concern, but I'm not sure of the internals of the ml5.js lib and what ml5.pix2pix(...) does.

---

I've put my revised code (including some other tweaks to the JS) up on https://jsfiddle.net/lecrte/jvohcw8r/16/ ... but it won't work there because the assets aren't available relative to the HTML file, and we can't load the edges2pikachu.pict direct from github.com due to CORS issues.