Author: Raja Qasim Khan

In the article “Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers”, many existing works, themes and applications of computer vision, and basic methods of representing computer vision.

To now directly address the questions asked of me, computer vision indeed differs gravely from human vision. While the goal of computer vision remains the same as that of human vision, id est to represent physical (extended) objects in a manner where no significant detail is lost, and so to identify and then perform computations upon the representations of this data (if so it may be needed).

Humans are unfathomably complex beings, with there being over 100 million rods (cells for low-light conditions and peripheral vision, source: Cleveland Clinic), and several million cones (cells for detail and color, source: Cleveland Clinic), many machines even by todays standards can never ever ever come close to us biological entities. Furthermore, operating at an average of 20 Watts (source: National Library of Health), our brains are incredibly efficient at managing and responding to input from the incredible complexity of our eyes, and every other sensory system and square inch of skin.

Now that I am done marveling at ourselves, I return to humiliate the computers. Computers are inefficient, slow, blocky, prone to faults and can really only function on binary numbers and logic (though in more recent years other number and logical systems are being explored). The challenge is both in the sensors scanning the environment and relaying this analog data to the computer. Next the challenge is for that analog data to be converted into a digital format (fundamentally 1s and 0s), and then for that data to be processed by a program in an efficient manner. Typically, videos from this sensory data are stored as “a stream of rectangular pixel buffers”, and according to the paper, this doesn’t really tell us much about what the computer is really being fed through the system.

The paper moves to mention several different schemes and standards that computer vision encoding may be analogous to, for representing real world data, and underscores how there is no unified convention when it comes to these schemes. Certain techniques that a basic algorithm may use to discern motion from stillness includes comparing two adjacent frames in a video to see what pixel values changes, as well as background subtraction. I now extend upon this independently, in that it is probably wiser to first subtract the background before measuring any sort of pixel value changes or points of reference, as we don’t want background pixel noise to impact accuracy.

What I really found interesting was how we may be able to implement basic interactions on these two – albeit simple – methods alone. Once the silhouette of a person has been detected, their motion or boundary can be used as a collider for free-falling objects, for example. Alternatively, we may even be able to recolor a person and their environment in grayscale, or B&W for intriguing stylistic effects. Perhaps it is so that it is only I who yearns for aged B&W technology. There is something I find oddly simple yet calming in such technology.

Alas, I have extended upon the core of that in the reading which held significance to me. Though I would like to mention, the more we try to personify computers, id est implementing traditionally biological processes to them, the more I marvel at our own biological complexity!

Midterm Idea:

I was inspired by Portal 2 by Valve in this endeavour, with my own twist on the gameplay and concept.

While I haven’t settled on a fitting name thus far for the project, the concept entails a “multi-faceted” reality, with a ray-gun that fires a bullet. When this bullet is incident on specific “panels” in the game, it may reflect off the surface of the panel. The objective is to reflect the bullet such that it reaches a designated mechanism to progress to the next level or finish the game.

The Twist:

This gets interesting in such that there are two realities that this special ray-gun, bullet and panels engage with. That is, the ray-gun may fire bullets in either one of Dimension ‘A’ or Dimension ‘B’. If the panel is of the same ‘dimension’ as the bullet, it will hit the panel and shatter it (this may unlock new areas or cause a loss if the wrong panel is destroyed). Alternatively, if the panel and the bullet differ in their ‘dimension’, then the bullet reflects off the panel and this can then be further reflected off other panel as the concept graphics below show:

 

The game will feature inspiration from very simple brutalist architecture, with 1K PBR Textures because I like a degree of realism in my games. Maps will generally be dimly lit, and there will be cool snippets of lore that players can find as they explore different parts of the map or progress though the game.

User Interaction:

Other than the aforementioned ray-gun, there will be WASD keys for forward-backward and left-right movement. Additionally, to look around them, users can drag their mouse towards the sides of the screen to pan their camera. Users can walk to all accessible parts of the map, which will be made sufficiently obvious to avoid confusion, as I am intending on using a darker theme for this game.

Users may use E to fire a bullet, and Q to switch the dimension setting of the ray-gun, id est the dimension of the bullet it fires.

Progression:

There are a total of two levels I plan to implement, and users are limited to three shots from their ray-gun before they have to restart the ENTIRE game because I want to convey a feeling of greater loss, not just that they spawn back at their current level.

Development & Implementation:

I have thus far coded in a very simple game engine to handle all objects. I am going to implement physics and collisions soon, along with more textures and the map.

There are many classes in this, but the two I shall focus on here are ‘BasePart’, which handles a simple 3D box or part of the map in the game, and ‘Workspace’, which is a class that contains all BasePart and other extended instances in the game, as well as contains the camera that displays the map to the user.

Major Concerns and What I’m Dreading:

This projects may seem overly ambitious, but the initial game engine setup has come through nicely. I am not looking forward to collision handling since that always is a nightmare.

Code and Credits:

Find here a link to my code: https://editor.p5js.org/rk5260/sketches/b6B9BVyoE

All code is written by myself (exception below), with many elements I first learnt how to use from the p5.js documentation, after which I implemented them myself.

Exception: The code for the PBR shaders was composed and modified from multiple existing shaders on github and p5.js documentation itself.

All PBR textures sources are credited in a credits.txt file in the PBR Texture’s directory. (Thus far its only GravelConcrete).

 

Concept:

I sought a degree of inspiration from Camillie Utterback’s Text Rain. One key difference, while Utterback made text fall under ‘gravity’ and react to people’s silhouette, I decided to map text onto continuous mathematical functions (see example below)
*The image above depicts a sine curve with the text ‘this is a simple sin function’.

Core Functionality and Features

The program contains a class appropriately called ‘fun’ (abbreviation for ‘function’) that contains attributes and methods, including a ‘display’ method to draw the curve onto the canvas. ‘fun’ also contains methods for changing the type of function – example from a ‘sin’ to a ‘log’ – and a method for fetching the function’s value from the parameter ‘t’ which in this case, spans the width of the canvas.

The ‘fun’ class once instantiated into an object contains data that alters how the drawn function appears. For example, the ‘sin’ function has the following ‘fun.function_data’:

let SineFunction = new fun("sin");

SineFunction.function_data[0] = 100;    // AMPLITUDE 'a'
SineFunction.function_data[1] = 0.01;   // Frequency 'r'
SineFunction.function_data[2] = 10;     // X Translation 'x'
SineFunction.function_data[3] = 200;    // Y Translation 'y'

/*
SineFunction.getF(t) {
  return a * Math.sin( r * t - x ) + y
}
*/

There are 7 basic types of mathematical functions that ‘fun’ can represent, namely sine (sin), cosine (cos), tangent (tan), hyperbolic sine (sinh), hyperbolic cosine (cosh), hyperbolic tangent (tanh), logarithmic (log).

Additionally, for more advanced generation, there is an additional class ‘algebricF’, that is used in a very similar way to ‘fun’, except that it can store up to two ‘fun’ or ‘algebricF’ classes in it, and performs a variety of specified algebraic operations on the class. Please do not mind the spelling mistake in the name of the class. The operations include addition, subtraction, multiplication, and division.

For interesting visuals, each drawn function’s text is assigned a color. The color is blue near the bottom of the screen, red near the top, and then this is blended with linear interpolation based on its ‘y’ position divided by the canvas ‘height’. A second linearly interpolated color is then chosen for the lateral span, from green at the left and purple (updated 20:20 GST, 16 FEB) at the right. The result of this interpolation is then linearly interpolated with the color of the vertical component, to grant a final color. The vertical color component gets priority because of how the third interpolation is carried out.

Demo

There are 3 demonstrations built into the code itself (courtesy of myself). They can be alternated between by pressing the keys ‘1’, ‘2’, ‘3’, or ‘0’ to disable all demonstrations. Click the screen to toggle whether the drawing moves or is paused.

Settings & Configurable Components

Certain settings and components may be configured in the Configuration section at the absolute top of the sketch. It is recommended that you read the comment or description given to each setting before playing around. It is also highly recommended to NOT alter any code beyond this unless you know what you are doing.

To create your own equations, declare an empty variable just before the setup function, and instantiate the correct ‘fun’ or ‘algebricF’ object inside of the setup function itself. Then, call its display method inside of the draw function. You may need to play around with the various components that constitute towards the function, especially amplitude and the y-translation.

Known Bugs

While there is a toggleable variable called ‘ADJUST_SCREEN’ in the Configuration section – which strives to adjust the screen to capture off-screen parts of the graph – it does not work correctly for all cases, thus it is recommended to use functions that stay within the screen if it is so that you actually want to see stuff.

Code & Additional Credits

As no external media or assistance (exception below) was used, I (Raja Qasim Khan) am the sole author and developer of the project’s code.

(Exception): https://p5js.org/reference/ was used for parts such as fetching date and time (for the canvas capture functionality) and the syntax of certain keywords in the language. No additional sources were employed.

Notes from the developer: I began by starting with the absolute basics of the project in the Processing 3 IDE for Python. Once the basics were done with, I re-wrote the code in JavaScript in p5.js and finished most of the remaining features there.

Please find the code below.

// CONFIGURATION
let TEXT_SIZE = 14; // size of the displayed text
let POINT_INCREMENT = 7; // the increment of character positioning in the display

// The gradient of colors in the Y axis
let MIN_COLOR_Y;
let MAX_COLOR_Y;

// The gradient of colors in the X axis
let MIN_COLOR_X;
let MAX_COLOR_X;

let MOVE_RESISTANCE = 1.2; // The resistance coefficient in moving the graph across the screen. -ve for backwards motion.

let ADJUST_SCREEN = false; // If the screen should translate to include outside points.

let PREBUILT_DISPLAY_NUMBER = 2; /*

Prebuilt by me for demo purposes:

0: don't display any demo.
1 : single 'cos' function.
2: multiple function demo.
3: algebric function demo.
4 show credit information.

*/

let OVERLAY_ENABLED = true; // whether the information overlay is enabled or not.

let BACKGROUND_COLOR = [255, 255, 255, 255]; // the background color RGBA

// GLOBALS
let current_offset = 0;
let isMoving = false;

// MAIN CODE

let F1;
let F2;

let F3;
let F4;

let FUN1;
let FUN2;
let FUN3;

let AF12;
let AF;

let CR;

function setup() {
  frameRate(30);
  createCanvas(800, 400);
  
  background(color(BACKGROUND_COLOR[0], BACKGROUND_COLOR[1], BACKGROUND_COLOR[2], BACKGROUND_COLOR[3]));
  
  F1 = new fun("cos");
  
  F1.function_data[0] = 100;
  F1.function_data[1] = 0.02;
  
  F1.function_data[2] = 0;
  F1.function_data[3] = 200;
  
  F2 = new fun("cos");
  
  F2.function_data[0] = -100;
  F2.function_data[1] = 0.02;
  
  F2.function_data[2] = 0;
  F2.function_data[3] = 200;
  
  F3 = new fun("tan");
  
  F3.function_data[0] = 20;
  F3.function_data[1] = 0.02;
  
  F3.function_data[2] = 0;
  F3.function_data[3] = 370;
  
  F4 = new fun("tan");
  
  F4.function_data[0] = 20;
  F4.function_data[1] = 0.02;
  
  F4.function_data[2] = 0;
  F4.function_data[3] = 30;
  
  FUN1 = new fun("sin");
  FUN1.function_data[0] = 100;
  FUN1.function_data[1] = 0.02;
  
  FUN1.function_data[2] = 0;
  FUN1.function_data[3] = 200/3;
  
  FUN2 = new fun("sin");
  FUN2.function_data[0] = 50;
  FUN2.function_data[1] = 0.04;
  
  FUN2.function_data[2] = 0;
  FUN2.function_data[3] = 200/3;
  
  FUN3 = new fun("cos");
  FUN3.function_data[0] = 100/3;
  FUN3.function_data[1] = 0.06;
  
  FUN3.function_data[2] = 0;
  FUN3.function_data[3] = 200/3;
  
  AF12 = new fun("algebricF");
  AF12.eq.fun1 = FUN1;
  AF12.eq.fun2 = FUN2;
  
  AF = new fun("algebricF");
  AF.eq.fun1 = AF12;
  AF.eq.fun2 = FUN3;
  
  CR = new fun("sin");
  
  CR.function_data[0] = 10;
  CR.function_data[1] = 0.005;
  
  CR.function_data[2] = 0;
  CR.function_data[3] = 200;
  
  MIN_COLOR_Y = color(0, 0, 255);
  MAX_COLOR_Y = color(255, 0, 0);
  
  MIN_COLOR_X = color(0, 255, 0);
  MAX_COLOR_X = color(255, 0, 255)
}

class algebricF {
  constructor() {
    /*
    (None) -> None
    
    A custom computation that MUST be an algebric combination of two 'fun's.
    
    can also be made of two algebricF's, or one algebricF.
    
    Operators: "+", "-", "/", "*"    
    */
    this.operator = "+";
    
    this.fun1 = "null";
    this.fun2 = "null";
  }
  
  getF(t) {
    if (this.operator == "+") {
      return this.fun1.getF(t) + this.fun2.getF(t);
      
    } else if (this.operator == "-") {
      return this.fun1.getF(t) - this.fun2.getF(t);
      
    } else if (this.operator == "/") {
      return this.fun1.getF(t) / this.fun2.getF(t);
      
    } else if (this.operator == "*") {
      
      return this.fun1.getF(t) * this.fun2.getF(t);
    }
  }
}

class fun {
  constructor(function_type) {
    /*
    (str) -> None
    
    Creates a new, displayable function.
    function_type s:
        "sin", "cos", "tan", "sinh", "cosh", "tanh", "log"
    */
    
    this.function_type = function_type;
    this.function_data = [];
    this.eq = new algebricF();
  }
  
  getF(t) {
    /*
    (float) -> float
    
    Returns the result of the function at 't'.
    */
    
    // Please do not mind the atrocity below. That is how I normally code else..if statements before deciding to switch to more readable ones for the sake of this class.
    
    if (this.function_type == "sin") {
      
      let a = this.function_data[0];
      let r = this.function_data[1];
      let x = this.function_data[2];
      let y = this.function_data[3];
      
      return a * Math.sin(r * t - x) - y;
    } else {
      
      if (this.function_type == "cos") {
        let a = this.function_data[0];
        let r = this.function_data[1];
        let x = this.function_data[2];
        let y = this.function_data[3];
      
        return a * Math.cos(r * t - x) - y;
      } else {
        
        if (this.function_type == "sinh") {
          let a = this.function_data[0];
          let r = this.function_data[1];
          let x = this.function_data[2];
          let y = this.function_data[3];
      
          return a * Math.sinh(r * t - x) - y;
        } else {
          
          if (this.function_type == "cosh") {
            let a = this.function_data[0];
            let r = this.function_data[1];
            let x = this.function_data[2];
            let y = this.function_data[3];
      
            return a * Math.cosh(r * t - x) - y;
          } else {
            
            if (this.function_type == "tan") {
              let a = this.function_data[0];
              let r = this.function_data[1];
              let x = this.function_data[2];
              let y = this.function_data[3];
      
              return a * Math.tan(r * t - x) - y;
            } else {
              
              if (this.function_type == "tanh") {
                let a = this.function_data[0];
                let r = this.function_data[1];
                let x = this.function_data[2];
                let y = this.function_data[3];
      
                return a * Math.tanh(r * t - x) - y;
              } else {
                
                if (this.function_type == "log") {
                  let b = this.function_data[0];
                  let r = this.function_data[1];
                  let x = this.function_data[2];
                  let y = this.function_data[3];
      
                  return r * Math.log(t - x) / Math.log(b) - y;
                } else {
                  
                  if (this.function_type == "algebricF") {
                    return this.eq.getF(t);
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  
  changeTo(new_function_type) {
    /*
    (str) -> None
    
    Changes the function type and replaces all function_data with default values.
    */
    
    this.function_type = new_function_type;
    
    if (this.function_type == "sin" || this.function_type == "cos" || this.function_type == "sinh" || this.function_type == "cosh" || this.function_type == "tan" || this.function_type == "tanh" || this.function_type == "log") {
      this.function_data = [
        10,
        0.1,
        0,
        50
      ];
    } else if (this.function_type == "algebricF") {
      this.function_data = [];
      this.eq = new algebricF();
    }
  }
  
  display(txt) {
    /*
    (str) -> None
    
    Draws the function on screen using characters from txt.
    */
    
    let entry = -this.getF(current_offset * MOVE_RESISTANCE);
    let exit = -this.getF(width + current_offset * MOVE_RESISTANCE)
    
    let Y_OFFSET = (entry + exit) / 2
    
    if (Y_OFFSET < 0) {
      // pass
    } else if (Y_OFFSET > height) {
      // pass
    } else {
      Y_OFFSET = 0;
    }
    
    if (!ADJUST_SCREEN) {
      Y_OFFSET = 0;
    }
    
    for (let i = 0; i <= Math.trunc(width / POINT_INCREMENT); i++) {
      
      let Y = -this.getF(i * POINT_INCREMENT + current_offset * MOVE_RESISTANCE) - Y_OFFSET; // Inverted because of the coordinate system.
      
      if (isNaN(Y)) {
        continue;
      }
      
      let selected_char = txt[Math.trunc(i) % txt.length];
      
      fill(lerpColor(lerpColor(MAX_COLOR_Y, MIN_COLOR_Y, abs(Y) / height), lerpColor(MIN_COLOR_X, MAX_COLOR_X, i * POINT_INCREMENT / width), abs(i*Y * POINT_INCREMENT)/(width*height)));
      textSize(TEXT_SIZE);
      text(selected_char, i * POINT_INCREMENT, Math.trunc(Y));
    }
  }
}

function draw() {
  background(color(BACKGROUND_COLOR[0], BACKGROUND_COLOR[1], BACKGROUND_COLOR[2], BACKGROUND_COLOR[3]));
  
  if (PREBUILT_DISPLAY_NUMBER == 2) {
    
    F1.display("from nothing we were raised and with nothing we return");
    F2.display("from nothing we were raised and to everything we built");
  
    F3.display("and return to gardens we may");
    F4.display("and return to flames we will");
  } else if (PREBUILT_DISPLAY_NUMBER == 3) {
    
    AF.display("this is an algebric function");
  } else if (PREBUILT_DISPLAY_NUMBER == 1) {
    
    F1.display("this is a simple sin function");
  } else if (PREBUILT_DISPLAY_NUMBER == 4) {
    
    CR.display("Raja Qasim Khan (rk5260)")
  }
  
  
  
  
  if (isMoving) { // This translates the graph to create a cool moving illusion.
    
    current_offset++; // a tanslation for the lateral movement of the function(s).
  } else if (OVERLAY_ENABLED) {
    
    // display overlay menu.
    fill(color(255-BACKGROUND_COLOR[0], 255-BACKGROUND_COLOR[1], 255-BACKGROUND_COLOR[2])); // the inverse color
    textSize(14);
    
    text("Click to toggle.", 100, 20);
    text("Demo (press key): ", 100, 34);
    text("0: Disable demo.", 100, 48);
    text("1: Single cos demo.", 100, 62);
    text("2: Multi-function demo.", 100, 76);
    text("3: Algebric demo.", 100, 90);
    text("4: Credits info", 100, 104);
    text("T: Disable overlay!", 100, 118);
    text("C: Capture canvas (image).", 100, 132);
  }
}

function mouseClicked() {
  
  isMoving = !isMoving;
}

function keyPressed() {
  
  // Switch displayed prebuilt simulation.
  if (keyCode == 48) {
    
    PREBUILT_DISPLAY_NUMBER = 0;
  } else if (keyCode == 49) {
    
    PREBUILT_DISPLAY_NUMBER = 1;
  } else if (keyCode == 50) {
    
    PREBUILT_DISPLAY_NUMBER = 2;
  } else if (keyCode == 51) {
    
    PREBUILT_DISPLAY_NUMBER = 3;
  } else if (keyCode == 52) {
    
    PREBUILT_DISPLAY_NUMBER = 4;
  } else if (keyCode == 84) {
    
    OVERLAY_ENABLED = !OVERLAY_ENABLED;
  } else if (keyCode == 67) {
    
    // capture screenshot of canvas.
    saveCanvas("rqk_TextualGraphics_HH" + hour().toString() + "_MM" + minute().toString() + "_SS" + second().toString() + "_dd" + day().toString() + "_mm" + month().toString() + "_yyyy" + year().toString())
  }
}

Link to project: https://editor.p5js.org/rk5260/sketches/Y8JIsLCxj

The Subjectivity of Linguistics and Definitions

It is a point that I most strongly resonate with, and one that I may struggle to articulate so fully as that which was present in Crawford’s reading, that I begin by producing a quote from said reading:

“… I take a more easygoing view of definitions. Any idea worthy of my attention is probably too big and too complicated to be reduced to some schoolmarmish formula.” (Crawford, 5)

As is to Crawford, to me linguistics is about a message. And it is so that the delivery of a message is as much an artform as is an artform about the delivery of a message. Oftentimes, I find definitions an easy way to brush over more nuanced topics.

As for interactive systems, I strive not to pen down a concrete definition, but to explore them based on their attributes and depth of interaction. Let us take a simple interactive system: a classic slot machine in the Ol’ Flamingo. Its got a lever for players to pull, makes cool sounds when you win, its got flashy lights and a degree of manual work in pulling the lever. It builds anticipation in the spinning of the slots. On the plus side (a win for the casino), the casino serves you drinks as you gamble away.

In essence, all five of your senses are deeply entrenched in this interactive system. And the drinks you’ve been served are likely to keep your brain engaged elsewhere from the fact that your credit rating isn’t looking awfully good at the moment. If I were to rate it on a scale of 1 to 10, with 10 being the strongest interactive system, it would most certainly be a good 8, for its pretty holistic in its nature, however cruel and money-depleting that nature may so be.

As did Crawford elaborate on interactive systems: “interaction: a cydic process in which two actors alternately listen, think, and speak”

In essence, the formerly described slot machine engages all of ones senses, it will never be a ten for one simple reason, that it lacks to spur intelligence. It does not really heighten one’s intellectual curiosity or thoughts. Its a repetitive action with the only randomness being how much your bank account depletes after every pull.

This is that which distinguishes between a good interactive system and a strong(er/est) interactive system. Id est a system that strives to not only address our superficial senses and thoughts, though in addition delves deep into a more nuanced realm of human thought and complex emotions.

It is that which is also present within Crawford’s formerly quoted definition.

I believe a p5 sketch could be honed but only so to an extent, for while we may invoke a multitude of senses, id est ones sight, perhaps emotion and intellectual curiosity, maybe even sound, it lacks touch, smell and taste. As someone who loves food oneself, many of my activities (in game design) and the real world do then to revolve around food, but the problem I run into is the same, we can not capture the actuality of every human sense in p5 sketches. Maybe that is the imperfect beauty of this medium, and so to me I will focus on honing the sights and sounds of my p5 sketches, with keeping in light a degree of naturality or randomness that helps to convey a deeper meaning, or perhaps is just there to perplex the observer!