Akyn the device that takes care of everything, so you don’t have to.

  • Describe your concept
    • Akyn is a Jarvis(Iron Man) like artificial intelligence that helps to make morning and night routines more pleasant. As a student, I can indeed say that many of us are lazy. We keep procrastinating a lot and as a result, get stressed out when the deadline approaches. Akyn is a way to change that. With only 2 buttons, you can escalate your morning and night routines to a new level.
    • As part of the morning routine, Akyn will turn on the AC for you and greet you in a respectful manner. Afterward, it will read out(over voice) the emails you have gotten over the night. And as a last action, it will play motivational music so there is no way you still lay in your bed.
    • When you go to sleep, Akyn will turn off the lights and AC for you. Afterward, Akyn voices over your alarm time for tomorrow and starts listing the events from your Google Calendar for tomorrow. This way you will start visualizing tomorrow starting from tonight.
  • How does the implementation work?
    • Description of interaction design
      • The interaction design is very simple. There are 2 buttons on the breadboard located right near your bed. Green button for morning routine and Blue button for night routine. By pressing one of them, the respectful routine gets triggered.
    • Description of Arduino code
      • The Arduino concept is very simple. There are 2 buttons and 2 servo motors connected.  In the setup, I connect the motors and start the handshake process with the p5.js. In the main loop, while there is a connection established, I keep waiting for the user to press either of the buttons. Once one of them is pressed the motor(s) start rotating which turns on/off AC/lights. Lastly, 0 or 1 is sent to the p5.js indicating which button is pressed.
      • #include <Servo.h>
        
        Servo servo_lights_off;
        Servo servo_AC;
        
        int pin_GM_button = 2;
        int pin_GN_button = 4;
        
        int pin_lights_off = 11;
        int pin_AC = 9;
        
        int state_GM_button = 0;
        int state_GN_button = 0;
        int prev_state_GM_button = 0;
        int prev_state_GN_button = 0;
        
        void setup() {
          servo_lights_off.attach(pin_lights_off);
          servo_AC.attach(pin_AC);
          Serial.begin(9600);
        
          // start the handshake
          while (Serial.available() <= 0) {
            Serial.println("-1"); // send a starting message
            delay(300);            // wait 1/3 second
          }
        }
        
        void loop() {
          while (Serial.available()) {
            state_GM_button = digitalRead(pin_GM_button);
            state_GN_button = digitalRead(pin_GN_button);
            // GM button is pressed
            if (state_GM_button != prev_state_GM_button && state_GM_button == 1) {
              // turn on the AC
              servo_AC.write(180);
              delay(1000);
              servo_AC.write(0);
              delay(1000);
              
              // send message to P5.js
              Serial.println("0");
            }
            // GN button is pressed
            if (state_GN_button != prev_state_GN_button && state_GN_button == 1) {
              // turn off the lights
              servo_lights_off.write(0);
              delay(1000);
              servo_lights_off.write(180);
              delay(1000);
              // turn off the AC, double click is needed to go to Night mode from Morning mode
              servo_AC.write(180);
              delay(1000);
              servo_AC.write(0);
              delay(2000);
              servo_AC.write(180);
              delay(1000);
              servo_AC.write(0);
              delay(2000);
        
              // send message to P5.js
              Serial.println("1");
            }
            
            // update the previous states of the buttons
            prev_state_GM_button = state_GM_button;
            prev_state_GN_button = state_GN_button;
          }
        }
    • Description of p5.js code
      • P5.js is where the main magic happens. The main interaction happens in the javascript file while the authorization happens in the index.html page.
      • Authorization
        • When the user starts the code after a specific amount of time(10 seconds for the user to have some time to read the instructions) authorization script is run. This script sends requests to Google Calendar API and Gmail API over the gapi library. As a result, I am now able to send get requests to the Google API. After the authorization, I load the calendar using a specific query indicating what kind of events I want to see. All of this is saved into the CALENDAR_EVENTS variable
          <script>
          
              const gapiLoadPromise = new Promise((resolve, reject) => {
                gapiLoadOkay = resolve;
                gapiLoadFail = reject;
              });
              const gisLoadPromise = new Promise((resolve, reject) => {
                gisLoadOkay = resolve;
                gisLoadFail = reject;
              });
          
              var tokenClient;
          
              (async () => {
                // First, load and initialize the gapi.client
                await gapiLoadPromise;
                await new Promise((resolve, reject) => {
                  // NOTE: the 'auth2' module is no longer loaded.
                  gapi.load('client', {callback: resolve, onerror: reject});
                });
                await gapi.client.init({
                  // NOTE: OAuth2 'scope' and 'client_id' parameters have moved to initTokenClient().
                })
                .then(function() {  // Load the Google API's discovery document.
                  gapi.client.load('https://www.googleapis.com/discovery/v1/apis/calendar/v3/rest');
                  gapi.client.load('https://www.googleapis.com/discovery/v1/apis/gmail/v1/rest');
                });
          
                // Now load the GIS client
                await gisLoadPromise;
                await new Promise((resolve, reject) => {
                  try {
                    tokenClient = google.accounts.oauth2.initTokenClient({
                        client_id: google_calendar_USER_ID,
                        scope: 'https://www.googleapis.com/auth/calendar.readonly https://www.googleapis.com/auth/gmail.readonly',
                        prompt: 'consent',
                        callback: '',  // defined at request time in await/promise scope.
                    });
                    resolve();
                  } catch (err) {
                    reject(err);
                  }
                });
              })();
          
              // get token in case error while authorizing appeared
              async function getToken(err) {
          
                if (err.result.error.code == 401 || (err.result.error.code == 403) &&
                    (err.result.error.status == "PERMISSION_DENIED")) {
          
                  // The access token is missing, invalid, or expired, prompt for user consent to obtain one.
                  await new Promise((resolve, reject) => {
                    try {
                      // Settle this promise in the response callback for requestAccessToken()
                      tokenClient.callback = (resp) => {
                        if (resp.error !== undefined) {
                          reject(resp);
                        }
                        // GIS has automatically updated gapi.client with the newly issued access token.
                        resolve(resp);
                      };
                      tokenClient.requestAccessToken();
                    } catch (err) {
                      console.log(err)
                    }
                  });
                } else {
                  // Errors unrelated to authorization: server errors, exceeding quota, bad requests, and so on.
                  throw new Error(err);
                }
              }
          
              function loadCalendar() {
                const tomorrow = new Date();
                tomorrow.setDate(tomorrow.getDate() + 1);
                tomorrow.setHours(0, 0, 0, 0);
          
                const after_tomorrow = new Date();
                after_tomorrow.setDate(after_tomorrow.getDate() + 2);
                after_tomorrow.setHours(0, 0, 0, 0);
          
                var calendar_query = { 'calendarId': 'primary', 'timeMin': tomorrow.toISOString(), 'timeMax' : after_tomorrow.toISOString(), 'singleEvents': true, 'orderBy': 'startTime'};
          
                // Try to fetch a list of Calendar events. If a valid access token is needed,
                // prompt to obtain one and then retry the original request.
                gapi.client.calendar.events.list(calendar_query)
                .then(calendarAPIResponse => CALENDAR_EVENTS = JSON.stringify(calendarAPIResponse))
                .catch(err  => getToken(err))  // for authorization errors obtain an access token
                .then(retry => gapi.client.calendar.events.list(calendar_query))
                .then(calendarAPIResponse => CALENDAR_EVENTS = JSON.stringify(calendarAPIResponse))
                .catch(err  => console.log(err));   // cancelled by user, timeout, etc.
              }
              
              // call loadCalendar after 3 seconds so that gapi.client initiates connection
              setTimeout(loadCalendar, 10000);
          
            </script>
      • Main logic
        • Now once everything is loaded, the client connects the Arduino and p5.js by simply clicking on the screen and selecting the Arduino port. From now on, everything gets triggered once the button is pressed.
        • Green button – goodMorning function gets called.
          1) Load Gmail information about unread emails
          2) Clear up the saved array, leaving only the necessary information to display
          3) P5.speech libraries voices over the amount emails received, the subject, and the sender info of each one as well. Upon completion, Mozart’s Eine kleine Nachtmusik classical music starts playing.

          /* voices over the Good Morning mode*/
          function goodMorning() {
            // load the unread emails
            loadGmail();
          
            // timeout to wait till the above functions finishes execution
            setTimeout(function() {
              clearEmails();
            }, 2000);
          
            setTimeout(function() {
              myVoice.speak(GM_PHRASES[Math.floor(Math.random()*GM_PHRASES.length)] + "You've got " + emails.length + " emails while you were sleeping.");
              if (emails.length === 0) {
                return;
              }
              emails.forEach((email, index) => {
                setTimeout(function() {
                  console.log(numberMapping[index + 1] + ' email is from ' + email.from + ' about ' + email.subject); // -> debug
                  myVoice.speak(numberMapping[index + 1] + ' email is from ' + email.from + ' about ' + email.subject);
                }, (index + 1) * 10000); // wait for the previous voice to finish before starting next one
              });
              setTimeout(function() {
                myVoice.speak("Turning on music!");
              }, (emails.length + 1) * 10000);
              setTimeout(function() {
                playMusic();
              }, (emails.length + 1) * 10000 + 3000); // wait till events finishes voicing and the previous line too.
            }, 6000); // this timer is the sum of the before called timers (1000 + 3000) + 1000 extra miliseconds
          }
        • Blue button – goodNight function gets called.
          1) Clear up the saved calendar events, leaving only necessary information to display
          2) Voice over the number of events and the time the alarm is set to.
          3) Voice over each of the events, time, and location

          /* voices over the events happening tomorrow*/
          function goodNight() {
            events = clearEvents();
            setTimeout(function() {
              const eventsLength = events.length - 1;
              let p = "Tomorrow you have " + eventsLength + " events. Your alarm is set to " + events[0]['startTime']+ " AM. " + GN_PHRASES[Math.floor(Math.random()*GN_PHRASES.length)];
              myVoice.speak(p);
              // console.log(p);
            }, 1500);
          
            // iterate thoruhg events and voice them over
            // timeout used since forEach does not wait for the voice to finish which gives undesired output
            for (let index = 1; index < events.length; index++) {
              const event = events[index];
              setTimeout(function() {
                console.log(numberMapping[index] + ' event is ' + event['eventName'] + ', happening from ' + event['startTime'] + ' till ' + event['endTime'] + ' at ' + event['location']);
                myVoice.speak(numberMapping[index] + ' event is ' + event['eventName'] + ', happening from ' + event['startTime'] + ' till ' + event['endTime'] + ' at ' + event['location']);
              }, index * 13000);
            }
          }
    • Description of communication between Arduino and p5.js
      • Due to the nature of the project, communication is one-sided. P5.js keeps waiting for Arduino to send 0 or 1 after which the main logic happens. Arduino does not need any input from p5.js all it does is wait for the buttons to be pressed.
  • What are some aspects of the project that you’re particularly proud of?
    • I am very proud of embedding Google API’s into my project. This is something I have not done before nor was it reached in any of the classes. Yet this is something very useful and applicable in a lot of spheres. By doing this, I was able to make the project actually helpful and not just some dummy toy. I believe that the use of this API together with the speech library and servo motors makes the project very unique.
      Also, coming up with balanced values for timeout is something to be proud of. All API requests need to be timeout so that the next functions can use the result of that get request. I experimented a lot with these values so that users do not wait too long and the response has enough time to arrive.
  • What are some areas for future improvement?
    • In the future, I would be happy to change the timeouts to Future Promises, so that the further functions only get triggered once the previous ones finish. This way it’s not hardcoded and works with any amount of response.
    • Working with wires was fun, however, it would be much more flexible and convenient to use either a Bluetooth connection or even a phone. With a little bit of research, I can substitute long and limited wires to a Bluetooth connection between the breadboard and Arduino. With more research, I will be able to write a telegram bot that sends commands directly to Arduino without any buttons.
  • Have people try your project without giving them any prompts/instructions and see how they use it
    • Are they able to figure it out? Where do they get confused and why? Do they understand the mapping between the controls and what happens in the experience?
      • I invited a few friends to try out my project. They really enjoyed the instructions screen and it was very straightforward to authorize Google Account. However, a useful comment I received was to specify the name of the port to which users have to press in order to establish a connection to the Arduino.
      • Further interaction was also very clear to my friends. They went near my bed and pressed either a blue or green button.
    • What parts of the experience are working well? What areas could be improved?
      • They enjoyed a lot the voiceover of the calendar and Gmail. They also liked my choice of music. One thing they stated can be improved is the choice of voices. Since p5 speech libraries hold more than 100 voices, people would like to pick one that they want to hear the most. I believe this is something that can be implemented, nevertheless, I did not want to complicate the project with voice pick since it does not align with the main goal of improvement of morning and night routines.
    • What parts of your project did you feel the need to explain? How could you make these areas clearer to someone experiencing your project for the first time?
      • I had to explain to them that once the button is pressed they have to wait till the whole routine is finished before pressing a button again. If someone rushes with a button press, the voice will queue up and some of them will be dropped which is highly undesirable. A possible solution for this is to lock the button action until the whole routine has finished(send value from p5.js to Arduino when everything is finished and keep waiting for a response on Arduino). This can be a great addition if I decide to improve the project in the future.

Author: Akhat Suleimenov

I am 22 years old. Alumni of 42 Software Engineering School in Silicon Valley. Born and raised in Astana, Kazakhstan. As hobbies, I workout, travel, and play volleyball, and chess.

Leave a Reply