Results and Closing Notes

In our last couple weeks for the project our Jaco 2 robot arm by Kinova was having software issue. We were able to get our system working, but not consistently. After working with the Kinova support team we finally fixed the software issue with the robot. The Semi-Autonomous Meal Assistance Robot (S.A.M.A.R) is fully functional. The Jaco arm is paired with a Microsoft Kinect V2 sensor. This combination detects when the users mouth is open, tracks the users head, brings the food to their mouth, and does not spill the contents it is carrying.

 An argument could be made that this system is too complex and that facial expression recognition could be replaced with a simple button instead. However, this is only the beginning of this concept and our project has developed the base of this technology. This technology is not just limited to meal assistance, the Jaco arm can easily be programmed to complete other tasks. The Kinect also tracks multiple face features which can lead to tasks triggered by a single facial expression or combinations of expressions. To expand the concept of our project, the Kinect also has voice control. The possibilities of ways to trigger the Jaco arm through the Kinect are endless.

Over my time doing research for assistive technology one of the biggest things that I have learned is that it is very difficult to design something for general use. There are many disabilities that are unique and therefore required unique devices depending on what the user is capable of and their needs. Our project can lead to libraries of task that are custom made to fit the needs of a specific user. The ways of triggering the tasks, such as facial expressions or voice control, can also be custom made to fit the capabilities of the user. Our project is only the beginning, I hope that this research is continued because there is a bright future ahead.

IMG_4768

This REU program has been a great experience for me. I really enjoyed the work that I completed with my group. Previously my experience with robots has been limited to manufacturing style systems but my experience at Stout has opened my eyes to a completely new world of robotics. I have learned a great deal about assistive technology and I’ve gotten a feel of what grad school is like. I have also enjoyed my time spent in the city of Menomonie. There are many scenic places to visit such as lake Menomin, Wakanda Park, Riverside park, and the Devils Punch Bowl park. Also for the most part, everything is within biking distance of Red Cedar Hall. I intend on returning to my university and complete my undergraduate degree in mechanical engineering. After I graduate I would like to start working in industry. Graduate school is something I would consider but perhaps further down the road. I’m excited to pursue more knowledge in robotics and can’t wait to see what the future holds for myself and the work that our group has done.

 

REU Reflection

Now that we have reached the end of the REU program, I am truly grateful for this experience. I was thanking my lucky stars that I took and intro to C++ course last spring, because the skills I learned were invaluable throughout this project. I was also able to build my knowledge of robotics, as I learned a lot more about the language used when developing robots and how to change and build upon existing code. Another important aspect of this project for me was working in a team. I have done lots of school projects in teams, but it is a whole different experience to work on a research project in a group.

I had a very nice time in Menomonie. The weather was for the most part beautiful, and I very much enjoyed myself here. It was really fun getting to know my roommate, Vanessa, and we had a lot of fun going to movies on Tuesday nights and Caribou on Thursdays, along with coffee breaks at Marion’s on occasion. A personal goal I had for myself was to get enough punches on my Marion’s punch card to get a free drink, and I was able to meet that goal. I got a free iced mocha if you were curious. Also I would highly recommend that drink.

After the program ends, I have just over three weeks until school starts for me. During that time, I will be working on a Flu study in my lab at the University of Minnesota. When school starts in the fall, I will be taking classes, working as a tutor, volunteering the lab, and working with a group of students to create the seat for my school’s solar vehicle project team. Something I’m greatly looking forward to in the fall is going to the SWE conference in Philadelphia.

Also, I would like to make a special internet shout out to Laura, who religiously read this blog.

Research Wrap Up

Things are winding down here at the Robotics REU. Though we had major technical difficulties, we were able to get the meal assistance robot working and some data collected. The final system didn’t end up entirely as I’d hoped, as when we tried testing it with someone with an upper extremity disability it didn’t work. When completed, I think this technology could have a huge impact. In the future, I hope that the meal assistance feature does more than just bring food to the user’s mouth, but also levels the utensil on its own.

I think there is a lot of different ways that this technology could improve in the future. A major advancement of our technology would be a way for the Kinect to determine the location of the base of the JACO arm. Currently, we need to directly measure the offset from the Kinect to the JACO in order for the meal assistance feature to bring food to the user’s mouth. If this were possible, the Kinect could be placed anywhere and the user would have more freedom with the system. In addition to this, the Kinect could be used to control not only the robot when eating, but also for other common movements the user wants the robot to do. For example, the user could smile to indicate that they want the robot to brush their teeth. I think there are a lot of ways future researchers could expand on our work in order to improve this technology.

REU Reflection & Wrap-Up

Today marks the last day of the REU program here at University of Wisconsin-Stout. During the last 8-weeks, Meng and I worked on developing robotics to help the blind navigate complex, indoor environments. We made significant progress towards our goal, and I learned a lot. The major players in our research project was the iRobot Create 2 development platform, BeagleBone-Black, wireless Bluetooth module, and a SLAM algorithm using a webcam.

Although we did not end with a finished and polished system, our progress is still significant and important. And, I do encourage this project to be continued during the next summer’s Robotics REU program. Being able to provide assistive robotics to those in need for a low-cost could change people’s lives because most assistive robots in today’s market are very expensive. The most impressive aspect that I took away from this research is the amount of creativity, technical ability, and user experience knowledge that one must develop in order to work in this area. Robotics truly is an intriguing interface between software, hardware, and human factors that makes it challenging and a fun place to be. This program introduced me to all of this, and for that I am truly grateful to the University of Wisconsin-Stout REU program and to the National Science Foundation as the granting organization.

Spending the last 8-weeks in Menomonie has flown by. I have really enjoyed spending time with all of the other REU researchers, exploring the UW-Stout campus, and getting to know different faculty. As a part of the program, I was introduced to a lot of different people including different engineering faculty, the great people at the Vocational Rehabilitation building, and the GRE tutors from the McNair program. This program has opened me up to how research can be conducted and has made me begin to think more seriously about my future and graduate school possibilities.

Lastly, I’d like to thank all of the REU mentors: Cheng Liu, Devin Berg, and Wei Shi. Thank you for all of the work it takes to apply for this NSF grant, organize the events, and provide a place for this to happen. A special thanks to Professor Cheng Liu who directly advised Meng and I in our research project.

Project Wrap Up and Final Thoughts

As I begin to wrap up my time in Menomonie, I would like to take some time to share my thoughts on my project and my experience here.

Project Wrap Up:

During the last 8 weeks, I have created a demonstration to show how my assistive robotic device to aid with catheter insertion would operate. From there I researched different tremor suppression techniques, which include using 1) sensors and actuators, 2) weight, 3) mass spring damper systems, and 4) an uncoupled device. Upon further investigation, I decided that my assistive robot should be used with an uncoupled device (that is using a stand similar to a camera tripod), to completely remove the device from the tremor motion allowing for smooth catheter insertion. From there, I created a mock up design of what the assistive robot and stand may look like. For future work, I hope to begin finalizing my design and prototyping the device, and then start testing the equipment. Initially testing would be done on catheter insertion dummies used in hospitals to teach nurses how to insert catheters. Using these dummies, I could see how well the device transfers the catheter from the device into the urethra, and initial improvements could be made from there.

With respects to my project, I found this experience to be a great learning opportunity, but also to be very challenging at times. Going through the engineering design process has showed me how expectations for completing a project do not always line up with the reality of the design process.  In all honesty, I thought I would have had a complete prototype built, tested and ready to present at my final presentation. But this was my expectation. In reality, I found that designing a new device takes more time than expected, requires more research than I thought, and you hit many more bumps in the road than predicted. Even though I did not accomplish as much as I had hoped, I was able to lay a foundation for future work to be built upon and I am proud of how much I was able to accomplish in the last 8 weeks. In addition, I am grateful for the opportunity to better understand the engineering design process. Prior to arriving, I had completed a few small group design activities, but nothing to this magnitude. I realized that previously I have not been given the opportunity to fully utilize the design process and I’m thankful that this experience has given me the chance to explore how this process works, and provided me with better insight on how to design a product. Overall, I believe that I have presented a great starting place for this design. I believe this product could improve someone’s life and I hope that the information I have laid out in my final paper would be beneficial to future work on this product.

I hope that this technology will eventually be prototyped and tested so that those who struggle with inserting a catheter can live a more independent life. Furthermore, I hope that the work I have laid out and my analysis on tremor suppression techniques will inspire others to find ways to help those who suffer from this neurological disorder, and will inspire new technological developments in creating tremor suppression devices.

Final Thoughts:

As I reflect on the REU program as a whole, I realize that I really did not have any expectations prior to arriving on campus, and thus my expectations were neither met nor proven wrong. I found this opportunity to be a great chance to get my feet wet with engineering experience, as well as a great way to push me to think outside the box and start thinking like an engineer. I feel honored to have meet colleagues who also share a passion and desire to help others. From my perspective, most of my fellow engineering students seemed to be more focused on the money that they will make or the things that they will have, rather than focusing on how their talents can help the human race. It was reassuring to see a group of talented individuals put their time and energy into products that matter and could help improve and individual’s life.

I also learned that I need to be more sure of myself as an engineer. When I first started out, I constantly seconded guessed each of my decisions. I realized though, that I need to be confident in what I am doing and not be afraid to take risks. The worst thing that can happen is failure, and in the end failure is really not a bad thing. Failure provides a learning experience for you to build upon and helps ensure you will not make the same mistakes again. In the end, I’ve learned that I should not worry as much, and just be confident and proud of my decision and this will help me complete my work faster and more efficiently. This is definitely a lesson I am going to take to heart moving forward because being confident in myself is going to make me a better engineer.

I have enjoyed my time in Menomonie. Though the town was a little smaller than what I am used to, it was nice to be in a quiet, tight knit community for the summer. In addition, being in Menomonie provided me with the opportunity to go to the Mall of America for the first time, and go to the Leinenkugel brewery, which I don’t think I would have ever done if I hadn’t been here. Overall, I have had a great experience in this town, and I am thankful for the time I was given to get to know my roommate, Erin. I don’t think I could have gotten through the summer without our weekly movie theater outings and Caribou/dollar store runs. I feel honored that I had the chance to get to know this talented and inspiring female engineer.

With regards to the future, I am preparing for my senior year at Marquette University, which begins in a few weeks. My senior year will hold many fun memories with friends, along with a lot of school work and spending many hours dedicated to my capstone project. Upon graduation, I hope to move back to Colorado to be closer to my family. I would like to work in industry for a few years to pay off my student loans, and then be sponsored by a company to go to graduate school. I hope to be working at a company that sees the value of the human person, and strives to make this world a better place.

REU Wrap-Up and Project Reflection

As my partner and I wrapped up our robotic project, I realized how fast time has flown by during this REU period. We did not end up with a fully functioning robot aide like we had originally planned, however, we did accomplish many of the tasks that dealt with the different parts of the robot system individually. Some of the major points that we were able to tackle included the iRobot to Beaglebone Black (BBB) connectivity, HC-05 Bluetooth module to BBB connectivity, iRobot communication and control with the smartphone app, and an LSD-SLAM algorithm implementation with a camera. Due to limited time and running into hardware issues at the end of the project period, we were not able to fully demonstrate all of our accomplishments.

The time and work put into this research will definitely not be wasted. To be able to see the different parts of the project working is already a good step forward. I think the impact of the research done behind our robotic aide will definitely be significant. Assistive robots are expensive, and by being able to show that we can create one with an iRobot Create 2 is very promising. Bluetooth wireless technology was also researched and implemented with the robotic aide. This technology is becoming huge so having dug our hands into the basics will give us a jump start as future developers. The potential of robot aides and wireless technology can definitely prove to be a benefit in helping people.

There is a lot of future work that can be done under our robotic aide for the blind project. First of all, it would be a good place to start by completely assembling everything that we had working into a fully functional robot system. Aside from that, I think implementing voice-control/command and  voice-feedback would be definitely helpful to the group of individuals that we intended this robot system for which were blind people. A robot that can output simple warning noises after detecting an oncoming obstacle or danger could be very helpful. Not just that but the robot could also be programmed to give audio-out directions as it leads the blind user inside a building. A walking-stick mount or something similar would also be needed as a link between the robot and blind user.

Personally, the Robotics REU here at the University of Wisconsin-Stout has been such an eye-opening experience. I attend this university as an engineering student and I learned so much about the Vocational Rehabilitation Institute on campus and the biomedical technology set in place here at the school, all of which was completely unknown to me prior. I learned a lot about planning and running a research project during the REU. One lesson I learned the hard way is to make copies of my work and to document working parts of the project right away just in case the worst happens.

I am satisfied with how far the research and project has come despite not having a completely working robot aide by the end of the REU. I am thankful for the mentors, who guided us and helped us through the whole research experience. I am also thankful for the engineers from outside universities that I have come to know through the past few weeks. It’s been a great summer here in Menomonie.

Currently, I will be planning on graduating in the fall semester. This opportunity has allowed for me to have a better understanding and a more in-depth experience on how research is conducted as a student. It also has given me a chance to discuss and learn about graduate school, which is something I have never really considered. I hope to use all that I have gained from this REU opportunity to help me in the very near future.

Wrap-up!

8 weeks flies by in no time when you’re working hard! In the case of the REU this was very evident. The title of my research ended with “Path Oriented Electric Wheelchair Steering Assistance”. The project was a success, although there is still plenty of work to make it complete. The prototype and proof of concept achieved its objective and can only improve. I was able to navigate the wheelchair around campus using nothing but my navigation system. The major hurdle in this project was getting the device to integrate with the wheelchair. After a few weeks of tinkering with the joystick controller, I found that it was too proprietary and wouldn’t let my apply my own steering directions. This caused me to change up my thinking and consider how to show that the concept would wok without actually controlling the wheelchair. I eventually went with an LED array output to determine if the wheelchair was on course or not. Overall it went well and I’m pleased with the result.

In order for this concept to be applied in commercial use, I’ll need to continue coordinating with external collaborators. I will be submitting the results of my research to The Journal of Open Engineering to spread the knowledge to others that may be able to use it. Since the project is entirely open-source, others will be able to recreate my results.

I picked up many skills over my time in Menomonie. I learned basic research skills such as documentation and time-management. In addition, the ability to communicate to others was very important during presentations, written and other ways. Finally, and most importantly, finding the distinction between turning a failed idea into a learning experience and working through a challenge came up everyday and fueled my continuation.

 

The UW-Stout camous is beautiful and well kept. The facilities are top notch, especially the electronics lab and Fablab. I was able to get 3d printed parts very quickly thanks to the great staff in the fablab and the electronics lab had everything I needed to test out the wheelchair joystick. The other part of campus I enjoyed were the gym facilities. I’m very active and having a 24 hour weight room right next to the dorm was great considering I was there everyday. We started a tradition early on between some of the REU researchers; the bunch of us went to the Waterfront restaurant for “Waterfront Wednesday Wings”. It’s a great place to cool off in the middle of the week and enjoy a good meal on the lake!

 

In closing, I’d like to thank UW-Stout for hosting this program as well as the advisors for organizing and especially to Dr. Devin Berg for helping me through the entirety of my 8 weeks here.

Technological Innovation – Bluetooth

One of technology’s biggest contributions is wireless 512px-Bluetooth.svg
communication. This form of communication involves information or data being transmitted between two or more devices without any form of an electrical conductor connection, such as wires or cables. The wireless transmission of information between two devices can range from a few feet to thousands of miles depending on the technology. Among the newer methods of achieving wireless communication is bluetooth. Bluetooth modules for electronic projects is relatively inexpensive and easy to use technology for short  range communication.

41LcDO6EztL._SY450_
HC-05 Bluetooth Module

Our Robotic Guide will be implemented with bluetooth to allow users to, for any reason, control the robots movement from an android app. Wireless communication will be made between the Beaglebone Black (BBB) controller and a smartphone. We are using the HC-05 bluetooth module which is wired to the BBB. Once the HC-05 and smartphone are paired, we are then able to send button press signals from the app to be read by the BBB. The BBB then translates the information to control the robot remotely. Future work on our robot may include voice commands and audible directions for blind or visually impaired users.

Bluetooth technology has been used in other ways to assist the blind/visually impaired with their daily routines. A smartphone app was designed to help provide step-by-step directions to a blind person navigating the London Underground. This technology for indoor navigation uses bluetooth beacons to transmit signals that can be detected by mobile devices. The app produces audio-out directions for the user and it also warns the user of upcoming stairs or escalators (Read more: http://www.bbc.com/news/health-31754365).

Progress Update #2

Week six is very busy and I am currently on schedule with the projects deliverables, so let me catch you up on the progress. “Industrial design philosophy is the rendering process from blue sky concept, prototype, to a finished product. Specifically how does one convey and deliver the idea and use lines to interpret those structures? How can we facilitate a smooth integration of linear creativity into modular design, the plausible idea of 3D from 2D?” An article from Architectural Services and Consultation descries the abstract and scale less form of line weight and hatching techniques using during concept sketching to emphasis translating geometrical information across plainer 2D weightless dimension. This graphical study places rigorous and iterative process on translating density into depth with the intent of exploring new spatial and qualitative effects while concentrating on latent patterns. They show this conceptually with outlined sketches with the 3D projected view from the 2D surface. Variable Projects is an award-winning design and research office with a focus on architecture elements, computations, and fabrications. Using this artist’s research and expressions I hope to incorporate their findings along with ours into a 3D desktop system to scan an image and replicate similar imaging processing.  (1) http://www.variableprojects.com/#/computational-drawings-1/

123

I have reached out and am currently working with the assistant professor of art and design, Dr. Kimberly Loken here at University of Wisconsin-Stout for her professional theory and how she conveys the idea and process to her lectures. With permission I have obtained some sample industrial design images created by the university student community that later will be vital in the test phase of the program I am developing. The theory is to create a new dynamic system incorporating processing with and a robotic system. The deliverable is achieved with a bit dot map file created when the user imports his/her fished industrial sketch image and the program deconstructs those lines into height profiles. Later (In development) the dot map file with be meshed to create a polysurface that the robot will follow via the extrusion head.

234

Since this is a niche market the application and need is but in high demand since little research is conducted. With this technology filler material is not consumed to build a full object but just the shell of the object. This allows for quick prototyping capability while obtaining the idea of what the artist is trying to convey.  Product design, pitching ideas, connecting people, problem solving, and sketching are target words for this project and application. Plausible future application would be to develop this towards the medical field for skin graphs and retina regeneration if micro stepping printing is achievable.

345.JPG

Waving arm around

Motion planning is required for moving and positioning the robot arm. Creating our own motion planning algorithms is a bit beyond the scope of our research. We are currently considering using the motion planning built into the arm through the Kinova SDK or the MoveIt! package from Robot Operating System [1]. We will be implementing the controller described by Y. Kuriyama [2] to prevent dropping food as it is transported from the plate to the mouth. Although that controller is specifically for liquid, we will attempt to use it for solid food. I. Naotunna et al. [3] mentions that the trajectory of the eating utensil should avoid approaching the face along the sagittal plane to avoid making the user feel uncomfortable. In layman’s terms, don’t move the fork straight at someone’s face, it makes them uncomfortable, move in at an angle to the face.

Controlling the robot arm is hard. Moving a 6-axis robot arm with a 3-axis joystick is rather awkward. We are investigating methods to make controlling the arm easier. Our idea is to make eating food with the arm easier by having the arm move autonomously for parts of the task of eating. I have been specifically focusing on implementing motion planning for the robot arm. The arm needs to be moved quickly and precisely to avoid injuring the user and to improve eating speed.

One extension to our work is catheter insertion. People who do not have full control of their body still produce waste. The robot arm can assist with catheter insertion. We will need to heavily modify the face recognition component of our current system.

[1] Ioan A. Sucan and Sachin Chitta, “MoveIt!”, [Online] Available: http://moveit.ros.org.
[2] Y. Kuriyama, K. Yano and M. Hamaguchi, “Trajectory planning for meal assist robot considering spilling avoidance”, 2008
[3] I. Naotunna, C. J. Perera, C. Sandaruwan, R. A. R. C. Gopura and T. D. Lalitharatne, “Meal assistance robots: A review on current status, challenges and future directions,” 2015 IEEE/SICE International Symposium on System Integration (SII), Nagoya, 2015, pp. 211-216.