Color Sensor Prototype: Detect Color Name from Objects

Pushbutton switch 12mm

RGB Diffused Common Anode

Standard LCD - 16x2 White on Blue

Breadboard (generic)

Resistor 100 ohm

Jumper wires (generic)

Photo resistor

Resistor 10k ohm

Arduino UNO

MLP Topology Workbench

<div class="se-component se-video-container __se__float-none" contenteditable="false" style="width: 100%;"><figure style="width: 100%; height: 56.25%; padding-bottom: 56.25%; margin: 0px;"><iframe frameborder="0" allowfullscreen="" src="https://www.youtube.com/embed/pc4IlCQ8TZM" data-proportion="true" data-percentage="100%,56.25%" data-size="100%,56.25%" data-align="none" style="width: 100%; height: 100%;" data-index="0" data-file-size="0" data-origin="100%,56.25%"></iframe><figcaption contenteditable="true"><div>See the color sensor working :)</div></figcaption></figure></div>This is a simplified version of the <a href="http://www.moretticb.com/blog/color-sensor-prototype-using-neural-networks/">full article</a>, for the sake of simplicity, so that the reproduction of this project can be done easily. Check also this project at <a href="http://www.moretticb.com/blog/color-sensor-prototype-using-neural-networks">my blog</a> for more implementation details.This article is divided in the sections listed on the right side menu. If you wanna jump to technical details, check the table of contents at the right, or download the code at the <a href="http://www.github.com/moretticb/ColorSensor">GitHub</a> repository. Before continuing your reading, check the project video to see what this project is really about :)<h3 >Project Overview</h3>Similar to color vision of the human eye, as well as based in light, the RGB model comprises more than 16 million colors, which are arranged in a 3D space, where integer values of components R (Red), G (Green) and B (Blue), ranging from 0 to 255, constitute coordinates of this space. From this model, color detection and recognition were performed with light-related electronic components and machine learning mechanisms; it is essentially the combination of an RGB LED and a CdS Cell (light sensor, or LDR), both isolated from ambient light. Such components, respectively, emit and sense the intensity of each light (red, green and blue) which was reflected from an object of a particular color.<h3 >Multi-Layer Perceptron</h3>Multi-Layer Perceptron (MLP) is a feedforward architecture of Artificial Neural Networks, having an input (non-neural) layer, hidden layers and an output layer. This network is trained by backpropagation algorithm, performing supervised learning (learning by examples).<figure><img src=https://projects.arduinocontent.cc/1d66995e-8013-4bfe-a591-8d2ca73b8675.png /><figcaption>Neural network for the color sensor</figcaption></figure>For this color sensor, <a href="http://www.moretticb.com/MTW/#3,6,10,s,s,e1e-7,r0.1,w2.75309|-11.47226|-3.31174|16.48123|19.50701|20.83178|7.11333|-6.42349|1.90722|6.49539|-27.71213|26.22820|-0.20637|-5.72456|-22.27807|30.06561|6.13926|-10.81428|28.51313|-9.78495|6.46702|0.05500|3.73036|4.14509|2.47902|0.01300|-3.58242|-16.36439|14.13336|-5.08929|1.63749|5.89483|1.41576|-3.31553|14.81429|-20.90657|-1.56866|1.91766|4.91018|4.03942|-10.84847|-5.64168|-4.13243|10.71144|3.75994|19.50770|17.72872|-3.21024|-2.47699|8.98845|5.19683|2.63604|17.35721|2.00543|11.71339|-5.45325|-6.94032|10.75201|0.66661|-7.26608|-3.58712|-9.92182|-12.68206|-15.45614|-13.74093|0.50826|15.17941|-11.14318|-19.08512|1.25124|22.00649|-4.22733|-0.44452|3.58902|0.64966|13.67560|-13.02688|-11.22907|-15.30070|-1.71819|6.73797|-28.17680|-2.50547|5.19797|7.00798|-2.86927|3.65035|18.02920|4.09836|10.48119|-2.56631|9.92777|2.34494|4.52433|">the neural network illustrated above</a> receives 3 inputs (RGB values), having one hidden layer with 6 neurons and an output layer with 10 neurons - just recalling: the output layer must have the same number of classes (colors, in this case), for a binarized output. For this tutorial, the network is already trained and able to recognize colors :). More details on training steps and the example dataset used for it, check out the full post in <a href="http://www.moretticb.com/blog/color-sensor-prototype-using-neural-networks/">my blog</a>.<h3 >Color Recognition</h3>This task can be performed using our trained MLP network. It allows classification and recognition of spatially separable patterns - very useful in this case. During its training step, the network mapped regions in the RGB color space illustrated below, so that every region isolated by hyperplanes represents a color. Therefore, every new RGB color pattern (represented as a 3D point) falls in a particular region, being classified as its respective color.<div id="gallery"><img id="gallery" src=https://projects.arduinocontent.cc/aa72f508-e07c-43a7-bc2f-36aa66f8b3de.png /><figcaption>Color space that a neural network shatters into regions</figcaption><img id="gallery" src=https://projects.arduinocontent.cc/ee1c01df-25c8-43be-a272-0fd88dd5053b.png /><figcaption> </figcaption></div>With the purpose of obtaining generalization with an MLP for a good recognition of RGB patterns, a training set (examples of colors with the desired output from the 3D space illustrated above) must be presented to the network for the training step. The training set used in this project is available at the project’s <a href="http://www.github.com/moretticb/ColorSensor">GitHub</a> repository. If you are keen on neural networks, keep reading this section and check out this <a href="http://www.moretticb.com/MTW/#3,6,10,s,s,e1e-7,r0.1,w2.75309|-11.47226|-3.31174|16.48123|19.50701|20.83178|7.11333|-6.42349|1.90722|6.49539|-27.71213|26.22820|-0.20637|-5.72456|-22.27807|30.06561|6.13926|-10.81428|28.51313|-9.78495|6.46702|0.05500|3.73036|4.14509|2.47902|0.01300|-3.58242|-16.36439|14.13336|-5.08929|1.63749|5.89483|1.41576|-3.31553|14.81429|-20.90657|-1.56866|1.91766|4.91018|4.03942|-10.84847|-5.64168|-4.13243|10.71144|3.75994|19.50770|17.72872|-3.21024|-2.47699|8.98845|5.19683|2.63604|17.35721|2.00543|11.71339|-5.45325|-6.94032|10.75201|0.66661|-7.26608|-3.58712|-9.92182|-12.68206|-15.45614|-13.74093|0.50826|15.17941|-11.14318|-19.08512|1.25124|22.00649|-4.22733|-0.44452|3.58902|0.64966|13.67560|-13.02688|-11.22907|-15.30070|-1.71819|6.73797|-28.17680|-2.50547|5.19797|7.00798|-2.86927|3.65035|18.02920|4.09836|10.48119|-2.56631|9.92777|2.34494|4.52433|">Playground</a> I developed, so you can use the dataset to train the network yourself and use it in the code :). Otherwise, skip to Implemetation details.Generalization will happen in the domain that the training set comprises, so it is worth to pay attention to min and max values of each component of the space! Do not feed the network with patterns outside this domain, otherwise the output is not expected to work correctly.<figure><img src=https://projects.arduinocontent.cc/93a00be7-afa1-4626-8adc-d7e10f3ef762.png /><figcaption>Dataset used for training</figcaption></figure>The dataset (all examples) contains 75 instances of color patterns ranging from 0 to 1. Initially ranging from 0 to 255, these instances were rescaled by simply dividing each value by 255, such that 0 &lt;= x1, x2, x3 &lt;= 1. As can be seen in the dataset, it is important to point out that only one neuron at the output layer must output 1, whereas the remaining ones must output zero. Owing to the fact that a trained network outputs float values, post-processing takes place as follows:<figure><img src=https://projects.arduinocontent.cc/e855dbfa-71c2-46c4-8353-eebf866e1077.png /><figcaption> </figcaption></figure> where yi is the output of the i-th neuron and max(y) is the is the largest output value. In practical terms, the neuron with the largest output gives 1 as output and the remaining ones give 0. Simple as that.<h3 >Electronic Circuit</h3>Arising from objects, all the detection procedure happens in the electronic circuit, encompassing computational activity running in an Atmega328, which is hooked up in Arduino boards. Check the wiring from the schematics below.<div id="gallery"><img id="gallery" src=https://projects.arduinocontent.cc/e96f3607-a792-485a-8042-a7d16dc2e40a.png /><figcaption>Schematics using a common anode RGB LED</figcaption><img id="gallery" src=https://projects.arduinocontent.cc/d668d667-c823-4708-83e8-2d1a2da95bae.png /><figcaption>Schematics using a common cathode RGB LED</figcaption></div>The code follows the scheme above that uses a common anode RGB LED. That said, check whether your RGB LED is also a common anode, otherwise just invert the logic in the code.Another important detail is that I am using only one resistor with the RGB LED. Since one color at a time will be lit, I put the resistor in the common anode, with an average resistance of the resistors that should have been with the cathodes - it is lazy, I know and I am sorry! When I went to buy project parts, they didn’t have everything I needed - but it is however very important to use the correct resistors with the cathodes in order to have fidelity in the collected RGB values with respect to RGB values in the computer. The way I did is not that bad, since the patterns are not distorted; they are just not the same colors we see in a computer screen (and as a human eye captures).It can be observed from the schematics the adjacency between the RGB LED and the CdS Cell. That is because they must be isolated from ambient light (an oldie black film tube is the perfect piece), so calibration (explained in <a href="http://www.moretticb.com/blog/color-sensor-prototype-using-neural-networks/#programming">Programming</a>) and recognition can be performed. Check the project video to see such components being isolated from ambient light.<h3 >Color Theory</h3>Color perception performed by the electronic circuit is based in color theory concepts. Since there are no lenses (yet) involved, only objects with opaque (and matte) material should be considered, avoiding to deal with specular reflection of the LED. Diffuse reflection on the other hand is the key to perform color detection with lights. From an incident light, it is reflected in irregular surfaces, not creating that glowish effect that ruins the function of the CdS Cell.Back to actual color theory, when light (of a certain color) reaches an object, it is reflected according to properties of that object’s color. For example, a red light reaching a yellow object will be reflected according to how much red exists in the composition of that yellow color - remember, we are talking about lights! - so it is expected to have a lot of red light being reflected, what makes sense when we think of the RGB composition of yellow (essentially red and green). However, when a blue light reaches the yellow object, no strong reflection is expected due to low presence of blue in the color composition.<figure><img src=https://projects.arduinocontent.cc/85124f88-0861-4308-acbf-a6e8255e03a9.png /><figcaption>Acquisition of RGB values for detection and calibration.</figcaption></figure>Considering an additive color system, in which white and black are respectively presence and absence of every colors (more details <a href="http://motion.kodak.com/KodakGCG/uploadedfiles/motion/US_plugins_acrobat_en_motion_education_colorTheory.pdf">here</a>), there can be measured (with the CdS Cell) maximum and minimum reflections of each light from the RGB LED which will reach colored objects. That said, it is possible to perform the calibration in electronic components involved in the circuit. This is another key to get fidelity in detection, as well as to ensure a stable detection of patterns (avoiding outliers) - here is a golden tip: after calibrating, try (hard!) not to move or touch neither the electronic components (specially when they are placed in a breadboard), nor the piece you are using (you must use) to isolate components from ambient light.<h3 >Programming</h3>For calibration and recognition, the color sensor executes three iterations, once a colored object is exposed to the RGB LED and the CdS Cell. In the first iteration, red light hits the object and the program waits CdS cell to stabilize its sensing; the analog input is then read and the reflection of the red light is stored. The program iterates twice more for green and blue colors. The figure shown in <a href="http://www.moretticb.com/blog/color-sensor-prototype-using-neural-networks/#color-theory">Color theory</a> gives a good visual explanation of this iterative process.Concerning calibration, the iterative process mentioned above is performed twice: once for black color and once for white color. As explained in <a href="http://www.moretticb.com/blog/color-sensor-prototype-using-neural-networks/#color-theory">Color theory</a>, this is for the detection of maximum and minimum - initially from near zero to near 1024, according to the reading resolution - reflections of red, green and blue lights, obtaining a true range to properly rescale to intervals [0, 255] (for informative purpose) and [0, 1] (the actual input to feed the neural network).The waiting time to establish reading of the light sensor can vary according to each electronic component, so it is good to give a good delay to ensure a steady sensing. In my case, I gave a 500-millisecond delay, but it is worth to initially use a bigger value and then decreasing it until the verge of a non steady behavior.In detection, the collected RGB values - ranging from 0 to 1 - feed an MLP, performing the actual color recognition. For the MLP running in Arduino, I am using <a href="http://www.moretticb.com/Neurona">Neurona</a> - a library I wrote to easily use ANNs in arduino, <a href="https://www.arduino.cc/en/Guide/Libraries">which can be installed from Arduino IDE with the Library Manager</a>. Check also <a href="http://www.moretticb.com/blog/neurona-neural-networks-for-arduino">this post</a> for more details.As mentioned earlier, the network is already trained and ready for detection in the project code. It was trained using the <a href="http://www.moretticb.com/MTW/#3,6,10,s,s,e1e-7,r0.1,w2.75309|-11.47226|-3.31174|16.48123|19.50701|20.83178|7.11333|-6.42349|1.90722|6.49539|-27.71213|26.22820|-0.20637|-5.72456|-22.27807|30.06561|6.13926|-10.81428|28.51313|-9.78495|6.46702|0.05500|3.73036|4.14509|2.47902|0.01300|-3.58242|-16.36439|14.13336|-5.08929|1.63749|5.89483|1.41576|-3.31553|14.81429|-20.90657|-1.56866|1.91766|4.91018|4.03942|-10.84847|-5.64168|-4.13243|10.71144|3.75994|19.50770|17.72872|-3.21024|-2.47699|8.98845|5.19683|2.63604|17.35721|2.00543|11.71339|-5.45325|-6.94032|10.75201|0.66661|-7.26608|-3.58712|-9.92182|-12.68206|-15.45614|-13.74093|0.50826|15.17941|-11.14318|-19.08512|1.25124|22.00649|-4.22733|-0.44452|3.58902|0.64966|13.67560|-13.02688|-11.22907|-15.30070|-1.71819|6.73797|-28.17680|-2.50547|5.19797|7.00798|-2.86927|3.65035|18.02920|4.09836|10.48119|-2.56631|9.92777|2.34494|4.52433|">Playground</a> I developed, embedding the adjusted weights to the code, so that Neurona library makes the network ready to operate.<h3 >Tests</h3>For informative purposes, some colors were extracted from the dataset to perform some recognition tests:<figure><img src=https://projects.arduinocontent.cc/f0f7adaf-828a-4e08-95ae-3240232fe166.png /><figcaption>Printed samples for testing the color sensor.</figcaption></figure>Numbers outside the figure are used for identification and numbers inside the figure indicate misclassifications, referencing which colors were classified instead. These colors were printed in sulfite paper with an inkjet printer - check the tiny paper squares in the video in the beginning of this post - so the objects consist of opaque material, proper for color detection.

Color sensor prototype

Color sensor schematics

Color Sensor Prototype: Detect Color Name from Objects

A sensor that recognizes the name of the color of an object based on light sensing and a neural network that learned from previous examples.