A Novel Concept of Electrical Stimulation of Touchscreens Used for Automated Verification of Mobile Devices

As the complexity and abundance of consumer electronic devices rises with the fast pace, efficient and reliable methods of their verification on the production line are extensively researched. This paper presents a novel concept of electrical stimulation of touchscreens which aims to fully automate the verification of devices with touchscreens on the final production line. Without mechanical moving parts, this approach is based on a stimulation board which constantly touches the entire screen and produces the required touches on the desired screen coordinates while being controlled by electric means only. Experiments were performed to choose the optimal material for the touch surfaces and determine the optimal size and separaton of the touch surfaces. Furthermore, the system for automated verification of touchscreen-based devices based on the proposed concept is described. The system supports single and multiple touches, line movements and multiple line movements. The system achieved more than 99 % reliability in producing controlled touches on screens up to a size of a tablet. DOI: 10.5755/j01.eie.25.2.23204


I. INTRODUCTION
In the recent years, fast progress of the consumer electronics industry has led to the saturation of production capacity for manual verification of devices [1].Complexity of the devices grows at a high rate and yet the verification systems for these devices must not lose pace and should ideally have a zero-tolerance for faulty verification result.No manual system, i.e. a system where humans are the part of, can have an errorless operation, which is why automated verification systems are increasingly researched to replace manual verification systems on the production lines.
One industry where the automated systems for verification on the final production line have already been implemented is the industry of television sets.Automated verification systems have a goal to optimize the speed, Manuscript received 12 December, 2017; accepted 7 August, 2018.This research was funded by a grant (No.TR32014) from the Ministry of Education, Science and Technological Development of the Republic of Serbia.
reliability and precision of performed tests by automating the verification parts of the final production line.For example, Rau [2] suggests the system for automated verification of digital television (TV) sets based on the measured image quality on the TV screen.In [3], Rama et al. present the flexible system for verification of digital TV sets which checks if the symbols on the messages appearing on the TV screen are correctly written.Recent decade has brought a massive expansion of consumer devices such as mobile phones and tablets which also require the verification step on their final production lines.Nowadays, these devices almost exclusively use the capacitive touchscreens for input and output [4].In order to design an automated system for verification of such devices, it is essential to design the automated way of stimulating the touchscreen in order to mimic the user input.
To the best of our knowledge, there have been little attempts to solve the problem of automated stimulation of touchscreens.Most of the research deals with the interface improvements, i.e. touch detection, human-device interaction, intelligence of the screen controllers and sensitivity improvements.Altinsoy and Merchel [5] developed an electro-tactile screen as an add-on device to the touchscreens in order to provide tactile feedback to the user.Kyung et al. [6] suggested the haptic interaction with the user using the compact add-on module for touchscreens.Nishino et al. [7] investigated additional interactions with the user to complement other senses, such as visual and auditory.Hafiz and Sawada [8] developed a tactile actuator that uses micro-vibrations generated from the memory wire.Review of industry solutions for automated systems for stimulation of touchscreen revealed that the mechanical approach is dominant.Solutions such as [9] use mechanical approach to stimulate the touchscreen, by using artificial fingers which are moved to the desired coordinates on the screen and lowered to touch the screen.While a mechanical approach is useful and provides one way of automating input generation for touchscreen-based devices, an electrical solution without mechanical parts would be more reliable and significantly improve on the size, complexity, price and speed of such devices.To the best of our knowledge, there have been no papers addressing this problem besides the preliminary work by the authors of this paper [10]- [12].This paper presents an approach for electrical stimulation of touchscreens.Unlike the mechanical solutions where the stimulation is based on moving the artificial finger, this paper proposes the stimulation board which constantly touches the screen and controllably distorts the electric field around the touchscreen at the desired coordinates without producing a touch on the rest of the screen.The distortion of the electric field is achieved by connecting the board to the long conductor which assumes the role of the finger.No mechanical parts are required for the system to operate and once placed, the board does not move from the touchscreen while stimulating the user moves on it.
The rest of the paper is organized as follows: Section II describes the model of touch used to design the concept of electrically stimulated touch.Section III describes the concept of electrical stimulation of touchscreens in more detail.In Section IV, the concept is applied to implement a stimulation board as a part of the system for automated verification of devices with touchscreens.In Section V, experimental results with the stimulation board are presented.Finally, Section VI gives conclusions.

II. THE MODEL OF TOUCH
Since the days of the first touchscreen designed by E.A. Johnson [13]- [14], touchscreens have been developed in multiple directions, with two becoming the most widely accepted: resistive and capacitive touchscreens.Each detects a change in a different electric quantity and the name suggests which one.Resistive touchscreens have recently lost the battle and capacitive technology dominates the industry of consumer electronic devices, namely mobile phones and tablets.This section will focus on designing a model of touch for capacitive touchscreens.
Capacitive touchscreens know the position of the user's finger by detecting the difference in the measured capacitance in the region of the touch.The finger acts as a conductor which disturbs the internal structure of the electric field on the screen, leading to a change in the measured capacitance across the screen.Change which is higher than the threshold value signalizes the controller of the screen that the touch occured at the coordinates of the detected disturbance.In order to be able to produce the effect of the touch, a conductive object, such as a user's finger, needs to touch the screen.
When the user touches the screen, their finger introduces the electric field disturbance which can be modelled as the grounded conductor introduced inside a capacitor.The way the conductor "shortens" the electric field lines in the model is equivalent to the way the finger "shortens" the electric field lines between rows and columns on the touchscreen.
The model in Fig. 1 shows introduction of the conductor of length x into the capacitor of permittivity ε, length a, width b and the distance between the capacitor plates d.The conductor is grounded and as such it assumes the role of one of the capacitor plates.Without loss of generality, we can assume that the introduced conductor acts as a negative plate, leaving the area to the right of the conductor without the electric field.The same result would be achieved if we assumed it to act as the positive plate.New capacitance of the system in Fig. 1 can be calculated from the equivalent electric model given in Fig. 2. The area with the introduced conductor has the capacitance 0 because it contains the serial connection of two capacitors, C l and C r, of which one has zero capacitance.The final capacitance C is therefore equal to the capacitance C new of the area A new without the introduced conductor, which is lower than the capacitance C old before introducing the conductor, since the area A new is smaller that the area of the capacitor without the introduced conductor, as calculated in ( 1)-( 2): Therefore, the decrease in the capacitance resulting from the finger touching the screen can successfully be modelled by introduction of the grounded conductor in the electric field of a capacitor.
The key property of the conductor in Fig. 1 is the fact that it is grounded.If the conductor were not grounded, the small conductor inside the capacitor would not change the capacitance as the negative charge on it would be concentrated on the left side and positive charge would be concentrated on the right side.The total capacitance of the entire system would remain the same because the electric field would remain in the entire volume of the capacitor, unchanged.Grounding the conductor forces it to be charged only one-way, as the opposite charges leave the capacitor region due to repelling forces between charges.

III. ELECTRICAL STIMULATION OF TOUCHSCREENS
Initial idea for electrical stimulation of touchscreens was based on mimicking the orientation of sensors on the screen and stimulating the screen across rows and columns, in the same configuration as the sensors.The first trial of controlled electrical stimulation of touchscreens involved two layers of orthogonal conductors (Fig. 3) [10].In this idea, touch would happen at the intersection of the activated row and column.While conceptually feasible, this approach did not lead to the acceptable solution due to various problems.Even though very thin conductive lines were used in the experiment (lines with thickness less than 1 mm), they were long and thick enough to stimulate the touchscreen uncontrollably over the entire area of the conductive surface.After initial failed attempts to stimulate the touchscreen by mimicking the sensor lines, the two-dimensional approach was changed to a point-approach.Namely, conductive surface of small length and width was chosen for the following experiments.The model described in Section II was used to lead to the following conclusion:  the screen will not be stimulated if approached by an insulator or a very small conductor which is not grounded,  the screen will be stimulated if approached by a grounded conductor.Hence, the stimulation of the touchscreen can be achieved by introducing a grounded conductor to the screen, touching it at the desired location where a touch wants to be achieved.Still, if we want to achieve fully electric stimulation, without the moving mechanical parts, it is unacceptable to base the system on moving the conductor towards and away from the screen.We need to achieve controllable stimulation without moving the conductor.
The key property of the model from Section II which allows controllability is the difference between conductors of different lengths.As listed previously, and confirmed experimentally, a very small conductor touching the screen will not produce a touch.The conductor needs to be grounded, i.e. very long, to produce the touch.Therefore, controllability can be achieved by changing the length of the conductor touching the screen, e.g. by switching the conductor's connection to the ground on or off.This concept is presented in Fig. 4. It has been experimentally confirmed by designing a conductive surface connected to a short conductor and a conductive surface connected to a long conductor.The former did not produce a touch and the latter produced it.Furthermore, it was experimentally confirmed by introducing a mechanical switch between the surface with the short conductor and the grounded conductor.When the switch was turned off, no touch was produced.When the switch was turned on, touch was produced.It is worth noting that the "grounded conductor" does not have to be actually grounded; a long conductor is good enough to act as the grounded conductor.
In order to make a controllable touch using the concept from Fig. 4, it was necessary to use the switching element which will be as close to the ideal switch as possible.Such a switch needed to have a very high resistance when turned off and a very low parasitic capacitance.These requirements are important in order not to produce a touch due to the electric properties of the switch.The critical state is the OFF state because in order to produce the touch, it is enough to be connected to the grounded conductor by any means.The resistance in the ON state does not have to be low to produce the low, but the critical requirement is that the resistance needs to be extremely high and the parasitic capacitance extremely low in the OFF state in order to effectively disconnect the touch surface from the grounded conductor.
Experiments were performed using the following elements in place of the switch:  Metal Oxide Semiconductor Field Effect Transistor (MOSFET);  Bipolar NPN transistor;  Electromechanical relay.
The first experiments were made using the MOSFET as the switching element, due to its good switching capabilities, wide usage in digital circuits and easiness in production of the printed circuit boards with MOSFETs.The second set of experiments was made using the bipolar NPN transistor.The conclusion of the first two sets of experiments was that semiconductor components were not controllable enough to be used as a switching device in the system with the concept presented in Fig. 4 due to their high parasitic capacitances.
The final set of experiments was performed using the electromechanical relays.Even though the goal was to reduce mechanical components of the system to a minimum, relay was chosen as an acceptable option because the mechanical part which is making the switch is completely electrically controllable and fast-enough not to introduce delays in the operation of the system.
The material for the touch surface is a critical point.The surface needs to ideally touch the screen on its entire area and yet be of material which will not damage the screen.For that reason, the thin metal surfaces were used in the system prototype.Experiments were performed with different surfaces and more details about the experiments, i.e. the surface area requirements for the reliable touch are given in Section V.

IV. STIMULATION BOARD AND SYSTEM FOR AUTOMATED VERIFICATION OF DEVICES WITH TOUCHSCREENS
After all the mentioned experiments, prototype stimulation board was designed to fit the devices with the maximum screen size of up to 10.1 inch, covering most of the currently available consumer devices: mobile phones and tablets.The touch surface contains 464 touch areas, each with the 7 mm × 7 mm surface area, distanced 1 mm in between.The touch areas are organized in 16 rows and 29 columns.The resolution of the touch is therefore 8 mm which is comparable to the width of the user's finger; an acceptable resolution for building the prototype.
The total surface area touching the screen in this prototype is shown in Fig. 5.This surface touches the screen the entire time stimulation is performed, while the switching elements connected to each surface control whether that surface will produce a touch at that point, or not.
During the first trials with the stimulation board, an interesting behavior was detected: if the two consecutive surfaces are stimulated, the screen does not detect the multitouch as the points are too close to each other.Instead, a single touch is detected at the point which is between the two stimulated surfaces.The reason for this is rooted in the behavior of the touchscreen controllerif the capacitance change is detected in nearby points, they are averaged to a single point.Otherwise, each touch would be detected as multiple touches because a single touch by a finger disturbes the electric field across a region of the finger, not at one point.Multitouch is detected only if the distance between the detected single touches is larger than certain threshold.
In the proposed prototype, two consecutive surfaces are close enough to produce a single touch in between, while two surfaces separated by at least one surface are far enough to produce multitouch.This goes well into our advantagewe can double the resolution in each dimension and produce touches at the points between the surfaces, by stimulating both consecutive surfaces.This means that the distance between touchable points is 4 mm instead of 8 mm in both dimensions and the total number of touchable points is almost quadruple the number of surfaces.Control of the stimulation board was achieved using the separate control board which architecture is presented in Fig. 6.State of the touch surfaces on the stimulation board is memorized in the series of shift registers which are filled via the complex programmable logic device (CPLD).Communication between the control software running on the personal computer (PC) and CPLD is achieved via Universal Serial Bus (USB) interface and Joint Test Action Group (JTAG) interface, using the custom-made integrated circuit in between.Shift registers are connected in a serial chain, receiving one bit at a time from CPLD.One write operation to shift registers defines states for all 464 relays on the stimulation board.After the shift registers receive all 464 bits, they are enabled and they send their state to the stimulation board.This enabling of the shift register output at the end of communication is essential in order to prevent junk touches during the reception of 464 bits of relay states.The control board is easily extendable to the larger number of touch surfaces, by simply adding more shift registers and re-configuring the CPLD.At the time of writing this paper, the control board and the control software running on PC support the following types of touchscreen stimulation:  single touch (press) at one point in the coordinate system of the stimulation board (one touch surface or two consecutive surfaces),  single release of the point which is pressed,  multi-touch (multi-press) at arbitrary number of points on the screen, provided they are at least 2 touch surfaces apart,  multi-release of the pressed points,  global release of all points, i.e. setting all relays to 0;  line movement,  multi-line movement, i.e. drawing multiple lines simultaneously, which is useful for producing zooming moves on the touchscreen.Stimulation board can be applied to design a system for automated verification of devices with touchscreens based on the similar concept as the previous work of the authors on implementation of the system for automated verification of television sets [15], [16].The concept of the system is given in Fig. 7. Control software on PC sends to the stimulation board the user movement it wants to reproduce.Stimulation board produces the touch and the device responds to the user input.Screen of the device is captured and sent back to PC for processing and verification of device's response to the user input.This can be used for verification of devices with touchscreen input on the final production line.As explained in [1], automated verification has advantages over manual verification and the proposed system achieves full automation of the verification process without mechanical moving components for producing user input on the touchscreen.

V. EXPERIMENTAL RESULTS
Concepts of electrical stimulation of touchscreens presented in this paper were tested on mobile phones and tablets of different screen sizes.The maximum screen size was 10.1 inch, for which the prototype system was implemented.Such a system can be used for verification of smaller devices, by using a subset of touch surfaces.
The first experiments aimed at testing different materials for touch surfaces.Materials in the experiment were: thin metal surfaces, conductive sponges and conductive rubber.The most stable were thin metal surfaces, but they have the lowest flexibility.On the other hand, conductive sponges and rubber were more flexible and more gentle to the screen, but they did not produce touches of significant reliability due to their lower density.The final choice for the prototype were thin metal layers.
After choosing the touch surface, the optimum surface area was seeked.The minimum surface required to produce the touch was discovered empirically.When the finger touches the screen, the touch surface is on the order of 50 mm 2 .In order to find out the minimum required surface, collection of touch surfaces was made such that the length and width of each surface ranged from 1 mm to 10 mm.Experiments showed that the minimum surface area which can produce a touch using the concept from Fig. 4 is 4 mm × 4 mm, but this surface area failed to produce the touch in some trials.The minimum surface area which produced the fully reliable touch, i.e. with no false negatives, was 7 mm × 7 mm.The surface area needs to be minimum possible in order to maximize the resolution of points on which touches can be produced.On the other hand, the surface area needs to be large enough to reliably produce the touch, because the small surface areas do not produce enough disturbance of the electric field.The material used in this experiment was a solid metal conductor.
The length of the conductive line was noticed to be very important because if the large conductor is brought to the vicinity of the screen, it will disturb the electric field itself and produce touches across the screen.Therefore, the length of the conductor between the touch surface and the switching element must be very short.In addition, the rest of the conductive elements, i.e. the bulk of the long grounded conductor, needs to be distanced from the screen in order not to disturb the touching.
Experiments with different switching elements were performed in order to find the switching component which produces the lowest number of false touches (false positives) and false non-touches (false negatives).Most switches had no problem switching the touch surface ON, but a few were successful in switching the circuit OFF and not producing a touch when it is not wanted, due to their internal electric properties, mostly high parasitic capacitance.
In order to test reliability of a controlled touch, the benchmark application which measures the frequency of the touch was used.This benchmark reports a frequency of ~60 Hz when the screen is constantly touched by a finger.The experiment was performed with a single touch surface constantly touching the screen, switched on or off from a long grounded conductor using one of the switch elements.Table I-Table III show benchmark results for MOSFET, bipolar transistor and electromechanical relay, respectively.
It can be seen from Table I that there is a significant difference between ON and OFF states of the MOSFET, but the OFF state is far from reliable, as it produces considerable number of false touches.This frequency must be 0 in an acceptable solution because the surface must never produce the touch on the screen if it is disconnected from the grounded conductor.Similarly, Table II shows that bipolar transistor is not an acceptable solution, but it is more reliable than MOSFET due to the lower frequency of touch in the OFF state and higher frequency of touch in the ON state.From Table III it can be seen that electromechanical relay is acceptable due to no touches produced when it is OFF and significant amount of touches, comparable to the touch by a finger, measured in the ON state.For that reason, relay was chosen as the only candidate for further experiments.
After experimentally confirming the feasibility of producing the electrically controlled touch on the screen, stimulation board was produced using thin metal layers as conductive surfaces and electromechanical relays as the switching elements.The prototype system made for final experimental verification is shown in Fig. 8. Touch surfaces are lined up on the upper part of the device, which is lowered to touch the screen of the mobile phone or tablet before running the tests.Operation of the stimulation board was verified using the drawing application shown in Fig. 5. Prototype was incrementally improved as it was difficult to overcome problems which emerge when using multiple surfaces for touching the same device.Lining up all the surfaces in such a way that they all touch the screen, do not touch each other and have perfect shape to be identical and perfectly flat to touch the screen with their entire surface, were just some of the challenges needed to be overcome for the final version of the prototype.For example, faulty surfaces were detected using the mentioned drawing application.In Fig. 5, result of one experiment is shown in which about 90 % of the surfaces worked and the other 10 % needed to be checked.The simple test which involved touching all points on the screen one by one was used for verifying the operation of the stimulation board.At the time of writing this paper, reliability of touches was over 99 % with only few cases without produced touch when needed, which is acceptable as the proof of concept of the proposed stimulation method.

VI. CONCLUSIONS
This paper described a concept, model and system based on electrical stimulation of the touchscreen in which the stimulation board imitates the user's touch on the screen in an electrically controllable way without any mechanical parts moving above the screen.The proposed concept and model were experimentally confirmed.The minimum surface area generating a single touch is 7 mm × 7 mm.The proposed system can stimulate screens up to 10.1 inch.The system does not produce false positives and produces the touch with more than 99 % reliability.It can be used as part of the system for verification of touchscreen-based devices.
Besides producing the system for verification already envisioned in Fig. 7, future work will consist of searching for better solutions for touch surfaces.As mentioned in the previous section, reliability of the presented solution is very high, but there is still room for improvement which may be brought by using different materials for touch surfaces.More detailed research is needed to seek for or design custom material which will conquer the final percentage of the reliability, which is always hardest to conquer, and give a 100 % reliable solution.As the devices themselves evolve and new ways of interacting with the user are researched [17], the verification system will to follow along with the evolution and support other types of user input, which is an entirely new challenge for the future.

Fig. 1 .
Fig. 1.Capacitor model of introducing the finger, represented as a conductor of length x, in the touchscreen's electric field E. Capacitor has length a, width b and the distance between the plates d.

Fig. 2 .
Fig. 2. Electric model of the capacitor with introduced grounded conductor.C new is the capacitance of the part above the introduced conductor; C l is the capacitance to the left of it and C r the capacitance to the right of it.
Fig. 3. Conductive lines used in the initial trials for electrical stimulation of touchscreens.

Fig. 4 .
Fig. 4. The concept of controllable stimulation of the touchscreen (left) and the implementation using the electromechanical relay as a switching element (right).

Fig. 5 .
Conductive touch surfaces on the prototype stimulation board (a), and the results of the touching test in which a point represents the successful touch (b).

Fig. 7 .
Fig. 7. Concept of the system for automated verification of devices with touchscreens.

Fig. 8 .
Fig. 8. Prototype of the testing device with stimulation board.

TABLE I .
TOUCH BENCHMARK USING MOSFET AS A SWITCH.