The second meeting of the working group developing “Body A” for the Avatar project took place on July 1, 2011. The participants were: Mr. D.I. Itskov, Mr. T.N. Shchukin, Mr. A.A. Frolov, Mr. V.L. Dunin-Barkovsky, Mr. V.G. Yakhno, Mr. V.A. Konyshev, Mr. L.A. Stankevich, Mr. V.D. Tsukerman, Mr. I.L. Kirilyuk, and Mr. V.V. Petrushenko.

At the start of the meeting, Mr. D.I. Itskov gave an account of how the "2045" Initiative is developing. He remarked that actor Steven Seagal will be visiting Moscow over the period of July 3-10 and will be taking part in a variety of events, including the shooting of a documentary about the Movement (at the current time, all the events planned have taken place).

Movement experts prepared an outline of plans for the creation of a specialized center or network of centers able to bring together the achievements of the Russia 2045 project. It was decided to send this letter to the prime minister of the Russian Federation, Vladimir Putin.

Mr. D.I. Itskov related the latest news, told about his recent trip to the U.S., about meetings with potential investors and with businessmen, actors, and other interested parties on the highest levels. It was decided not to include these people’s names in the report until they officially join the Movement.

Initiative coordinator Mr. T.N. Shchukin gave remarks on plans for the Movement to publish its own scientific journal, in which there will be articles by experts on robotics, cognitive science, modeling of living systems, mathematics, biology, anthropology, and philosophy.

Mr. T.N. Shchukin also spoke about the fact that the Movement’s new Internet portal, which is expected to launch at the end of July 2011, will, in addition to a social network, forums, a blog platform and a news section, also in the future have wiki-like information pages and an integrated “Virtual Laboratory”.

The Virtual Laboratory will allow scientific groups, associated labs, institutes, and simply participants in the Movement to take part in the experimental work of the Movement—in online experiments, in the creation and handling of databases—and to play games. Users will also be able to create individual digital avatars by building a personal profile based on psychodiagnostic data, individual EEG results, voice modeling, and on the results of the processing of photographs and texts. Diagnostics will be able to be undergone by way of connected serial neurointerfaces, and expertise and best practices will be able to be shared. In addition, the coordinator of the Movement remarked that in fall, a large-scale program of contests for scientific and manufacturing projects will begin with substantial monetary awards and the opportunity for the best projects to be included in the system of organization and financing of the Movement’s main tech projects.

Mr. T.N. Shchukin also spoke to the gathered experts about the start of work on “Body B”, about meetings with the head of the Scientific Research Institute of Gerontology, Mr. V.N. Shabalin, who led these efforts, and with Ms. Y.V. Tereshina, who is coordinating them from the institute’s side. Among the items discussed were such central areas of work as:

a)      the creation of a computer model of brain metabolism;

b)      the creation of perfusion liquid based on these models;

c)      the creation of a system to maintain the liquid’s composition (oxygenation, purification, etc.);

d)      the creation of a system of humoral and nerve brain stimulation, as well as of a system of communication between the brain and the outside environment.

Mr. Dunin-Barkovsky spoke about the start of work on “Body C”, in connection with which an international training seminar was held in the city of Sudak on computing neuroresearch. The seminar resulted in the Body C development group gaining its first workers, the first grants being distributed, and the first results received.

The following main areas of work on “Body C” in the realm of reverse engineering of the brain were set out:

A. By elements of nervous tissue structure—levels: molecular, membrane (ion channels, pumps, transmembrane transport systems), perisynaptic, neural, multi-neuron, and neuroglial.

B. By brain function hierarchy—levels: vegetative functions, circadian rhythms and sleep, neural mechanisms for movement and speech, mechanisms for perceiving sensory signals, mechanisms for feelings and emotions, neural mechanisms for language, thought and decision-making.

C. By anatomical hierarchy—levels: executive neurons, sensory receptors of all types, peripheral nervous system, spinal cord, brain stem, hypothalamus, thalamus, cerebellum, basal ganglia, hippocampus, and the cerebral cortex.

D. By transmissive, neuromodulatory and neurohumoral characteristics—the functions of every type of neurotransmitter and neuromodulator; mechanisms maintaining the timing and spatial dynamics of their transmission.

E. Issues of ontogeny and phylogeny of brain structure and function. Neurogenetics. Regulation of genome expression in brain cells.

F. Normal state and pathology: Everything previous must be examined both in the conditions of normal functioning and under the conditions of diseases and injuries.

G. Technical tools for neuroinformatics. Technical tools for receiving and submitting source data and results of modeling—computers (universal or specialized) and software for analyzing data and modeling neural systems. Neurochips of all types.

H. Systems of “synthetic” artificial intelligence. Monitoring of main trends and results of work on creation of systems of artificial intelligence not based on working mechanisms of living neural systems.

I. Systems of reproduction of psychic processes similar to those of humans. Discussion of methods and means of capturing individuality and its reproduction on artificial platforms.

Mr. T.N. Shchukin went on to propose to those present that they focus on the “Avatar-Body A” project and that they discuss the finalized distribution of responsibility and the composition of working groups for the project’s central tasks. The experts present at the meeting were given tables with the central tasks for unit A of the “Avatar” body for their confirmation. It was proposed to the experts that they complete work on drawing up approximate budgets for the sub-projects and working groups by the end of July.

The general action plan for every area of work makes the assumption that, if necessary, the most advanced technological models available for each area will be purchased—robots, biological prosthetic wrists, separate elements, models of electroactive polymers, and so on. The development of models in each area will be accompanied by optimization of work on a virtual nodal model by both the working groups and by the participants of the Movement using the Virtual Laboratory.

Also discussed was the parallel creation of prostheses of exoskeleton orthoses. A decision was made to focus on prostheses in the first stage and only later gradually move into the application of results for the creation of orthoses. There was a discussion of the necessity of returning to a MindMap of the project, away from tables, in order to simplify its integration and operation. It was decided to move the issue concerning the mathematical basis for the operational algorithms (neural networks, algorithms on the basis of analytical and imitational modeling, and so on) to discussion in the separate project groups.

After the introductory portion, it was proposed that group discussion be taken up, as a result of which the main areas of work were to be confirmed, the main stages of work on “Body A” to be defined, and those responsible for each stage determined.

The discussion resulted in the following definition of stages for the project and the following structure of working groups:

First Area: Hand

First Stage

Hand up to the elbow, providing for grip

It is expected in this area that the production of a working model of a hand with 15-20 degrees of freedom will be complete by the end of 2011.

The first master model, produced with the assistance of Mr. M. Y. Yablokov, is already complete, while the finalized model that takes into account certain adjustments should be complete toward the beginning of August. Models of electropolymer muscles have been ordered from Swiss manufacturers, and agreements have been made to create more effective propulsion devices on the basis of Russian technologies unknown in the West (including vacuum evaporation of electrode film on an elastic polymer). Mr. V.A. Konyshev has proposed using pneumatic muscles as provisional propulsion devices for the creation of working algorithms, and an e-skin covering based on an electroactive polymer to provide for base-level sensing. At the first stage, the hand will be controlled using a computer model and an EMG signal. Mr. V.A. Konyshev proposed utilizing the groundwork already done at the company Neurobotics on the registering of high-density electromyograms. The first interface will be an electromyographic interface, for the creation of which it will be necessary to isolate patterns from the recorded signal that correspond to different wrist movements. These data can be used to control the hand prosthesis using both the other hand and the registering of EMGs from the hand stump.

The working group for the first stage will include Mr. M.Y. Yablokov, Mr. A.A. Frolov, and Mr. V.A. Konyshev.

Second Stage

Hand to shoulder inclusive, providing for movement with a large range of degrees of freedom, hand-eye coordination, and autonomous execution of commands such as “pick up an object” with automatic formation of the structure of movement.

In contrast to the first stage, an entire range of new tasks are present:

1. Operation by a control point (the wrist)

2.Eye-motor coordination based on an evaluation of the environment using machine vision and kinetic-like radars.

3. Creation of individual movement based on an evaluation of the outside environment and proprioception data (the weight of parts of the hand, inertia, accounting for the weight of an object in the wrist, etc.).

4. Generation of coordinated movements of two hands.

Due to the accumulation of hand sense stimuli and the addition of the tasks of adapting a kinematic model of the hand and of creating machine vision and object and situation recognition, Mr. V.G. Yakhno, Mr. V.D. Tsukerman, and Mr. L.A. Stankevich will be joining the Hand project at this stage.

Second Area: Body

First Stage

A computer and a physical model of the body that provides for maintenance of balance while standing on a moving platform. It was proposed that the documentation for a similar project by a German colleague of Mr. A.A. Frolov, a Mr. Merner, be acquired. The possibility was discussed of using a human skeleton with artificially eliminated degrees of freedom in the spinal cord as a model for the construction. These degrees of freedom would gradually be added to the model and to the physical construction over the course of the model’s development.

Second Stage

Depending on the level of the model’s development, it is expected that the abilities to walk, run, and jump will be added to it. Depending on the results of the Hand project, the addition of hands and a head to the construction are planned, with a subsequent recalculation (retraining) of the model taking into account the new elements.

The working group for the project includes: Mr. A.A. Frolov as the team lead, Mr. V.L. Dunin-Barkovsky as a consultant in modeling the cerebellum function, as well as Mr. L.A. Stankevich as a robotics specialist. The group requires an engineering department and a programming department.

Third Area: Head (Face)

To provide for a human-like nature, the creation of an artificial face able to represent human facial expressions is required. Over the course of the discussion, the examined possibility of acquiring a Geminoid was discarded in favor of creating our own system on electroactive polymers (artificial muscles). It was proposed that a catalogue of expressions be adapted to create a catalogue of basic emotions for the artificial face using a basic range of expressive muscles. A gradual expansion of the catalogue will allow for the inclusion of human-like voice capability at the final stage.

Working group: Mr. V.A. Konyshev and Mr. M.Y. Yablokov.

Fourth Area: Control of the Avatar

Sub-Area 1: Invasive Interfaces

The work area related to the creation of invasive neurointerfaces was assigned to Mr. А.А. Frolov and Mr. V.A. Konyshev. The following main tasks are expected to be solved in this area:

? the development of a procedure for using an invasive interface;

? the use of an interface for recognizing speech signals on animals (for example, the sub-project “Speaking Ape”, which provides apes in childhood with an artificial vocalizer based on an invasive EEG interface, which they will be able to learn how to use in a similar way to sign language, the ability to use which is already possessed by other populations of experimental animals);

? the search for and development of new little-known means of registering: carbon nanotubes, mobile sensors, “sprouting” sensors, etc.

The decision was also made to pursue collaboration with mathematicians specializing in systems of pattern recognition with the goal of recruiting them to join the project to address the task of finding patterns in biological signals that correspond to movements of the hands and body, vocalization, and other similar tasks, as well as to pursue collaboration in the creation of adapted self-learning body control systems.

Sub-Area 2: Non-Invasive Interfaces

The participants in the meeting came to the opinion that the team lead for this area should be someone missing from the day’s proceedings, Mr. A.Y. Kaplan. Mr. T.N. Shchukin proposed discussing a range of what in his view are key research themes that must be investigated within this area in order to create a universal high-speed interface “computer-brain” (“robot-brain”) capable of being controlled by manipulators and arbitrarily shaped bodies.

There are two fundamentally different ways to create a “computer-brain” interface. The first is to look for signal indices (EEG (including EP components and other derivatives), Magnetoencephalography, etc.) that match the movements. Control of movement in the future will be done using the identified parameters or their patterns. The second is training a person to control (biofeedback or operant conditioning) by the chosen parameter that is (are) connected with the movements of a manipulator. There is an entire range of difficulties connected with the second area that limit its use.

a)      Inertia of indices. The essence of the problem lies in the fact that any steadily repeating phenomenon (index) in an EEG assumedly has a function in the system of work in the brain. The use of such a function “not for its intended purpose” can, first of all, worsen the execution of the primary function and, second of all, the training process itself is connected to competition between the new and old system—that is, “opposition” or “inertia” of the existing functional system to the new use.

Hypotheses:

1. The strength of “inertia” is proportional to the “size” of the phenomenon. The more subsystems that enter into the structure of the phenomenon (the index), the greater the strength of inertia. An optimal system may be some kind of autonomous unit (a separate cell, an ensemble, etc.).

Various readings of the measured signal according to these parameters have to be evaluated and an array of indices with the lowest “inertia” must be described.

2. The strength of “inertia” is proportional to the stability of the connections between this phenomenon and others. The more connections there are between this index and the surrounding systems, the greater the strength of inertia. For example, it’s likely that the signal readings connected to such activity as cognition will have less inertia than readings accompanying movement or perception. Various readings of the measured signal according to these parameters have to be evaluated and an array of indices with the lowest “inertia” must be described.

3. The strength of “inertia” is proportional to the change required for control. The greater the required deviation from typical functioning, the greater the inertia. The obvious method—summation of small deviations of various systems as a team—is the imposition of a pattern. The possibility of using small deviations as a team must be evaluated.

b)      The problem of forming a new skill. In addition to the search for appropriate indices for their use as elements of a team or as teamed-up effects, it is possible to create new unique indices specially for a task. The question arises of how to create this index. It’s obvious that a system of incentives and punishment likely will not work optimally in this case, since it will be difficult for the learner to establish a connection between the stimulus and the action required of him. What is needed is something like Galperin’s systematic formation of mental actions or a systematic formation of a new communication structure (similar to establishment of contact with a blind-deaf-mute).

Hypotheses:

1. The systematic formation approach is effective.

It is proposed to index the existing phenomena in the signal and mark out the “reserve zone”—that is, the large number of physiologically possible but not seen indices. A space must be chosen that is of a dimensionality corresponding to the dimensionality of the space of the orders (see hypothesis 2). After that, a system of systematic formation of actions (from the shared to the private, and/or independent training of separate elements and their subsequent assembly together) must be developed. The possibilities for systematic formation of complex self-regulating actions must be studied, as must the particularities of automation of actions’ sub-elements during the mastery of higher-level operations.

2. The space of orders is a new language, and the training of such should be similar to the creation and studying of a new language. It is proposed that a space of operational influences and EEG indices that correspond to the orders space (and, thus, to the structure of the activity) be created. A semantics of the space must be created in accordance with the space of the required orders. It is possible that a shared space should be arranged in accordance with the structure of the most stereotypical actions using, for instance, the hands. Separate actions, such as volitional movement with a finger, are to be created later than shared actions, taking into account a determination of unnecessary degrees of freedom.

3. Feedback should correspond to the structure of the control space. In the case of simple functions, control is achieved through output into the interface of feedback of the control function dynamic. The display into the interface is typically linear. In the case of work with patterns, such a system is absolutely non-functional. When control is undertaken with a single pattern, it is possible to create a system that puts out as feedback the level of similarity of the current EEG on the specified pattern. However, when there are more than one or two patterns, the feedback should reflect the closeness of the current position to multiple patterns simultaneously, i.e. it should correspond to the movements of the reference positions within the space. A feedback interface must be thought through and created that is able to display this space and this movement within it selectively, taking into account the goals of the activity. The hypothesis is that there is no need to display excess patterns if they are not “key pivot points” of the space.

4. Feedback will be more effective if multiple modalities are used in it simultaneously and if it includes near-threshold elements and sensory feedback elements from the action (signals from the hand sensors, for example). It is proposed that the possibilities of integrated feedback and of a combination of diverse modalities be studied.

5. The ability of an interface to adapt significantly increases its effectiveness. This refers to the fact that recognized or engineered patterns can change, just as can the entire space of positions as a whole—it can “drift” due to the influence of unaccounted-for factors. The pattern recognition system must be retrained and the patterns recalibrated in real time, based on the effectiveness of the action. For that, the pattern search mechanism must be supplemented by adaptive algorithms.

Fifth Area: Sensing of the Avatar

It was decided that responsibility for development of the sensory systems can be divided among the experts. Mr. V.G. Yakhno and Mr. V.A. Konyshev will take on the tasks involved in creating machine vision. Mr. M.Y. Yablokov will be responsible for the artificial nose and tongue, Mr. A.A. Frolov for the proprioception sensors, and Mr. M.Y. Yablokov for the sensing skin.

There is a range of tasks related to creating feedback—the receiving of sensory information by a person and the creation of a feeling in that person of presence in an external body on the basis of that information.

It was decided it would be reasonable to entrust Mr. A.Y. Kaplan with leadership in solving this task.

In addition to direct “sensory system-nervous system” interfaces, it is possible to create alternative sensory input points.

Hypotheses:

1. Sensory information from the hand (or the robot) should be delivered to a modality with the same dimensionality as that of the perception modality. Ideally, it will be delivered to the same modality. For that reason signals from the hand sensors can be projected on the skin of the hand in the form of a field of tactile or electro-stimulation. And the image from the cameras can be projected on the skin or the tongue using tactile or electro-stimulation. It was decided to examine the tasks involved in direct stimulation of the brain in order to create a virtual perceptual image separately.

2. Sensory information from the hand (or the robot) needn’t be delivered in the same dimensionality—it is possible to create a language of display of arbitrary dimensionality. The creation of a language of perception corresponds fully to the logic of the creation of a space for feedback corresponding to the team space. The feedback interface for controlling the robot can be partly or fully integrated with the system that delivers sensory information to the person from the robot as a whole.

As a result of the meeting it was also decided that by the end of July, a plan of action is to be presented for each of the discussed areas of work, including stages, deadlines, and resource demands—personnel resources, financial resources, organizational, informational, and infrastructure resources.

In addition, the decision was made to launch a series of monthly seminars in Moscow, each of which will be devoted to a separate topic related to work on Body A. Preliminarily, the proposed date for the first seminar is August 4, 2011. The preliminary topic is “Hand: Stage 1”. It was proposed that participants in the Movement and outside experts be invited (by advance registration) to join the discussion.