Tag Archive: Methodology Agenda


Validation of ADAS Nexyad Database

The NEXYAD company is currently developing the construction of a database for the validation of systems of driver assistance and driving delegation, (ADAS and Autonomous car) using the AGENDA methodology published in the 1990s by Gérard Yahiaoui (methodology initially intended to handle, among other things, the construction of learning databases and tests for the implementation of neural networks).

This database has two essential characteristics:

1) Real-life situations
     Indeed, the AGENDA methodology recommends describing the possible variations of signals and input images as factors of variability and their crosses.
     Example, for obstacle detection:
          . weather (dry weather, sunny weather, rain, fog)          . overall brightness (low, medium, high)
          . vehicle speed (low, moderate, high)
          . type of road (motorway, road with marking, road without marking, …)
          . coating (bitumen 1, bitumen 2, …, pavers)
          . day / night (car headlights and infrastructure lighting)
          . season (spring, summer, autumn, winter)
          . etc.

     Type of obstacle:
          – static obstacle
          . linked to the infrastructure: works terminals, tolls, …
          . related to users: tire on the roadway, package dropped from a truck, motorcyclist lying on the road following an accident, vehicle broken down stopped on the roadway, pedestrian stationary on the edge of the roadway (visible / hidden)
          – moving obstacle
          . truck, car, vulnerable (pedestrian, bike, motorcycle) with the typical trajectories (longitudinal in the direction of travel, longitudinal in the opposite direction of travel, lateral) and the position (opposite, right, left) .
          . Etc.

Real-life situations examples :
Validation of ADAS Nexyad database

We see that if we combine these factors, we can find quite quickly a very large number of cases. Now, the development of ADAS systems is complex, and it is necessary to proceed by successive iterations, from simple situations to complicated situations.
Our database allows this, since all records are described such as cross-referring of modalities of the factors of variability. We thus know exactly in which cases the system was tested or not.
The formalism of ‘cross-referring of modalities of the factors of variability’ makes it possible to use experimental designs, and in particular orthogonal fractional plans, to greatly reduce the number of cases to be tested while guaranteeing maximum coverage of real-life situations. In this context, we can develop an ADAS on an orthogonal fractional plane and test it for other orthogonal fractional planes, for example.

2) Ground reality
     It is a question of define the obstacles and elements of the infrastructure (markings, road edges, etc.) on the images so as to constitute a reference allow measuring the performance of the system.

          1.1, summer, cloudy, unmarked road, moderate speed, tire on roadway, dry weather
          1.2, summer, cloudy, unmarked road, moderate speed, package on roadway, dry weather
          2.1, summer, cloudy, unmarked road, moderate speed, visible static pedestrians at the edges of the roadway, time
          2.2, summer, cloudy, unmarked road, moderate speed, human lying on roadway, dry weather

     It is not certain that one can meet these few cases, even by driving 1 million km on open road!

Target of this database :
     NEXYAD starts its collection of images and data:
          . video (towards the front of the vehicle) color
          . accelerometers
          . gyros

The files are synchronized by the RT-MAPS tool of the company INTEMPORA.
The files are saved in RT-MAPS format and directly replayable on this tool.
NEXYAD is currently looking for contributors on this internal project. Contributors co fund and have in return free access to the database, unlimited in time. This contribution will accelerate the work of collection and labeling.
NEXYAD wishes to make this database available soon, free of charge to give material to the ADAS community and the autonomous vehicle, in a reduced version of the database, and in a paid way (in the form of subscriptions) for the complete base.

NEXYAD’s ambition is to propagate its methodological expertise and to enable everyone to evaluate the performance of vision systems for ADAS, be they systems developed by NEXYAD or others.

Methodology for ADAS Validation: The Potential Contribution of Other Scientific Fields Which Have Already Answered the Same Questions”, Gérard Yahiaoui, Pierre Da Silva Dias, CESA congress Dec 2014, Paris, Proc. Springer Verlag

“Methods and tools for ADAS validation”, Gérard Yahiaoui, Nicolas du Lac, Safetyweek congress, May 2015, Aschaffenburg


Conference on Machine Learning organized by SNCF

Very interesting conference on Machine Learning and Deep Learning.
SNCF RESEAUX welcomed a large audience last tuesday 28th march in Saint Denis.

French experts of A.I. field were introduced by Claude Solard SNCF réseau Chief Operating Officer and Jean-Jacques Thomas Director of Innovation.
Christophe Garcia, Full professor at INSA Lyon, deputy director of the LIRIS laboratory, gave an introduction to Neural Networks and Artificial Intelligence and answered to the audience’s questions.
Then, Gérard Yahiaoui, CEO of Nexyad, Vice President of the MOV’EO Cluster, explained the Criteria for use and Methodology for implementation: the methodology AGENDA.

SNCF Conference Gerard Yahiaoui
Gérard Yahiaoui CEO of Nexyad

From CEA List, Jean-Marc Philippe presented “Tools of the Network to the Optimized System”. The conference continued with a roundtable about the SNCF Cafeine Project or how to go from the industrial need to the applied solutions in SNCF Réseau; animated by Alain Rivero, SNCF Réseau Director of Cafeine Project; Xavier Roy, Head of Innovation of ITNovem; Michel Paindavoine, co-founder of Global Sensing Technologies and professor at University of Bourgogne; and Jean-Jacques Thomas Director of innovation of SNCF Réseau.

The CAFEINE Project proposes to use the Artificial Neural Networks to detect the defects of pantographs on moving trains, and also to read the number of each wagons, oars and containers.

Another roundtable was organized about Startups and A.I. with Kunthirvy Collin-Dy, Head of relationships with Startups of SNCF Réseau Robot Lab; and Yannick Gérard, Head of Project of DAVI.
Patrick Bastard, Director of Driving Assistance Systems engineering of Renault, ended the conference with a presentation on an opening to other fields of application : vehicles electrification, ADAS and autonomous driving.

Conference SNCF 3 speakers
From the left : Jean-Marc Philippe, Patrick Bastard et Christophe Garcia


Autonomous Vehicle TEST & DEVELOPMENT Symposium 2016

Autonomous Vehicle TEST & DEVELOPMENT Symposium 2016
31 may – 2 june 2016 Stuttgart, Germany.

Test & Development Symposium Stuttgart 2016

2016 Preliminary Conference Programme

Wesnesday 1st June

09:15 – 15:45 – Test and Validation Strategies for Autonomous Vehicles
Room B

09:45 – Building a relevant validation database for camera-based ADAS
Gérard Yahiaoui, President and CEO, Nexyad, France
Validation of camera-based artificial vision systems applied on open world is a very complex issue. An HD colour camera may generate more than 65 000 power 2 000 000 different images (information theory), so it is not possible to test every possible message. We propose a deterministic approach for building a validation database using the AGENDA methodology that was developed and published in the 1990s for neural network database (learn and test) design.



Driving Delegation: key elements

Driving Delegation: key elements for an artificial perception system
Publication of September 2, 2015
Authors : Gérard YAHIAOUI & Pierre DA SILVA DIAS

The automotive industry starts offering ADAS, and plans to propose in the near future partial or total driving delegation systems.

Main cases to be processed first may be:
. Highway driving, where the number of events per kilometer is small because the infrastructure has been designed to minimize path irregularities (little or no turns, every car in the same direction, wide track, geometric visibility up to several kilometers, enough little interactions between vehicles, at least when the traffic is flowing).
. The city, where infrastructure complexity is very large, where interactions between the road users are very strong, making detection a difficult tasks, but where speed of the vehicle is low.
In all cases, these future ADAS require developing advanced systems of perception.

Perception consist in detecting objects, clustering, and possibly tracking them in their own trajectory, from selected sensors (cameras, radar, lidar, slam, ultrasound, …)

It is usually presented as several phases :
. Detection: we perceive that “something” comes off the background, but we do not know what it this is. The Johnson criteria for detection give a theoretical limit of one period, or a minimum width of two pixels to detect a stationary object.
. segmentation and tracking: when zones are detected as being detached from the background (the landscape for image processing, the cluter for a radar, …), the detection must be agglomerated to track large enough objects that may have a meaning.
. Recognition: Recognition is to be able to say what it is. The Johnson criteria for human vision is about 6 periods (for stationnary objects) which gives 12 pixels.
. identification: identification gives, in the recognized class, the precise name of the object.

Detection is by far the most complex. It is potentially based on several principles:
. breakage hypothesis : we made a number of assumptions about world geography. We choose this hypothesis and make sure they are verified for the landscape (or cluter), and not for the objects to be detected. The non-validation of assumptions corresponds to a detection.
. the confrontation of a knowledge of the landscape or cluter: Comparing the “background” as it is supposed to appear in the absence of additional objects with said background which contains objects lead to detection of those objects.
. the knowledge of the shape of the objects to be detected: in this case the detection and pattern recognition are the same. System detects an object in its environment because it recognizes this object.
Human perception jointly implements the three principles.

Perceptions systems incorporate sensors and methods of processing, and are generally effective in a frame capture conditions, and little or not effective in the other frames. For example, a camera in the visible wavelenghts (and its image processing methods), will generally not be effective at night or in fog because “you can not see anything.”

No detection system can operate in any case when dealing with a real problem in the open world.

Designing a detection system then comprises two important phases:
. extend the maximum possible number of cases where the detection system works.
. have a diagnosis that allows to know when it is or when it is not in a position to that the perception system is effective.

We talk about performance (very efficient detection of all objects of interest), strength (number of cases where the collection system remains effective), and reliability (Situational Awareness in which one is and thus the confidence that can be placed in the collection system).
These three elements, performance, robustness, reliability, should be fully known in order to cooperate collection systems (for example, a camera and a radar).

NEXYAD proposed the Methodology AGENDA for characterizing life situations, using the formalism of orthogonal plans of experiments. The recognition of cases of functioning mode can be based on the description of life situations with this methodology. This gives a theoretical and practical framework for an estimation of robustness and reliability.
Performance is measured with statistical comparison operators: in general, it is considered the output of a detection system is a categorical variable with two categories: “detected” and “not detected”. This variable must be compared to a qualitative variable of reference that also has two modalities: “Presence of an object to
be detected” and “absence of objects to be detected.” The comparison can not be made by calculating a percentage (yet it is often that performance is measured this way), but it must use tools such as contingency table, the Khi2, normalized Khi2, khi2 in the box, etc …
To extend the life situations of the domain where the system detects objects correctly, we use to make cooperate several detection systems which use complementary types of sensors (eg in fog, we will trust in radar or infrared detection, but not detection by conventional camera).
A reliable system is one that is able to answer “I do not know”: in the case of driving delegation a system that could detect all objects so powerful, robust, and reliable in 30% of the time has a great value.
The delegation of driving frees 30% time of the driver, which is a real value proposition.

Safety is a discipline that encompasses many issues with the objective of ensuring the proper functioning of the system in all cases.
In particular, we must be vigilant concerning detection systems which require to have several measurement channels, such as stereovision.
If detection works only when you can have both cameras, then safety experts refuse such a system because two cameras means 2 times more likely that one fails.
We then see that perception system must have quite still usable “degraded mode” when simulating glitches sensors. A good design of a perception system for ADAS incorporates all these elements.

The race for performance that interests the engineers is rarely the real issue in industrial systems. A system that allows to delegate the driving in 30% of cases (eg clear overcast day dry) and “knows” when there is a case for which it works or does not work, can delegate driving and release the driver for 30% of the time.
This is a proposal for a very high value for the driver.
A system that works effectively in 99% of cases without knowing precisely when it works is absolutely unusable. No manufacturer will put such a system in operation for road safety applications.
The company NEXYAD has been working on these issues for twenty years, especially on road detection, obstacles detection, measurement of visibility (to describe cases where the detection is reliable, for example), the estimation of road safety (suitability driving style with the infrastructure).

NEXYAD developed:
. efficient and very robust basic bricks: RoadNex, ObstaNex, VisiNex onboard, SafetyNex
. a methodology for characterizing life situations in which it develops and tests an ADAS: AGENDA (Improvement performance, the recognition of cases of good performance, and validation of ADAS).
. know-how in collaboration between multiple perception systems.