Yoshua Bengio, along with Geoffrey Hinton and Yann Lecun, is considered one of the three people most responsible for the advancement of deep learning during the 1990s, 2000s, and now. Cited 139,000 times, he has been integral to some of the biggest breakthroughs in AI over the past 3 decades. ~ Description from Lex Fridman website
This one was a pretty short interview, about 45 mins, but several ideas were not picked up by the Singapore community from what I see. Putting down these ideas to explore further on my own. :)
Note 1: Credit Assignment
You might be wondering what is credit assignment we are talking about here? It is assigning weights to previous knowledge gained and these weights will affect the decision-making process.
In current machine learning/deep learning, the credit assignment can be achieved through backpropagation or LSTM. For the case of backpropagation, it is still unclear whether our brain undertakes such a method to manage the weights.
In the case of LSTM, there is still a limit to how much and how recent "memory" is stored. Compare to our brains, there is an arbitrary period of which memory is kept, where we can remember things for many years ago, perhaps during our childhood.
The current artificial neural network also takes a long time to change/update the weights whereas, for human's neural network, decision weights can be updated very quickly. The challenge is selecting what to remember and when to bring it up to the conscious level for decision making. This part is still not unraveled yet by the scientific community. Unraveling it can bring about architectural changes to the current artificial neural network to make it better.
Note 2: Data or Architecture?
Prof Yoshua mentioned that the current challenges are not about data or architecture anymore but rather the design of the training objectives. To have a breakthrough, it is time to visit the training objective, have it more directed and be able to get the neural network to search the semantic space better, understand cause & effect better (and usage perhaps?) and reward the "right" kind of exploration.
"More layers will not make it significantly better anymore."
Which is true if we believed that each layer is a level of abstraction of features. If you just explore things that are familiar to us like perhaps an apple, how many facts (abstraction level) do you know about it? Surely there is a limit to it.
Note 3: Where Symbolic AI failed? Where Deep Learning (currently) failed?
This is a pretty good discussion and "excited" more thoughts in me.
Prof Yoshua pointed out that currently symbolic AI or GOFAI failed was because if you look at how humans make a decision, no doubt there are reasons that can be extracted but there will be other factors that come into play. GOFAI needs these reasons/factors for decision making to be explicit so that they can be stated in rules or statements. This is the reason why GOFAI cannot handle uncertainty and uncertainty is all around us.
For deep learning, although it is successful at handling uncertainty but to make it better, we need "Disentangled Representation Learning" (another new keyword to explore! Nice!).
"Disentangled representation is an unsupervised learning technique that breaks down, or disentangles, each feature into narrowly defined variables and encodes them as separate dimensions. The goal is to mimic the quick intuition process of a human, using both “high” and “low” dimension reasoning." ~ from DeepAi.org
What needs to be dis-entangled is the relationship between low and high dimensions within the feature and also between the features as well. Through this, perhaps the cause and effect can be learned better. Currently, Artificial Neural Network is all tangled up without knowing how the features are linked and also dimensions within each feature how they are linked together i.e. very challenging to unravel them.
Note 4: Managing Bias in Data
This is a pretty short note and that is Prof Yoshua mentioned that in machine learning we can build less biased predictors. However, regulation should come in to ensure biases are managed, otherwise, no companies will manage on their own, given that they are faced with a scenario of choosing accuracy (affects profit) or reducing bias (social responsibility).
Note 5: Machine Teaching (Another Keyword!)
Machine teaching was a training method brought up during the interview. The concept is similar to mentoring but this time round, human teaches (mentor is a better word, I feel) the machine agent.
The human comes in to help point directions of learning for the machine agent, much like a tutorial system. This will ensure that a human is within the decision loop avoiding the paperclip challenge.
This is intriguing and definitely will try to read up more on how to structure the training process up but yes, a great training direction. But sad that I only hear about this from Lex Fridman's interview and nothing from elsewhere. :(
Note 6: Language Learning
For this note it is similar to the discussion with Tomaso Poggio, "How do we know if we have built an AGI?". But this discussion is from the perspective of language. How can be a build an agent that can hold a good conversation?
Prof Yoshua mentioned, we need to build agents that can understand the implications and context behind statements, i.e. what does the statement infer behind the scene, the invisible (so to speak) or context. What comes to my mind is sarcasm. If we can first, create an agent that understands sarcasm well enough, meaning it can understand the meaning behind the sarcastic remark and later, create an agent that can create sarcastic remarks, we might be on the right path to building AGI! :)
Other smaller notes included how Prof Yoshua sees the research community, his opinion on Ex Machina (and its wrong representation of the scientific community) and existential threats from AI. Existential threat wise, there is no need to worry for now but have to start researching into it. (I hold the same opinion as well...if anyone cares. :P )
If you are interested to discuss the content or any AI-related questions, feel free to link up with me on LinkedIn or my Twitter (@PSkoo). Do consider signing up for my newsletter to be updated with what I am learning and doing. :)