By Daniel Silver
Review Details
Reviewer has chosen not to be Anonymous
Overall Impression: Average
Content:
Technical Quality of the paper: Average
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes, but see detailed comments
Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Satisfactory
Level of English: Satisfactory
Overall presentation: Good
Detailed Comments:
The paper is entitled “A Computational Perspective …” but it seems to take more of a philosophical perspective only landing on the computational perspective to justify a conclusion based on the current hardware used to implement neural networks.
Section 2 of the paper provides an interesting overview of the symbolic-based relational logic AI and statistical relational learning using tensor-based deep learning methods; and gaps between them that are at the heart of the Neural-Symbolic Integration problem. This review is a helpful introduction. However, one of the key statements in this section is that “Hence, there is just no way of mapping that relational computation into any neural model with a fixed number of inputs, neurons in each layer, and overall depth, which is, however, the core design pattern in current deep learning”. Unfortunately, we are now discovering that RNNs and Transformer models can use representations of concepts (objects and relations) spread over time and space to accomplish this. Akin to the generative creation of text or code by ChatGPT, such neural models will likely be able to manipulate an “unbound set of related atoms that are true in a given world”. Again, these will often not yield optimal solutions, but they will be ones that work well enough in a PAC sense.
Regarding the key issue of variable binding in neural networks, there have been a number of solutions proposed. We refer the authors to the position paper by Klaus Greff, Sjoerd van Steenkiste, Jürgen Schmidhuber entitled “On the Binding Problem in Artificial Neural Networks”. https://arxiv.org/abs/2012.05208
The later portion of the paper focuses on relational machine learning and its difficulties for both symbolic and neural systems. In Section 3, the authors suggest that much of real-world data does not come in numeric tensor form but in relational data representations and is therefore more amenable to manipulation by symbols using techniques such as Inductive Logic Programming (ILP). I am not convinced that this is correct, because all human sensory systems receive data in the form of distributed representations - via our retinas, our finger tips, our olfactory system, etc spread out over time and space. These sensations enter our bodies through a finite set of senses that each have a finite set of receptors. Concepts and the symbols to which they refer are learned and recognized. Temporal and spatial relations (a form of concept) are also learned and recognized as a function of repeated exposure to examples of such. And they are rarely perfect, but they do get sub-optimally better over time with experience.
In the Section 3.3 on relational logic-based Lifted Graphical Models, it is acknowledge that similar principals are observed in deep neural network methods, but in Section 4 the authors suggest that the structure of the computation in these networks is the problem due to their fixed nature. The authors look at Recurrent networks (RNNs) including Graph Neural Networks as a way to overcome this problem, but ultimately are not clear end the end of Section 4 on where they stand on RNNs as a possible solution to learning and representing relations. Instead, the argument shifts to how recent research has focused on developing RNNs (or RNN replacements such as Transformers) that run on the current SIMD GPU technology. They are correct in their assessment of the simplifications made to these neural networks to work with this hardware – these simplifications do limit the potential for true RNNs to manipulate an “unbound set of related atoms that are true in a given world”. However, this is not an argument against the theoretical approach of using RNN architectures. It is an argument against the implementation of RNN acrchitectures that use the current hardware platforms. I would have preferred to have seen an argument that focused on the fundmentals of how we can bridge the strengths of symbolic reasoning and planning with the strengths of neural network learning.
Nonetheless, I found the paper thought provoking and for this reason, I think we should allow other researchers to read the thoughts of these authors following a revision that speaks more fundamentally to how the strengths of symbolic reasoning and planning might be combined with the strengths of neural network learning should we have the necessary hardware. In doing so, I would encourage the authors to consider the following perspective - The question of solving provably hard problems is not answered by current neural-based or by symbolic-based approaches alone. And it is not likely that they will be answered by combining current approaches. Such problems are by their nature hard, with optimal solutions often defying human intelligence. For such problems, optimal solutions are rarely found, instead we must settle for (human and artificial) systems that select suboptimal solutions that work well enough most of the time, in a PAC sense.
