By Gustav Šír
Review Details
Reviewer has chosen not to be Anonymous
Overall Impression: Good
Content:
Technical Quality of the paper: Excellent
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes, but see detailed comments
Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Good
Organization of the paper: Satisfactory
Level of English: Satisfactory
Overall presentation: Good
Detailed Comments:
I'm generally happy with the revision -- the authors addressed almost all of my previous concerns (incl. new experiments). What remains is mostly wording around the new parts, e.g. the results in Table 5, showing that the neural part actually contributes quite little on average, which the text still frames rather generously (I'd like to see this addressed a bit more honestly)...plus a few other things, none of which is major, and can be directly resolved with a few text edits.
My main previous concern was overclaiming, with which the authors did a thorough and honest job ("propositional" in the title, the scope of probabilistic-logic/circuit paradigm is explicitly stated, interpretability claims are separated/softened, etc.), plus the new experiments (incl. Sortino ablation -- shown to perform worse, fair enough!), and extended related wor are nice.
Now for the catching up with the new evidence in Table 5, which is great, but the accompanying Sec. 5.2 seems to describe it a bit too generously still:
- the reliable part of the gains seems to come from the classic propositional operators (literal insertion/removal)
- the "neural concept invention" (=the title) helps on the Image datasets with the pretrained ResNet, but hurts on the non-image data.
- note that, besides pretraining, the huge gap between validation and test (~overfitting) likely comes also from the much repeated candidate validation improvement acceptance test (end of the algorithm)
- so an honest summary seems that the symbolic revision does the main work, while the neural concepts are the a (nice, but fragile) add-on
- useful for pretrained image models, but ot in general... would be nice to see this scoping added somewhere (discussion/conclusion)
ad intro/related work - stating that KBANN/LRNNs/d-ILP "often treat the provided knowledge as fixed or rely primarily on learning new structure from data" contradicts your own (correct) related-work paragraph :)
- refining the provided knowledge was the whole point of KBANN...
- also the d-ILP shoudl cite "Learning explanatory rules from noisy data" by Evans & Grefenstette (the actual concept of d-ILP)
- btw. the LRNN structure learning is in the "Stacked Structure Learning for Lifted Relational Neural Networks" (that one does the theory revision...)
ad results: "Clearly outperforms" & "outperforms its competitors in both cases" is still a bit too strong (given Table 3)
Text hiccups: "Table 2 and 3 summarize..." is repeated (some post-hoc text edits, I guess)
----
Overall, the revision seems honest to me, the paper is now scoped corretly -- propositional theory revision within probabilistic-logic-based NeSy -- and there it is a solid contribution, I think. This is still a somewhat minor revision, but I trust the authors to incorporate the feedback successfully for the camera-ready, therefore let's not waste another round, and I mark it as Accept.
