Reviewer has chosen to be Anonymous

Overall Impression: Average

Content:
Technical Quality of the paper: Average
Originality of the paper: Yes
Adequacy of the bibliography: Yes, but see detailed comments

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Good
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Weak

Detailed Comments:

This paper discusses a very interesting topic, namely, an attempt to a neuro-symbolic approach for explanations. Moreover, it focuses on XUI, i.e., on the human-machine interaction component of explanations – which is often neglected in the literature. Finally, they aim at explaining symbolic artifacts (process models), which is also something important and massively neglected. So, I am very sympathetic to what the author are set out to achieve with this work.

That said, I think the paper has many shortcomings regarding its presentation (mostly) and organization. In the sequel, I elaborate on the most important points. Again, I am sympathetic to the general idea of the paper but I think the paper should undergo a major revision and another round of reviews before publication.

1) It is unclear what is the role of machine learning in the proposed approach. I understood that, by machine learning, the authors mean Inductive Logic Programming. That is completely fine. ILP is an useful machine learning method. However, it is unclear to me how ILP is used? Is it in Layer 2 to abstract the horn-like rules they present in the paper? In page 9, the authors write: “The system prompts the user to check and correct the calculation formula of the result node. In this case, the calculation formula r(v ) of the node v(onlineBookingEngine) must be specified in a domain-orientated (SIC. Domain-oriented would be better) manner by expanding it by adding customer data in relation to a booking request. The calculation formula must therefore be adapted as follows (additions in bold): r(vonlineBookingEngine) = ← (r(venterData) && r(vcustomerData) && r(vbookingRequest) == true”. The first sentence of this passage is ambiguous: is the user supposed to update the formula (not realistic) or does learning (formulate repair) kicks in here? Still related to this topic, sometimes in the paper, the authors talk about ML and ML experts, etc. Are they referring to ML in general, to ILP, something else?

2) There is some oddness in reconciling the results and prescriptions made in section 2 and what the authors are set out to do. Section 2 basically reduces the discussion of interaction to features – which is odd when one is primarily focused on explaining symbolic artifacts - I see that features are assembled in layer 3 but this should be much better explained. Another weakness in section 2 is that it assumes too much from the reader who is interested in NeSy systems but who is not necessarily knowledgeable of the literature of XAI. There are many technical terms used here without explanations (e.g., why-explanation, global and local explanation, counterfactual explanation, feature relevance, etc.).

3) The empirical evaluation presented is too thin and preliminary. They have interviewed six experts in a qualitative evaluation without a clear frame of comparison to alternatives. In page 13, they write: “A broader evaluation with various case studies and a larger survey participation is in preparation.” Perhaps we should wait for those results? The issue is that many of the claims of this paper are empirical claims (in p.9, the results are supposed to be “easily readable, comprehensible and understandable”).

4) Section 2 is an interesting result. However, it is frustrating that the mapping study does not look at explanation from a broader and more critical perspective. There is an immense literature on explanation that goes beyond explainable AI. According to this literature, some of these typical XAI strategies should not even be considered valid explanations, since they don't satisfy the most basic requirements for explanation. A paper that could be useful in this respect is (https://www.sciencedirect.com/science/article/pii/S0169023X24000491). In particular, because it also defends the need for explaining symbolic artifacts, as the authors are pursuing here.

5) Now, here is my main issue with the paper as it is: section 4. The explanation given there is way too succinct, cryptic, and hard to understand. The authors should spend time elaborating this (absolutely central) section of the paper. I think a running example (e.g., the one used in section 5) could help. They should better explain why this 5-layered architecture, and what really happens inside each of the layer. How is the analysis from 4 to 3 happens (algorithm 1 and its subsequent description are far from enough), how are the rules in L2 formed, verified and revised, how are the paths constructed what exactly is the algorithm here?), and so on.

6) In 5.1, given that we have two profiles here (administrators and domain experts), I was expecting to see at least an illustration of different explanations to these two profiles

7) P18: “In summary, it can be stated that different forms of explanation can support the explanatory power in the sense of comprehensible results.” -> True. However, the true discussion here shouldn’t be restricted to the concrete syntactical form of explanation presentation (e.g., visual, textural or tabular). The literature of explanation is rich on different understandings of what an explanation is. For example, one could combine pragmatic explanations with metaphorical and unificatory ones (again, the paper I mentioned above can perhaps help connecting to this literature).

5. P6: “ontologies frequently permit modeling flexibility, which enables the interpretation of modeled content and, consequently, makes the generic utilization and extension of the ontology for process analysis more challenging.” -> I honestly don’t know what the authors mean here. Ontologies are supposed to excluded unintended interpretations of content, not permit flexibility. The authors then continue “In such domain-specific cases, Noy and McGuinness propose the creation of an ontology from scratch”. I still don’t understand how this connects to the previous sentence. In any case, Noy and McGuinneess’ paper was meant as a very-basic introduction to ontologies (hence the title ‘Ontology 101’) not as a methodology for ontology engineering.

6. P9: “If, after three evaluation cycles, the value of the weighting of a result node assumes the value −1” -> Why three cycles?

Reviewer has chosen to be Anonymous

Overall Impression: Good

Content:
Technical Quality of the paper: Good
Originality of the paper: Yes
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Good
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Good

Detailed Comments:

The authors conduct a literature review of literature on user interfaces for explanations (XUI) and identify 29 explanation components. They then implement 12 of these components in a prototype system for analyzing business process logic that combines knowledge graph reasoning with machine learning feedback from users. While this paper does not feature a neurosymbolic system, it is topical to several areas stated in the journal special issue CFP. That aside, I do think those working in the area of neurosymbolic AI should pay attention to this paper - as the often-cited explainability capabilities of neurosymbolic systems typically rely on a system to present logic and/or knowledge graph reasoning to the user.

However, the authors should make several adjustments to improve understanding. I recommend that the authors change the following.

1. It is very hard to follow what the authors are doing without a unified example. Consider the following points:

1a. The first bit of an example is in Figure 1 - but little context for this example part, "explanation user interface," is provided. I would highly recommend that the authors consider a true running example, (e.g., a shaded boxes describing the example, or calling out when you are describing an example with italics and sub-headings "Example 1", etc.). This would make the scenario being described much clearer.

1b. Figures 2, 3, and 4 are illustrative examples, but are they all in the same scenario? Are they in the same scenario as Figure 1? This is not very clear. Again, here it would be of great benefit to have a running example, and unify figures 1-4 around a single contextual scenario that is presented to the reader.

1c. How does the example relate to the UI in section 5? It would be good to both mention (in section 4) how the design components interact with the overall system and perhaps continue the example in section 5 to point out how the design components make an impact.

2. The paper appears dis-joint. The XUI components described in section 3 are not mentioned in section 4. Section 4 is very technical, and the reader needs to better understand the context of this technical information with respect to XUI.

3. Regarding the selected design components, there is no discussion of how the design components are selected (which would have made section 3 seem more integrated in to the paper) and now discussion of how parts of the prototype implemented the specific design components (this would link section 5 back with the earlier portions of the paper).

4. I think there should probably be more discussion on the design component feedback from the users - I think the paragraph starting on line 10 (section 15) is a good start (though you should continue to use the x_ numbers for reference), but I think the analysis should be more in-depth. Did the authors have thesis as to why some of the components seen less often in the literature were viewed as more relevant to the users? Was it due to the specific scenario the users were looking at, or was it that the literature did not address XUI in the context of a logic-based system? Was the selection of those due to some hypothesis that they would be particularly relevant? These are the types of questions that would be interesting for discussion.

5. Overall, the ML feedback mechanism, while simple, should be discussed further in section 4, and also how it impacts user interface design in section 5. Likewise, the survey results relating to ML should be linked back to these points.

6. Additionally, there were some small mistakes:

In figure 1, the authors mis-spell duplicated (dublicated).

Line 23 of page 14 starts with a number (write out the word to start a sentence).

Reviewer has chosen to be Anonymous

Overall Impression: Weak

Content:
Technical Quality of the paper: Bad
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: No
Introduction: background and motivation: Limited
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Weak

Detailed Comments:

The paper tackles the role of Explainable AI methods in providing human-understandable explanations about the algorithmic decisions within a system. The authors investigate the design of user interfaces for explanations (XUI) for results in applying AI techniques. First of all, it is not very clear to which AI results the authors refer to. I might guess that they refer to applying process mining for analyzing event logs. (p. 2 lines 15-19) or is it how the business processes are checked for deficiencies and improvement measures by using To-Be knowledge stored in knowledge graphs? (p. 2 lines 20-22). It seems that the later might be more plausible based on the presented use case. This should be clarified from the beginning and also a more detailed description should be considered when presenting the process analysis procedure.
The authors performed a literature review in order to set a design catalog for XUI. It seems their research was limited to identifying XUIs that employed IML methods, although other methods can also support or replace IML in building effective XUIs, among which knowledge-based systems are quite efficient in providing path disclosure for reasoning results. This is a bit strange since they also employ a KB approach. On which basis was the following statement made, since the performed literature review was for another purpose? “A review of the literature revealed no freely available knowledge base in the form of an ontology that would be suitable for identifying weaknesses and improvement measures in business process analysis” (p.6 lines 13-14)
Further, having the knowledge base with nodes and relationships, it is not clear which is the mechanism that enables the “activated elements of a process”. The authors must present more precisely in which way “Different paths are traversed in a domain-specific knowledge graph during process analyses.”(p.8 line 17). I believe this is very important, especially for a paper that treats XAI solutions. The use of mathematical formalism might be intimidating in presenting the entire path navigation. This could be done by presenting the traversal route as small, digestible rules which are sequentially triggered. What triggers them must also be clearly presented. One or possible multiple scenarios should be better described (including a BPMN or Petri Net diagram) to understand the process analysis. The presented use case is not clearly explained: the diagram depicted in Fig 3 is erroneous and it does not seem that the authors made the mistakes on purpose (a tool like SAP Signavio would identify the errors). Moreover, the knowledge graph, as well, is not very clear – one should easily identify the entities and the schema elements, preferably with popular notations (rectangles for literal values, circles for nodes, distinguishing individuals’ nodes from classes’ nodes through a color): what are hexagons? What are rhombs? circles? ovals? What’s the relationship between them and the numbers visible on some of the edges?

Other minor remarks: some pleonastic expressions: “human-understandable XUI” (p.2 line 27), possible small typos like “is” instead of “its”? (p7 line 7) ; missing abbreviations (e.g. LIME p.4 line 19)