By Andreas Martin
Review Details
Reviewer has chosen not to be Anonymous
Overall Impression: Average
Content:
Technical Quality of the paper: Average
Originality of the paper: Yes
Adequacy of the bibliography: Yes, but see detailed comments
Presentation:
Adequacy of the abstract: No
Introduction: background and motivation: Limited
Organization of the paper: Needs improvement
Level of English: Unsatisfactory
Overall presentation: Weak
Detailed Comments:
The study is well-aligned with current trends in applying generative AI to business process analysis, but several significant issues limit the paper's clarity and overall contribution.
Key Strengths
- Timely Research: The paper addresses the intersection of generative AI and BPM, an area with considerable potential for advancing process analysis techniques.
- Visual Insights: Figures 5 and 6 are well-executed and hold practical value, especially for readers unfamiliar with BPM concepts.
Major Concerns and Recommendations
1. Uncertainty Regarding LLM Inferencing
- The methodology lacks clarity about how LLM inferencing was conducted. The abstract mentions the ChatGPT interface, but the manuscript inconsistently refers to “LLM services” and “GPT services,” leaving the reader unsure whether standard ChatGPT functionality or custom GPTs were used.
- Additionally, the specific models utilized are not disclosed. Clear documentation of the experimental setup, including the version of the LLM and its configuration, is essential for reproducibility and transparency.
2. State-of-the-Art and Background
- The section titled “Large Language Models and the BPM lifecycle” appears to function as a background chapter, but is far too brief for a journal article. It lacks critical engagement with existing literature and does not establish a clear research gap.
- A proper literature review should analyze the strengths and limitations of prior work, offering a critical stance and positioning the paper's contribution in the broader academic discourse.
3. Research Gap, Hypothesis, and Questions
- The paper fails to articulate its research gap clearly, hypotheses, or research questions. Without this foundation, the study's objectives and outcomes are ambiguous, making it difficult to evaluate its scientific merit.
- Explicitly stating these elements would provide focus and enable readers to assess the novelty and value of the research.
4. Presentation of Tables and Figures
- The tables are not adequately referenced in the text, and their presentation is suboptimal. Consider relocating tables to the appendix if they are not directly essential to the narrative.
- Figures 1–3 and potentially Figure 4 could also be moved to an appendix, as their content adds limited value to the main discussion. Conversely, Figures 5 and 6 could be expanded upon to better engage readers less familiar with BPM concepts.
5. Discussion Section
- The paper lacks a dedicated discussion section, which is crucial for interpreting findings and situating them within the context of existing work. The absence of a discussion makes it difficult for readers to gauge the added value of the study.
- A robust discussion should revisit the research questions (if stated) and critically evaluate the implications, limitations, and potential future directions of the findings.
6. Conclusion
- The conclusion does not sufficiently explore future research opportunities. For example, the potential of continuous pre-training on domain-specific knowledge graphs, meta-models, or XML/RDF schemas is an intriguing direction that is not mentioned but warrants inclusion.
- Introducing these perspectives would underline the study's relevance and inspire further exploration in the field.
Additional Recommendations
- Proofreading and Language Clarity: Issues such as incomplete sentences (e.g., the last sentence in the abstract) and unnecessarily long sentences hinder readability. Rigorous proofreading is needed.
- Terminology Regarding LLM Capabilities: Caution is advised when describing LLMs as “understanding” or “reasoning.” The paper should reflect recent literature, which highlights the limitations of LLMs in these areas.
- SPARQL Presentation: SPARQL snippets should not span across pages, as this disrupts the flow and readability of technical content.
Summary of Improvements Needed
1. Clarify the methodology, specifically the LLM interface and models used.
2. Expand the state-of-the-art section into a comprehensive literature review with a critical stance.
3. Explicitly define the research gap, hypotheses, or questions.
4. Improve the referencing and presentation of tables and figures, relocating less critical ones to the appendix.
5. Introduce a discussion section to interpret the findings and connect them to broader implications.
6. Enhance the conclusion by discussing future research directions, particularly pre-training on structured schemas.
