Reviewer has chosen to be Anonymous

Overall Impression: Weak

Content:
Technical Quality of the paper: Good
Originality of the paper: No
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Satisfactory
Level of English: Satisfactory
Overall presentation: Average

Detailed Comments:

The work presents an evaluation of neural models for KG node and relation extraction from unstructured text for the purpose of KG extraction. Overall, the presentation is reasonable and the work is relatively easy to follow. The review of related work good.

However, I have three major concerns about the work.

* The contribution is unclear, and novelty is also unclear. KG extraction from unstructured narrative text documents as a long history that, as the authors correctly point out, is grounded in KB/ontology learning. The methods for KG extraction compared by the authors are used very often in the literature, including the more novel GPT/LLM based models. The question thus arises, what exactly is this work adding to the state of the art in this area.

* Even more problematic is the small dataset. In my opinion, the obtained scores mean very little, if anything. The task on a single piece of news is likely just too simple, especially for the modern LLM based models. The choice of a news article may also favour LLMs. Overall, I think the dataset should be significantly larger and more heterogeneous (not just new articles, but also other kind of text documents, such as books, patents, legal documents, scientific articles).

* As the author highlight in the discussion, the results are mostly if not exclusively unsurprising and to be expected. What are we learning from the work?

Minor comments:

* The authors make a confusing use of KG creation and KG extraction, especially also in relation to IE. For instance, at the beginning of Section 2.2, the authors state that IE techniques are often used for extracting KGs just to speak of creating a KG further down. IMO, it makes more sense to speak of IE and KG construction. Indeed, one constructs a single KG (as a database) rather than extracting X small KGs from text.

* Page 1, Line 40: Reasoning relies on formal semantics (axioms) and as the authors point out, there are barely any extraction techniques that are capable of extracting axioms from unstructured text.

* Page 2, Line 30: KGs have certainly not "been around since the originating of the field of philosophy"! Ontologies are older, but obviously also not in an sense as understood in information science.

* Page 2, Footnote 1: Please deposity on a research data repository and identify by DOI; there is no guarantee of persistence in Github URLs

* Page 3, Line 16: No better example than the pizza ontology?

* Page 4, Line 36: What is future-proof in this context?

* Section 3: Please describe the dataset, especially in terms of size and other descriptive statistics.

* Page 6, Line 44: Check language "the different methods are described that are tested ..." (also elsewhere there are typos!)

* Page 8, Line 9: Is the sentence "Think carefully before you answer" in the prompt needed and if yes, why? GPT obviously doesn't think, but I can't tell whether this "instruction" influences performance.

* Please include DOIs in your references - I am stunned how often this still needs to be stated in reviews; as a reviewer, I do not want to have to copy and paste titles into a search engine and hope I get the right version of some work!

Reviewer has chosen to be Anonymous

Overall Impression: Weak

Content:
Technical Quality of the paper: Weak
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Satisfactory
Level of English: Satisfactory
Overall presentation: Good

Detailed Comments:

The authors explore the comparative performance of various relation extraction methods for creating knowledge graphs from unstructured text. They want to assess these techniques in a real-world scenario, specifically focusing on producing graphs that match the quality of those manually developed by domain experts. Through a comprehensive evaluation, the paper analyzes methods like REBEL, KnowGL, and GPT models on their ability to extract entities and relations from text while maintaining graph quality and expressiveness.

The study also highlights the strengths and limitations of these approaches, particularly in handling complex, real-world texts versus simplified ones. The authors aim to provide insights into the trade-offs between precision, recall, and graph density, making the research relevant for applications requiring scalable, high-quality knowledge graph development.

The paper is well presented, but severely limited from an empirical perspective, which is where I'll focus my critiques:

--The study relies on a single news article about the Nord Stream pipeline incident as the sole source of data, which undermines its generalizability. It fails to account for the variety and complexity of language structures present in broader datasets. I'm not sure why only this one news article was the sole scope of the work.

--The authors keep mentioning 'real world' but their idea of what happens in the real world is different from what I've witnessed in recent industry efforts. No one manually constructs KGs in the real world. There is manual work, yes, but it usually builds on first running an automated pipeline, followed by some kind of manual effort. I'm not seeing any of this discussed in the paper. Again, the reason the authors likely did not encounter this problem is because they are only focusing on one news article. They need to do this at a corpus level (i.e., at the level of thousands of news article). They don't need to manually annotate it themselves. The LDC makes available such corpora for such evaluations on a regular basis.

--The results are very uniform and unsurprising. A GPT model ends up performing the best on F-score in both scenarios. I'm not sure what there is to learn here that adds to our knowledge.

--Even if the authors are only going to focus on manually intensive cases as they did here, it behooves them to have tried different modalities (e.g., tweets, blog post, consulting report in PDF etc.). Comparing those differences would have been much more interesting.

--There is also a lack of methodological diversity here. In recent years, methods like chain-of-thought prompting have become interesting ways of prompting LLMs, including for constructing KGs. The paper does not seem to show awareness of such techniques, let alone try an evaluation of them.

The good news is that all of the above is addressable. The authors could choose to remain at small scale, but they should then significantly expand their number of case studies and methods. Or they could choose to focus on fewer methods, but aim for a more complete and high-impact 'bakeoff' which would justify the paper as an empirical survey. Of course, the best possible paper would do both. This paper unfortunately does neither, and hence, I cannot recommend acceptance or revision.