\section{Related Work}\label{sec:related_work}
\begin{comment}
- Review existing literature on neurosymbolic ontology reasoners and benchmark datasets.
- Discuss previous research related to introducing noise in machine learning and AI datasets.
- Identify gaps in the literature that your research aims to address.
\end{comment}

Neurosymbolic approaches integrate diverse reasoning techniques, resulting in multiple variations in their evaluation. In \secref{subsec:reasoning_techniques}, we provide a brief overview of neurosymbolic reasoning methods that are used for our experiments, followed by a discussion of most commonly used benchmark datasets in \secref{subsec:benchmark_datasets}.

\subsection{Reasoning Techniques}
\label{subsec:reasoning_techniques}

Henry Kautz, in his AAAI 2020 Robert S. Engelmore Memorial Award Lecture, discussed six categories of neurosymbolic AI systems as the ``Future of AI''~\citep{henry}. To showcase the variety in existing approaches, we categorize the reasoning methods used in our experiments into one of those categories. Henry Kautz defined the six categories of neurosymbolic AI identified as follows:

\begin{enumerate}
    \item \textbf{Symbolic Neuro symbolic:} Symbolic data are embedded as vectors using methods such as Word2Vec~\citep{word2vec}, processed by neural networks, and subsequently converted back into symbolic representations.
    \item \textbf{Symbolic[Neuro]:} A primarily symbolic framework that calls upon neural networks for specific tasks within a logical system; for example, in systems like AlphaGo~\citep{alphago}, symbolic search (Monte Carlo Tree Search) queries neural networks to evaluate board states and prioritize moves.
    \item \textbf{Neuro \textbar{} Symbolic:} In this category, the neural network and symbolic module solve complementary tasks and communicate frequently to guide each other. This represents a refined integration of neural and symbolic approaches, exemplified by the Neurosymbolic Concept Learner~\citep{DBLP:journals/corr/abs-1904-12584}.  
    \item \textbf{\boldmath$\mathbf{Neuro\!:\!Symbolic \rightarrow Neuro}$:} Symbolic knowledge is used to constrain, supervise, or regularize the training and operation of neural networks, without necessarily producing explicit symbolic outputs. This is demonstrated in Deep Learning For Symbolic Mathematics~\citep{lample2019deeplearningsymbolicmathematics}.
    \item \textbf{\boldmath$\mathbf{Neuro_{\mathrm{Symbolic}}}$:} Logical rules or knowledge bases are directly embedded into neural architectures, shaping their internal representations and guiding generalization, as in the Logic Tensor Network~\citep{DBLP:journals/corr/abs-2012-13635}.
    \item \textbf{Neuro[Symbolic]:} Neural networks that are capable of logical reasoning at certain points during execution fall into this category.   
\end{enumerate}

In \citep{owl2vec*}, the authors introduced \textit{OWL2Vec*}, which involves converting the symbolic input (i.e., ontologies and RDF graphs) to vectors, giving rise to \textit{Symbolic Neuro symbolic}. The method leverages random walk and word embedding techniques to encode the semantics of OWL ontologies. Unlike traditional KG embedding methods, \textit{OWL2Vec*} considers not only the graph structure but also lexical information and logical constructors inherent in OWL ontologies. This comprehensive approach enables \textit{OWL2Vec*} to capture nuanced relationships between concepts, making it suitable for tasks requiring fine-grained reasoning, such as ontology completion and prediction. The empirical evaluation conducted with three real-world datasets, i.e., HeLis~\citep{HeLis}, FoodOn~\citep{FoodOn} and Gene Ontology (GO)~\citep{GO}, demonstrates that \textit{OWL2Vec*} outperforms the state-of-the-art methods in class membership and class subsumption tasks. This suggests that \textit{OWL2Vec*} benefits from incorporating different aspects of ontology semantics, including graph structure, lexical information, and logical constructors.

In \citep{box2el}, the authors proposed a novel ontology embedding method called \textit{Box2EL} for \texttt{DL EL++}. The approach embeds symbolic reasoning inside neural engines, representing symbolic information in geometric or vector spaces and employing neural methods for reasoning tasks, resulting in the \textit{Neuro[Symbolic]} category. Specifically, they addressed the challenge of ontology completion in Description Logic (DL)-based OWL ontologies, which are widely used for knowledge representation. While classical deductive reasoning algorithms offer precise formal semantics for predicting missing facts in an ontology, recent years have seen a rise in interest in inductive reasoning techniques capable of deriving probable facts from an ontology. Inductive reasoning techniques, akin to those used in KG completion, involve learning ontology embeddings in a latent vector space while ensuring adherence to the semantics of the underlying DL. However, existing ontology embedding methods face shortcomings, particularly in faithfully modeling complex relations and role inclusion axioms, such as one-to-many, many-to-one, and many-to-many relations. This approach represents both concepts and roles as boxes (i.e., axis-aligned hyper-rectangles) and models inter-concept relationships using a bumping mechanism. The authors conduct an extensive experimental evaluation, achieving state-of-the-art results across a variety of datasets, i.e., GALEN~\citep{GALEN}, Gene Ontology (GO)~\citep{GO} and Anatomy (a.k.a. Uberon)~\citep{Anatomy}, on the tasks of subsumption prediction, role assertion prediction and approximating deductive reasoning. Table~\ref{comparison_box2el_owl2vec} summarizes the main differences between \textit{Box2EL} and \textit{OWL2Vec*}.

\begin{table}[h!]
\centering
\caption{Comparison of \textit{Box2EL} and \textit{OWL2Vec*}.}
\label{comparison_box2el_owl2vec}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}>{\raggedright}p{2cm}>{\raggedright}p{6.5cm}>{\raggedright\arraybackslash}p{6.5cm}@{}}
\toprule
& \textbf{Box2EL} & \textbf{OWL2Vec*} \\ \midrule
Core Idea & Embeds DL EL++ ontologies as \textbf{boxes} in vector space; \textbf{Neuro[Symbolic]} reasoning & Leverages ontologies/RDF graphs to generate vectors using \textbf{random walks + Word2Vec}; \textbf{Symbolic Neuro-Symbolic} reasoning \\
\addlinespace
Ontology Input & EL++ fragment ontologies & Full OWL ontologies (graph, lexical, and logical constructors) \\
\addlinespace
Embedding Type & Boxes/axis-aligned hyper-rectangles & Continuous vectors from word embeddings \\
\addlinespace
Strengths & Faithful to DL semantics; supports reasoning over complex relations & Combines structure, lexical, and logical information; strong performance on multiple datasets \\
\addlinespace
Weaknesses & Limited to EL++; requires DL-compliant ontologies & Embeddings approximate semantics; reasoning is probabilistic/pattern-based rather than formally guaranteed; embeddings are distributional, not geometric \\ 
\bottomrule
\end{tabular}
}
\end{table}

\subsection{Benchmark Datasets}
\label{subsec:benchmark_datasets}

There is a pressing need for standardized benchmark datasets for neurosymbolic reasoners to facilitate fair and consistent comparisons. Precisely, Singh et al.~\citep{Singh_Mutharaju_Bhatia_Tommasini} presented an overview of variations in neurosymbolic reasoning and evaluation approaches. Their overview reveals that similar works may differ significantly by employing distinct metrics and datasets to evaluate their contributions. For instance, the works of Makni et al.~\citep{noise-tolerant} and Ebrahimi et al.~\citep{pointer_nw} focus on RDFS entailment reasoning, aiming to replicate deductive reasoning processes. However, they adopt different metrics and datasets to assess the effectiveness and performance of their approaches. Such variations in evaluation criteria can lead to diverse insights and perspectives on the contributions within the field. Specifically, Makni et al.~\citep{noise-tolerant} used LUBM and a scientist dataset derived from DBpedia as benchmarks, evaluating performance with Precision, Recall, and F1 score. In contrast, Ebrahimi et al.~\citep{pointer_nw} used LUBM and synthetic data, employing exact matching accuracy as their metric.

The existing traditional benchmarks such as LUBM (Lehigh University Benchmark)~\citep{LUBM}, UOBM (University Ontology Benchmark)~\citep{UOBM} and OWL2Bench~\citep{owl2bench} lack suitability for evaluating neurosymbolic reasoners due to their narrow focus on conventional reasoning tasks. Traditional evaluations of reasoning systems often rely on metrics such as reasoning time, which may not align with the evaluation requirements of neurosymbolic reasoners. Although the ontologies of these benchmarks, along with those from the OWL Reasoner Evaluation (ORE) Competition~\citep{ORE}, can serve as initial datasets for neurosymbolic benchmarks, these datasets fall short of addressing the distinct challenges posed by neurosymbolic reasoning. 

To our knowledge, no benchmarks or evaluation frameworks have been designed to evaluate and compare neurosymbolic reasoning systems. Most reasoner evaluations are performed on different publicly available ontologies, including but not restricted to SNOMED CT\footnote{\url{https://bioportal.bioontology.org/ontologies/SNOMEDCT}}, Gene Ontology (GO)~\citep{GO} and GALEN~\citep{GALEN}, as well as other ontologies available in public repositories such as DBpedia~\citep{dbpedia}, YAGO~\citep{yago}, Wikidata~\citep{wikidata}, Claros\footnote{\url{https://www.clarosnet.org}}, NCBO Bioportal\footnote{\url{https://bioportal.bioontology.org/}} and AgroPortal\footnote{\url{http://agroportal.lirmm.fr/}}. However, these offer a limited set of ontologies for evaluation, which does not cover the full spectrum of possible scenarios.