\documentclass[runningheads]{llncs} % \usepackage{bibnames} \usepackage{graphicx} \usepackage{tabularx} \usepackage{amssymb} \usepackage{subcaption} \usepackage{algorithm2e} \usepackage{booktabs} \usepackage{multirow} \usepackage{multicol} \usepackage{pifont} \usepackage{array,etoolbox} \usepackage[svgnames,table]{xcolor} \usepackage{geometry} \usepackage{algorithm} \usepackage{algpseudocode} \newcommand{\triplet}[1]{\textlangle{}{#1}\textrangle} \begin{document} \title{Supplementary Material: \\ Advancing Surgical Visual Question Answering with Scene Graph Knowledge} \titlerunning{Advancing Surgical Visual Question Answering with Scene Graph Knowledge} \authorrunning{} \maketitle \section{SSG Dataset} \label{sec:full_training} In this section, we first illustrate the statistics of the SSG-VQA dataset and show that it is more challenging and diverse than the prior works. Then, we show the process of assigning location attributes to the surgical objects. Finally, we visualize some samples of the scene graphs that we use to generate question-answer pairs. \begin{table}[!htp] \centering \caption{Dataset statistics comparison. SSG-VQA is more challenging and balanced as it includes more attributes and complexities in the questions.} \label{table:dataset_stats} \begin{tabular}{c@{\hspace{0.5cm}}c@{\hspace{0.5cm}}c@{\hspace{0.5cm}}c} \hline Dataset & EndoVis-18-VQA & Cholec80-VQA & SSG-VQA \\ \hline Average Length in words & 5.8 & 2.0 & 12.8 \\ Average $\#$Questions per scene & 5.0 & 6.5 & 38.9\\ \hline $\#$Color & 0 & 0 & 7\\ $\#$Type & 0 & 0 & 2\\ $\#$Location & 4 & 0 & 6\\ $\#$Object & 9 & 7 & 13\\ $\#$Relation & 13 & 0 & 16 \\ \hline \end{tabular} \end{table} \subsection{Dataset Statistics} As shown in Tab.~\ref{table:dataset_stats}, SSG-VQA dataset contains the more challenging and diverse question-answer pairs. Specifically, the average length of question is $38.9$ words compared to $5$ words and $6.5$ words from EndoVis-18-VQA and Cholec80-VQA. Also, by using the question engine to automatically generate the question-answer pairs, SSG-VQA contains a more diverse and different pattern of question-answer pairs for each surgical scene. \subsection{Location Attribute and Triplet Annotation} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \begin{figure}[] \centering \includegraphics[width=1.00\linewidth]{supp_merge_ds.png} \caption{\textbf{SSG-VQA Dataset - location attribute:} (a) we use triplet annotations for action-related question-answer generation; (b) we include a spatial attribute for each object, e.g., top-right, bottom-left; (c) we use tool and anatomy bounding boxes to generate question-answer pairs.} \label{location_attribute} \end{figure} During the question-answer generation, we assign the location attributes to the detected surgical objects, including instruments and anatomies. As shown in Fig.~\ref{location_attribute}, we split the image into $9$ spatial blocks and assign the location attribute based on the locations of the surgical object in a given block. In addition to the spatial relations among surgical objects, we also use triplet annotation to connect the surgical objects with the actions, as shown in Fig. \ref{location_attribute} (a). \subsection{Scene Graphs} \begin{figure}[] \centering \includegraphics[width=0.9\linewidth]{supp_data.png} \caption{\textbf{SSG-VQA Dataset - scene graphs:} The questions are generated by exploiting the surgical scene graph. The surgical scene graph consists of spatial attributes and action relations.} \label{sample_scene} \end{figure} \begin{figure}[] \centering \includegraphics[width=1.00\linewidth]{supp_dataset.png} \caption{\textbf{SSG-VQA Dataset - different complexities:} We generate different type of question-answer pairs for each surgical scene image. The text in red indicates the reasoning step in the image, e.g., the fifth question requires two steps of reasoning to solve, therefore categorized into single-and type.} \label{complexity_qa} \end{figure} In this section, we show some scene graph samples that we use to generate question-answer pairs. As shown in Fig. \ref{sample_scene}, the surgical semantic scene graphs are directed graphs where the nodes represent the surgical objects and the edges represent the spatial and action relation between nodes. Also, we assign attributes to each surgical object, e.g., type, color, and location of the objects. Note that the spatial relations are fully connected among the objects. We show the important relations for a simpler demonstration. \subsection{Complexities of Question-answer Pairs} As the SSG-VQA dataset is generated with a question engine and question templates automatically, we control the complexity of question-answer pairs by altering the question templates. The question-answer pairs with different complexities also offer the diagnostic ability to pinpoint the weakness of the VQA model. As shown in Fig. \ref{complexity_qa}, we have $5$ kinds of question-answer pairs with $3$ kinds of complexities. Here, the complexity is measured based on how many reasoning steps are required to answer the question. For example, the fifth question-answer in Fig. \ref{complexity_qa} requires two steps of reasoning to solve, highlighted in \textcolor{red}{red}. As a result, this question is more complex than the others. \begin{figure}[] \centering \includegraphics[width=1\linewidth]{supp_answer2.png} \caption{\textbf{Qualitative Results:} It shows that the questions from our SSG-VQA dataset require visual reasoning to solve. Also, our SSG-VQA-Net with scene knowledge outperforms the prior baselines in terms of complex questions. } \label{qa:reasoning steps} \end{figure} \begin{figure}[] \centering \includegraphics[width=1.00\linewidth]{supp_triplet_qualitative.png} \caption{\textbf{Qualitative Results:} Comparison of SSG-VQA-Net with baseline surgical VQA models on the question-answer pairs based on surgical action triplets.} \label{qa:triplet} \end{figure} \begin{figure}[] \centering \includegraphics[width=1\linewidth]{supp_fig_qualitative.png} \caption{\textbf{Qualitative Results:} Comparison of SSG-VQA-Net with baseline surgical VQA models on the question-answer pairs based on spatial, color, counting, anatomy, and tool presence attributes.} \label{qa:attributes} \end{figure} \section{Qualitative Results} In this section, we compare our SSG-VQA-Net to the state-of-the-art surgical VQA models in terms of the question that require the understanding of attributes (Sec. \ref{supp_sec:attribute}), action triplet occurrence (Sec. \ref{supp_sec:triplet}) and reasoning steps (Sec. \ref{supp_sec:reasoning}) to solve. \subsection{Reasoning Steps} \label{supp_sec:reasoning} Fig.~\ref{qa:reasoning steps} shows that SSG-VQA-Net can perform multiple steps of reasoning to generate the correct answer to complex questions. Compared to the prior works, it achieves better results when the questions are more complex. \subsection{Action Triplets} \label{supp_sec:triplet} We also compare the SSG-VQA-Net model to the prior works on the questions that require an understanding of surgical action triplet. Fig.~\ref{qa:triplet} shows that the SSG-VQA-Net model can answer the questions that are generated based on the triplet annotations. \subsection{Attributes} \label{supp_sec:attribute} As shown in Fig.~\ref{qa:attributes}, the SSG-VQA-Net model outperforms the prior works when the questions contain attributes such as location and color. For example, the second subfigure in Fig.~\ref{qa:attributes} shows that the SSG-VQA-Net model understands the location attributes such as ``top-mid'' and ``bottom-mid'' to resolve the question. It contains more detailed knowledge about the surgical scene than the other methods. \section{Performance of the detection models used in SSG-QA-Dataset and SSG-QA-Net} \subsection{SSG-VQA dataset} For the construction of the SSG-VQA dataset, we train a \textcolor{red}{yolo} model on the m2cai16-tool-locations dataset and \textcolor{red}{deeplab} model on the CholecSeg8K dataset to construct the SSG-VQA dataset. We use these trained models to run the inference on the CholecT45 dataset, generating pseudo labels, i.e., segmentations for the anatomies and the bounding boxes for the tools. The segmentation output is further converted to the bounding boxes. We then use these bounding boxes to construct the scene graph and generate the question-answer pairs. For the testing set, we manually correct the predicted bounding boxes and the labels, ensuring the scene graphs are clean. Therefore, the generated question-answer pairs for testing are valid and reliable. In the table below, we show the performance of the \textcolor{red}{yolo} model and \textcolor{red}{deeplab} model to justify the quality of our SSG-VQA dataset. As shown in Tab.~\ref{table:yolo_ssgvqa}, the \textcolor{red}{yolo} model can accurately detect various surgical tools. The model demonstrates high precision, suggesting a low rate of false positives. This efficiency in detecting instruments shows the model's strength to construct high quality SSG-VQA dataset. \begin{table}[!htp] \centering \caption{Performance of the \textcolor{red}{yolo} model used in the construction of the SSG-VQA dataset.} \label{table:yolo_ssgvqa} \begin{tabular}{c@{\hspace{1cm}}c} \hline Metrics & \\ \hline Precision & 0.9521 \\ Recall & 0.9358 \\ \hline \end{tabular} \end{table} As shown in Tab.~\ref{table:deeplab_ssgvqa} below, the anatomy segmentation model,\textcolor{red}{deeplabv3}, achieves promising results in accurately segmenting most of the anatomy classes. However, a notable exception is observed in the segmentation of the cystic duct. We extract the bounding boxes from the segmentations to construct the scene graph. \begin{table}[!htp] \centering \caption{Performance of the segmentation model, \textcolor{red}{deeplabv3}, that we use to construct SSG-VQA dataset.} \label{table:deeplab_ssgvqa} \begin{tabular}{c@{\hspace{1cm}}c} \hline Class & Dice Score \\ \hline Abdominal wall cavity & 0.8477 \\ Liver & 0.9422 \\ Gut & 0.4369 \\ Omentum & 0.8658 \\ Gallbladder & 0.7436 \\ Cystic duct & 0.0146 \\ \hline \end{tabular} \end{table} \subsection{SSG-VQA-Net} In training the SSG-VQA-Net, a key step is training a new \textcolor{blue}{yolo} model using pseudo bounding boxes, generated from the prior \textcolor{red}{yolo} and \textcolor{red}{deeplab} models, simulating real-world performance in surgical object detection. This trained \textcolor{blue}{yolo} model is the core component of SSG-VQA-Net to serve as the object detection method to integrate scene graph knowledge. The design of SSG-VQA-Net relies solely on this \textcolor{blue}{yolo} model's performance without depending on additional segmentation or relationship prediction models. \begin{figure}[htbp] \centering \includegraphics[width=1.0\linewidth]{supp_confusion_f1.png} \caption{(a) Confusion matrix of the yolo model for SSG-VQA-Net. (b) F1 score variation with different confidence thresholds.} \label{confusion_f1} \end{figure} Fig.~\ref{confusion_f1} analyzes the detection results of this \textcolor{blue}{yolo} model in the validation set, which includes the confusion matrix for each class and the F1 score variation with different confidence thresholds. Fig.~\ref{confusion_f1} (a) shows the consistent mis-classification of the cystic duct as background, highlighting it as the most challenging class to detect. Furthermore, Fig.~\ref{confusion_f1} details the rationale behind our selection of hyper-parameters for the detection model, particularly choosing a confidence score threshold of 0.2 for bounding box generation. This choice ensures balanced detection accuracy and false positive rates, directly impacting the overall performance of the SSG-VQA-Net. \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{supp_pr.png} \caption{Class-wise average precision for trained detection model (yolo) that builds the SSG-VQA-Net.} \label{PR} \end{figure} We also provide a detailed analysis of the \textcolor{blue}{yolo} model's performance with class-wise Average Precision (AP) in Fig.~\ref{PR} and qualitative results on the validation set in Fig.~\ref{detection_qualitative}. The class-wise AP offers a detailed representation of the model's accuracy across different classes. It demonstrates the strengths and weaknesses that might require further refinement, especially in low AP classes. %These results visually represent the model's detection capabilities in real-world scenarios. \begin{figure}[htbp] \centering \includegraphics[width=1.0\linewidth]{supp_qualitative.jpg} \caption{Qualitative results of trained object detection model, \textcolor{blue}{yolo}, used in the SSG-VQA-Net.} \label{detection_qualitative} \end{figure} \end{document}