{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Merging Nodes Connected by Edges Labeled “I_CODE”, “N_Name”, \"R_exactMatch\", \"R_equivalentClass\", and \"PAYLOAD\"**\n",
        "\n",
        "This code takes a list of graph nodes and a list of graph edges as input. It then \"merges\" (see details below) all nodes connected by edges labeled “I_CODE”, “N_Name”, \"R_exactMatch\", \"R_equivalentClass\", and \"PAYLOAD\" and produces the modified node and edges lists as an output. \n",
        "\n",
        "After being processed by this code the [GBN node list](https://github.com/ncats/drug_rep/blob/main/Glioblastoma_Subgraph/GBN_Node_List.csv) contained 1,466 nodes and the [GBN edge list](https://github.com/ncats/drug_rep/blob/main/Glioblastoma_Subgraph/GBN_Edge_List.csv) contains 107,423 edges. A general overview of the steps taken to modify these lists is below:\n",
        " \n",
        "1. By iterating through the edge list, a 2D (Python) list of nodes to be merged is created (tbm_list). For each edge with the label “I_CODE”, “N_Name”, \"R_exactMatch\", \"R_equivalentClass\", or \"PAYLOAD\", the Source and Target nodes are recorded. If the Source node is already in tbm_list, then the Target node is appended to the Source node’s entry (and vice versa). If both the Source and Target nodes are already in tbm_list, then their respective entries are merged (if they are not already in the same entry). If neither the Source nor Target node are already in tbm_list, then a new entry is created containing them both. Thus, each entry in tbm_list contains a list of the Id numbers of nodes that all must be merged with each other. No node appears in more than one entry. For example, if  tbm_list = [[A, B], [C, D, E]], then nodes A and B must be merged into one node and nodes C, D, and E must be merged into another node. \n",
        "\n",
        "2. Next we iterate through tbm_list. For each entry in tbm_list, a new node is created (and appended to the end of the node list). The Id value of this new node is equal to the minimum of the Id values of the original nodes in the entry (the nodes being merged). The rest of the columns for this new node are equal to the concatenated values of the corresponding columns in the original nodes. Here an entry is also added to the dictionary fix_edge_dict for each original node, with the original node's original Id as the key and the new merged node's id as the value. This dictionary will be used to reconnect the remaining valid edges to the newly merged nodes. Finally, all of the original (pre-merge) nodes are deleted from the node list. \n",
        "\n",
        "3. Finally we iterate through the edge list again. If an edge has a Source value and a Target value that share an entry in tbm_list, the edge is deleted. If an edge has only one or the other (either a Source value present in tbm_list or a Target value present in tbm_list), then that value is updated to reflect the new merged node Id using fix_edge_dict. If an edge has a Source value and a Target value that are both present in tbm_list, but in different entries, then both the Source and Target values are updated to reflect the new merged node Id numbers using fix_edge_dict.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from google.colab import files\n",
        "import json\n",
        "import pandas as pd\n",
        "import os\n",
        "import numpy as np\n",
        "import networkx as nx\n",
        "import csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TDtTg2TCYDTF"
      },
      "outputs": [],
      "source": [
        "#read in edge list\n",
        "edge_list_df = pd.read_csv('/content/edge_list.csv') \n",
        "\n",
        "#read in node list \n",
        "node_list_df = pd.read_csv('/content/node_list.csv') "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#create empty to-be-merged list and Source/Target checks\n",
        "tbm_list = []\n",
        "Source = False\n",
        "Target = False\n",
        "\n",
        "#iterate through edge list\n",
        "for index, row in edge_list_df.iterrows():\n",
        "#if an edge has \"I_CODE\" or \"N_Name\" in the Neo4j \"type\" column,  \n",
        "  if row['type'] == 'I_CODE' or row['type'] == 'N_Name' or row['type'] == 'PAYLOAD' or row['type'] == 'R_equivalentClass' or row['type'] == 'R_exactMatch':\n",
        "  #see if it's \"Source\" value A is in the to-be-merged list already\n",
        "    if any(row['Source'] in subl for subl in tbm_list): \n",
        "      Source = True\n",
        "    else:\n",
        "      Source = False\n",
        "    #see if it's \"Target\" value B is in the to-be-merged list already\n",
        "    if any(row['Target'] in subl for subl in tbm_list):\n",
        "      Target = True\n",
        "    else:\n",
        "      Target = False\n",
        "    #if A is in the list but not B, add B to A's entry (and vice versa)\n",
        "    if Source == True and Target == False:\n",
        "      for i in range(len(tbm_list)):\n",
        "        if row['Source'] in tbm_list[i]:\n",
        "          tbm_list[i].append(row['Target'])\n",
        "    elif Source == False and Target == True:\n",
        "      for i in range(len(tbm_list)):\n",
        "        if row['Target'] in tbm_list[i]:\n",
        "          tbm_list[i].append(row['Source'])\n",
        "     #if both are in the list, merge those two entries together (if they aren't already the same)\n",
        "    elif Source == True and Target == True:\n",
        "      for i in range(len(tbm_list)):\n",
        "        if row['Source'] in tbm_list[i]:\n",
        "          sourcelist = tbm_list[i]\n",
        "          sourceIndex = i\n",
        "        if row['Target'] in tbm_list[i]:\n",
        "          targetlist = tbm_list[i]\n",
        "          targetIndex = i\n",
        "      if sourceIndex != targetIndex:\n",
        "        mergelist = list(set(sourcelist + targetlist))\n",
        "        tbm_list.append(mergelist)\n",
        "        tbm_list.pop(sourceIndex)\n",
        "        if sourceIndex < targetIndex:\n",
        "          tbm_list.pop(targetIndex-1)\n",
        "        if sourceIndex > targetIndex:\n",
        "          tbm_list.pop(targetIndex)\n",
        "    #if neither are in the list, create a new entry with A and B\n",
        "    elif Source == False and Target == False:\n",
        "      tbm_list.append([row['Source'], row['Target']])\n",
        "    Source = False\n",
        "    Target = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KV3lUzrAVTKA"
      },
      "outputs": [],
      "source": [
        "#make Id numbers the indices of the node list\n",
        "node_list_df_id_index = node_list_df.set_index(\"Id\", drop = False)\n",
        "\n",
        "#create fix edge dict and merge node dfs dataframe\n",
        "fix_edge_dict = {}\n",
        "merge_node_dfs = pd.DataFrame()\n",
        "min_id_tracker = 0 \n",
        "\n",
        "#iterate through to-be-merged list\n",
        "for i in range(len(tbm_list)):\n",
        "  merge_node_data = {}\n",
        "  #find min id value for new merged node id\n",
        "  for j in range(len(tbm_list[i])):\n",
        "    if j == 0:\n",
        "      min_id_tracker = tbm_list[i][j]\n",
        "    else:\n",
        "      min_id_tracker = min(min_id_tracker, tbm_list[i][j])\n",
        "    #create a new node (append to end of node list) with the minimum identity value as its identity, append the rest of the columns into one list\n",
        "    for c in list(node_list_df_id_index.columns):\n",
        "      if c != \"Id\":\n",
        "        if c in merge_node_data:\n",
        "          merge_node_data[c] = merge_node_data[c] + \", \" + str(node_list_df_id_index.at[tbm_list[i][j], c])\n",
        "        else:\n",
        "          merge_node_data[c] = str(node_list_df_id_index.at[tbm_list[i][j], c])\n",
        "  merge_node_data[\"Id\"] = min_id_tracker\n",
        "  merge_node_df = pd.DataFrame.from_dict(merge_node_data, orient='index').T\n",
        "  merge_node_dfs = merge_node_dfs.append(merge_node_df)\n",
        "\n",
        "  #add an entry to the fix edge dict for each original node, with the original node's original id as the key and the new merged node's id as the value\n",
        "  for k in range(len(tbm_list[i])):\n",
        "    fix_edge_dict[tbm_list[i][k]] = min_id_tracker\n",
        "    #delete all the original nodes from the node list \n",
        "    node_list_df_id_index.drop(labels = [tbm_list[i][k]], axis = 0, inplace = True)\n",
        "\n",
        "#append all merged nodes to node list\n",
        "node_list_df_id_index = node_list_df_id_index.append(merge_node_dfs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jFCazFt69aQK"
      },
      "outputs": [],
      "source": [
        "#iterate through edge list\n",
        "#If an edge has a Source value and a Target value that share an entry in tbm_list, the edge is deleted. \n",
        "#If an edge has only one or the other (either a Source value present in tbm_list or a Target value present in tbm_list), \n",
        "#then that value is updated to reflect the new merged node Id using fix_edge_dict. If an edge has a Source value and a Target \n",
        "#value that are both present in tbm_list, but in different entries, then both the Source and Target values are updated to reflect \n",
        "#the new merged node Id numbers using fix_edge_dict. \n",
        "\n",
        "for index, row in edge_list_df.iterrows():\n",
        "  #if statements eliminate case 1 (neither in tbm_list)\n",
        "  #case 2 (only source in tbm_list)\n",
        "  if any(row['Source'] in subl for subl in tbm_list) and not any(row['Target'] in subl for subl in tbm_list):\n",
        "    edge_list_df.at[index, \"Source\"] = fix_edge_dict[row['Source']]\n",
        "  #case 3 (only target in tbm_list)\n",
        "  if any(row['Target'] in subl for subl in tbm_list) and not any(row['Source'] in subl for subl in tbm_list):\n",
        "    edge_list_df.at[index, \"Target\"] = fix_edge_dict[row['Target']]\n",
        "  #case 4 and 5 (both in tbm_list)\n",
        "  if any(row['Source'] in subl for subl in tbm_list) and any(row['Target'] in subl for subl in tbm_list):\n",
        "    for i in range(len(tbm_list)):\n",
        "      #case 4 (both in same entry)\n",
        "      if row['Source'] in tbm_list[i] and row['Target'] in tbm_list[i]:\n",
        "        edge_list_df.drop([index], inplace = True)\n",
        "        break\n",
        "      #case 5 (both in different entries)\n",
        "      if row['Source'] in tbm_list[i] and row['Target'] not in tbm_list[i]:\n",
        "        edge_list_df.at[index, \"Source\"] = fix_edge_dict[row['Source']]\n",
        "      if row['Target'] in tbm_list[i] and row['Source'] not in tbm_list[i]:\n",
        "        edge_list_df.at[index, \"Target\"] = fix_edge_dict[row['Target']]"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Merge I_CODE and N_Name Duplicates.ipynb",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}