Skip to content
Dhruv-0905Public

About

AI-Powered Social Network Analytics with Apache Spark & Transformers

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

🧠 GraphMind: AI-Powered Social Network Analytics

An intelligent, scalable platform for analyzing and predicting patterns in large social networks using Apache Spark, advanced machine learning, and state-of-the-art NLP.


Project Overview

GraphMind is a production-ready, modular data application designed to extract deep insights from social network graphs. Built for scalability and advanced AI integration, it processes massive datasets of user connections and interactions, applying cutting-edge algorithms to:

  • Predict new friendships using Node2Vec embeddings and supervised machine learning.
  • Detect influential users through advanced graph centrality measures.
  • Analyze sentiment of user-generated content with Hugging Face Transformers.
  • Discover communities using state-of-the-art graph partitioning algorithms.
  • Visualize networks through an interactive, web-based dashboard.

Key Highlights

  • Dataset: ~88,000 edges, 4,039 nodes (scalable to millions)
  • Tech Stack: Python, Apache Spark, GraphFrames, NetworkX, Scikit-Learn, Hugging Face Transformers, Streamlit
  • Performance: Distributed processing with smart caching for lightning-fast user interactions

What It Does

GraphMind answers complex questions about social network dynamics by integrating several advanced analytical modules:

  1. Scalable Data Processing: Ingests and processes large graph datasets using Apache Spark for distributed computing.

  2. AI-Powered Link Prediction:

    • Uses Node2Vec to learn topological embeddings for each user
    • Trains a machine learning model to predict potential new friendships
    • Achieves 98.7% AUC score on test data
  3. Advanced Sentiment Analysis:

    • Employs pre-trained Hugging Face Transformer models
    • Performs nuanced sentiment analysis on user posts
    • Supports GPU acceleration (NVIDIA CUDA & AMD DirectML)
  4. Community Detection:

    • Uses Louvain algorithm to partition networks into communities
    • Identifies clusters and their characteristics
  5. Interactive Dashboard:

    • Web-based UI with parameter tuning
    • Real-time recommendations and visualizations
    • Static graph generation for stable rendering

Demonstration

Demo GIF

Architecture

GraphMind is built on a modern, modular Python stack designed for scalability and cutting-edge analytics:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    Streamlit Web Interface                  β”‚
β”‚          (Interactive UI with Caching & Parameter Tuning)   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                       β”‚
        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
        β”‚              β”‚              β”‚
        β–Ό              β–Ό              β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   Spark      β”‚ β”‚  NetworkX    β”‚ β”‚  ML Models   β”‚
β”‚  Ingestion   β”‚ β”‚  Analytics   β”‚ β”‚  (Node2Vec)  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
        β”‚              β”‚              β”‚
        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                       β”‚
                       β–Ό
            β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
            β”‚  Advanced Analytics  β”‚
            β”‚  β€’ Link Prediction   β”‚
            β”‚  β€’ Sentiment (BERT)  β”‚
            β”‚  β€’ Visualization     β”‚
            β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Module Breakdown:

  1. Data Ingestion (Apache Spark): Distributed data loading and cleaning
  2. Graph Modeling (NetworkX): Flexible in-memory analysis
  3. Feature Engineering (Node2Vec): State-of-the-art graph embeddings
  4. Machine Learning (Scikit-Learn): Logistic regression for link prediction
  5. NLP (Hugging Face): DistilBERT for sentiment analysis
  6. UI (Streamlit): Interactive web interface with smart caching

Installation & Setup

Prerequisites

  • Python: 3.9 or higher
  • Java: 8 or 11 (required by Apache Spark)
  • Apache Spark: 3.5.x
  • OS: Windows, macOS, or Linux

Step 1: Set Up Apache Spark

Download and Extract

  1. Download Apache Spark from spark.apache.org
  2. Extract to a directory (e.g., C:\spark on Windows or /opt/spark on Linux)

Configure Environment Variables

Windows:

setx SPARK_HOME "C:\spark"
setx HADOOP_HOME "C:\spark"
setx PATH "%PATH%;%SPARK_HOME%\bin"

macOS/Linux: Add to ~/.bashrc or ~/.zshrc:

export SPARK_HOME=/opt/spark
export HADOOP_HOME=$SPARK_HOME
export PATH=$SPARK_HOME/bin:$PATH

Step 2: Clone the Repository

git clone https://github.com/yourusername/GraphMind.git
cd GraphMind

Step 3: Create Virtual Environment

# Using Conda (recommended)
conda create -n graphmind python=3.9
conda activate graphmind

# OR using venv
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Step 4: Install Dependencies

pip install -r requirements.txt

Step 5: Prepare Data

Place your network edge list in data/edges.csv with format:

source,destination
1,2
1,3
2,4
...

(Optional) Add sentiment data in data/sentiment_posts.csv


Usage

Launch the Application

streamlit run src/app.py

The app will open in your browser at http://localhost:8501

First Run Behavior

⚠️ IMPORTANT: The first run will be SLOW (5-10 minutes)

You'll see --- CACHE MISS --- messages as the system:

  • Runs the Spark job
  • Trains Node2Vec embeddings
  • Trains link prediction model
  • Downloads Hugging Face models
  • Detects communities

After this initial setup, all subsequent interactions will be instant thanks to Streamlit's caching.

Using the Dashboard

  1. Link Prediction:

    • Enter a User ID
    • Click "Get Recommendations"
    • View top predicted friendships with probability scores
  2. Graph Visualization:

    • Select a user
    • Click "Generate Static Graph"
    • View the user's local network
  3. Parameter Tuning:

    • Use sidebar sliders to adjust:
      • Number of recommendations
      • Node2Vec dimensions
      • Walk parameters
    • Models automatically retrain when parameters change
  4. Community Analysis:

    • Expand "Show Top Communities"
    • View community size distribution
  5. Sentiment Analysis:

    • Expand "Run Advanced Sentiment Analysis"
    • Analyzes posts from top influencers
    • Displays average sentiment scores

Project Structure

GraphMind/
β”œβ”€β”€ data/                          # Data files
β”‚   β”œβ”€β”€ edges.csv                  # Network edge list
β”‚   └── sentiment_posts.csv        # User posts for sentiment
β”œβ”€β”€ src/                           # Source code
β”‚   β”œβ”€β”€ app.py                     # Main Streamlit application
β”‚   β”œβ”€β”€ module1_ingestion/         
β”‚   β”‚   └── ingest.py              # Spark data ingestion
β”‚   β”œβ”€β”€ module2_core_analytics/
β”‚   β”‚   └── distributed_analytics.py  # GraphFrames analytics
β”‚   β”œβ”€β”€ module3_advanced_analytics/
β”‚   β”‚   └── advanced_ml.py         # ML & NLP functions
β”‚   └── module4_visualization/
β”‚       β”œβ”€β”€ static_viz.py          # Static graph generation
β”‚       └── interactive_viz.py     # Interactive visualizations
β”œβ”€β”€ generated/                     # Output files (auto-created)
β”œβ”€β”€ requirements.txt               # Python dependencies
└── README.md                      # This file

Technical Details

Algorithms Used

Link Prediction Pipeline:

  1. Node2Vec random walk generation (2nd order biased walks)
  2. Word2Vec-style embedding learning (skip-gram)
  3. Feature engineering via Hadamard product
  4. Supervised classification with Logistic Regression

Sentiment Analysis:

  1. DistilBERT tokenization (max 512 tokens)
  2. Transformer inference with attention mechanism
  3. Binary classification (positive/negative)
  4. Aggregation to user-level sentiment

Community Detection:

  • Louvain modularity optimization
  • Multi-resolution clustering

Performance Metrics

  • Link Prediction AUC: 0.9871 (98.71% accuracy)
  • Graph Processing: Distributed across Spark executors
  • Sentiment Inference: GPU-accelerated when available

Important Notes & Troubleshooting

Cache Behavior

  • Cache persists within a single Streamlit session
  • Cache invalidates when:
    • App is restarted
    • Code files are modified
    • Hyperparameters are changed
  • This is normal behavior - just accept the first-run wait time

Common Issues

"Module 'torch' has no attribute 'is_available'"

  • Install torch-directml for AMD GPU: pip install torch-directml
  • Code auto-detects AMD/NVIDIA/CPU

Spark temp directory warnings

  • Harmless Windows file lock warnings
  • Safe to ignore

Interactive visualization freezes browser

  • Use static visualization instead
  • Interactive mode only works for very small subgraphs

Sentiment scores all zero

  • Check sentiment_posts.csv format
  • Ensure User IDs match graph node IDs
  • Verify 'text' column has valid content

Future Enhancements

GraphMind is designed to be extensible. Potential future directions:

Real-Time Streaming

  • Replace batch Spark with Spark Structured Streaming
  • Ingest live data from Apache Kafka
  • Continuous model updates

Advanced Visualization

  • Implement Plotly Dash Cytoscape for better interactive graphs
  • 3D network rendering with WebGL
  • Temporal network evolution animations

Enhanced ML Features

  • Graph Neural Networks (GNN) for link prediction
  • Multi-class sentiment analysis
  • Anomaly detection for spam/bots
  • Influence maximization algorithms

Deployment & Scaling

  • Dockerize application
  • Deploy on AWS/GCP/Azure
  • Kubernetes orchestration
  • RESTful API endpoints

Knowledge Graph Integration

  • Connect to external knowledge bases
  • Entity linking and disambiguation
  • Semantic network enrichment

Tech Stack & Dependencies

Core Technologies:

  • Python 3.9+
  • Apache Spark 3.5.x
  • Java 8/11

Key Libraries:

pyspark>=3.5.0
networkx>=3.0
node2vec>=0.4.6
scikit-learn>=1.3.0
transformers>=4.30.0
torch>=2.0.0
streamlit>=1.28.0
matplotlib>=3.7.0
pandas>=2.0.0

Full list in requirements.txt


Contributing

Contributions are welcome! Please:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/AmazingFeature)
  3. Commit changes (git commit -m 'Add AmazingFeature')
  4. Push to branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.


Authors

Created as a demonstration of modern big data analytics and AI integration for social network analysis.


Acknowledgments

  • Apache Spark community
  • Hugging Face for pre-trained models
  • NetworkX and Scikit-Learn teams
  • Streamlit for the amazing web framework

Contact

For questions, issues, or collaboration opportunities, please open an issue on GitHub.


⭐ If you find this project useful, please consider starring it!

About

AI-Powered Social Network Analytics with Apache Spark & Transformers

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages