An intelligent, scalable platform for analyzing and predicting patterns in large social networks using Apache Spark, advanced machine learning, and state-of-the-art NLP.
GraphMind is a production-ready, modular data application designed to extract deep insights from social network graphs. Built for scalability and advanced AI integration, it processes massive datasets of user connections and interactions, applying cutting-edge algorithms to:
- Predict new friendships using Node2Vec embeddings and supervised machine learning.
- Detect influential users through advanced graph centrality measures.
- Analyze sentiment of user-generated content with Hugging Face Transformers.
- Discover communities using state-of-the-art graph partitioning algorithms.
- Visualize networks through an interactive, web-based dashboard.
- Dataset: ~88,000 edges, 4,039 nodes (scalable to millions)
- Tech Stack: Python, Apache Spark, GraphFrames, NetworkX, Scikit-Learn, Hugging Face Transformers, Streamlit
- Performance: Distributed processing with smart caching for lightning-fast user interactions
GraphMind answers complex questions about social network dynamics by integrating several advanced analytical modules:
-
Scalable Data Processing: Ingests and processes large graph datasets using Apache Spark for distributed computing.
-
AI-Powered Link Prediction:
- Uses Node2Vec to learn topological embeddings for each user
- Trains a machine learning model to predict potential new friendships
- Achieves 98.7% AUC score on test data
-
Advanced Sentiment Analysis:
- Employs pre-trained Hugging Face Transformer models
- Performs nuanced sentiment analysis on user posts
- Supports GPU acceleration (NVIDIA CUDA & AMD DirectML)
-
Community Detection:
- Uses Louvain algorithm to partition networks into communities
- Identifies clusters and their characteristics
-
Interactive Dashboard:
- Web-based UI with parameter tuning
- Real-time recommendations and visualizations
- Static graph generation for stable rendering
GraphMind is built on a modern, modular Python stack designed for scalability and cutting-edge analytics:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Streamlit Web Interface β
β (Interactive UI with Caching & Parameter Tuning) β
ββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β
ββββββββββββββββΌβββββββββββββββ
β β β
βΌ βΌ βΌ
ββββββββββββββββ ββββββββββββββββ ββββββββββββββββ
β Spark β β NetworkX β β ML Models β
β Ingestion β β Analytics β β (Node2Vec) β
ββββββββββββββββ ββββββββββββββββ ββββββββββββββββ
β β β
ββββββββββββββββΌβββββββββββββββ
β
βΌ
ββββββββββββββββββββββββ
β Advanced Analytics β
β β’ Link Prediction β
β β’ Sentiment (BERT) β
β β’ Visualization β
ββββββββββββββββββββββββ
Module Breakdown:
- Data Ingestion (Apache Spark): Distributed data loading and cleaning
- Graph Modeling (NetworkX): Flexible in-memory analysis
- Feature Engineering (Node2Vec): State-of-the-art graph embeddings
- Machine Learning (Scikit-Learn): Logistic regression for link prediction
- NLP (Hugging Face): DistilBERT for sentiment analysis
- UI (Streamlit): Interactive web interface with smart caching
- Python: 3.9 or higher
- Java: 8 or 11 (required by Apache Spark)
- Apache Spark: 3.5.x
- OS: Windows, macOS, or Linux
- Download Apache Spark from spark.apache.org
- Extract to a directory (e.g.,
C:\sparkon Windows or/opt/sparkon Linux)
Windows:
setx SPARK_HOME "C:\spark"
setx HADOOP_HOME "C:\spark"
setx PATH "%PATH%;%SPARK_HOME%\bin"macOS/Linux:
Add to ~/.bashrc or ~/.zshrc:
export SPARK_HOME=/opt/spark
export HADOOP_HOME=$SPARK_HOME
export PATH=$SPARK_HOME/bin:$PATHgit clone https://github.com/yourusername/GraphMind.git
cd GraphMind# Using Conda (recommended)
conda create -n graphmind python=3.9
conda activate graphmind
# OR using venv
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activatepip install -r requirements.txtPlace your network edge list in data/edges.csv with format:
source,destination
1,2
1,3
2,4
...
(Optional) Add sentiment data in data/sentiment_posts.csv
streamlit run src/app.pyThe app will open in your browser at http://localhost:8501
You'll see --- CACHE MISS --- messages as the system:
- Runs the Spark job
- Trains Node2Vec embeddings
- Trains link prediction model
- Downloads Hugging Face models
- Detects communities
After this initial setup, all subsequent interactions will be instant thanks to Streamlit's caching.
-
Link Prediction:
- Enter a User ID
- Click "Get Recommendations"
- View top predicted friendships with probability scores
-
Graph Visualization:
- Select a user
- Click "Generate Static Graph"
- View the user's local network
-
Parameter Tuning:
- Use sidebar sliders to adjust:
- Number of recommendations
- Node2Vec dimensions
- Walk parameters
- Models automatically retrain when parameters change
- Use sidebar sliders to adjust:
-
Community Analysis:
- Expand "Show Top Communities"
- View community size distribution
-
Sentiment Analysis:
- Expand "Run Advanced Sentiment Analysis"
- Analyzes posts from top influencers
- Displays average sentiment scores
GraphMind/
βββ data/ # Data files
β βββ edges.csv # Network edge list
β βββ sentiment_posts.csv # User posts for sentiment
βββ src/ # Source code
β βββ app.py # Main Streamlit application
β βββ module1_ingestion/
β β βββ ingest.py # Spark data ingestion
β βββ module2_core_analytics/
β β βββ distributed_analytics.py # GraphFrames analytics
β βββ module3_advanced_analytics/
β β βββ advanced_ml.py # ML & NLP functions
β βββ module4_visualization/
β βββ static_viz.py # Static graph generation
β βββ interactive_viz.py # Interactive visualizations
βββ generated/ # Output files (auto-created)
βββ requirements.txt # Python dependencies
βββ README.md # This file
Link Prediction Pipeline:
- Node2Vec random walk generation (2nd order biased walks)
- Word2Vec-style embedding learning (skip-gram)
- Feature engineering via Hadamard product
- Supervised classification with Logistic Regression
Sentiment Analysis:
- DistilBERT tokenization (max 512 tokens)
- Transformer inference with attention mechanism
- Binary classification (positive/negative)
- Aggregation to user-level sentiment
Community Detection:
- Louvain modularity optimization
- Multi-resolution clustering
- Link Prediction AUC: 0.9871 (98.71% accuracy)
- Graph Processing: Distributed across Spark executors
- Sentiment Inference: GPU-accelerated when available
- Cache persists within a single Streamlit session
- Cache invalidates when:
- App is restarted
- Code files are modified
- Hyperparameters are changed
- This is normal behavior - just accept the first-run wait time
"Module 'torch' has no attribute 'is_available'"
- Install torch-directml for AMD GPU:
pip install torch-directml - Code auto-detects AMD/NVIDIA/CPU
Spark temp directory warnings
- Harmless Windows file lock warnings
- Safe to ignore
Interactive visualization freezes browser
- Use static visualization instead
- Interactive mode only works for very small subgraphs
Sentiment scores all zero
- Check
sentiment_posts.csvformat - Ensure User IDs match graph node IDs
- Verify 'text' column has valid content
GraphMind is designed to be extensible. Potential future directions:
- Replace batch Spark with Spark Structured Streaming
- Ingest live data from Apache Kafka
- Continuous model updates
- Implement Plotly Dash Cytoscape for better interactive graphs
- 3D network rendering with WebGL
- Temporal network evolution animations
- Graph Neural Networks (GNN) for link prediction
- Multi-class sentiment analysis
- Anomaly detection for spam/bots
- Influence maximization algorithms
- Dockerize application
- Deploy on AWS/GCP/Azure
- Kubernetes orchestration
- RESTful API endpoints
- Connect to external knowledge bases
- Entity linking and disambiguation
- Semantic network enrichment
Core Technologies:
- Python 3.9+
- Apache Spark 3.5.x
- Java 8/11
Key Libraries:
pyspark>=3.5.0
networkx>=3.0
node2vec>=0.4.6
scikit-learn>=1.3.0
transformers>=4.30.0
torch>=2.0.0
streamlit>=1.28.0
matplotlib>=3.7.0
pandas>=2.0.0
Full list in requirements.txt
Contributions are welcome! Please:
- Fork the repository
- Create a feature branch (
git checkout -b feature/AmazingFeature) - Commit changes (
git commit -m 'Add AmazingFeature') - Push to branch (
git push origin feature/AmazingFeature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
Created as a demonstration of modern big data analytics and AI integration for social network analysis.
- Apache Spark community
- Hugging Face for pre-trained models
- NetworkX and Scikit-Learn teams
- Streamlit for the amazing web framework
For questions, issues, or collaboration opportunities, please open an issue on GitHub.
β If you find this project useful, please consider starring it!
