nitowa a614991ff0 working graph implementation and improved shell scripts		2 年前
config/db	progress on mapping data, finding clusters, probably inefficient	2 年前
spark-packages	working graph implementation and improved shell scripts	2 年前
src/spark	working graph implementation and improved shell scripts	2 年前
.gitignore	working graph implementation and improved shell scripts	2 年前
README.md	working graph implementation and improved shell scripts	2 年前
clean.py	progress on mapping data, finding clusters, probably inefficient	2 年前
settings.json	working graph implementation and improved shell scripts	2 年前
setup.py	progress on mapping data, finding clusters, probably inefficient	2 年前
small_test_data.csv	progress on mapping data, finding clusters, probably inefficient	2 年前
start_services.sh	working graph implementation and improved shell scripts	2 年前
submit.sh	working graph implementation and improved shell scripts	2 年前
submit_graph.sh	working graph implementation and improved shell scripts	2 年前

Project Description

TODO

Installation

Prerequisites:

Python3
Apache spark 3.2 (https://spark.apache.org/downloads.html)
Cassandra DB (https://cassandra.apache.org//index.html, locally the docker build is recommended: https://hub.docker.com//cassandra)

For the graph implementation specifically you need to install graphframes manually since the official release is incompatible with spark 3.x (pull request pending). A prebuilt copy is supplied in the spark-packages directory.

graphframes (https://github.com/eejbyfeldt/graphframes/tree/spark-3.3)

Setting up

Modify settings.json to reflect your setup. If you are running everything locally you can use start_services.sh to turn everything on in one swoop.
Load the development database by running python3 setup.py from the project root.
Start the spark workload by either running submit.sh (slow) or submit_graph.sh (faster)