WikiMapping connects a Python data pipeline, a persistent graph in SQLite, a Flask API, and a JavaScript interface. The work spans data modeling, graph algorithms, API design, visualization, and the tradeoffs involved in making a large dataset usable in a browser.
01 · Crawling and graph modeling
The Python crawler uses Requests to fetch pages and Beautiful Soup to extract article links. Each page receives an integer ID; each hyperlink becomes a directed edge from its source page to its destination. An incoming link is a parent connection, and an outgoing link is a child connection.
SQLite stores a pages table with a unique title, ID, and exploration state, plus an edges table with unique source/target pairs. Duplicate inserts are ignored, and exploration state records completed or failed fetches so progress survives between sessions.
02 · Making the dataset manageable
The full crawl contains 10M+ pages and 350M+ directed links. A separate export tool ranks articles by incoming links in the source graph and retains the selected pages and the edges between them. SQLite performs grouping and sorting with disk-backed temporary storage.
The exporter opens the source read-only, preserves schema constraints and indexes, and can compact retained IDs to reduce memory use. The web server currently loads the 100,000-article database, keeping the interactive application separate from the much larger crawl.
03 · Fast graph access and a focused API
At startup, Flask loads title-to-ID maps and both forward and reverse adjacency lists. An article lookup uses a dictionary; inspecting its neighborhood or traversing a path reads the relevant lists without issuing a database query for each hop.
JSON endpoints expose graph subsets, article neighborhoods, rankings, autocomplete, random articles, and shortest paths. Inputs accept article titles or English Wikipedia URLs, normalize spaces and encoding, and return explicit errors for invalid inputs or missing articles.
04 · Shortest paths and playable runs
Bidirectional breadth-first search follows outgoing edges from the start and incoming edges from the target. It expands a complete layer from the smaller frontier, records predecessor maps, and reconstructs a shortest directed path when the searches meet. Worst-case traversal is O(V + E), with O(V) search state.
Speedruns turn that search into a game: the optimal route stays hidden while a player follows child links. The browser caches visited neighborhoods, supports backtracking, and compares the final route with the computed minimum. Shared challenge URLs store the two article choices.
05 · Interactive graph visualization
Graphology holds the browser's graph, ForceAtlas2 calculates a force-directed layout, and Sigma.js renders it using WebGL. The overview requests 50 highly linked articles by default, with the API supporting up to 500. Node size scales with incoming-link count using a sublinear power so large hubs remain readable.
Hover and selection highlight relationships; a draggable panel exposes parents and children. Article search uses a separate neighborhood view with directed arrows and lists of every returned connection. Those lists remain available if WebGL cannot initialize.
06 · Search and interface reliability
Autocomplete uses a cached, popularity-ordered title index. Results prioritize exact matches, then prefixes, then substrings. Article lists render in batches, while request cancellation prevents older responses from replacing a newer search.
The frontend uses HTML, CSS, and JavaScript modules, with Vite bundling each page. During development, Vite proxies API requests to Flask; after a build, Flask serves the generated pages and assets. Keyboard controls, visible focus states, loading feedback, and retry actions support the main flows.
07 · Verification and observability
Python tests use small SQLite fixtures to check ranking, URL normalization, validation, directed paths, disconnected nodes, and database exports. Pathfinding is also compared against a reference breadth-first search on generated graphs.
Playwright smoke checks exercise article search and speedruns, including autocomplete, shared challenges, and failed requests. The path API reports visited nodes, checked edges, expanded layers, and elapsed time, making the search's work visible in the interface.
08 · Engineering tradeoffs
Keeping both adjacency directions in memory uses additional RAM in exchange for quick neighborhood access and reverse traversal. The lists are indexed by page ID, so compact exports avoid allocating space for gaps in the original IDs.
Limiting the overview bounds the amount of rendering and layout work. ForceAtlas2 currently runs synchronously in the browser; moving layout into a worker or precomputing positions would be a next step for larger views. The cached autocomplete index still uses a linear scan, leaving room for a dedicated text index.
