Need Help: YeGods.Org Data Extraction

nhb42 · 发表于 2019-12-9 16:14:19

本帖最后由 nhb42 于 2019-12-9 16:20 编辑

I want to make a dictionary from this website: https://yegods.org/, but it seems that all the entries are in this directory: https://yegods.org/deity/, and couldn't extract data from here with my little knowledge of data crawling... I made dictionaries from several web encyclopedias, but for this site, I was only able to extract the info pages, including the glossary page. If anyone helps me make this website mdx version, I would be grateful to him/her.

Here's the mdx version I made from the Glossary page... https://drive.google.com/open?id=1Bd9JRYlrLNqkWBZN9WK7j73xyylbV2_P [I have also added the glossary mdx in the attachment]

Please, someone, crawl that whole website and I will give more reward (米)...

perhapz · 发表于 2019-12-9 16:14:20

本帖最后由 perhapz 于 2019-12-17 14:54 编辑

try to use code below in console of developer tools of Chrome

fetch("https://yegods.org/search/suggestions/a", {"credentials":"include","headers":{"accept":"*/*","accept-language":"en-US,en;q=0.9","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","x-requested-with":"XMLHttpRequest"},"referrer":"https://yegods.org/","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"POST","mode":"cors"}).then(v=>v.json()).then(json=>json.matches.map(k=>k.slug));

复制代码

Repeat from a-z => deduplication => get data from https://yegods.org/deity/*

nhb42 · 发表于 2019-12-17 16:20:11

perhapz 发表于 2019-12-17 14:24
try to use code below in console of developer tools of Chrome

Repeat from a-z => deduplication => g ...

wait... after googling I managed to open the script in the console and I got a list of entries... Now How do I get the data...?

perhapz · 发表于 2019-12-17 17:53:38

本帖最后由 perhapz 于 2019-12-17 17:58 编辑

Repeat the script from a-z to get all entries
Remove duplications
Use some data crawling tool to send GET requests to all entries: https://yegods.org/deity/entry-name
Get whatever you need from response

nhb42 · 发表于 2019-12-17 17:57:14

perhapz 发表于 2019-12-17 14:24
try to use code below in console of developer tools of Chrome

Repeat from a-z => deduplication => g ...

thank you very much, I've extracted the data with IDM using the list...

perhapz · 发表于 2019-12-17 18:02:55

nhb42 发表于 2019-12-17 17:57
thank you very much, I've extracted the data with IDM using the list...

welcome, thanks for the reward

[悬赏] Need Help: YeGods.Org Data Extraction

最佳答案

点评

评分

点评

评分

点评