pyscrapers.workers package

Submodules

pyscrapers.workers.audible module

Download books from audible

pyscrapers.workers.audible.add_book_ids(soup, book_ids) → bool

pyscrapers.workers.audible.add_links(soup, book_ids) → bool

pyscrapers.workers.audible.audible(_logger: Logger)

This does the downloads :param session: :param logger: :return:

pyscrapers.workers.audible.get_book_data(session: ExtSession, book_id: str) → OrderedDict

pyscrapers.workers.drumeo module

Download course material from drumeo

class pyscrapers.workers.drumeo.Course

Bases: object

This is an object representing one course

add_lesson(lesson)

Add a lesson to the course :param lesson: :return:

add_video(video, quality)

Add a video to the course :param video: :param quality: :return:

pyscrapers.workers.drumeo.download_course(course, session)

Download a course :param course: :param session: :return:

pyscrapers.workers.drumeo.get_course_details(course: Course, courses: bool, session)

Populate the Course type object :param course: :param courses: :param session: :return:

pyscrapers.workers.drumeo.get_course_urls(course, courses: bool, session)

pyscrapers.workers.drumeo.get_courses(pages, courses: bool, session)

Download the list of all the courses :param pages: :param courses: :param session: :return:

pyscrapers.workers.drumeo.get_link_re(courses: bool) → str

pyscrapers.workers.drumeo.get_members_url(courses: bool, number: int) → str

pyscrapers.workers.drumeo.get_number_of_pages(courses: bool, session: ExtSession) → int

Get the number of pages for all courses or pages :param courses: :param session: :return:

pyscrapers.workers.drumeo.get_url(courses: bool, page: int) → str

pyscrapers.workers.drumeo.get_videos(root, course, session)

Get all the videos pertaining to a certain course :param root: :param course: :param session: :return:

pyscrapers.workers.facebook module

download photos from facebook

pyscrapers.workers.facebook.scrape_facebook(user_id: str, session, url_set: UrlSet) → None

download photos from facebook :param user_id: :param session: :param url_set: :return:

pyscrapers.workers.getpocket module

getpocket.py

pyscrapers.workers.getpocket.getpocket_download(session: Session, _logger: Logger)

This does the heavy lifting :param session: :param _logger: :return:

pyscrapers.workers.instagram module

How does this work? When you fetch the page of a user on instagram you get an html with javascript embedded in it with a json object embedded in that. This json object describes the user, his id, his profile photo and the first 12 images for that user. If you want more you have to do a follow-up AJAX request to the server.

pyscrapers.workers.instagram.get_urls(logger, session, base, url_set, user_id)

pyscrapers.workers.instagram.is_rate_limit(response) → bool

Rate limit messages look like this: { “message”: “rate limited”, “status”: “fail” } :param response: :return:

pyscrapers.workers.instagram.scrape_instagram(user_id: str, session, url_set: UrlSet) → None

pyscrapers.workers.instagram_stories module

instagram_stories.py

pyscrapers.workers.instagram_stories.instagram_stories_download(_session: ExtSession, _logger: Logger)

This does the downloads :param session: :param logger: :return:

pyscrapers.workers.mamba_ru module

mamba_ru.py

pyscrapers.workers.mamba_ru.scrape_mambaru(user_id: str, session, url_set: UrlSet) → None

pyscrapers.workers.netflix module

netflix.py

pyscrapers.workers.netflix.netflix_download(_session: Session, _logger: Logger)

This does the downloads :param session: :param logger: :return:

pyscrapers.workers.sxyprn module

sxyprn.py

pyscrapers.workers.sxyprn.sxyprn_download(session: Session, logger: Logger)

This does the downloads :param session: :param logger: :return:

pyscrapers.workers.sxyprn.url_generator(url: str)

pyscrapers.workers.travelgirls module

travelgirls.py

pyscrapers.workers.travelgirls.scrape_travelgirls(user_id: str, session, url_set: UrlSet) → None

pyscrapers.workers.vk module

vk.py

pyscrapers.workers.vk.get_my_content(r)

the return from the server in vk is not a standard HTML. this is why we must cut it up and cant use the regular ‘get_real_content’ helper.

pyscrapers.workers.vk.get_total_images(logger, session, url, user_id)

pyscrapers.workers.vk.get_urls(base, got, json_obj, url_set)

pyscrapers.workers.vk.scrape_vk(user_id: str, session, url_set: UrlSet) → None

pyscrapers.workers.vk.yield_json_objs_and_base(r)

pyscrapers.workers.youtube_dl_handlers module

Module that handles the interaction with the youtube_dl library

References: - https://github.com/ytdl-org/youtube-dl/blob/master/README.md#embedding-youtube-dl - https://github.com/ytdl-org/youtube-dl/blob/3e4cedf9e8cd3157df2457df7274d0c842421945/youtube_dl/YoutubeDL.py#L137-L312

class pyscrapers.workers.youtube_dl_handlers.MyLogger

Bases: object

debug(msg)

error(msg)

warning(msg)

pyscrapers.workers.youtube_dl_handlers.youtube_dl_download_url(url: str) → None

pyscrapers.workers.youtube_dl_handlers.youtube_dl_download_urls(urls: list[str]) → None

pyscrapers.workers.youtube_dl_handlers.youtube_dl_handler() → None

Module contents

__init__.py