degoogedtube 1.0.2
degooged-tube
An adless, accountless, trackerless youtube interface with a sub box
INSTALLATION
ABOUT
CLI-USAGE
API-USAGE
DEVELOPMENT
SCRAPING
SANATIZATION
YOUTUBE-API-WRAPPERS
DATA-PIPELINE
INSTALLATION
Requires mpv for terminal playback
Install degooged-tube via pypi using pip:
pip install degooged-tube
ABOUT
Allows for youtube to be used from a terminal with no ads, without an account account, and while maintaining subbox functionality.
Can be embedded into other projects using the API in degooged_tube/ytApiHacking/__init__.py
All youtube API scraping is done internally, with the exception of getting the streaming link for videos, which is done with yt-dlp
CLI-USAGE
Launch in terminal with command
degooged-tube
Follow prompts to create a new 'user' (info is stored locally, has nothing to do with a youtube account), I recommend adding a few subs initially as it makes getting use to the cli easier.
General Interface
CLI is interactive, options are displayed on the bottom of the screen and can be selected by entering the letter in the brackets, IE w for (w)atch.
Some options IE (w)atch will show you a numbered list then prompt you for which number you want to watch, others IE (p)revious/(n)ext page will just preform the action.
API-USAGE
degooged_tube/ytApiHacking/__init__.py contains getter functions to return dataclasses, aswell as YtapiList of dataclasses containing scraped data, the data in the dataclasses is self explainitory and the YtapiList can be indexed like normal python lists, throwing IndexError if value is out of bounds. There is also YtApiList.getPaginated(pageNum, pageSize) which returns a list of elements (pageSize is upper bound)
DEVELOPMENT
To build for development run:
git clone https://github.com/PrinceOfPuppers/degooged-tube
cd degooged-tube
pip install -e .
This will build and install sync-dl in place, allowing you to work on the code without having to reinstall after changes.
Automated Testing
python test.py [options]
Will run all unit and integration tests, options are -u and -i to only run the unit/integration tests respectively.
SCRAPING
youtube will sometimes change their API, most of the time this is not an issue as the scraping engine is robust, however sometimes scrapers will be broken, as such here is a guide on how they work and how to repair them
The configuration of youtube-api hacking exists in degooged_tube/ytApiHacking/controlPanel.py
ScrapeJsonTree
The core of degooged-tube is a json scraping engine which is robust against changing apis, and easily repairable in the event of a scraper breaking.
It works by defining nodes in a json tree down to the data you wish to collect, you can specify as many or as few nodes as is needed.
Examples
"greetings":{
"hello": "there",
"howdy": {"name": "partner"},
"hi":[
{
"name": "alice",
"favColor": "blue",
"coolness": "megaRad"
},
{
"name": "bob",
"favColor": "green",
"coolness": 11
},
{
"name": "carol",
"favColor": "purple",
"coolness": ["super", "duper", "cool"]
},
{
"name": "dave",
"favColor": "purple"
}
]
}
Example Scrapers:
ScrapeNth("howdy") (N defaults to 1 or first)
-> we would just get {"howdy": {"name": "partner"}}
ScrapeNth("howdy", collapse = True)
-> we would just get {"name": "partner"}, collapse just returns the data for a node, rather than key value pairs
ScrapeAll("name")
-> we would just get {"name": ["partner", "alice", "bob", "carol", "dave"]}
ScrapeNth("hi", [ScrapeNth("name" )])
-> we would get {"hi": {"name": "alice"}}" because we would first get "hi", and then scrape for the first "name" in said data
ScrapeNth("hi", [ScrapeNth("name", collapse=True)])
-> we would get {"hi": "alice"}", "name" has been collapsed
ScrapeNth("hi", [ScrapeNth("name"), ScrapeNth("favColor")])
-> we would get {"hi": [{"name": "alice"}, {"favColor":"blue"}]}
ScrapeAll("name", collapse=True, dataCondition=lambda data: data[1] = 'a')
-> we would get ['carol', 'dave'] as dataCondition filters all data which fail the condition
ScrapeUnion([ScrapeNth("missing key",[], collapse=True), ScrapeNth("hello",[], collapse = True)])
-> we would get "there", however if "missing key" was present we would get the data for that instead
For more examples using all nodes and arguments, see degooged_tube/ytApiHacking/controlPanel.py.
For more involved examples of how this system is used in practice, see degooged_tube/tests/unitTests.py.
Node Types
ScrapeNth:
Scrapes and returns the data for n'th occurrence of the key
ScrapeAll:
Scrapes and returns the a list of data for all occurrence of the key
ScrapeLongest:
Scrapes and returns the data with largest len(data) for all occurrence of the key
ScrapeUnion([Node1, Node2, ...]):
Will scrape for each node and return the data first one which matches
ScrapeAllUnion/ScrapeAllUnionNode:
Used together as such: ScrapeAllUnion([ScrapeAllUnionNode, ScrapeAllUnionNode, ...]), will return a list containing all matches for any of the ScrapeAllUnionNode
ScrapeElement:
The generic type of any scrape node
SANATIZATION
After scraping the data is passed into the .fromData() constructor of a dataclass (ie VideoInfo.fromData()), all of which are located in degooged_tube/ytApiHacking/controlPanel.py where it is sanitized and any missing data can be filled with placeholder data
YOUTUBE-API-WRAPPERS
There are 3 Objects which which wrap the youtube-api and allow for easy access to resources
YtInitalPage
Constructed using YtInitalPage.fromUrl(url: str) gets a page and parses it for continuation chains and initial data
The method page.scrapeInitalData(dataFmt: Union[ScrapeElement, list[ScrapeElement]]) will run scrapeJsonTree on any initial data found and return it as a dict
YtApiList
Constructed from YtInitalPage, it behaves like a list and making calls to a continuation chain as items are indexed.
The continuation chain is decided using the apiUrl, and by duck-typing based on the scrapeFmt (following only chains which match the dataFmt).
Contains YtContIter, an iterator which wraps the continuation chains, deals the requests, as well as implements the aforementioned duck-typing
DATA-PIPELINE
pipeline under the hood (note this is not a callstack, for example processInfo is called in getter, its more a sequence diagram):
youtube-api
||
url/query -------------> getter --> YtInitalPage --+ +--> scrapeInitalData -------> scrapeJsonTree --> processInfo ---+---> fromData --> dataclass
| +-----+ |
+--> YtInitalPage --> getter -------------------+ +--> YtApiList/YtContIter ---> scrapeJsonTree --> onExtend ------+
||
youtube-api
Legend:
url/query:
url or search query
YtInitalPage:
created using fromUrl()
getter:
functions in degooged_tube/ytApiHacking/__init__.py prepended by get
scrapeInitalData:
a method of YtInitalPage is passes the requisite format to by getter
YtApiList\YtContIter:
YtApiList is instantiated by getter and passes the requisite format to by getter, YtContIter in instantiated as a component of YtApiList
scrapeJsonTree:
uses the format on data requested from the youtube-api
processInfo:
the functions in degooged_tube/ytApiHacking/__init__.py called process[BLANK]Info, typically just calls fromData on the requisite dataclass
onExtend:
the functions in degooged_tube/ytApiHacking/__init__.py called [BLANK]OnExtend, typically just calls fromData on the requisite dataclass (can be overridden to inject data using OnExtendKwargs in a few cases)
fromData:
the constructors in degooged_tube/ytApiHacking/controlPanel.py which create a dataclass to hold the scraped data (ie VideoInfo.fromData()). they also sanitize and any replaces missing fields with placeholder data if possible
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
There are no reviews.