+ - Using mkdocs
- Write and Deploy
@@ -182,9 +179,19 @@
- Run
pip3 install mkdocs to install mkdocs only for the current user,
or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly.
+Android/ChromeOS
+This might sound funny, but according to the growing amount of ChromeBooks and Android tablets with keyboard this might actually be useful.
+
+- Install the Tremux App from f-droid.
+- Launch Termux and type
apt update
+- Install python and git with the command:
apt install git python
+- Now install mkdocs with
pip install mkdocs.
+
+From here on everything will be the same as on Desktop. If you want to edit the files you can (besides vim or emacs which are available through Termux) use your favourite text editor on android. This is possible by opening the files with the Termux integration of the build in android file manager:
+
Update
Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system.
-Using mkdocs
+Using mkdocs
In order to extend this documentation you have to clone it from its git repository. When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs.
Write and Deploy
If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000. So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content.
diff --git a/img/termux_files.png b/img/termux_files.png
new file mode 100644
index 0000000..3851f68
Binary files /dev/null and b/img/termux_files.png differ
diff --git a/index.html b/index.html
index 347b715..74c3ccf 100644
--- a/index.html
+++ b/index.html
@@ -187,5 +187,5 @@ It focuses on making it possible for the creator of a scraper for a streaming se
diff --git a/search/search_index.json b/search/search_index.json
index a85783e..31e1bd4 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentation. This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Welcome to the NewPipe Documentation."},{"location":"#welcome-to-the-newpipe-documentation","text":"This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentation."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :) Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding of unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Running test in Android Studio Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially though) NOT allowed Content NSFL (Not Safe For Life) Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :)","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding of unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#running-test-in-android-studio","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially though)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL (Not Safe For Life) Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would collect it to form usable data for the front end. The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail, or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as an AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would collect it to form usable data for the front end. The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail, or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as an AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original URL. LinkHandler s are ReadOnly LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original URL. LinkHandler s are ReadOnly LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implement a service Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this. Required and optional parts Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries please tell us before you do it. Head of Service First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio. They come along with metainformation like a title, a description, next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only difference is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implement a service"},{"location":"03_Implement_a_service/#implement-a-service","text":"Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this.","title":"Implement a service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and optional parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries please tell us before you do it.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio. They come along with metainformation like a title, a description, next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only difference is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Run the changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps. Setup Android Studio First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up. Get the NewPipe code and run it. In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible . Run your changes on the Extractor In order to use the extractor in our app we use jitpack . This is a build service that can build maven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there. From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. Troubleshooting If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#run-the-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible .","title":"Get the NewPipe code and run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"In order to use the extractor in our app we use jitpack . This is a build service that can build maven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there. From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor.","title":"Run your changes on the Extractor"},{"location":"04_Run_changes_in_App/#troubleshooting","text":"If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Troubleshooting"},{"location":"05_documentation/","text":"Documentation The Documentation you are currently reading is written using mkdocs . It is a tool that will generate a static website based on files writting in markdown . Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex. Installation Mkdos is written in python and is distributed through the python internal package manager pip , thus you need to get python and pip running on your operating system first. Windows Download the latest Python3 . When running the setup program make sure to check Add Python 3.x to PATH . Install python Open powershell or cmd, and type: pip3 install mkdocs . MacOS MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager homebrew first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this. Install homebrew Install Python from homebrew, which will also install pip. Run this command: brew install python . Install mkdocs: pip3 install mkdocs Linux/*BSD Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly. Update Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system. Using mkdocs In order to extend this documentation you have to clone it from its git repository . When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs. Write and Deploy If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000 . So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page, so it will be up to date at the github pages simply type mkdocs gh-deploy . However please be aware that this will not push your changes to the master branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only priviliged maintainers can do this.","title":"Documentation"},{"location":"05_documentation/#documentation","text":"The Documentation you are currently reading is written using mkdocs . It is a tool that will generate a static website based on files writting in markdown . Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex.","title":"Documentation"},{"location":"05_documentation/#installation","text":"Mkdos is written in python and is distributed through the python internal package manager pip , thus you need to get python and pip running on your operating system first.","title":"Installation"},{"location":"05_documentation/#windows","text":"Download the latest Python3 . When running the setup program make sure to check Add Python 3.x to PATH . Install python Open powershell or cmd, and type: pip3 install mkdocs .","title":"Windows"},{"location":"05_documentation/#macos","text":"MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager homebrew first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this. Install homebrew Install Python from homebrew, which will also install pip. Run this command: brew install python . Install mkdocs: pip3 install mkdocs","title":"MacOS"},{"location":"05_documentation/#linuxbsd","text":"Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly.","title":"Linux/*BSD"},{"location":"05_documentation/#update","text":"Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system.","title":"Update"},{"location":"05_documentation/#using-mkdocs","text":"In order to extend this documentation you have to clone it from its git repository . When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs.","title":"Using mkdocs"},{"location":"05_documentation/#write-and-deploy","text":"If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000 . So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page, so it will be up to date at the github pages simply type mkdocs gh-deploy . However please be aware that this will not push your changes to the master branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only priviliged maintainers can do this.","title":"Write and Deploy"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentation. This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Welcome to the NewPipe Documentation."},{"location":"#welcome-to-the-newpipe-documentation","text":"This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentation."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :) Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding of unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Running test in Android Studio Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially though) NOT allowed Content NSFL (Not Safe For Life) Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :)","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding of unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#running-test-in-android-studio","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially though)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL (Not Safe For Life) Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would collect it to form usable data for the front end. The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail, or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as an AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would collect it to form usable data for the front end. The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail, or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as an AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original URL. LinkHandler s are ReadOnly LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original URL. LinkHandler s are ReadOnly LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implement a service Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this. Required and optional parts Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries please tell us before you do it. Head of Service First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio. They come along with metainformation like a title, a description, next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only difference is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implement a service"},{"location":"03_Implement_a_service/#implement-a-service","text":"Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this.","title":"Implement a service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and optional parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries please tell us before you do it.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio. They come along with metainformation like a title, a description, next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only difference is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Run the changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps. Setup Android Studio First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up. Get the NewPipe code and run it. In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible . Run your changes on the Extractor In order to use the extractor in our app we use jitpack . This is a build service that can build maven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there. From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. Troubleshooting If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#run-the-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible .","title":"Get the NewPipe code and run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"In order to use the extractor in our app we use jitpack . This is a build service that can build maven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there. From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor.","title":"Run your changes on the Extractor"},{"location":"04_Run_changes_in_App/#troubleshooting","text":"If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Troubleshooting"},{"location":"05_documentation/","text":"Documentation The Documentation you are currently reading is written using mkdocs . It is a tool that will generate a static website based on files writting in markdown . Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex. Installation Mkdos is written in python and is distributed through the python internal package manager pip , thus you need to get python and pip running on your operating system first. Windows Download the latest Python3 . When running the setup program make sure to check Add Python 3.x to PATH . Install python Open powershell or cmd, and type: pip3 install mkdocs . MacOS MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager homebrew first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this. Install homebrew Install Python from homebrew, which will also install pip. Run this command: brew install python . Install mkdocs: pip3 install mkdocs Linux/*BSD Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly. Android/ChromeOS This might sound funny, but according to the growing amount of ChromeBooks and Android tablets with keyboard this might actually be useful. Install the Tremux App from f-droid . Launch Termux and type apt update Install python and git with the command: apt install git python Now install mkdocs with pip install mkdocs . From here on everything will be the same as on Desktop. If you want to edit the files you can (besides vim or emacs which are available through Termux) use your favourite text editor on android. This is possible by opening the files with the Termux integration of the build in android file manager: Update Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system. Using mkdocs In order to extend this documentation you have to clone it from its git repository . When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs. Write and Deploy If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000 . So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page, so it will be up to date at the github pages simply type mkdocs gh-deploy . However please be aware that this will not push your changes to the master branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only priviliged maintainers can do this.","title":"Documentation"},{"location":"05_documentation/#documentation","text":"The Documentation you are currently reading is written using mkdocs . It is a tool that will generate a static website based on files writting in markdown . Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex.","title":"Documentation"},{"location":"05_documentation/#installation","text":"Mkdos is written in python and is distributed through the python internal package manager pip , thus you need to get python and pip running on your operating system first.","title":"Installation"},{"location":"05_documentation/#windows","text":"Download the latest Python3 . When running the setup program make sure to check Add Python 3.x to PATH . Install python Open powershell or cmd, and type: pip3 install mkdocs .","title":"Windows"},{"location":"05_documentation/#macos","text":"MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager homebrew first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this. Install homebrew Install Python from homebrew, which will also install pip. Run this command: brew install python . Install mkdocs: pip3 install mkdocs","title":"MacOS"},{"location":"05_documentation/#linuxbsd","text":"Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly.","title":"Linux/*BSD"},{"location":"05_documentation/#androidchromeos","text":"This might sound funny, but according to the growing amount of ChromeBooks and Android tablets with keyboard this might actually be useful. Install the Tremux App from f-droid . Launch Termux and type apt update Install python and git with the command: apt install git python Now install mkdocs with pip install mkdocs . From here on everything will be the same as on Desktop. If you want to edit the files you can (besides vim or emacs which are available through Termux) use your favourite text editor on android. This is possible by opening the files with the Termux integration of the build in android file manager:","title":"Android/ChromeOS"},{"location":"05_documentation/#update","text":"Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system.","title":"Update"},{"location":"05_documentation/#using-mkdocs","text":"In order to extend this documentation you have to clone it from its git repository . When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs.","title":"Using mkdocs"},{"location":"05_documentation/#write-and-deploy","text":"If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000 . So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page, so it will be up to date at the github pages simply type mkdocs gh-deploy . However please be aware that this will not push your changes to the master branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only priviliged maintainers can do this.","title":"Write and Deploy"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 8fd98fe..5e849a0 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ