diff --git a/03_Implement_a_service/index.html b/03_Implement_a_service/index.html
index ae0dedf..24f643e 100644
--- a/03_Implement_a_service/index.html
+++ b/03_Implement_a_service/index.html
@@ -139,6 +139,11 @@ This Page will describe how you can implement and add your own. Please make sure
Concept of Extractors
and the Concept of LinkHandler
before implementing your own Service.
+
Search
- SearchExtractor
diff --git a/index.html b/index.html
index 8376b6b..b34b85f 100644
--- a/index.html
+++ b/index.html
@@ -181,5 +181,5 @@ It focuses on making it possible for the creator of a scraper for a streaming se
diff --git a/search/search_index.json b/search/search_index.json
index 808a6db..85d1677 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentaiton. This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Welcome to the NewPipe Documentaiton."},{"location":"#welcome-to-the-newpipe-documentaiton","text":"This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentaiton."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's dive right in. ;D Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your ide and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Running test in Android Studio Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho) NOT allowed Content NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's dive right in. ;D","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your ide and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#running-test-in-android-studio","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as it's title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as it's title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointig to it, but it has one unique id that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the coresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical Usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the url. } @Override public String getUrl(String id) throws ParsingException { // Return the url based on the id given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support. SearchQueryHandler You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointig to it, but it has one unique id that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the coresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical Usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the url. } @Override public String getUrl(String id) throws ParsingException { // Return the url based on the id given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implement a service Services or better service connectors are the parts of NewPipe which communicative with an actual service like YouTube. This Page will describe how you can implement and add your own. Please make sure you red and understand the Concept of Extractors and the Concept of LinkHandler before implementing your own Service. Search SearchExtractor SearchQueryHandlerFactory Stream StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Channel ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Playlist PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Kiosk KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implement a service"},{"location":"03_Implement_a_service/#implement-a-service","text":"Services or better service connectors are the parts of NewPipe which communicative with an actual service like YouTube. This Page will describe how you can implement and add your own. Please make sure you red and understand the Concept of Extractors and the Concept of LinkHandler before implementing your own Service.","title":"Implement a service"},{"location":"03_Implement_a_service/#search","text":"SearchExtractor SearchQueryHandlerFactory","title":"Search"},{"location":"03_Implement_a_service/#stream","text":"StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#channel","text":"ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#playlist","text":"PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#kiosk","text":"KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Kiosk"},{"location":"04_Run_changes_in_App/","text":"Run the changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these changes. Setup Android Studio First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up. Get the NewPipe code and run it. In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible . Run your changes on the Extractor In order to use the extractor in our app we use jitpack . This is a build service that can build marven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there From here you can grap the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. Trouble shoot If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#run-the-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these changes.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible .","title":"Get the NewPipe code and run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"In order to use the extractor in our app we use jitpack . This is a build service that can build marven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there From here you can grap the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor.","title":"Run your changes on the Extractor"},{"location":"04_Run_changes_in_App/#trouble-shoot","text":"If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Trouble shoot"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentaiton. This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Welcome to the NewPipe Documentaiton."},{"location":"#welcome-to-the-newpipe-documentaiton","text":"This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentaiton."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's dive right in. ;D Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your ide and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Running test in Android Studio Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho) NOT allowed Content NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's dive right in. ;D","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your ide and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#running-test-in-android-studio","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as it's title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as it's title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointig to it, but it has one unique id that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the coresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical Usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the url. } @Override public String getUrl(String id) throws ParsingException { // Return the url based on the id given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support. SearchQueryHandler You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointig to it, but it has one unique id that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the coresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical Usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the url. } @Override public String getUrl(String id) throws ParsingException { // Return the url based on the id given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implement a service Services or better service connectors are the parts of NewPipe which communicative with an actual service like YouTube. This Page will describe how you can implement and add your own. Please make sure you red and understand the Concept of Extractors and the Concept of LinkHandler before implementing your own Service. ServiceList StreamingService ServiceInfo Search SearchExtractor SearchQueryHandlerFactory Stream StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Channel ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Playlist PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Kiosk KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implement a service"},{"location":"03_Implement_a_service/#implement-a-service","text":"Services or better service connectors are the parts of NewPipe which communicative with an actual service like YouTube. This Page will describe how you can implement and add your own. Please make sure you red and understand the Concept of Extractors and the Concept of LinkHandler before implementing your own Service. ServiceList StreamingService ServiceInfo","title":"Implement a service"},{"location":"03_Implement_a_service/#search","text":"SearchExtractor SearchQueryHandlerFactory","title":"Search"},{"location":"03_Implement_a_service/#stream","text":"StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#channel","text":"ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#playlist","text":"PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#kiosk","text":"KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Kiosk"},{"location":"04_Run_changes_in_App/","text":"Run the changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these changes. Setup Android Studio First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up. Get the NewPipe code and run it. In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible . Run your changes on the Extractor In order to use the extractor in our app we use jitpack . This is a build service that can build marven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there From here you can grap the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. Trouble shoot If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#run-the-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these changes.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible .","title":"Get the NewPipe code and run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"In order to use the extractor in our app we use jitpack . This is a build service that can build marven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there From here you can grap the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor.","title":"Run your changes on the Extractor"},{"location":"04_Run_changes_in_App/#trouble-shoot","text":"If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Trouble shoot"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 0b1826b..c5e04c0 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ