diff --git a/02_Concept_of_LinkHandler/index.html b/02_Concept_of_LinkHandler/index.html index f33e112..977a266 100644 --- a/02_Concept_of_LinkHandler/index.html +++ b/02_Concept_of_LinkHandler/index.html @@ -81,7 +81,7 @@
  • Implementation
    • -
  • ListLinkHandler and QueryLinkHandler
    • +
  • ListLinkHandler and SearchQueryHandler
    • @@ -168,10 +168,25 @@ LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/ } -

    ListLinkHandler and QueryLinkHandler

    -

    List based resources like channels and playlists can be sorted, for example by date, name, or by a certain name. -Therefore these type of resources don't just use a LinkHandler, but an extention called -ListLinkHandler

    +

    ListLinkHandler and SearchQueryHandler

    +

    List based resources like channels and playlists can be sorted, and filtered. +Therefore these type of resources don't just use a LinkHandler, but a class called +ListLinkHandler +which inherits from LinkHandler and adds the fields ContentFilter +which is used to filter by resource type like stream or playlist, and +SortFilter +which is used to sort by name, date or view count.

    +

    ListLinkHandler are also created by overriding the ListLinkHandlerFactory +additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override +getAvailableContentFilter() +and getAvailableSortFilter(). +Through these you can tell the front end which kind of filter your service support.

    +

    SearchQueryHandler

    +

    You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the +same search request might have a changing outcome deepening on the country or the time you send the request. This is +why the idea of an "ID" is replaced by a "SearchString" in the SearchQueryHandler +These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() +and getId() when overriding SearchQueryHandlerFactory.

    diff --git a/index.html b/index.html index f568be1..e3b6131 100644 --- a/index.html +++ b/index.html @@ -171,5 +171,5 @@ It focuses on making it possible for the creator of a scraper for a streaming se diff --git a/search/search_index.json b/search/search_index.json index a75a577..b7b7e06 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentaiton. This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Welcome to the NewPipe Documentaiton."},{"location":"#welcome-to-the-newpipe-documentaiton","text":"This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentaiton."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's dive right in. ;D Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your ide and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Running test in Android Studio Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho) NOT allowed Content NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's dive right in. ;D","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your ide and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#running-test-in-android-studio","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as it's title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as it's title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointig to it, but it has one unique id that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the coresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical Usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the url. } @Override public String getUrl(String id) throws ParsingException { // Return the url based on the id given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and QueryLinkHandler List based resources like channels and playlists can be sorted, for example by date, name, or by a certain name. Therefore these type of resources don't just use a LinkHandler, but an extention called ListLinkHandler","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointig to it, but it has one unique id that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the coresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical Usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the url. } @Override public String getUrl(String id) throws ParsingException { // Return the url based on the id given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-querylinkhandler","text":"List based resources like channels and playlists can be sorted, for example by date, name, or by a certain name. Therefore these type of resources don't just use a LinkHandler, but an extention called ListLinkHandler","title":"ListLinkHandler and QueryLinkHandler"}]} \ No newline at end of file +{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentaiton. This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Welcome to the NewPipe Documentaiton."},{"location":"#welcome-to-the-newpipe-documentaiton","text":"This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentaiton."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's dive right in. ;D Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your ide and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Running test in Android Studio Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho) NOT allowed Content NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's dive right in. ;D","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your ide and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#running-test-in-android-studio","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as it's title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as it's title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointig to it, but it has one unique id that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the coresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical Usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the url. } @Override public String getUrl(String id) throws ParsingException { // Return the url based on the id given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support. SearchQueryHandler You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointig to it, but it has one unique id that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the coresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical Usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the url. } @Override public String getUrl(String id) throws ParsingException { // Return the url based on the id given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 83ba5c2..18b0b2f 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ