diff --git a/00_Prepare_everything/index.html b/00_Prepare_everything/index.html index dfa3c22..4f5afc4 100644 --- a/00_Prepare_everything/index.html +++ b/00_Prepare_everything/index.html @@ -137,9 +137,14 @@

Prepare everything

-

Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor -service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. Let's -dive right in. ;D

+

Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor +service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. +The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our +Jdoc setup.

+

IMPORTANT!!! this is likely to be the worst documentation you have ever red, so do not hesitate to +report if +you find any (spelling)errors, incomplete parts or you simply don't understand something. We are an open community +and are open for everyone to help :)

Setup your dev environment

First and foremost you need to meet certain conditions in order to write your own service.

What you need to know

@@ -172,7 +177,7 @@ If all the checks are green you did everything right, and you are good to go to

Running test in Android Studio

Go to run > edit configurations > add new configuration and select "Gradle". As Gradle Project, select NewPipeExtractor. As task add "test". Now save and you should be able to run.

-

tests passed on idea

+

tests passed on idea

Inclusion criteria for services

After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules:

diff --git a/01_Concept_of_the_extractor/index.html b/01_Concept_of_the_extractor/index.html index 3fe574d..ee408fb 100644 --- a/01_Concept_of_the_extractor/index.html +++ b/01_Concept_of_the_extractor/index.html @@ -182,7 +182,7 @@ try { InfoItemsCollector. A InfoItemCollector will collect and assemble a list of InfoItem. For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit().

-

InfoItemsCollector_objectdiagram.svg

+

InfoItemsCollector_objectdiagram.svg

If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem.

diff --git a/04_Run_changes_in_App/index.html b/04_Run_changes_in_App/index.html index 264487a..f3b2fc6 100644 --- a/04_Run_changes_in_App/index.html +++ b/04_Run_changes_in_App/index.html @@ -134,7 +134,7 @@

Run the changes in the App

You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with -the actual fronted you need to follow these changes.

+the actual fronted you need to follow these steps.

Setup Android Studio

First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com, @@ -164,11 +164,11 @@ Your browser does not support the video tag.

If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. -image_sync_ok

+image_sync_ok

Trouble shoot

If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed. -jitpack failed to build

+jitpack failed to build

diff --git a/index.html b/index.html index 856ed5e..e7f62bf 100644 --- a/index.html +++ b/index.html @@ -180,6 +180,6 @@ It focuses on making it possible for the creator of a scraper for a streaming se diff --git a/search/search_index.json b/search/search_index.json index 0573be0..a1b6cfc 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentation. This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Welcome to the NewPipe Documentation."},{"location":"#welcome-to-the-newpipe-documentation","text":"This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentation."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. Let's dive right in. ;D Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Running test in Android Studio Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho) NOT allowed Content NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. Let's dive right in. ;D","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#running-test-in-android-studio","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implement a service Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this. Required and optional parts Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory methode of the your StreamingService implementation return null . The forntend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsiong: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link dectection in strings: AutoLink If you need to introduce new libraries please tell us before you do it. Head of Service First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio, they come along with metainformation like a title, a description, next/related videos, thumbnail and commends. For getting the url to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only diferens is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\" ets. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implement a service"},{"location":"03_Implement_a_service/#implement-a-service","text":"Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this.","title":"Implement a service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory methode of the your StreamingService implementation return null . The forntend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and optional parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsiong: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link dectection in strings: AutoLink If you need to introduce new libraries please tell us before you do it.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio, they come along with metainformation like a title, a description, next/related videos, thumbnail and commends. For getting the url to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only diferens is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\" ets. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Run the changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these changes. Setup Android Studio First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up. Get the NewPipe code and run it. In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible . Run your changes on the Extractor In order to use the extractor in our app we use jitpack . This is a build service that can build marven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. Trouble shoot If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#run-the-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these changes.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible .","title":"Get the NewPipe code and run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"In order to use the extractor in our app we use jitpack . This is a build service that can build marven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor.","title":"Run your changes on the Extractor"},{"location":"04_Run_changes_in_App/#trouble-shoot","text":"If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Trouble shoot"}]} \ No newline at end of file +{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentation. This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Welcome to the NewPipe Documentation."},{"location":"#welcome-to-the-newpipe-documentation","text":"This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentation."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever red, so do not hesitate to report if you find any (spelling)errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :) Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Running test in Android Studio Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho) NOT allowed Content NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever red, so do not hesitate to report if you find any (spelling)errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :)","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#running-test-in-android-studio","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially tho)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would take it and form usable data for the front end out of it. The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original url. LinkHandler are ReadOnly LinkHandler are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service support.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a changing outcome deepening on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, accept that you don't have to implement the methodes onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implement a service Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this. Required and optional parts Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory methode of the your StreamingService implementation return null . The forntend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsiong: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link dectection in strings: AutoLink If you need to introduce new libraries please tell us before you do it. Head of Service First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio, they come along with metainformation like a title, a description, next/related videos, thumbnail and commends. For getting the url to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only diferens is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\" ets. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implement a service"},{"location":"03_Implement_a_service/#implement-a-service","text":"Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this.","title":"Implement a service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory methode of the your StreamingService implementation return null . The forntend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and optional parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsiong: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link dectection in strings: AutoLink If you need to introduce new libraries please tell us before you do it.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio, they come along with metainformation like a title, a description, next/related videos, thumbnail and commends. For getting the url to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only diferens is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\" ets. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Run the changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps. Setup Android Studio First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up. Get the NewPipe code and run it. In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible . Run your changes on the Extractor In order to use the extractor in our app we use jitpack . This is a build service that can build marven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. Trouble shoot If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#run-the-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible .","title":"Get the NewPipe code and run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"In order to use the extractor in our app we use jitpack . This is a build service that can build marven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor.","title":"Run your changes on the Extractor"},{"location":"04_Run_changes_in_App/#trouble-shoot","text":"If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Trouble shoot"}]} \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index 71fb381..37f4dd1 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,32 +2,32 @@ None - 2018-11-03 + 2018-11-14 daily None - 2018-11-03 + 2018-11-14 daily None - 2018-11-03 + 2018-11-14 daily None - 2018-11-03 + 2018-11-14 daily None - 2018-11-03 + 2018-11-14 daily None - 2018-11-03 + 2018-11-14 daily \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 09a59eb..60ae5eb 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ