From 2c5e8bfa8810427847102be5b972393e9aef3b7c Mon Sep 17 00:00:00 2001 From: Christian Schabesberger Date: Thu, 21 Feb 2019 15:56:42 +0100 Subject: [PATCH] Deployed 01cfd0f with MkDocs version: 1.0.4 --- 05_releasing/index.html | 14 +++++++++----- index.html | 6 +++--- search/search_index.json | 2 +- sitemap.xml.gz | Bin 199 -> 199 bytes 4 files changed, 13 insertions(+), 9 deletions(-) diff --git a/05_releasing/index.html b/05_releasing/index.html index a292a4c..9700077 100644 --- a/05_releasing/index.html +++ b/05_releasing/index.html @@ -155,20 +155,24 @@ god as possible. Our whole release cycle is therefore designed around this issue and a version that fixes an issue that occurred because Youtube (or some other service) suddenly changed their website (mostly call this a shutdown). Lets first have a look how a regular release work, and then how the hotfix release work.

Regular releases

-

Regular releases are normal releases like they are done like in any other app. Releases are always stored on master branch. By means the latest commit on -master is always equal to the current releases. No development is done on master. This ensures that we always have one +

Regular releases are normal releases like they are done in any other app. Releases are always stored on master branch. By means the latest commit on +master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version.

Feature branching

-

For development the dev branch is used. Pushing to dev directly however is also not allowed since QA and testing should be done before pushing to dev. +

For development the dev branch is used. Pushing to dev directly however is also not allowed since QA and testing should be done before adding something to it. This ensures that also the dev version works as good a possible. So in order to change something on the app one may want to fork the dev branch and develop the changes in his own branch (this is called feature branching).

feature_branching

+

Make sure that both the dev branches as well as the master branches of the extractor and the frontend are compatible to each other. +If a change is done on the API to the extractor make sure that frontend is being made compatible to these changes. If the PR that should make the frontend compatible +again can not be merged please do not merge the corresponding PR on the extractor as well. This should make sure that any developer can run his changes +on the fronted at any time.

Merging features/bugfixes

After being done with the feature one should open up a Pull Reuqest to the dev branch here a maintainer can do Code review and Quality Assurance (QA). -If you are a maintainer please take care about the code architecture so corrosion or code shifting can be prevented. Please also preface core quality over functionality. +If you are a maintainer please take care about the code architecture so corrosion or code shifting can be prevented. Please also prefare code quality over functionality. So in short: cool function but bad code -> no merge. We should focus on leaving the code as clean as possible.

merge_feature_into_dev

-

At best you as a maintainer should build the app and put the signed apk into the description of that new pullrequest. This way other people can test the feature/bugfix and therefore help with QA.

+

At best you as a maintainer should build the app and put the signed apk into the description of that new pullrequest. This way other people can test the feature/bugfix and therefore help with QA. You may not need to do this every time. It is enough to do it on bigger pull requests.

After the maintainer merged the new feature into the dev branch he should add the title of the pullrequest or a summary of the changes into the release note.

Creating a new release

Once there are enough features together, and the maintainer feels like releasing he should create a new release. Here is a list of things he will want to do then. diff --git a/index.html b/index.html index ef3dd1e..53228c0 100644 --- a/index.html +++ b/index.html @@ -137,11 +137,11 @@

Welcome to the NewPipe Documentation.

-

This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor. +

This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor however it does also contain several notes about how to maintain NewPipe. It is an addition to our auto generated jdoc documentation.

Please be aware that it is also in an early state, so help and feedback is always welcome :D

Introduction

-

The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms.

+

The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms as well.

The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.

@@ -192,5 +192,5 @@ It focuses on making it possible for the creator of a scraper for a streaming se diff --git a/search/search_index.json b/search/search_index.json index abd9118..6e77518 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentation. This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Welcome to the NewPipe Documentation."},{"location":"#welcome-to-the-newpipe-documentation","text":"This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor . It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentation."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms. The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :) Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding of unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Importing the NewPipe Extractor in IntelliJ IDEA If you use IntelliJ IDEA you might want to know how to import the NewPipe extractor painlessly. This is what you want to do. git clone the extractor onto your computer locally. Start IntelliJ Idea, and click on Import Project . Select the root directory of the NewPipe Extractor Now it's important to select \" Import Project form external Model \" and then choose Gradle . In the next window make sure \" Use grale 'wrapper' task configuration \" is selected. Running test in Android Studio/IntelliJ IDEA Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially though) NOT allowed Content NSFL (Not Safe For Life) Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :)","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding of unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#importing-the-newpipe-extractor-in-intellij-idea","text":"If you use IntelliJ IDEA you might want to know how to import the NewPipe extractor painlessly. This is what you want to do. git clone the extractor onto your computer locally. Start IntelliJ Idea, and click on Import Project . Select the root directory of the NewPipe Extractor Now it's important to select \" Import Project form external Model \" and then choose Gradle . In the next window make sure \" Use grale 'wrapper' task configuration \" is selected.","title":"Importing the NewPipe Extractor in IntelliJ IDEA"},{"location":"00_Prepare_everything/#running-test-in-android-studiointellij-idea","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio/IntelliJ IDEA"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially though)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL (Not Safe For Life) Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would collect it to form usable data for the front end. The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail, or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as an AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would collect it to form usable data for the front end. The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail, or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as an AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original URL. LinkHandler s are ReadOnly LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. !!ATENTION!! Be careful when you implement content filter: No selected filter equals all filters selected. So if your get an empty content filter list in your extractor make sure you return everything. By means use if statements like this contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original URL. LinkHandler s are ReadOnly LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. !!ATENTION!! Be careful when you implement content filter: No selected filter equals all filters selected. So if your get an empty content filter list in your extractor make sure you return everything. By means use if statements like this contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implement a service Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this. Required and optional parts Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries please tell us before you do it. Head of Service First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio. They come along with metainformation like a title, a description, next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only difference is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implement a service"},{"location":"03_Implement_a_service/#implement-a-service","text":"Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this.","title":"Implement a service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and optional parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries please tell us before you do it.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio. They come along with metainformation like a title, a description, next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only difference is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Run the changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps. Setup Android Studio First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up. Get the NewPipe code and run it. In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible . Run your changes on the Extractor In order to use the extractor in our app we use jitpack . This is a build service that can build maven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there. From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. Troubleshooting If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#run-the-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible .","title":"Get the NewPipe code and run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"In order to use the extractor in our app we use jitpack . This is a build service that can build maven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there. From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor.","title":"Run your changes on the Extractor"},{"location":"04_Run_changes_in_App/#troubleshooting","text":"If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Troubleshooting"},{"location":"05_releasing/","text":"Releasing a new NewPipe version This site is ment for those who want to maintain NewPipe, or just want to know how releasing work. Difference between regular and hotfix release NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, this however has the disadvantage of the app to brake instantly when Youtube changes something. We can not know when this happen therefore we need to be prepared when it happens, and at lease reduce our downtime as god as possible. Our whole release cycle is therefore designed around this issue. So there is a difference between a release that is meant to introduce new features or fix minor bugs, and a version that fixes an issue that occurred because Youtube (or some other service) suddenly changed their website (mostly call this a shutdown). Lets first have a look how a regular release work, and then how the hotfix release work. Regular releases Regular releases are normal releases like they are done like in any other app. Releases are always stored on master branch. By means the latest commit on master is always equal to the current releases. No development is done on master. This ensures that we always have one branch with a stable/releasable version. Feature branching For development the dev branch is used. Pushing to dev directly however is also not allowed since QA and testing should be done before pushing to dev . This ensures that also the dev version works as good a possible. So in order to change something on the app one may want to fork the dev branch and develop the changes in his own branch (this is called feature branching). Merging features/bugfixes After being done with the feature one should open up a Pull Reuqest to the dev branch here a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer please take care about the code architecture so corrosion or code shifting can be prevented. Please also preface core quality over functionality. So in short: cool function but bad code -> no merge. We should focus on leaving the code as clean as possible. At best you as a maintainer should build the app and put the signed apk into the description of that new pullrequest. This way other people can test the feature/bugfix and therefore help with QA. After the maintainer merged the new feature into the dev branch he should add the title of the pullrequest or a summary of the changes into the release note . Creating a new release Once there are enough features together, and the maintainer feels like releasing he should create a new release. Here is a list of things he will want to do then. Be aware of the rule that a release should never be done on a frieday. For NewPipe this mean don't do a release if you don't have time for it!!! Fork the dev branch into a new release_x.y.z branch. Increase the version number Copy the release note from the github version draft into the corresponding fastlane file (see release note ). Open up a pullrequest form the new release_x.y.z branch into the master branch. Create an Issue pointing to the new Pullrequest. The reason for opening an issue is that from my perception more people are reading issues then they read pullrequests. Put the release-note into this pull request. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file NewPipe__RC1.apk . Zip it and post it into the head of the release issue. This way other people can test the release candidate. Test and QA the new version with the help of other people Leave the PR open for a few days and advertise people to help testing. While being in release phase no new pullrequests must be merged into dev branch. This procedure does not have to be done for the extractor as extractor will be tested together with the fronted. Quckfixes When issuing a new release you will most likely encounter new bugs. These bugs are called regressions as they where not there before. If you notice a regression during release phase you are allowed to push fixes directly into the release branch without having to fork a branch away from it. All maintainers (people who have write access to the release branch) have to be aware that they might be required to fix regressions so plan your release on a time when you have time for coding. Do not introduce new features while being in release phase. When you have pushed a quickfix you will want to updated the release candidate you put into the issue corresponding to the release pull request . Increment the version number in the filename of the Release candidate. e.g. NewPipe__RC2.apk etc. Don't update the actuall version number however :P . Releasing Once the glories day of all days has come, and you feel like fulfilling the ceremony of releasing. This is what you will want to do. After going through the release procedure of having created a new release and maybe having done quickfixes on the new release, you will want to do these steps: Hit merge Pullreqest Create a GPG signed tag with the name v0.x.y Merge dev into master on the extractor Create a GPG signed tag with the name v0.x.y on the extractor Make sure the draft name equals the tag name Make sure to not have forgotten enything Hit Publish Release Rebase quickfix changes back into dev if quickfixes where made Hotfix releases As described aboth NewPipe is a web crawler, and therefore might brake randomly. In order to keep the downtime of NewPipe as low as possible when such a shutdown happens we allow so called hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or other bugfixes. A hotfix may only focus on fixing what has caused the shutdown. Hotfix branch Hotfixes work on the master branch. The reason for this is because dev branch might have experimental changes that have not yet been tested properly enough to be released. Master however should always be at the latest stable version of NewPipe. If this one brakes due to a shutdown you may therefore want to fix that version. Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well. Releasing If you fixed the issue and found it to be tested and reviewed well enough you man release. Here you don't need to undergo the full release procedure of a regular release, which might take up to a few days. Keep in mind that if the hotfix might turn out to be broken after release you want to release another hotfix. Here it is important to release fast, and after all a less broken version of NewPipe is better then a full broken version \u00af\\_(\u30c4)_/\u00af. This is what you will want to do when releasing a hotfix version. Hit merge Pullreqest Create a GPG signed tag with the name v0.x.y Merge dev into master on the extractor Create a GPG signed tag with the name v0.x.y on the extractor Create a new release draft and write the down the fix into the release note Copy the release note into the fastlane directory of releases Increment the small minor version number and the versionCode Hit Publish Release Rebase the hotfix back into dev branch Versioning Versioning NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First I wanted this number to switch to 1 once NewPipe was feature complete. Now I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this (2 and beyond) well god knows what happens if we ever reach 1 \u00af\\_(\u30c4)_/\u00af . Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) will be incremented if there are just smaller bug fixes or features added to the app. Versioning the extractor The extractor is always released together with the app, therefore the version number of the extractor is the same as the one of the app. Version code In android an app can also have a versionCode . This code is a long integer and can be incremented by any value to show a device that a new version is there. For NewPipe the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps are reserved for our internal fdroid build server. Release notes Release notes should give the user an idea of what was changed on the app. The release nodes for NewPipe are stored in the github draft for a new release . When a maintainer wants to add change to the release note, but there is no draft for a new version he should create one. Changes can be categorized into three types. New : New features that god added to the app. Improved : Improvements to the app, or already existing features Fixes : Bugfixes When releasing a new version of NewPipe, before actually hitting release the maintainer should copy the release note from the draft and put it into a file called .txt (whereas needs to be the version code of the comming release). This file must be stored in the direcotry /fastlane/metadata/android/en-US/changelogs . This way fdroid will later be able to show the changes done on the app.","title":"Releasing a new NewPipe version"},{"location":"05_releasing/#releasing-a-new-newpipe-version","text":"This site is ment for those who want to maintain NewPipe, or just want to know how releasing work.","title":"Releasing a new NewPipe version"},{"location":"05_releasing/#difference-between-regular-and-hotfix-release","text":"NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, this however has the disadvantage of the app to brake instantly when Youtube changes something. We can not know when this happen therefore we need to be prepared when it happens, and at lease reduce our downtime as god as possible. Our whole release cycle is therefore designed around this issue. So there is a difference between a release that is meant to introduce new features or fix minor bugs, and a version that fixes an issue that occurred because Youtube (or some other service) suddenly changed their website (mostly call this a shutdown). Lets first have a look how a regular release work, and then how the hotfix release work.","title":"Difference between regular and hotfix release"},{"location":"05_releasing/#regular-releases","text":"Regular releases are normal releases like they are done like in any other app. Releases are always stored on master branch. By means the latest commit on master is always equal to the current releases. No development is done on master. This ensures that we always have one branch with a stable/releasable version.","title":"Regular releases"},{"location":"05_releasing/#feature-branching","text":"For development the dev branch is used. Pushing to dev directly however is also not allowed since QA and testing should be done before pushing to dev . This ensures that also the dev version works as good a possible. So in order to change something on the app one may want to fork the dev branch and develop the changes in his own branch (this is called feature branching).","title":"Feature branching"},{"location":"05_releasing/#merging-featuresbugfixes","text":"After being done with the feature one should open up a Pull Reuqest to the dev branch here a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer please take care about the code architecture so corrosion or code shifting can be prevented. Please also preface core quality over functionality. So in short: cool function but bad code -> no merge. We should focus on leaving the code as clean as possible. At best you as a maintainer should build the app and put the signed apk into the description of that new pullrequest. This way other people can test the feature/bugfix and therefore help with QA. After the maintainer merged the new feature into the dev branch he should add the title of the pullrequest or a summary of the changes into the release note .","title":"Merging features/bugfixes"},{"location":"05_releasing/#creating-a-new-release","text":"Once there are enough features together, and the maintainer feels like releasing he should create a new release. Here is a list of things he will want to do then. Be aware of the rule that a release should never be done on a frieday. For NewPipe this mean don't do a release if you don't have time for it!!! Fork the dev branch into a new release_x.y.z branch. Increase the version number Copy the release note from the github version draft into the corresponding fastlane file (see release note ). Open up a pullrequest form the new release_x.y.z branch into the master branch. Create an Issue pointing to the new Pullrequest. The reason for opening an issue is that from my perception more people are reading issues then they read pullrequests. Put the release-note into this pull request. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file NewPipe__RC1.apk . Zip it and post it into the head of the release issue. This way other people can test the release candidate. Test and QA the new version with the help of other people Leave the PR open for a few days and advertise people to help testing. While being in release phase no new pullrequests must be merged into dev branch. This procedure does not have to be done for the extractor as extractor will be tested together with the fronted.","title":"Creating a new release"},{"location":"05_releasing/#quckfixes","text":"When issuing a new release you will most likely encounter new bugs. These bugs are called regressions as they where not there before. If you notice a regression during release phase you are allowed to push fixes directly into the release branch without having to fork a branch away from it. All maintainers (people who have write access to the release branch) have to be aware that they might be required to fix regressions so plan your release on a time when you have time for coding. Do not introduce new features while being in release phase. When you have pushed a quickfix you will want to updated the release candidate you put into the issue corresponding to the release pull request . Increment the version number in the filename of the Release candidate. e.g. NewPipe__RC2.apk etc. Don't update the actuall version number however :P .","title":"Quckfixes"},{"location":"05_releasing/#releasing","text":"Once the glories day of all days has come, and you feel like fulfilling the ceremony of releasing. This is what you will want to do. After going through the release procedure of having created a new release and maybe having done quickfixes on the new release, you will want to do these steps: Hit merge Pullreqest Create a GPG signed tag with the name v0.x.y Merge dev into master on the extractor Create a GPG signed tag with the name v0.x.y on the extractor Make sure the draft name equals the tag name Make sure to not have forgotten enything Hit Publish Release Rebase quickfix changes back into dev if quickfixes where made","title":"Releasing"},{"location":"05_releasing/#hotfix-releases","text":"As described aboth NewPipe is a web crawler, and therefore might brake randomly. In order to keep the downtime of NewPipe as low as possible when such a shutdown happens we allow so called hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or other bugfixes. A hotfix may only focus on fixing what has caused the shutdown.","title":"Hotfix releases"},{"location":"05_releasing/#hotfix-branch","text":"Hotfixes work on the master branch. The reason for this is because dev branch might have experimental changes that have not yet been tested properly enough to be released. Master however should always be at the latest stable version of NewPipe. If this one brakes due to a shutdown you may therefore want to fix that version. Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well.","title":"Hotfix branch"},{"location":"05_releasing/#releasing_1","text":"If you fixed the issue and found it to be tested and reviewed well enough you man release. Here you don't need to undergo the full release procedure of a regular release, which might take up to a few days. Keep in mind that if the hotfix might turn out to be broken after release you want to release another hotfix. Here it is important to release fast, and after all a less broken version of NewPipe is better then a full broken version \u00af\\_(\u30c4)_/\u00af. This is what you will want to do when releasing a hotfix version. Hit merge Pullreqest Create a GPG signed tag with the name v0.x.y Merge dev into master on the extractor Create a GPG signed tag with the name v0.x.y on the extractor Create a new release draft and write the down the fix into the release note Copy the release note into the fastlane directory of releases Increment the small minor version number and the versionCode Hit Publish Release Rebase the hotfix back into dev branch","title":"Releasing"},{"location":"05_releasing/#versioning","text":"Versioning NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First I wanted this number to switch to 1 once NewPipe was feature complete. Now I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this (2 and beyond) well god knows what happens if we ever reach 1 \u00af\\_(\u30c4)_/\u00af . Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) will be incremented if there are just smaller bug fixes or features added to the app.","title":"Versioning"},{"location":"05_releasing/#versioning-the-extractor","text":"The extractor is always released together with the app, therefore the version number of the extractor is the same as the one of the app.","title":"Versioning the extractor"},{"location":"05_releasing/#version-code","text":"In android an app can also have a versionCode . This code is a long integer and can be incremented by any value to show a device that a new version is there. For NewPipe the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps are reserved for our internal fdroid build server.","title":"Version code"},{"location":"05_releasing/#release-notes","text":"Release notes should give the user an idea of what was changed on the app. The release nodes for NewPipe are stored in the github draft for a new release . When a maintainer wants to add change to the release note, but there is no draft for a new version he should create one. Changes can be categorized into three types. New : New features that god added to the app. Improved : Improvements to the app, or already existing features Fixes : Bugfixes When releasing a new version of NewPipe, before actually hitting release the maintainer should copy the release note from the draft and put it into a file called .txt (whereas needs to be the version code of the comming release). This file must be stored in the direcotry /fastlane/metadata/android/en-US/changelogs . This way fdroid will later be able to show the changes done on the app.","title":"Release notes"},{"location":"06_documentation/","text":"Documentation The Documentation you are currently reading is written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex. Installation Mkdos is written in python and is distributed through the python internal package manager pip , thus you need to get python and pip running on your operating system first. Windows Download the latest Python3 . When running the setup program make sure to check Add Python 3.x to PATH . Install python Open powershell or cmd, and type: pip3 install mkdocs . MacOS MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager homebrew first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this. Install homebrew Install Python from homebrew, which will also install pip. Run this command: brew install python . Install mkdocs: pip3 install mkdocs Linux/*BSD Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly. Android/ChromeOS This might sound funny, but according to the growing amount of ChromeBooks and Android tablets with keyboard this might actually be useful. Install the Tremux App from f-droid . Launch Termux and type apt update Install python and git with the command: apt install git python Now install mkdocs with pip install mkdocs . From here on everything will be the same as on Desktop. If you want to edit the files you can (besides vim or emacs which are available through Termux) use your favourite text editor on android. This is possible by opening the files with the Termux integration of the build in android file manager: Update Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system. Using mkdocs In order to extend this documentation you have to clone it from its git repository . When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs. Write and Deploy If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000 . So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page, so it will be up to date at the github pages simply type mkdocs gh-deploy . However please be aware that this will not push your changes to the master branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only priviliged maintainers can do this.","title":"Documentation"},{"location":"06_documentation/#documentation","text":"The Documentation you are currently reading is written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex.","title":"Documentation"},{"location":"06_documentation/#installation","text":"Mkdos is written in python and is distributed through the python internal package manager pip , thus you need to get python and pip running on your operating system first.","title":"Installation"},{"location":"06_documentation/#windows","text":"Download the latest Python3 . When running the setup program make sure to check Add Python 3.x to PATH . Install python Open powershell or cmd, and type: pip3 install mkdocs .","title":"Windows"},{"location":"06_documentation/#macos","text":"MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager homebrew first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this. Install homebrew Install Python from homebrew, which will also install pip. Run this command: brew install python . Install mkdocs: pip3 install mkdocs","title":"MacOS"},{"location":"06_documentation/#linuxbsd","text":"Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly.","title":"Linux/*BSD"},{"location":"06_documentation/#androidchromeos","text":"This might sound funny, but according to the growing amount of ChromeBooks and Android tablets with keyboard this might actually be useful. Install the Tremux App from f-droid . Launch Termux and type apt update Install python and git with the command: apt install git python Now install mkdocs with pip install mkdocs . From here on everything will be the same as on Desktop. If you want to edit the files you can (besides vim or emacs which are available through Termux) use your favourite text editor on android. This is possible by opening the files with the Termux integration of the build in android file manager:","title":"Android/ChromeOS"},{"location":"06_documentation/#update","text":"Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system.","title":"Update"},{"location":"06_documentation/#using-mkdocs","text":"In order to extend this documentation you have to clone it from its git repository . When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs.","title":"Using mkdocs"},{"location":"06_documentation/#write-and-deploy","text":"If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000 . So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page, so it will be up to date at the github pages simply type mkdocs gh-deploy . However please be aware that this will not push your changes to the master branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only priviliged maintainers can do this.","title":"Write and Deploy"}]} \ No newline at end of file +{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Documentation. This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor however it does also contain several notes about how to maintain NewPipe. It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms as well. The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Welcome to the NewPipe Documentation."},{"location":"#welcome-to-the-newpipe-documentation","text":"This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the NewPipe Extractor however it does also contain several notes about how to maintain NewPipe. It is an addition to our auto generated jdoc documentation . Please be aware that it is also in an early state, so help and feedback is always welcome :D","title":"Welcome to the NewPipe Documentation."},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App NewPipe for android, however it's system independent and also available for other platforms as well. The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Prepare everything Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :) Setup your dev environment First and foremost you need to meet certain conditions in order to write your own service. What you need to know Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding of unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project What you need to have A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter. Importing the NewPipe Extractor in IntelliJ IDEA If you use IntelliJ IDEA you might want to know how to import the NewPipe extractor painlessly. This is what you want to do. git clone the extractor onto your computer locally. Start IntelliJ Idea, and click on Import Project . Select the root directory of the NewPipe Extractor Now it's important to select \" Import Project form external Model \" and then choose Gradle . In the next window make sure \" Use grale 'wrapper' task configuration \" is selected. Running test in Android Studio/IntelliJ IDEA Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run. Inclusion criteria for services After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code. Allowed Content Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially though) NOT allowed Content NSFL (Not Safe For Life) Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"Prepare everything"},{"location":"00_Prepare_everything/#prepare-everything","text":"Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our Jdoc setup. IMPORTANT!!! this is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :)","title":"Prepare everything"},{"location":"00_Prepare_everything/#setup-your-dev-environment","text":"First and foremost you need to meet certain conditions in order to write your own service.","title":"Setup your dev environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding of unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project","title":"What you need to know"},{"location":"00_Prepare_everything/#what-you-need-to-have","text":"A dev environment/ide that supports: git java 8 gradle unit testing I highly recommend IDEA Community since it has everything we need. A Github account A lot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor , using the fork button . This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. Next import the cloned project into your IDE and run it. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.","title":"What you need to have"},{"location":"00_Prepare_everything/#importing-the-newpipe-extractor-in-intellij-idea","text":"If you use IntelliJ IDEA you might want to know how to import the NewPipe extractor painlessly. This is what you want to do. git clone the extractor onto your computer locally. Start IntelliJ Idea, and click on Import Project . Select the root directory of the NewPipe Extractor Now it's important to select \" Import Project form external Model \" and then choose Gradle . In the next window make sure \" Use grale 'wrapper' task configuration \" is selected.","title":"Importing the NewPipe Extractor in IntelliJ IDEA"},{"location":"00_Prepare_everything/#running-test-in-android-studiointellij-idea","text":"Go to run > edit configurations > add new configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As task add \"test\". Now save and you should be able to run.","title":"Running test in Android Studio/IntelliJ IDEA"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service you will need to submit it to our NewPipeExtractor repository. However in order to include your changes you need to follow these rules: Stick to our Code contribution guidelines Do not send services that present content we don't allow on NewPipe. You need to be willing to keep on maintaining your service after submission. Be patient and do the requested changes when one of our maintainers rejects your code.","title":"Inclusion criteria for services"},{"location":"00_Prepare_everything/#allowed-content","text":"Basically anything except NOT allowed content . Any kind of porn/NSFW that is allowed according to the US Porn act . Advertisement (may be handled specially though)","title":"Allowed Content"},{"location":"00_Prepare_everything/#not-allowed-content","text":"NSFL (Not Safe For Life) Porn that is not allowed according to US Porn act . Any form of violence Child pornography Media that harms others Media that violates human rights Copyright infringement/pirated media","title":"NOT allowed Content"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor Collector/Extractor pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would collect it to form usable data for the front end. The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the front end So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical implementation of a single data extractor The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields } Collector/Extractor pattern for lists Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } InfoItems encapsulated in pages When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail, or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as an AJAX request.","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern","text":"Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern you will find all over the code. It is called the extractor/collector pattern. The idea behind it is that the extractor would produce single pieces of data, and the collector would collect it to form usable data for the front end. The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"Collector/Extractor pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"So typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the front end"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if someting failed // get piece of information and return it } ... // More datafields }","title":"Typical implementation of a single data extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Sometimes information can be represented as a list. In NewPipe a list is represented by a InfoItemsCollector . A InfoItemCollector will collect and assemble a list of InfoItem . For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via commit() . If you are implementing a list for your service you need to extend InfoItem containing the extracted information, and implement an InfoItemExtractor that will return the data of one InfoItem. A common Implementation would look like this: private MyInfoItemCollector collectInfoItemsFromElement(Element e) { MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor pattern for lists"},{"location":"01_Concept_of_the_extractor/#infoitems-encapsulated-in-pages","text":"When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail, or its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. This is why a list in NewPipe lists are chopped down into smaller lists called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. Additional metainformation about the list such as its title a thumbnail or its creator, and extracting multiple pages can be handled by a ListExtractor , and it's ListExtractor.InfoItemsPage . For extracting list header information it behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular webpage, but all the others as an AJAX request.","title":"InfoItems encapsulated in pages"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of LinkHandler LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0 Importand notes about LinkHandler: A simple LinkHandler will contain the default URL, the ID and the original URL. LinkHandler s are ReadOnly LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } } ListLinkHandler and SearchQueryHandler List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. !!ATENTION!! Be careful when you implement content filter: No selected filter equals all filters selected. So if your get an empty content filter list in your extractor make sure you return everything. By means use if statements like this contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-linkhandler","text":"LinkHandler represent Links to resources like videos, search requests, channels, etc. The idea behind them is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube) https://youtu.be/oHg5SJYRHA0 https://m.youtube.com/watch?v=oHg5SJYRHA0","title":"Concept of LinkHandler"},{"location":"02_Concept_of_LinkHandler/#importand-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID and the original URL. LinkHandler s are ReadOnly LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of Type of Resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Importand notes about LinkHandler:"},{"location":"02_Concept_of_LinkHandler/#usage","text":"So the typical usage for getting a LinkHandler would look like this. LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHanlderFactory can handle this type of link } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources like channels and playlists can be sorted, and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler which inherits from LinkHandler and adds the fields ContentFilter which is used to filter by resource type like stream or playlist, and SortFilter which is used to sort by name, date or view count. !!ATENTION!! Be careful when you implement content filter: No selected filter equals all filters selected. So if your get an empty content filter list in your extractor make sure you return everything. By means use if statements like this contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implement a service Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this. Required and optional parts Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries please tell us before you do it. Head of Service First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio. They come along with metainformation like a title, a description, next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only difference is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implement a service"},{"location":"03_Implement_a_service/#implement-a-service","text":"Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube. This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before doing this.","title":"Implement a service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything. Some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels then. However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The parts of a service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and optional parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using regex to often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries please tell us before you do it.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all if you want to create a new service you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts required to be implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some meta information about your service. Such as the name, the capabilities, and your name as well as your email address for further notice and maintenance issues. Remember, after extending this class you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio. They come along with metainformation like a title, a description, next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such Streams, like a search result or a playlist. Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was pretty useless :) Parts required to be implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to as well display a suggestion while typing. Parts required to be implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend). A playlist may only contains StreamInfoItems , but no other InfoItem types. Parts required to be implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only difference is that it does not represent a simple list of streams, but a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor differs form the one of a playlist. Parts required to be implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators selection\" etc. Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought. Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user, since users of NewPipe can decide by the settings weather they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one Kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure every page you want to support has to be implemented as its own KioskExtractor . However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every Kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts required to be implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Run the changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps. Setup Android Studio First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up. Get the NewPipe code and run it. In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible . Run your changes on the Extractor In order to use the extractor in our app we use jitpack . This is a build service that can build maven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there. From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor. Troubleshooting If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#run-the-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it then with the actual fronted you need to follow these steps.","title":"Run the changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First of all you want to setup a working Android Studio environment. For this please download Studio from developer.android.com , and follow the instructions about how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it you must simply clone or download from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Please also make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under linux where you sometimes have to adjust the udev rules in order to make your device be accessible .","title":"Get the NewPipe code and run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"In order to use the extractor in our app we use jitpack . This is a build service that can build maven *.jar packages for android and java based on a github or gitlab repositories. To the extractor through jitpack, you need to push them to your online repository of your copy that you host either on github or gitlab . It's important to host it on one of both. Now copy your repository url in Http format, go to jitpack , and past it there. From here you can grab the latest commit via GET IT button. I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe with the new extractor.","title":"Run your changes on the Extractor"},{"location":"04_Run_changes_in_App/#troubleshooting","text":"If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it is red it already shows that the build failed.","title":"Troubleshooting"},{"location":"05_releasing/","text":"Releasing a new NewPipe version This site is ment for those who want to maintain NewPipe, or just want to know how releasing work. Difference between regular and hotfix release NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, this however has the disadvantage of the app to brake instantly when Youtube changes something. We can not know when this happen therefore we need to be prepared when it happens, and at lease reduce our downtime as god as possible. Our whole release cycle is therefore designed around this issue. So there is a difference between a release that is meant to introduce new features or fix minor bugs, and a version that fixes an issue that occurred because Youtube (or some other service) suddenly changed their website (mostly call this a shutdown). Lets first have a look how a regular release work, and then how the hotfix release work. Regular releases Regular releases are normal releases like they are done in any other app. Releases are always stored on master branch. By means the latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version. Feature branching For development the dev branch is used. Pushing to dev directly however is also not allowed since QA and testing should be done before adding something to it. This ensures that also the dev version works as good a possible. So in order to change something on the app one may want to fork the dev branch and develop the changes in his own branch (this is called feature branching). Make sure that both the dev branches as well as the master branches of the extractor and the frontend are compatible to each other. If a change is done on the API to the extractor make sure that frontend is being made compatible to these changes. If the PR that should make the frontend compatible again can not be merged please do not merge the corresponding PR on the extractor as well. This should make sure that any developer can run his changes on the fronted at any time. Merging features/bugfixes After being done with the feature one should open up a Pull Reuqest to the dev branch here a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer please take care about the code architecture so corrosion or code shifting can be prevented. Please also prefare code quality over functionality. So in short: cool function but bad code -> no merge. We should focus on leaving the code as clean as possible. At best you as a maintainer should build the app and put the signed apk into the description of that new pullrequest. This way other people can test the feature/bugfix and therefore help with QA. You may not need to do this every time. It is enough to do it on bigger pull requests. After the maintainer merged the new feature into the dev branch he should add the title of the pullrequest or a summary of the changes into the release note . Creating a new release Once there are enough features together, and the maintainer feels like releasing he should create a new release. Here is a list of things he will want to do then. Be aware of the rule that a release should never be done on a frieday. For NewPipe this mean don't do a release if you don't have time for it!!! Fork the dev branch into a new release_x.y.z branch. Increase the version number Copy the release note from the github version draft into the corresponding fastlane file (see release note ). Open up a pullrequest form the new release_x.y.z branch into the master branch. Create an Issue pointing to the new Pullrequest. The reason for opening an issue is that from my perception more people are reading issues then they read pullrequests. Put the release-note into this pull request. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file NewPipe__RC1.apk . Zip it and post it into the head of the release issue. This way other people can test the release candidate. Test and QA the new version with the help of other people Leave the PR open for a few days and advertise people to help testing. While being in release phase no new pullrequests must be merged into dev branch. This procedure does not have to be done for the extractor as extractor will be tested together with the fronted. Quckfixes When issuing a new release you will most likely encounter new bugs. These bugs are called regressions as they where not there before. If you notice a regression during release phase you are allowed to push fixes directly into the release branch without having to fork a branch away from it. All maintainers (people who have write access to the release branch) have to be aware that they might be required to fix regressions so plan your release on a time when you have time for coding. Do not introduce new features while being in release phase. When you have pushed a quickfix you will want to updated the release candidate you put into the issue corresponding to the release pull request . Increment the version number in the filename of the Release candidate. e.g. NewPipe__RC2.apk etc. Don't update the actuall version number however :P . Releasing Once the glories day of all days has come, and you feel like fulfilling the ceremony of releasing. This is what you will want to do. After going through the release procedure of having created a new release and maybe having done quickfixes on the new release, you will want to do these steps: Hit merge Pullreqest Create a GPG signed tag with the name v0.x.y Merge dev into master on the extractor Create a GPG signed tag with the name v0.x.y on the extractor Make sure the draft name equals the tag name Make sure to not have forgotten enything Hit Publish Release Rebase quickfix changes back into dev if quickfixes where made Hotfix releases As described aboth NewPipe is a web crawler, and therefore might brake randomly. In order to keep the downtime of NewPipe as low as possible when such a shutdown happens we allow so called hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or other bugfixes. A hotfix may only focus on fixing what has caused the shutdown. Hotfix branch Hotfixes work on the master branch. The reason for this is because dev branch might have experimental changes that have not yet been tested properly enough to be released. Master however should always be at the latest stable version of NewPipe. If this one brakes due to a shutdown you may therefore want to fix that version. Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well. Releasing If you fixed the issue and found it to be tested and reviewed well enough you man release. Here you don't need to undergo the full release procedure of a regular release, which might take up to a few days. Keep in mind that if the hotfix might turn out to be broken after release you want to release another hotfix. Here it is important to release fast, and after all a less broken version of NewPipe is better then a full broken version \u00af\\_(\u30c4)_/\u00af. This is what you will want to do when releasing a hotfix version. Hit merge Pullreqest Create a GPG signed tag with the name v0.x.y Merge dev into master on the extractor Create a GPG signed tag with the name v0.x.y on the extractor Create a new release draft and write the down the fix into the release note Copy the release note into the fastlane directory of releases Increment the small minor version number and the versionCode Hit Publish Release Rebase the hotfix back into dev branch Versioning Versioning NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First I wanted this number to switch to 1 once NewPipe was feature complete. Now I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this (2 and beyond) well god knows what happens if we ever reach 1 \u00af\\_(\u30c4)_/\u00af . Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) will be incremented if there are just smaller bug fixes or features added to the app. Versioning the extractor The extractor is always released together with the app, therefore the version number of the extractor is the same as the one of the app. Version code In android an app can also have a versionCode . This code is a long integer and can be incremented by any value to show a device that a new version is there. For NewPipe the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps are reserved for our internal fdroid build server. Release notes Release notes should give the user an idea of what was changed on the app. The release nodes for NewPipe are stored in the github draft for a new release . When a maintainer wants to add change to the release note, but there is no draft for a new version he should create one. Changes can be categorized into three types. New : New features that god added to the app. Improved : Improvements to the app, or already existing features Fixes : Bugfixes When releasing a new version of NewPipe, before actually hitting release the maintainer should copy the release note from the draft and put it into a file called .txt (whereas needs to be the version code of the comming release). This file must be stored in the direcotry /fastlane/metadata/android/en-US/changelogs . This way fdroid will later be able to show the changes done on the app.","title":"Releasing a new NewPipe version"},{"location":"05_releasing/#releasing-a-new-newpipe-version","text":"This site is ment for those who want to maintain NewPipe, or just want to know how releasing work.","title":"Releasing a new NewPipe version"},{"location":"05_releasing/#difference-between-regular-and-hotfix-release","text":"NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, this however has the disadvantage of the app to brake instantly when Youtube changes something. We can not know when this happen therefore we need to be prepared when it happens, and at lease reduce our downtime as god as possible. Our whole release cycle is therefore designed around this issue. So there is a difference between a release that is meant to introduce new features or fix minor bugs, and a version that fixes an issue that occurred because Youtube (or some other service) suddenly changed their website (mostly call this a shutdown). Lets first have a look how a regular release work, and then how the hotfix release work.","title":"Difference between regular and hotfix release"},{"location":"05_releasing/#regular-releases","text":"Regular releases are normal releases like they are done in any other app. Releases are always stored on master branch. By means the latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version.","title":"Regular releases"},{"location":"05_releasing/#feature-branching","text":"For development the dev branch is used. Pushing to dev directly however is also not allowed since QA and testing should be done before adding something to it. This ensures that also the dev version works as good a possible. So in order to change something on the app one may want to fork the dev branch and develop the changes in his own branch (this is called feature branching). Make sure that both the dev branches as well as the master branches of the extractor and the frontend are compatible to each other. If a change is done on the API to the extractor make sure that frontend is being made compatible to these changes. If the PR that should make the frontend compatible again can not be merged please do not merge the corresponding PR on the extractor as well. This should make sure that any developer can run his changes on the fronted at any time.","title":"Feature branching"},{"location":"05_releasing/#merging-featuresbugfixes","text":"After being done with the feature one should open up a Pull Reuqest to the dev branch here a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer please take care about the code architecture so corrosion or code shifting can be prevented. Please also prefare code quality over functionality. So in short: cool function but bad code -> no merge. We should focus on leaving the code as clean as possible. At best you as a maintainer should build the app and put the signed apk into the description of that new pullrequest. This way other people can test the feature/bugfix and therefore help with QA. You may not need to do this every time. It is enough to do it on bigger pull requests. After the maintainer merged the new feature into the dev branch he should add the title of the pullrequest or a summary of the changes into the release note .","title":"Merging features/bugfixes"},{"location":"05_releasing/#creating-a-new-release","text":"Once there are enough features together, and the maintainer feels like releasing he should create a new release. Here is a list of things he will want to do then. Be aware of the rule that a release should never be done on a frieday. For NewPipe this mean don't do a release if you don't have time for it!!! Fork the dev branch into a new release_x.y.z branch. Increase the version number Copy the release note from the github version draft into the corresponding fastlane file (see release note ). Open up a pullrequest form the new release_x.y.z branch into the master branch. Create an Issue pointing to the new Pullrequest. The reason for opening an issue is that from my perception more people are reading issues then they read pullrequests. Put the release-note into this pull request. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file NewPipe__RC1.apk . Zip it and post it into the head of the release issue. This way other people can test the release candidate. Test and QA the new version with the help of other people Leave the PR open for a few days and advertise people to help testing. While being in release phase no new pullrequests must be merged into dev branch. This procedure does not have to be done for the extractor as extractor will be tested together with the fronted.","title":"Creating a new release"},{"location":"05_releasing/#quckfixes","text":"When issuing a new release you will most likely encounter new bugs. These bugs are called regressions as they where not there before. If you notice a regression during release phase you are allowed to push fixes directly into the release branch without having to fork a branch away from it. All maintainers (people who have write access to the release branch) have to be aware that they might be required to fix regressions so plan your release on a time when you have time for coding. Do not introduce new features while being in release phase. When you have pushed a quickfix you will want to updated the release candidate you put into the issue corresponding to the release pull request . Increment the version number in the filename of the Release candidate. e.g. NewPipe__RC2.apk etc. Don't update the actuall version number however :P .","title":"Quckfixes"},{"location":"05_releasing/#releasing","text":"Once the glories day of all days has come, and you feel like fulfilling the ceremony of releasing. This is what you will want to do. After going through the release procedure of having created a new release and maybe having done quickfixes on the new release, you will want to do these steps: Hit merge Pullreqest Create a GPG signed tag with the name v0.x.y Merge dev into master on the extractor Create a GPG signed tag with the name v0.x.y on the extractor Make sure the draft name equals the tag name Make sure to not have forgotten enything Hit Publish Release Rebase quickfix changes back into dev if quickfixes where made","title":"Releasing"},{"location":"05_releasing/#hotfix-releases","text":"As described aboth NewPipe is a web crawler, and therefore might brake randomly. In order to keep the downtime of NewPipe as low as possible when such a shutdown happens we allow so called hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or other bugfixes. A hotfix may only focus on fixing what has caused the shutdown.","title":"Hotfix releases"},{"location":"05_releasing/#hotfix-branch","text":"Hotfixes work on the master branch. The reason for this is because dev branch might have experimental changes that have not yet been tested properly enough to be released. Master however should always be at the latest stable version of NewPipe. If this one brakes due to a shutdown you may therefore want to fix that version. Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well.","title":"Hotfix branch"},{"location":"05_releasing/#releasing_1","text":"If you fixed the issue and found it to be tested and reviewed well enough you man release. Here you don't need to undergo the full release procedure of a regular release, which might take up to a few days. Keep in mind that if the hotfix might turn out to be broken after release you want to release another hotfix. Here it is important to release fast, and after all a less broken version of NewPipe is better then a full broken version \u00af\\_(\u30c4)_/\u00af. This is what you will want to do when releasing a hotfix version. Hit merge Pullreqest Create a GPG signed tag with the name v0.x.y Merge dev into master on the extractor Create a GPG signed tag with the name v0.x.y on the extractor Create a new release draft and write the down the fix into the release note Copy the release note into the fastlane directory of releases Increment the small minor version number and the versionCode Hit Publish Release Rebase the hotfix back into dev branch","title":"Releasing"},{"location":"05_releasing/#versioning","text":"Versioning NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First I wanted this number to switch to 1 once NewPipe was feature complete. Now I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this (2 and beyond) well god knows what happens if we ever reach 1 \u00af\\_(\u30c4)_/\u00af . Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) will be incremented if there are just smaller bug fixes or features added to the app.","title":"Versioning"},{"location":"05_releasing/#versioning-the-extractor","text":"The extractor is always released together with the app, therefore the version number of the extractor is the same as the one of the app.","title":"Versioning the extractor"},{"location":"05_releasing/#version-code","text":"In android an app can also have a versionCode . This code is a long integer and can be incremented by any value to show a device that a new version is there. For NewPipe the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps are reserved for our internal fdroid build server.","title":"Version code"},{"location":"05_releasing/#release-notes","text":"Release notes should give the user an idea of what was changed on the app. The release nodes for NewPipe are stored in the github draft for a new release . When a maintainer wants to add change to the release note, but there is no draft for a new version he should create one. Changes can be categorized into three types. New : New features that god added to the app. Improved : Improvements to the app, or already existing features Fixes : Bugfixes When releasing a new version of NewPipe, before actually hitting release the maintainer should copy the release note from the draft and put it into a file called .txt (whereas needs to be the version code of the comming release). This file must be stored in the direcotry /fastlane/metadata/android/en-US/changelogs . This way fdroid will later be able to show the changes done on the app.","title":"Release notes"},{"location":"06_documentation/","text":"Documentation The Documentation you are currently reading is written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex. Installation Mkdos is written in python and is distributed through the python internal package manager pip , thus you need to get python and pip running on your operating system first. Windows Download the latest Python3 . When running the setup program make sure to check Add Python 3.x to PATH . Install python Open powershell or cmd, and type: pip3 install mkdocs . MacOS MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager homebrew first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this. Install homebrew Install Python from homebrew, which will also install pip. Run this command: brew install python . Install mkdocs: pip3 install mkdocs Linux/*BSD Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly. Android/ChromeOS This might sound funny, but according to the growing amount of ChromeBooks and Android tablets with keyboard this might actually be useful. Install the Tremux App from f-droid . Launch Termux and type apt update Install python and git with the command: apt install git python Now install mkdocs with pip install mkdocs . From here on everything will be the same as on Desktop. If you want to edit the files you can (besides vim or emacs which are available through Termux) use your favourite text editor on android. This is possible by opening the files with the Termux integration of the build in android file manager: Update Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system. Using mkdocs In order to extend this documentation you have to clone it from its git repository . When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs. Write and Deploy If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000 . So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page, so it will be up to date at the github pages simply type mkdocs gh-deploy . However please be aware that this will not push your changes to the master branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only priviliged maintainers can do this.","title":"Documentation"},{"location":"06_documentation/#documentation","text":"The Documentation you are currently reading is written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex.","title":"Documentation"},{"location":"06_documentation/#installation","text":"Mkdos is written in python and is distributed through the python internal package manager pip , thus you need to get python and pip running on your operating system first.","title":"Installation"},{"location":"06_documentation/#windows","text":"Download the latest Python3 . When running the setup program make sure to check Add Python 3.x to PATH . Install python Open powershell or cmd, and type: pip3 install mkdocs .","title":"Windows"},{"location":"06_documentation/#macos","text":"MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager homebrew first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this. Install homebrew Install Python from homebrew, which will also install pip. Run this command: brew install python . Install mkdocs: pip3 install mkdocs","title":"MacOS"},{"location":"06_documentation/#linuxbsd","text":"Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly.","title":"Linux/*BSD"},{"location":"06_documentation/#androidchromeos","text":"This might sound funny, but according to the growing amount of ChromeBooks and Android tablets with keyboard this might actually be useful. Install the Tremux App from f-droid . Launch Termux and type apt update Install python and git with the command: apt install git python Now install mkdocs with pip install mkdocs . From here on everything will be the same as on Desktop. If you want to edit the files you can (besides vim or emacs which are available through Termux) use your favourite text editor on android. This is possible by opening the files with the Termux integration of the build in android file manager:","title":"Android/ChromeOS"},{"location":"06_documentation/#update","text":"Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a linux/bsd system.","title":"Update"},{"location":"06_documentation/#using-mkdocs","text":"In order to extend this documentation you have to clone it from its git repository . When you cloned it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. here is a guide about how to use mkdocs.","title":"Using mkdocs"},{"location":"06_documentation/#write-and-deploy","text":"If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal webserver on port 8000 . So all you have to do is type localhost:8000 into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page, so it will be up to date at the github pages simply type mkdocs gh-deploy . However please be aware that this will not push your changes to the master branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only priviliged maintainers can do this.","title":"Write and Deploy"}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 8b1855c2c0ae489a9c354624605aab84b671ff2d..dd486206ec387a04d6893a38ca0513a04fe0b969 100644 GIT binary patch delta 14 VcmX@kc$|?