From 7e611329430ab5b4a9db3e5c55cac6f5eac7b8a7 Mon Sep 17 00:00:00 2001 From: TobiGr Date: Thu, 21 May 2020 18:17:59 +0100 Subject: [PATCH] Deployed ebd2c2e with MkDocs version: 1.0.4 --- 05_releasing/index.html | 8 ++++++++ index.html | 2 +- search/search_index.json | 2 +- sitemap.xml | 18 +++++++++--------- sitemap.xml.gz | Bin 202 -> 202 bytes 5 files changed, 19 insertions(+), 11 deletions(-) diff --git a/05_releasing/index.html b/05_releasing/index.html index 4e955f3..8727ac0 100644 --- a/05_releasing/index.html +++ b/05_releasing/index.html @@ -218,6 +218,10 @@ this is what you should do when releasing:

  • Make sure the draft name equals the tag name. draft_name
  • Make sure to not have forgotten anything.
  • Click "Publish Release".
    • +
  • Clone F-Droid / Data.
    • +
  • Add the new version in metadata/org.schabi.newpipe.yml.
    • +
  • Run fdroid signatures /path/to/newpipe.apk.
    • +
  • Create a MR.
  • Rebase quickfix changes back into dev if quickfixes were made.

    • rebase_back

    @@ -248,6 +252,10 @@ Here's what you do when releasing a hotfix:

  • Copy the release note into the fastlane directory of releases.
  • Increment the small minor version number and the versionCode.
  • Click "Publish Release".
    • +
  • Clone F-Droid / Data.
    • +
  • Add the new version in metadata/org.schabi.newpipe.yml.
    • +
  • Run fdroid signatures /path/to/newpipe.apk.
    • +
  • Create a MR.
  • Rebase the hotfix back into dev branch.

    • rebase_back_hotfix

    diff --git a/index.html b/index.html index 79294bb..f0c5baf 100644 --- a/index.html +++ b/index.html @@ -198,5 +198,5 @@ It focuses on making it possible for the creator of a scraper for a streaming se diff --git a/search/search_index.json b/search/search_index.json index 7b65309..32b2edc 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to NewPipe This site is/should be a beginner friendly tutorial and documentation for people who want to use or write services for the NewPipe Extractor . However, it also contains several notes about how to maintain NewPipe. It is an addition to our auto generated Jdoc documentation . Please be aware that it is in its early stages, so help and feedback is always appreciated :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming app NewPipe for Android. It is entirely independent from said platforms and also available for additional platforms as well. The beauty behind this framework is that it takes care of the extracting process, error handling etc. so you can focus on what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Welcome to NewPipe"},{"location":"#welcome-to-newpipe","text":"This site is/should be a beginner friendly tutorial and documentation for people who want to use or write services for the NewPipe Extractor . However, it also contains several notes about how to maintain NewPipe. It is an addition to our auto generated Jdoc documentation . Please be aware that it is in its early stages, so help and feedback is always appreciated :D","title":"Welcome to NewPipe"},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming app NewPipe for Android. It is entirely independent from said platforms and also available for additional platforms as well. The beauty behind this framework is that it takes care of the extracting process, error handling etc. so you can focus on what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Before You Start These documents will guide you through the process of understanding or creating your own Extractor service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube, SoundCloud and MediaCCC. The whole documentation consists of this page and Jdoc setup, which explains the general concept of the NewPipeExtractor. IMPORTANT!!! This is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :) Setting Up Your Dev Environment First and foremost, you need to meet the following conditions in order to write your own service. What You Need to Know: A basic understanding of Git Good Java knowledge A good understanding of web technology A basic understanding of unit testing and JUnit A thorough understanding of how to contribute to the NewPipe project Tools/Programs You Will Need: A dev environment/IDE that supports: Git Java 8 Gradle Unit testing IDEA Community (Strongly recommended, but not required) A Github account A lot of patience and excitement ;D After making sure all these conditions are provided, fork the NewPipeExtractor using the fork button . This is so you have a personal repository to develop on. Next, clone this repository into your local folder in which you want to work in. Then, import the cloned project into your IDE and run it. If all the checks are green, you did everything right! You can proceed to the next chapter. Importing the NewPipe Extractor in IntelliJ IDEA If you use IntelliJ IDEA, you should know the easy way of importing the NewPipe extractor. If you don't, here's how to do it: git clone the extractor onto your computer locally. Start IntelliJ Idea and click Import Project . Select the root directory of the NewPipe Extractor. Select \" Import Project from external Model \" and then choose Gradle . In the next window, select \" Use gradle 'wrapper' task configuration \". Running \"test\" in Android Studio/IntelliJ IDEA Go to Run > Edit Configurations > Add New Configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As a task, add \"test\". Now save and you should be able to run. Inclusion Criteria for Services After creating you own service, you will need to submit it to our NewPipeExtractor repository. However, in order to include your changes, you need to follow these rules: Stick to our code contribution guidelines . Do not send services that present content we don't allow on NewPipe. You must be willing to maintain your service after submission. Be patient and make the requested changes when one of our maintainers rejects your code. Content That is Permitted Any content that is not in the list of prohibited content . Any kind of pornography or NSFW content that does not violate US law. However, porn services will not be added to the official NewPipe app. Advertising, which may need to be approved beforehand. Content That is NOT Permitted Content that is considered NSFL (Not Safe For Life). Content that is prohibited by US federal law (Sexualization of minors, any form of violence, violations of human rights, etc). Copyrighted media, without the consent of the copyright holder/publisher.","title":"Before You Start"},{"location":"00_Prepare_everything/#before-you-start","text":"These documents will guide you through the process of understanding or creating your own Extractor service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube, SoundCloud and MediaCCC. The whole documentation consists of this page and Jdoc setup, which explains the general concept of the NewPipeExtractor. IMPORTANT!!! This is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :)","title":"Before You Start"},{"location":"00_Prepare_everything/#setting-up-your-dev-environment","text":"First and foremost, you need to meet the following conditions in order to write your own service.","title":"Setting Up Your Dev Environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"A basic understanding of Git Good Java knowledge A good understanding of web technology A basic understanding of unit testing and JUnit A thorough understanding of how to contribute to the NewPipe project","title":"What You Need to Know:"},{"location":"00_Prepare_everything/#toolsprograms-you-will-need","text":"A dev environment/IDE that supports: Git Java 8 Gradle Unit testing IDEA Community (Strongly recommended, but not required) A Github account A lot of patience and excitement ;D After making sure all these conditions are provided, fork the NewPipeExtractor using the fork button . This is so you have a personal repository to develop on. Next, clone this repository into your local folder in which you want to work in. Then, import the cloned project into your IDE and run it. If all the checks are green, you did everything right! You can proceed to the next chapter.","title":"Tools/Programs You Will Need:"},{"location":"00_Prepare_everything/#importing-the-newpipe-extractor-in-intellij-idea","text":"If you use IntelliJ IDEA, you should know the easy way of importing the NewPipe extractor. If you don't, here's how to do it: git clone the extractor onto your computer locally. Start IntelliJ Idea and click Import Project . Select the root directory of the NewPipe Extractor. Select \" Import Project from external Model \" and then choose Gradle . In the next window, select \" Use gradle 'wrapper' task configuration \".","title":"Importing the NewPipe Extractor in IntelliJ IDEA"},{"location":"00_Prepare_everything/#running-test-in-android-studiointellij-idea","text":"Go to Run > Edit Configurations > Add New Configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As a task, add \"test\". Now save and you should be able to run.","title":"Running \"test\" in Android Studio/IntelliJ IDEA"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service, you will need to submit it to our NewPipeExtractor repository. However, in order to include your changes, you need to follow these rules: Stick to our code contribution guidelines . Do not send services that present content we don't allow on NewPipe. You must be willing to maintain your service after submission. Be patient and make the requested changes when one of our maintainers rejects your code.","title":"Inclusion Criteria for Services"},{"location":"00_Prepare_everything/#content-that-is-permitted","text":"Any content that is not in the list of prohibited content . Any kind of pornography or NSFW content that does not violate US law. However, porn services will not be added to the official NewPipe app. Advertising, which may need to be approved beforehand.","title":"Content That is Permitted"},{"location":"00_Prepare_everything/#content-that-is-not-permitted","text":"Content that is considered NSFL (Not Safe For Life). Content that is prohibited by US federal law (Sexualization of minors, any form of violence, violations of human rights, etc). Copyrighted media, without the consent of the copyright holder/publisher.","title":"Content That is NOT Permitted"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor The Collector/Extractor Pattern Before you start coding your own service, you need to understand the basic concept of the extractor itself. There is a pattern you will find all over the code, called the extractor/collector pattern. The idea behind it is that the extractor would produce fragments of data, and the collector would collect them and assemble that data into a readable format for the front end. The collector also controls the parsing process, and takes care of error handling. So, if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of multiple methods, one method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the Front End A typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical Implementation of a Single Data Extractor The typical implementation of a single data extractor, on the other hand, would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if something failed // get piece of information and return it } ... // More datafields } Collector/Extractor Pattern for Lists Information can be represented as a list. In NewPipe, a list is represented by an InfoItemsCollector . An InfoItemsCollector will collect and assemble a list of InfoItem . For each item that should be extracted, a new Extractor must be created, and given to the InfoItemsCollector via commit() . If you are implementing a list in your service you need to implement an InfoItemExtractor , that will be able to retrieve data for one and only one InfoItem. This extractor will then be comitted to the InfoItemsCollector that can collect the type of InfoItems you want to generate. A common implementation would look like this: private SomeInfoItemCollector collectInfoItemsFromElement(Element e) { // See *Some* as something like Stream or Channel // e.g. StreamInfoItemsCollector, and ChannelInfoItemsCollector are provided by NP SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } ListExtractor There is more to know about lists: When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail, and its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. Both of these Problems are fixed by the ListExtractor which takes care about extracting additional metadata about the list, and by chopping down lists into several pages, so called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. For extracting list header information a ListExtractor behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular web page, but all the others as an AJAX request. An InfoItemsPage itself has two constructors which take these parameters: - The InfoitemsCollector of the list that the page should represent - A nextPageUrl which represents the url of the following page (may be null if not page follows). - Optionally errors which is a list of Exceptions that may have happened during extracton. Here is a simplified reference implementation of a list extractor that only extracts pages, but not metadata: class MyListExtractor extends ListExtractor { ... private Document document; ... public InfoItemsPage getPage(pageUrl) throws ExtractionException { SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); document = myFunctionToGetThePageHTMLWhatever(pageUrl); //remember this part from the simple list extraction for(final Element li : document.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return new InfoItemsPage(collector, myFunctionToGetTheNextPageUrl(document)); } public InfoItemsPage getInitialPage() { //document here got initialized by the fetch() function. return getPage(getTheCurrentPageUrl(document)); } ... }","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#the-collectorextractor-pattern","text":"Before you start coding your own service, you need to understand the basic concept of the extractor itself. There is a pattern you will find all over the code, called the extractor/collector pattern. The idea behind it is that the extractor would produce fragments of data, and the collector would collect them and assemble that data into a readable format for the front end. The collector also controls the parsing process, and takes care of error handling. So, if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of multiple methods, one method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"The Collector/Extractor Pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"A typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the Front End"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor, on the other hand, would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if something failed // get piece of information and return it } ... // More datafields }","title":"Typical Implementation of a Single Data Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Information can be represented as a list. In NewPipe, a list is represented by an InfoItemsCollector . An InfoItemsCollector will collect and assemble a list of InfoItem . For each item that should be extracted, a new Extractor must be created, and given to the InfoItemsCollector via commit() . If you are implementing a list in your service you need to implement an InfoItemExtractor , that will be able to retrieve data for one and only one InfoItem. This extractor will then be comitted to the InfoItemsCollector that can collect the type of InfoItems you want to generate. A common implementation would look like this: private SomeInfoItemCollector collectInfoItemsFromElement(Element e) { // See *Some* as something like Stream or Channel // e.g. StreamInfoItemsCollector, and ChannelInfoItemsCollector are provided by NP SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor Pattern for Lists"},{"location":"01_Concept_of_the_extractor/#listextractor","text":"There is more to know about lists: When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail, and its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. Both of these Problems are fixed by the ListExtractor which takes care about extracting additional metadata about the list, and by chopping down lists into several pages, so called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. For extracting list header information a ListExtractor behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular web page, but all the others as an AJAX request. An InfoItemsPage itself has two constructors which take these parameters: - The InfoitemsCollector of the list that the page should represent - A nextPageUrl which represents the url of the following page (may be null if not page follows). - Optionally errors which is a list of Exceptions that may have happened during extracton. Here is a simplified reference implementation of a list extractor that only extracts pages, but not metadata: class MyListExtractor extends ListExtractor { ... private Document document; ... public InfoItemsPage getPage(pageUrl) throws ExtractionException { SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); document = myFunctionToGetThePageHTMLWhatever(pageUrl); //remember this part from the simple list extraction for(final Element li : document.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return new InfoItemsPage(collector, myFunctionToGetTheNextPageUrl(document)); } public InfoItemsPage getInitialPage() { //document here got initialized by the fetch() function. return getPage(getTheCurrentPageUrl(document)); } ... }","title":"ListExtractor"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of the LinkHandler The LinkHandler represent links to resources like videos, search requests, channels, etc. The idea is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (the default URL for YouTube) https://youtu.be/oHg5SJYRHA0 (the shortened link) https://m.youtube.com/watch?v=oHg5SJYRHA0 (the link for mobile devices) Important notes about LinkHandler A simple LinkHandler will contain the default URL, the ID, and the original URL. LinkHandler s are read only. LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage The typical usage for obtaining a LinkHandler would look like this: LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHandlerFactory can handle this type of link. } } ListLinkHandler and SearchQueryHandler List based resources, like channels and playlists, can be sorted and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler , which inherits from LinkHandler and adds the field ContentFilter , which is used to filter by resource type, like stream or playlist, and SortFilter , which is used to sort by name, date, or view count. !!ATTENTION!! Be careful when you implement a content filter: No selected filter equals all filters selected. If your get an empty content filter list in your extractor, make sure you return everything. By all means, use \"if\" statements like contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler . These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of the LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-the-linkhandler","text":"The LinkHandler represent links to resources like videos, search requests, channels, etc. The idea is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (the default URL for YouTube) https://youtu.be/oHg5SJYRHA0 (the shortened link) https://m.youtube.com/watch?v=oHg5SJYRHA0 (the link for mobile devices)","title":"Concept of the LinkHandler"},{"location":"02_Concept_of_LinkHandler/#important-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID, and the original URL. LinkHandler s are read only. LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Important notes about LinkHandler"},{"location":"02_Concept_of_LinkHandler/#usage","text":"The typical usage for obtaining a LinkHandler would look like this: LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHandlerFactory can handle this type of link. } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources, like channels and playlists, can be sorted and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler , which inherits from LinkHandler and adds the field ContentFilter , which is used to filter by resource type, like stream or playlist, and SortFilter , which is used to sort by name, date, or view count. !!ATTENTION!! Be careful when you implement a content filter: No selected filter equals all filters selected. If your get an empty content filter list in your extractor, make sure you return everything. By all means, use \"if\" statements like contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler . These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implementing a Service Services, or better service connectors, are the parts of NewPipe which communicate with an actual service like YouTube. This page will describe how you can implement and add your own services to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before continuing. Required and Optional Parts Your service does not have to implement everything; some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels. However, if you start to implement one of the optional parts of the list below, you will have to implement all of its parts/classes. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The Parts of a Service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already includes a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks, Regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using Regex too often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries, please tell us before you do so. Head of Service First of all, if you want to create a new service, you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts Required to be Implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add and should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some metadata about your service such as the name, capabilities, the author's name, and their email address for further notice and maintenance issues. Remember, after extending this class, you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way, your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio. They have metadata like a title, a description, next/related videos, a thumbnail and comments. To obtain the URL to the actual stream data, as well as its metadata, StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such streams, like a search result or a playlist. Since every streaming service (obviously) provides streams, this is required to implement. Otherwise, your service was pretty useless :) Parts Required to be Implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support suggestions as you type, you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to also display a suggestion while typing. Parts Required to be Implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you might not have to be concerned over locally saved playlists, those will be handled by the frontend). A playlist may only contain StreamInfoItems , but no other InfoItem types. Parts Required to be Implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only difference is that it does not only represent a simple list of streams, but also a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the ChannelExtractor differs from the one of a playlist. Parts Required to be Implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators Selection\" etc. Kiosks are controversial; many people may not like them. If you also don't like them, please consider your users and refrain from denying support for them. Your service would look pretty empty if you select it and no video is being displayed. Also, you should not override the preference of the user, since users of NewPipe can decide by the settings whether they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure, every page you want to support has to be implemented as its own KioskExtractor . However, if the pages are similar, you can use the same implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts Required to be Implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implementing a Service"},{"location":"03_Implement_a_service/#implementing-a-service","text":"Services, or better service connectors, are the parts of NewPipe which communicate with an actual service like YouTube. This page will describe how you can implement and add your own services to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before continuing.","title":"Implementing a Service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything; some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels. However, if you start to implement one of the optional parts of the list below, you will have to implement all of its parts/classes. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The Parts of a Service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and Optional Parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already includes a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks, Regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using Regex too often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries, please tell us before you do so.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all, if you want to create a new service, you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts Required to be Implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add and should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some metadata about your service such as the name, capabilities, the author's name, and their email address for further notice and maintenance issues. Remember, after extending this class, you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way, your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio. They have metadata like a title, a description, next/related videos, a thumbnail and comments. To obtain the URL to the actual stream data, as well as its metadata, StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such streams, like a search result or a playlist. Since every streaming service (obviously) provides streams, this is required to implement. Otherwise, your service was pretty useless :) Parts Required to be Implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support suggestions as you type, you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to also display a suggestion while typing. Parts Required to be Implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you might not have to be concerned over locally saved playlists, those will be handled by the frontend). A playlist may only contain StreamInfoItems , but no other InfoItem types. Parts Required to be Implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only difference is that it does not only represent a simple list of streams, but also a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the ChannelExtractor differs from the one of a playlist. Parts Required to be Implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators Selection\" etc. Kiosks are controversial; many people may not like them. If you also don't like them, please consider your users and refrain from denying support for them. Your service would look pretty empty if you select it and no video is being displayed. Also, you should not override the preference of the user, since users of NewPipe can decide by the settings whether they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure, every page you want to support has to be implemented as its own KioskExtractor . However, if the pages are similar, you can use the same implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts Required to be Implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Testing Your Changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it with the actual fronted, you need to follow these steps. Setup Android Studio First, you'll want to set up a working Android Studio environment. To do this, download Studio from developer.android.com , and follow the instructions on how to set it up. Get the NewPipe Code and Run it. In order to get it, you simply clone or download it from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Also, make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under Linux, where you sometimes have to adjust the udev rules in order to make your device accessible . Run Your Changes on the Extractor There are several ways to test your extractor version in NewPipe. We will show you the most convenient ones: Using local folder In NewPipe app root folder, edit settings.gradle file and add this: includeBuild('../NewPipeExtractor') { dependencySubstitution { substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor') } } includeBuild should have the relative path as argument. ../NewPipeExtractor means one folder up in hierarchy, and the folder is exactly named NewPipeExtractor . If that's not the case, edit this part. Using JitPack Another way is to use JitPack . This is a build service that can build maven *.jar packages for Android and Java based on GitHub or GitLab repositories. To use the extractor through JitPack, you need to push it to your online repository of your copy that you host either on GitHub or GitLab . It's important to host it on one of both. To copy your repository URL in HTTP format, go to JitPack and paste it there. From here, you can grab the latest commit via GET IT button. I recommend not to use a SNAPSHOT, since I am not sure when snapshot is built. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well, then you should only see a screen with OK signs. Now you can compile and run NewPipe with the new extractor. Troubleshooting If something went wrong on JitPack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it's red, it means that the build failed.","title":"Testing Your Changes in the App"},{"location":"04_Run_changes_in_App/#testing-your-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it with the actual fronted, you need to follow these steps.","title":"Testing Your Changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First, you'll want to set up a working Android Studio environment. To do this, download Studio from developer.android.com , and follow the instructions on how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it, you simply clone or download it from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Also, make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under Linux, where you sometimes have to adjust the udev rules in order to make your device accessible .","title":"Get the NewPipe Code and Run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"There are several ways to test your extractor version in NewPipe. We will show you the most convenient ones:","title":"Run Your Changes on the Extractor"},{"location":"04_Run_changes_in_App/#using-local-folder","text":"In NewPipe app root folder, edit settings.gradle file and add this: includeBuild('../NewPipeExtractor') { dependencySubstitution { substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor') } } includeBuild should have the relative path as argument. ../NewPipeExtractor means one folder up in hierarchy, and the folder is exactly named NewPipeExtractor . If that's not the case, edit this part.","title":"Using local folder"},{"location":"04_Run_changes_in_App/#using-jitpack","text":"Another way is to use JitPack . This is a build service that can build maven *.jar packages for Android and Java based on GitHub or GitLab repositories. To use the extractor through JitPack, you need to push it to your online repository of your copy that you host either on GitHub or GitLab . It's important to host it on one of both. To copy your repository URL in HTTP format, go to JitPack and paste it there. From here, you can grab the latest commit via GET IT button. I recommend not to use a SNAPSHOT, since I am not sure when snapshot is built. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well, then you should only see a screen with OK signs. Now you can compile and run NewPipe with the new extractor.","title":"Using JitPack"},{"location":"04_Run_changes_in_App/#troubleshooting","text":"If something went wrong on JitPack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it's red, it means that the build failed.","title":"Troubleshooting"},{"location":"05_releasing/","text":"Releasing a New NewPipe Version This site is meant for those who want to maintain NewPipe, or just want to know how releasing works. Differences Between Regular and Hotfix Releases NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, this however has the disadvantage of the app to break instantly when YouTube changes something. We do not know when this happen. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as much as possible. The entire release cycle is therefore designed around this issue. There is a difference between a release that introduces new features and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown). Lets have a look at the characteristics of a regular release , and then the characteristics of a hotfix release . Regular Releases Regular releases are normal releases like they are done in any other app. Releases are always stored on master branch. The latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version. Feature Branching When developing, the dev branch is used. Pushing to dev directly, however, is not allowed, since QA and testing should be done first before adding something to it. This ensures that the dev version works as stable a possible. In order to change something on the app, one may want to fork the dev branch and develop the changes in their own branch (this is called feature branching). Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other. If a change is done on the API to the extractor, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes on the fronted at any time. Merging Features/Bugfixes After finishing a feature, one should open up a Pull Request to the dev branch. From here, a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer, please take care about the code architecture so corrosion or code shifting can be prevented. Please also prioritize code quality over functionality. In short: cool function but bad code = no merge. Focus on leaving the code as clean as possible. You, as a maintainer, should build the app and put the signed APK into the description of that new pull request. This way, other people can test the feature/bugfix and help with QA. You may not need to do this every time. It is enough to do it on bigger pull requests. After the maintainer merges the new feature into the dev branch, he should add the title of the pull request or a summary of the changes into the release notes . Creating a New Release Once there are enough features together, and the maintainers believe that NewPipe is ready for a new release, they should create a new release. Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: Don't do a release if you don't have time for it!!! Below is a list of things you will want to do: Fork the dev branch into a new release_x.y.z branch. Increase the version number Merge Weblate changes from the dev branch at https://hosted.weblate.org/git/newpipe/strings/ . Copy the release notes from the GitHub version draft into the corresponding fastlane file (see release notes ). Open up a pull request form the new release_x.y.z branch into the master branch. Create an issue pointing to the new pull request. The reason for opening an issue is that from my perception, people read issues more than pull requests. Put the release-note into this pull request. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file NewPipe__RC1.apk . Zip it and post it to the head of the release issue. This way, others can test the release candidate. Test and QA the new version with the help of others. Leave the PR open for a few days and advertise it to help testing. While being in release phase no new pull requests must be merged into dev branch. This procedure does not have to be done for the extractor as extractor will be tested together with the fronted. Quickfixes When issuing a new release, you will most likely encounter bugs that might not have existed in previous versions. These are called regressions . If you find a regression during release phase, you are allowed to push fixes directly into the release branch without having to fork a branch away from it. All maintainers have to be aware that they might be required to fix regressions, so plan your release at a time when you are available. Do not introduce new features during the release phase. When you have pushed a quickfix, you will want to update the release candidate you put into the issue corresponding to the release pull request . Increment the version number in the filename of the release candidate. e.g. NewPipe__RC2.apk etc. Don't update the actual version number. :P Releasing Once the glorious day of all days has come, and you fulfill the ceremony of releasing. After going through the release procedure of creating a new release and maybe a few quickfixes on the new release, this is what you should do when releasing: Click \"Merge Pull Request\". Create a GPG signed tag with the name v0.x.y . Merge dev into master on the extractor. Create a GPG signed tag with the name v0.x.y on the extractor. Make sure the draft name equals the tag name. Make sure to not have forgotten anything. Click \"Publish Release\". Rebase quickfix changes back into dev if quickfixes were made. Hotfix Releases As aforementioned, NewPipe is a web crawler and could break at any moment. In order to keep the downtime of NewPipe as low as possible, when such a shutdown happens, we allow hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or unrelated bugfixes. A hotfix may only focus on fixing what caused the shutdown. Hotfix Branch Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch. Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well. Releasing If you fixed the issue and found it to be tested and reviewed well enough, you may release it. You don't need to undergo the full release procedure of a regular release, which takes more time to release. Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix. It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better then a non-functional version \u00af\\_(\u30c4)_/\u00af. Here's what you do when releasing a hotfix: Click \"Merge Pull Request\" Create a GPG signed tag with the name v0.x.y . Merge dev into master on the extractor. Create a GPG signed tag with the name v0.x.y on the extractor. Create a new release draft and write the down the fix into the release notes. Copy the release note into the fastlane directory of releases. Increment the small minor version number and the versionCode . Click \"Publish Release\". Rebase the hotfix back into dev branch. Version Nomenclature The version nomenclature of NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. \u00af\\_(\u30c4)_/\u00af Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) will be incremented if there are bug fixes or minor features added to the app. Version Nomenclature of the Extractor The extractor is always released together with the app, therefore the version number of the extractor is identical to the one of NewPipe itself. Version Code In Android, an app can also have a versionCode . This code is a long integer and can be incremented by any value to show a device that a new version is there. For NewPipe, the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps are reserved for our internal F-Droid build server. Release Notes Release notes should tell what was changed in the new version of the app. The release nodes for NewPipe are stored in the GitHub draft for a new release . When a maintainer wants to add changes to the release note, but there is no draft for a new version, they should create one. Changes can be categorized into three types: New : New features that got added to the app. Improved : Improvements to the app or existing features Fixes : Bugfixes When releasing a new version of NewPipe, before actually clicking \"Release\", the maintainer should copy the release notes from the draft and put it into a file called .txt (whereas needs to be the version code of the incoming release). This file must be stored in the directory /fastlane/metadata/android/en-US/changelogs . This way, F-Droid will be able to show the changes done to the app.","title":"Releasing a New NewPipe Version"},{"location":"05_releasing/#releasing-a-new-newpipe-version","text":"This site is meant for those who want to maintain NewPipe, or just want to know how releasing works.","title":"Releasing a New NewPipe Version"},{"location":"05_releasing/#differences-between-regular-and-hotfix-releases","text":"NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, this however has the disadvantage of the app to break instantly when YouTube changes something. We do not know when this happen. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as much as possible. The entire release cycle is therefore designed around this issue. There is a difference between a release that introduces new features and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown). Lets have a look at the characteristics of a regular release , and then the characteristics of a hotfix release .","title":"Differences Between Regular and Hotfix Releases"},{"location":"05_releasing/#regular-releases","text":"Regular releases are normal releases like they are done in any other app. Releases are always stored on master branch. The latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version.","title":"Regular Releases"},{"location":"05_releasing/#feature-branching","text":"When developing, the dev branch is used. Pushing to dev directly, however, is not allowed, since QA and testing should be done first before adding something to it. This ensures that the dev version works as stable a possible. In order to change something on the app, one may want to fork the dev branch and develop the changes in their own branch (this is called feature branching). Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other. If a change is done on the API to the extractor, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes on the fronted at any time.","title":"Feature Branching"},{"location":"05_releasing/#merging-featuresbugfixes","text":"After finishing a feature, one should open up a Pull Request to the dev branch. From here, a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer, please take care about the code architecture so corrosion or code shifting can be prevented. Please also prioritize code quality over functionality. In short: cool function but bad code = no merge. Focus on leaving the code as clean as possible. You, as a maintainer, should build the app and put the signed APK into the description of that new pull request. This way, other people can test the feature/bugfix and help with QA. You may not need to do this every time. It is enough to do it on bigger pull requests. After the maintainer merges the new feature into the dev branch, he should add the title of the pull request or a summary of the changes into the release notes .","title":"Merging Features/Bugfixes"},{"location":"05_releasing/#creating-a-new-release","text":"Once there are enough features together, and the maintainers believe that NewPipe is ready for a new release, they should create a new release. Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: Don't do a release if you don't have time for it!!! Below is a list of things you will want to do: Fork the dev branch into a new release_x.y.z branch. Increase the version number Merge Weblate changes from the dev branch at https://hosted.weblate.org/git/newpipe/strings/ . Copy the release notes from the GitHub version draft into the corresponding fastlane file (see release notes ). Open up a pull request form the new release_x.y.z branch into the master branch. Create an issue pointing to the new pull request. The reason for opening an issue is that from my perception, people read issues more than pull requests. Put the release-note into this pull request. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file NewPipe__RC1.apk . Zip it and post it to the head of the release issue. This way, others can test the release candidate. Test and QA the new version with the help of others. Leave the PR open for a few days and advertise it to help testing. While being in release phase no new pull requests must be merged into dev branch. This procedure does not have to be done for the extractor as extractor will be tested together with the fronted.","title":"Creating a New Release"},{"location":"05_releasing/#quickfixes","text":"When issuing a new release, you will most likely encounter bugs that might not have existed in previous versions. These are called regressions . If you find a regression during release phase, you are allowed to push fixes directly into the release branch without having to fork a branch away from it. All maintainers have to be aware that they might be required to fix regressions, so plan your release at a time when you are available. Do not introduce new features during the release phase. When you have pushed a quickfix, you will want to update the release candidate you put into the issue corresponding to the release pull request . Increment the version number in the filename of the release candidate. e.g. NewPipe__RC2.apk etc. Don't update the actual version number. :P","title":"Quickfixes"},{"location":"05_releasing/#releasing","text":"Once the glorious day of all days has come, and you fulfill the ceremony of releasing. After going through the release procedure of creating a new release and maybe a few quickfixes on the new release, this is what you should do when releasing: Click \"Merge Pull Request\". Create a GPG signed tag with the name v0.x.y . Merge dev into master on the extractor. Create a GPG signed tag with the name v0.x.y on the extractor. Make sure the draft name equals the tag name. Make sure to not have forgotten anything. Click \"Publish Release\". Rebase quickfix changes back into dev if quickfixes were made.","title":"Releasing"},{"location":"05_releasing/#hotfix-releases","text":"As aforementioned, NewPipe is a web crawler and could break at any moment. In order to keep the downtime of NewPipe as low as possible, when such a shutdown happens, we allow hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or unrelated bugfixes. A hotfix may only focus on fixing what caused the shutdown.","title":"Hotfix Releases"},{"location":"05_releasing/#hotfix-branch","text":"Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch. Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well.","title":"Hotfix Branch"},{"location":"05_releasing/#releasing_1","text":"If you fixed the issue and found it to be tested and reviewed well enough, you may release it. You don't need to undergo the full release procedure of a regular release, which takes more time to release. Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix. It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better then a non-functional version \u00af\\_(\u30c4)_/\u00af. Here's what you do when releasing a hotfix: Click \"Merge Pull Request\" Create a GPG signed tag with the name v0.x.y . Merge dev into master on the extractor. Create a GPG signed tag with the name v0.x.y on the extractor. Create a new release draft and write the down the fix into the release notes. Copy the release note into the fastlane directory of releases. Increment the small minor version number and the versionCode . Click \"Publish Release\". Rebase the hotfix back into dev branch.","title":"Releasing"},{"location":"05_releasing/#version-nomenclature","text":"The version nomenclature of NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. \u00af\\_(\u30c4)_/\u00af Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) will be incremented if there are bug fixes or minor features added to the app.","title":"Version Nomenclature"},{"location":"05_releasing/#version-nomenclature-of-the-extractor","text":"The extractor is always released together with the app, therefore the version number of the extractor is identical to the one of NewPipe itself.","title":"Version Nomenclature of the Extractor"},{"location":"05_releasing/#version-code","text":"In Android, an app can also have a versionCode . This code is a long integer and can be incremented by any value to show a device that a new version is there. For NewPipe, the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps are reserved for our internal F-Droid build server.","title":"Version Code"},{"location":"05_releasing/#release-notes","text":"Release notes should tell what was changed in the new version of the app. The release nodes for NewPipe are stored in the GitHub draft for a new release . When a maintainer wants to add changes to the release note, but there is no draft for a new version, they should create one. Changes can be categorized into three types: New : New features that got added to the app. Improved : Improvements to the app or existing features Fixes : Bugfixes When releasing a new version of NewPipe, before actually clicking \"Release\", the maintainer should copy the release notes from the draft and put it into a file called .txt (whereas needs to be the version code of the incoming release). This file must be stored in the directory /fastlane/metadata/android/en-US/changelogs . This way, F-Droid will be able to show the changes done to the app.","title":"Release Notes"},{"location":"06_documentation/","text":"About This Documentation The documentation you are currently reading was written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like HTML or LaTeX. Installation Mkdocs is written in Python and is distributed through the Python internal package manager pip , thus you need to get python and pip running on your operating system first. Windows Download the latest Python3 version. When running the setup program, make sure to tick, \"Add Python 3.x to PATH\". Install Python. Open PowerShell or cmd.exe and type: pip3 install mkdocs . MacOS MacOS already includes Python, however, pip is still missing. The easiest and most nondestructive way is to install the MacOS package manager, homebrew , first. The advantage of homebrew is that it will only modify your home directory, and not the root dir, so your OS will not be tampered with. Install homebrew . Install Python from Homebrew, which will also install pip. Enter this command: brew install python . Install mkdocs: pip3 install mkdocs . Linux/*BSD Linux/*BSD also has Python pre-installed. Most distributions also contain pip by default. If it is not installed, you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly. Android/ChromeOS This might sound funny, but according to the growing amount of Chromebooks and Android tablets with keyboards, this might actually be useful. Install the Termux App from F-Droid . Launch Termux and type apt update Install Python and Git with the command: apt install git python Install mkdocs with pip install mkdocs . From herein, everything will be the same as on Desktop. If you want to edit the files, you can (besides vim or emacs which are available through Termux) use your preferred text editor on Android. This is possible by opening the files with the Termux integration of the build in android file manager: Updating Sometimes, mkdocs changes the way of how it serves, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To check, simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a Linux/BSD* system. Using mkdocs In order to extend this documentation, you have to clone it from its GitHub repository . When you clone it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. Here is a guide about how to use mkdocs. Write and Deploy If you are writing a documentation page and want a live preview of it, you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal web server on port 8000 . So all you have to do is type localhost:8000 into the address bar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page so it will be up to date at the GitHub pages , simply type mkdocs gh-deploy . However, please be aware that this will not push your changes to the master branch of the repository. So, you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only privileged maintainers can do this.","title":"About This Documentation"},{"location":"06_documentation/#about-this-documentation","text":"The documentation you are currently reading was written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like HTML or LaTeX.","title":"About This Documentation"},{"location":"06_documentation/#installation","text":"Mkdocs is written in Python and is distributed through the Python internal package manager pip , thus you need to get python and pip running on your operating system first.","title":"Installation"},{"location":"06_documentation/#windows","text":"Download the latest Python3 version. When running the setup program, make sure to tick, \"Add Python 3.x to PATH\". Install Python. Open PowerShell or cmd.exe and type: pip3 install mkdocs .","title":"Windows"},{"location":"06_documentation/#macos","text":"MacOS already includes Python, however, pip is still missing. The easiest and most nondestructive way is to install the MacOS package manager, homebrew , first. The advantage of homebrew is that it will only modify your home directory, and not the root dir, so your OS will not be tampered with. Install homebrew . Install Python from Homebrew, which will also install pip. Enter this command: brew install python . Install mkdocs: pip3 install mkdocs .","title":"MacOS"},{"location":"06_documentation/#linuxbsd","text":"Linux/*BSD also has Python pre-installed. Most distributions also contain pip by default. If it is not installed, you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly.","title":"Linux/*BSD"},{"location":"06_documentation/#androidchromeos","text":"This might sound funny, but according to the growing amount of Chromebooks and Android tablets with keyboards, this might actually be useful. Install the Termux App from F-Droid . Launch Termux and type apt update Install Python and Git with the command: apt install git python Install mkdocs with pip install mkdocs . From herein, everything will be the same as on Desktop. If you want to edit the files, you can (besides vim or emacs which are available through Termux) use your preferred text editor on Android. This is possible by opening the files with the Termux integration of the build in android file manager:","title":"Android/ChromeOS"},{"location":"06_documentation/#updating","text":"Sometimes, mkdocs changes the way of how it serves, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To check, simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a Linux/BSD* system.","title":"Updating"},{"location":"06_documentation/#using-mkdocs","text":"In order to extend this documentation, you have to clone it from its GitHub repository . When you clone it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. Here is a guide about how to use mkdocs.","title":"Using mkdocs"},{"location":"06_documentation/#write-and-deploy","text":"If you are writing a documentation page and want a live preview of it, you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal web server on port 8000 . So all you have to do is type localhost:8000 into the address bar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page so it will be up to date at the GitHub pages , simply type mkdocs gh-deploy . However, please be aware that this will not push your changes to the master branch of the repository. So, you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only privileged maintainers can do this.","title":"Write and Deploy"},{"location":"07_maintainers_view/","text":"Maintainers' Section These are some basic principles that we want maintainers to follow when maintaining NewPipe. Keep it Streamlined NewPipe is a media player for devices on the Android platform, thus it is intended to be used for entertainment. This means it does not have to be some professional application, and it does not have to be complicated to be used. However NewPipe might not focus on the casual user completely as there are some features designed for more experienced users which may require some knowledge about code, however in essence NewPipe should be easy to use, even for an average Android user. Don't add too many special features. NewPipe does not have to be an airplane cockpit. Do not try to fill every single niche that might exist. If people wanted more advanced features, they would use professional tools. If you add too much functionality, you add complexity, and complexity scares away the average user. Focus on NewPipe's scope as a media player for the end user, and only as such. Usability of the user interface should be prioritized. Try to make it comply with material design guidelines . Bugfixes ] Disclaimer: This is a meme. Please don't take it personally. Always prioritize fixing bugs , as the best application with the best features does not help much if it is broken, or annoying to use. Now if a program is in an early stage it is quite understandable that many things break. This is one reason why NewPipe still has no \"1\" in the beginning of its version number. By now, NewPipe is in a stage where there should be a strong focus on stability. If there are multiple Pull Requests open, check the ones with bugfixes first. Do not add too many features every version, as every feature will inevitably introduce more bugs. It is OK if PRs remain open for a while, but don't leave them open for too long. If there are bugs that are stale, or open for a while bump them from time to time, so devs know that there is still something left to fix. Never merge PRs with known issues. From our perception the community does not like to fix bugs, this is why you as a maintainer should especially focus on pursuing bugs. Features Features are also something that can cause a headache. You should not blindly say yes to features, even if they are small, but you should also not immediately say no. If you are not sure, try the feature, look into the code, speak with the developer, and then make a decision. When considering a feature, ask yourself the following questions: Was the feature requested by only a few, or by many? Avoid introducing niche features to satisfy a small handful of users. Was the code rushed and messy, and could a cleaner solution be made? A pull request that adds a frequently requested feature could implement the feature in a messy way. Such PRs should not be merged as it will likely cause problems later down the line, either through problems of extending the feature by introducing many bugs, or simply by breaking the architecture or the philosophy of NewPipe. Does the amount of code justify the feature's purpose? Use critical thinking when considering new features and question whether that features makes sense, is useful, and if it would benefit NewPipe's users. Pull Requests If a PR contains more than one feature/bugfix, be cautious. The more stuff a PR changes, the longer it will take to be added. There also might be things that seem to not have any issues, but other things will, and this would prevent you from merging a PR. This is why it is encouraged to keep one change per pull request, and you should insist that contributors divide such PRs into multiple smaller PRs when possible. Community When you talk to the community, stay friendly and respectful with good etiquette. When you have a bad day, just don't go to GitHub (advice from our experience ;D ).","title":"Maintainers' Section"},{"location":"07_maintainers_view/#maintainers-section","text":"These are some basic principles that we want maintainers to follow when maintaining NewPipe.","title":"Maintainers' Section"},{"location":"07_maintainers_view/#keep-it-streamlined","text":"NewPipe is a media player for devices on the Android platform, thus it is intended to be used for entertainment. This means it does not have to be some professional application, and it does not have to be complicated to be used. However NewPipe might not focus on the casual user completely as there are some features designed for more experienced users which may require some knowledge about code, however in essence NewPipe should be easy to use, even for an average Android user. Don't add too many special features. NewPipe does not have to be an airplane cockpit. Do not try to fill every single niche that might exist. If people wanted more advanced features, they would use professional tools. If you add too much functionality, you add complexity, and complexity scares away the average user. Focus on NewPipe's scope as a media player for the end user, and only as such. Usability of the user interface should be prioritized. Try to make it comply with material design guidelines .","title":"Keep it Streamlined"},{"location":"07_maintainers_view/#bugfixes","text":"] Disclaimer: This is a meme. Please don't take it personally. Always prioritize fixing bugs , as the best application with the best features does not help much if it is broken, or annoying to use. Now if a program is in an early stage it is quite understandable that many things break. This is one reason why NewPipe still has no \"1\" in the beginning of its version number. By now, NewPipe is in a stage where there should be a strong focus on stability. If there are multiple Pull Requests open, check the ones with bugfixes first. Do not add too many features every version, as every feature will inevitably introduce more bugs. It is OK if PRs remain open for a while, but don't leave them open for too long. If there are bugs that are stale, or open for a while bump them from time to time, so devs know that there is still something left to fix. Never merge PRs with known issues. From our perception the community does not like to fix bugs, this is why you as a maintainer should especially focus on pursuing bugs.","title":"Bugfixes"},{"location":"07_maintainers_view/#features","text":"Features are also something that can cause a headache. You should not blindly say yes to features, even if they are small, but you should also not immediately say no. If you are not sure, try the feature, look into the code, speak with the developer, and then make a decision. When considering a feature, ask yourself the following questions: Was the feature requested by only a few, or by many? Avoid introducing niche features to satisfy a small handful of users. Was the code rushed and messy, and could a cleaner solution be made? A pull request that adds a frequently requested feature could implement the feature in a messy way. Such PRs should not be merged as it will likely cause problems later down the line, either through problems of extending the feature by introducing many bugs, or simply by breaking the architecture or the philosophy of NewPipe. Does the amount of code justify the feature's purpose? Use critical thinking when considering new features and question whether that features makes sense, is useful, and if it would benefit NewPipe's users.","title":"Features"},{"location":"07_maintainers_view/#pull-requests","text":"If a PR contains more than one feature/bugfix, be cautious. The more stuff a PR changes, the longer it will take to be added. There also might be things that seem to not have any issues, but other things will, and this would prevent you from merging a PR. This is why it is encouraged to keep one change per pull request, and you should insist that contributors divide such PRs into multiple smaller PRs when possible.","title":"Pull Requests"},{"location":"07_maintainers_view/#community","text":"When you talk to the community, stay friendly and respectful with good etiquette. When you have a bad day, just don't go to GitHub (advice from our experience ;D ).","title":"Community"}]} \ No newline at end of file +{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to NewPipe This site is/should be a beginner friendly tutorial and documentation for people who want to use or write services for the NewPipe Extractor . However, it also contains several notes about how to maintain NewPipe. It is an addition to our auto generated Jdoc documentation . Please be aware that it is in its early stages, so help and feedback is always appreciated :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming app NewPipe for Android. It is entirely independent from said platforms and also available for additional platforms as well. The beauty behind this framework is that it takes care of the extracting process, error handling etc. so you can focus on what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Welcome to NewPipe"},{"location":"#welcome-to-newpipe","text":"This site is/should be a beginner friendly tutorial and documentation for people who want to use or write services for the NewPipe Extractor . However, it also contains several notes about how to maintain NewPipe. It is an addition to our auto generated Jdoc documentation . Please be aware that it is in its early stages, so help and feedback is always appreciated :D","title":"Welcome to NewPipe"},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming app NewPipe for Android. It is entirely independent from said platforms and also available for additional platforms as well. The beauty behind this framework is that it takes care of the extracting process, error handling etc. so you can focus on what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Before You Start These documents will guide you through the process of understanding or creating your own Extractor service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube, SoundCloud and MediaCCC. The whole documentation consists of this page and Jdoc setup, which explains the general concept of the NewPipeExtractor. IMPORTANT!!! This is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :) Setting Up Your Dev Environment First and foremost, you need to meet the following conditions in order to write your own service. What You Need to Know: A basic understanding of Git Good Java knowledge A good understanding of web technology A basic understanding of unit testing and JUnit A thorough understanding of how to contribute to the NewPipe project Tools/Programs You Will Need: A dev environment/IDE that supports: Git Java 8 Gradle Unit testing IDEA Community (Strongly recommended, but not required) A Github account A lot of patience and excitement ;D After making sure all these conditions are provided, fork the NewPipeExtractor using the fork button . This is so you have a personal repository to develop on. Next, clone this repository into your local folder in which you want to work in. Then, import the cloned project into your IDE and run it. If all the checks are green, you did everything right! You can proceed to the next chapter. Importing the NewPipe Extractor in IntelliJ IDEA If you use IntelliJ IDEA, you should know the easy way of importing the NewPipe extractor. If you don't, here's how to do it: git clone the extractor onto your computer locally. Start IntelliJ Idea and click Import Project . Select the root directory of the NewPipe Extractor. Select \" Import Project from external Model \" and then choose Gradle . In the next window, select \" Use gradle 'wrapper' task configuration \". Running \"test\" in Android Studio/IntelliJ IDEA Go to Run > Edit Configurations > Add New Configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As a task, add \"test\". Now save and you should be able to run. Inclusion Criteria for Services After creating you own service, you will need to submit it to our NewPipeExtractor repository. However, in order to include your changes, you need to follow these rules: Stick to our code contribution guidelines . Do not send services that present content we don't allow on NewPipe. You must be willing to maintain your service after submission. Be patient and make the requested changes when one of our maintainers rejects your code. Content That is Permitted Any content that is not in the list of prohibited content . Any kind of pornography or NSFW content that does not violate US law. However, porn services will not be added to the official NewPipe app. Advertising, which may need to be approved beforehand. Content That is NOT Permitted Content that is considered NSFL (Not Safe For Life). Content that is prohibited by US federal law (Sexualization of minors, any form of violence, violations of human rights, etc). Copyrighted media, without the consent of the copyright holder/publisher.","title":"Before You Start"},{"location":"00_Prepare_everything/#before-you-start","text":"These documents will guide you through the process of understanding or creating your own Extractor service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube, SoundCloud and MediaCCC. The whole documentation consists of this page and Jdoc setup, which explains the general concept of the NewPipeExtractor. IMPORTANT!!! This is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :)","title":"Before You Start"},{"location":"00_Prepare_everything/#setting-up-your-dev-environment","text":"First and foremost, you need to meet the following conditions in order to write your own service.","title":"Setting Up Your Dev Environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"A basic understanding of Git Good Java knowledge A good understanding of web technology A basic understanding of unit testing and JUnit A thorough understanding of how to contribute to the NewPipe project","title":"What You Need to Know:"},{"location":"00_Prepare_everything/#toolsprograms-you-will-need","text":"A dev environment/IDE that supports: Git Java 8 Gradle Unit testing IDEA Community (Strongly recommended, but not required) A Github account A lot of patience and excitement ;D After making sure all these conditions are provided, fork the NewPipeExtractor using the fork button . This is so you have a personal repository to develop on. Next, clone this repository into your local folder in which you want to work in. Then, import the cloned project into your IDE and run it. If all the checks are green, you did everything right! You can proceed to the next chapter.","title":"Tools/Programs You Will Need:"},{"location":"00_Prepare_everything/#importing-the-newpipe-extractor-in-intellij-idea","text":"If you use IntelliJ IDEA, you should know the easy way of importing the NewPipe extractor. If you don't, here's how to do it: git clone the extractor onto your computer locally. Start IntelliJ Idea and click Import Project . Select the root directory of the NewPipe Extractor. Select \" Import Project from external Model \" and then choose Gradle . In the next window, select \" Use gradle 'wrapper' task configuration \".","title":"Importing the NewPipe Extractor in IntelliJ IDEA"},{"location":"00_Prepare_everything/#running-test-in-android-studiointellij-idea","text":"Go to Run > Edit Configurations > Add New Configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As a task, add \"test\". Now save and you should be able to run.","title":"Running \"test\" in Android Studio/IntelliJ IDEA"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service, you will need to submit it to our NewPipeExtractor repository. However, in order to include your changes, you need to follow these rules: Stick to our code contribution guidelines . Do not send services that present content we don't allow on NewPipe. You must be willing to maintain your service after submission. Be patient and make the requested changes when one of our maintainers rejects your code.","title":"Inclusion Criteria for Services"},{"location":"00_Prepare_everything/#content-that-is-permitted","text":"Any content that is not in the list of prohibited content . Any kind of pornography or NSFW content that does not violate US law. However, porn services will not be added to the official NewPipe app. Advertising, which may need to be approved beforehand.","title":"Content That is Permitted"},{"location":"00_Prepare_everything/#content-that-is-not-permitted","text":"Content that is considered NSFL (Not Safe For Life). Content that is prohibited by US federal law (Sexualization of minors, any form of violence, violations of human rights, etc). Copyrighted media, without the consent of the copyright holder/publisher.","title":"Content That is NOT Permitted"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor The Collector/Extractor Pattern Before you start coding your own service, you need to understand the basic concept of the extractor itself. There is a pattern you will find all over the code, called the extractor/collector pattern. The idea behind it is that the extractor would produce fragments of data, and the collector would collect them and assemble that data into a readable format for the front end. The collector also controls the parsing process, and takes care of error handling. So, if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of multiple methods, one method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the Front End A typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical Implementation of a Single Data Extractor The typical implementation of a single data extractor, on the other hand, would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if something failed // get piece of information and return it } ... // More datafields } Collector/Extractor Pattern for Lists Information can be represented as a list. In NewPipe, a list is represented by an InfoItemsCollector . An InfoItemsCollector will collect and assemble a list of InfoItem . For each item that should be extracted, a new Extractor must be created, and given to the InfoItemsCollector via commit() . If you are implementing a list in your service you need to implement an InfoItemExtractor , that will be able to retrieve data for one and only one InfoItem. This extractor will then be comitted to the InfoItemsCollector that can collect the type of InfoItems you want to generate. A common implementation would look like this: private SomeInfoItemCollector collectInfoItemsFromElement(Element e) { // See *Some* as something like Stream or Channel // e.g. StreamInfoItemsCollector, and ChannelInfoItemsCollector are provided by NP SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } ListExtractor There is more to know about lists: When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail, and its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. Both of these Problems are fixed by the ListExtractor which takes care about extracting additional metadata about the list, and by chopping down lists into several pages, so called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. For extracting list header information a ListExtractor behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular web page, but all the others as an AJAX request. An InfoItemsPage itself has two constructors which take these parameters: - The InfoitemsCollector of the list that the page should represent - A nextPageUrl which represents the url of the following page (may be null if not page follows). - Optionally errors which is a list of Exceptions that may have happened during extracton. Here is a simplified reference implementation of a list extractor that only extracts pages, but not metadata: class MyListExtractor extends ListExtractor { ... private Document document; ... public InfoItemsPage getPage(pageUrl) throws ExtractionException { SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); document = myFunctionToGetThePageHTMLWhatever(pageUrl); //remember this part from the simple list extraction for(final Element li : document.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return new InfoItemsPage(collector, myFunctionToGetTheNextPageUrl(document)); } public InfoItemsPage getInitialPage() { //document here got initialized by the fetch() function. return getPage(getTheCurrentPageUrl(document)); } ... }","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#the-collectorextractor-pattern","text":"Before you start coding your own service, you need to understand the basic concept of the extractor itself. There is a pattern you will find all over the code, called the extractor/collector pattern. The idea behind it is that the extractor would produce fragments of data, and the collector would collect them and assemble that data into a readable format for the front end. The collector also controls the parsing process, and takes care of error handling. So, if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of multiple methods, one method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"The Collector/Extractor Pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"A typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the Front End"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor, on the other hand, would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataFiled() throws ExtractionException { //The exception needs to be thrown if something failed // get piece of information and return it } ... // More datafields }","title":"Typical Implementation of a Single Data Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Information can be represented as a list. In NewPipe, a list is represented by an InfoItemsCollector . An InfoItemsCollector will collect and assemble a list of InfoItem . For each item that should be extracted, a new Extractor must be created, and given to the InfoItemsCollector via commit() . If you are implementing a list in your service you need to implement an InfoItemExtractor , that will be able to retrieve data for one and only one InfoItem. This extractor will then be comitted to the InfoItemsCollector that can collect the type of InfoItems you want to generate. A common implementation would look like this: private SomeInfoItemCollector collectInfoItemsFromElement(Element e) { // See *Some* as something like Stream or Channel // e.g. StreamInfoItemsCollector, and ChannelInfoItemsCollector are provided by NP SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor Pattern for Lists"},{"location":"01_Concept_of_the_extractor/#listextractor","text":"There is more to know about lists: When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail, and its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. Both of these Problems are fixed by the ListExtractor which takes care about extracting additional metadata about the list, and by chopping down lists into several pages, so called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. For extracting list header information a ListExtractor behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular web page, but all the others as an AJAX request. An InfoItemsPage itself has two constructors which take these parameters: - The InfoitemsCollector of the list that the page should represent - A nextPageUrl which represents the url of the following page (may be null if not page follows). - Optionally errors which is a list of Exceptions that may have happened during extracton. Here is a simplified reference implementation of a list extractor that only extracts pages, but not metadata: class MyListExtractor extends ListExtractor { ... private Document document; ... public InfoItemsPage getPage(pageUrl) throws ExtractionException { SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); document = myFunctionToGetThePageHTMLWhatever(pageUrl); //remember this part from the simple list extraction for(final Element li : document.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return new InfoItemsPage(collector, myFunctionToGetTheNextPageUrl(document)); } public InfoItemsPage getInitialPage() { //document here got initialized by the fetch() function. return getPage(getTheCurrentPageUrl(document)); } ... }","title":"ListExtractor"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of the LinkHandler The LinkHandler represent links to resources like videos, search requests, channels, etc. The idea is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (the default URL for YouTube) https://youtu.be/oHg5SJYRHA0 (the shortened link) https://m.youtube.com/watch?v=oHg5SJYRHA0 (the link for mobile devices) Important notes about LinkHandler A simple LinkHandler will contain the default URL, the ID, and the original URL. LinkHandler s are read only. LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage The typical usage for obtaining a LinkHandler would look like this: LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHandlerFactory can handle this type of link. } } ListLinkHandler and SearchQueryHandler List based resources, like channels and playlists, can be sorted and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler , which inherits from LinkHandler and adds the field ContentFilter , which is used to filter by resource type, like stream or playlist, and SortFilter , which is used to sort by name, date, or view count. !!ATTENTION!! Be careful when you implement a content filter: No selected filter equals all filters selected. If your get an empty content filter list in your extractor, make sure you return everything. By all means, use \"if\" statements like contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler . These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of the LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-the-linkhandler","text":"The LinkHandler represent links to resources like videos, search requests, channels, etc. The idea is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (the default URL for YouTube) https://youtu.be/oHg5SJYRHA0 (the shortened link) https://m.youtube.com/watch?v=oHg5SJYRHA0 (the link for mobile devices)","title":"Concept of the LinkHandler"},{"location":"02_Concept_of_LinkHandler/#important-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID, and the original URL. LinkHandler s are read only. LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Important notes about LinkHandler"},{"location":"02_Concept_of_LinkHandler/#usage","text":"The typical usage for obtaining a LinkHandler would look like this: LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHandlerFactory can handle this type of link. } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources, like channels and playlists, can be sorted and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler , which inherits from LinkHandler and adds the field ContentFilter , which is used to filter by resource type, like stream or playlist, and SortFilter , which is used to sort by name, date, or view count. !!ATTENTION!! Be careful when you implement a content filter: No selected filter equals all filters selected. If your get an empty content filter list in your extractor, make sure you return everything. By all means, use \"if\" statements like contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler . These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implementing a Service Services, or better service connectors, are the parts of NewPipe which communicate with an actual service like YouTube. This page will describe how you can implement and add your own services to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before continuing. Required and Optional Parts Your service does not have to implement everything; some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels. However, if you start to implement one of the optional parts of the list below, you will have to implement all of its parts/classes. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The Parts of a Service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already includes a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks, Regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using Regex too often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries, please tell us before you do so. Head of Service First of all, if you want to create a new service, you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts Required to be Implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add and should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some metadata about your service such as the name, capabilities, the author's name, and their email address for further notice and maintenance issues. Remember, after extending this class, you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way, your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio. They have metadata like a title, a description, next/related videos, a thumbnail and comments. To obtain the URL to the actual stream data, as well as its metadata, StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such streams, like a search result or a playlist. Since every streaming service (obviously) provides streams, this is required to implement. Otherwise, your service was pretty useless :) Parts Required to be Implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support suggestions as you type, you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to also display a suggestion while typing. Parts Required to be Implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you might not have to be concerned over locally saved playlists, those will be handled by the frontend). A playlist may only contain StreamInfoItems , but no other InfoItem types. Parts Required to be Implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only difference is that it does not only represent a simple list of streams, but also a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the ChannelExtractor differs from the one of a playlist. Parts Required to be Implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators Selection\" etc. Kiosks are controversial; many people may not like them. If you also don't like them, please consider your users and refrain from denying support for them. Your service would look pretty empty if you select it and no video is being displayed. Also, you should not override the preference of the user, since users of NewPipe can decide by the settings whether they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure, every page you want to support has to be implemented as its own KioskExtractor . However, if the pages are similar, you can use the same implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts Required to be Implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implementing a Service"},{"location":"03_Implement_a_service/#implementing-a-service","text":"Services, or better service connectors, are the parts of NewPipe which communicate with an actual service like YouTube. This page will describe how you can implement and add your own services to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before continuing.","title":"Implementing a Service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything; some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels. However, if you start to implement one of the optional parts of the list below, you will have to implement all of its parts/classes. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The Parts of a Service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and Optional Parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already includes a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks, Regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using Regex too often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries, please tell us before you do so.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all, if you want to create a new service, you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts Required to be Implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add and should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some metadata about your service such as the name, capabilities, the author's name, and their email address for further notice and maintenance issues. Remember, after extending this class, you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way, your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio. They have metadata like a title, a description, next/related videos, a thumbnail and comments. To obtain the URL to the actual stream data, as well as its metadata, StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such streams, like a search result or a playlist. Since every streaming service (obviously) provides streams, this is required to implement. Otherwise, your service was pretty useless :) Parts Required to be Implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support suggestions as you type, you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to also display a suggestion while typing. Parts Required to be Implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you might not have to be concerned over locally saved playlists, those will be handled by the frontend). A playlist may only contain StreamInfoItems , but no other InfoItem types. Parts Required to be Implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only difference is that it does not only represent a simple list of streams, but also a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the ChannelExtractor differs from the one of a playlist. Parts Required to be Implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators Selection\" etc. Kiosks are controversial; many people may not like them. If you also don't like them, please consider your users and refrain from denying support for them. Your service would look pretty empty if you select it and no video is being displayed. Also, you should not override the preference of the user, since users of NewPipe can decide by the settings whether they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure, every page you want to support has to be implemented as its own KioskExtractor . However, if the pages are similar, you can use the same implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts Required to be Implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Testing Your Changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it with the actual fronted, you need to follow these steps. Setup Android Studio First, you'll want to set up a working Android Studio environment. To do this, download Studio from developer.android.com , and follow the instructions on how to set it up. Get the NewPipe Code and Run it. In order to get it, you simply clone or download it from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Also, make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under Linux, where you sometimes have to adjust the udev rules in order to make your device accessible . Run Your Changes on the Extractor There are several ways to test your extractor version in NewPipe. We will show you the most convenient ones: Using local folder In NewPipe app root folder, edit settings.gradle file and add this: includeBuild('../NewPipeExtractor') { dependencySubstitution { substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor') } } includeBuild should have the relative path as argument. ../NewPipeExtractor means one folder up in hierarchy, and the folder is exactly named NewPipeExtractor . If that's not the case, edit this part. Using JitPack Another way is to use JitPack . This is a build service that can build maven *.jar packages for Android and Java based on GitHub or GitLab repositories. To use the extractor through JitPack, you need to push it to your online repository of your copy that you host either on GitHub or GitLab . It's important to host it on one of both. To copy your repository URL in HTTP format, go to JitPack and paste it there. From here, you can grab the latest commit via GET IT button. I recommend not to use a SNAPSHOT, since I am not sure when snapshot is built. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well, then you should only see a screen with OK signs. Now you can compile and run NewPipe with the new extractor. Troubleshooting If something went wrong on JitPack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it's red, it means that the build failed.","title":"Testing Your Changes in the App"},{"location":"04_Run_changes_in_App/#testing-your-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it with the actual fronted, you need to follow these steps.","title":"Testing Your Changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First, you'll want to set up a working Android Studio environment. To do this, download Studio from developer.android.com , and follow the instructions on how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it, you simply clone or download it from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Also, make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under Linux, where you sometimes have to adjust the udev rules in order to make your device accessible .","title":"Get the NewPipe Code and Run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"There are several ways to test your extractor version in NewPipe. We will show you the most convenient ones:","title":"Run Your Changes on the Extractor"},{"location":"04_Run_changes_in_App/#using-local-folder","text":"In NewPipe app root folder, edit settings.gradle file and add this: includeBuild('../NewPipeExtractor') { dependencySubstitution { substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor') } } includeBuild should have the relative path as argument. ../NewPipeExtractor means one folder up in hierarchy, and the folder is exactly named NewPipeExtractor . If that's not the case, edit this part.","title":"Using local folder"},{"location":"04_Run_changes_in_App/#using-jitpack","text":"Another way is to use JitPack . This is a build service that can build maven *.jar packages for Android and Java based on GitHub or GitLab repositories. To use the extractor through JitPack, you need to push it to your online repository of your copy that you host either on GitHub or GitLab . It's important to host it on one of both. To copy your repository URL in HTTP format, go to JitPack and paste it there. From here, you can grab the latest commit via GET IT button. I recommend not to use a SNAPSHOT, since I am not sure when snapshot is built. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well, then you should only see a screen with OK signs. Now you can compile and run NewPipe with the new extractor.","title":"Using JitPack"},{"location":"04_Run_changes_in_App/#troubleshooting","text":"If something went wrong on JitPack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it's red, it means that the build failed.","title":"Troubleshooting"},{"location":"05_releasing/","text":"Releasing a New NewPipe Version This site is meant for those who want to maintain NewPipe, or just want to know how releasing works. Differences Between Regular and Hotfix Releases NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, this however has the disadvantage of the app to break instantly when YouTube changes something. We do not know when this happen. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as much as possible. The entire release cycle is therefore designed around this issue. There is a difference between a release that introduces new features and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown). Lets have a look at the characteristics of a regular release , and then the characteristics of a hotfix release . Regular Releases Regular releases are normal releases like they are done in any other app. Releases are always stored on master branch. The latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version. Feature Branching When developing, the dev branch is used. Pushing to dev directly, however, is not allowed, since QA and testing should be done first before adding something to it. This ensures that the dev version works as stable a possible. In order to change something on the app, one may want to fork the dev branch and develop the changes in their own branch (this is called feature branching). Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other. If a change is done on the API to the extractor, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes on the fronted at any time. Merging Features/Bugfixes After finishing a feature, one should open up a Pull Request to the dev branch. From here, a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer, please take care about the code architecture so corrosion or code shifting can be prevented. Please also prioritize code quality over functionality. In short: cool function but bad code = no merge. Focus on leaving the code as clean as possible. You, as a maintainer, should build the app and put the signed APK into the description of that new pull request. This way, other people can test the feature/bugfix and help with QA. You may not need to do this every time. It is enough to do it on bigger pull requests. After the maintainer merges the new feature into the dev branch, he should add the title of the pull request or a summary of the changes into the release notes . Creating a New Release Once there are enough features together, and the maintainers believe that NewPipe is ready for a new release, they should create a new release. Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: Don't do a release if you don't have time for it!!! Below is a list of things you will want to do: Fork the dev branch into a new release_x.y.z branch. Increase the version number Merge Weblate changes from the dev branch at https://hosted.weblate.org/git/newpipe/strings/ . Copy the release notes from the GitHub version draft into the corresponding fastlane file (see release notes ). Open up a pull request form the new release_x.y.z branch into the master branch. Create an issue pointing to the new pull request. The reason for opening an issue is that from my perception, people read issues more than pull requests. Put the release-note into this pull request. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file NewPipe__RC1.apk . Zip it and post it to the head of the release issue. This way, others can test the release candidate. Test and QA the new version with the help of others. Leave the PR open for a few days and advertise it to help testing. While being in release phase no new pull requests must be merged into dev branch. This procedure does not have to be done for the extractor as extractor will be tested together with the fronted. Quickfixes When issuing a new release, you will most likely encounter bugs that might not have existed in previous versions. These are called regressions . If you find a regression during release phase, you are allowed to push fixes directly into the release branch without having to fork a branch away from it. All maintainers have to be aware that they might be required to fix regressions, so plan your release at a time when you are available. Do not introduce new features during the release phase. When you have pushed a quickfix, you will want to update the release candidate you put into the issue corresponding to the release pull request . Increment the version number in the filename of the release candidate. e.g. NewPipe__RC2.apk etc. Don't update the actual version number. :P Releasing Once the glorious day of all days has come, and you fulfill the ceremony of releasing. After going through the release procedure of creating a new release and maybe a few quickfixes on the new release, this is what you should do when releasing: Click \"Merge Pull Request\". Create a GPG signed tag with the name v0.x.y . Merge dev into master on the extractor. Create a GPG signed tag with the name v0.x.y on the extractor. Make sure the draft name equals the tag name. Make sure to not have forgotten anything. Click \"Publish Release\". Clone F-Droid / Data . Add the new version in metadata/org.schabi.newpipe.yml . Run fdroid signatures /path/to/newpipe.apk . Create a MR. Rebase quickfix changes back into dev if quickfixes were made. Hotfix Releases As aforementioned, NewPipe is a web crawler and could break at any moment. In order to keep the downtime of NewPipe as low as possible, when such a shutdown happens, we allow hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or unrelated bugfixes. A hotfix may only focus on fixing what caused the shutdown. Hotfix Branch Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch. Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well. Releasing If you fixed the issue and found it to be tested and reviewed well enough, you may release it. You don't need to undergo the full release procedure of a regular release, which takes more time to release. Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix. It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better then a non-functional version \u00af\\_(\u30c4)_/\u00af. Here's what you do when releasing a hotfix: Click \"Merge Pull Request\" Create a GPG signed tag with the name v0.x.y . Merge dev into master on the extractor. Create a GPG signed tag with the name v0.x.y on the extractor. Create a new release draft and write the down the fix into the release notes. Copy the release note into the fastlane directory of releases. Increment the small minor version number and the versionCode . Click \"Publish Release\". Clone F-Droid / Data . Add the new version in metadata/org.schabi.newpipe.yml . Run fdroid signatures /path/to/newpipe.apk . Create a MR. Rebase the hotfix back into dev branch. Version Nomenclature The version nomenclature of NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. \u00af\\_(\u30c4)_/\u00af Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) will be incremented if there are bug fixes or minor features added to the app. Version Nomenclature of the Extractor The extractor is always released together with the app, therefore the version number of the extractor is identical to the one of NewPipe itself. Version Code In Android, an app can also have a versionCode . This code is a long integer and can be incremented by any value to show a device that a new version is there. For NewPipe, the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps are reserved for our internal F-Droid build server. Release Notes Release notes should tell what was changed in the new version of the app. The release nodes for NewPipe are stored in the GitHub draft for a new release . When a maintainer wants to add changes to the release note, but there is no draft for a new version, they should create one. Changes can be categorized into three types: New : New features that got added to the app. Improved : Improvements to the app or existing features Fixes : Bugfixes When releasing a new version of NewPipe, before actually clicking \"Release\", the maintainer should copy the release notes from the draft and put it into a file called .txt (whereas needs to be the version code of the incoming release). This file must be stored in the directory /fastlane/metadata/android/en-US/changelogs . This way, F-Droid will be able to show the changes done to the app.","title":"Releasing a New NewPipe Version"},{"location":"05_releasing/#releasing-a-new-newpipe-version","text":"This site is meant for those who want to maintain NewPipe, or just want to know how releasing works.","title":"Releasing a New NewPipe Version"},{"location":"05_releasing/#differences-between-regular-and-hotfix-releases","text":"NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, this however has the disadvantage of the app to break instantly when YouTube changes something. We do not know when this happen. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as much as possible. The entire release cycle is therefore designed around this issue. There is a difference between a release that introduces new features and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown). Lets have a look at the characteristics of a regular release , and then the characteristics of a hotfix release .","title":"Differences Between Regular and Hotfix Releases"},{"location":"05_releasing/#regular-releases","text":"Regular releases are normal releases like they are done in any other app. Releases are always stored on master branch. The latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version.","title":"Regular Releases"},{"location":"05_releasing/#feature-branching","text":"When developing, the dev branch is used. Pushing to dev directly, however, is not allowed, since QA and testing should be done first before adding something to it. This ensures that the dev version works as stable a possible. In order to change something on the app, one may want to fork the dev branch and develop the changes in their own branch (this is called feature branching). Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other. If a change is done on the API to the extractor, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes on the fronted at any time.","title":"Feature Branching"},{"location":"05_releasing/#merging-featuresbugfixes","text":"After finishing a feature, one should open up a Pull Request to the dev branch. From here, a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer, please take care about the code architecture so corrosion or code shifting can be prevented. Please also prioritize code quality over functionality. In short: cool function but bad code = no merge. Focus on leaving the code as clean as possible. You, as a maintainer, should build the app and put the signed APK into the description of that new pull request. This way, other people can test the feature/bugfix and help with QA. You may not need to do this every time. It is enough to do it on bigger pull requests. After the maintainer merges the new feature into the dev branch, he should add the title of the pull request or a summary of the changes into the release notes .","title":"Merging Features/Bugfixes"},{"location":"05_releasing/#creating-a-new-release","text":"Once there are enough features together, and the maintainers believe that NewPipe is ready for a new release, they should create a new release. Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: Don't do a release if you don't have time for it!!! Below is a list of things you will want to do: Fork the dev branch into a new release_x.y.z branch. Increase the version number Merge Weblate changes from the dev branch at https://hosted.weblate.org/git/newpipe/strings/ . Copy the release notes from the GitHub version draft into the corresponding fastlane file (see release notes ). Open up a pull request form the new release_x.y.z branch into the master branch. Create an issue pointing to the new pull request. The reason for opening an issue is that from my perception, people read issues more than pull requests. Put the release-note into this pull request. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file NewPipe__RC1.apk . Zip it and post it to the head of the release issue. This way, others can test the release candidate. Test and QA the new version with the help of others. Leave the PR open for a few days and advertise it to help testing. While being in release phase no new pull requests must be merged into dev branch. This procedure does not have to be done for the extractor as extractor will be tested together with the fronted.","title":"Creating a New Release"},{"location":"05_releasing/#quickfixes","text":"When issuing a new release, you will most likely encounter bugs that might not have existed in previous versions. These are called regressions . If you find a regression during release phase, you are allowed to push fixes directly into the release branch without having to fork a branch away from it. All maintainers have to be aware that they might be required to fix regressions, so plan your release at a time when you are available. Do not introduce new features during the release phase. When you have pushed a quickfix, you will want to update the release candidate you put into the issue corresponding to the release pull request . Increment the version number in the filename of the release candidate. e.g. NewPipe__RC2.apk etc. Don't update the actual version number. :P","title":"Quickfixes"},{"location":"05_releasing/#releasing","text":"Once the glorious day of all days has come, and you fulfill the ceremony of releasing. After going through the release procedure of creating a new release and maybe a few quickfixes on the new release, this is what you should do when releasing: Click \"Merge Pull Request\". Create a GPG signed tag with the name v0.x.y . Merge dev into master on the extractor. Create a GPG signed tag with the name v0.x.y on the extractor. Make sure the draft name equals the tag name. Make sure to not have forgotten anything. Click \"Publish Release\". Clone F-Droid / Data . Add the new version in metadata/org.schabi.newpipe.yml . Run fdroid signatures /path/to/newpipe.apk . Create a MR. Rebase quickfix changes back into dev if quickfixes were made.","title":"Releasing"},{"location":"05_releasing/#hotfix-releases","text":"As aforementioned, NewPipe is a web crawler and could break at any moment. In order to keep the downtime of NewPipe as low as possible, when such a shutdown happens, we allow hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or unrelated bugfixes. A hotfix may only focus on fixing what caused the shutdown.","title":"Hotfix Releases"},{"location":"05_releasing/#hotfix-branch","text":"Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch. Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well.","title":"Hotfix Branch"},{"location":"05_releasing/#releasing_1","text":"If you fixed the issue and found it to be tested and reviewed well enough, you may release it. You don't need to undergo the full release procedure of a regular release, which takes more time to release. Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix. It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better then a non-functional version \u00af\\_(\u30c4)_/\u00af. Here's what you do when releasing a hotfix: Click \"Merge Pull Request\" Create a GPG signed tag with the name v0.x.y . Merge dev into master on the extractor. Create a GPG signed tag with the name v0.x.y on the extractor. Create a new release draft and write the down the fix into the release notes. Copy the release note into the fastlane directory of releases. Increment the small minor version number and the versionCode . Click \"Publish Release\". Clone F-Droid / Data . Add the new version in metadata/org.schabi.newpipe.yml . Run fdroid signatures /path/to/newpipe.apk . Create a MR. Rebase the hotfix back into dev branch.","title":"Releasing"},{"location":"05_releasing/#version-nomenclature","text":"The version nomenclature of NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. \u00af\\_(\u30c4)_/\u00af Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) will be incremented if there are bug fixes or minor features added to the app.","title":"Version Nomenclature"},{"location":"05_releasing/#version-nomenclature-of-the-extractor","text":"The extractor is always released together with the app, therefore the version number of the extractor is identical to the one of NewPipe itself.","title":"Version Nomenclature of the Extractor"},{"location":"05_releasing/#version-code","text":"In Android, an app can also have a versionCode . This code is a long integer and can be incremented by any value to show a device that a new version is there. For NewPipe, the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps are reserved for our internal F-Droid build server.","title":"Version Code"},{"location":"05_releasing/#release-notes","text":"Release notes should tell what was changed in the new version of the app. The release nodes for NewPipe are stored in the GitHub draft for a new release . When a maintainer wants to add changes to the release note, but there is no draft for a new version, they should create one. Changes can be categorized into three types: New : New features that got added to the app. Improved : Improvements to the app or existing features Fixes : Bugfixes When releasing a new version of NewPipe, before actually clicking \"Release\", the maintainer should copy the release notes from the draft and put it into a file called .txt (whereas needs to be the version code of the incoming release). This file must be stored in the directory /fastlane/metadata/android/en-US/changelogs . This way, F-Droid will be able to show the changes done to the app.","title":"Release Notes"},{"location":"06_documentation/","text":"About This Documentation The documentation you are currently reading was written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like HTML or LaTeX. Installation Mkdocs is written in Python and is distributed through the Python internal package manager pip , thus you need to get python and pip running on your operating system first. Windows Download the latest Python3 version. When running the setup program, make sure to tick, \"Add Python 3.x to PATH\". Install Python. Open PowerShell or cmd.exe and type: pip3 install mkdocs . MacOS MacOS already includes Python, however, pip is still missing. The easiest and most nondestructive way is to install the MacOS package manager, homebrew , first. The advantage of homebrew is that it will only modify your home directory, and not the root dir, so your OS will not be tampered with. Install homebrew . Install Python from Homebrew, which will also install pip. Enter this command: brew install python . Install mkdocs: pip3 install mkdocs . Linux/*BSD Linux/*BSD also has Python pre-installed. Most distributions also contain pip by default. If it is not installed, you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly. Android/ChromeOS This might sound funny, but according to the growing amount of Chromebooks and Android tablets with keyboards, this might actually be useful. Install the Termux App from F-Droid . Launch Termux and type apt update Install Python and Git with the command: apt install git python Install mkdocs with pip install mkdocs . From herein, everything will be the same as on Desktop. If you want to edit the files, you can (besides vim or emacs which are available through Termux) use your preferred text editor on Android. This is possible by opening the files with the Termux integration of the build in android file manager: Updating Sometimes, mkdocs changes the way of how it serves, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To check, simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a Linux/BSD* system. Using mkdocs In order to extend this documentation, you have to clone it from its GitHub repository . When you clone it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. Here is a guide about how to use mkdocs. Write and Deploy If you are writing a documentation page and want a live preview of it, you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal web server on port 8000 . So all you have to do is type localhost:8000 into the address bar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page so it will be up to date at the GitHub pages , simply type mkdocs gh-deploy . However, please be aware that this will not push your changes to the master branch of the repository. So, you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only privileged maintainers can do this.","title":"About This Documentation"},{"location":"06_documentation/#about-this-documentation","text":"The documentation you are currently reading was written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like HTML or LaTeX.","title":"About This Documentation"},{"location":"06_documentation/#installation","text":"Mkdocs is written in Python and is distributed through the Python internal package manager pip , thus you need to get python and pip running on your operating system first.","title":"Installation"},{"location":"06_documentation/#windows","text":"Download the latest Python3 version. When running the setup program, make sure to tick, \"Add Python 3.x to PATH\". Install Python. Open PowerShell or cmd.exe and type: pip3 install mkdocs .","title":"Windows"},{"location":"06_documentation/#macos","text":"MacOS already includes Python, however, pip is still missing. The easiest and most nondestructive way is to install the MacOS package manager, homebrew , first. The advantage of homebrew is that it will only modify your home directory, and not the root dir, so your OS will not be tampered with. Install homebrew . Install Python from Homebrew, which will also install pip. Enter this command: brew install python . Install mkdocs: pip3 install mkdocs .","title":"MacOS"},{"location":"06_documentation/#linuxbsd","text":"Linux/*BSD also has Python pre-installed. Most distributions also contain pip by default. If it is not installed, you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly.","title":"Linux/*BSD"},{"location":"06_documentation/#androidchromeos","text":"This might sound funny, but according to the growing amount of Chromebooks and Android tablets with keyboards, this might actually be useful. Install the Termux App from F-Droid . Launch Termux and type apt update Install Python and Git with the command: apt install git python Install mkdocs with pip install mkdocs . From herein, everything will be the same as on Desktop. If you want to edit the files, you can (besides vim or emacs which are available through Termux) use your preferred text editor on Android. This is possible by opening the files with the Termux integration of the build in android file manager:","title":"Android/ChromeOS"},{"location":"06_documentation/#updating","text":"Sometimes, mkdocs changes the way of how it serves, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To check, simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a Linux/BSD* system.","title":"Updating"},{"location":"06_documentation/#using-mkdocs","text":"In order to extend this documentation, you have to clone it from its GitHub repository . When you clone it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. Here is a guide about how to use mkdocs.","title":"Using mkdocs"},{"location":"06_documentation/#write-and-deploy","text":"If you are writing a documentation page and want a live preview of it, you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal web server on port 8000 . So all you have to do is type localhost:8000 into the address bar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page so it will be up to date at the GitHub pages , simply type mkdocs gh-deploy . However, please be aware that this will not push your changes to the master branch of the repository. So, you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only privileged maintainers can do this.","title":"Write and Deploy"},{"location":"07_maintainers_view/","text":"Maintainers' Section These are some basic principles that we want maintainers to follow when maintaining NewPipe. Keep it Streamlined NewPipe is a media player for devices on the Android platform, thus it is intended to be used for entertainment. This means it does not have to be some professional application, and it does not have to be complicated to be used. However NewPipe might not focus on the casual user completely as there are some features designed for more experienced users which may require some knowledge about code, however in essence NewPipe should be easy to use, even for an average Android user. Don't add too many special features. NewPipe does not have to be an airplane cockpit. Do not try to fill every single niche that might exist. If people wanted more advanced features, they would use professional tools. If you add too much functionality, you add complexity, and complexity scares away the average user. Focus on NewPipe's scope as a media player for the end user, and only as such. Usability of the user interface should be prioritized. Try to make it comply with material design guidelines . Bugfixes ] Disclaimer: This is a meme. Please don't take it personally. Always prioritize fixing bugs , as the best application with the best features does not help much if it is broken, or annoying to use. Now if a program is in an early stage it is quite understandable that many things break. This is one reason why NewPipe still has no \"1\" in the beginning of its version number. By now, NewPipe is in a stage where there should be a strong focus on stability. If there are multiple Pull Requests open, check the ones with bugfixes first. Do not add too many features every version, as every feature will inevitably introduce more bugs. It is OK if PRs remain open for a while, but don't leave them open for too long. If there are bugs that are stale, or open for a while bump them from time to time, so devs know that there is still something left to fix. Never merge PRs with known issues. From our perception the community does not like to fix bugs, this is why you as a maintainer should especially focus on pursuing bugs. Features Features are also something that can cause a headache. You should not blindly say yes to features, even if they are small, but you should also not immediately say no. If you are not sure, try the feature, look into the code, speak with the developer, and then make a decision. When considering a feature, ask yourself the following questions: Was the feature requested by only a few, or by many? Avoid introducing niche features to satisfy a small handful of users. Was the code rushed and messy, and could a cleaner solution be made? A pull request that adds a frequently requested feature could implement the feature in a messy way. Such PRs should not be merged as it will likely cause problems later down the line, either through problems of extending the feature by introducing many bugs, or simply by breaking the architecture or the philosophy of NewPipe. Does the amount of code justify the feature's purpose? Use critical thinking when considering new features and question whether that features makes sense, is useful, and if it would benefit NewPipe's users. Pull Requests If a PR contains more than one feature/bugfix, be cautious. The more stuff a PR changes, the longer it will take to be added. There also might be things that seem to not have any issues, but other things will, and this would prevent you from merging a PR. This is why it is encouraged to keep one change per pull request, and you should insist that contributors divide such PRs into multiple smaller PRs when possible. Community When you talk to the community, stay friendly and respectful with good etiquette. When you have a bad day, just don't go to GitHub (advice from our experience ;D ).","title":"Maintainers' Section"},{"location":"07_maintainers_view/#maintainers-section","text":"These are some basic principles that we want maintainers to follow when maintaining NewPipe.","title":"Maintainers' Section"},{"location":"07_maintainers_view/#keep-it-streamlined","text":"NewPipe is a media player for devices on the Android platform, thus it is intended to be used for entertainment. This means it does not have to be some professional application, and it does not have to be complicated to be used. However NewPipe might not focus on the casual user completely as there are some features designed for more experienced users which may require some knowledge about code, however in essence NewPipe should be easy to use, even for an average Android user. Don't add too many special features. NewPipe does not have to be an airplane cockpit. Do not try to fill every single niche that might exist. If people wanted more advanced features, they would use professional tools. If you add too much functionality, you add complexity, and complexity scares away the average user. Focus on NewPipe's scope as a media player for the end user, and only as such. Usability of the user interface should be prioritized. Try to make it comply with material design guidelines .","title":"Keep it Streamlined"},{"location":"07_maintainers_view/#bugfixes","text":"] Disclaimer: This is a meme. Please don't take it personally. Always prioritize fixing bugs , as the best application with the best features does not help much if it is broken, or annoying to use. Now if a program is in an early stage it is quite understandable that many things break. This is one reason why NewPipe still has no \"1\" in the beginning of its version number. By now, NewPipe is in a stage where there should be a strong focus on stability. If there are multiple Pull Requests open, check the ones with bugfixes first. Do not add too many features every version, as every feature will inevitably introduce more bugs. It is OK if PRs remain open for a while, but don't leave them open for too long. If there are bugs that are stale, or open for a while bump them from time to time, so devs know that there is still something left to fix. Never merge PRs with known issues. From our perception the community does not like to fix bugs, this is why you as a maintainer should especially focus on pursuing bugs.","title":"Bugfixes"},{"location":"07_maintainers_view/#features","text":"Features are also something that can cause a headache. You should not blindly say yes to features, even if they are small, but you should also not immediately say no. If you are not sure, try the feature, look into the code, speak with the developer, and then make a decision. When considering a feature, ask yourself the following questions: Was the feature requested by only a few, or by many? Avoid introducing niche features to satisfy a small handful of users. Was the code rushed and messy, and could a cleaner solution be made? A pull request that adds a frequently requested feature could implement the feature in a messy way. Such PRs should not be merged as it will likely cause problems later down the line, either through problems of extending the feature by introducing many bugs, or simply by breaking the architecture or the philosophy of NewPipe. Does the amount of code justify the feature's purpose? Use critical thinking when considering new features and question whether that features makes sense, is useful, and if it would benefit NewPipe's users.","title":"Features"},{"location":"07_maintainers_view/#pull-requests","text":"If a PR contains more than one feature/bugfix, be cautious. The more stuff a PR changes, the longer it will take to be added. There also might be things that seem to not have any issues, but other things will, and this would prevent you from merging a PR. This is why it is encouraged to keep one change per pull request, and you should insist that contributors divide such PRs into multiple smaller PRs when possible.","title":"Pull Requests"},{"location":"07_maintainers_view/#community","text":"When you talk to the community, stay friendly and respectful with good etiquette. When you have a bad day, just don't go to GitHub (advice from our experience ;D ).","title":"Community"}]} \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index 348e484..8068605 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,47 +2,47 @@ None - 2020-04-19 + 2020-05-21 daily None - 2020-04-19 + 2020-05-21 daily None - 2020-04-19 + 2020-05-21 daily None - 2020-04-19 + 2020-05-21 daily None - 2020-04-19 + 2020-05-21 daily None - 2020-04-19 + 2020-05-21 daily None - 2020-04-19 + 2020-05-21 daily None - 2020-04-19 + 2020-05-21 daily None - 2020-04-19 + 2020-05-21 daily \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index dbd18804eaddb7752a2b254599de96ecb192befe..54a5abd827d3dfb70a024fd4e51007f725b0de21 100644 GIT binary patch literal 202 zcmV;*05$&~iwFqEx5i!q|8r?{Wo=<_E_iKh0PU1P4#FT1MfW`gVPA$eF;PS5&ZR3o z0HN5DP^f@bZ!fjgcmp?`&HVZEGjBJne)I<2l@Hq43KxW8q^xv}wrz#ar$fHQHP6`1 zo1h{}VH+yk#W?H%#&IM;N7aiV5a(M6vP(mN#RN)rm}e-}bYND987KMCS;b&dA5(Ws%m4w?t$oY^yF_U3nw4c{2EMW(s^UD{uv_!2bf@Exir?0(V(NC1wNw E0J%C_umAu6 literal 202 zcmV;*05$&~iwFqY!<=3M|8r?{Wo=<_E_iKh0PU1962c%5MSD-d&<9Ay!Z?I