1 line
134 KiB
JSON
1 line
134 KiB
JSON
{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to the NewPipe Development Docs This site is/should be a beginner friendly tutorial and documentation for people who want to use or write services for the NewPipe Extractor . However, it also contains several notes about how to maintain NewPipe. It is an addition to our auto generated Jdoc documentation . Please be aware that it is in its early stages, so help and feedback is always appreciated :D Introduction The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming app NewPipe for Android. It is entirely independent from said platforms and also available for additional platforms as well. The beauty behind this framework is that it takes care of the extracting process, error handling etc. so you can focus on what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Welcome to the NewPipe Development Docs"},{"location":"#welcome-to-the-newpipe-development-docs","text":"This site is/should be a beginner friendly tutorial and documentation for people who want to use or write services for the NewPipe Extractor . However, it also contains several notes about how to maintain NewPipe. It is an addition to our auto generated Jdoc documentation . Please be aware that it is in its early stages, so help and feedback is always appreciated :D","title":"Welcome to the NewPipe Development Docs"},{"location":"#introduction","text":"The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming app NewPipe for Android. It is entirely independent from said platforms and also available for additional platforms as well. The beauty behind this framework is that it takes care of the extracting process, error handling etc. so you can focus on what is important: Scraping the website. It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.","title":"Introduction"},{"location":"00_Prepare_everything/","text":"Before You Start These documents will guide you through the process of understanding or creating your own Extractor service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube, SoundCloud and MediaCCC. The whole documentation consists of this page and Jdoc setup, which explains the general concept of the NewPipeExtractor. IMPORTANT!!! This is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :) Setting Up Your Dev Environment First and foremost, you need to meet the following conditions in order to write your own service. What You Need to Know: A basic understanding of Git Good Java knowledge A good understanding of web technology A basic understanding of unit testing and JUnit A thorough understanding of how to contribute to the NewPipe project Tools/Programs You Will Need: A dev environment/IDE that supports: Git Java 8 Gradle Unit testing IDEA Community (Strongly recommended, but not required) A Github account A lot of patience and excitement ;D After making sure all these conditions are provided, fork the NewPipeExtractor using the fork button . This is so you have a personal repository to develop on. Next, clone this repository into your local folder in which you want to work in. Then, import the cloned project into your IDE and run it. If all the checks are green, you did everything right! You can proceed to the next chapter. Importing the NewPipe Extractor in IntelliJ IDEA If you use IntelliJ IDEA, you should know the easy way of importing the NewPipe extractor. If you don't, here's how to do it: git clone the extractor onto your computer locally. Start IntelliJ Idea and click Import Project . Select the root directory of the NewPipe Extractor. Select \" Import Project from external Model \" and then choose Gradle . In the next window, select \" Use gradle 'wrapper' task configuration \". Running \"test\" in Android Studio/IntelliJ IDEA Go to Run > Edit Configurations > Add New Configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As a task, add \"test\". Now save and you should be able to run. Inclusion Criteria for Services After creating you own service, you will need to submit it to our NewPipeExtractor repository. However, in order to include your changes, you need to follow these rules: Stick to our code contribution guidelines . Do not send services that present content we don't allow on NewPipe. You must be willing to maintain your service after submission. Be patient and make the requested changes when one of our maintainers rejects your code. Content That is Permitted Any content that is not in the list of prohibited content . Any kind of pornography or NSFW content that does not violate US law. However, porn services will not be added to the official NewPipe app. Advertising, which may need to be approved beforehand. Content That is NOT Permitted Content that is considered NSFL (Not Safe For Life). Content that is prohibited by US federal law (Sexualization of minors, any form of violence, violations of human rights, etc). Copyrighted media, without the consent of the copyright holder/publisher.","title":"Before You Start"},{"location":"00_Prepare_everything/#before-you-start","text":"These documents will guide you through the process of understanding or creating your own Extractor service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube, SoundCloud and MediaCCC. The whole documentation consists of this page and Jdoc setup, which explains the general concept of the NewPipeExtractor. IMPORTANT!!! This is likely to be the worst documentation you have ever read, so do not hesitate to report if you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community and are open for everyone to help :)","title":"Before You Start"},{"location":"00_Prepare_everything/#setting-up-your-dev-environment","text":"First and foremost, you need to meet the following conditions in order to write your own service.","title":"Setting Up Your Dev Environment"},{"location":"00_Prepare_everything/#what-you-need-to-know","text":"A basic understanding of Git Good Java knowledge A good understanding of web technology A basic understanding of unit testing and JUnit A thorough understanding of how to contribute to the NewPipe project","title":"What You Need to Know:"},{"location":"00_Prepare_everything/#toolsprograms-you-will-need","text":"A dev environment/IDE that supports: Git Java 8 Gradle Unit testing IDEA Community (Strongly recommended, but not required) A Github account A lot of patience and excitement ;D After making sure all these conditions are provided, fork the NewPipeExtractor using the fork button . This is so you have a personal repository to develop on. Next, clone this repository into your local folder in which you want to work in. Then, import the cloned project into your IDE and run it. If all the checks are green, you did everything right! You can proceed to the next chapter.","title":"Tools/Programs You Will Need:"},{"location":"00_Prepare_everything/#importing-the-newpipe-extractor-in-intellij-idea","text":"If you use IntelliJ IDEA, you should know the easy way of importing the NewPipe extractor. If you don't, here's how to do it: git clone the extractor onto your computer locally. Start IntelliJ Idea and click Import Project . Select the root directory of the NewPipe Extractor. Select \" Import Project from external Model \" and then choose Gradle . In the next window, select \" Use gradle 'wrapper' task configuration \".","title":"Importing the NewPipe Extractor in IntelliJ IDEA"},{"location":"00_Prepare_everything/#running-test-in-android-studiointellij-idea","text":"Go to Run > Edit Configurations > Add New Configuration and select \"Gradle\". As Gradle Project, select NewPipeExtractor. As a task, add \"test\". Now save and you should be able to run.","title":"Running \"test\" in Android Studio/IntelliJ IDEA"},{"location":"00_Prepare_everything/#inclusion-criteria-for-services","text":"After creating you own service, you will need to submit it to our NewPipeExtractor repository. However, in order to include your changes, you need to follow these rules: Stick to our code contribution guidelines . Do not send services that present content we don't allow on NewPipe. You must be willing to maintain your service after submission. Be patient and make the requested changes when one of our maintainers rejects your code.","title":"Inclusion Criteria for Services"},{"location":"00_Prepare_everything/#content-that-is-permitted","text":"Any content that is not in the list of prohibited content . Any kind of pornography or NSFW content that does not violate US law. However, porn services will not be added to the official NewPipe app. Advertising, which may need to be approved beforehand.","title":"Content That is Permitted"},{"location":"00_Prepare_everything/#content-that-is-not-permitted","text":"Content that is considered NSFL (Not Safe For Life). Content that is prohibited by US federal law (Sexualization of minors, any form of violence, violations of human rights, etc). Copyrighted media, without the consent of the copyright holder/publisher.","title":"Content That is NOT Permitted"},{"location":"01_Concept_of_the_extractor/","text":"Concept of the Extractor The Collector/Extractor Pattern Before you start coding your own service, you need to understand the basic concept of the extractor itself. There is a pattern you will find all over the code, called the extractor/collector pattern. The idea behind it is that the extractor would produce fragments of data, and the collector would collect them and assemble that data into a readable format for the front end. The collector also controls the parsing process, and takes care of error handling. So, if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of multiple methods, one method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors. Usage in the Front End A typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction } Typical Implementation of a Single Data Extractor The typical implementation of a single data extractor, on the other hand, would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataField() throws ExtractionException { //The exception needs to be thrown if something failed // get piece of information and return it } ... // More datafields } Collector/Extractor Pattern for Lists Information can be represented as a list. In NewPipe, a list is represented by an InfoItemsCollector . An InfoItemsCollector will collect and assemble a list of InfoItem . For each item that should be extracted, a new Extractor must be created, and given to the InfoItemsCollector via commit() . If you are implementing a list in your service you need to implement an InfoItemExtractor , that will be able to retrieve data for one and only one InfoItem. This extractor will then be comitted to the InfoItemsCollector that can collect the type of InfoItems you want to generate. A common implementation would look like this: private SomeInfoItemCollector collectInfoItemsFromElement(Element e) { // See *Some* as something like Stream or Channel // e.g. StreamInfoItemsCollector, and ChannelInfoItemsCollector are provided by NP SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; } ListExtractor There is more to know about lists: When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail, and its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. Both of these Problems are fixed by the ListExtractor which takes care about extracting additional metadata about the list, and by chopping down lists into several pages, so called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. For extracting list header information a ListExtractor behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular web page, but all the others as an AJAX request. An InfoItemsPage itself has two constructors which take these parameters: - The InfoitemsCollector of the list that the page should represent - A nextPageUrl which represents the url of the following page (may be null if not page follows). - Optionally errors which is a list of Exceptions that may have happened during extracton. Here is a simplified reference implementation of a list extractor that only extracts pages, but not metadata: class MyListExtractor extends ListExtractor { ... private Document document; ... public InfoItemsPage<SomeInfoItem> getPage(pageUrl) throws ExtractionException { SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); document = myFunctionToGetThePageHTMLWhatever(pageUrl); //remember this part from the simple list extraction for(final Element li : document.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return new InfoItemsPage<SomeInfoItem>(collector, myFunctionToGetTheNextPageUrl(document)); } public InfoItemsPage<SomeInfoItem> getInitialPage() { //document here got initialized by the fetch() function. return getPage(getTheCurrentPageUrl(document)); } ... }","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#concept-of-the-extractor","text":"","title":"Concept of the Extractor"},{"location":"01_Concept_of_the_extractor/#the-collectorextractor-pattern","text":"Before you start coding your own service, you need to understand the basic concept of the extractor itself. There is a pattern you will find all over the code, called the extractor/collector pattern. The idea behind it is that the extractor would produce fragments of data, and the collector would collect them and assemble that data into a readable format for the front end. The collector also controls the parsing process, and takes care of error handling. So, if the extractor fails at any point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of multiple methods, one method for every data field the collector wants to have. The collectors are provided by NewPipe. You need to take care of the extractors.","title":"The Collector/Extractor Pattern"},{"location":"01_Concept_of_the_extractor/#usage-in-the-front-end","text":"A typical call for retrieving data from a website would look like this: Info info; try { // Create a new Extractor with a given context provided as parameter. Extractor extractor = new Extractor(some_meta_info); // Retrieves the data form extractor and builds info package. info = Info.getInfo(extractor); } catch(Exception e) { // handle errors when collector decided to break up extraction }","title":"Usage in the Front End"},{"location":"01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor","text":"The typical implementation of a single data extractor, on the other hand, would look like this: class MyExtractor extends FutureExtractor { public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) { super(requiredInfo, forExtraction); ... } @Override public void fetch() { // Actually fetch the page data here } @Override public String someDataField() throws ExtractionException { //The exception needs to be thrown if something failed // get piece of information and return it } ... // More datafields }","title":"Typical Implementation of a Single Data Extractor"},{"location":"01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists","text":"Information can be represented as a list. In NewPipe, a list is represented by an InfoItemsCollector . An InfoItemsCollector will collect and assemble a list of InfoItem . For each item that should be extracted, a new Extractor must be created, and given to the InfoItemsCollector via commit() . If you are implementing a list in your service you need to implement an InfoItemExtractor , that will be able to retrieve data for one and only one InfoItem. This extractor will then be comitted to the InfoItemsCollector that can collect the type of InfoItems you want to generate. A common implementation would look like this: private SomeInfoItemCollector collectInfoItemsFromElement(Element e) { // See *Some* as something like Stream or Channel // e.g. StreamInfoItemsCollector, and ChannelInfoItemsCollector are provided by NP SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); for(final Element li : element.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return collector; }","title":"Collector/Extractor Pattern for Lists"},{"location":"01_Concept_of_the_extractor/#listextractor","text":"There is more to know about lists: When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail, and its creator. Such info can be called list header . When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. Both of these Problems are fixed by the ListExtractor which takes care about extracting additional metadata about the list, and by chopping down lists into several pages, so called InfoItemsPage s. Each page has its own URL, and needs to be extracted separately. For extracting list header information a ListExtractor behaves like a regular extractor. For handling InfoItemsPages it adds methods such as: getInitialPage() which will return the first page of InfoItems. getNextPageUrl() If a second Page of InfoItems is available this will return the URL pointing to them. getPage() returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the getNextPageUrl() method of the previous page. The reason why the first page is handled special is because many Websites such as YouTube will load the first page of items like a regular web page, but all the others as an AJAX request. An InfoItemsPage itself has two constructors which take these parameters: - The InfoitemsCollector of the list that the page should represent - A nextPageUrl which represents the url of the following page (may be null if not page follows). - Optionally errors which is a list of Exceptions that may have happened during extracton. Here is a simplified reference implementation of a list extractor that only extracts pages, but not metadata: class MyListExtractor extends ListExtractor { ... private Document document; ... public InfoItemsPage<SomeInfoItem> getPage(pageUrl) throws ExtractionException { SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId()); document = myFunctionToGetThePageHTMLWhatever(pageUrl); //remember this part from the simple list extraction for(final Element li : document.children()) { collector.commit(new InfoItemExtractor() { @Override public String getName() throws ParsingException { ... } @Override public String getUrl() throws ParsingException { ... } ... } return new InfoItemsPage<SomeInfoItem>(collector, myFunctionToGetTheNextPageUrl(document)); } public InfoItemsPage<SomeInfoItem> getInitialPage() { //document here got initialized by the fetch() function. return getPage(getTheCurrentPageUrl(document)); } ... }","title":"ListExtractor"},{"location":"02_Concept_of_LinkHandler/","text":"Concept of the LinkHandler The LinkHandler represent links to resources like videos, search requests, channels, etc. The idea is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (the default URL for YouTube) https://youtu.be/oHg5SJYRHA0 (the shortened link) https://m.youtube.com/watch?v=oHg5SJYRHA0 (the link for mobile devices) Important notes about LinkHandler A simple LinkHandler will contain the default URL, the ID, and the original URL. LinkHandler s are read only. LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. Usage The typical usage for obtaining a LinkHandler would look like this: LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\"); Implementation In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHandlerFactory can handle this type of link. } } ListLinkHandler and SearchQueryHandler List based resources, like channels and playlists, can be sorted and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler , which inherits from LinkHandler and adds the field ContentFilter , which is used to filter by resource type, like stream or playlist, and SortFilter , which is used to sort by name, date, or view count. !!ATTENTION!! Be careful when you implement a content filter: No selected filter equals all filters selected. If your get an empty content filter list in your extractor, make sure you return everything. By all means, use \"if\" statements like contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports. SearchQueryHandler You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler . These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"Concept of the LinkHandler"},{"location":"02_Concept_of_LinkHandler/#concept-of-the-linkhandler","text":"The LinkHandler represent links to resources like videos, search requests, channels, etc. The idea is that a video can have multiple links pointing to it, but it has one unique ID that represents it, like this example: oHg5SJYRHA0 can be represented as: https://www.youtube.com/watch?v=oHg5SJYRHA0 (the default URL for YouTube) https://youtu.be/oHg5SJYRHA0 (the shortened link) https://m.youtube.com/watch?v=oHg5SJYRHA0 (the link for mobile devices)","title":"Concept of the LinkHandler"},{"location":"02_Concept_of_LinkHandler/#important-notes-about-linkhandler","text":"A simple LinkHandler will contain the default URL, the ID, and the original URL. LinkHandler s are read only. LinkHandler s are also used to determine which part of the extractor can handle a certain link. In order to get one you must either call fromUrl() or fromId() of the the corresponding LinkHandlerFactory . Every type of resource has its own LinkHandlerFactory . Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.","title":"Important notes about LinkHandler"},{"location":"02_Concept_of_LinkHandler/#usage","text":"The typical usage for obtaining a LinkHandler would look like this: LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandler myVideo = myLinkHandlerFactory.fromUrl(\"https://my.service.com/the_video\");","title":"Usage"},{"location":"02_Concept_of_LinkHandler/#implementation","text":"In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg: class MyStreamLinkHandlerFactory extends LinkHandlerFactory { @Override public String getId(String url) throws ParsingException { // Return the ID based on the URL. } @Override public String getUrl(String id) throws ParsingException { // Return the URL based on the ID given. } @Override public boolean onAcceptUrl(String url) throws ParsingException { // Return true if this LinkHandlerFactory can handle this type of link. } }","title":"Implementation"},{"location":"02_Concept_of_LinkHandler/#listlinkhandler-and-searchqueryhandler","text":"List based resources, like channels and playlists, can be sorted and filtered. Therefore these type of resources don't just use a LinkHandler, but a class called ListLinkHandler , which inherits from LinkHandler and adds the field ContentFilter , which is used to filter by resource type, like stream or playlist, and SortFilter , which is used to sort by name, date, or view count. !!ATTENTION!! Be careful when you implement a content filter: No selected filter equals all filters selected. If your get an empty content filter list in your extractor, make sure you return everything. By all means, use \"if\" statements like contentFilter.contains(\"video\") || contentFilter.isEmpty() . ListLinkHandler are also created by overriding the ListLinkHandlerFactory additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override getAvailableContentFilter() and getAvailableSortFilter() . Through these you can tell the front end which kind of filter your service supports.","title":"ListLinkHandler and SearchQueryHandler"},{"location":"02_Concept_of_LinkHandler/#searchqueryhandler","text":"You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the same search request might have a different outcome depending on the country or the time you send the request. This is why the idea of an \"ID\" is replaced by a \"SearchString\" in the SearchQueryHandler . These work like regular ListLinkHandler, except that you don't have to implement the methods onAcceptUrl() and getId() when overriding SearchQueryHandlerFactory .","title":"SearchQueryHandler"},{"location":"03_Implement_a_service/","text":"Implementing a Service Services, or better service connectors, are the parts of NewPipe which communicate with an actual service like YouTube. This page will describe how you can implement and add your own services to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before continuing. Required and Optional Parts Your service does not have to implement everything; some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels. However, if you start to implement one of the optional parts of the list below, you will have to implement all of its parts/classes. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The Parts of a Service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional) Allowed Libraries The NewPipe Extractor already includes a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks, Regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using Regex too often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries, please tell us before you do so. Head of Service First of all, if you want to create a new service, you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts Required to be Implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add and should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some metadata about your service such as the name, capabilities, the author's name, and their email address for further notice and maintenance issues. Remember, after extending this class, you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way, your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one. Stream Streams are considered single entities of video or audio. They have metadata like a title, a description, next/related videos, a thumbnail and comments. To obtain the URL to the actual stream data, as well as its metadata, StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such streams, like a search result or a playlist. Since every streaming service (obviously) provides streams, this is required to implement. Otherwise, your service was pretty useless :) Parts Required to be Implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory Search The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support suggestions as you type, you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to also display a suggestion while typing. Parts Required to be Implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional) Playlist Playlists are lists of streams provided by the service (you might not have to be concerned over locally saved playlists, those will be handled by the frontend). A playlist may only contain StreamInfoItems , but no other InfoItem types. Parts Required to be Implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory Channel A Channel is mostly a Playlist , the only difference is that it does not only represent a simple list of streams, but also a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the ChannelExtractor differs from the one of a playlist. Parts Required to be Implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory Kiosk A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators Selection\" etc. Kiosks are controversial; many people may not like them. If you also don't like them, please consider your users and refrain from denying support for them. Your service would look pretty empty if you select it and no video is being displayed. Also, you should not override the preference of the user, since users of NewPipe can decide by the settings whether they want to see the kiosk page or not. Multiple Kiosks Most services will implement more than one kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure, every page you want to support has to be implemented as its own KioskExtractor . However, if the pages are similar, you can use the same implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts Required to be Implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Implementing a Service"},{"location":"03_Implement_a_service/#implementing-a-service","text":"Services, or better service connectors, are the parts of NewPipe which communicate with an actual service like YouTube. This page will describe how you can implement and add your own services to the extractor. Please make sure you read and understand the Concept of Extractors and the Concept of LinkHandler before continuing.","title":"Implementing a Service"},{"location":"03_Implement_a_service/#required-and-optional-parts","text":"Your service does not have to implement everything; some parts are optional. This is because not all services support every feature other services support. For example, it might be that a certain service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding factory method of the your StreamingService implementation return null . The frontend will handle the lack of having channels. However, if you start to implement one of the optional parts of the list below, you will have to implement all of its parts/classes. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself. The Parts of a Service: Head of Service Stream Search Playlist (optional) Channel (optional) Kiosk (optional)","title":"Required and Optional Parts"},{"location":"03_Implement_a_service/#allowed-libraries","text":"The NewPipe Extractor already includes a lot of usable tools and external libraries that should make extracting easy. For some specific (tiny) tasks, Regex is allowed. Here you can take a look at the Parser , which will give you a little help with that. Use Regex with care!!! Avoid it as often as possible. It's better to ask us to introduce a new library than start using Regex too often. Html/XML Parsing: jsoup JSON Parsing: nanojson JavaScript Parsing/Execution: Mozilla Rhino Link detection in strings: AutoLink If you need to introduce new libraries, please tell us before you do so.","title":"Allowed Libraries"},{"location":"03_Implement_a_service/#head-of-service","text":"First of all, if you want to create a new service, you should create a new package below org.schabi.newpipe.services , with the name of your service as package name. Parts Required to be Implemented: StreamingService ServiceInfo StreamingService is a factory class that will return objects of all important parts of your service. Every extractor, handler, and info type you add and should be part of your implementation, must be instantiated using an instance of this class. You can see it as a factory for all objects of your implementation. ServiceInfo will return some metadata about your service such as the name, capabilities, the author's name, and their email address for further notice and maintenance issues. Remember, after extending this class, you need to return an instance of it by through your implementation of StreamingService.getServiceInfo() . When these two classes are extended by you, you need to add your StreamingService to the ServiceList of NewPipe. This way, your service will become an official part of the NewPipe Extractor. Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor. So when adding your service just give it the ID of the previously last service in the list incremented by one.","title":"Head of Service"},{"location":"03_Implement_a_service/#stream","text":"Streams are considered single entities of video or audio. They have metadata like a title, a description, next/related videos, a thumbnail and comments. To obtain the URL to the actual stream data, as well as its metadata, StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will extract one item in a list of items representing such streams, like a search result or a playlist. Since every streaming service (obviously) provides streams, this is required to implement. Otherwise, your service was pretty useless :) Parts Required to be Implemented: StreamExtractor StreamInfoItemExtractor LinkHandlerFactory","title":"Stream"},{"location":"03_Implement_a_service/#search","text":"The SearchExtractor is also required to be implemented. It will take a search query represented as SearchQueryHandler and return a list of search results. Since many services support suggestions as you type, you will also want to implement a SuggestionExtractor . This will make it possible for the frontend to also display a suggestion while typing. Parts Required to be Implemented: SearchExtractor SearchQueryHandlerFactory SuggestionExtractor (optional)","title":"Search"},{"location":"03_Implement_a_service/#playlist","text":"Playlists are lists of streams provided by the service (you might not have to be concerned over locally saved playlists, those will be handled by the frontend). A playlist may only contain StreamInfoItems , but no other InfoItem types. Parts Required to be Implemented: PlaylistExtractor PlayListInfoItemExtractor ListLinkHandlerFactory","title":"Playlist"},{"location":"03_Implement_a_service/#channel","text":"A Channel is mostly a Playlist , the only difference is that it does not only represent a simple list of streams, but also a user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the ChannelExtractor differs from the one of a playlist. Parts Required to be Implemented: ChannelExtractor ChannelInfoItemExtractor ListLinkHandlerFactory","title":"Channel"},{"location":"03_Implement_a_service/#kiosk","text":"A kiosk is a list of InfoItems which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content displayed on the main page of a video platform. A kiosk could be something like \"Top 20\", \"Charts\", \"News\", \"Creators Selection\" etc. Kiosks are controversial; many people may not like them. If you also don't like them, please consider your users and refrain from denying support for them. Your service would look pretty empty if you select it and no video is being displayed. Also, you should not override the preference of the user, since users of NewPipe can decide by the settings whether they want to see the kiosk page or not.","title":"Kiosk"},{"location":"03_Implement_a_service/#multiple-kiosks","text":"Most services will implement more than one kiosk, so a service might have a \"Top 20\" for different categories like \"Country Music\", \"Techno\", etc. This is why the extractor will let you implement multiple KioskExtractors . Since different kiosk pages might also differ with their HTML structure, every page you want to support has to be implemented as its own KioskExtractor . However, if the pages are similar, you can use the same implementation, but set the page type when you instantiate your KioskExtractor through the KioskList.KioskExtractorFactory . Every kiosk you implement needs to be added to your KioskList which you return with your StreamingService implementation. It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service. An example implementation of the getKioskList() could look like this: @Override public KioskList getKioskList() throws ExtractionException { KioskList list = new KioskList(getServiceId()); list.addKioskEntry(new KioskList.KioskExtractorFactory() { @Override public KioskExtractor createNewKiosk(StreamingService streamingService, String url, String id, Localization local) throws ExtractionException { return new YoutubeTrendingExtractor(YoutubeService.this, new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local); } }, new YoutubeTrendingLinkHandlerFactory(), \"Trending\"); list.setDefaultKiosk(\"Trending\"); return list; } Parts Required to be Implemented: KioskList.KioskExtractorFactory KioskExtractor ListLinkHandlerFactory","title":"Multiple Kiosks"},{"location":"04_Run_changes_in_App/","text":"Testing Your Changes in the App You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it with the actual frontend, you need to follow these steps. Setup Android Studio First, you'll want to set up a working Android Studio environment. To do this, download Studio from developer.android.com , and follow the instructions on how to set it up. Get the NewPipe Code and Run it. In order to get it, you simply clone or download it from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Also, make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under Linux, where you sometimes have to adjust the udev rules in order to make your device accessible . Run Your Changes on the Extractor There are several ways to test your extractor version in NewPipe. We will show you the most convenient ones: Using local folder In NewPipe app root folder, edit settings.gradle file and add this: includeBuild('../NewPipeExtractor') { dependencySubstitution { substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor') } } includeBuild should have the relative path as argument. ../NewPipeExtractor means one folder up in hierarchy, and the folder is exactly named NewPipeExtractor . If that's not the case, edit this part. Using JitPack Another way is to use JitPack . This is a build service that can build maven *.jar packages for Android and Java based on GitHub or GitLab repositories. To use the extractor through JitPack, you need to push it to your online repository of your copy that you host either on GitHub or GitLab . It's important to host it on one of both. To copy your repository URL in HTTP format, go to JitPack and paste it there. From here, you can grab the latest commit via GET IT button. I recommend not to use a SNAPSHOT, since I am not sure when snapshot is built. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:<commit>' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well, then you should only see a screen with OK signs. Now you can compile and run NewPipe with the new extractor. Troubleshooting If something went wrong on JitPack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it's red, it means that the build failed.","title":"Testing Your Changes in the App"},{"location":"04_Run_changes_in_App/#testing-your-changes-in-the-app","text":"You should develop and test your changes with the JUnit environment that is provided by the NewPipe Extractor and IDEA. If you want to try it with the actual frontend, you need to follow these steps.","title":"Testing Your Changes in the App"},{"location":"04_Run_changes_in_App/#setup-android-studio","text":"First, you'll want to set up a working Android Studio environment. To do this, download Studio from developer.android.com , and follow the instructions on how to set it up.","title":"Setup Android Studio"},{"location":"04_Run_changes_in_App/#get-the-newpipe-code-and-run-it","text":"In order to get it, you simply clone or download it from the current dev branch github.com/TeamNewPipe/NewPipe.git . You can then build and run it following these instructions . Also, make sure you are comfortable with adb since you might experience some trouble running your compiled app on a real device, especially under Linux, where you sometimes have to adjust the udev rules in order to make your device accessible .","title":"Get the NewPipe Code and Run it."},{"location":"04_Run_changes_in_App/#run-your-changes-on-the-extractor","text":"There are several ways to test your extractor version in NewPipe. We will show you the most convenient ones:","title":"Run Your Changes on the Extractor"},{"location":"04_Run_changes_in_App/#using-local-folder","text":"In NewPipe app root folder, edit settings.gradle file and add this: includeBuild('../NewPipeExtractor') { dependencySubstitution { substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor') } } includeBuild should have the relative path as argument. ../NewPipeExtractor means one folder up in hierarchy, and the folder is exactly named NewPipeExtractor . If that's not the case, edit this part.","title":"Using local folder"},{"location":"04_Run_changes_in_App/#using-jitpack","text":"Another way is to use JitPack . This is a build service that can build maven *.jar packages for Android and Java based on GitHub or GitLab repositories. To use the extractor through JitPack, you need to push it to your online repository of your copy that you host either on GitHub or GitLab . It's important to host it on one of both. To copy your repository URL in HTTP format, go to JitPack and paste it there. From here, you can grab the latest commit via GET IT button. I recommend not to use a SNAPSHOT, since I am not sure when snapshot is built. An \"implementation\" string will be generated for you. Copy this string and replace the implementation 'com.github.TeamNewPipe:NewPipeExtractor:<commit>' line in the file /app/build.gradle with it. Your browser does not support the video tag. If everything synced well, then you should only see a screen with OK signs. Now you can compile and run NewPipe with the new extractor.","title":"Using JitPack"},{"location":"04_Run_changes_in_App/#troubleshooting","text":"If something went wrong on JitPack site, you can check their build log, by selecting the commit you tried to build and click on that little paper symbol next to the GET IT button. If it's red, it means that the build failed.","title":"Troubleshooting"},{"location":"05_Mock_tests/","text":"Mock Tests A web crawler is, by its very nature, dependent on the external service which it is crawling. In order to have a reliable CI pipeline, this external dependency needs to be removed. For this, there is a system in place to automatically save the requests made to a service, as well as the responses received. These can then be used in the CI pipeline to reliably test changes made to the Extractor, and avoid test failures due to API changes on the side of the service. Multiple downloader implementations There are multiple implementations of the abstract class Downloader DownloaderTestImpl is used for running the test against the actual service. RecordingDownloader is used to save the request and response to a file, thus creating the mock. MockDownloader is used to answer requests using the saved mocks. Usage There are 2 ways to specify which downloader should be used. First one is passing the -Ddownloader=<value> argument from the command line, where value can be one of DownloaderType . The main use case is in the CI pipeline, like this: ./gradlew check --stacktrace -Ddownloader=MOCK . Other than that it can also be used to mass generate mocks by specifying which package should be tested. For example, if one wanted to update all YouTube mocks: gradle clean test --tests 'org.schabi.newpipe.extractor.services.youtube.*' -Ddownloader=RECORDING The second way is changing the field DownloaderFactory.DEFAULT_DOWNLOADER . The default value is DownloaderType.REAL which should not be changed on the master branch. Locally one can change this to DownloaderType.RECORDING , run the tests and commit the generated mocks. This is the main use case for when developing locally. Mock only tests There are some things which cannot ever be tested reliably against an actual service. For example, tests for an upcoming livestream would fail after the livestream starts. For this, there is a marker interface MockOnly , and a custom TestRule MockOnlyRule . It skips the tests in the CI pipeline if they are not run with mocks. See MockOnlyRule for further details. Example usage: public static class TestClass { @Rule public MockOnlyRule rule = new MockOnlyRule(); @MockOnly @Test public void myTest() throws Exception { //assertions } }","title":"Mock Tests"},{"location":"05_Mock_tests/#mock-tests","text":"A web crawler is, by its very nature, dependent on the external service which it is crawling. In order to have a reliable CI pipeline, this external dependency needs to be removed. For this, there is a system in place to automatically save the requests made to a service, as well as the responses received. These can then be used in the CI pipeline to reliably test changes made to the Extractor, and avoid test failures due to API changes on the side of the service.","title":"Mock Tests"},{"location":"05_Mock_tests/#multiple-downloader-implementations","text":"There are multiple implementations of the abstract class Downloader DownloaderTestImpl is used for running the test against the actual service. RecordingDownloader is used to save the request and response to a file, thus creating the mock. MockDownloader is used to answer requests using the saved mocks.","title":"Multiple downloader implementations"},{"location":"05_Mock_tests/#usage","text":"There are 2 ways to specify which downloader should be used. First one is passing the -Ddownloader=<value> argument from the command line, where value can be one of DownloaderType . The main use case is in the CI pipeline, like this: ./gradlew check --stacktrace -Ddownloader=MOCK . Other than that it can also be used to mass generate mocks by specifying which package should be tested. For example, if one wanted to update all YouTube mocks: gradle clean test --tests 'org.schabi.newpipe.extractor.services.youtube.*' -Ddownloader=RECORDING The second way is changing the field DownloaderFactory.DEFAULT_DOWNLOADER . The default value is DownloaderType.REAL which should not be changed on the master branch. Locally one can change this to DownloaderType.RECORDING , run the tests and commit the generated mocks. This is the main use case for when developing locally.","title":"Usage"},{"location":"05_Mock_tests/#mock-only-tests","text":"There are some things which cannot ever be tested reliably against an actual service. For example, tests for an upcoming livestream would fail after the livestream starts. For this, there is a marker interface MockOnly , and a custom TestRule MockOnlyRule . It skips the tests in the CI pipeline if they are not run with mocks. See MockOnlyRule for further details. Example usage: public static class TestClass { @Rule public MockOnlyRule rule = new MockOnlyRule(); @MockOnly @Test public void myTest() throws Exception { //assertions } }","title":"Mock only tests"},{"location":"06_releasing/","text":"Releasing a New NewPipe Version This site is meant for those who want to maintain NewPipe, or just want to know how releasing works. Differences Between Regular and Hotfix Releases Depending on the service, NewPipe Extractor uses web crawling or internal APIs. Both are subject to arbitrary changes by the service providers, like YouTube, SoundCloud or PeerTube. When they change something, NewPipe Extractor and thus NewPipe break instantly. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as much as possible. The entire release cycle is therefore designed around this issue. There is a difference between a release that introduces new features and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown). Let's have a look at the characteristics of a regular release , and then the characteristics of a hotfix release . Regular Releases Regular releases are normal releases as they are done in any other app. Releases are always stored and tagged on master branch. The latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version. Feature Branching The dev branch is used for development. Pushing to dev directly, however, is not allowed, since QA and testing should be done before adding something to the branch. This ensures that the development version works as stable a possible. In order to change something on the app, one may want to fork the dev branch and develop the changes in their own branch (this is called feature branching). Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other. If the extractor's API is modified, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes on the fronted at any time. Merging Features/Bugfixes After finishing a feature, one should open up a Pull Request to the dev branch. From here, a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer, please take care about the code architecture so corrosion or code shifting can be prevented. Please also prioritize code quality over functionality. In short: cool function but bad code = no merge. Focus on keeping the code as clean as possible. An APK for testing is provided by GitHub Actions for every PR. Please ensure that this APK is tested thoroughly to prevent introducing regressions. Testing features needs to take into account that NewPipe is used on a brought variety of Android versions from KitKat to the latest, on custom ROMs like Lineage OS, CalyxOS or /e/ and different devices like phones, tablets and TVs. Sometimes, the content of a PR changes over the time. Modify the PR's title if it does not represent the introduced changes anymore. After a maintainer merged the new feature into the dev branch, they should add the PR's title or a summary of the changes into the release notes . Normal Releases Once there are enough changes, and the maintainers believe that NewPipe is ready for a new version, they should prepare a new release. Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: Don't do a release if you don't have time for it!!! By following the steps listed in Release instructions , you can publish a stable version of NewPipe. Hotfix Releases As aforementioned, NewPipe heavily relies on external components and might break at a random point of time. In order to keep the NewPipe's downtime as low as possible, when such a shutdown happens, we allow hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or unrelated bugfixes. A hotfix may only focus on fixing what caused the shutdown. Hotfix Branch Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch. Of course, you are not allowed to push to master directly, so you need to create a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well. Releasing If you fixed the issue and found it to be tested and reviewed well enough, you may publish a new version. You don't need to undergo the full release procedure of a regular release, which takes too much time. Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix. It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better than a non-functional version \u00af\\_(\u30c4)_/\u00af. Here's what you do when releasing a hotfix: Merge the corresponding pull request in the extractor. Publish the new extractor version . Update the extractor version in the app's build.gradle file. Create a new release draft and put some info on the fix into the release note . Copy the release notes into the fastlane directory to create a changelog file . Increment the small minor version number and the versionCode . Generate a release APK ( gradlew assembleRelease ) and sign it (or get it signed by one of the other maintainers). Add the signed APK to the GitHub release notes. Click \"Publish Release\" . Publish the new version on F-Droid . Merge the changes from master into dev . Update the changelog for the website . Extractor releases In general, the release process for extractor versions is not that complicated compared to app releases. The extractor has (in difference to the app) a decent test coverage. Additionally, the latest extractor version is typically tested in the app's latest dev version. Therefore, a long test phase is not needed when creating extractor releases. To create a new extractor version , update the version in the extractor's build.gradle file as well as the version names in the README. Merge the dev branch into master . The same that applies the app's release notes also applies to the extractor's release notes. When publishing an extractor release via GitHub on the master branch, a new JavaDoc version is generated and published automatically. Pleas keep an eye on the GitHub Action which is responsible for that. If changes in that release introduced invalid JavaDoc, the build fails and needs to be fixed. For this reason, you should check locally if there are any problems with the JavaDoc generation before publishing the new version. Version Nomenclature The version nomenclature of NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. \u00af\\_(\u30c4)_/\u00af Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) is incremented when bug fixes or minor features are added to the app. Version Nomenclature of the Extractor Previously, the extractor was released together with the app, therefore the version number of the extractor was identical to the one of NewPipe itself. We try to combine efforts to make NewPipe Extractor more independent of the app. The extractor is used by multiple other applications and therefore releasing extractor updates should not be coupled to app releases. However, maintainers need to keep an eye on making the app compatible with extractor changes. Release Notes Release notes should tell what was changed in the new version of the app. The release notes for NewPipe are stored in the GitHub draft for a new release . When a maintainer wants to add changes to the release notes, but there is no draft for a new version, they should create one. Changes can be categorized into five basic types: New : New features that got added to the app. Improved : Improvements to the app or existing features Fixed : Bugfixes Translation : New translations Development : Changes which address things \"under the hood\", which do not have any recognizable effect to the user; e.g. dependency updates or changes to the build process When adding a PR to the release notes, increase the PR counter at the top of the draft and put the number before the PR summary / title. This helps the blog post authors to keep easily track of new PRs. Remove the numbers before publishing a new version :) If there is a blog post covering the changes in more detail, make sure to link it on the top of the release notes. It would be a pity, if only a few people read the blog post after our wonderful writers put so much effort into creating it. Changelog file Maintainers need to provide a changelog file for each release. A changelog file is used by F-Droid to give a quick summary of the most important changes for a release. This file is placed in the /fastlane/metadata/android/en-US/changelogs directory and named <versionCode>.txt (whereas <versionCode> is the version code of the incoming release). Changelog files must not exceed 500 bytes. Be aware that the changelog is translated into multiple languages. A changelog written in English which almost hits 500 bytes can hardly be translated completely within this limit. This causes troubles for translators, because Weblate enforces the 500 bytes limit, too. For this reason it is recommended to keep the changelog at 400 bytes. When creating the changelog file be aware of changes which were done in the extractor as well. Before pushing the changelog to NewPipe's repo, ask other maintainers to review it. After pushing the changelog to NewPipe's GitHub repo, updating Weblate is necessary. This enables translators to work on localized versions of the changelog before a release is tagged and published. Publish on F-Droid NewPipe is and supports open source software. For this reason, the preferred way to distribute the app is F-Droid . F-Droid is a catalogue of FOSS apps and also comes with an Android client which handles app updates. There are two ways to install NewPipe via F-Droid. Through the main F-Droid repository NewPipe's metadata file can be found in F-Droid's data repository on GitLab: https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/org.schabi.newpipe.yml This file is automatically updated by a bot when a new release is published on GitHub. It can take a few days until all new apps on F-Droid are built, signed and published. F-Droid also supports reproducible builds . Reproducible builds or deterministic builds allow someone else to retrieve the exact same binary as we get when building the app (except the signing of course). When the reproducible build feature is enabled for an app in F-Droid, they compare their binary to one provided by the author. If both are identical, F-Droid does not only publish the binary signed by themselves, but also the one signed by the author. Currently, NewPipe's builds are not deterministic, and we therefore cannot use that feature. Once the builds are deterministic again, the following steps need to be done to publish a new version on F-Droid: Install fdroidserver on your device . Clone the F-Droid Data repo from https://gitlab.com/fdroid/fdroiddata Add the new version to metadata/org.schabi.newpipe.yml Run fdroid signatures /path/to/newpipe.apk on the signed APK from within the repo. Create a MR on GitLab. An example commit containing all required changes can be found here . Through NewPipe's F-Droid repository F-Droid needs NewPipe's own F-Droid repo is available at https://archive.newpipe.net/fdroid/repo It is updated and maintained by @TheAssassin .","title":"Releasing a New NewPipe Version"},{"location":"06_releasing/#releasing-a-new-newpipe-version","text":"This site is meant for those who want to maintain NewPipe, or just want to know how releasing works.","title":"Releasing a New NewPipe Version"},{"location":"06_releasing/#differences-between-regular-and-hotfix-releases","text":"Depending on the service, NewPipe Extractor uses web crawling or internal APIs. Both are subject to arbitrary changes by the service providers, like YouTube, SoundCloud or PeerTube. When they change something, NewPipe Extractor and thus NewPipe break instantly. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as much as possible. The entire release cycle is therefore designed around this issue. There is a difference between a release that introduces new features and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown). Let's have a look at the characteristics of a regular release , and then the characteristics of a hotfix release .","title":"Differences Between Regular and Hotfix Releases"},{"location":"06_releasing/#regular-releases","text":"Regular releases are normal releases as they are done in any other app. Releases are always stored and tagged on master branch. The latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version.","title":"Regular Releases"},{"location":"06_releasing/#feature-branching","text":"The dev branch is used for development. Pushing to dev directly, however, is not allowed, since QA and testing should be done before adding something to the branch. This ensures that the development version works as stable a possible. In order to change something on the app, one may want to fork the dev branch and develop the changes in their own branch (this is called feature branching). Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other. If the extractor's API is modified, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes on the fronted at any time.","title":"Feature Branching"},{"location":"06_releasing/#merging-featuresbugfixes","text":"After finishing a feature, one should open up a Pull Request to the dev branch. From here, a maintainer can do Code review and Quality Assurance (QA) . If you are a maintainer, please take care about the code architecture so corrosion or code shifting can be prevented. Please also prioritize code quality over functionality. In short: cool function but bad code = no merge. Focus on keeping the code as clean as possible. An APK for testing is provided by GitHub Actions for every PR. Please ensure that this APK is tested thoroughly to prevent introducing regressions. Testing features needs to take into account that NewPipe is used on a brought variety of Android versions from KitKat to the latest, on custom ROMs like Lineage OS, CalyxOS or /e/ and different devices like phones, tablets and TVs. Sometimes, the content of a PR changes over the time. Modify the PR's title if it does not represent the introduced changes anymore. After a maintainer merged the new feature into the dev branch, they should add the PR's title or a summary of the changes into the release notes .","title":"Merging Features/Bugfixes"},{"location":"06_releasing/#normal-releases","text":"Once there are enough changes, and the maintainers believe that NewPipe is ready for a new version, they should prepare a new release. Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: Don't do a release if you don't have time for it!!! By following the steps listed in Release instructions , you can publish a stable version of NewPipe.","title":"Normal Releases"},{"location":"06_releasing/#hotfix-releases","text":"As aforementioned, NewPipe heavily relies on external components and might break at a random point of time. In order to keep the NewPipe's downtime as low as possible, when such a shutdown happens, we allow hotfixes . A hotfix allows work on the master branch instead of the dev branch. A hotfix MUST NOT contain any features or unrelated bugfixes. A hotfix may only focus on fixing what caused the shutdown.","title":"Hotfix Releases"},{"location":"06_releasing/#hotfix-branch","text":"Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch. Of course, you are not allowed to push to master directly, so you need to create a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well.","title":"Hotfix Branch"},{"location":"06_releasing/#releasing","text":"If you fixed the issue and found it to be tested and reviewed well enough, you may publish a new version. You don't need to undergo the full release procedure of a regular release, which takes too much time. Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix. It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better than a non-functional version \u00af\\_(\u30c4)_/\u00af. Here's what you do when releasing a hotfix: Merge the corresponding pull request in the extractor. Publish the new extractor version . Update the extractor version in the app's build.gradle file. Create a new release draft and put some info on the fix into the release note . Copy the release notes into the fastlane directory to create a changelog file . Increment the small minor version number and the versionCode . Generate a release APK ( gradlew assembleRelease ) and sign it (or get it signed by one of the other maintainers). Add the signed APK to the GitHub release notes. Click \"Publish Release\" . Publish the new version on F-Droid . Merge the changes from master into dev . Update the changelog for the website .","title":"Releasing"},{"location":"06_releasing/#extractor-releases","text":"In general, the release process for extractor versions is not that complicated compared to app releases. The extractor has (in difference to the app) a decent test coverage. Additionally, the latest extractor version is typically tested in the app's latest dev version. Therefore, a long test phase is not needed when creating extractor releases. To create a new extractor version , update the version in the extractor's build.gradle file as well as the version names in the README. Merge the dev branch into master . The same that applies the app's release notes also applies to the extractor's release notes. When publishing an extractor release via GitHub on the master branch, a new JavaDoc version is generated and published automatically. Pleas keep an eye on the GitHub Action which is responsible for that. If changes in that release introduced invalid JavaDoc, the build fails and needs to be fixed. For this reason, you should check locally if there are any problems with the JavaDoc generation before publishing the new version.","title":"Extractor releases"},{"location":"06_releasing/#version-nomenclature","text":"The version nomenclature of NewPipe is simple. Major : The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. \u00af\\_(\u30c4)_/\u00af Minor : The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app. Small Minor : The small minor (the number after the second dot) is incremented when bug fixes or minor features are added to the app.","title":"Version Nomenclature"},{"location":"06_releasing/#version-nomenclature-of-the-extractor","text":"Previously, the extractor was released together with the app, therefore the version number of the extractor was identical to the one of NewPipe itself. We try to combine efforts to make NewPipe Extractor more independent of the app. The extractor is used by multiple other applications and therefore releasing extractor updates should not be coupled to app releases. However, maintainers need to keep an eye on making the app compatible with extractor changes.","title":"Version Nomenclature of the Extractor"},{"location":"06_releasing/#release-notes","text":"Release notes should tell what was changed in the new version of the app. The release notes for NewPipe are stored in the GitHub draft for a new release . When a maintainer wants to add changes to the release notes, but there is no draft for a new version, they should create one. Changes can be categorized into five basic types: New : New features that got added to the app. Improved : Improvements to the app or existing features Fixed : Bugfixes Translation : New translations Development : Changes which address things \"under the hood\", which do not have any recognizable effect to the user; e.g. dependency updates or changes to the build process When adding a PR to the release notes, increase the PR counter at the top of the draft and put the number before the PR summary / title. This helps the blog post authors to keep easily track of new PRs. Remove the numbers before publishing a new version :) If there is a blog post covering the changes in more detail, make sure to link it on the top of the release notes. It would be a pity, if only a few people read the blog post after our wonderful writers put so much effort into creating it.","title":"Release Notes"},{"location":"06_releasing/#changelog-file","text":"Maintainers need to provide a changelog file for each release. A changelog file is used by F-Droid to give a quick summary of the most important changes for a release. This file is placed in the /fastlane/metadata/android/en-US/changelogs directory and named <versionCode>.txt (whereas <versionCode> is the version code of the incoming release). Changelog files must not exceed 500 bytes. Be aware that the changelog is translated into multiple languages. A changelog written in English which almost hits 500 bytes can hardly be translated completely within this limit. This causes troubles for translators, because Weblate enforces the 500 bytes limit, too. For this reason it is recommended to keep the changelog at 400 bytes. When creating the changelog file be aware of changes which were done in the extractor as well. Before pushing the changelog to NewPipe's repo, ask other maintainers to review it. After pushing the changelog to NewPipe's GitHub repo, updating Weblate is necessary. This enables translators to work on localized versions of the changelog before a release is tagged and published.","title":"Changelog file"},{"location":"06_releasing/#publish-on-f-droid","text":"NewPipe is and supports open source software. For this reason, the preferred way to distribute the app is F-Droid . F-Droid is a catalogue of FOSS apps and also comes with an Android client which handles app updates. There are two ways to install NewPipe via F-Droid. Through the main F-Droid repository NewPipe's metadata file can be found in F-Droid's data repository on GitLab: https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/org.schabi.newpipe.yml This file is automatically updated by a bot when a new release is published on GitHub. It can take a few days until all new apps on F-Droid are built, signed and published. F-Droid also supports reproducible builds . Reproducible builds or deterministic builds allow someone else to retrieve the exact same binary as we get when building the app (except the signing of course). When the reproducible build feature is enabled for an app in F-Droid, they compare their binary to one provided by the author. If both are identical, F-Droid does not only publish the binary signed by themselves, but also the one signed by the author. Currently, NewPipe's builds are not deterministic, and we therefore cannot use that feature. Once the builds are deterministic again, the following steps need to be done to publish a new version on F-Droid: Install fdroidserver on your device . Clone the F-Droid Data repo from https://gitlab.com/fdroid/fdroiddata Add the new version to metadata/org.schabi.newpipe.yml Run fdroid signatures /path/to/newpipe.apk on the signed APK from within the repo. Create a MR on GitLab. An example commit containing all required changes can be found here . Through NewPipe's F-Droid repository F-Droid needs NewPipe's own F-Droid repo is available at https://archive.newpipe.net/fdroid/repo It is updated and maintained by @TheAssassin .","title":"Publish on F-Droid"},{"location":"07_release_instructions/","text":"Release instructions for normal releases This page contains detailed instructions for normal releases. Refer to Releasing a New NewPipe Version for other information about releases. Preliminary steps Permissions Have admin rights on Weblate You should be able to access Weblate's Maintenance page Tip: if the correct page does not show up when clicking that URL, make sure you are logged in ;-) Have at least maintainer rights on the NewPipe and NewPipeExtractor repos Repositories Have a cloned NewPipe local repository (for the rest of the page, origin is assumed to be the remote at github.com/TeamNewPipe/NewPipe ) Add the weblate remote to the same local repository (the URL used below can be found on the Maintenance page on Weblate) git remote add weblate https://hosted.weblate.org/git/newpipe/strings/ Make sure there are no pending changes git clean -fdx to discard them all ( CAUTION ) Switch to the dev branch and make sure it is up-to-date with the remote: git checkout dev git pull origin dev Version name and conventions Find the version code of the next release by looking for versionCode in app/build.gradle : You will add 1 to that value (from now on called NEW_VERSION_CODE ) to get the new value (but do not edit the file yet) Choose the version number of the next release according to semantic versioning (from now on called X.X.X ) Identification Have gpg installed and usable on your PC Have a GPG key, which can be used to verify that a file is really from you Pull changes from Weblate Go to Weblate's Maintenance tab Press the Lock button to prevent translators from translating while you are creating commits; remember to Unlock later! Press the Update button to update Weblate with the latest changes on NewPipe's dev branch Press the Commit button, if needed, to make sure Weblate creates a commit for translations which have not been committed yet Now go back to the local git repository In case you followed these steps before, delete the weblate-dev branch git branch -D weblate-dev Fetch new changes from the weblate remote git fetch weblate Create a new branch starting from weblate/dev , named weblate-dev , and switch to it git checkout -b weblate-dev weblate/dev If you run git log --oneline --graph you should see a Weblate commit on top, and then all of the commits currently on the dev branch: * cmt12hash (HEAD -> weblate-dev, weblate/dev) Translated using Weblate (...) * cmt89hash (origin/dev, dev) Commit message ... Switch back to the dev branch git checkout dev Merge weblate-dev into dev : git merge weblate-dev Create a changelog Finalize the draft changelog kept on GitHub , in case there are still some things to fill in Remove the temporary instructions, and the numbers before - which keep track of the order in which the PRs were merged, as that info is useful only for the blog post writers Before removing that information, you may want to send the original changelog to the blogpost writers Create a new English changelog in the fastlane/metadata/android/en-US/changelogs/ folder The file should be named NEW_VERSION_CODE.txt , using the new version code found in the Preliminary steps The file should have this structure (sections with no points can be removed): New \u2022 ... Improved \u2022 ... Fixed \u2022 ... Make sure you use the \u2022 for points (it looks nicer than - ) Capitalize the first letter in each point Use English verbs as if you were asking someone to do something, so for example use \"Fix abc\" and not \"Fixed abc\"; this allows saving a few characters and using a consistent style Prepend [SERVICE] to service-only changes (e.g. \"\u2022 [YouTube] Add mixes\") Summarize only the most important changes from the draft release kept on GitHub (it contains all merged pull requests) Make sure the file size is at most 500 bytes , in order to fit F-Droid's changelog size limit (!) Tip: removing the newline at the end of the file saves 1 byte ;-) Commit the file on the dev branch (try to stick to the provided commit message template) git add fastlane/metadata/android/en-US/changelogs/NEW_VERSION_CODE.txt git commit -m \"Add changelog for vX.X.X (NEW_VERSION_CODE)\" Push the changelog to Weblate Now there should be two new commits (the Weblate and changelog ones) on your local dev branch, which are not on NewPipe's remote dev branch. If you are an admin of the NewPipe repo, just push the changes to the remote dev git push origin dev If you are not an admin, create a pull request normally and ask someone with maintainer access to merge it Go to Weblate's Maintenance tab Press the Update button to update Weblate with the commit you just pushed on NewPipe's dev branch Press the Unlock button to allow translators to translate the changelog and possibly other components ( do not forget this step! ) Note that we had to do this process on NewPipe's dev branch because: Weblate's components are connected to NewPipe's dev branch, and will update changes from there Weblate's git repo is not writable, so there is no way to push commits there manually Creating the release branch Create a new branch starting from dev , named release-X.X.X , and switch to it git checkout dev git checkout -b release-X.X.X Edit the app/build.gradle file to update the extractor Set the NewPipeExtractor dependency version to a suitable one (usually the last commit in the NewPipeExtractor repo ) Commit the extractor update (if you used a specific version, append to VERSION to the commit message) git add app/build.gradle git commit -m \"Update NewPipeExtractor\" Edit the app/build.gradle file to bump the release Set versionCode to NEW_VERSION_CODE , i.e. increment the value by 1 as described in the Preliminary steps Set versionName to \"X.X.X\" Commit the version bump (try to stick to the provided commit message template) git add app/build.gradle git commit -m \"Release vX.X.X (NEW_VERSION_CODE)\" Push the newly created branch to the NewPipe repo git push origin release-X.X.X Creating the Pull Request Create a Pull Request (PR) from the new branch you just pushed If you used the correct branch name you should be able to use this URL, after changing the X.X.X: https://github.com/TeamNewPipe/NewPipe/pull/new/release-X.X.X Make sure the PR has master as the base branch and release-X.X.X as the compare branch The PR title should be \"Release vX.X.X (NEW_VERSION_CODE)\" Remove the entire PR template, and instead put these two lines in the description (the ISSUE_NUMBER will be replaced later): Do **not** report regressions here, but rather in the corresponding issue: #ISSUE_NUMBER The changelog is also there. Once you have created the PR, note down its number (from now on called PR_NUMBER ) In case some issue would be fixed when the release PR is merged, link them using the \"Development\" tab on the right, or add a \"Fixes #....\" in the PR description Check out #8231 for reference Creating the issue Create an issue Click here to open one without a template The issue title should be \"Release vX.X.X (please TEST!)\" The issue should have some sections, in the same order as provided below, with ## before titles The ## Testing for regressions section should contain the following lines; more information about how to obtain the APK are given at Testing APKs Debug APK (built by our CI in #PR_NUMBER): ... Please report **only regressions** (i.e. new issues) here, not issues that were already present in previous releases! An optional ## TODO section should contain a list of things that still need to be done before releasing, for example regressions that need to be fixed, or a reminder to merge the Weblate changelogs before releasing (use - [ ] to create checkbox lists) The ## NewPipeExtractor version section should contain a link to the NewPipeExtractor release this new NewPipe version will ship with (i.e. the one set in Creating the release branch ); choose one of these lines as a template This version of NewPipe will ship with [NewPipeExtractor version NPE_VERSION](https://github.com/TeamNewPipe/NewPipeExtractor/releases/tag/NPE_VERSION) This version of NewPipe will ship with [NewPipeExtractor commit FIRST_7_DIGITS_OF_NPE_COMMIT](https://github.com/TeamNewPipe/NewPipeExtractor/commit/NPE_COMMIT) Create the App changelog section using the template below. Copy the draft Markdown changelog kept on GitHub (you finalized it earlier in Create a changelog ) to the clipboard and paste it where specified below (make sure to leave a newline above, otherwise Markdown breaks): <details><summary><h2>App changelog </h2></summary><p> INSERT_COPIED_CHANGELOG_HERE </details> Once you have created the issue, pin it using the \"Pin issue\" button on the right Update the ISSUE_NUMBER in the pull request description Check out #8230 for reference Testing APKs The first time you open the release issue, and then each time some changes are made to the release PR, you should provide a debug APK in the ## Testing for regressions section. Wait for the Continuous Integration (CI) to finish testing the PR, then download the resulting debug APK artifact from the \"Checks\" tab Rename it to NewPipe_vX.X.X_RC1_debug.apk where RC1 should be incremented to RC2 and so on each time a new APK is provided Zip it and make sure the .zip file has the same name as the .apk it contains Upload it in the issue description, replacing the ... placeholder used above Sometimes it might be needed to also provide a release APK. In this case follow the same steps as above, with these differences: Make sure you are on the release-X.X.X branch Build the release APK yourself in Android Studio and sign it with your keys Temporarily edit the app/build.gradle file and add System.properties.put(\"packageSuffix\", \"vX_X_X\") at the top of the android -> buildTypes -> release block, which ensures that the application has a different package name than the official one Build and sign an APK via \"Build -> Generate Signed Bundle / APK...\" Make sure it installs correctly on your device Use this naming scheme: NewPipe_vX.X.X_RC1_release.apk Add a line to the ## Testing for regressions section, of this form: Release APK (built and signed by @YOUR_GITHUB_USERNAME): ... Taking care of regressions (quickfixes) The release issue and pull request should stay open for roughly one week , so that people can test the provided APKs and give feedback. If a regression is reported by some user, it should possibly be solved before releasing, otherwise the app would become more broken after each release. A regression is a bug now present in some code that used to run well in the last release, but was then modified in this release (supposedly to fix something else) and is now broken. So the following do not classify as regressions: some videos stop working because YouTube made some changes; the newly introduced big feature XYZ is still not perfect and has some bugs; a random crash reproducible also on previous versions... You get the point. Before releasing, try to fix any regressions that are reported, but avoid fixing non-regressions, since those should be treated with the same care and attention as all other issues. As a Release Manager, you might be required to fix regressions, so plan your release at a time when you are available. Pull requests fixing regressions should target the release-X.X.X branch, not the dev branch! When merging those PRs, also provide a new Release Candidate APK. Finally merging the pull request Once enough time has passed and all regressions and TODOs have been solved, you can proceed with the actual release. The following points include merging weblate changes again. In the local repository, check out the release branch and make sure it is up-to-date with the remote git checkout release-X.X.X git pull origin release-X.X.X Go to Weblate's Maintenance tab Press Lock ; remember to Unlock later! Press Update Press Commit , if needed Now go back to the local git repository Delete the weblate-dev branch, just in case git branch -D weblate-dev Fetch changes from Weblate (in particular you should see the weblate/dev remote branch being updated) git fetch weblate Obtain the hash of the last commit on the weblate/dev remote branch git log -n 1 --pretty=\"format:%H\" weblate/dev Cherry pick the hash you obtained above into the release branch (the one you are currently on) git cherry-pick HASH Push the changes to the remote branch git push origin release-X.X.X Merge the PR you created before Delete the GitHub remote branch associated with the PR, i.e. release-X.X.X (there should be a button in the PR) Close the issue you created before Merge dev back into master (since the PR merged changes onto master ) git checkout master git pull origin master git checkout dev git pull origin dev git merge master git push origin dev or create another temporary PR and merge it immediately Go to Weblate's Maintenance tab Press Unlock Creating the APK Now on the remote master branch there is the release code which you need to turn into an APK. In the local repository, check out the master branch and make sure it is up-to-date with the remote git checkout master git pull origin master Open the local project in Android Studio Run the Gradle clean task using Android Studio's interface, in order to clean up temporary/cache files that may interfere with reproducible builds Double press Ctrl, type gradle clean , press Enter Make sure leftover files from building RC releases are actually removed, in order to avoid confusion rm -rf ./app/release Run the Gradle assembleRelease task using Android Studio's interface: it will start the process of building an unsigned APK Double press Ctrl, type gradle assembleRelease , press Enter After a while you should find the APK under ./app/build/outputs/apk/release/app-release-unsigned.apk Having the APK signed by @TheAssassin Currently @TheAssassin is the only holder of NewPipe's APK signing keys. Therefore you should send the unsigned APK to him, after which he will sign it and send it back to you. He will also then publish the signed APK in NewPipe's F-Droid repo. Rename app-release-unsigned.apk to NewPipe_vX.X.X.apk Generate a signature for the APK file gpg -b NewPipe_vX.X.X.apk will generate NewPipe_vX.X.X.apk.sig It will also output 'using \"FINGERPRINT\" as default secret key for signing'; keep track of the FINGERPRINT part Send an email to @TheAssassin and attach both NewPipe_vX.X.X.apk and NewPipe_vX.X.X.apk.sig If @TheAssassin does not already know it, send him your PGP key FINGERPRINT you obtained before You should not send it using email this time, but using another service through which @TheAssassin can be almost sure it is really you (this is a sort of 2FA) For example, you can send it on the IRC group, or create a GitHub gist with the fingerprint and then give that link to @TheAssassin Notify him on IRC that you have sent him an email He will send you back the signed APK Make sure its name is still NewPipe_vX.X.X.apk (rename if it's not the case) Install it on your device to see if everything went well (note that installation will work only if your currently installed version of newpipe comes from NewPipe's F-Droid repo or GitHub) Tell @TheAssassin to \"push the buttons\", i.e. publish the signed APK in NewPipe's F-Droid repo. Publishing the release Go to the draft changelog kept on GitHub Set vX.X.X as the tag name Set vX.X.X as the release title Set master as the \"Target:\" branch Attach the signed APK @TheAssassin sent you Publish the release Profit :-D Blog post The blog post writers need an up-to-date list of merged PRs numbered in chronological order. This is so that they can keep track of what changes have already been detailed in the draft blog post, and which ones still need to be added. So make sure that there is always at least one up-to-date \"master copy\" of the draft release notes available for them to review. The blog post should ideally be published before the GitHub release is made (so that the link to it works!), but in case of some delay, it is fine to let the blog post come later. It is far more important to get the release into users' hands sooner. In order for the blog post to be published, ask @TheAssassin to \"press the buttons\" again Once the blog post is ready, add this block of text on top of the release notes on GitHub: [:arrow_right: :arrow_right: :arrow_right: Read the blog post :arrow_left: :arrow_left: :arrow_left:](LINK_TO_BLOG_POST)","title":"Release instructions for normal releases"},{"location":"07_release_instructions/#release-instructions-for-normal-releases","text":"This page contains detailed instructions for normal releases. Refer to Releasing a New NewPipe Version for other information about releases.","title":"Release instructions for normal releases"},{"location":"07_release_instructions/#preliminary-steps","text":"","title":"Preliminary steps"},{"location":"07_release_instructions/#permissions","text":"Have admin rights on Weblate You should be able to access Weblate's Maintenance page Tip: if the correct page does not show up when clicking that URL, make sure you are logged in ;-) Have at least maintainer rights on the NewPipe and NewPipeExtractor repos","title":"Permissions"},{"location":"07_release_instructions/#repositories","text":"Have a cloned NewPipe local repository (for the rest of the page, origin is assumed to be the remote at github.com/TeamNewPipe/NewPipe ) Add the weblate remote to the same local repository (the URL used below can be found on the Maintenance page on Weblate) git remote add weblate https://hosted.weblate.org/git/newpipe/strings/ Make sure there are no pending changes git clean -fdx to discard them all ( CAUTION ) Switch to the dev branch and make sure it is up-to-date with the remote: git checkout dev git pull origin dev","title":"Repositories"},{"location":"07_release_instructions/#version-name-and-conventions","text":"Find the version code of the next release by looking for versionCode in app/build.gradle : You will add 1 to that value (from now on called NEW_VERSION_CODE ) to get the new value (but do not edit the file yet) Choose the version number of the next release according to semantic versioning (from now on called X.X.X )","title":"Version name and conventions"},{"location":"07_release_instructions/#identification","text":"Have gpg installed and usable on your PC Have a GPG key, which can be used to verify that a file is really from you","title":"Identification"},{"location":"07_release_instructions/#pull-changes-from-weblate","text":"Go to Weblate's Maintenance tab Press the Lock button to prevent translators from translating while you are creating commits; remember to Unlock later! Press the Update button to update Weblate with the latest changes on NewPipe's dev branch Press the Commit button, if needed, to make sure Weblate creates a commit for translations which have not been committed yet Now go back to the local git repository In case you followed these steps before, delete the weblate-dev branch git branch -D weblate-dev Fetch new changes from the weblate remote git fetch weblate Create a new branch starting from weblate/dev , named weblate-dev , and switch to it git checkout -b weblate-dev weblate/dev If you run git log --oneline --graph you should see a Weblate commit on top, and then all of the commits currently on the dev branch: * cmt12hash (HEAD -> weblate-dev, weblate/dev) Translated using Weblate (...) * cmt89hash (origin/dev, dev) Commit message ... Switch back to the dev branch git checkout dev Merge weblate-dev into dev : git merge weblate-dev","title":"Pull changes from Weblate"},{"location":"07_release_instructions/#create-a-changelog","text":"Finalize the draft changelog kept on GitHub , in case there are still some things to fill in Remove the temporary instructions, and the numbers before - which keep track of the order in which the PRs were merged, as that info is useful only for the blog post writers Before removing that information, you may want to send the original changelog to the blogpost writers Create a new English changelog in the fastlane/metadata/android/en-US/changelogs/ folder The file should be named NEW_VERSION_CODE.txt , using the new version code found in the Preliminary steps The file should have this structure (sections with no points can be removed): New \u2022 ... Improved \u2022 ... Fixed \u2022 ... Make sure you use the \u2022 for points (it looks nicer than - ) Capitalize the first letter in each point Use English verbs as if you were asking someone to do something, so for example use \"Fix abc\" and not \"Fixed abc\"; this allows saving a few characters and using a consistent style Prepend [SERVICE] to service-only changes (e.g. \"\u2022 [YouTube] Add mixes\") Summarize only the most important changes from the draft release kept on GitHub (it contains all merged pull requests) Make sure the file size is at most 500 bytes , in order to fit F-Droid's changelog size limit (!) Tip: removing the newline at the end of the file saves 1 byte ;-) Commit the file on the dev branch (try to stick to the provided commit message template) git add fastlane/metadata/android/en-US/changelogs/NEW_VERSION_CODE.txt git commit -m \"Add changelog for vX.X.X (NEW_VERSION_CODE)\"","title":"Create a changelog"},{"location":"07_release_instructions/#push-the-changelog-to-weblate","text":"Now there should be two new commits (the Weblate and changelog ones) on your local dev branch, which are not on NewPipe's remote dev branch. If you are an admin of the NewPipe repo, just push the changes to the remote dev git push origin dev If you are not an admin, create a pull request normally and ask someone with maintainer access to merge it Go to Weblate's Maintenance tab Press the Update button to update Weblate with the commit you just pushed on NewPipe's dev branch Press the Unlock button to allow translators to translate the changelog and possibly other components ( do not forget this step! ) Note that we had to do this process on NewPipe's dev branch because: Weblate's components are connected to NewPipe's dev branch, and will update changes from there Weblate's git repo is not writable, so there is no way to push commits there manually","title":"Push the changelog to Weblate"},{"location":"07_release_instructions/#creating-the-release-branch","text":"Create a new branch starting from dev , named release-X.X.X , and switch to it git checkout dev git checkout -b release-X.X.X Edit the app/build.gradle file to update the extractor Set the NewPipeExtractor dependency version to a suitable one (usually the last commit in the NewPipeExtractor repo ) Commit the extractor update (if you used a specific version, append to VERSION to the commit message) git add app/build.gradle git commit -m \"Update NewPipeExtractor\" Edit the app/build.gradle file to bump the release Set versionCode to NEW_VERSION_CODE , i.e. increment the value by 1 as described in the Preliminary steps Set versionName to \"X.X.X\" Commit the version bump (try to stick to the provided commit message template) git add app/build.gradle git commit -m \"Release vX.X.X (NEW_VERSION_CODE)\" Push the newly created branch to the NewPipe repo git push origin release-X.X.X","title":"Creating the release branch"},{"location":"07_release_instructions/#creating-the-pull-request","text":"Create a Pull Request (PR) from the new branch you just pushed If you used the correct branch name you should be able to use this URL, after changing the X.X.X: https://github.com/TeamNewPipe/NewPipe/pull/new/release-X.X.X Make sure the PR has master as the base branch and release-X.X.X as the compare branch The PR title should be \"Release vX.X.X (NEW_VERSION_CODE)\" Remove the entire PR template, and instead put these two lines in the description (the ISSUE_NUMBER will be replaced later): Do **not** report regressions here, but rather in the corresponding issue: #ISSUE_NUMBER The changelog is also there. Once you have created the PR, note down its number (from now on called PR_NUMBER ) In case some issue would be fixed when the release PR is merged, link them using the \"Development\" tab on the right, or add a \"Fixes #....\" in the PR description Check out #8231 for reference","title":"Creating the Pull Request"},{"location":"07_release_instructions/#creating-the-issue","text":"Create an issue Click here to open one without a template The issue title should be \"Release vX.X.X (please TEST!)\" The issue should have some sections, in the same order as provided below, with ## before titles The ## Testing for regressions section should contain the following lines; more information about how to obtain the APK are given at Testing APKs Debug APK (built by our CI in #PR_NUMBER): ... Please report **only regressions** (i.e. new issues) here, not issues that were already present in previous releases! An optional ## TODO section should contain a list of things that still need to be done before releasing, for example regressions that need to be fixed, or a reminder to merge the Weblate changelogs before releasing (use - [ ] to create checkbox lists) The ## NewPipeExtractor version section should contain a link to the NewPipeExtractor release this new NewPipe version will ship with (i.e. the one set in Creating the release branch ); choose one of these lines as a template This version of NewPipe will ship with [NewPipeExtractor version NPE_VERSION](https://github.com/TeamNewPipe/NewPipeExtractor/releases/tag/NPE_VERSION) This version of NewPipe will ship with [NewPipeExtractor commit FIRST_7_DIGITS_OF_NPE_COMMIT](https://github.com/TeamNewPipe/NewPipeExtractor/commit/NPE_COMMIT) Create the App changelog section using the template below. Copy the draft Markdown changelog kept on GitHub (you finalized it earlier in Create a changelog ) to the clipboard and paste it where specified below (make sure to leave a newline above, otherwise Markdown breaks): <details><summary><h2>App changelog </h2></summary><p> INSERT_COPIED_CHANGELOG_HERE </details> Once you have created the issue, pin it using the \"Pin issue\" button on the right Update the ISSUE_NUMBER in the pull request description Check out #8230 for reference","title":"Creating the issue"},{"location":"07_release_instructions/#testing-apks","text":"The first time you open the release issue, and then each time some changes are made to the release PR, you should provide a debug APK in the ## Testing for regressions section. Wait for the Continuous Integration (CI) to finish testing the PR, then download the resulting debug APK artifact from the \"Checks\" tab Rename it to NewPipe_vX.X.X_RC1_debug.apk where RC1 should be incremented to RC2 and so on each time a new APK is provided Zip it and make sure the .zip file has the same name as the .apk it contains Upload it in the issue description, replacing the ... placeholder used above Sometimes it might be needed to also provide a release APK. In this case follow the same steps as above, with these differences: Make sure you are on the release-X.X.X branch Build the release APK yourself in Android Studio and sign it with your keys Temporarily edit the app/build.gradle file and add System.properties.put(\"packageSuffix\", \"vX_X_X\") at the top of the android -> buildTypes -> release block, which ensures that the application has a different package name than the official one Build and sign an APK via \"Build -> Generate Signed Bundle / APK...\" Make sure it installs correctly on your device Use this naming scheme: NewPipe_vX.X.X_RC1_release.apk Add a line to the ## Testing for regressions section, of this form: Release APK (built and signed by @YOUR_GITHUB_USERNAME): ...","title":"Testing APKs"},{"location":"07_release_instructions/#taking-care-of-regressions-quickfixes","text":"The release issue and pull request should stay open for roughly one week , so that people can test the provided APKs and give feedback. If a regression is reported by some user, it should possibly be solved before releasing, otherwise the app would become more broken after each release. A regression is a bug now present in some code that used to run well in the last release, but was then modified in this release (supposedly to fix something else) and is now broken. So the following do not classify as regressions: some videos stop working because YouTube made some changes; the newly introduced big feature XYZ is still not perfect and has some bugs; a random crash reproducible also on previous versions... You get the point. Before releasing, try to fix any regressions that are reported, but avoid fixing non-regressions, since those should be treated with the same care and attention as all other issues. As a Release Manager, you might be required to fix regressions, so plan your release at a time when you are available. Pull requests fixing regressions should target the release-X.X.X branch, not the dev branch! When merging those PRs, also provide a new Release Candidate APK.","title":"Taking care of regressions (quickfixes)"},{"location":"07_release_instructions/#finally-merging-the-pull-request","text":"Once enough time has passed and all regressions and TODOs have been solved, you can proceed with the actual release. The following points include merging weblate changes again. In the local repository, check out the release branch and make sure it is up-to-date with the remote git checkout release-X.X.X git pull origin release-X.X.X Go to Weblate's Maintenance tab Press Lock ; remember to Unlock later! Press Update Press Commit , if needed Now go back to the local git repository Delete the weblate-dev branch, just in case git branch -D weblate-dev Fetch changes from Weblate (in particular you should see the weblate/dev remote branch being updated) git fetch weblate Obtain the hash of the last commit on the weblate/dev remote branch git log -n 1 --pretty=\"format:%H\" weblate/dev Cherry pick the hash you obtained above into the release branch (the one you are currently on) git cherry-pick HASH Push the changes to the remote branch git push origin release-X.X.X Merge the PR you created before Delete the GitHub remote branch associated with the PR, i.e. release-X.X.X (there should be a button in the PR) Close the issue you created before Merge dev back into master (since the PR merged changes onto master ) git checkout master git pull origin master git checkout dev git pull origin dev git merge master git push origin dev or create another temporary PR and merge it immediately Go to Weblate's Maintenance tab Press Unlock","title":"Finally merging the pull request"},{"location":"07_release_instructions/#creating-the-apk","text":"Now on the remote master branch there is the release code which you need to turn into an APK. In the local repository, check out the master branch and make sure it is up-to-date with the remote git checkout master git pull origin master Open the local project in Android Studio Run the Gradle clean task using Android Studio's interface, in order to clean up temporary/cache files that may interfere with reproducible builds Double press Ctrl, type gradle clean , press Enter Make sure leftover files from building RC releases are actually removed, in order to avoid confusion rm -rf ./app/release Run the Gradle assembleRelease task using Android Studio's interface: it will start the process of building an unsigned APK Double press Ctrl, type gradle assembleRelease , press Enter After a while you should find the APK under ./app/build/outputs/apk/release/app-release-unsigned.apk","title":"Creating the APK"},{"location":"07_release_instructions/#having-the-apk-signed-by-theassassin","text":"Currently @TheAssassin is the only holder of NewPipe's APK signing keys. Therefore you should send the unsigned APK to him, after which he will sign it and send it back to you. He will also then publish the signed APK in NewPipe's F-Droid repo. Rename app-release-unsigned.apk to NewPipe_vX.X.X.apk Generate a signature for the APK file gpg -b NewPipe_vX.X.X.apk will generate NewPipe_vX.X.X.apk.sig It will also output 'using \"FINGERPRINT\" as default secret key for signing'; keep track of the FINGERPRINT part Send an email to @TheAssassin and attach both NewPipe_vX.X.X.apk and NewPipe_vX.X.X.apk.sig If @TheAssassin does not already know it, send him your PGP key FINGERPRINT you obtained before You should not send it using email this time, but using another service through which @TheAssassin can be almost sure it is really you (this is a sort of 2FA) For example, you can send it on the IRC group, or create a GitHub gist with the fingerprint and then give that link to @TheAssassin Notify him on IRC that you have sent him an email He will send you back the signed APK Make sure its name is still NewPipe_vX.X.X.apk (rename if it's not the case) Install it on your device to see if everything went well (note that installation will work only if your currently installed version of newpipe comes from NewPipe's F-Droid repo or GitHub) Tell @TheAssassin to \"push the buttons\", i.e. publish the signed APK in NewPipe's F-Droid repo.","title":"Having the APK signed by @TheAssassin"},{"location":"07_release_instructions/#publishing-the-release","text":"Go to the draft changelog kept on GitHub Set vX.X.X as the tag name Set vX.X.X as the release title Set master as the \"Target:\" branch Attach the signed APK @TheAssassin sent you Publish the release Profit :-D","title":"Publishing the release"},{"location":"07_release_instructions/#blog-post","text":"The blog post writers need an up-to-date list of merged PRs numbered in chronological order. This is so that they can keep track of what changes have already been detailed in the draft blog post, and which ones still need to be added. So make sure that there is always at least one up-to-date \"master copy\" of the draft release notes available for them to review. The blog post should ideally be published before the GitHub release is made (so that the link to it works!), but in case of some delay, it is fine to let the blog post come later. It is far more important to get the release into users' hands sooner. In order for the blog post to be published, ask @TheAssassin to \"press the buttons\" again Once the blog post is ready, add this block of text on top of the release notes on GitHub: [:arrow_right: :arrow_right: :arrow_right: Read the blog post :arrow_left: :arrow_left: :arrow_left:](LINK_TO_BLOG_POST)","title":"Blog post"},{"location":"08_documentation/","text":"About This Documentation The documentation you are currently reading was written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like HTML or LaTeX. Installation Mkdocs is written in Python and is distributed through the Python internal package manager pip , thus you need to get python and pip running on your operating system first. Windows Download the latest Python3 version. When running the setup program, make sure to tick, \"Add Python 3.x to PATH\". Install Python. Open PowerShell or cmd.exe and type: pip3 install mkdocs . MacOS MacOS already includes Python, however, pip is still missing. The easiest and most nondestructive way is to install the MacOS package manager, homebrew , first. The advantage of homebrew is that it will only modify your home directory, and not the root dir, so your OS will not be tampered with. Install homebrew . Install Python from Homebrew, which will also install pip. Enter this command: brew install python . Install mkdocs: pip3 install mkdocs . Linux/*BSD Linux/*BSD also has Python pre-installed. Most distributions also contain pip by default. If it is not installed, you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly. Android/ChromeOS This might sound funny, but according to the growing amount of Chromebooks and Android tablets with keyboards, this might actually be useful. Install the Termux App from F-Droid . Launch Termux and type apt update Install Python and Git with the command: apt install git python Install mkdocs with pip install mkdocs . From herein, everything will be the same as on Desktop. If you want to edit the files, you can (besides vim or emacs which are available through Termux) use your preferred text editor on Android. This is possible by opening the files with the Termux integration of the build in android file manager: Updating Sometimes, mkdocs changes the way of how it serves, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To check, simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a Linux/BSD* system. Using mkdocs In order to extend this documentation, you have to clone it from its GitHub repository . When you clone it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. Here is a guide about how to use mkdocs. Write and Deploy If you are writing a documentation page and want a live preview of it, you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal web server on port 8000 . So all you have to do is type localhost:8000 into the address bar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page so it will be up to date at the GitHub pages , simply type mkdocs gh-deploy . However, please be aware that this will not push your changes to the master branch of the repository. So, you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only privileged maintainers can do this.","title":"About This Documentation"},{"location":"08_documentation/#about-this-documentation","text":"The documentation you are currently reading was written using mkdocs . It is a tool that will generate a static website based on markdown files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like HTML or LaTeX.","title":"About This Documentation"},{"location":"08_documentation/#installation","text":"Mkdocs is written in Python and is distributed through the Python internal package manager pip , thus you need to get python and pip running on your operating system first.","title":"Installation"},{"location":"08_documentation/#windows","text":"Download the latest Python3 version. When running the setup program, make sure to tick, \"Add Python 3.x to PATH\". Install Python. Open PowerShell or cmd.exe and type: pip3 install mkdocs .","title":"Windows"},{"location":"08_documentation/#macos","text":"MacOS already includes Python, however, pip is still missing. The easiest and most nondestructive way is to install the MacOS package manager, homebrew , first. The advantage of homebrew is that it will only modify your home directory, and not the root dir, so your OS will not be tampered with. Install homebrew . Install Python from Homebrew, which will also install pip. Enter this command: brew install python . Install mkdocs: pip3 install mkdocs .","title":"MacOS"},{"location":"08_documentation/#linuxbsd","text":"Linux/*BSD also has Python pre-installed. Most distributions also contain pip by default. If it is not installed, you may need to figure out how to install pip3 through the package manager of your system. Install pip3 with these commands according to distributions: Ubuntu/Mint : apt install python3-pip Fedora/CentOS : sudo dnf install python3-pip Arch/Manjaro : sudo pacman -S python-pip openSuse : sudo zypper install python-pip *BSD : You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it. Run pip3 install mkdocs to install mkdocs only for the current user, or run sudo pip3 install mkdocs to install mkdocs systemwide. Last one has the higher chance to work properly.","title":"Linux/*BSD"},{"location":"08_documentation/#androidchromeos","text":"This might sound funny, but according to the growing amount of Chromebooks and Android tablets with keyboards, this might actually be useful. Install the Termux App from F-Droid . Launch Termux and type apt update Install Python and Git with the command: apt install git python Install mkdocs with pip install mkdocs . From herein, everything will be the same as on Desktop. If you want to edit the files, you can (besides vim or emacs which are available through Termux) use your preferred text editor on Android. This is possible by opening the files with the Termux integration of the build in android file manager:","title":"Android/ChromeOS"},{"location":"08_documentation/#updating","text":"Sometimes, mkdocs changes the way of how it serves, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To check, simply run pip3 install --upgrade mkdocs or sudo pip3 install --upgrade mkdocs if you installed pip system wide on a Linux/BSD* system.","title":"Updating"},{"location":"08_documentation/#using-mkdocs","text":"In order to extend this documentation, you have to clone it from its GitHub repository . When you clone it, you will find a mkdocs.yml file, and a docs directory inside. The yaml file is the config file while in the directory docs the documentation files are stored. Here is a guide about how to use mkdocs.","title":"Using mkdocs"},{"location":"08_documentation/#write-and-deploy","text":"If you are writing a documentation page and want a live preview of it, you can enter the root directory of this documentation project, and then run mkdocs serve this will start the mkdocs internal web server on port 8000 . So all you have to do is type localhost:8000 into the address bar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content. If you want to deploy the page so it will be up to date at the GitHub pages , simply type mkdocs gh-deploy . However, please be aware that this will not push your changes to the master branch of the repository. So, you still have to commit and push your changes to the actual git repository of this documentation. Please be aware that only privileged maintainers can do this.","title":"Write and Deploy"},{"location":"09_maintainers_view/","text":"Maintainers' Section These are some basic principles that we want maintainers to follow when maintaining NewPipe. Keep it Streamlined NewPipe is a media player for devices on the Android platform, thus it is intended to be used for entertainment. This means it does not have to be some professional application, and it does not have to be complicated to be used. However NewPipe might not focus on the casual user completely as there are some features designed for more experienced users which may require some knowledge about code, however in essence NewPipe should be easy to use, even for an average Android user. Don't add too many special features. NewPipe does not have to be an airplane cockpit. Do not try to fill every single niche that might exist. If people wanted more advanced features, they would use professional tools. If you add too much functionality, you add complexity, and complexity scares away the average user. Focus on NewPipe's scope as a media player for the end user, and only as such. Usability of the user interface should be prioritized. Try to make it comply with material design guidelines . Bugfixes ] Disclaimer: This is a meme. Please don't take it personally. Always prioritize fixing bugs , as the best application with the best features does not help much if it is broken, or annoying to use. Now if a program is in an early stage it is quite understandable that many things break. This is one reason why NewPipe still has no \"1\" in the beginning of its version number. By now, NewPipe is in a stage where there should be a strong focus on stability. If there are multiple Pull Requests open, check the ones with bugfixes first. Do not add too many features every version, as every feature will inevitably introduce more bugs. It is OK if PRs remain open for a while, but don't leave them open for too long. If there are bugs that are stale, or open for a while bump them from time to time, so devs know that there is still something left to fix. Never merge PRs with known issues. From our perception the community does not like to fix bugs, this is why you as a maintainer should especially focus on pursuing bugs. Features Features are also something that can cause a headache. You should not blindly say yes to features, even if they are small, but you should also not immediately say no. If you are not sure, try the feature, look into the code, speak with the developer, and then make a decision. When considering a feature, ask yourself the following questions: Was the feature requested by only a few, or by many? Avoid introducing niche features to satisfy a small handful of users. Was the code rushed and messy, and could a cleaner solution be made? A pull request that adds a frequently requested feature could implement the feature in a messy way. Such PRs should not be merged as it will likely cause problems later down the line, either through problems of extending the feature by introducing many bugs, or simply by breaking the architecture or the philosophy of NewPipe. Does the amount of code justify the feature's purpose? Use critical thinking when considering new features and question whether that features makes sense, is useful, and if it would benefit NewPipe's users. Pull Requests If a PR contains more than one feature/bugfix, be cautious. The more stuff a PR changes, the longer it will take to be added. There also might be things that seem to not have any issues, but other things will, and this would prevent you from merging a PR. This is why it is encouraged to keep one change per pull request, and you should insist that contributors divide such PRs into multiple smaller PRs when possible. Community When you talk to the community, stay friendly and respectful with good etiquette. When you have a bad day, just don't go to GitHub (advice from our experience ;D ). Managing translations via Weblate NewPipe is translated via Weblate . There are two different components which are open for translation: The app strings . The fastlane metadata ; this includes the F-Droid store description and changelogs. Maintainers can access more options to handle Weblate via the Manage > Repository Maintenance button or via the Weblate CLI . These options include basic access to Git operations like commit and rebase as well as locking Weblate to prevent further changes to translations. HINT: When updating Weblate via the web interface, please use the \"Update > Rebase\" option. Update Weblate Weblate is based on NewPipe's dev branch and is configured to automatically update its repository to be in sync with NewPipe. However, Weblate does not update its branch often, therefore it is better to update it manually after changing strings in NewPipe. To do thus manually, commit the Weblate changes and rebase the repository. Sometimes conflicts need to be resoled while rebasing the repository. Conflicts need to be addressed ASAP, because Weblate is automatically locked once conflicts occur. To do so, merge the changes from Weblate into NewPipe . If Weblate does not recognize the new commit by itself, ask Weblate to rebase once more. Weblate unlocks the translations when all conflicts are resolved and no errors are detected. Merge changes from Weblate into NewPipe Weblate does not push the translation changes to NewPipe automatically. Doing this manually, allows the maintainers to do a quick review of the changes. Before merging weblate changes into NewPipe, make sure to commit all Weblate changes and lock the Weblate to prevent modifications while you update Weblate. To merge the changes into NewPipe, checkout Weblate's dev branch. You have read access to Weblate's repository via https://hosted.weblate.org/git/newpipe/strings/ . If there are conflicts when rebasing weblate, resolve them. Check the following things: - Is there a translation for a new language? If yes, register the language with the app's langauge selector - Use Analyse > Inspect Code in Android Studio to find unused strings and potential bugs introduced by Weblate. Pay attention to plurals in Asian languages. They are broken by Weblate on a regular basis. Push the changes to NewPipe's dev branch, update Weblate and unlock it.","title":"Maintainers' Section"},{"location":"09_maintainers_view/#maintainers-section","text":"These are some basic principles that we want maintainers to follow when maintaining NewPipe.","title":"Maintainers' Section"},{"location":"09_maintainers_view/#keep-it-streamlined","text":"NewPipe is a media player for devices on the Android platform, thus it is intended to be used for entertainment. This means it does not have to be some professional application, and it does not have to be complicated to be used. However NewPipe might not focus on the casual user completely as there are some features designed for more experienced users which may require some knowledge about code, however in essence NewPipe should be easy to use, even for an average Android user. Don't add too many special features. NewPipe does not have to be an airplane cockpit. Do not try to fill every single niche that might exist. If people wanted more advanced features, they would use professional tools. If you add too much functionality, you add complexity, and complexity scares away the average user. Focus on NewPipe's scope as a media player for the end user, and only as such. Usability of the user interface should be prioritized. Try to make it comply with material design guidelines .","title":"Keep it Streamlined"},{"location":"09_maintainers_view/#bugfixes","text":"] Disclaimer: This is a meme. Please don't take it personally. Always prioritize fixing bugs , as the best application with the best features does not help much if it is broken, or annoying to use. Now if a program is in an early stage it is quite understandable that many things break. This is one reason why NewPipe still has no \"1\" in the beginning of its version number. By now, NewPipe is in a stage where there should be a strong focus on stability. If there are multiple Pull Requests open, check the ones with bugfixes first. Do not add too many features every version, as every feature will inevitably introduce more bugs. It is OK if PRs remain open for a while, but don't leave them open for too long. If there are bugs that are stale, or open for a while bump them from time to time, so devs know that there is still something left to fix. Never merge PRs with known issues. From our perception the community does not like to fix bugs, this is why you as a maintainer should especially focus on pursuing bugs.","title":"Bugfixes"},{"location":"09_maintainers_view/#features","text":"Features are also something that can cause a headache. You should not blindly say yes to features, even if they are small, but you should also not immediately say no. If you are not sure, try the feature, look into the code, speak with the developer, and then make a decision. When considering a feature, ask yourself the following questions: Was the feature requested by only a few, or by many? Avoid introducing niche features to satisfy a small handful of users. Was the code rushed and messy, and could a cleaner solution be made? A pull request that adds a frequently requested feature could implement the feature in a messy way. Such PRs should not be merged as it will likely cause problems later down the line, either through problems of extending the feature by introducing many bugs, or simply by breaking the architecture or the philosophy of NewPipe. Does the amount of code justify the feature's purpose? Use critical thinking when considering new features and question whether that features makes sense, is useful, and if it would benefit NewPipe's users.","title":"Features"},{"location":"09_maintainers_view/#pull-requests","text":"If a PR contains more than one feature/bugfix, be cautious. The more stuff a PR changes, the longer it will take to be added. There also might be things that seem to not have any issues, but other things will, and this would prevent you from merging a PR. This is why it is encouraged to keep one change per pull request, and you should insist that contributors divide such PRs into multiple smaller PRs when possible.","title":"Pull Requests"},{"location":"09_maintainers_view/#community","text":"When you talk to the community, stay friendly and respectful with good etiquette. When you have a bad day, just don't go to GitHub (advice from our experience ;D ).","title":"Community"},{"location":"09_maintainers_view/#managing-translations-via-weblate","text":"NewPipe is translated via Weblate . There are two different components which are open for translation: The app strings . The fastlane metadata ; this includes the F-Droid store description and changelogs. Maintainers can access more options to handle Weblate via the Manage > Repository Maintenance button or via the Weblate CLI . These options include basic access to Git operations like commit and rebase as well as locking Weblate to prevent further changes to translations. HINT: When updating Weblate via the web interface, please use the \"Update > Rebase\" option.","title":"Managing translations via Weblate"},{"location":"09_maintainers_view/#update-weblate","text":"Weblate is based on NewPipe's dev branch and is configured to automatically update its repository to be in sync with NewPipe. However, Weblate does not update its branch often, therefore it is better to update it manually after changing strings in NewPipe. To do thus manually, commit the Weblate changes and rebase the repository. Sometimes conflicts need to be resoled while rebasing the repository. Conflicts need to be addressed ASAP, because Weblate is automatically locked once conflicts occur. To do so, merge the changes from Weblate into NewPipe . If Weblate does not recognize the new commit by itself, ask Weblate to rebase once more. Weblate unlocks the translations when all conflicts are resolved and no errors are detected.","title":"Update Weblate"},{"location":"09_maintainers_view/#merge-changes-from-weblate-into-newpipe","text":"Weblate does not push the translation changes to NewPipe automatically. Doing this manually, allows the maintainers to do a quick review of the changes. Before merging weblate changes into NewPipe, make sure to commit all Weblate changes and lock the Weblate to prevent modifications while you update Weblate. To merge the changes into NewPipe, checkout Weblate's dev branch. You have read access to Weblate's repository via https://hosted.weblate.org/git/newpipe/strings/ . If there are conflicts when rebasing weblate, resolve them. Check the following things: - Is there a translation for a new language? If yes, register the language with the app's langauge selector - Use Analyse > Inspect Code in Android Studio to find unused strings and potential bugs introduced by Weblate. Pay attention to plurals in Asian languages. They are broken by Weblate on a regular basis. Push the changes to NewPipe's dev branch, update Weblate and unlock it.","title":"Merge changes from Weblate into NewPipe"}]} |