Merge pull request #9 from snappyapple632/master
Rewrite of the documentation.
This commit is contained in:
Christian Schabesberger
GitHub
commit
ccf87875b4
|
|
@ -1,86 +1,82 @@
|
|||
# Prepare everything
|
||||
# Before You Start
|
||||
|
||||
Welcome to the NewPipeExtractor documentation. This tutorial will guide you through the process of creating your own Extractor
|
||||
service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud.
|
||||
These documents will guide you through the process of creating your own Extractor
|
||||
service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube and SoundCloud.
|
||||
The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our
|
||||
[Jdoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) setup.
|
||||
|
||||
__IMPORTANT!!!__ this is likely to be the worst documentation you have ever read, so do not hesitate to
|
||||
__IMPORTANT!!!__ This is likely to be the worst documentation you have ever read, so do not hesitate to
|
||||
[report](https://github.com/teamnewpipe/documentation/issues) if
|
||||
you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community
|
||||
and are open for everyone to help :)
|
||||
|
||||
## Setup your dev environment
|
||||
## Setting Up Your Dev Environment
|
||||
|
||||
First and foremost you need to meet certain conditions in order to write your own service.
|
||||
First and foremost, you need to meet the following conditions in order to write your own service.
|
||||
|
||||
### What you need to know
|
||||
### What You Need to Know:
|
||||
|
||||
- Basic understanding of __[git](https://try.github.io)__
|
||||
- A basic understanding of __[git](https://try.github.io)__
|
||||
- Good __[Java](https://whatpixel.com/best-java-books/)__ knowledge
|
||||
- Good understanding of __[web technology](https://www.w3schools.com/)__
|
||||
- Basic understanding of __[unit testing](https://www.vogella.com/tutorials/JUnit/article.html)__ and __[JUnit](https://junit.org/)__
|
||||
- Flawless understanding of how to [contribute](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md#code-contribution) to the __NewPipe project__
|
||||
- A good understanding of __[web technology](https://www.w3schools.com/)__
|
||||
- A basic understanding of __[unit testing](https://www.vogella.com/tutorials/JUnit/article.html)__ and __[JUnit](https://junit.org/)__
|
||||
- A thorough understanding of how to [contribute](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md#code-contribution) to the __NewPipe project__
|
||||
|
||||
### What you need to have
|
||||
### Tools/Programs You Will Need:
|
||||
|
||||
- A dev environment/ide that supports:
|
||||
- __[git](https://git-scm.com/downloads/guis)__
|
||||
- __[java 8](https://www.java.com/en/download/faq/java8.xml)__
|
||||
- __[gradle](https://gradle.org/)__
|
||||
- __[unit testing](https://junit.org/junit5/)__
|
||||
- I highly recommend [IDEA Community](https://www.jetbrains.com/idea/) since it has everything we need.
|
||||
- __[Java 8](https://www.java.com/en/download/faq/java8.xml)__
|
||||
- __[Gradle](https://gradle.org/)__
|
||||
- __[Unit testing](https://junit.org/junit5/)__
|
||||
- [IDEA Community](https://www.jetbrains.com/idea/) (Strongly recommended, but not required)
|
||||
- A __[Github](https://github.com/)__ account
|
||||
- A lot of patience and excitement ;D
|
||||
|
||||
After making sure all these conditions are provided fork the [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor),
|
||||
After making sure all these conditions are provided, fork the [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor)
|
||||
using the [fork button](https://github.com/TeamNewPipe/NewPipeExtractor#fork-destination-box).
|
||||
This way you have your own working repository. Now clone this repository into your local folder in which you want to work in.
|
||||
Next import the cloned project into your [IDE](https://www.jetbrains.com/help/idea/configuring-projects.html#importing-project)
|
||||
and [run](https://www.jetbrains.com/help/idea/performing-tests.html) it.
|
||||
If all the checks are green you did everything right, and you are good to go to move on to the next chapter.
|
||||
This is so you have a personal repository to develop on. Next, clone this repository into your local folder in which you want to work in.
|
||||
Then, import the cloned project into your [IDE](https://www.jetbrains.com/help/idea/configuring-projects.html#importing-project)
|
||||
and [run it.](https://www.jetbrains.com/help/idea/performing-tests.html)
|
||||
If all the checks are green, you did everything right! You can proceed to the next chapter.
|
||||
|
||||
### Importing the NewPipe Extractor in IntelliJ IDEA
|
||||
If you use IntelliJ IDEA you might want to know how to import the NewPipe extractor painlessly. This is what you want to do.
|
||||
If you use IntelliJ IDEA, you should know the easy way of importing the NewPipe extractor. If you don't, here's how to do it:
|
||||
|
||||
1. `git clone` the extractor onto your computer locally.
|
||||
2. Start IntelliJ Idea, and click on `Import Project`.
|
||||
2. Start IntelliJ Idea and click `Import Project`.
|
||||
3. Select the root directory of the NewPipe Extractor
|
||||
4. Now it's important to select "__Import Project form external Model__" and then choose __Gradle__.
|
||||
4. Select "__Import Project from external Model__" and then choose __Gradle__.
|
||||
|
||||
5. In the next window make sure "__Use grale 'wrapper' task configuration__" is selected.
|
||||
5. In the next window, select "__Use gradle 'wrapper' task configuration__".
|
||||
|
||||
|
||||
### Running test in Android Studio/IntelliJ IDEA
|
||||
### Running "test" in Android Studio/IntelliJ IDEA
|
||||
|
||||
Go to _run_ > _edit configurations_ > _add new configuration_ and select "Gradle".
|
||||
As Gradle Project, select NewPipeExtractor. As task add "test". Now save and you should be able to run.
|
||||
Go to _Run_ > _Edit Configurations_ > _Add New Configuration_ and select "Gradle".
|
||||
As Gradle Project, select NewPipeExtractor. As a task, add "test". Now save and you should be able to run.
|
||||
|
||||
|
||||
|
||||
# Inclusion criteria for services
|
||||
# Inclusion Criteria for Services
|
||||
|
||||
After creating you own service you will need to submit it to our [NewPipeExtractor](https://github.com/teamnewpipe/newpipeextractor)
|
||||
repository. However in order to include your changes you need to follow these rules:
|
||||
After creating you own service, you will need to submit it to our [NewPipeExtractor](https://github.com/teamnewpipe/newpipeextractor)
|
||||
repository. However, in order to include your changes, you need to follow these rules:
|
||||
|
||||
1. Stick to our [Code contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md#code-contribution)
|
||||
2. Do not send services that present content we [don't allow](#not-allowed-content) on NewPipe.
|
||||
3. You need to be willing to keep on maintaining your service after submission.
|
||||
4. Be patient and do the requested changes when one of our maintainers rejects your code.
|
||||
2. Do not send services that present content we [don't allow](#content-that-is-not-permitted) on NewPipe.
|
||||
3. You must be willing to maintain your service after submission.
|
||||
4. Be patient and make the requested changes when one of our maintainers rejects your code.
|
||||
|
||||
## Allowed Content
|
||||
## Content That is Permitted:
|
||||
|
||||
- Basically anything except [NOT allowed content](#not-allowed-content).
|
||||
- Any kind of porn/NSFW that is allowed according to the [US Porn act](https://www.justice.gov/archive/opa/pr/2003/April/03_ag_266.htm).
|
||||
- Advertisement (may be handled specially though)
|
||||
- Any content that is not in the [list of prohibited content](#content-that-is-not-permitted).
|
||||
- Any kind of pornography or NSFW content that does not violate US law.
|
||||
- Advertising, which may need to be approved beforehand.
|
||||
|
||||
## NOT allowed Content
|
||||
## Content That is NOT Permitted:
|
||||
|
||||
- NSFL (Not Safe For Life)
|
||||
- Porn that is not allowed according to [US Porn act](https://www.justice.gov/archive/opa/pr/2003/April/03_ag_266.htm).
|
||||
- Any form of violence
|
||||
- Child pornography
|
||||
- Media that harms others
|
||||
- Media that violates human rights
|
||||
- Copyright infringement/pirated media
|
||||
- Content that is considered NSFL (Not Safe For Life)
|
||||
- Content that is prohibited by US federal law (Sexualization of minors, any form of violence, violations of human rights, etc).
|
||||
- Copyrighted media, without the consent of the copyright holder/publisher.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,19 +1,19 @@
|
|||
# Concept of the Extractor
|
||||
|
||||
## Collector/Extractor pattern
|
||||
## The Collector/Extractor Pattern
|
||||
|
||||
Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern
|
||||
you will find all over the code. It is called the __extractor/collector__ pattern. The idea behind it is that
|
||||
Before you start coding your own service, you need to understand the basic concept of the extractor itself. There is a pattern
|
||||
you will find all over the code, called the __extractor/collector__ pattern. The idea behind it is that
|
||||
the [extractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html)
|
||||
would produce single pieces of data, and the collector would collect it to form usable data for the front end.
|
||||
The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any
|
||||
would produce fragments of data, and the collector would collect them and assemble that data into a readable format for the front end.
|
||||
The collector also controls the parsing process, and takes care of error handling. So, if the extractor fails at any
|
||||
point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of
|
||||
many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe.
|
||||
multiple methods, one method for every data field the collector wants to have. The collectors are provided by NewPipe.
|
||||
You need to take care of the extractors.
|
||||
|
||||
### Usage in the front end
|
||||
### Usage in the Front End
|
||||
|
||||
So typical call for retrieving data from a website would look like this:
|
||||
A typical call for retrieving data from a website would look like this:
|
||||
``` java
|
||||
Info info;
|
||||
try {
|
||||
|
|
@ -26,9 +26,9 @@ try {
|
|||
}
|
||||
```
|
||||
|
||||
### Typical implementation of a single data extractor
|
||||
### Typical Implementation of a Single Data Extractor
|
||||
|
||||
The typical implementation of a single data extractor on the other hand would look like this:
|
||||
The typical implementation of a single data extractor, on the other hand, would look like this:
|
||||
``` java
|
||||
class MyExtractor extends FutureExtractor {
|
||||
|
||||
|
|
@ -53,20 +53,20 @@ class MyExtractor extends FutureExtractor {
|
|||
}
|
||||
```
|
||||
|
||||
## Collector/Extractor pattern for lists
|
||||
## Collector/Extractor Pattern for Lists
|
||||
|
||||
Sometimes information can be represented as a list. In NewPipe a list is represented by a
|
||||
Information can be represented as a list. In NewPipe, a list is represented by a
|
||||
[InfoItemsCollector](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItemsCollector.html).
|
||||
A InfoItemCollector will collect and assemble a list of [InfoItem](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItem.html).
|
||||
For each item that should be extracted a new Extractor must be created, and given to the InfoItemCollector via [commit()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItemsCollector.html#commit-E-).
|
||||
For each item that should be extracted, a new Extractor must be created, and given to the InfoItemCollector via [commit()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItemsCollector.html#commit-E-).
|
||||
|
||||
|
||||
|
||||
If you are implementing a list for your service you need to extend InfoItem containing the extracted information,
|
||||
and implement an [InfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html)
|
||||
If you are implementing a list for your service you need to extend InfoItem containing the extracted information
|
||||
and implement an [InfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html),
|
||||
that will return the data of one InfoItem.
|
||||
|
||||
A common Implementation would look like this:
|
||||
A common implementation would look like this:
|
||||
```
|
||||
private MyInfoItemCollector collectInfoItemsFromElement(Element e) {
|
||||
MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId());
|
||||
|
|
@ -90,19 +90,18 @@ private MyInfoItemCollector collectInfoItemsFromElement(Element e) {
|
|||
|
||||
```
|
||||
|
||||
## InfoItems encapsulated in pages
|
||||
## InfoItems Encapsulated in Pages
|
||||
|
||||
When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail,
|
||||
or its creator. Such info can be called __list header__.
|
||||
When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail,
|
||||
and its creator. Such info can be called __list header__.
|
||||
|
||||
When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down.
|
||||
|
||||
This is why a list in NewPipe lists are chopped down into smaller lists called [InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html)s. Each page has its own URL, and needs to be extracted separately.
|
||||
|
||||
Additional metainformation about the list such as its title a thumbnail
|
||||
or its creator, and extracting multiple pages can be handled by a
|
||||
Additional metadata about the list and extracting multiple pages can be handled by a
|
||||
[ListExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html),
|
||||
and it's [ListExtractor.InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html).
|
||||
and its [ListExtractor.InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html).
|
||||
|
||||
For extracting list header information it behaves like a regular extractor. For handling `InfoItemsPages` it adds methods
|
||||
such as:
|
||||
|
|
@ -116,7 +115,7 @@ such as:
|
|||
|
||||
|
||||
The reason why the first page is handled special is because many Websites such as YouTube will load the first page of
|
||||
items like a regular webpage, but all the others as an AJAX request.
|
||||
items like a regular web page, but all the others as an AJAX request.
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,27 +1,27 @@
|
|||
# Concept of LinkHandler
|
||||
# Concept of the LinkHandler
|
||||
|
||||
[LinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandler.html)
|
||||
represent Links to resources like videos, search requests, channels, etc.
|
||||
The idea behind them is that a video can have multiple links pointing to it, but it has
|
||||
The [LinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandler.html)
|
||||
represent links to resources like videos, search requests, channels, etc.
|
||||
The idea is that a video can have multiple links pointing to it, but it has
|
||||
one unique ID that represents it, like this example:
|
||||
|
||||
[oHg5SJYRHA0](https://www.youtube.com/watch?v=oHg5SJYRHA0) can be represented as:
|
||||
|
||||
- https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube)
|
||||
- https://youtu.be/oHg5SJYRHA0
|
||||
- https://m.youtube.com/watch?v=oHg5SJYRHA0
|
||||
- [https://www.youtube.com/watch?v=oHg5SJYRHA0](https://www.youtube.com/watch?v=oHg5SJYRHA0) (the default URL for YouTube)
|
||||
- [https://youtu.be/oHg5SJYRHA0](https://youtu.be/oHg5SJYRHA0) (the shortened link)
|
||||
- [https://m.youtube.com/watch?v=oHg5SJYRHA0](https://m.youtube.com/watch?v=oHg5SJYRHA0) (the link for mobile devices)
|
||||
|
||||
### Importand notes about LinkHandler:
|
||||
- A simple `LinkHandler` will contain the default URL, the ID and the original URL.
|
||||
- `LinkHandler`s are ReadOnly
|
||||
- A simple `LinkHandler` will contain the default URL, the ID, and the original URL.
|
||||
- `LinkHandler`s are read only.
|
||||
- `LinkHandler`s are also used to determine which part of the extractor can handle a certain link.
|
||||
- In order to get one you must either call
|
||||
[fromUrl()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromUrl-java.lang.String-) or [fromId()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromId-java.lang.String-) of the the corresponding `LinkHandlerFactory`.
|
||||
- Every type of Type of Resource has its own `LinkHandlerFactory`. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.
|
||||
- Every type of resource has its own `LinkHandlerFactory`. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.
|
||||
|
||||
### Usage
|
||||
|
||||
So the typical usage for getting a LinkHandler would look like this.
|
||||
The typical usage for obtaining a LinkHandler would look like this:
|
||||
```java
|
||||
LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory();
|
||||
LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_video");
|
||||
|
|
@ -29,7 +29,7 @@ LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_v
|
|||
|
||||
### Implementation
|
||||
|
||||
In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg:
|
||||
In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg:
|
||||
|
||||
```java
|
||||
class MyStreamLinkHandlerFactory extends LinkHandlerFactory {
|
||||
|
|
@ -53,15 +53,16 @@ class MyStreamLinkHandlerFactory extends LinkHandlerFactory {
|
|||
|
||||
### ListLinkHandler and SearchQueryHandler
|
||||
|
||||
List based resources like channels and playlists can be sorted, and filtered.
|
||||
List based resources, like channels and playlists, can be sorted and filtered.
|
||||
Therefore these type of resources don't just use a LinkHandler, but a class called
|
||||
[ListLinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html)
|
||||
which inherits from LinkHandler and adds the fields [ContentFilter](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html#contentFilters)
|
||||
which is used to filter by resource type like stream or playlist, and
|
||||
[SortFilter](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html#sortFilter)
|
||||
which is used to sort by name, date or view count.
|
||||
[ListLinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html),
|
||||
which inherits from LinkHandler and adds the field [ContentFilter](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html#contentFilters),
|
||||
which is used to filter by resource type, like stream or playlist, and
|
||||
[SortFilter](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html#sortFilter),
|
||||
which is used to sort by name, date, or view count.
|
||||
|
||||
__!!ATENTION!!__ Be careful when you implement content filter: No selected filter equals all filters selected. So if your get an empty content filter list in your extractor make sure you return everything. By means use if statements like this `contentFilter.contains("video") || contentFilter.isEmpty()`.
|
||||
__!!ATTENTION!!__ Be careful when you implement a content filter: No selected filter equals all filters selected. If your get an empty content filter list in your extractor, make sure you return everything. By all means, use "if"
|
||||
statements like `contentFilter.contains("video") || contentFilter.isEmpty()`.
|
||||
|
||||
ListLinkHandler are also created by overriding the [ListLinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html)
|
||||
additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override
|
||||
|
|
|
|||
|
|
@ -1,22 +1,21 @@
|
|||
# Implement a service
|
||||
# Implementing a Service
|
||||
|
||||
Services or better service connectors are the parts of NewPipe which communicate with an actual service like YouTube.
|
||||
This Page will describe how you can implement and add your own to the extractor. Please make sure you read and understand the
|
||||
Services, or better service connectors, are the parts of NewPipe which communicate with an actual service like YouTube.
|
||||
This page will describe how you can implement and add your own services to the extractor. Please make sure you read and understand the
|
||||
[Concept of Extractors](https://teamnewpipe.github.io/documentation/01_Concept_of_the_extractor/)
|
||||
and the [Concept of LinkHandler](https://teamnewpipe.github.io/documentation/02_Concept_of_LinkHandler/)
|
||||
before doing this.
|
||||
before continuing.
|
||||
|
||||
### Required and optional parts
|
||||
Your service does not have to implement everything. Some parts are optional.
|
||||
### Required and Optional Parts
|
||||
Your service does not have to implement everything; some parts are optional.
|
||||
This is because not all services support every feature other services support. For example, it might be that a certain
|
||||
service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding
|
||||
factory method of the your __StreamingService__ implementation return __null__. The frontend will handle the lack of
|
||||
having channels then.
|
||||
having channels.
|
||||
|
||||
However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes
|
||||
of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself.
|
||||
However, if you start to implement one of the optional parts of the list below, you will have to implement all of its parts/classes. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself.
|
||||
|
||||
__The parts of a service:__
|
||||
__The Parts of a Service:__
|
||||
|
||||
- [Head of Service](#head-of-service)
|
||||
- [Stream](#stream)
|
||||
|
|
@ -27,56 +26,56 @@ __The parts of a service:__
|
|||
|
||||
### Allowed Libraries
|
||||
|
||||
The NewPipe Extractor already comes along with a lot of usable tools and external libraries that should make extracting easy.
|
||||
For some specific (tiny) tasks regex is allowed. Here you can take a look at the
|
||||
The NewPipe Extractor already includes a lot of usable tools and external libraries that should make extracting easy.
|
||||
For some specific (tiny) tasks, Regex is allowed. Here you can take a look at the
|
||||
[Parser](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/utils/Parser.html),
|
||||
which will give you a little help with that. __Use Regex with care!!!__ Avoid it as often as possible. It's better to
|
||||
ask us to introduce a new library than start using regex to often.
|
||||
ask us to introduce a new library than start using Regex to often.
|
||||
|
||||
- Html/XML Parsing: [jsoup](https://jsoup.org/apidocs/overview-summary.html)
|
||||
- JSON Parsing: [nanojson](https://github.com/mmastrac/nanojson#parser-example)
|
||||
- JavaScript Parsing/Execution: [Mozilla Rhino](https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Rhino/Documentation)
|
||||
- Link detection in strings: [AutoLink](https://github.com/robinst/autolink-java)
|
||||
|
||||
If you need to introduce new libraries please tell us before you do it.
|
||||
If you need to introduce new libraries, please tell us before you do so.
|
||||
|
||||
### Head of Service
|
||||
|
||||
First of all if you want to create a new service you should create a new package below `org.schabi.newpipe.services`
|
||||
First of all, if you want to create a new service, you should create a new package below `org.schabi.newpipe.services`
|
||||
, with the name of your service as package name.
|
||||
|
||||
__Parts required to be implemented:__
|
||||
__Parts Required to be Implemented:__
|
||||
|
||||
- [StreamingService](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.html)
|
||||
- [ServiceInfo](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.ServiceInfo.html)
|
||||
|
||||
[StreamingService](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.html)
|
||||
is a factory class that will return objects of all important parts of your service.
|
||||
Every extractor, handler, and info type you add, and which should be part of your implementation, must be instantiated using an
|
||||
Every extractor, handler, and info type you add and should be part of your implementation, must be instantiated using an
|
||||
instance of this class. You can see it as a factory for all objects of your implementation.
|
||||
|
||||
[ServiceInfo](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.ServiceInfo.html)
|
||||
will return some meta information about your service. Such as the name, the capabilities, and your name as well as your
|
||||
email address for further notice and maintenance issues. Remember, after extending this class you need to return an
|
||||
will return some metadata about your service such as the name, capabilities, the author's name, and their
|
||||
email address for further notice and maintenance issues. Remember, after extending this class, you need to return an
|
||||
instance of it by through your implementation of
|
||||
[`StreamingService.getServiceInfo()`](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.html#getServiceInfo--).
|
||||
|
||||
When these two classes are extended by you, you need to add your `StreamingService` to the
|
||||
[ServiceList](https://github.com/TeamNewPipe/NewPipeExtractor/blob/49c2eb51859a58e4bb5ead2d9d0771408f7d59d6/extractor/src/main/java/org/schabi/newpipe/extractor/ServiceList.java#L23)
|
||||
of NewPipe. This way your service will become an official part of the NewPipe Extractor.
|
||||
of NewPipe. This way, your service will become an official part of the NewPipe Extractor.
|
||||
Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor.
|
||||
So when adding your service just give it the ID of the previously last service in the list incremented by one.
|
||||
|
||||
### Stream
|
||||
|
||||
Streams are considered single entities of video or audio. They come along with metainformation like a title, a description,
|
||||
next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation
|
||||
Streams are considered single entities of video or audio. They have metadata like a title, a description,
|
||||
next/related videos, a thumbnail and comments. To obtain the URL to the actual stream data, as well as its metadata,
|
||||
StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will
|
||||
extract one item in a list of items representing such Streams, like a search result or a playlist.
|
||||
Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was
|
||||
extract one item in a list of items representing such streams, like a search result or a playlist.
|
||||
Since every streaming service (obviously) provides streams, this is required to implement. Otherwise, your service was
|
||||
pretty useless :)
|
||||
|
||||
__Parts required to be implemented:__
|
||||
__Parts Required to be Implemented:__
|
||||
|
||||
- [StreamExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamExtractor.html)
|
||||
- [StreamInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamInfoItemExtractor.html)
|
||||
|
|
@ -85,31 +84,31 @@ __Parts required to be implemented:__
|
|||
### Search
|
||||
The SearchExtractor is also required to be implemented. It will take a search query represented as
|
||||
[SearchQueryHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandler.html)
|
||||
and return a list of search results. Since many services support a suggestion popup while you type you will also want to implement
|
||||
a __SuggestionExtractor__. This will make it possible for the frontend to as well display a suggestion while typing.
|
||||
and return a list of search results. Since many services support suggestions as you type, you will also want to implement
|
||||
a __SuggestionExtractor__. This will make it possible for the frontend to also display a suggestion while typing.
|
||||
|
||||
__Parts required to be implemented:__
|
||||
__Parts Required to be Implemented:__
|
||||
|
||||
- [SearchExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/search/SearchExtractor.html)
|
||||
- [SearchQueryHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandlerFactory.html)
|
||||
- [SuggestionExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/SuggestionExtractor.html) _(optional)_
|
||||
|
||||
### Playlist
|
||||
Playlists are lists of streams provided by the service (you may not have to take care about locally saved playlists. This will be handled by the frontend).
|
||||
A playlist may only contains __StreamInfoItems__, but no other __InfoItem__ types.
|
||||
Playlists are lists of streams provided by the service (you might not have to be concerned over locally saved playlists, those will be handled by the frontend).
|
||||
A playlist may only contain __StreamInfoItems__, but no other __InfoItem__ types.
|
||||
|
||||
__Parts required to be implemented:__
|
||||
__Parts Required to be Implemented:__
|
||||
|
||||
- [PlaylistExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/playlist/PlaylistExtractor.html)
|
||||
- [PlayListInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/playlist/PlaylistExtractor.html)
|
||||
- [ListLinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html)
|
||||
|
||||
### Channel
|
||||
A Channel is mostly a [Playlist](#playlist), the only difference is that it does not represent a simple list of streams, but a
|
||||
user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor
|
||||
differs form the one of a playlist.
|
||||
A Channel is mostly a [Playlist](#playlist), the only difference is that it does not only represent a simple list of streams, but also a
|
||||
user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the ChannelExtractor
|
||||
differs from the one of a playlist.
|
||||
|
||||
__Parts required to be implemented:__
|
||||
__Parts Required to be Implemented:__
|
||||
|
||||
- [ChannelExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html)
|
||||
- [ChannelInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html)
|
||||
|
|
@ -117,19 +116,19 @@ __Parts required to be implemented:__
|
|||
|
||||
### Kiosk
|
||||
A kiosk is a list of __InfoItems__ which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content
|
||||
displayed on the main page of a video platform. A kiosk could be something like "Top 20", "Charts", "News", "Creators selection" etc.
|
||||
Kiosk are controversial, many people may not like them. If you don't like them as well please don't refuse to add them thought.
|
||||
Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user,
|
||||
since users of NewPipe can decide by the settings weather they want to see the kiosk page or not.
|
||||
displayed on the main page of a video platform. A kiosk could be something like "Top 20", "Charts", "News", "Creators Selection" etc.
|
||||
Kiosks are controversial; many people may not like them. If you also don't like them, please consider your users and refrain from denying support for them.
|
||||
Your service would look pretty empty if you select it and no video is being displayed. Also, you should not override the preference of the user,
|
||||
since users of NewPipe can decide by the settings whether they want to see the kiosk page or not.
|
||||
|
||||
#### Multiple Kiosks
|
||||
Most services will implement more than one Kiosk, so a service might have a "Top 20" for different categories like "Country Music", "Techno", etc.
|
||||
Most services will implement more than one kiosk, so a service might have a "Top 20" for different categories like "Country Music", "Techno", etc.
|
||||
This is why the extractor will let you implement multiple __KioskExtractors__. Since different kiosk pages might also differ
|
||||
with their HTML structure every page you want to support has to be implemented as its own __KioskExtractor__.
|
||||
However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your __KioskExtractor__
|
||||
with their HTML structure, every page you want to support has to be implemented as its own __KioskExtractor__.
|
||||
However, if the pages are similar, you can use the same implementation, but set the page type when you instantiate your __KioskExtractor__
|
||||
through the __KioskList.KioskExtractorFactory__.
|
||||
|
||||
Every Kiosk you implement needs to be added to your __KioskList__ which you return with your
|
||||
Every kiosk you implement needs to be added to your __KioskList__ which you return with your
|
||||
__StreamingService__ implementation.
|
||||
|
||||
It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service.
|
||||
|
|
@ -160,7 +159,7 @@ public KioskList getKioskList() throws ExtractionException {
|
|||
|
||||
```
|
||||
|
||||
__Parts required to be implemented:__
|
||||
__Parts Required to be Implemented:__
|
||||
|
||||
- [KioskList.KioskExtractorFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/kiosk/KioskList.KioskExtractorFactory.html)
|
||||
- [KioskExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/kiosk/KioskExtractor.html)
|
||||
|
|
|
|||
|
|
@ -1,35 +1,35 @@
|
|||
# Run the changes in the App
|
||||
# Testing Your Changes in the App
|
||||
|
||||
You should develop and test your changes with the JUnit environment that is
|
||||
provided by the NewPipe Extractor and IDEA. If you want to try it then with
|
||||
the actual fronted you need to follow these steps.
|
||||
provided by the NewPipe Extractor and IDEA. If you want to try it with
|
||||
the actual fronted, you need to follow these steps.
|
||||
|
||||
### Setup Android Studio
|
||||
|
||||
First of all you want to setup a working Android Studio environment. For this please
|
||||
First, you'll want to set up a working Android Studio environment. To do this,
|
||||
download Studio from [developer.android.com](https://developer.android.com/studio/),
|
||||
and follow the [instructions](https://developer.android.com/studio/install) about how to set it up.
|
||||
and follow the [instructions](https://developer.android.com/studio/install) on how to set it up.
|
||||
|
||||
### Get the NewPipe code and run it.
|
||||
### Get the NewPipe Code and Run it.
|
||||
|
||||
In order to get it you must simply clone or download from the current `dev` branch
|
||||
In order to get it, you simply clone or download it from the current `dev` branch
|
||||
[github.com/TeamNewPipe/NewPipe.git](https://github.com/TeamNewPipe/NewPipe/archive/dev.zip).
|
||||
You can then build and run it following [these instructions](https://developer.android.com/studio/run/).
|
||||
Please also make sure you are comfortable with [adb](https://en.droidwiki.org/wiki/Android_Debug_Bridge) since
|
||||
you might experience some trouble running your compiled app on a real device, especially under linux where you
|
||||
Also, make sure you are comfortable with [adb](https://en.droidwiki.org/wiki/Android_Debug_Bridge) since
|
||||
you might experience some trouble running your compiled app on a real device, especially under Linux, where you
|
||||
sometimes have to adjust the udev rules in order to
|
||||
[make your device be accessible](https://www.janosgyerik.com/adding-udev-rules-for-usb-debugging-android-devices/).
|
||||
[make your device accessible](https://www.janosgyerik.com/adding-udev-rules-for-usb-debugging-android-devices/).
|
||||
|
||||
### Run your changes on the Extractor
|
||||
### Run Your Changes on the Extractor
|
||||
|
||||
In order to use the extractor in our app we use [jitpack](https://jitpack.io). This is a build service that can build
|
||||
maven *.jar packages for android and java based on a github or gitlab repositories.
|
||||
In order to use the extractor in our app, we use [jitpack](https://jitpack.io). This is a build service that can build
|
||||
maven *.jar packages for Android and Java based on GitHub or GitLab repositories.
|
||||
|
||||
To the extractor through jitpack, you need to push them to your online repository of
|
||||
your copy that you host either on [github](https://github.com) or [gitlab](https://gitlab.com). It's important to host
|
||||
it on one of both. Now copy your repository url in Http format, go to [jitpack](https://jitpack.io/), and past it there.
|
||||
From here you can grab the latest commit via `GET IT` button.
|
||||
I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An "implementation" string will be generated
|
||||
To use the extractor through jitpack, you need to push it to your online repository of
|
||||
your copy that you host either on [GitHub](https://github.com) or [GitLab](https://gitlab.com). It's important to host
|
||||
it on one of both. To copy your repository URL in HTTP format, go to [jitpack](https://jitpack.io/) and paste it there.
|
||||
From here, you can grab the latest commit via `GET IT` button.
|
||||
I recomend not to use a SNAPSHOT, since I am not sure when snapshot is built. An "implementation" string will be generated
|
||||
for you. Copy this string and replace the `implementation 'com.github.TeamNewPipe:NewPipeExtractor:<commit>'` line in
|
||||
the file [/app/build.gradle](https://github.com/TeamNewPipe/NewPipe/blob/dev/app/build.gradle#L58) with it.
|
||||
|
||||
|
|
@ -38,7 +38,7 @@ the file [/app/build.gradle](https://github.com/TeamNewPipe/NewPipe/blob/dev/app
|
|||
Your browser does not support the video tag.
|
||||
</video>
|
||||
|
||||
If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe
|
||||
If everything synced well, then you should only see a screen with OK signs. Now you can compile and run NewPipe
|
||||
with the new extractor.
|
||||
|
||||
|
||||
|
|
@ -46,5 +46,5 @@ with the new extractor.
|
|||
### Troubleshooting
|
||||
|
||||
If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and
|
||||
click on that little paper symbol next to the `GET IT` button. If it is red it already shows that the build failed.
|
||||
click on that little paper symbol next to the `GET IT` button. If it's red, it means that the build failed.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,166 +1,167 @@
|
|||
# Releasing a new NewPipe version
|
||||
# Releasing a New NewPipe Version
|
||||
|
||||
This site is ment for those who want to maintain NewPipe, or just want to know how releasing work.
|
||||
This site is meant for those who want to maintain NewPipe, or just want to know how releasing works.
|
||||
|
||||
|
||||
|
||||
## Difference between regular and hotfix release
|
||||
## Differences Between Regular and Hotfix Releases
|
||||
|
||||
NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website,
|
||||
this however has the disadvantage of the app to brake instantly when Youtube changes something.
|
||||
We can not know when this happen therefore we need to be prepared when it happens, and at lease reduce our downtime as
|
||||
god as possible. Our whole release cycle is therefore designed around this issue.
|
||||
this however has the disadvantage of the app to break instantly when YouTube changes something.
|
||||
We do not know when this happen. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as
|
||||
much as possible. The entire release cycle is therefore designed around this issue.
|
||||
|
||||
So there is a difference between a release that is meant to introduce new features or fix minor bugs,
|
||||
and a version that fixes an issue that occurred because Youtube (or some other service) suddenly changed their website (mostly call this a shutdown).
|
||||
Lets first have a look how a regular release work, and then how the hotfix release work.
|
||||
There is a difference between a release that introduces new features
|
||||
and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown).
|
||||
Lets have a look at the characteristics of a __regular release__, and then the characteristics of a __hotfix release__.
|
||||
|
||||
## Regular releases
|
||||
## Regular Releases
|
||||
|
||||
Regular releases are normal releases like they are done in any other app. Releases are always stored on __master__ branch. By means the latest commit on
|
||||
Regular releases are normal releases like they are done in any other app. Releases are always stored on __master__ branch. The latest commit on
|
||||
__master__ is always equal to the currently released version. No development is done on master. This ensures that we always have one
|
||||
branch with a stable/releasable version.
|
||||
|
||||
### Feature branching
|
||||
For development the __dev__ branch is used. Pushing to __dev__ directly however is also not allowed since QA and testing should be done before adding something to it.
|
||||
This ensures that also the dev version works as good a possible.
|
||||
So in order to change something on the app one may want to __fork__ the dev branch and develop the changes in his own branch (this is called feature branching).
|
||||
### Feature Branching
|
||||
When developing, the __dev__ branch is used. Pushing to __dev__ directly, however, is not allowed, since QA and testing should be done first before adding something to it.
|
||||
This ensures that the dev version works as stable a possible.
|
||||
In order to change something on the app, one may want to __fork__ the dev branch and develop the changes in their own branch (this is called feature branching).
|
||||
|
||||
|
||||
|
||||
Make sure that both the dev branches as well as the master branches of the extractor and the frontend are compatible to each other.
|
||||
If a change is done on the API to the extractor make sure that frontend is being made compatible to these changes. If the PR that should make the frontend compatible
|
||||
again can not be merged please do not merge the corresponding PR on the extractor as well. This should make sure that any developer can run his changes
|
||||
Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other.
|
||||
If a change is done on the API to the extractor, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible
|
||||
again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes
|
||||
on the fronted at any time.
|
||||
|
||||
### Merging features/bugfixes
|
||||
### Merging Features/Bugfixes
|
||||
|
||||
After being done with the feature one should open up a __Pull Reuqest__ to the dev branch here a maintainer can do __Code review__ and __Quality Assurance (QA)__.
|
||||
If you are a maintainer please take care about the code architecture so __corrosion__ or __code shifting__ can be prevented. Please also prefare code quality over functionality.
|
||||
So in short: cool function but bad code -> no merge. We should focus on leaving the code as clean as possible.
|
||||
After finishing a feature, one should open up a __Pull Reuqest__ to the dev branch. From here, a maintainer can do __Code review__ and __Quality Assurance (QA)__.
|
||||
If you are a maintainer, please take care about the code architecture so __corrosion__ or __code shifting__ can be prevented. Please also prioritize code quality over functionality.
|
||||
In short: cool function but bad code = no merge. Focus on leaving the code as clean as possible.
|
||||
|
||||
|
||||
|
||||
At best you as a maintainer should build the app and put the signed apk into the description of that new pullrequest. This way other people can test the feature/bugfix and therefore help with QA. _You may not need to do this every time. It is enough to do it on bigger pull requests._
|
||||
You, as a maintainer, should build the app and put the signed APK into the description of that new pull request. This way, other people can test the feature/bugfix and help with QA. _You may not need to do this every time. It is enough to do it on bigger pull requests._
|
||||
|
||||
After the maintainer merged the new feature into the dev branch he should add the title of the pullrequest or a summary of the changes into the [release note](#release-notes).
|
||||
After the maintainer merges the new feature into the dev branch, he should add the title of the pull request or a summary of the changes into the [release notes](#release-notes).
|
||||
|
||||
### Creating a new release
|
||||
### Creating a New Release
|
||||
|
||||
Once there are enough features together, and the maintainer feels like releasing he should create a new release. Here is a list of things he will want to do then.
|
||||
Be aware of the rule that a release should never be done on a frieday. For NewPipe this mean don't do a release if you don't have time for it!!!
|
||||
Once there are enough features together, and the maintainers believe that NewPipe is ready for a new release, they should create a new release.
|
||||
Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: __Don't do a release if you don't have time for it!!!__
|
||||
Below is a list of things you will want to do:
|
||||
|
||||
1. Fork the __dev__ branch into a new __release_x.y.z__ branch.
|
||||
2. Increase the [version number](#versioning)
|
||||
3. Merge [weblate](https://hosted.weblate.org/projects/newpipe/) changes from `dev` branch at `https://hosted.weblate.org/git/newpipe/strings/`.
|
||||
4. Copy the [release note](#release-notes) from the github version draft into the corresponding fastlane file (see [release note](#release-notes)).
|
||||
5. Open up a pullrequest form the new __release_x.y.z__ branch into the __master__ branch.
|
||||
6. Create an Issue pointing to the new Pullrequest. The reason for opening an issue is that from my perception more people are reading issues then they read pullrequests. Put the release-note into this pull request.
|
||||
2. Increase the [version number](#version-nomenclature)
|
||||
3. Merge [weblate](https://hosted.weblate.org/projects/newpipe/) changes from the `dev` branch at `https://hosted.weblate.org/git/newpipe/strings/`.
|
||||
4. Copy the [release notes](#release-notes) from the GitHub version draft into the corresponding fastlane file (see [release notes](#release-notes)).
|
||||
5. Open up a pull request form the new __release_x.y.z__ branch into the __master__ branch.
|
||||
6. Create an issue pointing to the new pull request. The reason for opening an issue is that from my perception, people read issues more than pull requests. Put the release-note into this pull request.
|
||||
7. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file `NewPipe_<versionNumber>_RC1.apk`.
|
||||
Zip it and post it into the head of the release issue. This way other people can test the release candidate.
|
||||
8. Test and QA the new version with the help of other people
|
||||
9. Leave the PR open for a few days and advertise people to help testing.
|
||||
Zip it and post it to the head of the release issue. This way, others can test the release candidate.
|
||||
8. Test and QA the new version with the help of others.
|
||||
9. Leave the PR open for a few days and advertise it to help testing.
|
||||
|
||||
While being in release phase no new pullrequests must be merged into __dev__ branch.
|
||||
While being in release phase no new pull requests must be merged into __dev__ branch.
|
||||
|
||||
This procedure does not have to be done for the extractor as extractor will be tested together with the fronted.
|
||||
|
||||
### Quckfixes
|
||||
### Quickfixes
|
||||
|
||||
When issuing a new release you will most likely encounter new bugs. These bugs are called __regressions__ as they where not there before.
|
||||
If you notice a regression during release phase you are allowed to push fixes directly into the release branch without having to fork a branch away from it.
|
||||
All maintainers (people who have write access to the release branch) have to be aware that they might be required to fix regressions so plan your release on a time when
|
||||
you have time for coding. Do not introduce new features while being in release phase.
|
||||
When issuing a new release, you will most likely encounter bugs that might not have existed in previous versions. These are called __regressions__.
|
||||
If you find a regression during release phase, you are allowed to push fixes directly into the release branch without having to fork a branch away from it.
|
||||
All maintainers have to be aware that they might be required to fix regressions, so plan your release at a time when
|
||||
you are available. Do not introduce new features during the release phase.
|
||||
|
||||
When you have pushed a quickfix you will want to updated the __release candidate__ you put into the __issue__ corresponding to the __release pull request__.
|
||||
Increment the version number in the filename of the Release candidate. e.g. `NewPipe_<versionNumber>_RC2.apk` etc. _Don't update the actuall version number however :P_.
|
||||
When you have pushed a quickfix, you will want to update the __release candidate__ you put into the __issue__ corresponding to the __release pull request__.
|
||||
Increment the version number in the filename of the release candidate. e.g. `NewPipe_<versionNumber>_RC2.apk` etc. _Don't update the actual version number. :P_
|
||||
|
||||
|
||||
|
||||
### Releasing
|
||||
|
||||
Once the glories day of all days has come, and you feel like fulfilling the ceremony of releasing. This is what you will want to do.
|
||||
After going through the release procedure of having [created a new release](#create_a_new_release) and maybe having done [quickfixes](#quickfixes) on the new release,
|
||||
you will want to do these steps:
|
||||
Once the glorious day of all days has come, and you fulfill the ceremony of releasing.
|
||||
After going through the release procedure of [creating a new release](#create_a_new_release) and maybe a few [quickfixes](#quickfixes) on the new release,
|
||||
this is what you should do when releasing:
|
||||
|
||||
1. Hit merge Pullreqest
|
||||
2. Create a GPG signed tag with the name `v0.x.y`
|
||||
3. Merge __dev__ into master on the extractor
|
||||
4. Create a GPG signed tag with the name `v0.x.y` on the extractor
|
||||
5. Make sure the draft name equals the tag name 
|
||||
6. Make sure to not have forgotten enything
|
||||
7. Hit `Publish Release`
|
||||
8. Rebase quickfix changes back into __dev__ if quickfixes where made
|
||||
1. Click "Merge Pull Request".
|
||||
2. Create a GPG signed tag with the name `v0.x.y`.
|
||||
3. Merge __dev__ into master on the extractor.
|
||||
4. Create a GPG signed tag with the name `v0.x.y` on the extractor.
|
||||
5. Make sure the draft name equals the tag name. 
|
||||
6. Make sure to not have forgotten anything.
|
||||
7. Click "Publish Release".
|
||||
8. Rebase quickfix changes back into __dev__ if quickfixes were made.
|
||||
|
||||
|
||||
|
||||
## Hotfix releases
|
||||
## Hotfix Releases
|
||||
|
||||
|
||||
|
||||
As described aboth NewPipe is a web crawler, and therefore might brake randomly. In order to keep the downtime of NewPipe as low as possible when such a shutdown happens
|
||||
we allow so called __hotfixes__.
|
||||
As aforementioned, NewPipe is a web crawler and could break at any moment. In order to keep the downtime of NewPipe as low as possible, when such a shutdown happens,
|
||||
we allow __hotfixes__.
|
||||
|
||||
|
||||
|
||||
- A hotfix allows work on the master branch instead of the dev branch.
|
||||
- A hotfix MUST __NOT__ contain any features or other bugfixes.
|
||||
- A hotfix may only focus on fixing what has caused the shutdown.
|
||||
- A hotfix MUST __NOT__ contain any features or unrelated bugfixes.
|
||||
- A hotfix may only focus on fixing what caused the shutdown.
|
||||
|
||||
### Hotfix branch
|
||||
### Hotfix Branch
|
||||
|
||||
Hotfixes work on the master branch. The reason for this is because dev branch might have experimental changes that have not yet been tested properly enough to be released. Master however should always be at the latest stable version of NewPipe. If this one brakes due to a shutdown you may therefore want to fix that version.
|
||||
Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch.
|
||||
Of course you are not allowed to push to master directly so you will have to open up a __hotfix__ branch. _If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well._
|
||||
|
||||
|
||||
|
||||
### Releasing
|
||||
|
||||
If you fixed the issue and found it to be tested and reviewed well enough you man release. Here you don't need to undergo the full release procedure of a regular release, which might take up to a few days.
|
||||
Keep in mind that if the hotfix might turn out to be broken after release you want to release another hotfix.
|
||||
Here it is important to release fast, and after all a less broken version of NewPipe is better then a full broken version ¯\\\_(ツ)\_/¯.
|
||||
This is what you will want to do when releasing a hotfix version.
|
||||
If you fixed the issue and found it to be tested and reviewed well enough, you may release it. You don't need to undergo the full release procedure of a regular release, which takes more time to release.
|
||||
Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix.
|
||||
It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better then a non-functional version ¯\\\_(ツ)\_/¯.
|
||||
Here's what you do when releasing a hotfix:
|
||||
|
||||
1. Hit merge Pullreqest
|
||||
2. Create a GPG signed tag with the name `v0.x.y`
|
||||
3. Merge __dev__ into master on the extractor
|
||||
4. Create a GPG signed tag with the name `v0.x.y` on the extractor
|
||||
5. Create a new release draft and write the down the fix into the release note
|
||||
6. Copy the [release note](#release_notes) into the fastlane directory of releases
|
||||
7. Increment the __small minor__ version number and the `versionCode`
|
||||
8. Hit `Publish Release`
|
||||
9. Rebase the hotfix back into __dev__ branch
|
||||
1. Click "Merge Pull Request"
|
||||
2. Create a GPG signed tag with the name `v0.x.y`.
|
||||
3. Merge __dev__ into master on the extractor.
|
||||
4. Create a GPG signed tag with the name `v0.x.y` on the extractor.
|
||||
5. Create a new release draft and write the down the fix into the release notes.
|
||||
6. Copy the [release note](#release-notes) into the fastlane directory of releases.
|
||||
7. Increment the __small minor__ version number and the `versionCode`.
|
||||
8. Click "Publish Release".
|
||||
9. Rebase the hotfix back into __dev__ branch.
|
||||
|
||||
|
||||
|
||||
## Versioning
|
||||
## Version Nomenclature
|
||||
|
||||
Versioning NewPipe is simple.
|
||||
The version nomenclature of NewPipe is simple.
|
||||
|
||||
- __Major__: The __major__ version number (the number before the first dot) was 0 for years. The reason for this changed over time. First I wanted this number to
|
||||
switch to 1 once NewPipe was feature complete. Now I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this (2 and beyond) well god knows what happens if we ever reach 1 ¯\\\_(ツ)\_/¯ .
|
||||
- __Major__: The __major__ version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to
|
||||
switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. ¯\\\_(ツ)\_/¯
|
||||
- __Minor__: The __minor__ version number (the number after the first dot) will be incremented if there is a major feature added to the app.
|
||||
- __Small Minor__: The small minor (the number after the second dot) will be incremented if there are just smaller bug fixes or features added to the app.
|
||||
- __Small Minor__: The small minor (the number after the second dot) will be incremented if there are bug fixes or minor features added to the app.
|
||||
|
||||
|
||||
#### Versioning the extractor
|
||||
#### Version Nomenclature of the Extractor
|
||||
|
||||
The extractor is always released together with the app, therefore the version number of the extractor is the same as the one of the app.
|
||||
The extractor is always released together with the app, therefore the version number of the extractor is identical to the one of NewPipe itself.
|
||||
|
||||
#### Version code
|
||||
In android an app can also have a [versionCode](https://developer.android.com/studio/publish/versioning). This code is a `long integer` and can be incremented by any value to show a device that a new version is there.
|
||||
For NewPipe the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps
|
||||
are reserved for our internal fdroid build server.
|
||||
#### Version Code
|
||||
In Android, an app can also have a [versionCode](https://developer.android.com/studio/publish/versioning). This code is a `long integer` and can be incremented by any value to show a device that a new version is there.
|
||||
For NewPipe, the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps
|
||||
are reserved for our internal F-Droid build server.
|
||||
|
||||
## Release notes
|
||||
Release notes should give the user an idea of what was changed on the app. The release nodes for NewPipe are stored in the [github draft for a new release](https://github.com/TeamNewPipe/NewPipe/releases/tag/v0.14.0). When a maintainer wants to add change to the release note, but there is no draft for a new version he should create one.
|
||||
## Release Notes
|
||||
Release notes should tell what was changed in the new version of the app. The release nodes for NewPipe are stored in the [GitHub draft for a new release](https://github.com/TeamNewPipe/NewPipe/releases/tag/v0.14.0). When a maintainer wants to add changes to the release note, but there is no draft for a new version, they should create one.
|
||||
|
||||
Changes can be categorized into three types.
|
||||
Changes can be categorized into three types:
|
||||
|
||||
- __New__: New features that god added to the app.
|
||||
- __Improved__: Improvements to the app, or already existing features
|
||||
- __Improved__: Improvements to the app or existing features
|
||||
- __Fixes__: Bugfixes
|
||||
|
||||
When releasing a new version of NewPipe, before actually hitting release the maintainer should copy the release note from the draft and put it into a file called
|
||||
`<versionCode>.txt` (whereas `<versionCode>` needs to be the version code of the comming release). This file must be stored in the direcotry [`/fastlane/metadata/android/en-US/changelogs`](https://github.com/teamnewpipe/newpipe/tree/dev/fastlane/metadata/android/en-US/changelogs). This way fdroid will later be able to show the
|
||||
changes done on the app.
|
||||
When releasing a new version of NewPipe, before actually clicking "Release", the maintainer should copy the release notes from the draft and put it into a file called
|
||||
`<versionCode>.txt` (whereas `<versionCode>` needs to be the version code of the incoming release). This file must be stored in the directory [`/fastlane/metadata/android/en-US/changelogs`](https://github.com/teamnewpipe/newpipe/tree/dev/fastlane/metadata/android/en-US/changelogs). This way, F-Droid will be able to show the
|
||||
changes done to the app.
|
||||
|
|
|
|||
|
|
@ -1,33 +1,33 @@
|
|||
# Documentation
|
||||
# About This Documentation
|
||||
|
||||
|
||||
The Documentation you are currently reading is written using [mkdocs](https://www.mkdocs.org/). It is a tool that will generate a static website based on [markdown](https://www.markdownguide.org/) files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like html or latex.
|
||||
The documentation you are currently reading was written using [mkdocs](https://www.mkdocs.org/). It is a tool that will generate a static website based on [markdown](https://www.markdownguide.org/) files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like HTML or LaTeX.
|
||||
|
||||
## Installation
|
||||
|
||||
Mkdos is written in [python](https://www.python.org/) and is distributed through the python internal package manager [pip](https://pypi.org/project/pip/), thus you need to get python and pip running on your operating system first.
|
||||
Mkdocs is written in [Python](https://www.python.org/) and is distributed through the Python internal package manager [pip](https://pypi.org/project/pip/), thus you need to get python and pip running on your operating system first.
|
||||
|
||||
## Windows
|
||||
|
||||
1. Download the latest [Python3](https://www.python.org/downloads/windows/).
|
||||
2. When running the setup program make sure to check `Add Python 3.x to PATH`.
|
||||
1. Download the latest [Python3](https://www.python.org/downloads/windows/) version.
|
||||
2. When running the setup program, make sure to tick, "Add Python 3.x to PATH".
|
||||
|
||||
3. Install python
|
||||
4. Open powershell or cmd, and type: `pip3 install mkdocs`.
|
||||
3. Install Python.
|
||||
4. Open PowerShell or cmd.exe and type: `pip3 install mkdocs`.
|
||||
|
||||
## MacOS
|
||||
|
||||
MacOS already comes along with python, however pip is still missing. The easyest, and most nondistructive way is to install the MacOS package manager [homebrew](https://brew.sh/index_de) first. The advantage of homebrew are that it will only modify your home directory, but not the root dir. So your OS will not be touched by this.
|
||||
MacOS already includes Python, however, pip is still missing. The easiest and most nondestructive way is to install the MacOS package manager, [homebrew](https://brew.sh/index_de), first. The advantage of homebrew is that it will only modify your home directory, and not the root dir, so your OS will not be tampered with.
|
||||
|
||||
1. Install [homebrew](https://brew.sh/index_de)
|
||||
2. Install Python from homebrew, which will also install pip. Run this command:
|
||||
1. Install [homebrew](https://brew.sh/index_de).
|
||||
2. Install Python from homebrew, which will also install pip. Enter this command:
|
||||
`brew install python`.
|
||||
3. Install mkdocs:
|
||||
`pip3 install mkdocs`
|
||||
|
||||
## Linux/*BSD
|
||||
|
||||
Also Linux/*BSD comes pre installed with python. Most distributions also deliver pip by default. If its not installed you may need to figure out how to install pip3 through the package manager of your system.
|
||||
Linux/*BSD also has Python preinstalled. Most distributions also contain pip by default. If it is not installed, you may need to figure out how to install pip3 through the package manager of your system.
|
||||
|
||||
1. Install pip3 with these commands according to distributions:
|
||||
- __Ubuntu/Mint__: `apt install python3-pip`
|
||||
|
|
@ -39,26 +39,26 @@ Also Linux/*BSD comes pre installed with python. Most distributions also deliver
|
|||
or run `sudo pip3 install mkdocs` to install mkdocs systemwide. Last one has the higher chance to work properly.
|
||||
|
||||
## Android/ChromeOS
|
||||
This might sound funny, but according to the growing amount of ChromeBooks and Android tablets with keyboard this might actually be useful.
|
||||
This might sound funny, but according to the growing amount of Chromebooks and Android tablets with keyboards, this might actually be useful.
|
||||
|
||||
1. Install the [Tremux App](https://termux.com/) from [f-droid](https://f-droid.org/packages/com.termux/).
|
||||
1. Install the [Termux App](https://termux.com/) from [f-droid](https://f-droid.org/packages/com.termux/).
|
||||
2. Launch Termux and type `apt update`
|
||||
3. Install python and git with the command: `apt install git python`
|
||||
4. Now install mkdocs with `pip install mkdocs`.
|
||||
3. Install Python and git with the command: `apt install git python`
|
||||
4. Install mkdocs with `pip install mkdocs`.
|
||||
|
||||
From here on everything will be the same as on Desktop. If you want to edit the files you can (besides vim or emacs which are available through Termux) use your favourite text editor on android. This is possible by opening the files with the Termux integration of the build in android file manager:
|
||||
From herein, everything will be the same as on Desktop. If you want to edit the files, you can (besides vim or emacs which are available through Termux) use your preferred text editor on Android. This is possible by opening the files with the Termux integration of the build in android file manager:
|
||||
|
||||
|
||||
|
||||
## Update
|
||||
Sometimes mkdocs changes the way how to serve, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To ensure this simply run `pip3 install --upgrade mkdocs` or `sudo pip3 install --upgrade mkdocs` if you installed pip system wide on a linux/bsd system.
|
||||
## Updating
|
||||
Sometimes, mkdocs changes the way of how it serves, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To check, simply run `pip3 install --upgrade mkdocs` or `sudo pip3 install --upgrade mkdocs` if you installed pip system wide on a Linux/BSD* system.
|
||||
|
||||
## Using mkdocs
|
||||
In order to extend this documentation you have to clone it from its [git repository](https://github.com/TeamNewPipe/documentation). When you cloned it, you will find a [mkdocs.yml](https://github.com/TeamNewPipe/documentation/blob/master/mkdocs.yml) file, and a [docs](https://github.com/TeamNewPipe/documentation/tree/master/docs) directory inside. The yaml file is the [config file](https://www.mkdocs.org/user-guide/configuration/) while in the directory docs the documentation files are stored. [here](https://www.mkdocs.org/user-guide/writing-your-docs/) is a guide about how to use mkdocs.
|
||||
In order to extend this documentation, you have to clone it from its [GitHub repository](https://github.com/TeamNewPipe/documentation). When you clone it, you will find a [mkdocs.yml](https://github.com/TeamNewPipe/documentation/blob/master/mkdocs.yml) file, and a [docs](https://github.com/TeamNewPipe/documentation/tree/master/docs) directory inside. The yaml file is the [config file](https://www.mkdocs.org/user-guide/configuration/) while in the directory docs the documentation files are stored. [Here](https://www.mkdocs.org/user-guide/writing-your-docs/) is a guide about how to use mkdocs.
|
||||
|
||||
## Write and Deploy
|
||||
If you are writing a documentation page, and want a live preview of it you can enter the root directory of this documentation project, and then run `mkdocs serve` this will start the mkdocs internal webserver on port `8000`. So all you have to do is type `localhost:8000` into the addressbar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content.
|
||||
If you are writing a documentation page and want a live preview of it, you can enter the root directory of this documentation project, and then run `mkdocs serve` this will start the mkdocs internal web server on port `8000`. So all you have to do is type `localhost:8000` into the address bar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content.
|
||||
|
||||
If you want to deploy the page, so it will be up to date at the [github pages](https://teamnewpipe.github.io/documentation/) simply type `mkdocs gh-deploy`. However please be aware that this will not push your changes to the `master` branch of the repository. So you still have to commit and push your changes to the actual git repository of this documentation. _Please be aware that only priviliged maintainers can do this._
|
||||
If you want to deploy the page so it will be up to date at the [GitHub pages](https://teamnewpipe.github.io/documentation/), simply type `mkdocs gh-deploy`. However, please be aware that this will not push your changes to the `master` branch of the repository. So, you still have to commit and push your changes to the actual git repository of this documentation. _Please be aware that only privileged maintainers can do this._
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,17 +1,17 @@
|
|||
# Welcome to the NewPipe Documentation.
|
||||
# Welcome to NewPipe.
|
||||
|
||||
<img width=150 src="https://raw.githubusercontent.com/TeamNewPipe/NewPipe/dev/assets/new_pipe_icon_5.png"/>
|
||||
|
||||
|
||||
This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the [NewPipe Extractor](https://github.com/TeamNewPipe/NewPipeExtractor) however it does also contain several notes about how to maintain NewPipe.
|
||||
This site is/should be a beginner friendly tutorial and documentation for people who want to use or write services for the [NewPipe Extractor](https://github.com/TeamNewPipe/NewPipeExtractor). However, it also contains several notes about how to maintain NewPipe.
|
||||
It is an addition to our auto generated [jdoc documentation](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/).
|
||||
|
||||
Please be aware that it is also in an early state, so help and [feedback](https://github.com/TeamNewPipe/documentation/issues) is always welcome :D
|
||||
Please be aware that it is in its early stages, so help and [feedback](https://github.com/TeamNewPipe/documentation/issues) is always appreciated :D
|
||||
|
||||
|
||||
## Introduction
|
||||
|
||||
The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App [NewPipe](https://newpipe.schabi.org) for android, however it's system independent and also available for other platforms as well.
|
||||
The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming app [NewPipe](https://newpipe.schabi.org) for Android. It is entirely independent from said platforms and also available for additional platforms as well.
|
||||
|
||||
The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website.
|
||||
The beauty behind this framework is that it takes care of the extracting process, error handling etc. so you can focus on what is important: Scraping the website.
|
||||
It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.
|
||||
|
|
|
|||
Loading…
Reference in New Issue