be-gay-do-crimes
/
newpipe-documentation
mirror of https://github.com/TeamNewPipe/documentation
1
0
Fork 0
Code Issues Releases Wiki Activity

Merge pull request #1 from snappyapple632/snappyapple632-patch-1 

Fixed spelling, grammar, structure, etc.
This commit is contained in:
snappyapple632 2019-02-27 22:54:57 -08:00 committed by GitHub
parent c4b5bd6c99 09eea974ca
commit 6ce54e70d9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 755 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

82
00_Prepare_everything.md Normal file
View File

@ -0,0 +1,82 @@
# Before You Start
These documents will guide you through the process of creating your own Extractor
service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube and SoundCloud.
The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our
[Jdoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) setup.
__IMPORTANT!!!__ This is likely to be the worst documentation you have ever read, so do not hesitate to
[report](https://github.com/teamnewpipe/documentation/issues) if
you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community
and are open for everyone to help :)
## Setting Up Your Dev Environment
First and foremost, you need to meet the following conditions in order to write your own service.
### What You Need to Know:
- A basic understanding of __[git](https://try.github.io)__
- Good __[Java](https://whatpixel.com/best-java-books/)__ knowledge
- A good understanding of __[web technology](https://www.w3schools.com/)__
- A basic understanding of __[unit testing](https://www.vogella.com/tutorials/JUnit/article.html)__ and __[JUnit](https://junit.org/)__
- A thorough understanding of how to [contribute](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md#code-contribution) to the __NewPipe project__
### Tools/Programs You Will Need:
- A dev environment/ide that supports:
- __[git](https://git-scm.com/downloads/guis)__
- __[Java 8](https://www.java.com/en/download/faq/java8.xml)__
- __[Gradle](https://gradle.org/)__
- __[Unit testing](https://junit.org/junit5/)__
- [IDEA Community](https://www.jetbrains.com/idea/) (Strongly recommended, but not required)
- A __[Github](https://github.com/)__ account
- A lot of patience and excitement ;D
After making sure all these conditions are provided, fork the [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor)
using the [fork button](https://github.com/TeamNewPipe/NewPipeExtractor#fork-destination-box).
This is so you have a personal repository to develop on. Next, clone this repository into your local folder in which you want to work in.
Then, import the cloned project into your [IDE](https://www.jetbrains.com/help/idea/configuring-projects.html#importing-project)
and [run it.](https://www.jetbrains.com/help/idea/performing-tests.html)
If all the checks are green, you did everything right! You can proceed to the next chapter.
### Importing the NewPipe Extractor in IntelliJ IDEA
If you use IntelliJ IDEA, you should know the easy way of importing the NewPipe extractor. If you don't, here's how to do it:
1. `git clone` the extractor onto your computer locally.
2. Start IntelliJ Idea and click `Import Project`.
3. Select the root directory of the NewPipe Extractor
4. Select "__Import Project form external Model__" and then choose __Gradle__.
![import from gradle image](img/select_gradle.png)
5. In the next window, select "__Use gradle 'wrapper' task configuration__".
![use gradle 'wrapper' task configuration checkbox](img/select_gradle_wrapper.png)
### Running "test" in Android Studio/IntelliJ IDEA
Go to _Run_ > _Edit Configurations_ > _Add New Configuration_ and select "Gradle".
As Gradle Project, select NewPipeExtractor. As a task, add "test". Now save and you should be able to run.
![tests passed on idea](img/prepare_tests_passed.png)
# Inclusion Criteria for Services
After creating you own service, you will need to submit it to our [NewPipeExtractor](https://github.com/teamnewpipe/newpipeextractor)
repository. However, in order to include your changes, you need to follow these rules:
1. Stick to our [Code contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md#code-contribution)
2. Do not send services that present content we [don't allow](#content-that-is-not-permitted) on NewPipe.
3. You must be willing to maintain your service after submission.
4. Be patient and make the requested changes when one of our maintainers rejects your code.
## Content That is Permitted:
- Any content that is not in the [list of prohibited content](#content-that-is-not-permitted).
- Any kind of pornography or NSFW content that does not violate US law.
- Advertising, which may need to be approved beforehand.
## Content That is NOT Permitted:
- Content that is considered NSFL (Not Safe For Life)
- Content that is prohibited by US federal law (Sexualization of minors, etc).
- Copyrighted media, without the consent of the copyright holder/publisher.

121
01_Concept_of_the_extractor.md Normal file
View File

@ -0,0 +1,121 @@
# Concept of the Extractor
## The Collector/Extractor Pattern
Before you start coding your own service, you need to understand the basic concept of the extractor itself. There is a pattern
you will find all over the code, called the __extractor/collector__ pattern. The idea behind it is that
the [extractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html)
would produce fragments of data, and the collector would collect them and assemble that data into a readable format for the front end.
The collector also controls the parsing process, and takes care of error handling. So, if the extractor fails at any
point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of
multiple methods, one method for every data field the collector wants to have. The collectors are provided by NewPipe.
You need to take care of the extractors.
### Usage in the Front End
A typical call for retrieving data from a website would look like this:
``` java
Info info;
try {
// Create a new Extractor with a given context provided as parameter.
Extractor extractor = new Extractor(some_meta_info);
// Retrieves the data form extractor and builds info package.
info = Info.getInfo(extractor);
} catch(Exception e) {
// handle errors when collector decided to break up extraction
}
```
### Typical Implementation of a Single Data Extractor
The typical implementation of a single data extractor, on the other hand, would look like this:
``` java
class MyExtractor extends FutureExtractor {
public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) {
super(requiredInfo, forExtraction);
...
}
@Override
public void fetch() {
// Actually fetch the page data here
}
@Override
public String someDataFiled()
throws ExtractionException { //The exception needs to be thrown if someting failed
// get piece of information and return it
}
... // More datafields
}
```
## Collector/Extractor Pattern for Lists
Information can be represented as a list. In NewPipe, a list is represented by a
[InfoItemsCollector](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItemsCollector.html).
A InfoItemCollector will collect and assemble a list of [InfoItem](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItem.html).
For each item that should be extracted, a new Extractor must be created, and given to the InfoItemCollector via [commit()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItemsCollector.html#commit-E-).
![InfoItemsCollector_objectdiagram.svg](img/InfoItemsCollector_objectdiagram.svg)
If you are implementing a list for your service you need to extend InfoItem containing the extracted information
and implement an [InfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html),
that will return the data of one InfoItem.
A common implementation would look like this:
```
private MyInfoItemCollector collectInfoItemsFromElement(Element e) {
MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId());
for(final Element li : element.children()) {
collector.commit(new InfoItemExtractor() {
@Override
public String getName() throws ParsingException {
...
}
@Override
public String getUrl() throws ParsingException {
...
}
...
}
return collector;
}
```
## InfoItems Encapsulated in Pages
When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail,
and its creator. Such info can be called __list header__.
When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down.
This is why a list in NewPipe lists are chopped down into smaller lists called [InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html)s. Each page has its own URL, and needs to be extracted separately.
Additional metadata about the list and extracting multiple pages can be handled by a
[ListExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html),
and its [ListExtractor.InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html).
For extracting list header information it behaves like a regular extractor. For handling `InfoItemsPages` it adds methods
such as:
- [getInitialPage()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html#getInitialPage--)
which will return the first page of InfoItems.
- [getNextPageUrl()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html#getNextPageUrl--)
If a second Page of InfoItems is available this will return the URL pointing to them.
- [getPage()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html#getPage-java.lang.String-)
returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the `getNextPageUrl()` method of the previous page.
The reason why the first page is handled special is because many Websites such as YouTube will load the first page of
items like a regular web page, but all the others as an AJAX request.

88
02_Concept_of_LinkHandler.md Normal file
View File

@ -0,0 +1,88 @@
# Concept of the LinkHandler
The [LinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandler.html)
represent links to resources like videos, search requests, channels, etc.
The idea is that a video can have multiple links pointing to it, but it has
one unique ID that represents it, like this example:
[oHg5SJYRHA0](https://www.youtube.com/watch?v=oHg5SJYRHA0) can be represented as:
- [https://www.youtube.com/watch?v=oHg5SJYRHA0](https://www.youtube.com/watch?v=oHg5SJYRHA0) (the default URL for YouTube)
- [https://youtu.be/oHg5SJYRHA0](https://youtu.be/oHg5SJYRHA0) (the shortened link)
- [https://m.youtube.com/watch?v=oHg5SJYRHA0](https://m.youtube.com/watch?v=oHg5SJYRHA0) (the link for mobile devices)
### Importand notes about LinkHandler:
- A simple `LinkHandler` will contain the default URL, the ID, and the original URL.
- `LinkHandler`s are read only.
- `LinkHandler`s are also used to determine which part of the extractor can handle a certain link.
- In order to get one you must either call
[fromUrl()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromUrl-java.lang.String-) or [fromId()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromId-java.lang.String-) of the the corresponding `LinkHandlerFactory`.
- Every type of resource has its own `LinkHandlerFactory`. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.
### Usage
The typical usage for obtaining a LinkHandler would look like this:
```java
LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory();
LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_video");
```
### Implementation
In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg:
```java
class MyStreamLinkHandlerFactory extends LinkHandlerFactory {
@Override
public String getId(String url) throws ParsingException {
// Return the ID based on the URL.
}
@Override
public String getUrl(String id) throws ParsingException {
// Return the URL based on the ID given.
}
@Override
public boolean onAcceptUrl(String url) throws ParsingException {
// Return true if this LinkHanlderFactory can handle this type of link
}
}
```
### ListLinkHandler and SearchQueryHandler
List based resources, like channels and playlists, can be sorted and filtered.
Therefore these type of resources don't just use a LinkHandler, but a class called
[ListLinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html),
which inherits from LinkHandler and adds the field [ContentFilter](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html#contentFilters),
which is used to filter by resource type, like stream or playlist, and
[SortFilter](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandler.html#sortFilter),
which is used to sort by name, date, or view count.
__!!ATTENTION!!__ Be careful when you implement a content filter: No selected filter equals all filters selected. If your get an empty content filter list in your extractor, make sure you return everything. By all means, use "if"
statements like `contentFilter.contains("video") || contentFilter.isEmpty()`.
ListLinkHandler are also created by overriding the [ListLinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html)
additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override
[getAvailableContentFilter()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html#getAvailableContentFilter--)
and [getAvailableSortFilter()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html#getAvailableSortFilter--).
Through these you can tell the front end which kind of filter your service supports.
#### SearchQueryHandler
You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the
same search request might have a different outcome depending on the country or the time you send the request. This is
why the idea of an "ID" is replaced by a "SearchString" in the [SearchQueryHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandler.html)
These work like regular ListLinkHandler, except that you don't have to implement the methods `onAcceptUrl()`
and `getId()` when overriding [SearchQueryHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandlerFactory.html).

166
03_Implement_a_service.md Normal file
View File

@ -0,0 +1,166 @@
# Implementing a Service
Services, or better service connectors, are the parts of NewPipe which communicate with an actual service like YouTube.
This page will describe how you can implement and add your own services to the extractor. Please make sure you read and understand the
[Concept of Extractors](https://teamnewpipe.github.io/documentation/01_Concept_of_the_extractor/)
and the [Concept of LinkHandler](https://teamnewpipe.github.io/documentation/02_Concept_of_LinkHandler/)
before continuing.
### Required and Optional Parts
Your service does not have to implement everything; some parts are optional.
This is because not all services support every feature other services support. For example, it might be that a certain
service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding
factory method of the your __StreamingService__ implementation return __null__. The frontend will handle the lack of
having channels.
However, if you start to implement one of the optional parts of the list below, you will have to implement all of its parts/classes. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself.
__The Parts of a Service:__
- [Head of Service](#head-of-service)
- [Stream](#stream)
- [Search](#search)
- [Playlist](#playlist) _(optional)_
- [Channel](#channel) _(optional)_
- [Kiosk](#kiosk) _(optional)_
### Allowed Libraries
The NewPipe Extractor already includes a lot of usable tools and external libraries that should make extracting easy.
For some specific (tiny) tasks, Regex is allowed. Here you can take a look at the
[Parser](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/utils/Parser.html),
which will give you a little help with that. __Use Regex with care!!!__ Avoid it as often as possible. It's better to
ask us to introduce a new library than start using Regex to often.
- Html/XML Parsing: [jsoup](https://jsoup.org/apidocs/overview-summary.html)
- JSON Parsing: [nanojson](https://github.com/mmastrac/nanojson#parser-example)
- JavaScript Parsing/Execution: [Mozilla Rhino](https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Rhino/Documentation)
- Link detection in strings: [AutoLink](https://github.com/robinst/autolink-java)
If you need to introduce new libraries, please tell us before you do so.
### Head of Service
First of all, if you want to create a new service, you should create a new package below `org.schabi.newpipe.services`
, with the name of your service as package name.
__Parts Required to be Implemented:__
- [StreamingService](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.html)
- [ServiceInfo](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.ServiceInfo.html)
[StreamingService](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.html)
is a factory class that will return objects of all important parts of your service.
Every extractor, handler, and info type you add and should be part of your implementation, must be instantiated using an
instance of this class. You can see it as a factory for all objects of your implementation.
[ServiceInfo](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.ServiceInfo.html)
will return some metadata about your service such as the name, capabilities, the author's name, and their
email address for further notice and maintenance issues. Remember, after extending this class, you need to return an
instance of it by through your implementation of
[`StreamingService.getServiceInfo()`](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.html#getServiceInfo--).
When these two classes are extended by you, you need to add your `StreamingService` to the
[ServiceList](https://github.com/TeamNewPipe/NewPipeExtractor/blob/49c2eb51859a58e4bb5ead2d9d0771408f7d59d6/extractor/src/main/java/org/schabi/newpipe/extractor/ServiceList.java#L23)
of NewPipe. This way, your service will become an official part of the NewPipe Extractor.
Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor.
So when adding your service just give it the ID of the previously last service in the list incremented by one.
### Stream
Streams are considered single entities of video or audio. They have metadata like a title, a description,
next/related videos, a thumbnail and comments. To obtain the URL to the actual stream data, as well as its metadata,
StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will
extract one item in a list of items representing such streams, like a search result or a playlist.
Since every streaming service (obviously) provides streams, this is required to implement. Otherwise, your service was
pretty useless :)
__Parts Required to be Implemented:__
- [StreamExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamExtractor.html)
- [StreamInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamInfoItemExtractor.html)
- [LinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html)
### Search
The SearchExtractor is also required to be implemented. It will take a search query represented as
[SearchQueryHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandler.html)
and return a list of search results. Since many services support suggestions as you type, you will also want to implement
a __SuggestionExtractor__. This will make it possible for the frontend to also display a suggestion while typing.
__Parts Required to be Implemented:__
- [SearchExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/search/SearchExtractor.html)
- [SearchQueryHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandlerFactory.html)
- [SuggestionExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/SuggestionExtractor.html) _(optional)_
### Playlist
Playlists are lists of streams provided by the service (you might not have to be concerned over locally saved playlists, those will be handled by the frontend).
A playlist may only contain __StreamInfoItems__, but no other __InfoItem__ types.
__Parts Required to be Implemented:__
- [PlaylistExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/playlist/PlaylistExtractor.html)
- [PlayListInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/playlist/PlaylistExtractor.html)
- [ListLinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html)
### Channel
A Channel is mostly a [Playlist](#playlist), the only difference is that it does not only represent a simple list of streams, but also a
user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the ChannelExtractor
differs from the one of a playlist.
__Parts Required to be Implemented:__
- [ChannelExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html)
- [ChannelInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html)
- [ListLinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html)
### Kiosk
A kiosk is a list of __InfoItems__ which will be displayed on the main page of NewPipe. A kiosk is mostly similar to the content
displayed on the main page of a video platform. A kiosk could be something like "Top 20", "Charts", "News", "Creators Selection" etc.
Kiosks are controversial; many people may not like them. If you also don't like them, please consider your users and refrain from denying support for them.
Your service would look pretty empty if you select it and no video is being displayed. Also, you should not override the preference of the user,
since users of NewPipe can decide by the settings whether they want to see the kiosk page or not.
#### Multiple Kiosks
Most services will implement more than one kiosk, so a service might have a "Top 20" for different categories like "Country Music", "Techno", etc.
This is why the extractor will let you implement multiple __KioskExtractors__. Since different kiosk pages might also differ
with their HTML structure, every page you want to support has to be implemented as its own __KioskExtractor__.
However, if the pages are similar, you can use the same implementation, but set the page type when you instantiate your __KioskExtractor__
through the __KioskList.KioskExtractorFactory__.
Every kiosk you implement needs to be added to your __KioskList__ which you return with your
__StreamingService__ implementation.
It is also important to set the default kiosk. This will be the kiosk that will be shown by the first start of your service.
An example implementation of the
[getKioskList()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/StreamingService.html)
could look like this:
```java
@Override
public KioskList getKioskList() throws ExtractionException {
KioskList list = new KioskList(getServiceId());
list.addKioskEntry(new KioskList.KioskExtractorFactory() {
@Override
public KioskExtractor createNewKiosk(StreamingService streamingService,
String url,
String id,
Localization local)
throws ExtractionException {
return new YoutubeTrendingExtractor(YoutubeService.this,
new YoutubeTrendingLinkHandlerFactory().fromUrl(url), id, local);
}
}, new YoutubeTrendingLinkHandlerFactory(), "Trending");
list.setDefaultKiosk("Trending");
return list;
}
```
__Parts Required to be Implemented:__
- [KioskList.KioskExtractorFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/kiosk/KioskList.KioskExtractorFactory.html)
- [KioskExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/kiosk/KioskExtractor.html)
- [ListLinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html)

50
04_Run_changes_in_App.md Normal file
View File

@ -0,0 +1,50 @@
# Testing Your Changes in the App
You should develop and test your changes with the JUnit environment that is
provided by the NewPipe Extractor and IDEA. If you want to try it with
the actual fronted, you need to follow these steps.
### Setup Android Studio
First, you'll want to set up a working Android Studio environment. To do this,
download Studio from [developer.android.com](https://developer.android.com/studio/),
and follow the [instructions](https://developer.android.com/studio/install) on how to set it up.
### Get the NewPipe Code and Run it.
In order to get it, you simply clone or download it from the current `dev` branch
[github.com/TeamNewPipe/NewPipe.git](https://github.com/TeamNewPipe/NewPipe/archive/dev.zip).
You can then build and run it following [these instructions](https://developer.android.com/studio/run/).
Also, make sure you are comfortable with [adb](https://en.droidwiki.org/wiki/Android_Debug_Bridge) since
you might experience some trouble running your compiled app on a real device, especially under Linux, where you
sometimes have to adjust the udev rules in order to
[make your device accessible](https://www.janosgyerik.com/adding-udev-rules-for-usb-debugging-android-devices/).
### Run Your Changes on the Extractor
In order to use the extractor in our app, we use [jitpack](https://jitpack.io). This is a build service that can build
maven *.jar packages for Android and Java based on GitHub or GitLab repositories.
To use the extractor through jitpack, you need to push it to your online repository of
your copy that you host either on [GitHub](https://github.com) or [GitLab](https://gitlab.com). It's important to host
it on one of both. To copy your repository URL in HTTP format, go to [jitpack](https://jitpack.io/) and paste it there.
From here, you can grab the latest commit via `GET IT` button.
I recomend not to use a SNAPSHOT, since I am not sure when snapshot is built. An "implementation" string will be generated
for you. Copy this string and replace the `implementation 'com.github.TeamNewPipe:NewPipeExtractor:<commit>'` line in
the file [/app/build.gradle](https://github.com/TeamNewPipe/NewPipe/blob/dev/app/build.gradle#L58) with it.
<video width="600" controls>
<source src="../media/how_to_jitpack.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
If everything synced well, then you should only see a screen with OK signs. Now you can compile and run NewPipe
with the new extractor.
![image_sync_ok](img/sync_ok.png)
### Troubleshooting
If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and
click on that little paper symbol next to the `GET IT` button. If it's red, it means that the build failed.
![jitpack failed to build](img/jitpack_fail.png)

167
05_releasing.md Normal file
View File

@ -0,0 +1,167 @@
# Releasing a New NewPipe Version
This site is meant for those who want to maintain NewPipe, or just want to know how releasing works.
![one does not simply push to master](img/onedoes.jpg)
## Differences Between Regular and Hotfix Releases
NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website,
this however has the disadvantage of the app to break instantly when YouTube changes something.
We do not know when this happen. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as
much as possible. The entire release cycle is therefore designed around this issue.
There is a difference between a release that introduces new features
and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown).
Lets have a look at the characteristics of a __regular release__, and then the characteristics of a __hotfix release__.
## Regular Releases
Regular releases are normal releases like they are done in any other app. Releases are always stored on __master__ branch. The latest commit on
__master__ is always equal to the currently released version. No development is done on master. This ensures that we always have one
branch with a stable/releasable version.
### Feature Branching
When developing, the __dev__ branch is used. Pushing to __dev__ directly, however, is not allowed, since QA and testing should be done first before adding something to it.
This ensures that the dev version works as stable a possible.
In order to change something on the app, one may want to __fork__ the dev branch and develop the changes in their own branch (this is called feature branching).
![feature_branching](img/feature_branch.svg)
Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other.
If a change is done on the API to the extractor, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible
again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes
on the fronted at any time.
### Merging Features/Bugfixes
After finishing a feature, one should open up a __Pull Reuqest__ to the dev branch. From here, a maintainer can do __Code review__ and __Quality Assurance (QA)__.
If you are a maintainer, please take care about the code architecture so __corrosion__ or __code shifting__ can be prevented. Please also prioritize code quality over functionality.
In short: cool function but bad code = no merge. Focus on leaving the code as clean as possible.
![merge_feature_into_dev](img/merge_into_dev.svg)
You, as a maintainer, should build the app and put the signed APK into the description of that new pull request. This way, other people can test the feature/bugfix and help with QA. _You may not need to do this every time. It is enough to do it on bigger pull requests._
After the maintainer merges the new feature into the dev branch, he should add the title of the pull request or a summary of the changes into the [release notes](#release-notes).
### Creating a New Release
Once there are enough features together, and the maintainers believe that NewPipe is ready for a new release, they should create a new release.
Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: __Don't do a release if you don't have time for it!!!__
Below is a list of things you will want to do:
1. Fork the __dev__ branch into a new __release_x.y.z__ branch.
2. Increase the [version number](#versioning)
3. Merge [weblate](https://hosted.weblate.org/projects/newpipe/) changes from the `dev` branch at `https://hosted.weblate.org/git/newpipe/strings/`.
4. Copy the [release notes](#release-notes) from the GitHub version draft into the corresponding fastlane file (see [release notes](#release-notes)).
5. Open up a pull request form the new __release_x.y.z__ branch into the __master__ branch.
6. Create an issue pointing to the new pull request. The reason for opening an issue is that from my perception, people read issues more than pull requests. Put the release-note into this pull request.
7. Build a signed release version of NewPipe using schabis signing keys. This is a release candidate (RC). Name the build apk file `NewPipe_<versionNumber>_RC1.apk`.
Zip it and post it to the head of the release issue. This way, others can test the release candidate.
8. Test and QA the new version with the help of others.
9. Leave the PR open for a few days and advertise it to help testing.
While being in release phase no new pull requests must be merged into __dev__ branch.
This procedure does not have to be done for the extractor as extractor will be tested together with the fronted.
### Quickfixes
When issuing a new release, you will most likely encounter bugs that might not have existed in previous versions. These are called __regressions__.
If you find a regression during release phase, you are allowed to push fixes directly into the release branch without having to fork a branch away from it.
All maintainers have to be aware that they might be required to fix regressions, so plan your release at a time when
you are available. Do not introduce new features during the release phase.
When you have pushed a quickfix, you will want to update the __release candidate__ you put into the __issue__ corresponding to the __release pull request__.
Increment the version number in the filename of the release candidate. e.g. `NewPipe_<versionNumber>_RC2.apk` etc. _Don't update the actual version number. :P_
![release_branch](img/release_branch.svg)
### Releasing
Once the glorious day of all days has come, and you fulfill the ceremony of releasing.
After going through the release procedure of [creating a new release](#create_a_new_release) and maybe a few [quickfixes](#quickfixes) on the new release,
this is what you should do when releasing:
1. Click "Merge Pull Request".
2. Create a GPG signed tag with the name `v0.x.y`.
3. Merge __dev__ into master on the extractor.
4. Create a GPG signed tag with the name `v0.x.y` on the extractor.
5. Make sure the draft name equals the tag name. ![draft_name](img/draft_name.png)
6. Make sure to not have forgotten enything
7. Click "Publish Release".
8. Rebase quickfix changes back into __dev__ if quickfixes were made.
![rebase_back](img/rebase_back_release.svg)
## Hotfix Releases
![this_is_fine](img/could_not_decrypt.png)
As aforementioned, NewPipe is a web crawler and could break at any moment. In order to keep the downtime of NewPipe as low as possible, when such a shutdown happens,
we allow __hotfixes__.
- A hotfix allows work on the master branch instead of the dev branch.
- A hotfix MUST __NOT__ contain any features or unrelated bugfixes.
- A hotfix may only focus on fixing what caused the shutdown.
### Hotfix Branch
Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch.
Of course you are not allowed to push to master directly so you will have to open up a __hotfix__ branch. _If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well._
![hotfix_branch](img/hotfix_branch.svg)
### Releasing
If you fixed the issue and found it to be tested and reviewed well enough, you may release it. You don't need to undergo the full release procedure of a regular release, which takes more time to release.
Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix.
It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better then a non-functional version ¯\\\_(ツ)\_/¯.
Here's what you do when releasing a hotfix:
1. Click "Merge Pull Request"
2. Create a GPG signed tag with the name `v0.x.y`.
3. Merge __dev__ into master on the extractor.
4. Create a GPG signed tag with the name `v0.x.y` on the extractor.
5. Create a new release draft and write the down the fix into the release notes.
6. Copy the [release note](#release_notes) into the fastlane directory of releases.
7. Increment the __small minor__ version number and the `versionCode`.
8. Click "Publish Release".
9. Rebase the hotfix back into __dev__ branch.
![rebase_back_hotfix](img/rebase_back_hotfix.svg)
## Version Nomenclature
The version nomenclature of NewPipe is simple.
- __Major__: The __major__ version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to
switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. ¯\\\_(ツ)\_/¯
- __Minor__: The __minor__ version number (the number after the first dot) will be incremented if there is a major feature added to the app.
- __Small Minor__: The small minor (the number after the second dot) will be incremented if there are bug fixes or minor features added to the app.
#### Version Nomenclature of the Extractor
The extractor is always released together with the app, therefore the version number of the extractor is identical to the one of NewPipe itself.
#### Version Code
In Android, an app can also have a [versionCode](https://developer.android.com/studio/publish/versioning). This code is a `long integer` and can be incremented by any value to show a device that a new version is there.
For NewPipe, the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps
are reserved for our internal F-Droid build server.
## Release Notes
Release notes should tell what was changed in the new version of the app. The release nodes for NewPipe are stored in the [GitHub draft for a new release](https://github.com/TeamNewPipe/NewPipe/releases/tag/v0.14.0). When a maintainer wants to add changes to the release note, but there is no draft for a new version, they should create one.
Changes can be categorized into three types:
- __New__: New features that god added to the app.
- __Improved__: Improvements to the app or existing features
- __Fixes__: Bugfixes
When releasing a new version of NewPipe, before actually clicking "Release", the maintainer should copy the release notes from the draft and put it into a file called
`<versionCode>.txt` (whereas `<versionCode>` needs to be the version code of the incoming release). This file must be stored in the directory [`/fastlane/metadata/android/en-US/changelogs`](https://github.com/teamnewpipe/newpipe/tree/dev/fastlane/metadata/android/en-US/changelogs). This way, F-Droid will be able to show the
changes done to the app.

64
06_documentation.md Normal file
View File

@ -0,0 +1,64 @@
# About This Documentation
The documentation you are currently reading was written using [mkdocs](https://www.mkdocs.org/). It is a tool that will generate a static website based on [markdown](https://www.markdownguide.org/) files. Markdown has the advantage that it is simple to read and write, and that there are several tools that can translate a markdown file into languages like HTML or LaTeX.
## Installation
Mkdocs is written in [Python](https://www.python.org/) and is distributed through the Python internal package manager [pip](https://pypi.org/project/pip/), thus you need to get python and pip running on your operating system first.
## Windows
1. Download the latest [Python3](https://www.python.org/downloads/windows/) version.
2. When running the setup program, make sure to tick, "Add Python 3.x to PATH".
![check_path](img/check_path.png)
3. Install Python.
4. Open PowerShell or cmd.exe and type: `pip3 install mkdocs`.
## MacOS
MacOS already includes Python, however, pip is still missing. The easiest and most nondestructive way is to install the MacOS package manager, [homebrew](https://brew.sh/index_de), first. The advantage of homebrew is that it will only modify your home directory, and not the root dir, so your OS will not be tampered with.
1. Install [homebrew](https://brew.sh/index_de).
2. Install Python from homebrew, which will also install pip. Enter this command:
`brew install python`.
3. Install mkdocs:
`pip3 install mkdocs`
## Linux/*BSD
Linux/*BSD also has Python preinstalled. Most distributions also contain pip by default. If it is not installed, you may need to figure out how to install pip3 through the package manager of your system.
1. Install pip3 with these commands according to distributions:
- __Ubuntu/Mint__: `apt install python3-pip`
- __Fedora/CentOS__: `sudo dnf install python3-pip`
- __Arch/Manjaro__: `sudo pacman -S python-pip`
- __openSuse__: `sudo zypper install python-pip`
- __*BSD__: You are already advanced enough to know how you can force the bits on your disk to become pip by meditating upon it.
2. Run `pip3 install mkdocs` to install mkdocs only for the current user,
or run `sudo pip3 install mkdocs` to install mkdocs systemwide. Last one has the higher chance to work properly.
## Android/ChromeOS
This might sound funny, but according to the growing amount of Chromebooks and Android tablets with keyboards, this might actually be useful.
1. Install the [Termux App](https://termux.com/) from [f-droid](https://f-droid.org/packages/com.termux/).
2. Launch Termux and type `apt update`
3. Install Python and git with the command: `apt install git python`
4. Install mkdocs with `pip install mkdocs`.
From herein, everything will be the same as on Desktop. If you want to edit the files, you can (besides vim or emacs which are available through Termux) use your preferred text editor on Android. This is possible by opening the files with the Termux integration of the build in android file manager:
![termux_files](img/termux_files.png)
## Updating
Sometimes, mkdocs changes the way of how it serves, or the syntax will differ. This is why you should make sure to always run the latest version of mkdocs. To check. simply run `pip3 install --upgrade mkdocs` or `sudo pip3 install --upgrade mkdocs` if you installed pip system wide on a Linux/BSD* system.
## Using mkdocs
In order to extend this documentation, you have to clone it from its [GitHub repository](https://github.com/TeamNewPipe/documentation). When you clone it, you will find a [mkdocs.yml](https://github.com/TeamNewPipe/documentation/blob/master/mkdocs.yml) file, and a [docs](https://github.com/TeamNewPipe/documentation/tree/master/docs) directory inside. The yaml file is the [config file](https://www.mkdocs.org/user-guide/configuration/) while in the directory docs the documentation files are stored. [here](https://www.mkdocs.org/user-guide/writing-your-docs/) is a guide about how to use mkdocs.
## Write and Deploy
If you are writing a documentation page and want a live preview of it, you can enter the root directory of this documentation project, and then run `mkdocs serve` this will start the mkdocs internal web server on port `8000`. So all you have to do is type `localhost:8000` into the address bar of your browser, and here you go. If you modify a file, and save it, mkdocs will reload the page and show you the new content.
If you want to deploy the page so it will be up to date at the [GitHub pages](https://teamnewpipe.github.io/documentation/), simply type `mkdocs gh-deploy`. However, please be aware that this will not push your changes to the `master` branch of the repository. So, you still have to commit and push your changes to the actual git repository of this documentation. _Please be aware that only privileged maintainers can do this._

17
index.md Normal file
View File

@ -0,0 +1,17 @@
# Welcome to NewPipe.
<img width=150 src="https://raw.githubusercontent.com/TeamNewPipe/NewPipe/dev/assets/new_pipe_icon_5.png"/>
This site is/should be a beginner friendly tutorial and documentation for people who want to use or write services for the [NewPipe Extractor](https://github.com/TeamNewPipe/NewPipeExtractor). However, it also contains several notes about how to maintain NewPipe.
It is an addition to our auto generated [jdoc documentation](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/).
Please be aware that it is in its early stages, so help and [feedback](https://github.com/TeamNewPipe/documentation/issues) is always appreciated :D
## Introduction
The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming app [NewPipe](https://newpipe.schabi.org) for Android. It is entirely independent from said platforms and also available for additional platforms as well.
The beauty behind this framework is that it takes care of the extracting process, error handling etc. so you can focus on what is important: Enjoying the content.
It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.