be-gay-do-crimes
/
newpipe-documentation
mirror of https://github.com/TeamNewPipe/documentation
1
0
Fork 0
Code Issues Releases Wiki Activity

fix conflict

This commit is contained in:
Christian Schabesberger 2018-09-21 22:39:17 +02:00
parent a9c99bf4c6 ad3276e909
commit 32e3e06524
6 changed files with 35 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
docs/00_Prepare_everything.md
View File

@ -1,7 +1,7 @@
# Prepare everything # Prepare everything
Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor
service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. Let's
dive right in. ;D dive right in. ;D
## Setup your dev environment ## Setup your dev environment
@ -23,14 +23,14 @@ First and foremost you need to meet certain conditions in order to write your ow
- __[java 8](https://www.java.com/en/download/faq/java8.xml)__ - __[java 8](https://www.java.com/en/download/faq/java8.xml)__
- __[gradle](https://gradle.org/)__ - __[gradle](https://gradle.org/)__
- __[unit testing](https://junit.org/junit5/)__ - __[unit testing](https://junit.org/junit5/)__
- I highly recomend [IDEA Community](https://www.jetbrains.com/idea/) since it has everything we need. - I highly recommend [IDEA Community](https://www.jetbrains.com/idea/) since it has everything we need.
- A __[github](https://github.com/)__ account - A __[Github](https://github.com/)__ account
- A loot of patience and excitement ;D - A lot of patience and excitement ;D
After making sure all these conditions are provided fork the [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor), After making sure all these conditions are provided fork the [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor),
using the [fork button](https://github.com/TeamNewPipe/NewPipeExtractor#fork-destination-box). using the [fork button](https://github.com/TeamNewPipe/NewPipeExtractor#fork-destination-box).
This way you have your own working repository. Now clone this repository into your local folder in which you want to work in. This way you have your own working repository. Now clone this repository into your local folder in which you want to work in.
Next import the cloned project into your [ide](https://www.jetbrains.com/help/idea/configuring-projects.html#importing-project) Next import the cloned project into your [IDE](https://www.jetbrains.com/help/idea/configuring-projects.html#importing-project)
and [run](https://www.jetbrains.com/help/idea/performing-tests.html) it. and [run](https://www.jetbrains.com/help/idea/performing-tests.html) it.
If all the checks are green you did everything right, and you are good to go to move on to the next chapter. If all the checks are green you did everything right, and you are good to go to move on to the next chapter.

6
docs/01_Concept_of_the_extractor.md
View File

@ -5,7 +5,7 @@
Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern
you will find all over the code. It is called the __extractor/collector__ pattern. The idea behind it is that you will find all over the code. It is called the __extractor/collector__ pattern. The idea behind it is that
the [extractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html) the [extractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html)
would produce single peaces of data, and the collector would take it and form usable data for the front end out of it. would produce single pieces of data, and the collector would take it and form usable data for the front end out of it.
The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any
point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of
many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe. many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe.
@ -99,7 +99,7 @@ When a website shows a long list of items it usually does not load the whole lis
This is why a list in NewPipe lists are chopped down into smaller lists called [InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html)s. Each page has its own URL, and needs to be extracted separately. This is why a list in NewPipe lists are chopped down into smaller lists called [InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html)s. Each page has its own URL, and needs to be extracted separately.
Additional metainformation about the list such as it's title a thumbnail Additional metainformation about the list such as its title a thumbnail
or its creator, and extracting multiple pages can be handled by a or its creator, and extracting multiple pages can be handled by a
[ListExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html), [ListExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html),
and it's [ListExtractor.InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html). and it's [ListExtractor.InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html).
@ -115,7 +115,7 @@ such as:
returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the `getNextPageUrl()` method of the previous page. returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the `getNextPageUrl()` method of the previous page.
The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of The reason why the first page is handled special is because many Websites such as YouTube will load the first page of
items like a regular webpage, but all the others as AJAX request. items like a regular webpage, but all the others as AJAX request.

18
docs/02_Concept_of_LinkHandler.md
View File

@ -2,12 +2,12 @@
[LinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandler.html) [LinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandler.html)
represent Links to resources like videos, search requests, channels, etc. represent Links to resources like videos, search requests, channels, etc.
The idea behind them is that a video can have multiple links pointig to it, but it has The idea behind them is that a video can have multiple links pointing to it, but it has
one unique id that represents it, like this example: one unique ID that represents it, like this example:
[oHg5SJYRHA0](https://www.youtube.com/watch?v=oHg5SJYRHA0) can be represented as: [oHg5SJYRHA0](https://www.youtube.com/watch?v=oHg5SJYRHA0) can be represented as:
- https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube) - https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube)
- https://youtu.be/oHg5SJYRHA0 - https://youtu.be/oHg5SJYRHA0
- https://m.youtube.com/watch?v=oHg5SJYRHA0 - https://m.youtube.com/watch?v=oHg5SJYRHA0
@ -16,12 +16,12 @@ one unique id that represents it, like this example:
- `LinkHandler` are ReadOnly - `LinkHandler` are ReadOnly
- LinkHandler are also used to determine which part of the extractor can handle a certain link. - LinkHandler are also used to determine which part of the extractor can handle a certain link.
- In order to get one you must either call - In order to get one you must either call
[fromUrl()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromUrl-java.lang.String-) or [fromId()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromId-java.lang.String-) of the the coresponding `LinkHandlerFactory`. [fromUrl()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromUrl-java.lang.String-) or [fromId()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromId-java.lang.String-) of the the corresponding `LinkHandlerFactory`.
- Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc. - Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.
### Usage ### Usage
So the typical Usage for getting a LinkHandler would look like this. So the typical usage for getting a LinkHandler would look like this.
```java ```java
LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory(); LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory();
LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_video"); LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_video");
@ -29,19 +29,19 @@ LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_v
### Implementation ### Implementation
In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg: In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg:
```java ```java
class MyStreamLinkHandlerFactory extends LinkHandlerFactory { class MyStreamLinkHandlerFactory extends LinkHandlerFactory {
@Override @Override
public String getId(String url) throws ParsingException { public String getId(String url) throws ParsingException {
// Return the ID based on the url. // Return the ID based on the URL.
} }
@Override @Override
public String getUrl(String id) throws ParsingException { public String getUrl(String id) throws ParsingException {
// Return the url based on the id given. // Return the URL based on the ID given.
} }
@Override @Override
@ -70,7 +70,7 @@ Through these you can tell the front end which kind of filter your service suppo
#### SearchQueryHandler #### SearchQueryHandler
You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the
same search request might have a changing outcome deepening on the country or the time you send the request. This is same search request might have a changing outcome deepening on the country or the time you send the request. This is
why the idea of an "ID" is replaced by a "SearchString" in the [SearchQueryHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandler.html) why the idea of an "ID" is replaced by a "SearchString" in the [SearchQueryHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandler.html)
These work like regular ListLinkHandler, accept that you don't have to implement the methodes `onAcceptUrl()` These work like regular ListLinkHandler, accept that you don't have to implement the methodes `onAcceptUrl()`

20
docs/03_Implement_a_service.md
View File

@ -1,11 +1,20 @@
# Implement a service # Implement a service
Services or better service connectors are the parts of NewPipe which communicative with an actual service like YouTube. Services or better service connectors are the parts of NewPipe which communicative with an actual service like YouTube.
This Page will describe how you can implement and add your own. Please make sure you red and understand the This Page will describe how you can implement and add your own. Please make sure you read and understand the
[Concept of Extractors](https://teamnewpipe.github.io/documentation/01_Concept_of_the_extractor/) [Concept of Extractors](https://teamnewpipe.github.io/documentation/01_Concept_of_the_extractor/)
and the [Concept of LinkHandler](https://teamnewpipe.github.io/documentation/02_Concept_of_LinkHandler/) and the [Concept of LinkHandler](https://teamnewpipe.github.io/documentation/02_Concept_of_LinkHandler/)
before implementing your own Service. before implementing your own Service.
__The parts of a service:__
- [Head of Service](#head-of-service)
- [Stream](#stream)
- [Search](#search)
- [Channel](#channel) _(optional)_
- [Playlist](#playlist) _(optional)_
- [Kiosk](#kiosk) _(optional)_
### Allowed Libraries ### Allowed Libraries
The NewPipe Extractor already comes a long with a lot of usable tools and external libraries that should make extracting easy. The NewPipe Extractor already comes a long with a lot of usable tools and external libraries that should make extracting easy.
@ -46,15 +55,16 @@ of NewPipe. This way the will become an official part of the NewPipe Extractor.
Every service has an ID, which will be set when this list gets created. You set this Id by entering it in the constructor. Every service has an ID, which will be set when this list gets created. You set this Id by entering it in the constructor.
So when adding your service just give it the ID of the previously last service in the list incremented by one. So when adding your service just give it the ID of the previously last service in the list incremented by one.
### Search
- [SearchExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/search/SearchExtractor.html)
- [SearchQueryHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandlerFactory.html)
### Stream ### Stream
- [StreamExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamExtractor.html) - [StreamExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamExtractor.html)
- [StreamInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamInfoItemExtractor.html) - [StreamInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamInfoItemExtractor.html)
- [LinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html) - [LinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html)
### Search
- [SearchExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/search/SearchExtractor.html)
- [SearchQueryHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandlerFactory.html)
- [SuggestionExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/SuggestionExtractor.html) _(optional)_
### Channel ### Channel
- [ChannelExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html) - [ChannelExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html)
- [ChannelInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html) - [ChannelInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html)

2
docs/04_Run_changes_in_App.md
View File

@ -28,7 +28,7 @@ marven *.jar packages for android and java based on a github or gitlab repositor
To the extractor through jitpack, you need to push them to your online repository of To the extractor through jitpack, you need to push them to your online repository of
your copy that you host either on [github](https://github.com) or [gitlab](https://gitlab.com). It's important to host your copy that you host either on [github](https://github.com) or [gitlab](https://gitlab.com). It's important to host
it on one of both. Now copy your repository url in Http format, go to [jitpack](https://jitpack.io/), and past it there it on one of both. Now copy your repository url in Http format, go to [jitpack](https://jitpack.io/), and past it there
From here you can grap the latest commit via `GET IT` button. From here you can grab the latest commit via `GET IT` button.
I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An "implementation" string will be generated I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An "implementation" string will be generated
for you. Copy this string and replace the `implementation 'com.github.TeamNewPipe:NewPipeExtractor:<commit>'` line in for you. Copy this string and replace the `implementation 'com.github.TeamNewPipe:NewPipeExtractor:<commit>'` line in
the file [/app/build.gradle](https://github.com/TeamNewPipe/NewPipe/blob/dev/app/build.gradle#L58) with it. the file [/app/build.gradle](https://github.com/TeamNewPipe/NewPipe/blob/dev/app/build.gradle#L58) with it.

4
docs/index.md
View File

@ -1,4 +1,4 @@
# Welcome to the NewPipe Documentaiton. # Welcome to the NewPipe Documentation.
This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the [NewPipe Extractor](https://github.com/TeamNewPipe/NewPipeExtractor). This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the [NewPipe Extractor](https://github.com/TeamNewPipe/NewPipeExtractor).
It is an addition to our auto generated [jdoc documentation](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/). It is an addition to our auto generated [jdoc documentation](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/).
@ -8,7 +8,7 @@ Please be aware that it is also in an early state, so help and [feedback](https:
## Introduction ## Introduction
The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App [NewPipe](https://newpipe.schabi.org) for android, however it's system independent and also available for other platforms. The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular YouTube and streaming App [NewPipe](https://newpipe.schabi.org) for android, however it's system independent and also available for other platforms.
The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website. The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website.
It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code. It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.