fix conflict
This commit is contained in:
Christian Schabesberger
commit
32e3e06524
|
|
@ -1,7 +1,7 @@
|
|||
# Prepare everything
|
||||
|
||||
Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor
|
||||
service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's
|
||||
service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud. Let's
|
||||
dive right in. ;D
|
||||
|
||||
## Setup your dev environment
|
||||
|
|
@ -23,14 +23,14 @@ First and foremost you need to meet certain conditions in order to write your ow
|
|||
- __[java 8](https://www.java.com/en/download/faq/java8.xml)__
|
||||
- __[gradle](https://gradle.org/)__
|
||||
- __[unit testing](https://junit.org/junit5/)__
|
||||
- I highly recomend [IDEA Community](https://www.jetbrains.com/idea/) since it has everything we need.
|
||||
- A __[github](https://github.com/)__ account
|
||||
- A loot of patience and excitement ;D
|
||||
- I highly recommend [IDEA Community](https://www.jetbrains.com/idea/) since it has everything we need.
|
||||
- A __[Github](https://github.com/)__ account
|
||||
- A lot of patience and excitement ;D
|
||||
|
||||
After making sure all these conditions are provided fork the [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor),
|
||||
using the [fork button](https://github.com/TeamNewPipe/NewPipeExtractor#fork-destination-box).
|
||||
This way you have your own working repository. Now clone this repository into your local folder in which you want to work in.
|
||||
Next import the cloned project into your [ide](https://www.jetbrains.com/help/idea/configuring-projects.html#importing-project)
|
||||
Next import the cloned project into your [IDE](https://www.jetbrains.com/help/idea/configuring-projects.html#importing-project)
|
||||
and [run](https://www.jetbrains.com/help/idea/performing-tests.html) it.
|
||||
If all the checks are green you did everything right, and you are good to go to move on to the next chapter.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern
|
||||
you will find all over the code. It is called the __extractor/collector__ pattern. The idea behind it is that
|
||||
the [extractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html)
|
||||
would produce single peaces of data, and the collector would take it and form usable data for the front end out of it.
|
||||
would produce single pieces of data, and the collector would take it and form usable data for the front end out of it.
|
||||
The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any
|
||||
point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of
|
||||
many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe.
|
||||
|
|
@ -99,7 +99,7 @@ When a website shows a long list of items it usually does not load the whole lis
|
|||
|
||||
This is why a list in NewPipe lists are chopped down into smaller lists called [InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html)s. Each page has its own URL, and needs to be extracted separately.
|
||||
|
||||
Additional metainformation about the list such as it's title a thumbnail
|
||||
Additional metainformation about the list such as its title a thumbnail
|
||||
or its creator, and extracting multiple pages can be handled by a
|
||||
[ListExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html),
|
||||
and it's [ListExtractor.InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html).
|
||||
|
|
@ -115,7 +115,7 @@ such as:
|
|||
returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the `getNextPageUrl()` method of the previous page.
|
||||
|
||||
|
||||
The reason why the first page is handled speciall is because many Websites such as Youtube will load the first page of
|
||||
The reason why the first page is handled special is because many Websites such as YouTube will load the first page of
|
||||
items like a regular webpage, but all the others as AJAX request.
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -2,12 +2,12 @@
|
|||
|
||||
[LinkHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandler.html)
|
||||
represent Links to resources like videos, search requests, channels, etc.
|
||||
The idea behind them is that a video can have multiple links pointig to it, but it has
|
||||
one unique id that represents it, like this example:
|
||||
The idea behind them is that a video can have multiple links pointing to it, but it has
|
||||
one unique ID that represents it, like this example:
|
||||
|
||||
[oHg5SJYRHA0](https://www.youtube.com/watch?v=oHg5SJYRHA0) can be represented as:
|
||||
|
||||
- https://www.youtube.com/watch?v=oHg5SJYRHA0 (default url for youtube)
|
||||
- https://www.youtube.com/watch?v=oHg5SJYRHA0 (default URL for YouTube)
|
||||
- https://youtu.be/oHg5SJYRHA0
|
||||
- https://m.youtube.com/watch?v=oHg5SJYRHA0
|
||||
|
||||
|
|
@ -16,12 +16,12 @@ one unique id that represents it, like this example:
|
|||
- `LinkHandler` are ReadOnly
|
||||
- LinkHandler are also used to determine which part of the extractor can handle a certain link.
|
||||
- In order to get one you must either call
|
||||
[fromUrl()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromUrl-java.lang.String-) or [fromId()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromId-java.lang.String-) of the the coresponding `LinkHandlerFactory`.
|
||||
[fromUrl()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromUrl-java.lang.String-) or [fromId()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromId-java.lang.String-) of the the corresponding `LinkHandlerFactory`.
|
||||
- Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.
|
||||
|
||||
### Usage
|
||||
|
||||
So the typical Usage for getting a LinkHandler would look like this.
|
||||
So the typical usage for getting a LinkHandler would look like this.
|
||||
```java
|
||||
LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory();
|
||||
LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_video");
|
||||
|
|
@ -29,19 +29,19 @@ LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_v
|
|||
|
||||
### Implementation
|
||||
|
||||
In order to Use LinkHandler for your service you must override the apropriate LinkHandlerFactory. eg:
|
||||
In order to Use LinkHandler for your service you must override the appropriate LinkHandlerFactory. eg:
|
||||
|
||||
```java
|
||||
class MyStreamLinkHandlerFactory extends LinkHandlerFactory {
|
||||
|
||||
@Override
|
||||
public String getId(String url) throws ParsingException {
|
||||
// Return the ID based on the url.
|
||||
// Return the ID based on the URL.
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getUrl(String id) throws ParsingException {
|
||||
// Return the url based on the id given.
|
||||
// Return the URL based on the ID given.
|
||||
}
|
||||
|
||||
@Override
|
||||
|
|
@ -70,7 +70,7 @@ Through these you can tell the front end which kind of filter your service suppo
|
|||
|
||||
#### SearchQueryHandler
|
||||
|
||||
You can not point to a search request with an ID like you point to a playlist or a channel, simply because one and the
|
||||
You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the
|
||||
same search request might have a changing outcome deepening on the country or the time you send the request. This is
|
||||
why the idea of an "ID" is replaced by a "SearchString" in the [SearchQueryHandler](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandler.html)
|
||||
These work like regular ListLinkHandler, accept that you don't have to implement the methodes `onAcceptUrl()`
|
||||
|
|
|
|||
|
|
@ -1,11 +1,20 @@
|
|||
# Implement a service
|
||||
|
||||
Services or better service connectors are the parts of NewPipe which communicative with an actual service like YouTube.
|
||||
This Page will describe how you can implement and add your own. Please make sure you red and understand the
|
||||
This Page will describe how you can implement and add your own. Please make sure you read and understand the
|
||||
[Concept of Extractors](https://teamnewpipe.github.io/documentation/01_Concept_of_the_extractor/)
|
||||
and the [Concept of LinkHandler](https://teamnewpipe.github.io/documentation/02_Concept_of_LinkHandler/)
|
||||
before implementing your own Service.
|
||||
|
||||
__The parts of a service:__
|
||||
|
||||
- [Head of Service](#head-of-service)
|
||||
- [Stream](#stream)
|
||||
- [Search](#search)
|
||||
- [Channel](#channel) _(optional)_
|
||||
- [Playlist](#playlist) _(optional)_
|
||||
- [Kiosk](#kiosk) _(optional)_
|
||||
|
||||
### Allowed Libraries
|
||||
|
||||
The NewPipe Extractor already comes a long with a lot of usable tools and external libraries that should make extracting easy.
|
||||
|
|
@ -46,15 +55,16 @@ of NewPipe. This way the will become an official part of the NewPipe Extractor.
|
|||
Every service has an ID, which will be set when this list gets created. You set this Id by entering it in the constructor.
|
||||
So when adding your service just give it the ID of the previously last service in the list incremented by one.
|
||||
|
||||
### Search
|
||||
- [SearchExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/search/SearchExtractor.html)
|
||||
- [SearchQueryHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandlerFactory.html)
|
||||
|
||||
### Stream
|
||||
- [StreamExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamExtractor.html)
|
||||
- [StreamInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/stream/StreamInfoItemExtractor.html)
|
||||
- [LinkHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html)
|
||||
|
||||
### Search
|
||||
- [SearchExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/search/SearchExtractor.html)
|
||||
- [SearchQueryHandlerFactory](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandlerFactory.html)
|
||||
- [SuggestionExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/SuggestionExtractor.html) _(optional)_
|
||||
|
||||
### Channel
|
||||
- [ChannelExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html)
|
||||
- [ChannelInfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/channel/ChannelExtractor.html)
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ marven *.jar packages for android and java based on a github or gitlab repositor
|
|||
To the extractor through jitpack, you need to push them to your online repository of
|
||||
your copy that you host either on [github](https://github.com) or [gitlab](https://gitlab.com). It's important to host
|
||||
it on one of both. Now copy your repository url in Http format, go to [jitpack](https://jitpack.io/), and past it there
|
||||
From here you can grap the latest commit via `GET IT` button.
|
||||
From here you can grab the latest commit via `GET IT` button.
|
||||
I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An "implementation" string will be generated
|
||||
for you. Copy this string and replace the `implementation 'com.github.TeamNewPipe:NewPipeExtractor:<commit>'` line in
|
||||
the file [/app/build.gradle](https://github.com/TeamNewPipe/NewPipe/blob/dev/app/build.gradle#L58) with it.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Welcome to the NewPipe Documentaiton.
|
||||
# Welcome to the NewPipe Documentation.
|
||||
|
||||
This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the [NewPipe Extractor](https://github.com/TeamNewPipe/NewPipeExtractor).
|
||||
It is an addition to our auto generated [jdoc documentation](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/).
|
||||
|
|
@ -8,7 +8,7 @@ Please be aware that it is also in an early state, so help and [feedback](https:
|
|||
|
||||
## Introduction
|
||||
|
||||
The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular Youtube and streaming App [NewPipe](https://newpipe.schabi.org) for android, however it's system independent and also available for other platforms.
|
||||
The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular YouTube and streaming App [NewPipe](https://newpipe.schabi.org) for android, however it's system independent and also available for other platforms.
|
||||
|
||||
The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website.
|
||||
It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.
|
||||
|
|
|
|||
Loading…
Reference in New Issue