be-gay-do-crimes
/
newpipe-documentation
mirror of https://github.com/TeamNewPipe/documentation
1
0
Fork 0
Code Issues Releases Wiki Activity

Deployed 479eaaf with MkDocs version: 1.0.4

This commit is contained in:
Christian Schabesberger 2018-12-14 09:57:47 +01:00
parent a5f9f3d560
commit 6dea049dfb
9 changed files with 40 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
00_Prepare_everything/index.html
View File

@ -141,9 +141,9 @@
service with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SoundCloud.
The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our
<a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/">Jdoc</a> setup.</p>
<p><strong>IMPORTANT!!!</strong> this is likely to be the worst documentation you have ever red, so do not hesitate to
<p><strong>IMPORTANT!!!</strong> this is likely to be the worst documentation you have ever read, so do not hesitate to
<a href="https://github.com/teamnewpipe/documentation/issues">report</a> if
you find any (spelling)errors, incomplete parts or you simply don't understand something. We are an open community
you find any spelling errors, incomplete parts or you simply don't understand something. We are an open community
and are open for everyone to help :)</p>
<h2 id="setup-your-dev-environment">Setup your dev environment</h2>
<p>First and foremost you need to meet certain conditions in order to write your own service.</p>
@ -152,7 +152,7 @@ and are open for everyone to help :)</p>
<li>Basic understanding of <strong><a href="https://try.github.io">git</a></strong></li>
<li>Good <strong><a href="https://whatpixel.com/best-java-books/">Java</a></strong> knowledge</li>
<li>Good understanding of <strong><a href="https://www.w3schools.com/">web technology</a></strong></li>
<li>Basic understanding about <strong><a href="https://www.vogella.com/tutorials/JUnit/article.html">unit testing</a></strong> and <strong><a href="https://junit.org/">JUnit</a></strong></li>
<li>Basic understanding of <strong><a href="https://www.vogella.com/tutorials/JUnit/article.html">unit testing</a></strong> and <strong><a href="https://junit.org/">JUnit</a></strong></li>
<li>Flawless understanding of how to <a href="https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md#code-contribution">contribute</a> to the <strong>NewPipe project</strong></li>
</ul>
<h3 id="what-you-need-to-have">What you need to have</h3>
@ -191,11 +191,11 @@ As Gradle Project, select NewPipeExtractor. As task add "test". Now save and you
<ul>
<li>Basically anything except <a href="#not-allowed-content">NOT allowed content</a>.</li>
<li>Any kind of porn/NSFW that is allowed according to the <a href="https://www.justice.gov/archive/opa/pr/2003/April/03_ag_266.htm">US Porn act</a>.</li>
<li>Advertisement (may be handled specially tho)</li>
<li>Advertisement (may be handled specially though)</li>
</ul>
<h2 id="not-allowed-content">NOT allowed Content</h2>
<ul>
<li>NSFL</li>
<li>NSFL (Not Safe For Life)</li>
<li>Porn that is not allowed according to <a href="https://www.justice.gov/archive/opa/pr/2003/April/03_ag_266.htm">US Porn act</a>.</li>
<li>Any form of violence</li>
<li>Child pornography</li>

10
01_Concept_of_the_extractor/index.html
View File

@ -134,9 +134,9 @@
<p>Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern
you will find all over the code. It is called the <strong>extractor/collector</strong> pattern. The idea behind it is that
the <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html">extractor</a>
would produce single pieces of data, and the collector would take it and form usable data for the front end out of it.
The collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any
point the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of
would produce single pieces of data, and the collector would collect it to form usable data for the front end.
The collector also controls the parsing process, and takes care of error handling. So if the extractor fails at any
point, the collector will decide whether or not it should continue parsing. This requires the extractor to be made out of
many small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe.
You need to take care of the extractors.</p>
<h3 id="usage-in-the-front-end">Usage in the front end</h3>
@ -210,7 +210,7 @@ that will return the data of one InfoItem.</p>
</code></pre>
<h2 id="infoitems-encapsulated-in-pages">InfoItems encapsulated in pages</h2>
<p>When a streaming site shows a list of items it usually offers some additional information about that list, like it's title a thumbnail
<p>When a streaming site shows a list of items it usually offers some additional information about that list, like it's title, a thumbnail,
or its creator. Such info can be called <strong>list header</strong>.</p>
<p>When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down. </p>
<p>This is why a list in NewPipe lists are chopped down into smaller lists called <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html">InfoItemsPage</a>s. Each page has its own URL, and needs to be extracted separately.</p>
@ -229,7 +229,7 @@ such as:</p>
returns a ListExtractor.InfoItemsPage by its URL which was retrieved by the <code>getNextPageUrl()</code> method of the previous page.</li>
</ul>
<p>The reason why the first page is handled special is because many Websites such as YouTube will load the first page of
items like a regular webpage, but all the others as AJAX request.</p>
items like a regular webpage, but all the others as an AJAX request.</p>
</div>
</div>

14
02_Concept_of_LinkHandler/index.html
View File

@ -144,12 +144,12 @@ one unique ID that represents it, like this example:</p>
</ul>
<h3 id="importand-notes-about-linkhandler">Importand notes about LinkHandler:</h3>
<ul>
<li>A simple <code>LinkHandler</code> will contain the default URL, the ID and the original url.</li>
<li><code>LinkHandler</code> are ReadOnly</li>
<li>LinkHandler are also used to determine which part of the extractor can handle a certain link.</li>
<li>A simple <code>LinkHandler</code> will contain the default URL, the ID and the original URL.</li>
<li><code>LinkHandler</code>s are ReadOnly</li>
<li><code>LinkHandler</code>s are also used to determine which part of the extractor can handle a certain link.</li>
<li>In order to get one you must either call
<a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromUrl-java.lang.String-">fromUrl()</a> or <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/LinkHandlerFactory.html#fromId-java.lang.String-">fromId()</a> of the the corresponding <code>LinkHandlerFactory</code>.</li>
<li>Every type of Type of Resource has its own LinkHandlerFactory. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.</li>
<li>Every type of Type of Resource has its own <code>LinkHandlerFactory</code>. Eg. YoutubeStreamLinkHandler, YoutubeChannelLinkHandler, etc.</li>
</ul>
<h3 id="usage">Usage</h3>
<p>So the typical usage for getting a LinkHandler would look like this.</p>
@ -190,12 +190,12 @@ which is used to sort by name, date or view count.</p>
additionally to the abstract methods this factory inherits from the LinkHandlerFactory you can override
<a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html#getAvailableContentFilter--">getAvailableContentFilter()</a>
and <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html#getAvailableSortFilter--">getAvailableSortFilter()</a>.
Through these you can tell the front end which kind of filter your service support.</p>
Through these you can tell the front end which kind of filter your service supports.</p>
<h4 id="searchqueryhandler">SearchQueryHandler</h4>
<p>You cannot point to a search request with an ID like you point to a playlist or a channel, simply because one and the
same search request might have a changing outcome deepening on the country or the time you send the request. This is
same search request might have a different outcome depending on the country or the time you send the request. This is
why the idea of an "ID" is replaced by a "SearchString" in the <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandler.html">SearchQueryHandler</a>
These work like regular ListLinkHandler, accept that you don't have to implement the methodes <code>onAcceptUrl()</code>
These work like regular ListLinkHandler, except that you don't have to implement the methods <code>onAcceptUrl()</code>
and <code>getId()</code> when overriding <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/SearchQueryHandlerFactory.html">SearchQueryHandlerFactory</a>.</p>
</div>

14
03_Implement_a_service/index.html
View File

@ -149,7 +149,7 @@ before doing this.</p>
<p>Your service does not have to implement everything. Some parts are optional.
This is because not all services support every feature other services support. For example, it might be that a certain
service does not support channels. If so, you can leave out the implementation of channels, and make the corresponding
factory methode of the your <strong>StreamingService</strong> implementation return <strong>null</strong>. The forntend will handle the lack of
factory method of the your <strong>StreamingService</strong> implementation return <strong>null</strong>. The frontend will handle the lack of
having channels then.</p>
<p>However if you start to implement one of the optional parts of the list below, you have to implement all parts/classes
of it. NewPipe will crash if you only implement the extractor for the list item of a channel, but not the channel extractor itself.</p>
@ -170,9 +170,9 @@ which will give you a little help with that. <strong>Use Regex with care!!!</str
ask us to introduce a new library than start using regex to often.</p>
<ul>
<li>Html/XML Parsing: <a href="https://jsoup.org/apidocs/overview-summary.html">jsoup</a></li>
<li>JSON Parsiong: <a href="https://github.com/mmastrac/nanojson#parser-example">nanojson</a></li>
<li>JSON Parsing: <a href="https://github.com/mmastrac/nanojson#parser-example">nanojson</a></li>
<li>JavaScript Parsing/Execution: <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Rhino/Documentation">Mozilla Rhino</a></li>
<li>Link dectection in strings: <a href="https://github.com/robinst/autolink-java">AutoLink</a></li>
<li>Link detection in strings: <a href="https://github.com/robinst/autolink-java">AutoLink</a></li>
</ul>
<p>If you need to introduce new libraries please tell us before you do it.</p>
<h3 id="head-of-service">Head of Service</h3>
@ -198,8 +198,8 @@ of NewPipe. This way your service will become an official part of the NewPipe Ex
Every service has an ID, which will be set when this list gets created. You need to set this ID by entering it in the constructor.
So when adding your service just give it the ID of the previously last service in the list incremented by one.</p>
<h3 id="stream">Stream</h3>
<p>Streams are considered single entities of video or audio, they come along with metainformation like a title, a description,
next/related videos, thumbnail and commends. For getting the url to the actual stream data as well as this metainformation
<p>Streams are considered single entities of video or audio. They come along with metainformation like a title, a description,
next/related videos, a thumbnail and comments. For getting the URL to the actual stream data as well as this metainformation
StreamExtractor is used. The LinkHandlerFactory will represent a link to such a stream. StreamInfoItemExtractor will
extract one item in a list of items representing such Streams, like a search result or a playlist.
Since every Streaming service (obviously) provides streams this is required to implement. Otherwise your service was
@ -231,7 +231,7 @@ A playlist may only contains <strong>StreamInfoItems</strong>, but no other <str
<li><a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/linkhandler/ListLinkHandlerFactory.html">ListLinkHandlerFactory</a></li>
</ul>
<h3 id="channel">Channel</h3>
<p>A Channel is mostly a <a href="#playlist">Playlist</a>, the only diferens is that it does not represent a simple list of streams, but a
<p>A Channel is mostly a <a href="#playlist">Playlist</a>, the only difference is that it does not represent a simple list of streams, but a
user, a channel, or any entity that could be represented as a user. This is why the metadata supported by the channel extractor
differs form the one of a playlist.</p>
<p><strong>Parts required to be implemented:</strong></p>
@ -247,7 +247,7 @@ Kiosk are controversial, many people may not like them. If you don't like them a
Your service would look pretty empty if you select it and no video is being displayed. Also you should not override the preference of the user,
since users of NewPipe can decide by the settings weather they want to see the kiosk page or not.</p>
<h4 id="multiple-kiosks">Multiple Kiosks</h4>
<p>Most services will implement more than one Kiosk, so a service might have a "Top 20" for different categories like "Country Music", "Techno" ets.
<p>Most services will implement more than one Kiosk, so a service might have a "Top 20" for different categories like "Country Music", "Techno", etc.
This is why the extractor will let you implement multiple <strong>KioskExtractors</strong>. Since different kiosk pages might also differ
with their HTML structure every page you want to support has to be implemented as its own <strong>KioskExtractor</strong>.
However if the pages are similar you can use the same Implementation, but set the page type when you instantiate your <strong>KioskExtractor</strong>

8
04_Run_changes_in_App/index.html
View File

@ -91,7 +91,7 @@
<li><a class="toctree-l3" href="#run-your-changes-on-the-extractor">Run your changes on the Extractor</a></li>
<li><a class="toctree-l3" href="#trouble-shoot">Trouble shoot</a></li>
<li><a class="toctree-l3" href="#troubleshooting">Troubleshooting</a></li>
</ul>
@ -149,10 +149,10 @@ sometimes have to adjust the udev rules in order to
<a href="https://www.janosgyerik.com/adding-udev-rules-for-usb-debugging-android-devices/">make your device be accessible</a>.</p>
<h3 id="run-your-changes-on-the-extractor">Run your changes on the Extractor</h3>
<p>In order to use the extractor in our app we use <a href="https://jitpack.io">jitpack</a>. This is a build service that can build
marven *.jar packages for android and java based on a github or gitlab repositories. </p>
maven *.jar packages for android and java based on a github or gitlab repositories. </p>
<p>To the extractor through jitpack, you need to push them to your online repository of
your copy that you host either on <a href="https://github.com">github</a> or <a href="https://gitlab.com">gitlab</a>. It's important to host
it on one of both. Now copy your repository url in Http format, go to <a href="https://jitpack.io/">jitpack</a>, and past it there
it on one of both. Now copy your repository url in Http format, go to <a href="https://jitpack.io/">jitpack</a>, and past it there.
From here you can grab the latest commit via <code>GET IT</code> button.
I recomend not to use SNAPSHOT, since I am not sure when snapshot is build. An "implementation" string will be generated
for you. Copy this string and replace the <code>implementation 'com.github.TeamNewPipe:NewPipeExtractor:&lt;commit&gt;'</code> line in
@ -165,7 +165,7 @@ Your browser does not support the video tag.
<p>If everything synced well then you should only see a screen with OK signs. Now you can go on an compile and run NewPipe
with the new extractor.</p>
<p><img alt="image_sync_ok" src="../img/sync_ok.png" /></p>
<h3 id="trouble-shoot">Trouble shoot</h3>
<h3 id="troubleshooting">Troubleshooting</h3>
<p>If something went wrong on jitpack site, you can check their build log, by selecting the commit you tried to build and
click on that little paper symbol next to the <code>GET IT</code> button. If it is red it already shows that the build failed.
<img alt="jitpack failed to build" src="../img/jitpack_fail.png" /></p>

10
index.html
View File

@ -127,13 +127,13 @@
<h1 id="welcome-to-the-newpipe-documentation">Welcome to the NewPipe Documentation.</h1>
<p><img width=150 src="https://raw.githubusercontent.com/TeamNewPipe/NewPipe/dev/assets/new_pipe_icon_5.png"/></p>
<p>This side is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the <a href="https://github.com/TeamNewPipe/NewPipeExtractor">NewPipe Extractor</a>.
<p>This site is/should be a beginner friendly tutorial and documentation for people who want to use, or write services for the <a href="https://github.com/TeamNewPipe/NewPipeExtractor">NewPipe Extractor</a>.
It is an addition to our auto generated <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/">jdoc documentation</a>.</p>
<p>Please be aware that it is also in an early state, so help and <a href="https://github.com/TeamNewPipe/documentation/issues">feedback</a> is always welcome :D</p>
<h2 id="introduction">Introduction</h2>
<p>The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accedes like a normal API. The extractor is the core of the popular YouTube and streaming App <a href="https://newpipe.schabi.org">NewPipe</a> for android, however it's system independent and also available for other platforms. </p>
<p>The beauty behind this framework is it takes care about the extraction process, error handling etc., so you can take care about what is important: Scraping the website.
It focuses on making it possible for the creator of a scraper for a streaming service to create best outcome by least amount of written code.</p>
<p>The NewPipeExtractor is a Java framework for scraping video platform websites in a way that they can be accessed like a normal API. The extractor is the core of the popular YouTube and streaming App <a href="https://newpipe.schabi.org">NewPipe</a> for android, however it's system independent and also available for other platforms. </p>
<p>The beauty behind this framework is it takes care of the extracting process, error handling etc., so you can take care about what is important: Scraping the website.
It focuses on making it possible for the creator of a scraper for a streaming service to create the best outcome with the least amount of written code.</p>
</div>
</div>
@ -182,5 +182,5 @@ It focuses on making it possible for the creator of a scraper for a streaming se
<!--
MkDocs version : 1.0.4
Build Date UTC : 2018-11-16 18:23:26
Build Date UTC : 2018-12-14 08:57:47
-->

2
search/search_index.json
View File

File diff suppressed because one or more lines are too long

12
sitemap.xml
View File

@ -2,32 +2,32 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>None</loc>
<lastmod>2018-11-16</lastmod>
<lastmod>2018-12-14</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2018-11-16</lastmod>
<lastmod>2018-12-14</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2018-11-16</lastmod>
<lastmod>2018-12-14</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2018-11-16</lastmod>
<lastmod>2018-12-14</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2018-11-16</lastmod>
<lastmod>2018-12-14</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2018-11-16</lastmod>
<lastmod>2018-12-14</lastmod>
<changefreq>daily</changefreq>
</url>
</urlset>

BIN
sitemap.xml.gz
View File

Binary file not shown.