74 lines
11 KiB
JSON
74 lines
11 KiB
JSON
{
|
|
"docs": [
|
|
{
|
|
"location": "/",
|
|
"text": "Welcome to NewPipe Tutorial\n\n\nThis side is a beginner tutorial for people who want to use, or write services for the \nNewPipe Extractor\n.\nThis however is not the \ndocumentation\n for it.\n\n\nThis tutorial and the documentation are in an early state. So \nfeedback\n is always welcome :D\n\n\nStay Tuned\n\n\nTune yourself to stay ;D",
|
|
"title": "Welcome to NewPipe Tutorial"
|
|
},
|
|
{
|
|
"location": "/#welcome-to-newpipe-tutorial",
|
|
"text": "This side is a beginner tutorial for people who want to use, or write services for the NewPipe Extractor .\nThis however is not the documentation for it. This tutorial and the documentation are in an early state. So feedback is always welcome :D",
|
|
"title": "Welcome to NewPipe Tutorial"
|
|
},
|
|
{
|
|
"location": "/#stay-tuned",
|
|
"text": "Tune yourself to stay ;D",
|
|
"title": "Stay Tuned"
|
|
},
|
|
{
|
|
"location": "/00_Prepare_everything/",
|
|
"text": "Prepare everything\n\n\nWelcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor\nservice with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's\ndive right. ;D\n\n\nSetup your dev environment\n\n\nFirst and foremost you need to meet certain conditions in order to write your own service.\n\n\nWhat you need to know\n\n\n\n\nBasic understanding of \ngit\n\n\nGood \nJava\n knowledge\n\n\nGood understanding of \nweb technology\n\n\nBasic understanding about \nunit testing\n and \nJUnit\n\n\nFlawless understanding of how to \ncontribute\n to the \nNewPipe project\n\n\n\n\nWhat you need to have\n\n\n\n\nA dev environment/ide that supports:\n\n\ngit\n\n\njava 8\n\n\ngradle\n\n\nI highly recomend \nIDEA Community\n since it has everything we need.\n\n\n\n\n\n\nA \ngithub\n account\n\n\nA loot of patience and excitement ;D\n\n\n\n\nAfter making sure all these conditions are provided fork the \nNewPipeExtractor\n,\nusing the \nfork button\n.\nThis way you have your own working repository. Now clone this repository into your local folder in which you want to work in.\nNext import the cloned project into your \nide\n\nand \nrun\n it.\nIf all the checks are green you did everything right, and you are good to go to move on to the next chapter.",
|
|
"title": "Prepare everything"
|
|
},
|
|
{
|
|
"location": "/00_Prepare_everything/#prepare-everything",
|
|
"text": "Welcome to the NewPipe tutorial. This tutorial will guide you through the process of creating your own NewPipeExtractor\nservice with which NewPipe will gain support for a dedicated streaming service like YouTube, Vimeo or SournCloud. Let's\ndive right. ;D",
|
|
"title": "Prepare everything"
|
|
},
|
|
{
|
|
"location": "/00_Prepare_everything/#setup-your-dev-environment",
|
|
"text": "First and foremost you need to meet certain conditions in order to write your own service.",
|
|
"title": "Setup your dev environment"
|
|
},
|
|
{
|
|
"location": "/00_Prepare_everything/#what-you-need-to-know",
|
|
"text": "Basic understanding of git Good Java knowledge Good understanding of web technology Basic understanding about unit testing and JUnit Flawless understanding of how to contribute to the NewPipe project",
|
|
"title": "What you need to know"
|
|
},
|
|
{
|
|
"location": "/00_Prepare_everything/#what-you-need-to-have",
|
|
"text": "A dev environment/ide that supports: git java 8 gradle I highly recomend IDEA Community since it has everything we need. A github account A loot of patience and excitement ;D After making sure all these conditions are provided fork the NewPipeExtractor ,\nusing the fork button .\nThis way you have your own working repository. Now clone this repository into your local folder in which you want to work in.\nNext import the cloned project into your ide \nand run it.\nIf all the checks are green you did everything right, and you are good to go to move on to the next chapter.",
|
|
"title": "What you need to have"
|
|
},
|
|
{
|
|
"location": "/01_Concept_of_the_extractor/",
|
|
"text": "Concept of the Extractor\n\n\nCollector/Extractor pattern\n\n\nBefore we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern\nyou will find all over the code. It is called the \nextractor/collector\n pattern. The idea behind this pattern is that\nthe \nextractor\n\nwould produce single peaces of data, and the collector would take it and form usable data for the front end out of it.\nThe collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any\npoint the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of\nmany small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe.\nYou need to take care of the extractors.\n\n\nUsage in the front end\n\n\nSo typical call for retrieving data from a website would look like this:\n\n\nInfo info;\ntry {\n // Create a new Extractor with a given context provided as parameter.\n Extractor extractor = new Extractor(some_meta_info);\n // Retrieves the data form extractor and builds info package.\n info = Info.getInfo(extractor);\n} catch(Exception e) {\n // handle errors when collector decided to break up extraction\n}\n\n\n\n\nTypical implementation of a single data extractor\n\n\nThe typical implementation of a single data extractor on the other hand would look like this:\n\n\nclass MyExtractor extends FutureExtractor {\n\n public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) {\n super(requiredInfo, forExtraction);\n\n ...\n }\n\n @Override\n public void fetch() {\n // Actually fetch the page data here\n }\n\n @Override\n public String someDataFiled() \n throws ExtractionException { //The exception needs to be thrown if someting failed\n // get piece of information and return it\n }\n\n ... // More datafields\n}\n\n\n\n\nCollector/Extractor pattern for lists\n\n\nSometimes information can not be represented as a structure, but as a list. In NewPipe an item of a list is called\n\nInfoItem\n. In order\nto get such items a \nInfoItemsCollector\n\nis used. For each item that should be extracted a new Extractor will be given to the InfoItemCollector via \ncommit()\n.\n\n\n\n\nWhen a streaming site shows a list it usually offers some additional information about that list, like it's title, a thumbnail\nor its creator. Such info can be called \nlist header\n.\n\n\nAlso if you open a list in a web browser the website usually does not load the whole list, but only a part\nof it. In order to get more you may have to click on a next page button, or scroll down. This is why a list in\nNewPipe is coped down into \nInfoItemPage\ns. Each Page has its own URL, and needs to be extracted separately.\n\n\nList header information and extracting multiple pages of an InfoItem list can be handled by a\n\nListExtractor",
|
|
"title": "Concept of the Extractor"
|
|
},
|
|
{
|
|
"location": "/01_Concept_of_the_extractor/#concept-of-the-extractor",
|
|
"text": "",
|
|
"title": "Concept of the Extractor"
|
|
},
|
|
{
|
|
"location": "/01_Concept_of_the_extractor/#collectorextractor-pattern",
|
|
"text": "Before we can start coding our own service we need to understand the basic concept of the extractor. There is a pattern\nyou will find all over the code. It is called the extractor/collector pattern. The idea behind this pattern is that\nthe extractor \nwould produce single peaces of data, and the collector would take it and form usable data for the front end out of it.\nThe collector also controls the parsing process, and takes care about error handling. So if the extractor fails at any\npoint the collector will decide whether it should continue parsing or not. This requires the extractor to be made out of\nmany small methods. One method for every data field the collector wants to have. The collectors are provided by NewPipe.\nYou need to take care of the extractors.",
|
|
"title": "Collector/Extractor pattern"
|
|
},
|
|
{
|
|
"location": "/01_Concept_of_the_extractor/#usage-in-the-front-end",
|
|
"text": "So typical call for retrieving data from a website would look like this: Info info;\ntry {\n // Create a new Extractor with a given context provided as parameter.\n Extractor extractor = new Extractor(some_meta_info);\n // Retrieves the data form extractor and builds info package.\n info = Info.getInfo(extractor);\n} catch(Exception e) {\n // handle errors when collector decided to break up extraction\n}",
|
|
"title": "Usage in the front end"
|
|
},
|
|
{
|
|
"location": "/01_Concept_of_the_extractor/#typical-implementation-of-a-single-data-extractor",
|
|
"text": "The typical implementation of a single data extractor on the other hand would look like this: class MyExtractor extends FutureExtractor {\n\n public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) {\n super(requiredInfo, forExtraction);\n\n ...\n }\n\n @Override\n public void fetch() {\n // Actually fetch the page data here\n }\n\n @Override\n public String someDataFiled() \n throws ExtractionException { //The exception needs to be thrown if someting failed\n // get piece of information and return it\n }\n\n ... // More datafields\n}",
|
|
"title": "Typical implementation of a single data extractor"
|
|
},
|
|
{
|
|
"location": "/01_Concept_of_the_extractor/#collectorextractor-pattern-for-lists",
|
|
"text": "Sometimes information can not be represented as a structure, but as a list. In NewPipe an item of a list is called InfoItem . In order\nto get such items a InfoItemsCollector \nis used. For each item that should be extracted a new Extractor will be given to the InfoItemCollector via commit() . When a streaming site shows a list it usually offers some additional information about that list, like it's title, a thumbnail\nor its creator. Such info can be called list header . Also if you open a list in a web browser the website usually does not load the whole list, but only a part\nof it. In order to get more you may have to click on a next page button, or scroll down. This is why a list in\nNewPipe is coped down into InfoItemPage s. Each Page has its own URL, and needs to be extracted separately. List header information and extracting multiple pages of an InfoItem list can be handled by a ListExtractor",
|
|
"title": "Collector/Extractor pattern for lists"
|
|
}
|
|
]
|
|
} |