A typical call for retrieving data from a website would look like this:
-Info info;
+Info info;
try {
// Create a new Extractor with a given context provided as parameter.
Extractor extractor = new Extractor(some_meta_info);
@@ -155,10 +155,9 @@ try {
// handle errors when collector decided to break up extraction
}
-
Typical Implementation of a Single Data Extractor
The typical implementation of a single data extractor, on the other hand, would look like this:
-class MyExtractor extends FutureExtractor {
+class MyExtractor extends FutureExtractor {
public MyExtractor(RequiredInfo requiredInfo, ForExtraction forExtraction) {
super(requiredInfo, forExtraction);
@@ -180,7 +179,6 @@ try {
... // More datafields
}
-
Collector/Extractor Pattern for Lists
Information can be represented as a list. In NewPipe, a list is represented by an
InfoItemsCollector.
@@ -213,7 +211,6 @@ that will be able to retrieve data for one and only one InfoItem. This extractor
}
-
ListExtractor
There is more to know about lists:
@@ -313,17 +310,18 @@ items like a regular web page, but all the others as an AJAX request.
The typical usage for obtaining a LinkHandler would look like this:
-LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory();
+LinkHandlerFactory myLinkHandlerFactory = new MyStreamLinkHandlerFactory();
LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/the_video");
-
Implementation
In order to use LinkHandler for your service, you must override the appropriate LinkHandlerFactory. eg:
-class MyStreamLinkHandlerFactory extends LinkHandlerFactory {
+class MyStreamLinkHandlerFactory extends LinkHandlerFactory {
@Override
public String getId(String url) throws ParsingException {
@@ -179,7 +178,6 @@ LinkHandler myVideo = myLinkHandlerFactory.fromUrl("https://my.service.com/
}
}
-
ListLinkHandler and SearchQueryHandler
List based resources, like channels and playlists, can be sorted and filtered.
Therefore these type of resources don't just use a LinkHandler, but a class called
@@ -234,17 +232,18 @@ and getId() when overriding
-
-
-
- « Previous
-
-
- Next »
-
-
+
+
+
+ « Previous
+
+
+ Next »
+
+
An example implementation of the getKioskList() could look like this:
-@Override
+@Override
public KioskList getKioskList() throws ExtractionException {
KioskList list = new KioskList(getServiceId());
@@ -280,7 +280,6 @@ public KioskList getKioskList() throws ExtractionException {
}
-
Parts Required to be Implemented:
- KioskList.KioskExtractorFactory
@@ -320,17 +319,18 @@ public KioskList getKioskList() throws ExtractionException {
includeBuild should have the relative path as argument. ../NewPipeExtractor means one folder up in hierarchy,
and the folder is exactly named NewPipeExtractor. If that's not the case, edit this part.
GET IT button. If it'
This site is meant for those who want to maintain NewPipe, or just want to know how releasing works.
NewPipe is a web crawler. That means it does not use a web API, but instead tries to scrape the data from the website, -this however has the disadvantage of the app to break instantly when YouTube changes something. -We do not know when this happen. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as +
Depending on the service, NewPipe Extractor uses web crawling or internal APIs. +Both are subject to arbitrary changes by the service providers, like YouTube, SoundCloud or PeerTube. +When they change something, NewPipe Extractor and thus NewPipe break instantly. +Therefore, maintainers need to act quickly when it happens, and reduce our downtime as much as possible. The entire release cycle is therefore designed around this issue.
There is a difference between a release that introduces new features -and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown). -Lets have a look at the characteristics of a regular release, and then the characteristics of a hotfix release.
+and a release that fixes an issue that occurred because YouTube, or some other service, +changed their website (typically called a shutdown). +Let's have a look at the characteristics of a regular release, +and then the characteristics of a hotfix release.Regular releases are normal releases like they are done in any other app. Releases are always stored on master branch. The latest commit on -master is always equal to the currently released version. No development is done on master. This ensures that we always have one -branch with a stable/releasable version.
+Regular releases are normal releases as they are done in any other app. +Releases are always stored and tagged on master branch. The latest commit on +master is always equal to the currently released version. No development is done on master. +This ensures that we always have one branch with a stable/releasable version.
When developing, the dev branch is used. Pushing to dev directly, however, is not allowed, since QA and testing should be done first before adding something to it. -This ensures that the dev version works as stable a possible. -In order to change something on the app, one may want to fork the dev branch and develop the changes in their own branch (this is called feature branching).
+The dev branch is used for development. Pushing to dev directly, however, is not allowed, +since QA and testing should be done before adding something to the branch. +This ensures that the development version works as stable a possible. +In order to change something on the app, one may want to fork the dev branch +and develop the changes in their own branch (this is called feature branching).
Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other. -If a change is done on the API to the extractor, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible -again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes -on the fronted at any time.
+Make sure that both the dev branches, as well as the master branches of the extractor +and the frontend, are compatible with each other. +If the extractor's API is modified, make sure that frontend is compatible, +or changed to become compatible, with these changes. +If the PR that should make the frontend compatible +again can not be merged, please do not merge the corresponding PR on the extractor either. +This should make sure that any developer can run his changes on the fronted at any time.
After finishing a feature, one should open up a Pull Request to the dev branch. From here, a maintainer can do Code review and Quality Assurance (QA). -If you are a maintainer, please take care about the code architecture so corrosion or code shifting can be prevented. Please also prioritize code quality over functionality. -In short: cool function but bad code = no merge. Focus on leaving the code as clean as possible.
+After finishing a feature, one should open up a Pull Request to the dev branch.
+From here, a maintainer can do Code review and Quality Assurance (QA).
+If you are a maintainer, please take care about the code architecture
+so corrosion or code shifting can be prevented.
+Please also prioritize code quality over functionality.
+In short: cool function but bad code = no merge. Focus on keeping the code as clean as possible.
You, as a maintainer, should build the app and put the signed APK into the description of that new pull request. This way, other people can test the feature/bugfix and help with QA. You may not need to do this every time. It is enough to do it on bigger pull requests.
-After the maintainer merges the new feature into the dev branch, he should add the title of the pull request or a summary of the changes into the release notes.
+An APK for testing is provided by GitHub Actions for every PR. +Please ensure that this APK is tested thoroughly to prevent introducing regressions. +Testing features needs to take into account that NewPipe is used on a brought variety +of Android versions from KitKat to the latest, on custom ROMs like Lineage OS, CalyxOS or /e/ +and different devices like phones, tablets and TVs.
+Sometimes, the content of a PR changes over the time. +Modify the PR's title if it does not represent the introduced changes anymore. +After a maintainer merged the new feature into the dev branch, +they should add the PR's title or a summary of the changes into the release notes.
Once there are enough features together, and the maintainers believe that NewPipe is ready for a new release, they should create a new release. -Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: Don't do a release if you don't have time for it!!! -Below is a list of things you will want to do:
+Once there are enough changes, and the maintainers believe that NewPipe is ready
+for a new version, they should prepare a new release.
+Be aware of the rule that a release should never be done on a Friday.
+For NewPipe, this means: Don't do a release if you don't have time for it!!!
By following the steps listed below, you can publish a stable version of NewPipe:
dev branch at https://hosted.weblate.org/git/newpipe/strings/.NewPipe_<versionNumber>_RC1.apk.
- Zip it and post it to the head of the release issue. This way, others can test the release candidate.build.gradle file.releaseCandidate and debug build types.
+ Name the build apk files NewPipe_<versionCode>_RC1.apk and NewPipe_<versionCode>_debug_RC1.apk.
+ Zip and post them to the head of the pull request and issue. This way, others can test the release candidate.
+ Release (candidate) and debug APKs of the latest published NewPipe version
+ should also be provided to allow testing the upgrade process.While being in release phase no new pull requests must be merged into dev branch.
-This procedure does not have to be done for the extractor as extractor will be tested together with the fronted.
+New features can be merged into dev while the release candidate is tested. +PRs which aim to fix regressions of the upcoming release need to target the release/x.y.z branch. +Read Quickfixes for more info.
+The changelogs are translated during the test phase. +Therefore, the translations need to be merged from Weblate once more. +The translation commit is cherry-picked into the release branch.
+Once testing is done and the release branch does not contain critical regressions, and you think the update is ready, +proceed with releasing the new version.
When issuing a new release, you will most likely encounter bugs that might not have existed in previous versions. These are called regressions. -If you find a regression during release phase, you are allowed to push fixes directly into the release branch without having to fork a branch away from it. -All maintainers have to be aware that they might be required to fix regressions, so plan your release at a time when -you are available. Do not introduce new features during the release phase.
-When you have pushed a quickfix, you will want to update the release candidate you put into the issue corresponding to the release pull request.
-Increment the version number in the filename of the release candidate. e.g. NewPipe_<versionNumber>_RC2.apk etc. Don't update the actual version number. :P
When issuing a new release, you will most likely encounter bugs +that might not have existed in previous versions. +These are called regressions. +If you find a regression during release phase, +you are allowed to push fixes directly into the release branch +without having to fork a branch away from it. +Maintainers have to be aware that they might be required to fix regressions, +so plan your release at a time when you are available.
+When you have pushed a quickfix, you need to provide an updated release candidate.
+Increment the version number in the filename of the release candidate. e.g. NewPipe_<versionNumber>_RC2.apk etc.
+Don't update the actual version number. :P
Once the glorious day of all days has come, and you fulfill the ceremony of releasing. -After going through the release procedure of creating a new release and maybe a few quickfixes on the new release, +After going through the release procedure of creating a new release +and maybe a few quickfixes on the new release, this is what you should do when releasing:
v0.x.y.v0.x.y on the extractor.
metadata/org.schabi.newpipe.yml.fdroid signatures /path/to/newpipe.apk.
As aforementioned, NewPipe is a web crawler and could break at any moment. In order to keep the downtime of NewPipe as low as possible, when such a shutdown happens, +
As aforementioned, NewPipe heavily relies on external components and might break at a random point of time. +In order to keep the NewPipe's downtime as low as possible, when such a shutdown happens, we allow hotfixes.
Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch. -Of course you are not allowed to push to master directly so you will have to open up a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well.
+Hotfixes work on the master branch. +The dev branch has experimental changes that might have not been tested properly enough to be released, +if at all. The master branch should always be the latest stable version of NewPipe. +If the master branch breaks due to a shutdown, you should fix the master branch. +Of course, you are not allowed to push to master directly, +so you need to create a hotfix branch. +If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well.
If you fixed the issue and found it to be tested and reviewed well enough, you may release it. You don't need to undergo the full release procedure of a regular release, which takes more time to release. +
If you fixed the issue and found it to be tested and reviewed well enough, you may publish a new version. +You don't need to undergo the full release procedure of a regular release, which takes too much time. Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix. -It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better then a non-functional version ¯\_(ツ)_/¯. +It is important to release quickly for the sake of keeping NewPipe alive, and after all, +a slightly broken version of NewPipe is better than a non-functional version ¯\_(ツ)_/¯. Here's what you do when releasing a hotfix:
v0.x.y.v0.x.y on the extractor.build.gradle file.versionCode.metadata/org.schabi.newpipe.yml.fdroid signatures /path/to/newpipe.apk.gradlew assembleRelease) and sign it (or get it signed by one of the other maintainers).
In general, the release process for extractor versions is not that complicated compared to app releases. +The extractor has (in difference to the app) a decent test coverage. +Additionally, the latest extractor version is typically tested in the app's latest dev version. +Therefore, a long test phase is not needed when creating extractor releases.
+To create a new extractor version, update the version in the extractor's build.gradle file
+as well as the version names in the README.
+Merge the dev branch into master.
+The same that applies the app's release notes also applies to the extractor's release notes.
When publishing an extractor release via GitHub on the master branch, +a new JavaDoc version +is generated and published automatically. +Pleas keep an eye on the GitHub Action which is responsible for that. +If changes in that release introduced invalid JavaDoc, the build fails and needs to be fixed. +For this reason, you should check locally if there are any problems with the JavaDoc generation before publishing the new version.
The version nomenclature of NewPipe is simple.
The extractor is always released together with the app, therefore the version number of the extractor is identical to the one of NewPipe itself.
-In Android, an app can also have a versionCode. This code is a long integer and can be incremented by any value to show a device that a new version is there.
-For NewPipe, the version code will be incremented by 10 regardless of the change of the major or minor version number. The version codes between the 10 steps
-are reserved for our internal F-Droid build server.
Previously, the extractor was released together with the app, +therefore the version number of the extractor was identical to the one of NewPipe itself.
+We try to combine efforts to make NewPipe Extractor more independent of the app. +The extractor is used by multiple other applications +and therefore releasing extractor updates should not be coupled to app releases. +However, maintainers need to keep an eye on making the app compatible with extractor changes.
Release notes should tell what was changed in the new version of the app. The release nodes for NewPipe are stored in the GitHub draft for a new release. When a maintainer wants to add changes to the release note, but there is no draft for a new version, they should create one.
-Changes can be categorized into three types:
+Release notes should tell what was changed in the new version of the app. +The release notes for NewPipe are stored in the +GitHub draft for a new release. +When a maintainer wants to add changes to the release notes, +but there is no draft for a new version, they should create one.
+Changes can be categorized into five basic types:
When releasing a new version of NewPipe, before actually clicking "Release", the maintainer should copy the release notes from the draft and put it into a file called
-<versionCode>.txt (whereas <versionCode> needs to be the version code of the incoming release). This file must be stored in the directory /fastlane/metadata/android/en-US/changelogs. This way, F-Droid will be able to show the
-changes done to the app.
When adding a PR to the release notes, increase the PR counter at the top of the draft +and put the number before the PR summary / title. +This helps the blog post authors to keep easily track of new PRs. +Remove the numbers before publishing a new version :)
+If there is a blog post covering the changes in more detail, +make sure to link it on the top of the release notes. +It would be a pity, if only a few people read the blog post +after our wonderful writers put so much effort into creating it.
+Maintainers need to provide a changelog file for each release.
+A changelog file is used by F-Droid to give a quick summary of the most important changes for a release.
+This file is placed in the
+/fastlane/metadata/android/en-US/changelogs
+directory and named <versionCode>.txt (whereas <versionCode> is the version code of the incoming release).
+Changelog files must not exceed 500 bytes.
+Be aware that the changelog is translated into multiple languages.
+A changelog written in English which almost hits 500 bytes can hardly be translated completely within this limit.
+This causes troubles for translators, because Weblate enforces the 500 bytes limit, too.
+For this reason it is recommended to keep the changelog at 400 bytes.
When creating the changelog file be aware of changes which were done in the extractor as well.
+Before pushing the changelog to NewPipe's repo, ask other maintainers to review it.
+After pushing the changelog to NewPipe's GitHub repo, updating Weblate is necessary.
+This enables translators to work on localized versions of the changelog before a release is tagged and published.
NewPipe is and supports open source software. +For this reason, the preferred way to distribute the app is F-Droid. +F-Droid is a catalogue of FOSS apps and also comes with an Android client which handles app updates. +There are two ways to install NewPipe via F-Droid.
+fdroidserver on your device.https://gitlab.com/fdroid/fdroiddatametadata/org.schabi.newpipe.ymlfdroid signatures /path/to/newpipe.apk on the signed APK from within the repo.sudo pip3 install mkdocs to install mkdocs systemwide. Last
When you talk to the community, stay friendly and respectful with good etiquette. When you have a bad day, just don't go to GitHub (advice from our experience ;D ).
+NewPipe is translated via Weblate. +There are two different components which are open for translation:
+strings.Maintainers can access more options to handle Weblate via the +Manage > Repository Maintenance button +or via the Weblate CLI. These options include +basic access to Git operations like commit and rebase +as well as locking Weblate to prevent further changes to translations.
+
+HINT: When updating Weblate via the web interface, please use the "Update > Rebase" option.
Weblate is based on NewPipe's dev branch and is configured to automatically update its repository to be in sync with NewPipe.
+However, Weblate does not update its branch often, therefore it is better to update it manually after changing strings in NewPipe.
To do thus manually, commit the Weblate changes and rebase the repository. +Sometimes conflicts need to be resoled while rebasing the repository. +Conflicts need to be addressed ASAP, because Weblate is automatically locked once conflicts occur. +To do so, merge the changes from Weblate into NewPipe. +If Weblate does not recognize the new commit by itself, ask Weblate to rebase once more. +Weblate unlocks the translations when all conflicts are resolved and no errors are detected.
+Weblate does not push the translation changes to NewPipe automatically. +Doing this manually, allows the maintainers to do a quick review of the changes.
+Before merging weblate changes into NewPipe, make sure to commit all Weblate changes and
+lock the Weblate to prevent modifications while you update Weblate.
+To merge the changes into NewPipe, checkout Weblate's dev branch.
+You have read access to Weblate's repository via https://hosted.weblate.org/git/newpipe/strings/.
+If there are conflicts when rebasing weblate, resolve them.
Check the following things:
+- Is there a translation for a new language? If yes, register the language with the app's langauge selector
+- Use Analyse > Inspect Code in Android Studio to find unused strings and potential bugs introduced by Weblate.
+ Pay attention to plurals in Asian languages. They are broken by Weblate on a regular basis.
Push the changes to NewPipe's dev branch, update Weblate and unlock it.