Compare commits

...

309 Commits
v0.22.7 ... dev

Author SHA1 Message Date
TobiGr 8e92227b2e Fix JDoc 2024-11-24 17:15:36 +01:00
Stypox eebcc46255
Release v0.24.3 2024-11-24 17:07:47 +01:00
Stypox 9fb03f6c87
Merge pull request #1192 from TeamNewPipe/user-agent
[tests] Update user agent
2024-11-18 17:13:55 +01:00
TobiGr e4a1a6ecd8 Fix tests 2024-11-17 21:38:47 +01:00
TobiGr 727e791602 [YouTube] Update mocks 2024-11-17 21:38:46 +01:00
TobiGr d635d4db2a Make RecordingDownloader more resillient against ReCaptchaExceptions
Implement a max number of requests per minute to prevent hitting reate limits and triggering ReCaptchaExceptions. This slows down the RecordingDownloader significantly and can be adjusted if needed. A request ist retried once when facing a ReCaptchaException.
2024-11-17 21:38:46 +01:00
Stypox ea1a1d1375
Merge pull request #1242 from TeamNewPipe/revert-1205-feature-branch
Revert "Refactored Identifiers"
2024-11-16 14:01:10 +01:00
Stypox c00d0a7028
Revert "Refactored Identifiers (#1205)"
This reverts commit 0de224124b.
2024-11-16 14:00:38 +01:00
Stypox d3d5f2b3f0
Merge pull request #1240 from AudricV/yt_fix-playlists-items-extraction
[YouTube] Add support for new playlist items data structure
2024-11-14 16:37:35 +01:00
congyuluo 0de224124b
Refactored Identifiers (#1205)
Extractor.pageFetched -> Extractor.isPageFetched
Stream.equalStats(Stream) is renamed to Stream.areStatsEqual(Stream)
Stream.getBitrate() is renamed to Stream.getBitRate()
2024-11-13 10:01:20 +01:00
AudricV 183563cc9e
[YouTube] Add support for playlists lockupViewModels
This new data type, A/B tested or rolled out at the time the changes
are commited, is present on multiple surfaces.
2024-11-10 21:44:06 +01:00
AudricV f52d2269fc
[YouTube] Move channel verified status check from badges to a method
This method will be used in more places such as the new playlist item
data.

Support for a new icon used in artist channels has been also added.
2024-11-10 20:18:04 +01:00
TobiGr 667c867ad8 Update user agent to Firefox ESR 128 2024-11-10 17:27:22 +01:00
Tobi 169098432b
Merge pull request #1239 from AudricV/yt_fix-shorts-thumbnails-extraction
[YouTube] Fix Shorts' thumbnails extraction in their channel tab
2024-11-06 09:42:13 +01:00
AudricV 06b2c8e2aa
[YouTube] Fix Shorts' thumbnails extraction in their channel tab
Wrong methods were used to access and extract the thumbnails' data.
This has been fixed with this commit.
2024-11-06 09:31:42 +01:00
Tobi c343e31ed2
Merge pull request #1236 from Thompson3142/fix_scrubbing_seekbar_preview_crash
Add documentation for faulty framesets
2024-10-27 14:42:06 +01:00
TobiGr 1f26c12098 Use JDoc and inherit doc 2024-10-27 09:45:15 +01:00
Tobi 6af22e3e45
Merge pull request #1237 from AudricV/yt_more-audio-track-types-support
[YouTube] Add support for automatic dubbed and secondary audio tracks
2024-10-27 09:19:01 +01:00
AudricV 8a3350f79d
[YouTube] Add support for automatic dubbed and secondary tracks 2024-10-26 20:32:39 +02:00
Thompson3142 542867ff4d Add documentation for faulty framesets 2024-10-24 19:36:35 +02:00
Tobi abba78cf9d
Merge pull request #1235 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.11.3
Bump org.junit:junit-bom from 5.11.2 to 5.11.3
2024-10-22 20:53:10 +02:00
dependabot[bot] 534bbc90cf
Bump org.junit:junit-bom from 5.11.2 to 5.11.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.11.2 to 5.11.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.11.2...r5.11.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-22 09:39:48 +00:00
Tobi f169885dbc
Merge pull request #1219 from floriegl/fix-case-sensitive-jitpack-coordinates
Fix for JitPack case sensitive coordinates in README
2024-10-10 15:28:04 +02:00
Tobi 18c9f1fd38
Merge pull request #1233 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.11.2
Bump org.junit:junit-bom from 5.11.0 to 5.11.2
2024-10-05 20:22:02 +02:00
dependabot[bot] fb81eaab82
Bump org.junit:junit-bom from 5.11.0 to 5.11.2
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.11.0 to 5.11.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.11.0...r5.11.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-05 18:17:38 +00:00
Tobi 5431069588
Merge pull request #1231 from AudricV/yt_fix-n-param-decode-function-extraction
[YouTube] Fix extraction of n param deobfuscation function name
2024-10-05 20:16:15 +02:00
Tobi 743a4000b8
Merge pull request #1207 from TeamNewPipe/fix/peertube-certain-domains
[PeerTube] Fix parsing ID for instances whose domain ends with a or c
2024-10-05 19:20:42 +02:00
AudricV 69ff271be1
[YouTube] Fix extraction of n param deobfuscation function name
This commit adds two new regular expressions to parse the n parameter function.

It also improves existing regular expressions by using the constant representing
multiple characters instead of adding the one or multiple characters
token manually in each regex for everything and not only function names.
2024-09-29 16:01:10 +02:00
Audric V. eb30316a36
Merge pull request #1222 from AudricV/yt_fix-videos-channel-tab-linkhandler-serialization
[YouTube] Fix serialization of Videos channel tab when it is already fetched
2024-09-29 15:58:43 +02:00
AudricV 42c1afaf87
[YouTube] Fix serialization of Videos channel tab when already fetched
Also remove usage of Optional as fields as it is not a good practice. This
simplifies in some places channel info extraction code.
2024-09-29 15:35:21 +02:00
Audric V. 596bce294d
Merge pull request #1221 from AudricV/yt_support-new-shorts-ui-data
[YouTube] Fix extraction of Shorts in channels and remove visitor data usage
2024-09-29 14:54:07 +02:00
AudricV f9ffdd91d5
[YouTube] Update YoutubeChannelTabExtractorTest.Shorts test class mocks 2024-09-08 17:51:08 +02:00
AudricV 34f28fc1f0
[YouTube] Remove visitorData usage for shorts continuations
It isn't required anymore and not used by extractor anymore since commit
5a6da5f43e, as the wrong page ID is used as a
visitor data (the VerifiedStatus value as a string).
2024-09-08 17:41:23 +02:00
AudricV f926fbcf35
[YouTube] Add support for shortsLockupViewModels
This new UI data type is replacing the reelItemRenderer one.
2024-09-08 17:21:40 +02:00
floriegl 36cc17c789
Update dependency coordinates due to case sensitivity in JitPack 2024-09-04 15:33:05 +02:00
TobiGr 6e3a4a6d9d [SoundCloud] Fix test: title changed 2024-08-15 12:22:37 +02:00
Tobi 70d6a06bf2
Merge pull request #1211 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.11.0
Bump org.junit:junit-bom from 5.10.3 to 5.11.0
2024-08-15 12:12:40 +02:00
dependabot[bot] 1278517492
Bump org.junit:junit-bom from 5.10.3 to 5.11.0
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.3 to 5.11.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.3...r5.11.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-15 09:10:12 +00:00
TobiGr bcacfc53c5 [PeerTube] Fix parsing id for instances whose domain ends with a or c
The pattern to detect the channel ID was faulty and e.g. the ID detected for "https://kolektiva.media/video-channels/documentary_channel" was "a/video-channels" which is wrong.  A new pattern is added to distinguish between URLs and potential IDs because IDs must not start with a "/" while IDs inside an URL must.

Fixes TeamNewPipe/NewPipe#11369
2024-08-02 18:19:45 +02:00
TobiGr 6963385176 Fix issue template dir name 2024-08-02 12:50:30 +02:00
opusforlife2 5f1ba8cf7d
Add issue templates (#1204)
* Create bug report template

* Create feature request template

* Create config.yml

---------

Co-authored-by: TobiGr <tobigr@users.noreply.github.com>
2024-08-02 12:44:11 +02:00
Stypox 176da72cb4
Merge branch 'dev' 2024-07-25 18:29:07 +02:00
Stypox 530c157d4d
Release v0.24.2 2024-07-25 18:25:10 +02:00
Stypox 996eb046aa
Merge pull request #1203 from AudricV/yt_support-shows-and-pageheader-on-user-channels
[YouTube] Support shows and page header on user channels
2024-07-25 18:03:13 +02:00
AudricV 8db724943d
[YouTube] Update mocks of YoutubeChannelTabExtractorTest.Shorts test
For some reason, mocks of the continuation were not parsed. All mocks of the
test have been updated to fix the issue.
2024-07-25 17:51:43 +02:00
AudricV 76956ec95f
[YouTube] Fix VSauce test of YoutubeChannelExtractorTest test class
The channel description has been changed and some expected words have been
removed.
2024-07-25 17:51:43 +02:00
AudricV 10704dfc94
[YouTube] Fix NPE when getting channel header for videos channel tab 2024-07-25 17:51:43 +02:00
AudricV 8be64574e4
[YouTube] Support pageHeader on user channels
Also move duplicate strings into constants and add a missing default switch
case.
2024-07-25 17:51:42 +02:00
AudricV df26badd4a
[YouTube] Add common methods to get ID, name and age gate object of channels
Also move duplicate strings into constants and support pageHeader channel
header in user channels on YoutubeChannelHelper methods.
2024-07-24 19:51:58 +02:00
AudricV 5a6da5f43e
[YouTube] Support shows in channels and provide verified status to items
Also fix naming of info items' collection methods.
2024-07-24 19:51:58 +02:00
AudricV 9d5201f40e
[YouTube] Add support for showRenderers in search results 2024-07-24 19:51:58 +02:00
AudricV 37178bd007
[YouTube] Add base implementation for show InfoItems
As there are multiple show UI elements which share a lot of common data, a base
implementation, an abstract class named YoutubeBaseShowInfoItemExtractor, has
been created to handle common cases.
2024-07-24 19:51:57 +02:00
AudricV 5879190ada
[YouTube] Move channel header's verified status code to YoutubeChannelHelper
Also throw an exception when we cannot get the verified status of a channel in
YoutubeChannelExtractor due to a missing channelHeader, if the channel has no
channelAgeGateRenderer.
2024-07-24 19:51:56 +02:00
AudricV 9fa8d4c0b4
[YouTube] Support playlists as URL navigation endpoints 2024-07-24 18:47:38 +02:00
AudricV c99d94b615
[YouTube] Do not get twice runs array in YoutubeParsingHelper
The runs object was computed twice in getTextFromObject and getUrlFromObject
methods, leading to unneeded search costs. This has been avoided by storing the
array in method variables.
2024-07-24 18:47:30 +02:00
Stypox 2d36945b39
Merge pull request #1197 from AudricV/yt_innertube-clients-changes-for-streams
[YouTube] Workaround HTTP 403s on streaming URLs of WEB client (after some time or instantly for some JavaScript players), update clients info
2024-07-24 15:45:32 +02:00
AudricV d73de6b12d
[YouTube] Don't provide streaming URLs which have an non-decoded n param
This param used to throttle bandwidth of streaming URLs which have this
parameter when the correct value is not provided but it is not the case
anymore, as the streaming URLs return now an HTTP response code 403 in
this case.
2024-07-23 20:48:39 +02:00
AudricV 22f818109f
[YouTube] Fix JavaScript n parameter decoding function name extraction
This commits fixes extraction of the function name decoding the n parameter for
HTML5 clients' streaming URLs for YouTube base JavaScript player 3400486c.

Two new regexes have been added to the existing ones. All regexes and what they
extract has been documented.
2024-07-23 20:43:56 +02:00
AudricV 480f5e223e
[YouTube] Update mocks 2024-07-23 20:43:54 +02:00
AudricV 986a76494c
[YouTube] Fix some YoutubeStreamExtractorDefaultTest tests
- Fix typo in folder name of DescriptionTestPewdiepie test;
- Fix constant usage of DownloaderTestImpl as download implementation for
UnlistedTest and CCLicensed tests.
2024-07-23 20:43:54 +02:00
AudricV 1c07764b4f
[YouTube] Fix YoutubeSearchExtractorTest.CrisisResources
The "blue whale" search query does not return a crisis resource panel anymore,
so it was changed to a different word, "suicide".
2024-07-23 20:43:54 +02:00
AudricV a13510b962
[YouTube] Update clients info 2024-07-23 20:43:53 +02:00
AudricV f4931d8bbd
[YouTube] Workaround 403s on streaming URLs of WEB client after some time
These changes work around an anti-bot token, for which its requirement is A/B
tested on the WEB client. In this test, streaming URLs of this client return
HTTP errors 403 if the token is not provided after some time.

It also allows to not fetch the JavaScript player for non-age restricted
videos, reducing data usage.

The TVHTML5 embed client is now only fetched in the case of age-restricted
videos.

The methods forceFetchAndroidClient and forceFetchIosClient of
YoutubeStreamExtractor have been removed, as they are now not needed anymore.

These changes also break the extraction of appropriate error states for private
and deleted videos and invalid video IDs.
2024-07-23 20:43:48 +02:00
Tobi 312e91048c
Merge pull request #1177 from TeamNewPipe/bandcamp-upgrade-to-https
[Bandcamp] Upgrade incoming links to HTTPS
2024-07-22 11:52:52 +02:00
Fynn Godau f441036ed2 [Bandcamp] Upgrade incoming links to HTTPS 2024-07-22 11:48:02 +02:00
Marco Sirabella 02e14b8931
Add support for on.soundcloud.com urls (#1179)
Add support for on.soundcloud.com urls

Fixes #1178

Co-authored-by: TobiGr <tobigr@users.noreply.github.com>
2024-07-22 11:22:47 +02:00
Tobi 0e15f9ac1b
Merge pull request #1201 from TeamNewPipe/bandcamp-radio-v3
[Bandcamp] Show additional info in radio kiosk
2024-07-22 08:27:27 +02:00
Tobi 87af6bb223
Merge pull request #1200 from TeamNewPipe/bandcamp-fix-null-url-catenation
[Bandcamp] Null-safe url catenation in track playlist
2024-07-22 08:26:16 +02:00
Fynn Godau 227c6894a7 [Bandcamp] Show additional info in radio kiosk
A newer version of the radio list API delivers additional fields. Thanks
@crisp5.
2024-07-21 23:59:10 +02:00
Fynn Godau 97955e5e41 [Bandcamp] Null-safe url catenation in track playlist
Previously, if no URL was provided due to the track being a preorder or
unpaid track that doesn't have its own public page, we would catenate
the artist URL and null to a nonsensical string.

This invalid URL would also be used to try fetching a thumbnail URL.

The downstream behavior of the NewPipe client is only marginally
improved, as it now no longer throws visible exceptions due to the
missing thumbnails, but still gives an unfriendly error upon clicking
the item that has a `null` URL.
2024-07-21 23:35:57 +02:00
Tobi 4aaab63c12
Merge pull request #1199 from TeamNewPipe/bandcamp-update-artist-detection
[Bandcamp] Update artist page detection
2024-07-21 22:04:53 +02:00
Isira Seneviratne 9a29f9ee2d
Use the new URL encode/decode methods introduced in Java 10 (#1196)
* Use Java 10 URLDecoder/URLEncoder methods

* Simplify compatParseMap
2024-07-21 19:45:28 +05:30
Fynn Godau de9fb7cb28 [Bandcamp] Update artist page detection
Bandcamp changed the way the footer is rendered. Therefore, we check for
a link to the cart page instead.
2024-07-21 15:37:21 +02:00
XiangRongLin d39fc43282
[Youtube] Adjust throttling function extraction to changes (#1191)
* [Youtube] Adjust throttling function extraction to changes

---------

Co-authored-by: Stypox <stypox@pm.me>
2024-07-11 11:23:53 +02:00
XiangRongLin 592f1596e6
[Youtube] Adjust throttling function extraction to changes (#1191)
* [Youtube] Adjust throttling function extraction to changes

---------

Co-authored-by: Stypox <stypox@pm.me>
2024-07-11 11:20:33 +02:00
Tobi c3c6de85bc
Merge pull request #1180 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.11.0
Bump com.google.code.gson:gson from 2.10.1 to 2.11.0
2024-06-29 11:27:13 +02:00
dependabot[bot] 383000f10d
Bump com.google.code.gson:gson from 2.10.1 to 2.11.0
Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.10.1 to 2.11.0.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10.1...gson-parent-2.11.0)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-29 09:22:41 +00:00
Tobi 2c7076930c
Merge pull request #1186 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.3
Bump org.junit:junit-bom from 5.10.2 to 5.10.3
2024-06-29 11:20:50 +02:00
dependabot[bot] 11a31721c5
Bump org.junit:junit-bom from 5.10.2 to 5.10.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.2 to 5.10.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.2...r5.10.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-28 09:50:44 +00:00
Tobi 90183056b5
Merge pull request #1183 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.6
Bump com.github.spotbugs:spotbugs-annotations from 4.8.5 to 4.8.6
2024-06-22 23:35:46 +02:00
dependabot[bot] 2efea787d2
Bump com.github.spotbugs:spotbugs-annotations from 4.8.5 to 4.8.6
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.5 to 4.8.6.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.5...4.8.6)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-18 09:35:16 +00:00
TobiGr fafd471606 [PeerTube] Fix test for like count
Number changed
2024-05-08 19:25:12 +02:00
TobiGr 4f477ad72b [PeerTube] Fix testing comment content
The comment is not available anymore.
2024-05-08 19:25:12 +02:00
Tobi 964e429978
Merge pull request #1166 from TeamNewPipe/dependabot/github_actions/peaceiris/actions-gh-pages-4
Bump peaceiris/actions-gh-pages from 3 to 4
2024-05-08 19:24:40 +02:00
dependabot[bot] 10c6965a28
Bump peaceiris/actions-gh-pages from 3 to 4
Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages) from 3 to 4.
- [Release notes](https://github.com/peaceiris/actions-gh-pages/releases)
- [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md)
- [Commits](https://github.com/peaceiris/actions-gh-pages/compare/v3...v4)

---
updated-dependencies:
- dependency-name: peaceiris/actions-gh-pages
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-08 17:16:08 +00:00
TobiGr e54f38f5e7 [PeerTube] Fix test
UploaderName was changed by user
2024-05-08 19:13:49 +02:00
Tobi 5dd5c7a65b
Merge pull request #1173 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.5
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
2024-05-08 19:12:40 +02:00
Tobi 37438ff82f
Merge pull request #1174 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.15
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
2024-05-08 19:02:57 +02:00
TobiGr bba3b6c69b version 0.24.0 2024-05-08 13:01:06 +02:00
TobiGr b40a5784ed Fix JDoc
YoutubePostLiveStreamDvrDashManifestCreator.java:90: error: unexpected end tag: </p>
2024-05-08 12:41:36 +02:00
dependabot[bot] c6da4004e2
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.15.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:05 +00:00
dependabot[bot] f26e84d39f
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.3 to 4.8.5.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.3...4.8.5)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:03 +00:00
Tobi ec3e8378c6
Merge pull request #1171 from TeamNewPipe/fix/jdoc
Fix JavaDocs and add Checkstyle exception for line length in JavaDocs' links
2024-04-23 21:22:12 +02:00
TobiGr 8d2a7a5281 [PeerTube] Fix test 2024-04-23 20:02:29 +02:00
TobiGr 7c29dbc965 Fix JDoc
Add Checkstyle exception for LineLength in JDoc links.

See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/8804403691/job/24164703883
2024-04-23 19:55:51 +02:00
Stypox fbe9e6223a
Merge pull request #1168 from AudricV/yt_upd-cver-rm-keys-and-do-fixes
[YouTube] Update clients versions, restore access to some streams and more
2024-04-20 11:54:24 +02:00
Stypox 4e9e7cb29c
Improve assertTabsContain() to also check size 2024-04-20 11:48:36 +02:00
Stypox 9d0dd36034
[YouTube] Create constants for client names/versions 2024-04-20 11:43:54 +02:00
Stypox d4e6d22e64
[YouTube] Improve meta info code for review 2024-04-20 11:43:08 +02:00
AudricV 74bf000473
[YouTube] Update mocks 2024-04-11 18:59:11 +02:00
AudricV f9792cf3a9
[YouTube] Fix InteractiveTabbedHeader.testTabs test
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.

As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.

Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
2024-04-11 18:59:10 +02:00
AudricV f40fc0aa4f
[YouTube] Add support for new crisis meta info action data
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.

This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
2024-04-11 18:59:09 +02:00
AudricV 2a3c6f80d2
[YouTube] Fix YoutubeStreamExtractorRelatedMixTest 2024-04-10 21:19:03 +02:00
AudricV 657b4377aa
[YouTube] Fix YoutubeStreamExtractorDefaultTest tests 2024-04-10 21:19:03 +02:00
AudricV 7bf50bf1cb
[YouTube] Update Android client player parameters
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
2024-04-10 21:19:03 +02:00
AudricV 27dc1b1f50
[YouTube] Remove usage of API keys for InnerTube requests, bump versions
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.

Clients and device OS versions have been bumped to their latest stable version
known.

Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
2024-04-10 21:19:02 +02:00
AudricV e380bb4bc3
[YouTube] Add missing prettyPrint query parameter to mixes continuations 2024-04-10 19:06:36 +02:00
Audric V 6c3c2e25d7
Merge pull request #1163 from AudricV/yt-fix_comments_extraction
[YouTube] Support new comments data
2024-04-10 18:19:59 +02:00
Stypox 02274d5395
[YouTube] Avoid XSS attacks in description or comments 2024-04-08 11:21:31 +02:00
Stypox 3f7b2653e3
[YouTube] Add YoutubeDescriptionHelperTest 2024-04-08 11:21:31 +02:00
Stypox a90237816a
[YouTube] Cleanup description helper
Remove unneeded isClose field, and make constants private
2024-04-08 11:21:31 +02:00
Stypox b80c3f5d51
[YouTube] Replace link text with accessibility label 2024-04-08 00:14:28 +02:00
Stypox 09732d6785
[YouTube] Add support for styles in attributed descriptions
Also refactor descriptions parsing.
2024-04-04 21:14:27 +02:00
AudricV 293c3e9e47
[YouTube] Support new A/B tested comments data
Also improve current comments code by removing outdated comment
renderer data.
2024-04-04 21:14:26 +02:00
Stypox e5b30ae8c3
Merge pull request #1151 from Profpatsch/localization-return-optional
LocaleCompat.forLanguageTag: return Optional if parsing fails
2024-03-29 13:50:48 +01:00
Stypox 23fc7aa209
Throw ParsingException instead of IllegalArg 2024-03-29 13:44:42 +01:00
Stypox fb468a23f4
Merge pull request #1142 from TeamNewPipe/peertube-v6
[PeerTube] Add support for PeerTube v6 features
2024-03-29 12:25:38 +01:00
Stypox 6589e2c15d
Merge pull request #1148 from Stypox/mediaccc-channel-tab-handler
[MediaCCC] Allow obtaining channel tab link handler
2024-03-28 13:45:05 +01:00
Tobi ad71864b23
Merge pull request #1162 from Stypox/fix-comment-description-npe
Make getCommentText @Nonnull
2024-03-27 20:04:17 +01:00
Stypox c57016b79b
Make getCommentText @Nonnull 2024-03-27 15:26:06 +01:00
Tobi adcc1f17ee
Merge pull request #1160 from TeamNewPipe/fix/tests
Fix some failing unit tests and detect new account termination messages for YouTube
2024-03-20 15:17:55 +01:00
TobiGr 51ddacc81d [SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage
Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page
2024-03-20 15:10:39 +01:00
TobiGr 8392d50ba6 Update mocks for YoutubeChannelExtractorTest.NotAvailable 2024-03-20 14:59:44 +01:00
TobiGr aaccfecda8 [YouTube] Detect new account termination messages 2024-03-20 14:57:41 +01:00
TobiGr 73f0c63a9d [PeerTube] Fix tests for "What is PeerTube?" video 2024-03-20 14:44:06 +01:00
Tobi 896a55e319
Merge pull request #1139 from TeamNewPipe/dependabot/github_actions/actions/upload-artifact-4
Bump actions/upload-artifact from 3 to 4
2024-03-18 08:59:35 +01:00
dependabot[bot] e58fc652e0
Bump actions/upload-artifact from 3 to 4
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-18 07:53:24 +00:00
Tobi e3f2c9aec7
Merge pull request #1153 from TeamNewPipe/dependabot/github_actions/actions/cache-4
Bump actions/cache from 3 to 4
2024-03-18 08:51:24 +01:00
Tobi 6b0fc14c04
Merge pull request #1156 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.2
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
2024-03-18 08:46:13 +01:00
dependabot[bot] d579b608e5
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.0 to 5.10.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-05 09:15:26 +00:00
TobiGr fe47a4311f [PeerTube] Add test for segments and framesets 2024-01-29 10:22:06 +01:00
TobiGr 15e0e74b48 [PeerTube] Add support for stream frames/storyboards extraction
Implement PeerTubeStreamExtractor.getFrames()
2024-01-29 10:22:06 +01:00
dependabot[bot] da04eded5d
Bump actions/cache from 3 to 4
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-18 09:32:42 +00:00
Profpatsch 7408173246 LocaleCompat.forLanguageTag: return Optional if parsing fails
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.

So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
2024-01-07 14:31:34 +01:00
Stypox aaf3231fc7
[MediaCCC] Fix lambda link handler keeping reference to extractor
This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway.
2023-12-30 23:23:19 +01:00
Stypox 137e924035
[MediaCCC] Add ChannelTabExtractorTest 2023-12-30 22:53:51 +01:00
Stypox cc9ade962e
[MediaCCC] Allow obtaining channel tab extractor from scratch
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
2023-12-30 22:53:27 +01:00
Stypox 3402cdb666
Merge pull request #1147 from petlyh/youtube-albums
[YouTube] Add Albums channel tab
2023-12-30 21:58:19 +01:00
petlyh 6dc25f7b97
[YouTube] Add Albums channel tab mocks 2023-12-30 14:46:39 +01:00
petlyh 4408e2d0ac
[YouTube] Add Albums channel tab 2023-12-30 14:01:30 +01:00
TobiGr 9ab932e394 Rename testDoNotAcceptNonURLs() -> assertDoNotAcceptNonURLs() 2023-12-29 16:38:11 +01:00
Stypox 9d66debf3c
Merge pull request #1132 from TeamNewPipe/dependabot/github_actions/actions/setup-java-4
Bump actions/setup-java from 3 to 4
2023-12-29 16:20:46 +01:00
Tobi 038ebdedc4
Merge pull request #1143 from petlyh/peertube-non-urls
Avoid PeerTube accepting non-URLs
2023-12-29 12:47:30 +01:00
TobiGr 61d237de02 [PeerTube] Test onAccept(String URL) in LinkHandlerFactories for non-URLs 2023-12-29 12:45:02 +01:00
petlyh 2b2c1546d1 Avoid PeerTube accepting non-URLs 2023-12-29 12:27:39 +01:00
Tobi 1e93b1dc20
Merge pull request #1135 from Stypox/yt-emergency-info
[YouTube] Implement emergency meta info
2023-12-29 12:01:40 +01:00
dependabot[bot] 3400af99b3
Bump actions/setup-java from 3 to 4
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:57:47 +00:00
Tobi 1f8a044462
Merge pull request #1138 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.3
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
2023-12-29 11:56:18 +01:00
dependabot[bot] 1470aa7303
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.0 to 4.8.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.0...4.8.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:53:28 +00:00
TobiGr 8f9ebdcb77 [PeerTube] Fix failing PeertubeTrendingLinkHandlerFactoryTest
The factory was updated in #1144
2023-12-29 11:52:19 +01:00
Stypox 1553931027
Merge pull request #1145 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.17.2
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
2023-12-29 11:27:01 +01:00
Stypox b2ec1b15fb
Merge pull request #1144 from dragfyre/patch-1
Update PeertubeTrendingLinkHandlerFactory.java
2023-12-29 11:21:08 +01:00
dependabot[bot] 151ee99da3
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.2 to 1.17.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.2...jsoup-1.17.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 09:47:00 +00:00
dragfyre 65e7bc5b95
Update PeertubeTrendingLinkHandlerFactory.java
correcting Peertube local trending api URL (per #10685 in main NewPipe repo); see https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getVideos
2023-12-28 14:50:31 +07:00
Stypox f276caf54a
Release v0.23.1 2023-12-21 21:59:21 +01:00
Stypox fc54fb2fdb
Merge pull request #1140 from Stypox/yt-shorts-no-duration
[YouTube] Always return -1 as duration of Shorts returned inside reel items
2023-12-21 21:52:40 +01:00
Stypox 0518487d26
Fix SearchInfo's non-null MetaInfo being null when initialized or when an extraction error occurs (#1141)
The meta info might have been null either when SearchInfo is first initialized, or when extractor.getMetaInfo() throws an exception in getInfo().
This caused NewPipe to crash instead of showing a nice error in https://www.reddit.com/r/youtube/comments/184ttmw/what_exactly_about_blue_whales_has_youtube_so/.
2023-12-21 21:39:25 +01:00
Stypox 5b59a1a8c5
[YouTube] Move meta info extraction to separate file
YoutubeParsingHelper was longer than 2000 lines which caused checkstyle issues
2023-12-21 21:19:08 +01:00
Stypox b8e12dd76c
[YouTube] Implement emergency meta info
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").

It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association

Also add a test that just checks if a meta info is properly extracted
2023-12-21 21:19:08 +01:00
Stypox 83c1737f70
[YouTube] Update shorts channel tab mocks 2023-12-21 20:54:16 +01:00
Stypox 2938067c2c
[YouTube] Shorts don't provide a duration anymore 2023-12-21 20:41:01 +01:00
Stypox 91419ec6e8
Release v0.23.0 2023-12-10 15:59:32 +01:00
Audric V 678c98f24c
Merge pull request #1127 from AudricV/yt_improvements-and-fixes
[YouTube] Make some improvements and fixes
2023-12-09 14:07:31 +01:00
AudricV ec0194cfbf
[YouTube] Update mocks 2023-12-08 21:46:52 +01:00
AudricV 00a0f1a103
[YouTube] Add a blocking crisis resources bypass in search results test
This test only tests that search results are returned, when no content filters
are provided and crisis resources blocking search results should be returned.

Searches with blocking crisis resources and content filters should work too, as
the bypass has been implemented for them.
2023-12-08 21:46:49 +01:00
AudricV 06838d7245
[YouTube] Fix YoutubeSearchQHTest tests
As search parameters to bypass crisis resources blocking search results have
been implemented, they need to be added to search tests, in order to pass
them.
2023-12-08 21:46:49 +01:00
AudricV 642bb01388
[YouTube] Fix testSubscriberCount method of YoutubeChannelExtractorTest.CarouselHeader
The subscriber count is now lower than the expected count as some people
unsubscribed to the Sports system channel. The expected count has been so
lowered.
2023-12-08 21:46:49 +01:00
AudricV bedc9e5bc0
[YouTube] Remove Channels channel tab in tests 2023-12-08 21:46:48 +01:00
AudricV 5fa22ae25b
[YouTube] Enable and change testRelatedItems method of YoutubePlaylistExtractorTest.LearningPlaylist
This change makes the method test whether there is no more items instead of
being disabled.
2023-12-08 21:46:48 +01:00
AudricV 29dc7625f2
[YouTube] Fix YoutubeSearchExtractorTest.Suggestion
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.

URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
2023-12-08 21:46:48 +01:00
AudricV 56ab35423e
[YouTube] Fix potential NullPointerException in YoutubeSearchExtractor.getSearchSuggestion 2023-12-08 21:46:48 +01:00
AudricV 9dc1eab28c
[YouTube] Fix expected upload dates of StreamExtractor tests
YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.

This commit makes the StreamExtractor tests use these more precise dates.
2023-12-08 21:46:48 +01:00
AudricV ad3d187ac7
[YouTube] Fix testRelatedItems method of YoutubeStreamExtractorRelatedMixTest
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
2023-12-08 21:46:47 +01:00
AudricV e111814401
[YouTube] Fix YoutubeStreamExtractorDefaultTest.RatingsDisabledTest
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.

The test class's mocks will be updated in a different commit.
2023-12-08 21:46:47 +01:00
AudricV fc45941ead
[YouTube] Fix YoutubeChannelExtractorTest tests
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
2023-12-08 21:46:47 +01:00
AudricV 0bcb241c38
[YouTube] Fix expected data in YoutubeStreamExtractorRelatedMixTest
Video's title and tags have been changed by its uploader, so they have to be
updated.

Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
2023-12-08 21:46:47 +01:00
AudricV 6ba8251be1
[YouTube] Bypass crisis resources blocking search results
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.

This commit changes search parameters to include the property allowing to show
search results.
2023-12-08 21:46:47 +01:00
AudricV 7dea2d0d27
[YouTube] Remove Channels channel tab support
This tab has been removed by YouTube.
2023-12-08 21:46:47 +01:00
AudricV 3782d9a02a
[YouTube] Support new A/B tested like data and avoid like count conversion from integer to long
Also make minor improvements to current like data extraction and remove
previous like count data support, as it is not returned anymore.
2023-12-08 21:46:46 +01:00
AudricV b71ce1123f
[YouTube] Extract only search results corresponding to a search type
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.

Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
2023-12-08 21:46:46 +01:00
AudricV ff8ed7247f
[YouTube] Switch to new consent cookie
Also move the documentation of the consent in its setter method in order to be
accessible publicly and improve it.
2023-12-08 21:46:46 +01:00
AudricV ec838d7421
[YouTube] Add missing prettyPrint query parameter to some test InnerTube requests
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.

Also remove an unneeded ParsingException exception throwing declaration in a
test method.
2023-12-08 21:46:46 +01:00
AudricV 2c941794c0
[YouTube] Add utcOffsetMinutes to all InnerTube payloads
This should make returned dates consistent between timezones and countries on
which the extractor is ran.

It was previously only set on YouTube Music search continuations.
2023-12-08 21:46:46 +01:00
AudricV d97c9e0db1
[YouTube] Improve payloads and URLs of InnerTube requests
For every InnerTube request:
- Always add a `request` object with the following properties:
  - "internalExperimentFlags" set to an empty array;
  - "useSsl" set to "true";
  - "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.

For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
2023-12-08 21:46:45 +01:00
AudricV 8a9ebcc373
[YouTube] Update InnerTube clients' version and devices' OS version and model 2023-12-08 21:46:45 +01:00
Tobi eac850ca10
Merge pull request #1114 from FineFindus/feat/comment-author-is-owner
[YouTube] Add channel owner to comments
2023-10-25 09:51:12 +02:00
Tobi 5ab1f784e8
Merge pull request #1117 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.2
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
2023-10-21 19:26:36 +02:00
dependabot[bot] 9d7bcba050
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.1 to 1.16.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.1...jsoup-1.16.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-20 09:13:21 +00:00
dependabot[bot] e26065148a Bump com.github.spotbugs:spotbugs-annotations from 4.7.3 to 4.8.0
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.3 to 4.8.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.3...4.8.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 14:19:10 +02:00
FineFindus 34b05a0dda
feat(youtube/comments): support creator replies 2023-10-09 16:33:43 +02:00
TobiGr 0821f09114
Add missing mocks 2023-10-09 16:33:43 +02:00
FineFindus c1784a4bdb
[YouTube] Add channel owner to comments 2023-10-09 16:33:43 +02:00
TobiGr f9846352ea Fix wrong `@Nullable` annotation 2023-10-09 16:02:57 +02:00
Tobi d6f5cba6e2
Merge pull request #1111 from FineFindus/feat/creator-reply
Add `hasCreatorReply()` to CommentsInfoItem
2023-10-09 12:45:56 +02:00
TobiGr 9d63c75623 Add missing mocks 2023-10-09 11:24:39 +02:00
TobiGr d49f8411d7 [PeerTube] Implement CommentsInfoItemExtractor.hasCreatorReply() 2023-10-09 02:47:12 +02:00
Stypox bb132167d5
Merge pull request #1113 from AudricV/snd_fix-non-jpg-images
[SoundCloud] Fix extraction of non-JPG images
2023-10-02 19:40:57 +02:00
AudricV c98695fcea
[SoundCloud] Fix extraction of non-JPG images
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.

Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.

As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
2023-10-01 20:33:25 +02:00
AudricV ac00459c1a
Change requirement of image extensions in ImageSuffix class' Javadoc to a possibility
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.

With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
2023-09-30 21:11:09 +02:00
FineFindus dd7b2d9798
feat(youtube/comments): support creator replies 2023-09-25 10:40:45 +02:00
Youssif Shaaban Alsager 917554acc4
[YouTube] Add support for ultralow audio formats (#1063) 2023-09-24 19:04:34 +02:00
Tobi 8b0068f8f4
Merge pull request #1110 from christian-2hu/chore/update-copyright
chore: Update copyright notices
2023-09-23 00:29:19 +02:00
Christian fc67d49f59 Update copyright notices
Update copyright notices to comply to GPLv3 and change NewPipe to NewPipe Extractor on some notices that were not updated.
2023-09-22 19:10:15 -03:00
Stypox 289db1178a
Merge pull request #1108 from AudricV/yt_refactor-js-usage
[YouTube] Refactor JavaScript usage and fix extraction of obfuscated signature deobfuscation function
2023-09-22 10:41:57 +02:00
AudricV 6ed22099a2
[YouTube] Update stream mocks 2023-09-21 21:59:34 +02:00
AudricV 714b141ecb
[YouTube] Catch any exception when extracting something from JavaScript's base player 2023-09-21 21:59:33 +02:00
AudricV 588c6a8422
[YouTube] Quote signature deobfuscation function name and add semicolon only where needed 2023-09-21 21:59:33 +02:00
AudricV 1fa85ec6ca
[YouTube] Add tests for signature timestamp extraction and signature deobfuscation function extraction and execution 2023-09-21 21:59:33 +02:00
AudricV a04bc320de
[YouTube] Convert signature timestamp to integer
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.

As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
2023-09-21 21:59:32 +02:00
AudricV 7de3753a81
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.

For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.

The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.

YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.

If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.

All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.

Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-21 21:59:32 +02:00
AudricV 6884d191cd
[YouTube] Add utility class around signatures and fix signature deobfuscation function extraction
The goal of this class is to decouple the extraction of signature timestamp and
signature deobfuscation function from YoutubeStreamExtractor.

The extraction of the signature deobfuscation function has been also adapted to
support the latest YouTube player versions.

This new class, YoutubeSignatureUtils, doens't store anything temporary such as
a copy of the player code, which has to be passed where required. It is not
public, as it will be used by a JavaScript player manager class in the future,
in order to handle in a better way fetching, caching and resetting cache of the
player code.
2023-09-21 21:59:26 +02:00
Tobi 3be76a6406
Merge pull request #1107 from Isira-Seneviratne/Locale_forLanguageTag
Use Locale.forLanguageTag() in tests
2023-09-18 16:48:30 +02:00
TobiGr 17790328cd Improve doc 2023-09-18 16:44:51 +02:00
Isira Seneviratne 4bc8ae7812 Use Locale.forLanguageTag() in tests 2023-09-18 08:59:13 +05:30
Tobi 90aed06a63
Merge pull request #1105 from TeamNewPipe/fix/bandcamp-streame-extractor-test
[Badcamp] Fix StreamExtractorTest
2023-09-18 01:49:04 +02:00
TobiGr cf49f4a31c [Badcamp] Fix StreamExtractorTest
The song was renamed and the URL changed
2023-09-17 23:58:07 +02:00
Tobi 7c7ceaceab
Merge pull request #1103 from TeamNewPipe/dependabot/github_actions/actions/checkout-4
Bump actions/checkout from 3 to 4
2023-09-17 22:42:01 +02:00
dependabot[bot] 72c475d944
Bump actions/checkout from 3 to 4
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 10:00:13 +00:00
Stypox 1f08d28ae5
Merge pull request #889 from AudricV/multiple-images-support
Multiple images support
2023-08-13 11:35:11 +02:00
AudricV e8bfd20170
[MediaCCC] Apply changes in extractor tests
Also remove some public test methods modifiers.
2023-08-12 22:56:33 +02:00
AudricV 0292c4f3e8
[Bandcamp] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.

BandcampChannelExtractorTest.testLength has been removed as the test is always
true.
2023-08-12 22:56:32 +02:00
AudricV 2578f22054
[Bandcamp] Add utility test method to test images
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains f4.bcbits.com/img and ends with .jpg or .png.

To do so, a new non-instantiable final class has been added: BandcampTestUtils.
2023-08-12 22:56:32 +02:00
AudricV ba5315c72d
[PeerTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and improve some code.
2023-08-12 22:56:32 +02:00
AudricV 1d72bac53d
[SoundCloud] Apply changes in extractor tests 2023-08-12 22:56:32 +02:00
AudricV 93a210394d
[YouTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
2023-08-12 22:56:31 +02:00
AudricV 2c436d428c
[YouTube] Add utility test method to test images in YoutubeTestsUtils
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains the string yt.

The JavaDoc of the class has been also updated to reflect the changes made in
it (it is now more general).
2023-08-12 22:56:31 +02:00
AudricV d381f3b70b
Update avatar, banners and thumbnail methods' name and apply changes in DefaultStreamExtractorTest 2023-08-12 22:56:31 +02:00
AudricV 434e885708
Add utility methods in ExtractorAsserts to check whether a collection is empty and to test image collections
Two new methods have been added in ExtractorAsserts to check if a collection is
empty:

- assertNotEmpty(String, Collection<?>), checking:
  - the non nullity of the collection;
  - its non emptiness (if that's not case, an exception will be thrown using
    the provided message).

- assertNotEmpty(Collection<?>), calling assertNotEmpty(String, Collection<?>)
  with null as the value of the string argument.

A new one has been added to this assertion class to check the contrary:
assertEmpty(Collection<?>), checking emptiness of the collection only if it is
not null.

Three new methods have been added in ExtractorAsserts as utility test methods
for image collections:

- assertContainsImageUrlInImageCollection(String, Collection<Image>), checking
that:
  - the provided URL and image collection are not null;
  - the image collection contains at least one image which has the provided
    string value as its URL (which is a string) property.

- assertContainsOnlyEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - they have the same size;
  - each image of the first collection has its equivalent in the second one.
    This means that the properties of an image in the first collection must be
    equal in an image of the second one.

- assertNotOnlyContainsEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - one of the following conditions is met:
    - they have different sizes;
    - an image of the first collection has not its equivalent in the second one.
      This means that the properties of an image in the first collection must
      be not equal in an image of the second one.

These methods will be used by services extractors tests (and default ones) to
test image collections.
2023-08-12 22:56:31 +02:00
AudricV 5158472852
Apply changes in DefaultTests and add utility method to test image lists
This new method, defaultTestImageList(List<Image), will check that the image
list is not null.

For each image, it will test that its URL is secure and its height and width
are more than or equal to their relevant unknown constants in the Image class
(HEIGHT_UNKNOWN and WIDTH_UNKNOWN).
2023-08-12 22:56:31 +02:00
AudricV 70fb3aa38e
Update BaseExtractorTests image methods' name
Also suppress unused warnings in BaseStreamExtractorTest, like it is done on
other BaseExtractorTests interfaces.
2023-08-12 22:56:30 +02:00
AudricV e16d521b7b
[MediaCCC] Apply changes in Extractors
Also remove usage of the conference logo as the banner of a conference, as it
is a logo and not a banner.
2023-08-12 22:56:30 +02:00
AudricV 306068a63b
[MediaCCC] Apply changes in InfoItemExtractors 2023-08-12 22:56:30 +02:00
AudricV 2f40861428
[MediaCCC] Add utility methods to get image lists from conference logos and streams
These three new methods, added in MediaCCCParsingHelper,
getImageListFromImageUrl(String), getThumbnailsFromStreamItem(JsonObject) and
getThumbnailsFromLiveStreamItem(JsonObject) (the last two are based on a common
method, getThumbnailsFromObject(JsonObject, String, String)), return an empty
list if the case no image URL could be extracted.

Images returned have their height and width unknown and a resolution level
depending on the image key of the JSON API response.
2023-08-12 22:56:30 +02:00
AudricV 71cda03c4c
[Bandcamp] Apply changes in Extractors 2023-08-12 22:56:29 +02:00
AudricV 7e01eaac33
[Bandcamp] Apply changes in InfoItemExtractors 2023-08-12 22:56:29 +02:00
AudricV 4b80d737a4
[Bandcamp] Add utility methods to get multiple images
Bandcamp images work with image IDs, which provide different resolutions.

Images on Bandcamp are not always squares, and some IDs respect aspect ratios
where some others not.

The extractor will only use the ones which preserve aspect ratio and will not
provide original images, for performance and size purposes.

Because of this aspect ratio preservation constraint, only one dimension will
be known at a time.

The image IDs with their respective dimension used are:

- 10: 1200w;
- 101: 90h;
- 170: 422h;
- 171: 646h;
- 20: 1024w;
- 200: 420h;
- 201: 280h;
- 202: 140h;
- 204: 360h;
- 205: 240h;
- 206: 180h;
- 207: 120h;
- 43: 100h;
- 44: 200h.

(Where w represents the width of the image and h the height of the image)

Note that these dimensions are theoretical because if the image size is less
than the dimensions of the image ID, it will be not upscaled but kept to its
original size.

All these resolutions are stored in a private static list of ThumbnailSuffixes
in BandcampExtractorHelper, in which the methods to get mutliple images have
been added:

- getImagesFromImageUrl(String): public method to get images from an image URL;
- getImagesFromImageId(long, boolean): public method to get images from an
  image ID;
- getImagesFromImageBaseUrl(String): private utility method to get images from
  the static list of ThumbnailSuffixes from a given image base URL, containing
  the path to the image, a "a" letter if it comes from an album, its ID and an
  underscore.

Some existing methods have been also edited:

- the documentation of getImageUrl(long, boolean) has been changed to reflect
  the Bandcamp images findings;
- getThumbnailUrlFromSearchResult has been renamed to
  getImagesFromSearchResult, and a documentation has been added to this method.

The method replaceHttpWithHttps of the Utils class has been also used in
BandcampExtractorHelper instead of doing manually what the method does.
2023-08-12 22:56:29 +02:00
AudricV 4e6fb368bc
[PeerTube] Apply changes in Extractors and remove usages of default avatar picture
The default avatar picture was used when no profile picture was found, but it
was removed and split in multiple images.

Thumbnails' size is not known, as this data is not provided by the API.
2023-08-12 22:56:29 +02:00
AudricV 0a6011a50e
[PeerTube] Apply changes in InfoItemExtractors
Also lower the visibility of attributes of channels and playlists InfoItems to
private.
2023-08-12 22:56:29 +02:00
AudricV 6f8331524b
[PeerTube] Add utility method to get thumbnails of playlists and videos
This method, getThumbnailsFromPlaylistOrVideoItem, has been added in
PeertubeParsingHelper and returns the two image variants for playlists and
videos.
2023-08-12 22:56:28 +02:00
AudricV 81c0d80a54
[PeerTube] Add utility methods to get avatars and banners of accounts and channels
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
  getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
  getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
  JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
  JsonArray).
2023-08-12 22:56:28 +02:00
AudricV 31da5beb51
[SoundCloud] Apply changes in Extractors 2023-08-12 22:56:28 +02:00
AudricV a3a74cd566
[SoundCloud] Apply changes in InfoItemExtractors and return track user avatars as uploader avatars in SoundcloudStreamInfoItemExtractor 2023-08-12 22:56:28 +02:00
AudricV 7f818217d2
[SoundCloud] Add utility methods to get images from track JSON objects and image URLs
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):

- for artworks and avatars:
  - mini: 16x16;
  - t20x20: 20x20;
  - small: 32x32;
  - badge: 47x47;
  - t50x50: 50x50;
  - t60x60: 60x60;
  - t67x67: 67x67;
  - large: 100x100;
  - t120x120: 120x120;
  - t200x200: 200x200;
  - t240x240: 240x240;
  - t250x250: 250x250;
  - t300x300: 300x300;
  - t500x500: 500x500.

- for visuals/user banners:
  - t1240x260: 1240x260;
  - t2480x520: 2480x520.

Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
2023-08-12 22:56:28 +02:00
AudricV 266cd1f76b
[YouTube] Apply changes in YoutubeMusicSearchExtractor and split its InfoItemExtractors into separate classes
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).

These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.

The code of YoutubeMusicSearchExtractor have been also improved a bit.
2023-08-12 22:56:27 +02:00
AudricV c1981ed54f
[YouTube] Apply changes in Extractors except YoutubeMusicSearchExtractor
Also improve a bit some code related to the changes.
2023-08-12 22:56:27 +02:00
AudricV 4cc99f9ce1
[YouTube] Apply changes in InfoItemExtractors except YouTube Music ones 2023-08-12 22:56:27 +02:00
AudricV adfad086ac
[YouTube] Add utility methods to get images from InfoItems and thumbnails arrays
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.

These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
2023-08-12 22:56:27 +02:00
AudricV d56b880cae
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Infos 2023-08-12 22:56:26 +02:00
AudricV 9d8098576e
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Extractors 2023-08-12 22:56:26 +02:00
AudricV 0f4a5a8184
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemsCollectors 2023-08-12 22:56:26 +02:00
AudricV ca1d4a6fa4
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemExtractors 2023-08-12 22:56:26 +02:00
AudricV 2f3ee8a3f2
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItems 2023-08-12 22:56:25 +02:00
AudricV 78ce65769f
Add an ImageSuffix class to the extractor
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.

This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
2023-08-12 22:56:25 +02:00
AudricV d85454186a
Add an Image class to the extractor
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.

Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.

Getters of these properties are available and the constructor needs these four
properties.
2023-08-12 22:56:25 +02:00
Stypox 7294675aea
Merge pull request #1093 from AudricV/yt_support-shorts-ui-playlists
[YouTube] Support Shorts UI in playlists
2023-08-12 11:11:36 +02:00
Stypox 93a90b816d
Merge pull request #1094 from AudricV/yt_support-more-channel-headers
[YouTube] Support more channel headers
2023-08-12 11:08:30 +02:00
Stypox 44b664af15
[YouTube] Simplify Optional chains in channel 2023-08-12 11:02:51 +02:00
AudricV 2f7bfd3e7f
[YouTube] Add mocks of interactiveTabbedHeaderRenderer channel header test 2023-08-08 19:12:29 +02:00
AudricV b147904571
[YouTube] Add test for interactiveTabbedHeaderRenderer channel header
This test uses the Minecraft game topic channel.
2023-08-08 19:12:28 +02:00
AudricV 1852031a0b
[YouTube] Support pageHeaderRenderer and interactiveTabbedHeaderRenderer channel headers
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.

Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
2023-08-08 19:12:27 +02:00
AudricV 698c710685
Do not require knowledge of uploader in default StreamInfoItems tests
This change is required as some services can return no uploader info, such as
YouTube for playlists with a Shorts UI.
2023-08-07 19:43:15 +02:00
AudricV 8237052ef5
Fix wrong assertion in assertNotEmpty
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
2023-08-07 19:43:09 +02:00
AudricV 162c261577
[YouTube] Add mocks of the playlist with Shorts UI test 2023-08-07 19:07:53 +02:00
AudricV e2f4ee47b9
[YouTube] Add a playlist with Shorts UI test
The system Shorts videos uploads playlist of the YouTube official channel has
been chosen for this test.
2023-08-07 19:06:09 +02:00
AudricV e6f371fb94
[YouTube] Support Shorts UI in playlists
Also remove an outdated A/B test comment.
2023-08-07 19:01:08 +02:00
Stypox 7936987955
Merge pull request #1092 from Stypox/channel-tabs-improvements
Channel tabs code improvements
2023-08-06 21:44:10 +02:00
Stypox 6d2227111f
[YouTube] Assert that videos tab is ready after channel fetching 2023-08-06 21:14:57 +02:00
Stypox ee625c325c
Inherit from DefaultListExtractorTest in channel tab tests 2023-08-06 21:14:56 +02:00
Stypox 276c293889
Rename assertTabsContain 2023-08-06 21:14:56 +02:00
Stypox 9d3761a371
[YouTube] Directly use playlist collector in channel tabs wrapper
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
2023-08-06 21:13:25 +02:00
Stypox 95a3cc0a17
Merge pull request #1083 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.0
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
2023-08-06 18:58:27 +02:00
Stypox e34b4f1978
[YouTube] Avoid using Consumer 2023-08-06 13:02:31 +02:00
Stypox ef67c7cd74
[YouTube] Simplify usage of channel header json 2023-08-06 13:02:31 +02:00
Stypox a104cf3227
[YouTube] Fix docs in channel helper 2023-08-06 13:02:31 +02:00
Stypox bb47f05f89
Merge pull request #1091 from Stypox/update-mocks
[YouTube] Update stream mocks after #1087
2023-08-06 12:58:24 +02:00
Stypox 468bcc045d
[YouTube] Update mocks after #1087 2023-08-06 12:33:04 +02:00
Stypox 35f3a4ad01
Merge pull request #1082 from AudricV/channel-tabs-and-tags-support
Add support for channel tabs and channel tags
2023-08-06 12:21:42 +02:00
AudricV e7d64099a7
[YouTube] Update channel mocks and add channel tabs mocks 2023-08-06 12:15:06 +02:00
AudricV 684101c47d
[YouTube] Implement age-restricted channels support, link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV eaf2600ce0
[SoundCloud] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 0ee2072de5
[PeerTube] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV d3801dd0e9
[MediaCCC] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 8baec04611
[Bandcamp] Implement link handlers and channels tabs and tags changes on tests
Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV e0ba29cd19
Add utility method to assert that given channel tabs are in the ones returned by a channel extractor
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 18846baba7
Add tabs and tags methods in tests interfaces and annotate all methods with the Test JUnit annotation
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
ThetaDev c70a0e3543
Add a test for textual durations parsing using TimeAgoParser's patterns 2023-08-06 12:15:06 +02:00
AudricV 7366eab156
[YouTube] Add support for channel tabs and tags and age-restricted channels
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.

Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.

The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.

Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.

A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.

All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:15:04 +02:00
AudricV 4586067934
Add utility method to parse textual durations using TimeAgoParser's patterns
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:13:33 +02:00
AudricV d4bfe791ee
[SoundCloud] Add tabs support for users
Support of tracks, playlists and albums has been added for users.

Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:13:32 +02:00
AudricV 6f7d1f079f
[Bandcamp] Add tabs support for artists
Support of tracks and albums has been added for artists.

Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:19 +02:00
AudricV 1e8474b22d
[PeerTube] Add tabs support for accounts and video channels
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.

The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:15 +02:00
AudricV 652c2c8408
Add a ListLinkHandler which can be used to be returned from ChannelInfo.getTabs() when a specific tab's data has already been fetched
This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.

Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:12 +02:00
AudricV de823a6b68
Add an UnsupportedTabException exception class
This class makes easier for LinkHandlerFactory implementations to declare an
UnsupportedOperationException.
2023-08-06 12:11:12 +02:00
AudricV 76fb9dcdd7
Add UnsupportedOperationException to exceptions which can be thrown by getId and getUrl methods of LinkHandlerFactory and its base implementations
This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
2023-08-06 12:11:12 +02:00
AudricV 946eb9bd91
Add structure of channel tags
Tags' getters and/or setters have been added in ChannelExtractor and
ChannelInfo to do so.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:11:12 +02:00
AudricV 356a888d6c
Add structure of channel tabs
This commit introduces the following breaking changes:

- Three new classes have been added:
  - ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
  extracts InfoItems from a channel tab;
  - ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
  a ChannelTabExtractor and returns them as a ChannelTabInfo;
  - ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:11 +02:00
Stypox 3faaf4301c
Merge pull request #1087 from AudricV/yt_js-extractor-improvements-and-fixes
[YouTube] Improve and fix YoutubeJavaScriptExtractor
2023-08-06 12:01:00 +02:00
Stypox 8fb6ba36fa
Merge pull request #1081 from TeamNewPipe/fix/sc/search-next-page
[SoundCloud] Detect whether there are any more search results
2023-08-06 11:49:35 +02:00
Stypox 2947257111
[SoundCloud] Properly calculate if results have finished 2023-08-06 11:38:22 +02:00
Stypox 485bfbca9d
[SoundCloud] Move try-catch inside getOffsetFromUrl 2023-08-06 11:35:37 +02:00
Stypox 7c70fef197
Merge pull request #1089 from TeamNewPipe/ccc
[media.ccc.de] Only extract kiosk live stream rooms if they are streaming
2023-08-06 10:12:04 +02:00
Stypox de0a9bb797
Merge pull request #1088 from FireMasterK/pseudo-rng
Replace cryptographically secure random with regular random
2023-08-05 11:37:49 +02:00
TobiGr 340095515d Make Kiosk IDs accessible if possible 2023-08-05 03:18:40 +02:00
TobiGr fe27d6a0ec [media.ccc.de] Only extract live streams if the conference is streaming 2023-08-05 01:53:43 +02:00
Kavin 25082d78b0
Replace SecureRandom with Random 2023-08-03 23:00:02 +01:00
TobiGr aa6c17dc77 [SoundCloud] Deduplicate some code 2023-08-03 14:41:30 +02:00
TobiGr 2fb9922a15 [SoundCloud] Detect whether there are any more search results
Add test for this edge case.
2023-08-03 14:37:13 +02:00
AudricV a3d160edab
[YouTube] Improve and fix YoutubeJavaScriptExtractor
- Enhance documentation;
- Fix the regular expression fallback on HTML embed watch page;
- Use HTML scripts tag search first instead of the regular expression approach,
now used as a last resort;
- Compile regular expressions only once, in order to improve the performance of
subsequent extraction calls when clearing the cache;
- Provide original exceptions when fetching or parsing pages on which the base
JavaScript's player could be found failed, allowing clients to detect network
errors when they are the cause of the failures for instance;
- Remove delegate method which was not taking a video ID and hardcoding one, as
we can provide the video ID in all cases or do not provide a video ID at worse;
- Rename and make extraction methods package-private, as they are not intended
to be used publicly.

These breaking internal changes have been applied where needed, in
YoutubeJavaScriptExtractorTest and YoutubeStreamExtractor (in which an unneeded
initStsFromPlayerJsIfNeeded call have been removed).
2023-08-02 23:05:08 +02:00
dependabot[bot] bda3a3fc5d
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.3 to 5.10.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.3...r5.10.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-24 09:53:05 +00:00
668 changed files with 68730 additions and 34639 deletions

93
.github/ISSUE_TEMPLATE/bug_report.yml vendored Normal file
View File

@ -0,0 +1,93 @@
name: Bug report
description: Create a bug report to help us improve
labels: [bug]
body:
- type: markdown
attributes:
value: |
Thank you for helping to make the NewPipe Extractor better by reporting a bug. :hugs:
Please fill in as much information as possible about your bug so that we don't have to play "information ping-pong" and can help you immediately.
- type: checkboxes
id: checklist
attributes:
label: "Checklist"
options:
- label: "I am able to reproduce the bug with the latest version given here: [CLICK THIS LINK](https://github.com/TeamNewPipe/NewPipeExtractor/releases/latest)."
required: true
- label: "I am aware that this issue is being opened for the NewPipe Extractor, NOT the [app](https://github.com/TeamNewPipe/NewPipe), and my bug report will be dismissed otherwise."
required: true
- label: "I made sure that there are *no existing issues* - [open](https://github.com/TeamNewPipe/NewPipeExtractor/issues) or [closed](https://github.com/TeamNewPipe/NewPipeExtractor/issues?q=is%3Aissue+is%3Aclosed) - which I could contribute my information to."
required: true
- label: "I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise."
required: true
- label: "This issue contains only one bug."
required: true
- label: "I have read and understood the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md)."
required: true
- type: input
id: extractor-version
attributes:
label: Affected version
description: "In which NewPipe Extractor version did you encounter the bug?"
placeholder: "x.xx.x"
validations:
required: true
- type: textarea
id: steps-to-reproduce
attributes:
label: Steps to reproduce the bug
description: |
What did you do for the bug to show up?
If you can't cause the bug to show up again reliably (and hence don't have a proper set of steps to give us), please still try to give as many details as possible on how you think you encountered the bug.
placeholder: |
1. Init NewPipe with 'NewPipe.init(...)'
2. Create a StreamExtractor for xyz: 'StreamExtractor e = YouTube.getStreamExtractor(xyz.com)'
3. Get the description 'e.getDescription()'
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: Expected behavior
description: |
Tell us what you expect to happen.
- type: textarea
id: actual-behavior
attributes:
label: Actual behavior
description: |
Tell us what happens with the steps given above.
- type: textarea
id: screen-media
attributes:
label: Screenshots/Screen recordings
description: |
A picture or video is worth a thousand words.
If applicable, add screenshots or a screen recording to help explain your problem.
GitHub supports uploading them directly in the text box.
If your file is too big for Github to accept, try to compress it (ZIP-file) or feel free to paste a link to an image/video hoster here instead.
- type: textarea
id: logs
attributes:
label: Logs
description: |
If your bug includes a log you think we need to see, paste it here.
- type: textarea
id: additional-information
attributes:
label: Additional information
description: |
Any other information you'd like to include, for instance that
* your cat disabled your network connection
* ...

8
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: 💬 Matrix
url: https://matrix.to/#/#newpipe:matrix.newpipe-ev.de
about: Chat with us via Matrix for quick Q/A
- name: 💬 IRC
url: https://web.libera.chat/#newpipe
about: Chat with us via IRC for quick Q/A

50
.github/ISSUE_TEMPLATE/enhancement.yml vendored Normal file
View File

@ -0,0 +1,50 @@
name: Feature request
description: Suggest an idea for this project
labels: [enhancement]
body:
- type: markdown
attributes:
value: |
Thank you for helping to make the NewPipe Extractor better by suggesting a feature. :hugs:
Your ideas are highly welcome! The Extractor is made for you, the downstream developers, after all.
- type: checkboxes
id: checklist
attributes:
label: "Checklist"
options:
- label: "I am aware that this issue is being opened for the NewPipe Extractor, NOT the [app](https://github.com/TeamNewPipe/NewPipe), and my feature request will be dismissed otherwise."
required: true
- label: "I made sure that there are *no existing issues* - [open](https://github.com/TeamNewPipe/NewPipeExtractor/issues) or [closed](https://github.com/TeamNewPipe/NewPipeExtractor/issues?q=is%3Aissue+is%3Aclosed) - which I could contribute my information to."
required: true
- label: "I have taken the time to fill in all the required details. I understand that the feature request will be dismissed otherwise."
required: true
- label: "This issue contains only one feature request."
required: true
- label: "I have read and understood the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md)."
required: true
- type: textarea
id: feature-description
attributes:
label: Feature description
description: |
Explain how you want the Extractor's behavior to change to suit your needs.
validations:
required: true
- type: textarea
id: why-is-the-feature-requested
attributes:
label: Why do you want this feature?
description: |
Describe any problem or limitation you come across while using the Extractor which would be solved by this feature.
validations:
required: true
- type: textarea
id: additional-information
attributes:
label: Additional information
description: Any other information you'd like to include, for instance sketches, mockups, pictures of cats, etc.

View File

@ -17,16 +17,16 @@ jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: set up JDK 11
uses: actions/setup-java@v3
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
@ -44,7 +44,7 @@ jobs:
fi
- name: Upload test reports when failure occurs
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
if: failure()
with:
name: NewPipeExtractor-test-reports

View File

@ -13,16 +13,16 @@ jobs:
build-and-deploy-docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: set up JDK 11
uses: actions/setup-java@v3
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
@ -32,7 +32,7 @@ jobs:
run: ./gradlew aggregatedJavadocs
- name: Deploy JavaDocs
uses: peaceiris/actions-gh-pages@v3
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./build/docs

View File

@ -1,6 +1,6 @@
# NewPipe Extractor
[![CI](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml/badge.svg?branch=dev&event=schedule)](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml) [![JIT Pack Badge](https://jitpack.io/v/TeamNewPipe/NewPipeExtractor.svg)](https://jitpack.io/#TeamNewPipe/NewPipeExtractor) [JDoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) • [Documentation](https://teamnewpipe.github.io/documentation/)
[![CI](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml/badge.svg?branch=dev&event=schedule)](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml) [![JIT Pack Badge](https://jitpack.io/v/teamnewpipe/NewPipeExtractor.svg)](https://jitpack.io/#teamnewpipe/NewPipeExtractor) [JDoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) • [Documentation](https://teamnewpipe.github.io/documentation/)
NewPipe Extractor is a library for extracting things from streaming sites. It is a core component of [NewPipe](https://github.com/TeamNewPipe/NewPipe), but could be used independently.
@ -11,7 +11,7 @@ NewPipe Extractor is available at JitPack's Maven repo.
If you're using Gradle, you could add NewPipe Extractor as a dependency with the following steps:
1. Add `maven { url 'https://jitpack.io' }` to the `repositories` in your `build.gradle`.
2. Add `implementation 'com.github.TeamNewPipe:NewPipeExtractor:INSERT_VERSION_HERE'` to the `dependencies` in your `build.gradle`. Replace `INSERT_VERSION_HERE` with the [latest release](https://github.com/TeamNewPipe/NewPipeExtractor/releases/latest).
2. Add `implementation 'com.github.teamnewpipe:NewPipeExtractor:INSERT_VERSION_HERE'` to the `dependencies` in your `build.gradle`. Replace `INSERT_VERSION_HERE` with the [latest release](https://github.com/TeamNewPipe/NewPipeExtractor/releases/latest).
3. If you are using tools to minimize your project, make sure to keep the files below, by e.g. adding the following lines to your proguard file:
```
## Rules for NewPipeExtractor
@ -21,7 +21,7 @@ If you're using Gradle, you could add NewPipe Extractor as a dependency with the
-dontwarn org.mozilla.javascript.tools.**
```
**Note:** To use NewPipe Extractor in Android projects with a `minSdk` below 26, [API desugaring](https://developer.android.com/studio/write/java8-support#library-desugaring) is required. If the `minSdk` is below 19, the `desugar_jdk_libs_nio` artifact is required, which requires Android Gradle Plugin (AGP) version 7.4.0.
**Note:** To use NewPipe Extractor in Android projects with a `minSdk` below 33, [core library desugaring](https://developer.android.com/studio/write/java8-support#library-desugaring) with the `desugar_jdk_libs_nio` artifact is required.
### Testing changes
@ -30,7 +30,7 @@ To test changes quickly you can build the library locally. A good approach would
```groovy
includeBuild('../NewPipeExtractor') {
dependencySubstitution {
substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor')
substitute module('com.github.teamnewpipe:NewPipeExtractor') with project(':extractor')
}
}
```
@ -40,7 +40,7 @@ Another approach would be to use the local Maven repository, here's a gist of ho
1. Add `mavenLocal()` in your project `repositories` list (usually as the first entry to give priority above the others).
2. It's _recommended_ that you change the `version` of this library (e.g. `LOCAL_SNAPSHOT`).
3. Run gradle's `ìnstall` task to deploy this library to your local repository (using the wrapper, present in the root of this project: `./gradlew install`)
4. Change the dependency version used in your project to match the one you chose in step 2 (`implementation 'com.github.TeamNewPipe:NewPipeExtractor:LOCAL_SNAPSHOT'`)
4. Change the dependency version used in your project to match the one you chose in step 2 (`implementation 'com.github.teamnewpipe:NewPipeExtractor:LOCAL_SNAPSHOT'`)
> Tip for Android Studio users: After you make changes and run the `install` task, use the menu option `File → "Sync with File System"` to refresh the library in your project.

View File

@ -8,7 +8,7 @@ allprojects {
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
version 'v0.22.7'
version 'v0.24.3'
group 'com.github.TeamNewPipe'
repositories {
@ -28,8 +28,8 @@ allprojects {
ext {
nanojsonVersion = "1d9e1aea9049fc9f85e68b43ba39fe7be1c1f751"
spotbugsVersion = "4.7.3"
junitVersion = "5.9.3"
spotbugsVersion = "4.8.6"
junitVersion = "5.11.3"
checkstyleVersion = "10.4"
}
}

View File

@ -38,6 +38,8 @@
<module name="LineLength">
<property name="max" value="100"/>
<property name="fileExtensions" value="java"/>
<!-- Also allow links in javadocs to be longer (the default would just cover imports) -->
<property name="ignorePattern" value="^((package|import) .*)|( *\* (@see )?&lt;a href ?\= ?&quot;.*&quot;&gt;)$"/>
</module>
<!-- Checks for whitespace -->

View File

@ -26,12 +26,12 @@ dependencies {
implementation project(':timeago-parser')
implementation "com.github.TeamNewPipe:nanojson:$nanojsonVersion"
implementation 'org.jsoup:jsoup:1.16.1'
implementation 'org.jsoup:jsoup:1.17.2'
implementation "com.github.spotbugs:spotbugs-annotations:$spotbugsVersion"
// do not upgrade to 1.7.14, since in 1.7.14 Rhino uses the `SourceVersion` class, which is not
// available on Android (even when using desugaring), and `NoClassDefFoundError` is thrown
implementation 'org.mozilla:rhino:1.7.13'
implementation 'org.mozilla:rhino:1.7.15'
checkstyle "com.puppycrawl.tools:checkstyle:$checkstyleVersion"
@ -41,5 +41,5 @@ dependencies {
testImplementation 'org.junit.jupiter:junit-jupiter-params'
testImplementation "com.squareup.okhttp3:okhttp:3.12.13"
testImplementation 'com.google.code.gson:gson:2.10.1'
testImplementation 'com.google.code.gson:gson:2.11.0'
}

View File

@ -0,0 +1,211 @@
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.Objects;
/**
* Class representing images in the extractor.
*
* <p>
* An image has four properties: its URL, its height, its width and its estimated quality level.
* </p>
*
* <p>
* Depending of the services, the height, the width or both properties may be not known.
* Implementations <b>must use</b> the relevant unknown constants in this case
* ({@link #HEIGHT_UNKNOWN} and {@link #WIDTH_UNKNOWN}), to ensure properly the lack of knowledge
* of one or both of these properties to extractor clients.
* </p>
*
* <p>
* They should also respect the ranges defined in the estimated image resolution levels as much as
* possible, to ensure consistency to extractor clients.
* </p>
*/
public final class Image implements Serializable {
/**
* Constant representing that the height of an {@link Image} is unknown.
*/
public static final int HEIGHT_UNKNOWN = -1;
/**
* Constant representing that the width of an {@link Image} is unknown.
*/
public static final int WIDTH_UNKNOWN = -1;
@Nonnull
private final String url;
private final int height;
private final int width;
@Nonnull
private final ResolutionLevel estimatedResolutionLevel;
/**
* Construct an {@link Image} instance.
*
* @param url the URL to the image, which should be not null or empty
* @param height the image's height
* @param width the image's width
* @param estimatedResolutionLevel the image's estimated resolution level, which must not be
* null
* @throws NullPointerException if {@code estimatedResolutionLevel} is null
*/
public Image(@Nonnull final String url,
final int height,
final int width,
@Nonnull final ResolutionLevel estimatedResolutionLevel)
throws NullPointerException {
this.url = url;
this.height = height;
this.width = width;
this.estimatedResolutionLevel = Objects.requireNonNull(
estimatedResolutionLevel, "estimatedResolutionLevel is null");
}
/**
* Get the URL of this {@link Image}.
*
* @return the {@link Image}'s URL.
*/
@Nonnull
public String getUrl() {
return url;
}
/**
* Get the height of this {@link Image}.
*
* <p>
* If it is unknown, {@link #HEIGHT_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s height or {@link #HEIGHT_UNKNOWN}
*/
public int getHeight() {
return height;
}
/**
* Get the width of this {@link Image}.
*
* <p>
* If it is unknown, {@link #WIDTH_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s width or {@link #WIDTH_UNKNOWN}
*/
public int getWidth() {
return width;
}
/**
* Get the estimated resolution level of this image.
*
* <p>
* If it is unknown, {@link ResolutionLevel#UNKNOWN} is returned instead.
* </p>
*
* @return the estimated resolution level, which is never {@code null}
* @see ResolutionLevel
*/
@Nonnull
public ResolutionLevel getEstimatedResolutionLevel() {
return estimatedResolutionLevel;
}
/**
* Get a string representation of this {@link Image} instance.
*
* <p>
* The representation will be in the following format, where {@code url}, {@code height},
* {@code width} and {@code estimatedResolutionLevel} represent the corresponding properties:
* <br>
* <br>
* {@code Image {url=url, height='height, width=width,
* estimatedResolutionLevel=estimatedResolutionLevel}'}
* </p>
*
* @return a string representation of this {@link Image} instance
*/
@Nonnull
@Override
public String toString() {
return "Image {" + "url=" + url + ", height=" + height + ", width=" + width
+ ", estimatedResolutionLevel=" + estimatedResolutionLevel + "}";
}
/**
* The estimated resolution level of an {@link Image}.
*
* <p>
* Some services don't return the size of their images, but we may know for a specific image
* type that a service returns, according to real data, an approximation of the resolution
* level.
* </p>
*/
public enum ResolutionLevel {
/**
* The high resolution level.
*
* <p>
* This level applies to images with a height greater than or equal to 720px.
* </p>
*/
HIGH,
/**
* The medium resolution level.
*
* <p>
* This level applies to images with a height between 175px inclusive and 720px exclusive.
* </p>
*/
MEDIUM,
/**
* The low resolution level.
*
* <p>
* This level applies to images with a height between 1px inclusive and 175px exclusive.
* </p>
*/
LOW,
/**
* The unknown resolution level.
*
* <p>
* This value is returned when the extractor doesn't know what resolution level an image
* could have, for example if the extractor loops in an array of images with different
* resolution levels without knowing the height.
* </p>
*/
UNKNOWN;
/**
* Get a {@link ResolutionLevel} based from the given height.
*
* @param heightPx the height from which returning the good {@link ResolutionLevel}
* @return the {@link ResolutionLevel} corresponding to the height provided. See the
* {@link ResolutionLevel} values for details about what value is returned.
*/
public static ResolutionLevel fromHeight(final int heightPx) {
if (heightPx <= 0) {
return UNKNOWN;
}
if (heightPx < 175) {
return LOW;
}
if (heightPx < 720) {
return MEDIUM;
}
return HIGH;
}
}
}

View File

@ -1,33 +1,36 @@
package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
public abstract class InfoItem implements Serializable {
private final InfoType infoType;
private final int serviceId;
private final String url;
private final String name;
private String thumbnailUrl;
@Nonnull
private List<Image> thumbnails = List.of();
public InfoItem(final InfoType infoType,
final int serviceId,
@ -55,12 +58,13 @@ public abstract class InfoItem implements Serializable {
return name;
}
public void setThumbnailUrl(final String thumbnailUrl) {
this.thumbnailUrl = thumbnailUrl;
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
public String getThumbnailUrl() {
return thumbnailUrl;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
@Override

View File

@ -2,8 +2,12 @@ package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
public interface InfoItemExtractor {
String getName() throws ParsingException;
String getUrl() throws ParsingException;
String getThumbnailUrl() throws ParsingException;
@Nonnull
List<Image> getThumbnails() throws ParsingException;
}

View File

@ -12,21 +12,21 @@ import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemExtractor>

View File

@ -3,23 +3,23 @@ package org.schabi.newpipe.extractor;
/*
* Created by Adam Howard on 08/11/15.
*
* Copyright (c) Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com> 2015
* Copyright (c) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com>
*
* MediaFormat.java is part of NewPipe.
* MediaFormat.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import javax.annotation.Nonnull;
@ -51,7 +51,7 @@ public enum MediaFormat {
WEBMA_OPUS(0x200, "WebM Opus", "webm", "audio/webm"),
AIFF (0x600, "AIFF", "aiff", "audio/aiff"),
/**
* Same as {@link MediaFormat.AIFF}, just with the shorter suffix/file extension
* Same as {@link MediaFormat#AIFF}, just with the shorter suffix/file extension
*/
AIF (0x600, "AIFF", "aif", "audio/aiff"),
WAV (0x700, "WAV", "wav", "audio/wav"),

View File

@ -15,21 +15,21 @@ import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 23.08.15.
*
* Copyright (C) Christian Schabesberger 2015 <chris.schabesberger@mailbox.org>
* NewPipe.java is part of NewPipe.
* Copyright (C) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* NewPipe Extractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.downloader.Downloader;

View File

@ -9,21 +9,21 @@ import org.schabi.newpipe.extractor.services.youtube.YoutubeService;
import java.util.List;
/*
* Copyright (C) Christian Schabesberger 2018 <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe.
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**

View File

@ -1,6 +1,7 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
@ -28,21 +29,21 @@ import java.util.Collections;
import java.util.List;
/*
* Copyright (C) Christian Schabesberger 2018 <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe.
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class StreamingService {
@ -140,6 +141,14 @@ public abstract class StreamingService {
*/
public abstract ListLinkHandlerFactory getChannelLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for channel tabs.
* If support for channel tabs is not given null must be returned.
*
* @return an instance of a ListLinkHandlerFactory for channels or null
*/
public abstract ListLinkHandlerFactory getChannelTabLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for playlists.
* If support for playlists is not given null must be returned.
@ -204,6 +213,15 @@ public abstract class StreamingService {
public abstract ChannelExtractor getChannelExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must create a new instance of a ChannelTabExtractor implementation.
*
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelTabExtractor
*/
public abstract ChannelTabExtractor getChannelTabExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must crete a new instance of a PlaylistExtractor implementation.
* @param linkHandler is pointing to the playlist which should be handled by this new instance.
@ -262,6 +280,20 @@ public abstract class StreamingService {
return getChannelExtractor(getChannelLHFactory().fromUrl(url));
}
public ChannelTabExtractor getChannelTabExtractorFromId(final String id, final String tab)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), ""));
}
public ChannelTabExtractor getChannelTabExtractorFromIdAndBaseUrl(final String id,
final String tab,
final String baseUrl)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), "", baseUrl));
}
public PlaylistExtractor getPlaylistExtractor(final String url) throws ExtractionException {
return getPlaylistExtractor(getPlaylistLHFactory().fromUrl(url));
}

View File

@ -1,47 +1,58 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
/*
* Created by Christian Schabesberger on 25.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public abstract class ChannelExtractor extends ListExtractor<StreamInfoItem> {
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.Extractor;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
import java.util.List;
public abstract class ChannelExtractor extends Extractor {
public static final long UNKNOWN_SUBSCRIBER_COUNT = -1;
public ChannelExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
protected ChannelExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
public abstract String getAvatarUrl() throws ParsingException;
public abstract String getBannerUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getAvatars() throws ParsingException;
@Nonnull
public abstract List<Image> getBanners() throws ParsingException;
public abstract String getFeedUrl() throws ParsingException;
public abstract long getSubscriberCount() throws ParsingException;
public abstract String getDescription() throws ParsingException;
public abstract String getParentChannelName() throws ParsingException;
public abstract String getParentChannelUrl() throws ParsingException;
public abstract String getParentChannelAvatarUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getParentChannelAvatars() throws ParsingException;
public abstract boolean isVerified() throws ParsingException;
@Nonnull
public abstract List<ListLinkHandler> getTabs() throws ParsingException;
@Nonnull
public List<String> getTags() throws ParsingException {
return List.of();
}
}

View File

@ -1,47 +1,45 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
/*
* Created by Christian Schabesberger on 31.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public class ChannelInfo extends ListInfo<StreamInfoItem> {
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.Info;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nonnull;
public class ChannelInfo extends Info {
public ChannelInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final ListLinkHandler listLinkHandler) {
super(serviceId, id, url, originalUrl, name, listLinkHandler.getContentFilters(),
listLinkHandler.getSortFilter());
final String name) {
super(serviceId, id, url, originalUrl, name);
}
public static ChannelInfo getInfo(final String url) throws IOException, ExtractionException {
@ -55,13 +53,6 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return getInfo(extractor);
}
public static InfoItemsPage<StreamInfoItem> getMoreItems(final StreamingService service,
final String url,
final Page page)
throws IOException, ExtractionException {
return service.getChannelExtractor(url).getPage(page);
}
public static ChannelInfo getInfo(final ChannelExtractor extractor)
throws IOException, ExtractionException {
@ -71,35 +62,32 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
final String originalUrl = extractor.getOriginalUrl();
final String name = extractor.getName();
final ChannelInfo info =
new ChannelInfo(serviceId, id, url, originalUrl, name, extractor.getLinkHandler());
final ChannelInfo info = new ChannelInfo(serviceId, id, url, originalUrl, name);
try {
info.setAvatarUrl(extractor.getAvatarUrl());
info.setAvatars(extractor.getAvatars());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setBannerUrl(extractor.getBannerUrl());
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setFeedUrl(extractor.getFeedUrl());
} catch (final Exception e) {
info.addError(e);
}
final InfoItemsPage<StreamInfoItem> itemsPage =
ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPage(itemsPage.getNextPage());
try {
info.setSubscriberCount(extractor.getSubscriberCount());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setDescription(extractor.getDescription());
} catch (final Exception e) {
@ -119,7 +107,7 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
}
try {
info.setParentChannelAvatarUrl(extractor.getParentChannelAvatarUrl());
info.setParentChannelAvatars(extractor.getParentChannelAvatars());
} catch (final Exception e) {
info.addError(e);
}
@ -130,19 +118,36 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
info.addError(e);
}
try {
info.setTabs(extractor.getTabs());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTags(extractor.getTags());
} catch (final Exception e) {
info.addError(e);
}
return info;
}
private String avatarUrl;
private String parentChannelName;
private String parentChannelUrl;
private String parentChannelAvatarUrl;
private String bannerUrl;
private String feedUrl;
private long subscriberCount = -1;
private String description;
private String[] donationLinks;
@Nonnull
private List<Image> avatars = List.of();
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> parentChannelAvatars = List.of();
private boolean verified;
private List<ListLinkHandler> tabs = List.of();
private List<String> tags = List.of();
public String getParentChannelName() {
return parentChannelName;
@ -160,28 +165,31 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
this.parentChannelUrl = parentChannelUrl;
}
public String getParentChannelAvatarUrl() {
return parentChannelAvatarUrl;
@Nonnull
public List<Image> getParentChannelAvatars() {
return parentChannelAvatars;
}
public void setParentChannelAvatarUrl(final String parentChannelAvatarUrl) {
this.parentChannelAvatarUrl = parentChannelAvatarUrl;
public void setParentChannelAvatars(@Nonnull final List<Image> parentChannelAvatars) {
this.parentChannelAvatars = parentChannelAvatars;
}
public String getAvatarUrl() {
return avatarUrl;
@Nonnull
public List<Image> getAvatars() {
return avatars;
}
public void setAvatarUrl(final String avatarUrl) {
this.avatarUrl = avatarUrl;
public void setAvatars(@Nonnull final List<Image> avatars) {
this.avatars = avatars;
}
public String getBannerUrl() {
return bannerUrl;
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBannerUrl(final String bannerUrl) {
this.bannerUrl = bannerUrl;
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getFeedUrl() {
@ -223,4 +231,22 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
public void setVerified(final boolean verified) {
this.verified = verified;
}
@Nonnull
public List<ListLinkHandler> getTabs() {
return tabs;
}
public void setTabs(@Nonnull final List<ListLinkHandler> tabs) {
this.tabs = tabs;
}
@Nonnull
public List<String> getTags() {
return tags;
}
public void setTags(@Nonnull final List<String> tags) {
this.tags = tags;
}
}

View File

@ -5,21 +5,21 @@ import org.schabi.newpipe.extractor.InfoItem;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ChannelInfoItem extends InfoItem {

View File

@ -6,21 +6,21 @@ import org.schabi.newpipe.extractor.exceptions.ParsingException;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public interface ChannelInfoItemExtractor extends InfoItemExtractor {

View File

@ -1,28 +1,28 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
public final class ChannelInfoItemsCollector
extends InfoItemsCollector<ChannelInfoItem, ChannelInfoItemExtractor> {
public ChannelInfoItemsCollector(final int serviceId) {
@ -47,7 +47,7 @@ public final class ChannelInfoItemsCollector
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}

View File

@ -0,0 +1,25 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
/**
* A {@link ListExtractor} of {@link InfoItem}s for tabs of channels.
*/
public abstract class ChannelTabExtractor extends ListExtractor<InfoItem> {
protected ChannelTabExtractor(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public String getName() {
return getLinkHandler().getContentFilters().get(0);
}
}

View File

@ -0,0 +1,70 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelInfo;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
public class ChannelTabInfo extends ListInfo<InfoItem> {
public ChannelTabInfo(final int serviceId,
@Nonnull final ListLinkHandler linkHandler) {
super(serviceId, linkHandler, linkHandler.getContentFilters().get(0));
}
/**
* Get a {@link ChannelTabInfo} instance from the given service and tab handler.
*
* @param service streaming service
* @param linkHandler Channel tab handler (from {@link ChannelInfo})
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler)
throws ExtractionException, IOException {
final ChannelTabExtractor extractor = service.getChannelTabExtractor(linkHandler);
extractor.fetchPage();
return getInfo(extractor);
}
/**
* Get a {@link ChannelTabInfo} instance from a {@link ChannelTabExtractor}.
*
* @param extractor an extractor where {@code fetchPage()} was already got called on
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final ChannelTabExtractor extractor) {
final ChannelTabInfo info =
new ChannelTabInfo(extractor.getServiceId(), extractor.getLinkHandler());
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (final Exception e) {
info.addError(e);
}
final ListExtractor.InfoItemsPage<InfoItem> page
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(page.getItems());
info.setNextPage(page.getNextPage());
return info;
}
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(
@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler,
@Nonnull final Page page) throws ExtractionException, IOException {
return service.getChannelTabExtractor(linkHandler).getPage(page);
}
}

View File

@ -0,0 +1,17 @@
package org.schabi.newpipe.extractor.channel.tabs;
/**
* Constants of channel tabs supported by the extractor.
*/
public final class ChannelTabs {
public static final String VIDEOS = "videos";
public static final String TRACKS = "tracks";
public static final String SHORTS = "shorts";
public static final String LIVESTREAMS = "livestreams";
public static final String CHANNELS = "channels";
public static final String PLAYLISTS = "playlists";
public static final String ALBUMS = "albums";
private ChannelTabs() {
}
}

View File

@ -1,18 +1,23 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public class CommentsInfoItem extends InfoItem {
private String commentId;
private Description commentText;
@Nonnull
private Description commentText = Description.EMPTY_DESCRIPTION;
private String uploaderName;
private String uploaderAvatarUrl;
@Nonnull
private List<Image> uploaderAvatars = List.of();
private String uploaderUrl;
private boolean uploaderVerified;
private String textualUploadDate;
@ -26,6 +31,8 @@ public class CommentsInfoItem extends InfoItem {
private int replyCount;
@Nullable
private Page replies;
private boolean isChannelOwner;
private boolean creatorReply;
public static final int NO_LIKE_COUNT = -1;
public static final int NO_STREAM_POSITION = -1;
@ -44,11 +51,12 @@ public class CommentsInfoItem extends InfoItem {
this.commentId = commentId;
}
@Nonnull
public Description getCommentText() {
return commentText;
}
public void setCommentText(final Description commentText) {
public void setCommentText(@Nonnull final Description commentText) {
this.commentText = commentText;
}
@ -60,12 +68,13 @@ public class CommentsInfoItem extends InfoItem {
this.uploaderName = uploaderName;
}
public String getUploaderAvatarUrl() {
return uploaderAvatarUrl;
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatarUrl(final String uploaderAvatarUrl) {
this.uploaderAvatarUrl = uploaderAvatarUrl;
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getUploaderUrl() {
@ -94,8 +103,8 @@ public class CommentsInfoItem extends InfoItem {
}
/**
* @return the comment's like count
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable
* @return the comment's like count or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is
* unavailable
*/
public int getLikeCount() {
return likeCount;
@ -167,4 +176,22 @@ public class CommentsInfoItem extends InfoItem {
public Page getReplies() {
return this.replies;
}
public void setChannelOwner(final boolean channelOwner) {
this.isChannelOwner = channelOwner;
}
public boolean isChannelOwner() {
return isChannelOwner;
}
public void setCreatorReply(final boolean creatorReply) {
this.creatorReply = creatorReply;
}
public boolean hasCreatorReply() {
return creatorReply;
}
}

View File

@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
@ -8,7 +9,9 @@ import org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeCommentsI
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public interface CommentsInfoItemExtractor extends InfoItemExtractor {
@ -42,6 +45,7 @@ public interface CommentsInfoItemExtractor extends InfoItemExtractor {
/**
* The text of the comment
*/
@Nonnull
default Description getCommentText() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
@ -77,8 +81,9 @@ public interface CommentsInfoItemExtractor extends InfoItemExtractor {
return "";
}
default String getUploaderAvatarUrl() throws ParsingException {
return "";
@Nonnull
default List<Image> getUploaderAvatars() throws ParsingException {
return List.of();
}
/**
@ -130,4 +135,18 @@ public interface CommentsInfoItemExtractor extends InfoItemExtractor {
default Page getReplies() throws ParsingException {
return null;
}
/**
* Whether the comment was made by the channel owner.
*/
default boolean isChannelOwner() throws ParsingException {
return false;
}
/**
* Whether the comment was replied to by the creator.
*/
default boolean hasCreatorReply() throws ParsingException {
return false;
}
}

View File

@ -36,7 +36,7 @@ public final class CommentsInfoItemsCollector
addError(e);
}
try {
resultItem.setUploaderAvatarUrl(extractor.getUploaderAvatarUrl());
resultItem.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
addError(e);
}
@ -66,7 +66,7 @@ public final class CommentsInfoItemsCollector
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
@ -101,6 +101,20 @@ public final class CommentsInfoItemsCollector
addError(e);
}
try {
resultItem.setChannelOwner(extractor.isChannelOwner());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setCreatorReply(extractor.hasCreatorReply());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 30.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ExtractionException extends Exception {

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 12.09.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class FoundAdException extends ParsingException {

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 31.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by beneth <bmauduit@beneth.fr> on 07.12.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ReCaptchaException extends ExtractionException {

View File

@ -0,0 +1,7 @@
package org.schabi.newpipe.extractor.exceptions;
public final class UnsupportedTabException extends UnsupportedOperationException {
public UnsupportedTabException(final String unsupportedTab) {
super("Unsupported tab " + unsupportedTab);
}
}

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.InfoItem;

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.ListExtractor;

View File

@ -8,21 +8,21 @@ import java.util.Objects;
/*
* Created by Christian Schabesberger on 26.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class LinkHandlerFactory {
@ -31,13 +31,14 @@ public abstract class LinkHandlerFactory {
// To Override
///////////////////////////////////
public abstract String getId(String url) throws ParsingException;
public abstract String getId(String url) throws ParsingException, UnsupportedOperationException;
public abstract String getUrl(String id) throws ParsingException;
public abstract String getUrl(String id) throws ParsingException, UnsupportedOperationException;
public abstract boolean onAcceptUrl(String url) throws ParsingException;
public String getUrl(final String id, final String baseUrl) throws ParsingException {
public String getUrl(final String id, final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id);
}

View File

@ -14,12 +14,13 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
///////////////////////////////////
public abstract String getUrl(String id, List<String> contentFilter, String sortFilter)
throws ParsingException;
throws ParsingException, UnsupportedOperationException;
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter,
final String baseUrl) throws ParsingException {
final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id, contentFilter, sortFilter);
}
@ -72,7 +73,7 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
*
* @return the url corresponding to id without any filters applied
*/
public String getUrl(final String id) throws ParsingException {
public String getUrl(final String id) throws ParsingException, UnsupportedOperationException {
return getUrl(id, new ArrayList<>(0), "");
}

View File

@ -0,0 +1,54 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
/**
* A {@link ListLinkHandler} which can be used to be returned from {@link
* org.schabi.newpipe.extractor.channel.ChannelInfo#getTabs() ChannelInfo#getTabs()} when a
* specific tab's data has already been fetched.
*
* <p>
* This class allows passing a builder for a {@link ChannelTabExtractor} that can hold references
* to variables.
* </p>
*
* <p>
* Note: a service that wishes to use this class in one of its {@link
* org.schabi.newpipe.extractor.channel.ChannelExtractor ChannelExtractor}s must also add the
* following snippet of code in the service's
* {@link StreamingService#getChannelTabExtractor(ListLinkHandler)}:
* <pre>
* if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
* return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
* }
* </pre>
*/
public class ReadyChannelTabListLinkHandler extends ListLinkHandler {
public interface ChannelTabExtractorBuilder extends Serializable {
@Nonnull
ChannelTabExtractor build(@Nonnull StreamingService service,
@Nonnull ListLinkHandler linkHandler);
}
private final ChannelTabExtractorBuilder extractorBuilder;
public ReadyChannelTabListLinkHandler(
final String url,
final String channelId,
@Nonnull final String channelTab,
@Nonnull final ChannelTabExtractorBuilder extractorBuilder) {
super(url, url, channelId, List.of(channelTab), "");
this.extractorBuilder = extractorBuilder;
}
@Nonnull
public ChannelTabExtractor getChannelTabExtractor(@Nonnull final StreamingService service) {
return extractorBuilder.build(service, new ListLinkHandler(this));
}
}

View File

@ -13,7 +13,7 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
@Override
public abstract String getUrl(String query, List<String> contentFilter, String sortFilter)
throws ParsingException;
throws ParsingException, UnsupportedOperationException;
@SuppressWarnings("unused")
public String getSearchString(final String url) {
@ -25,7 +25,7 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
///////////////////////////////////
@Override
public String getId(final String url) {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return getSearchString(url);
}

View File

@ -11,10 +11,12 @@ import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class Localization implements Serializable {
public static final Localization DEFAULT = new Localization("en", "GB");
@ -26,20 +28,28 @@ public class Localization implements Serializable {
/**
* @param localizationCodeList a list of localization code, formatted like {@link
* #getLocalizationCode()}
* @throws IllegalArgumentException If any of the localizationCodeList is formatted incorrectly
* @return list of Localization objects
*/
@Nonnull
public static List<Localization> listFrom(final String... localizationCodeList) {
final List<Localization> toReturn = new ArrayList<>();
for (final String localizationCode : localizationCodeList) {
toReturn.add(fromLocalizationCode(localizationCode));
toReturn.add(fromLocalizationCode(localizationCode)
.orElseThrow(() -> new IllegalArgumentException(
"Not a localization code: " + localizationCode
)));
}
return Collections.unmodifiableList(toReturn);
}
/**
* @param localizationCode a localization code, formatted like {@link #getLocalizationCode()}
* @return A Localization, if the code was valid.
*/
public static Localization fromLocalizationCode(final String localizationCode) {
return fromLocale(LocaleCompat.forLanguageTag(localizationCode));
@Nonnull
public static Optional<Localization> fromLocalizationCode(final String localizationCode) {
return LocaleCompat.forLanguageTag(localizationCode).map(Localization::fromLocale);
}
public Localization(@Nonnull final String languageCode, @Nullable final String countryCode) {
@ -61,10 +71,6 @@ public class Localization implements Serializable {
return countryCode == null ? "" : countryCode;
}
public Locale asLocale() {
return new Locale(getLanguageCode(), getCountryCode());
}
public static Localization fromLocale(@Nonnull final Locale locale) {
return new Localization(locale.getLanguage(), locale.getCountry());
}
@ -72,6 +78,8 @@ public class Localization implements Serializable {
/**
* Return a formatted string in the form of: {@code language-Country}, or
* just {@code language} if country is {@code null}.
*
* @return A correctly formatted localizationCode for this localization.
*/
public String getLocalizationCode() {
return languageCode + (countryCode == null ? "" : "-" + countryCode);

View File

@ -7,14 +7,21 @@ import org.schabi.newpipe.extractor.utils.Parser;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.util.regex.MatchResult;
/**
* A helper class that is meant to be used by services that need to parse upload dates in the
* format '2 days ago' or similar.
* A helper class that is meant to be used by services that need to parse durations such as
* {@code 23 seconds} and/or upload dates in the format {@code 2 days ago} or similar.
*/
public class TimeAgoParser {
private static final Pattern DURATION_PATTERN = Pattern.compile("(?:(\\d+) )?([A-z]+)");
private final PatternsHolder patternsHolder;
private final OffsetDateTime now;
@ -60,6 +67,48 @@ public class TimeAgoParser {
return getResultFor(parseTimeAgoAmount(textualDate), parseChronoUnit(textualDate));
}
/**
* Parses a textual duration into a duration computer number.
*
* @param textualDuration the textual duration to parse
* @return the textual duration parsed, as a primitive {@code long}
* @throws ParsingException if the textual duration could not be parsed
*/
public long parseDuration(final String textualDuration) throws ParsingException {
// We can't use Matcher.results, as it is only available on Android 14 and above
final Matcher matcher = DURATION_PATTERN.matcher(textualDuration);
final List<MatchResult> results = new ArrayList<>();
while (matcher.find()) {
results.add(matcher.toMatchResult());
}
return results.stream()
.map(match -> {
final String digits = match.group(1);
final String word = match.group(2);
int amount;
try {
amount = Integer.parseInt(digits);
} catch (final NumberFormatException ignored) {
amount = 1;
}
final ChronoUnit unit;
try {
unit = parseChronoUnit(word);
} catch (final ParsingException ignored) {
return 0L;
}
return amount * unit.getDuration().getSeconds();
})
.filter(n -> n > 0)
.reduce(Long::sum)
.orElseThrow(() -> new ParsingException(
"Could not parse duration \"" + textualDuration + "\""));
}
private int parseTimeAgoAmount(final String textualDate) {
try {
return Integer.parseInt(textualDate.replaceAll("\\D+", ""));

View File

@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
@ -9,6 +10,9 @@ import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import javax.annotation.Nonnull;
import java.util.Collections;
import java.util.List;
public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
public PlaylistExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
@ -17,7 +21,8 @@ public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
public abstract String getUploaderUrl() throws ParsingException;
public abstract String getUploaderName() throws ParsingException;
public abstract String getUploaderAvatarUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getUploaderAvatars() throws ParsingException;
public abstract boolean isUploaderVerified() throws ParsingException;
public abstract long getStreamCount() throws ParsingException;
@ -26,15 +31,13 @@ public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
public abstract Description getDescription() throws ParsingException;
@Nonnull
public String getThumbnailUrl() throws ParsingException {
return "";
public List<Image> getThumbnails() throws ParsingException {
return Collections.emptyList();
}
@Nonnull
public String getBannerUrl() throws ParsingException {
// Banner can't be handled by frontend right now.
// Whoever is willing to implement this should also implement it in the frontend.
return "";
public List<Image> getBanners() throws ParsingException {
return List.of();
}
@Nonnull
@ -48,8 +51,8 @@ public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
}
@Nonnull
public String getSubChannelAvatarUrl() throws ParsingException {
return "";
public List<Image> getSubChannelAvatars() throws ParsingException {
return List.of();
}
public PlaylistInfo.PlaylistType getPlaylistType() throws ParsingException {

View File

@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
@ -12,6 +13,7 @@ import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@ -109,26 +111,23 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
info.addError(e);
}
try {
info.setThumbnailUrl(extractor.getThumbnailUrl());
info.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
info.setUploaderUrl("");
uploaderParsingErrors.add(e);
}
try {
info.setUploaderName(extractor.getUploaderName());
} catch (final Exception e) {
info.setUploaderName("");
uploaderParsingErrors.add(e);
}
try {
info.setUploaderAvatarUrl(extractor.getUploaderAvatarUrl());
info.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
info.setUploaderAvatarUrl("");
uploaderParsingErrors.add(e);
}
try {
@ -142,12 +141,12 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelAvatarUrl(extractor.getSubChannelAvatarUrl());
info.setSubChannelAvatars(extractor.getSubChannelAvatars());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setBannerUrl(extractor.getBannerUrl());
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
@ -171,32 +170,38 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
return info;
}
private String thumbnailUrl;
private String bannerUrl;
private String uploaderUrl;
private String uploaderName;
private String uploaderAvatarUrl;
private String uploaderUrl = "";
private String uploaderName = "";
private String subChannelUrl;
private String subChannelName;
private String subChannelAvatarUrl;
private long streamCount = 0;
private Description description;
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> subChannelAvatars = List.of();
@Nonnull
private List<Image> thumbnails = List.of();
@Nonnull
private List<Image> uploaderAvatars = List.of();
private long streamCount;
private PlaylistType playlistType;
public String getThumbnailUrl() {
return thumbnailUrl;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
public void setThumbnailUrl(final String thumbnailUrl) {
this.thumbnailUrl = thumbnailUrl;
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
public String getBannerUrl() {
return bannerUrl;
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBannerUrl(final String bannerUrl) {
this.bannerUrl = bannerUrl;
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getUploaderUrl() {
@ -215,12 +220,13 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
this.uploaderName = uploaderName;
}
public String getUploaderAvatarUrl() {
return uploaderAvatarUrl;
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatarUrl(final String uploaderAvatarUrl) {
this.uploaderAvatarUrl = uploaderAvatarUrl;
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getSubChannelUrl() {
@ -239,12 +245,13 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
this.subChannelName = subChannelName;
}
public String getSubChannelAvatarUrl() {
return subChannelAvatarUrl;
@Nonnull
public List<Image> getSubChannelAvatars() {
return subChannelAvatars;
}
public void setSubChannelAvatarUrl(final String subChannelAvatarUrl) {
this.subChannelAvatarUrl = subChannelAvatarUrl;
public void setSubChannelAvatars(@Nonnull final List<Image> subChannelAvatars) {
this.subChannelAvatars = subChannelAvatars;
}
public long getStreamCount() {

View File

@ -32,7 +32,7 @@ public class PlaylistInfoItemsCollector
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}

View File

@ -19,7 +19,7 @@ public class SearchInfo extends ListInfo<InfoItem> {
private final String searchString;
private String searchSuggestion;
private boolean isCorrectedSearch;
private List<MetaInfo> metaInfo;
private List<MetaInfo> metaInfo = List.of();
public SearchInfo(final int serviceId,
final SearchQueryHandler qIHandler,

View File

@ -12,6 +12,7 @@ import static org.schabi.newpipe.extractor.services.bandcamp.extractors.Bandcamp
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.kiosk.KioskList;
@ -19,11 +20,13 @@ import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelTabExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampCommentsExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor;
@ -34,6 +37,7 @@ import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSearchE
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSuggestionExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampCommentsLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampFeaturedLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampPlaylistLinkHandlerFactory;
@ -58,27 +62,32 @@ public class BandcampService extends StreamingService {
@Override
public LinkHandlerFactory getStreamLHFactory() {
return new BandcampStreamLinkHandlerFactory();
return BandcampStreamLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelLHFactory() {
return new BandcampChannelLinkHandlerFactory();
return BandcampChannelLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelTabLHFactory() {
return BandcampChannelTabLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getPlaylistLHFactory() {
return new BandcampPlaylistLinkHandlerFactory();
return BandcampPlaylistLinkHandlerFactory.getInstance();
}
@Override
public SearchQueryHandlerFactory getSearchQHFactory() {
return new BandcampSearchQueryHandlerFactory();
return BandcampSearchQueryHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getCommentsLHFactory() {
return new BandcampCommentsLinkHandlerFactory();
return BandcampCommentsLinkHandlerFactory.getInstance();
}
@Override
@ -98,27 +107,27 @@ public class BandcampService extends StreamingService {
@Override
public KioskList getKioskList() throws ExtractionException {
final KioskList kioskList = new KioskList(this);
final ListLinkHandlerFactory h = BandcampFeaturedLinkHandlerFactory.getInstance();
try {
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampFeaturedExtractor(
BandcampService.this,
new BandcampFeaturedLinkHandlerFactory().fromUrl(FEATURED_API_URL),
h.fromUrl(FEATURED_API_URL),
kioskId
),
new BandcampFeaturedLinkHandlerFactory(),
h,
KIOSK_FEATURED
);
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampRadioExtractor(
BandcampService.this,
new BandcampFeaturedLinkHandlerFactory().fromUrl(RADIO_API_URL),
h.fromUrl(RADIO_API_URL),
kioskId
),
new BandcampFeaturedLinkHandlerFactory(),
h,
KIOSK_RADIO
);
@ -136,6 +145,15 @@ public class BandcampService extends StreamingService {
return new BandcampChannelExtractor(this, linkHandler);
}
@Override
public ChannelTabExtractor getChannelTabExtractor(final ListLinkHandler linkHandler) {
if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
} else {
return new BandcampChannelTabExtractor(this, linkHandler);
}
}
@Override
public PlaylistExtractor getPlaylistExtractor(final ListLinkHandler linkHandler) {
return new BandcampPlaylistExtractor(this, linkHandler);

View File

@ -0,0 +1,62 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampAlbumInfoItemExtractor implements PlaylistInfoItemExtractor {
private final JsonObject albumInfoItem;
private final String uploaderUrl;
public BandcampAlbumInfoItemExtractor(final JsonObject albumInfoItem,
final String uploaderUrl) {
this.albumInfoItem = albumInfoItem;
this.uploaderUrl = uploaderUrl;
}
@Override
public String getName() throws ParsingException {
return albumInfoItem.getString("title");
}
@Override
public String getUrl() throws ParsingException {
return BandcampExtractorHelper.getStreamUrlFromIds(
albumInfoItem.getLong("band_id"),
albumInfoItem.getLong("item_id"),
albumInfoItem.getString("item_type"));
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return BandcampExtractorHelper.getImagesFromImageId(albumInfoItem.getLong("art_id"), true);
}
@Override
public String getUploaderName() throws ParsingException {
return albumInfoItem.getString("band_name");
}
@Override
public String getUploaderUrl() {
return uploaderUrl;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
return ListExtractor.ITEM_COUNT_UNKNOWN;
}
}

View File

@ -2,26 +2,36 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getArtistDetails;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.jsoup.Jsoup;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampDiscographStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javax.annotation.Nonnull;
@ -35,32 +45,33 @@ public class BandcampChannelExtractor extends ChannelExtractor {
super(service, linkHandler);
}
@Nonnull
@Override
public String getAvatarUrl() {
if (channelInfo.getLong("bio_image_id") == 0) {
return "";
}
return BandcampExtractorHelper.getImageUrl(channelInfo.getLong("bio_image_id"), false);
public List<Image> getAvatars() {
return getImagesFromImageId(channelInfo.getLong("bio_image_id"), false);
}
@Nonnull
@Override
public String getBannerUrl() throws ParsingException {
public List<Image> getBanners() throws ParsingException {
/*
* Mobile API does not return the header or not the correct header.
* Therefore, we need to query the website
*/
try {
final String html = getDownloader()
.get(replaceHttpWithHttps(channelInfo.getString("bandcamp_url")))
.responseBody();
.get(replaceHttpWithHttps(channelInfo.getString("bandcamp_url")))
.responseBody();
return Stream.of(Jsoup.parse(html).getElementById("customHeader"))
.filter(Objects::nonNull)
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.findFirst()
.orElse(""); // no banner available
.filter(url -> !url.isEmpty())
.map(url -> new Image(
replaceHttpWithHttps(url), HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.UNKNOWN))
.collect(Collectors.toUnmodifiableList());
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not download artist web site", e);
@ -95,9 +106,10 @@ public class BandcampChannelExtractor extends ChannelExtractor {
return null;
}
@Nonnull
@Override
public String getParentChannelAvatarUrl() {
return null;
public List<Image> getParentChannelAvatars() {
return List.of();
}
@Override
@ -107,35 +119,53 @@ public class BandcampChannelExtractor extends ChannelExtractor {
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() throws ParsingException {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
public List<ListLinkHandler> getTabs() throws ParsingException {
final JsonArray discography = channelInfo.getArray("discography");
final TabExtractorBuilder builder = new TabExtractorBuilder(discography);
for (int i = 0; i < discography.size(); i++) {
// A discograph is as an item appears in a discography
final JsonObject discograph = discography.getObject(i);
final List<ListLinkHandler> tabs = new ArrayList<>();
if (!discograph.getString("item_type").equals("track")) {
boolean foundTrackItem = false;
boolean foundAlbumItem = false;
for (final Object discographyItem : discography) {
if (foundTrackItem && foundAlbumItem) {
break;
}
if (!(discographyItem instanceof JsonObject)) {
continue;
}
collector.commit(new BandcampDiscographStreamInfoItemExtractor(discograph, getUrl()));
final JsonObject discographyJsonItem = (JsonObject) discographyItem;
final String itemType = discographyJsonItem.getString("item_type");
if (!foundTrackItem && "track".equals(itemType)) {
foundTrackItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.TRACKS),
getId(),
ChannelTabs.TRACKS,
builder));
}
if (!foundAlbumItem && "album".equals(itemType)) {
foundAlbumItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.ALBUMS),
getId(),
ChannelTabs.ALBUMS,
builder));
}
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page) {
return null;
return Collections.unmodifiableList(tabs);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
channelInfo = BandcampExtractorHelper.getArtistDetails(getId());
channelInfo = getArtistDetails(getId());
}
@Nonnull
@ -143,4 +173,20 @@ public class BandcampChannelExtractor extends ChannelExtractor {
public String getName() {
return channelInfo.getString("name");
}
private static final class TabExtractorBuilder
implements ReadyChannelTabListLinkHandler.ChannelTabExtractorBuilder {
private final JsonArray discography;
TabExtractorBuilder(final JsonArray discography) {
this.discography = discography;
}
@Nonnull
@Override
public ChannelTabExtractor build(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
return BandcampChannelTabExtractor.fromDiscography(service, linkHandler, discography);
}
}
}

View File

@ -3,9 +3,15 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampChannelInfoItemExtractor implements ChannelInfoItemExtractor {
private final Element resultInfo;
@ -26,9 +32,10 @@ public class BandcampChannelInfoItemExtractor implements ChannelInfoItemExtracto
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return BandcampExtractorHelper.getThumbnailUrlFromSearchResult(searchResult);
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromSearchResult(searchResult);
}
@Override

View File

@ -0,0 +1,95 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampDiscographStreamInfoItemExtractor;
import javax.annotation.Nonnull;
import java.io.IOException;
public class BandcampChannelTabExtractor extends ChannelTabExtractor {
private JsonArray discography;
private final String filter;
public BandcampChannelTabExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
final String tab = linkHandler.getContentFilters().get(0);
switch (tab) {
case ChannelTabs.TRACKS:
filter = "track";
break;
case ChannelTabs.ALBUMS:
filter = "album";
break;
default:
throw new IllegalArgumentException("Unsupported channel tab: " + tab);
}
}
public static BandcampChannelTabExtractor fromDiscography(final StreamingService service,
final ListLinkHandler linkHandler,
final JsonArray discography) {
final BandcampChannelTabExtractor tabExtractor =
new BandcampChannelTabExtractor(service, linkHandler);
tabExtractor.discography = discography;
return tabExtractor;
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader) throws ParsingException {
if (discography == null) {
discography = BandcampExtractorHelper.getArtistDetails(getId())
.getArray("discography");
}
}
@Nonnull
@Override
public InfoItemsPage<InfoItem> getInitialPage() throws IOException, ExtractionException {
final MultiInfoItemsCollector collector = new MultiInfoItemsCollector(getServiceId());
for (final Object discograph : discography) {
// A discograph is as an item appears in a discography
if (!(discograph instanceof JsonObject)) {
continue;
}
final JsonObject discographJsonObject = (JsonObject) discograph;
final String itemType = discographJsonObject.getString("item_type", "");
if (!itemType.equals(filter)) {
continue;
}
switch (itemType) {
case "track":
collector.commit(new BandcampDiscographStreamInfoItemExtractor(
discographJsonObject, getUrl()));
break;
case "album":
collector.commit(new BandcampAlbumInfoItemExtractor(
discographJsonObject, getUrl()));
break;
}
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<InfoItem> getPage(final Page page) {
return null;
}
}

View File

@ -1,13 +1,17 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.comments.CommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import java.util.List;
public class BandcampCommentsInfoItemExtractor implements CommentsInfoItemExtractor {
private final JsonObject review;
@ -28,11 +32,13 @@ public class BandcampCommentsInfoItemExtractor implements CommentsInfoItemExtrac
return url;
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return getUploaderAvatarUrl();
public List<Image> getThumbnails() throws ParsingException {
return getUploaderAvatars();
}
@Nonnull
@Override
public Description getCommentText() throws ParsingException {
return new Description(review.getString("why"), Description.PLAIN_TEXT);
@ -43,8 +49,9 @@ public class BandcampCommentsInfoItemExtractor implements CommentsInfoItemExtrac
return review.getString("name");
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
return getImageUrl(review.getLong("image_id"), false);
public List<Image> getUploaderAvatars() {
return getImagesFromImageId(review.getLong("image_id"), false);
}
}

View File

@ -6,25 +6,82 @@ import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import com.grack.nanojson.JsonWriter;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.utils.ImageSuffix;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nullable;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.time.DateTimeException;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
public final class BandcampExtractorHelper {
/**
* List of image IDs which preserve aspect ratio with their theoretical dimension known.
*
* <p>
* Bandcamp images are not always squares, so images which preserve aspect ratio are only used.
* </p>
*
* <p>
* One of the direct consequences of this specificity is that only one dimension of images is
* known at time, depending of the image ID.
* </p>
*
* <p>
* Note also that dimensions are only theoretical because if the image size is less than the
* dimensions of the image ID, it will be not upscaled but kept to its original size.
* </p>
*
* <p>
* IDs come from <a href="https://gist.github.com/f2k1de/06f5fd0ae9c919a7c3693a44ee522213">the
* GitHub Gist "Bandcamp File Format Parameters" by f2k1de</a>
* </p>
*/
private static final List<ImageSuffix> IMAGE_URL_SUFFIXES_AND_RESOLUTIONS = List.of(
// ID | HEIGHT | WIDTH
new ImageSuffix("10.jpg", HEIGHT_UNKNOWN, 1200, ResolutionLevel.HIGH),
new ImageSuffix("101.jpg", 90, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("170.jpg", 422, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
// 180 returns the same image aspect ratio and size as 171
new ImageSuffix("171.jpg", 646, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("20.jpg", HEIGHT_UNKNOWN, 1024, ResolutionLevel.HIGH),
// 203 returns the same image aspect ratio and size as 200
new ImageSuffix("200.jpg", 420, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("201.jpg", 280, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("202.jpg", 140, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("204.jpg", 360, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("205.jpg", 240, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("206.jpg", 180, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("207.jpg", 120, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("43.jpg", 100, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("44.jpg", 200, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM));
private static final String IMAGE_URL_APPENDIX_AND_EXTENSION_REGEX = "_\\d+\\.\\w+";
private static final String IMAGES_DOMAIN_AND_PATH = "https://f4.bcbits.com/img/";
public static final String BASE_URL = "https://bandcamp.com";
public static final String BASE_API_URL = BASE_URL + "/api";
@ -44,7 +101,7 @@ public final class BandcampExtractorHelper {
+ "&tralbum_id=" + itemId + "&tralbum_type=" + itemType.charAt(0))
.responseBody();
return Utils.replaceHttpWithHttps(JsonParser.object().from(jsonString)
return replaceHttpWithHttps(JsonParser.object().from(jsonString)
.getString("bandcamp_url"));
} catch (final JsonParserException | ReCaptchaException | IOException e) {
@ -55,8 +112,7 @@ public final class BandcampExtractorHelper {
/**
* Fetch artist details from mobile endpoint.
* <a href="https://notabug.org/fynngodau/bandcampDirect/wiki/
* rewindBandcamp+%E2%80%93+Fetching+artist+details">
* <a href="https://notabug.org/fynngodau/bandcampDirect/wiki/rewindBandcamp+%E2%80%93+Fetching+artist+details">
* More technical info.</a>
*/
public static JsonObject getArtistDetails(final String id) throws ParsingException {
@ -76,40 +132,58 @@ public final class BandcampExtractorHelper {
}
/**
* Generate image url from image ID.
* <p>
* The appendix "_10" was chosen because it provides images sized 1200x1200. Other integer
* values are possible as well (e.g. 0 is a very large resolution, possibly the original).
* Generate an image url from an image ID.
*
* @param id The image ID
* @param album True if this is the cover of an album or track
* @return URL of image with this ID sized 1200x1200
* <p>
* The image ID {@code 10} was chosen because it provides images wide up to 1200px (when
* the original image width is more than or equal this resolution).
* </p>
*
* <p>
* Other integer values are possible as well (e.g. 0 is a very large resolution, possibly the
* original); see {@link #IMAGE_URL_SUFFIXES_AND_RESOLUTIONS} for more details about image
* resolution IDs.
* </p>
*
* @param id the image ID
* @param isAlbum whether the image is the cover of an album or a track
* @return a URL of the image with this ID with a width up to 1200px
*/
public static String getImageUrl(final long id, final boolean album) {
return "https://f4.bcbits.com/img/" + (album ? 'a' : "") + id + "_10.jpg";
@Nonnull
public static String getImageUrl(final long id, final boolean isAlbum) {
return IMAGES_DOMAIN_AND_PATH + (isAlbum ? 'a' : "") + id + "_10.jpg";
}
/**
* @return <code>true</code> if the given URL looks like it comes from a bandcamp custom domain
* or if it comes from <code>bandcamp.com</code> itself
* or a <code>*.bandcamp.com</code> subdomain
*/
public static boolean isSupportedDomain(final String url) throws ParsingException {
public static boolean isArtistDomain(final String url) throws ParsingException {
// Accept all bandcamp.com URLs
if (url.toLowerCase().matches("https?://.+\\.bandcamp\\.com(/.*)?")) {
return true;
}
// Reject non-artist bandcamp.com URLs
if (url.toLowerCase().matches("https?://bandcamp\\.com(/.*)?")) {
return false;
}
try {
// Test other URLs for whether they contain a footer that links to bandcamp
return Jsoup.parse(NewPipe.getDownloader().get(url).responseBody())
.getElementById("pgFt")
.getElementById("pgFt-inner")
.getElementById("footer-logo-wrapper")
.getElementById("footer-logo")
.getElementsByClass("hiddenAccess")
.text().equals("Bandcamp");
} catch (final NullPointerException e) {
return Jsoup.parse(
NewPipe.getDownloader()
.get(Utils.replaceHttpWithHttps(url))
.responseBody()
)
.getElementsByClass("cart-wrapper")
.get(0)
.getElementsByTag("a")
.get(0)
.attr("href")
.equals("https://bandcamp.com/cart");
} catch (final NullPointerException | IndexOutOfBoundsException e) {
return false;
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not determine whether URL is custom domain "
@ -136,13 +210,94 @@ public final class BandcampExtractorHelper {
}
}
@Nullable
public static String getThumbnailUrlFromSearchResult(final Element searchResult) {
return searchResult.getElementsByClass("art").stream()
/**
* Get a list of images from a search result {@link Element}.
*
* <p>
* This method will call {@link #getImagesFromImageUrl(String)} using the first non null and
* non empty image URL found from the {@code src} attribute of {@code img} HTML elements, or an
* empty string if no valid image URL was found.
* </p>
*
* @param searchResult a search result {@link Element}
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where no valid image URL was found
*/
@Nonnull
public static List<Image> getImagesFromSearchResult(@Nonnull final Element searchResult) {
return getImagesFromImageUrl(searchResult.getElementsByClass("art")
.stream()
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.filter(string -> !string.isEmpty())
.filter(imageUrl -> !isNullOrEmpty(imageUrl))
.findFirst()
.orElse(null);
.orElse(""));
}
/**
* Get all images which have resolutions preserving aspect ratio from an image URL.
*
* <p>
* This method will remove the image ID and its extension from the end of the URL and then call
* {@link #getImagesFromImageBaseUrl(String)}.
* </p>
*
* @param imageUrl the full URL of an image provided by Bandcamp, such as in its HTML code
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where the image URL has been not extracted (and so is null or empty)
*/
@Nonnull
public static List<Image> getImagesFromImageUrl(@Nullable final String imageUrl) {
if (isNullOrEmpty(imageUrl)) {
return List.of();
}
return getImagesFromImageBaseUrl(
imageUrl.replaceFirst(IMAGE_URL_APPENDIX_AND_EXTENSION_REGEX, "_"));
}
/**
* Get all images which have resolutions preserving aspect ratio from an image ID.
*
* <p>
* This method will call {@link #getImagesFromImageBaseUrl(String)}.
* </p>
*
* @param id the id of an image provided by Bandcamp
* @param isAlbum whether the image is the cover of an album
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where the image ID has been not extracted (and so equal to 0)
*/
@Nonnull
public static List<Image> getImagesFromImageId(final long id, final boolean isAlbum) {
if (id == 0) {
return List.of();
}
return getImagesFromImageBaseUrl(IMAGES_DOMAIN_AND_PATH + (isAlbum ? 'a' : "") + id + "_");
}
/**
* Get all images resolutions preserving aspect ratio from a base image URL.
*
* <p>
* Base image URLs are images containing the image path, a {@code a} letter if it comes from an
* album, its ID and an underscore.
* </p>
*
* <p>
* Images resolutions returned are the ones of {@link #IMAGE_URL_SUFFIXES_AND_RESOLUTIONS}.
* </p>
*
* @param baseUrl the base URL of the image
* @return an unmodifiable and non-empty list of {@link Image}s
*/
@Nonnull
private static List<Image> getImagesFromImageBaseUrl(@Nonnull final String baseUrl) {
return IMAGE_URL_SUFFIXES_AND_RESOLUTIONS.stream()
.map(imageSuffix -> new Image(baseUrl + imageSuffix.getSuffix(),
imageSuffix.getHeight(), imageSuffix.getWidth(),
imageSuffix.getResolutionLevel()))
.collect(Collectors.toUnmodifiableList());
}
}

View File

@ -1,6 +1,7 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor.getAlbumInfoJson;
import static org.schabi.newpipe.extractor.utils.JsonUtils.getJsonData;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;
@ -13,6 +14,7 @@ import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -28,6 +30,7 @@ import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import java.io.IOException;
import java.util.Objects;
import java.util.List;
import javax.annotation.Nonnull;
@ -74,11 +77,11 @@ public class BandcampPlaylistExtractor extends PlaylistExtractor {
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
public List<Image> getThumbnails() throws ParsingException {
if (albumJson.isNull("art_id")) {
return "";
return List.of();
} else {
return getImageUrl(albumJson.getLong("art_id"), true);
return getImagesFromImageId(albumJson.getLong("art_id"), true);
}
}
@ -94,12 +97,14 @@ public class BandcampPlaylistExtractor extends PlaylistExtractor {
return albumJson.getString("artist");
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
return document.getElementsByClass("band-photo").stream()
public List<Image> getUploaderAvatars() {
return getImagesFromImageUrl(document.getElementsByClass("band-photo")
.stream()
.map(element -> element.attr("src"))
.findFirst()
.orElse("");
.orElse(""));
}
@Override
@ -154,7 +159,7 @@ public class BandcampPlaylistExtractor extends PlaylistExtractor {
} else {
// Pretend every track has the same cover art as the album
collector.commit(new BandcampPlaylistStreamInfoItemExtractor(
track, getUploaderUrl(), getThumbnailUrl()));
track, getUploaderUrl(), getThumbnails()));
}
}

View File

@ -1,9 +1,13 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampPlaylistInfoItemExtractor implements PlaylistInfoItemExtractor {
private final Element searchResult;
@ -46,8 +50,9 @@ public class BandcampPlaylistInfoItemExtractor implements PlaylistInfoItemExtrac
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public String getThumbnailUrl() {
return BandcampExtractorHelper.getThumbnailUrlFromSearchResult(searchResult);
public List<Image> getThumbnails() {
return getImagesFromSearchResult(searchResult);
}
}

View File

@ -1,9 +1,14 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
public class BandcampPlaylistInfoItemFeaturedExtractor implements PlaylistInfoItemExtractor {
@ -40,12 +45,14 @@ public class BandcampPlaylistInfoItemFeaturedExtractor implements PlaylistInfoIt
@Override
public String getUrl() {
return featuredStory.getString("item_url").replaceAll("http://", "https://");
return Utils.replaceHttpWithHttps(featuredStory.getString("item_url"));
}
@Nonnull
@Override
public String getThumbnailUrl() {
return featuredStory.has("art_id") ? getImageUrl(featuredStory.getLong("art_id"), true)
: getImageUrl(featuredStory.getLong("item_art_id"), true);
public List<Image> getThumbnails() {
return featuredStory.has("art_id")
? getImagesFromImageId(featuredStory.getLong("art_id"), true)
: getImagesFromImageId(featuredStory.getLong("item_art_id"), true);
}
}

View File

@ -24,7 +24,7 @@ import static org.schabi.newpipe.extractor.services.bandcamp.extractors.Bandcamp
public class BandcampRadioExtractor extends KioskExtractor<StreamInfoItem> {
public static final String KIOSK_RADIO = "Radio";
public static final String RADIO_API_URL = BASE_API_URL + "/bcweekly/1/list";
public static final String RADIO_API_URL = BASE_API_URL + "/bcweekly/3/list";
private JsonObject json = null;

View File

@ -3,15 +3,20 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.parseDate;
public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
@ -30,6 +35,12 @@ public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
return 0;
}
@Nullable
@Override
public String getShortDescription() {
return show.getString("desc");
}
@Nullable
@Override
public String getTextualUploadDate() {
@ -39,7 +50,7 @@ public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
@Nullable
@Override
public DateWrapper getUploadDate() throws ParsingException {
return BandcampExtractorHelper.parseDate(getTextualUploadDate());
return parseDate(getTextualUploadDate());
}
@Override
@ -52,9 +63,10 @@ public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
return BASE_URL + "/?show=" + show.getInt("id");
}
@Nonnull
@Override
public String getThumbnailUrl() {
return getImageUrl(show.getLong("image_id"), false);
public List<Image> getThumbnails() {
return getImagesFromImageId(show.getLong("image_id"), false);
}
@Override
@ -69,8 +81,8 @@ public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
@Override
public String getUploaderName() {
// JSON does not contain uploader name
return "";
// The "title" field contains the title of the series, e.g. "Bandcamp Weekly".
return show.getString("title");
}
@Override
@ -78,12 +90,6 @@ public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
return "";
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;

View File

@ -3,6 +3,7 @@ package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
@ -11,6 +12,8 @@ import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
@ -97,14 +100,16 @@ public class BandcampRadioStreamExtractor extends BandcampStreamExtractor {
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return getImageUrl(showInfo.getLong("show_image_id"), false);
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageId(showInfo.getLong("show_image_id"), false);
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
return BASE_URL + "/img/buttons/bandcamp-button-circle-whitecolor-512.png";
public List<Image> getUploaderAvatars() {
return Collections.singletonList(
new Image(BASE_URL + "/img/buttons/bandcamp-button-circle-whitecolor-512.png",
512, 512, ResolutionLevel.MEDIUM));
}
@Nonnull

View File

@ -3,10 +3,14 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
/**
* Extracts recommended albums from tracks' website
@ -28,9 +32,10 @@ public class BandcampRelatedPlaylistInfoItemExtractor implements PlaylistInfoIte
return relatedAlbum.getElementsByClass("album-link").attr("abs:href");
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return relatedAlbum.getElementsByClass("album-art").attr("src");
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageUrl(relatedAlbum.getElementsByClass("album-art").attr("src"));
}
@Override

View File

@ -2,8 +2,11 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.parseDate;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParserException;
@ -11,6 +14,7 @@ import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -98,7 +102,7 @@ public class BandcampStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public String getUrl() throws ParsingException {
return albumJson.getString("url").replace("http://", "https://");
return replaceHttpWithHttps(albumJson.getString("url"));
}
@Nonnull
@ -116,26 +120,27 @@ public class BandcampStreamExtractor extends StreamExtractor {
@Nullable
@Override
public DateWrapper getUploadDate() throws ParsingException {
return BandcampExtractorHelper.parseDate(getTextualUploadDate());
return parseDate(getTextualUploadDate());
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
public List<Image> getThumbnails() throws ParsingException {
if (albumJson.isNull("art_id")) {
return "";
return List.of();
}
return getImageUrl(albumJson.getLong("art_id"), true);
return getImagesFromImageId(albumJson.getLong("art_id"), true);
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
return document.getElementsByClass("band-photo").stream()
public List<Image> getUploaderAvatars() {
return getImagesFromImageUrl(document.getElementsByClass("band-photo")
.stream()
.map(element -> element.attr("src"))
.findFirst()
.orElse("");
.orElse(""));
}
@Nonnull

View File

@ -1,10 +1,14 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import javax.annotation.Nullable;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
public class BandcampDiscographStreamInfoItemExtractor extends BandcampStreamInfoItemExtractor {
@ -20,12 +24,6 @@ public class BandcampDiscographStreamInfoItemExtractor extends BandcampStreamInf
return discograph.getString("band_name");
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public String getName() {
return discograph.getString("title");
@ -40,11 +38,10 @@ public class BandcampDiscographStreamInfoItemExtractor extends BandcampStreamInf
);
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return BandcampExtractorHelper.getImageUrl(
discograph.getLong("art_id"), true
);
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageId(discograph.getLong("art_id"), true);
}
@Override

View File

@ -3,19 +3,21 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import javax.annotation.Nullable;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
public class BandcampPlaylistStreamInfoItemExtractor extends BandcampStreamInfoItemExtractor {
private final JsonObject track;
private String substituteCoverUrl;
private List<Image> substituteCovers;
private final StreamingService service;
public BandcampPlaylistStreamInfoItemExtractor(final JsonObject track,
@ -24,13 +26,14 @@ public class BandcampPlaylistStreamInfoItemExtractor extends BandcampStreamInfoI
super(uploaderUrl);
this.track = track;
this.service = service;
substituteCovers = Collections.emptyList();
}
public BandcampPlaylistStreamInfoItemExtractor(final JsonObject track,
final String uploaderUrl,
final String substituteCoverUrl) {
final List<Image> substituteCovers) {
this(track, uploaderUrl, (StreamingService) null);
this.substituteCoverUrl = substituteCoverUrl;
this.substituteCovers = substituteCovers;
}
@Override
@ -40,7 +43,12 @@ public class BandcampPlaylistStreamInfoItemExtractor extends BandcampStreamInfoI
@Override
public String getUrl() {
return getUploaderUrl() + track.getString("title_link");
final String relativeUrl = track.getString("title_link");
if (relativeUrl != null) {
return getUploaderUrl() + relativeUrl;
} else {
return null;
}
}
@Override
@ -56,28 +64,23 @@ public class BandcampPlaylistStreamInfoItemExtractor extends BandcampStreamInfoI
return "";
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
/**
* Each track can have its own cover art. Therefore, unless a substitute is provided,
* the thumbnail is extracted using a stream extractor.
*/
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
if (substituteCoverUrl != null) {
return substituteCoverUrl;
} else {
public List<Image> getThumbnails() throws ParsingException {
if (substituteCovers.isEmpty() && getUrl() != null) {
try {
final StreamExtractor extractor = service.getStreamExtractor(getUrl());
extractor.fetchPage();
return extractor.getThumbnailUrl();
return extractor.getThumbnails();
} catch (final ExtractionException | IOException e) {
throw new ParsingException("could not download cover art location", e);
throw new ParsingException("Could not download cover art location", e);
}
}
return substituteCovers;
}
}

View File

@ -1,10 +1,13 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import javax.annotation.Nullable;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampSearchStreamInfoItemExtractor extends BandcampStreamInfoItemExtractor {
@ -29,12 +32,6 @@ public class BandcampSearchStreamInfoItemExtractor extends BandcampStreamInfoIte
}
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public String getName() throws ParsingException {
return resultInfo.getElementsByClass("heading").text();
@ -45,9 +42,10 @@ public class BandcampSearchStreamInfoItemExtractor extends BandcampStreamInfoIte
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return BandcampExtractorHelper.getThumbnailUrlFromSearchResult(searchResult);
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromSearchResult(searchResult);
}
@Override

View File

@ -10,6 +10,7 @@ import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.IOException;
import java.util.List;
@ -17,13 +18,23 @@ import java.util.List;
/**
* Artist do have IDs that are useful
*/
public class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
public final class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampChannelLinkHandlerFactory INSTANCE
= new BandcampChannelLinkHandlerFactory();
private BandcampChannelLinkHandlerFactory() {
}
public static BandcampChannelLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
try {
final String response = NewPipe.getDownloader().get(url).responseBody();
final String response = NewPipe.getDownloader().get(Utils.replaceHttpWithHttps(url))
.responseBody();
// Use band data embedded in website to extract ID
final JsonObject bandData = JsonUtils.getJsonData(response, "data-band");
@ -41,16 +52,13 @@ public class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
*/
@Override
public String getUrl(final String id, final List<String> contentFilter, final String sortFilter)
throws ParsingException {
try {
return BandcampExtractorHelper.getArtistDetails(id)
.getString("bandcamp_url")
.replace("http://", "https://");
} catch (final NullPointerException e) {
throws ParsingException, UnsupportedOperationException {
final JsonObject artistDetails = BandcampExtractorHelper.getArtistDetails(id);
if (artistDetails.getBoolean("error")) {
throw new ParsingException(
"JSON does not contain URL (invalid id?) or is otherwise invalid", e);
"JSON does not contain a channel URL (invalid id?) or is otherwise invalid");
}
return Utils.replaceHttpWithHttps(artistDetails.getString("bandcamp_url"));
}
/**
@ -61,22 +69,21 @@ public class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
final String lowercaseUrl = url.toLowerCase();
// https: | | artist.bandcamp.com | releases
// 0 1 2 3
// https: | | artist.bandcamp.com | releases - music - album - track ( | name)
// 0 1 2 3 (4)
final String[] splitUrl = lowercaseUrl.split("/");
// URL is too short
if (splitUrl.length < 3) {
if (splitUrl.length != 3 && splitUrl.length != 4) {
return false;
}
// Must have "releases" or "music" as segment after url or none at all
if (splitUrl.length > 3 && !(
splitUrl[3].equals("releases") || splitUrl[3].equals("music")
)) {
// Must have "releases", "music", "album" or "track" as segment after URL or none at all
if (splitUrl.length == 4 && !(splitUrl[3].equals("releases")
|| splitUrl[3].equals("music")
|| splitUrl[3].equals("album")
|| splitUrl[3].equals("track"))) {
return false;
} else {
if (splitUrl[2].equals("daily.bandcamp.com")) {
// Refuse links to daily.bandcamp.com as that is not an artist
@ -84,7 +91,7 @@ public class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
}
// Test whether domain is supported
return BandcampExtractorHelper.isSupportedDomain(lowercaseUrl);
return BandcampExtractorHelper.isArtistDomain(lowercaseUrl);
}
}
}

View File

@ -0,0 +1,72 @@
package org.schabi.newpipe.extractor.services.bandcamp.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.exceptions.UnsupportedTabException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import javax.annotation.Nonnull;
import java.util.List;
public final class BandcampChannelTabLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampChannelTabLinkHandlerFactory INSTANCE
= new BandcampChannelTabLinkHandlerFactory();
private BandcampChannelTabLinkHandlerFactory() {
}
public static BandcampChannelTabLinkHandlerFactory getInstance() {
return INSTANCE;
}
/**
* Get a tab's URL suffix.
*
* <p>
* These URLs don't actually exist on the Bandcamp website, as both albums and tracks are
* listed on the main page, but redirect to the main page, which is perfect for us as we need a
* unique URL for each tab.
* </p>
*
* @param tab the tab value, which must not be null
* @return a URL suffix
* @throws UnsupportedTabException if the tab is not supported
*/
@Nonnull
public static String getUrlSuffix(@Nonnull final String tab) throws UnsupportedTabException {
switch (tab) {
case ChannelTabs.TRACKS:
return "/track";
case ChannelTabs.ALBUMS:
return "/album";
}
throw new UnsupportedTabException(tab);
}
@Override
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return BandcampChannelLinkHandlerFactory.getInstance().getId(url);
}
@Override
public String getUrl(final String id, final List<String> contentFilter, final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return BandcampChannelLinkHandlerFactory.getInstance().getUrl(id)
+ getUrlSuffix(contentFilter.get(0));
}
@Override
public boolean onAcceptUrl(final String url) throws ParsingException {
return BandcampChannelLinkHandlerFactory.getInstance().onAcceptUrl(url);
}
@Override
public String[] getAvailableContentFilter() {
return new String[]{
ChannelTabs.TRACKS,
ChannelTabs.ALBUMS,
};
}
}

View File

@ -3,6 +3,7 @@ package org.schabi.newpipe.extractor.services.bandcamp.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.utils.Utils;
import java.util.List;
@ -10,11 +11,21 @@ import java.util.List;
* Like in {@link BandcampStreamLinkHandlerFactory}, tracks have no meaningful IDs except for
* their URLs
*/
public class BandcampCommentsLinkHandlerFactory extends ListLinkHandlerFactory {
public final class BandcampCommentsLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampCommentsLinkHandlerFactory INSTANCE
= new BandcampCommentsLinkHandlerFactory();
private BandcampCommentsLinkHandlerFactory() {
}
public static BandcampCommentsLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
return url;
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return Utils.replaceHttpWithHttps(url);
}
@Override
@ -29,13 +40,14 @@ public class BandcampCommentsLinkHandlerFactory extends ListLinkHandlerFactory {
}
// Test whether domain is supported
return BandcampExtractorHelper.isSupportedDomain(url);
return BandcampExtractorHelper.isArtistDomain(url);
}
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
return id;
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return Utils.replaceHttpWithHttps(id);
}
}

View File

@ -2,6 +2,7 @@
package org.schabi.newpipe.extractor.services.bandcamp.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.utils.Utils;
@ -13,12 +14,23 @@ import static org.schabi.newpipe.extractor.services.bandcamp.extractors.Bandcamp
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.KIOSK_RADIO;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.RADIO_API_URL;
public class BandcampFeaturedLinkHandlerFactory extends ListLinkHandlerFactory {
public final class BandcampFeaturedLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampFeaturedLinkHandlerFactory INSTANCE =
new BandcampFeaturedLinkHandlerFactory();
private BandcampFeaturedLinkHandlerFactory() {
}
public static BandcampFeaturedLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
if (id.equals(KIOSK_FEATURED)) {
return FEATURED_API_URL; // doesn't have a website
} else if (id.equals(KIOSK_RADIO)) {
@ -29,7 +41,7 @@ public class BandcampFeaturedLinkHandlerFactory extends ListLinkHandlerFactory {
}
@Override
public String getId(final String url) {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
final String fixedUrl = Utils.replaceHttpWithHttps(url);
if (BandcampExtractorHelper.isRadioUrl(fixedUrl) || fixedUrl.equals(RADIO_API_URL)) {
return KIOSK_RADIO;

View File

@ -5,23 +5,36 @@ package org.schabi.newpipe.extractor.services.bandcamp.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.utils.Utils;
import java.util.List;
/**
* Just as with streams, the album ids are essentially useless for us.
*/
public class BandcampPlaylistLinkHandlerFactory extends ListLinkHandlerFactory {
public final class BandcampPlaylistLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampPlaylistLinkHandlerFactory INSTANCE
= new BandcampPlaylistLinkHandlerFactory();
private BandcampPlaylistLinkHandlerFactory() {
}
public static BandcampPlaylistLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return getUrl(url);
}
@Override
public String getUrl(final String url,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
return url;
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return Utils.replaceHttpWithHttps(url);
}
/**
@ -36,6 +49,6 @@ public class BandcampPlaylistLinkHandlerFactory extends ListLinkHandlerFactory {
}
// Test whether domain is supported
return BandcampExtractorHelper.isSupportedDomain(url);
return BandcampExtractorHelper.isArtistDomain(url);
}
}

View File

@ -8,18 +8,25 @@ import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.UnsupportedEncodingException;
import java.util.List;
public class BandcampSearchQueryHandlerFactory extends SearchQueryHandlerFactory {
public final class BandcampSearchQueryHandlerFactory extends SearchQueryHandlerFactory {
private static final BandcampSearchQueryHandlerFactory INSTANCE
= new BandcampSearchQueryHandlerFactory();
private BandcampSearchQueryHandlerFactory() {
}
public static BandcampSearchQueryHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getUrl(final String query,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
try {
return BASE_URL + "/search?q=" + Utils.encodeUrlUtf8(query) + "&page=1";
} catch (final UnsupportedEncodingException e) {
throw new ParsingException("query \"" + query + "\" could not be encoded", e);
}
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return BASE_URL + "/search?q=" + Utils.encodeUrlUtf8(query) + "&page=1";
}
}

View File

@ -5,6 +5,7 @@ package org.schabi.newpipe.extractor.services.bandcamp.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.utils.Utils;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
@ -14,14 +15,24 @@ import static org.schabi.newpipe.extractor.services.bandcamp.extractors.Bandcamp
*
* <p>Radio (bandcamp weekly) shows do have ids.</p>
*/
public class BandcampStreamLinkHandlerFactory extends LinkHandlerFactory {
public final class BandcampStreamLinkHandlerFactory extends LinkHandlerFactory {
private static final BandcampStreamLinkHandlerFactory INSTANCE
= new BandcampStreamLinkHandlerFactory();
private BandcampStreamLinkHandlerFactory() {
}
public static BandcampStreamLinkHandlerFactory getInstance() {
return INSTANCE;
}
/**
* @see BandcampStreamLinkHandlerFactory
*/
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
if (BandcampExtractorHelper.isRadioUrl(url)) {
return url.split("bandcamp.com/\\?show=")[1];
} else {
@ -34,11 +45,12 @@ public class BandcampStreamLinkHandlerFactory extends LinkHandlerFactory {
* @see BandcampStreamLinkHandlerFactory
*/
@Override
public String getUrl(final String input) {
public String getUrl(final String input)
throws ParsingException, UnsupportedOperationException {
if (input.matches("\\d+")) {
return BASE_URL + "/?show=" + input;
} else {
return input;
return Utils.replaceHttpWithHttps(input);
}
}
@ -60,6 +72,6 @@ public class BandcampStreamLinkHandlerFactory extends LinkHandlerFactory {
}
// Test whether domain is supported
return BandcampExtractorHelper.isSupportedDomain(url);
return BandcampExtractorHelper.isArtistDomain(url);
}
}

View File

@ -6,6 +6,7 @@ import static java.util.Arrays.asList;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.kiosk.KioskList;
@ -13,10 +14,12 @@ import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCChannelTabExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCConferenceExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCConferenceKiosk;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCLiveStreamExtractor;
@ -27,8 +30,6 @@ import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCSearch
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCStreamExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferenceLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferencesListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCLiveListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCRecentListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCSearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCStreamLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
@ -47,12 +48,19 @@ public class MediaCCCService extends StreamingService {
@Override
public LinkHandlerFactory getStreamLHFactory() {
return new MediaCCCStreamLinkHandlerFactory();
return MediaCCCStreamLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelLHFactory() {
return new MediaCCCConferenceLinkHandlerFactory();
return MediaCCCConferenceLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelTabLHFactory() {
// there is just one channel tab in MediaCCC, the one containing conferences, so there is
// no need for a specific channel tab link handler, but we can just use the channel one
return MediaCCCConferenceLinkHandlerFactory.getInstance();
}
@Override
@ -62,7 +70,7 @@ public class MediaCCCService extends StreamingService {
@Override
public SearchQueryHandlerFactory getSearchQHFactory() {
return new MediaCCCSearchQueryHandlerFactory();
return MediaCCCSearchQueryHandlerFactory.getInstance();
}
@Override
@ -78,6 +86,18 @@ public class MediaCCCService extends StreamingService {
return new MediaCCCConferenceExtractor(this, linkHandler);
}
@Override
public ChannelTabExtractor getChannelTabExtractor(final ListLinkHandler linkHandler) {
if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
// conference data has already been fetched, let the ReadyChannelTabListLinkHandler
// create a MediaCCCChannelTabExtractor with that data
return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
} else {
// conference data has not been fetched yet, so pass null instead
return new MediaCCCChannelTabExtractor(this, linkHandler, null);
}
}
@Override
public PlaylistExtractor getPlaylistExtractor(final ListLinkHandler linkHandler) {
return null;
@ -91,40 +111,41 @@ public class MediaCCCService extends StreamingService {
@Override
public KioskList getKioskList() throws ExtractionException {
final KioskList list = new KioskList(this);
final ListLinkHandlerFactory h = MediaCCCConferencesListLinkHandlerFactory.getInstance();
// add kiosks here e.g.:
try {
list.addKioskEntry(
(streamingService, url, kioskId) -> new MediaCCCConferenceKiosk(
MediaCCCService.this,
new MediaCCCConferencesListLinkHandlerFactory().fromUrl(url),
h.fromUrl(url),
kioskId
),
new MediaCCCConferencesListLinkHandlerFactory(),
"conferences"
h,
MediaCCCConferenceKiosk.KIOSK_ID
);
list.addKioskEntry(
(streamingService, url, kioskId) -> new MediaCCCRecentKiosk(
MediaCCCService.this,
new MediaCCCRecentListLinkHandlerFactory().fromUrl(url),
h.fromUrl(url),
kioskId
),
new MediaCCCRecentListLinkHandlerFactory(),
"recent"
h,
MediaCCCRecentKiosk.KIOSK_ID
);
list.addKioskEntry(
(streamingService, url, kioskId) -> new MediaCCCLiveStreamKiosk(
MediaCCCService.this,
new MediaCCCLiveListLinkHandlerFactory().fromUrl(url),
h.fromUrl(url),
kioskId
),
new MediaCCCLiveListLinkHandlerFactory(),
"live"
h,
MediaCCCLiveStreamKiosk.KIOSK_ID
);
list.setDefaultKiosk("recent");
list.setDefaultKiosk(MediaCCCRecentKiosk.KIOSK_ID);
} catch (final Exception e) {
throw new ExtractionException(e);
}

View File

@ -0,0 +1,69 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems.MediaCCCStreamInfoItemExtractor;
import java.io.IOException;
import java.util.Objects;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
/**
* MediaCCC does not really have channel tabs, but rather a list of videos for each conference,
* so this class just acts as a videos channel tab extractor.
*/
public class MediaCCCChannelTabExtractor extends ChannelTabExtractor {
@Nullable
private JsonObject conferenceData;
/**
* @param conferenceData will be not-null if conference data has already been fetched by
* {@link MediaCCCConferenceExtractor}. Otherwise, if this parameter is
* {@code null}, conference data will be fetched anew.
*/
public MediaCCCChannelTabExtractor(final StreamingService service,
final ListLinkHandler linkHandler,
@Nullable final JsonObject conferenceData) {
super(service, linkHandler);
this.conferenceData = conferenceData;
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws ExtractionException, IOException {
if (conferenceData == null) {
// only fetch conference data if we don't have it already
conferenceData = MediaCCCConferenceExtractor.fetchConferenceData(downloader, getId());
}
}
@Nonnull
@Override
public ListExtractor.InfoItemsPage<InfoItem> getInitialPage() {
final MultiInfoItemsCollector collector =
new MultiInfoItemsCollector(getServiceId());
Objects.requireNonNull(conferenceData) // will surely be != null after onFetchPage
.getArray("events")
.stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.forEach(event -> collector.commit(new MediaCCCStreamInfoItemExtractor(event)));
return new ListExtractor.InfoItemsPage<>(collector, null);
}
@Override
public ListExtractor.InfoItemsPage<InfoItem> getPage(final Page page) {
return ListExtractor.InfoItemsPage.emptyPage();
}
}

View File

@ -1,23 +1,27 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonArray;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getImageListFromLogoImageUrl;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems.MediaCCCStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferenceLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
import java.io.IOException;
public class MediaCCCConferenceExtractor extends ChannelExtractor {
private JsonObject conferenceData;
@ -27,14 +31,29 @@ public class MediaCCCConferenceExtractor extends ChannelExtractor {
super(service, linkHandler);
}
@Override
public String getAvatarUrl() {
return conferenceData.getString("logo_url");
static JsonObject fetchConferenceData(@Nonnull final Downloader downloader,
@Nonnull final String conferenceId)
throws IOException, ExtractionException {
final String conferenceUrl
= MediaCCCConferenceLinkHandlerFactory.CONFERENCE_API_ENDPOINT + conferenceId;
try {
return JsonParser.object().from(downloader.get(conferenceUrl).responseBody());
} catch (final JsonParserException jpe) {
throw new ExtractionException("Could not parse json returned by URL: " + conferenceUrl);
}
}
@Nonnull
@Override
public String getBannerUrl() {
return conferenceData.getString("logo_url");
public List<Image> getAvatars() {
return getImageListFromLogoImageUrl(conferenceData.getString("logo_url"));
}
@Nonnull
@Override
public List<Image> getBanners() {
return Collections.emptyList();
}
@Override
@ -62,9 +81,10 @@ public class MediaCCCConferenceExtractor extends ChannelExtractor {
return "";
}
@Nonnull
@Override
public String getParentChannelAvatarUrl() {
return "";
public List<Image> getParentChannelAvatars() {
return Collections.emptyList();
}
@Override
@ -74,30 +94,18 @@ public class MediaCCCConferenceExtractor extends ChannelExtractor {
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
final JsonArray events = conferenceData.getArray("events");
for (int i = 0; i < events.size(); i++) {
collector.commit(new MediaCCCStreamInfoItemExtractor(events.getObject(i)));
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page) {
return InfoItemsPage.emptyPage();
public List<ListLinkHandler> getTabs() throws ParsingException {
// avoid keeping a reference to MediaCCCConferenceExtractor inside the lambda
final JsonObject theConferenceData = conferenceData;
return List.of(new ReadyChannelTabListLinkHandler(getUrl(), getId(), ChannelTabs.VIDEOS,
(service, linkHandler) ->
new MediaCCCChannelTabExtractor(service, linkHandler, theConferenceData)));
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
final String conferenceUrl
= MediaCCCConferenceLinkHandlerFactory.CONFERENCE_API_ENDPOINT + getId();
try {
conferenceData = JsonParser.object().from(downloader.get(conferenceUrl).responseBody());
} catch (final JsonParserException jpe) {
throw new ExtractionException("Could not parse json returnd by url: " + conferenceUrl);
}
conferenceData = fetchConferenceData(downloader, getId());
}
@Nonnull

View File

@ -21,6 +21,8 @@ import java.io.IOException;
import javax.annotation.Nonnull;
public class MediaCCCConferenceKiosk extends KioskExtractor<ChannelInfoItem> {
public static final String KIOSK_ID = "conferences";
private JsonObject doc;
public MediaCCCConferenceKiosk(final StreamingService streamingService,

View File

@ -1,11 +1,13 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getThumbnailsFromLiveStreamItem;
import static org.schabi.newpipe.extractor.stream.AudioStream.UNKNOWN_BITRATE;
import static org.schabi.newpipe.extractor.stream.Stream.ID_UNKNOWN;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -77,8 +79,8 @@ public class MediaCCCLiveStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return room.getString("thumb");
public List<Image> getThumbnails() throws ParsingException {
return getThumbnailsFromLiveStreamItem(room);
}
@Nonnull

View File

@ -16,6 +16,8 @@ import javax.annotation.Nonnull;
import java.io.IOException;
public class MediaCCCLiveStreamKiosk extends KioskExtractor<StreamInfoItem> {
public static final String KIOSK_ID = "live";
private JsonArray doc;
public MediaCCCLiveStreamKiosk(final StreamingService streamingService,
@ -36,13 +38,16 @@ public class MediaCCCLiveStreamKiosk extends KioskExtractor<StreamInfoItem> {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
for (int c = 0; c < doc.size(); c++) {
final JsonObject conference = doc.getObject(c);
final JsonArray groups = conference.getArray("groups");
for (int g = 0; g < groups.size(); g++) {
final String group = groups.getObject(g).getString("group");
final JsonArray rooms = groups.getObject(g).getArray("rooms");
for (int r = 0; r < rooms.size(); r++) {
final JsonObject room = rooms.getObject(r);
collector.commit(new MediaCCCLiveStreamKioskExtractor(conference, group, room));
if (conference.getBoolean("isCurrentlyStreaming")) {
final JsonArray groups = conference.getArray("groups");
for (int g = 0; g < groups.size(); g++) {
final String group = groups.getObject(g).getString("group");
final JsonArray rooms = groups.getObject(g).getArray("rooms");
for (int r = 0; r < rooms.size(); r++) {
final JsonObject room = rooms.getObject(r);
collector.commit(new MediaCCCLiveStreamKioskExtractor(
conference, group, room));
}
}
}
@ -59,6 +64,6 @@ public class MediaCCCLiveStreamKiosk extends KioskExtractor<StreamInfoItem> {
@Nonnull
@Override
public String getName() throws ParsingException {
return "live";
return KIOSK_ID;
}
}

View File

@ -1,12 +1,17 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getThumbnailsFromLiveStreamItem;
public class MediaCCCLiveStreamKioskExtractor implements StreamInfoItemExtractor {
@ -32,9 +37,10 @@ public class MediaCCCLiveStreamKioskExtractor implements StreamInfoItemExtractor
return roomInfo.getString("link");
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return roomInfo.getString("thumb");
public List<Image> getThumbnails() throws ParsingException {
return getThumbnailsFromLiveStreamItem(roomInfo);
}
@Override
@ -75,12 +81,6 @@ public class MediaCCCLiveStreamKioskExtractor implements StreamInfoItemExtractor
return "https://media.ccc.de/c/" + conferenceInfo.getString("slug");
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;

View File

@ -1,21 +1,33 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.time.OffsetDateTime;
import java.time.format.DateTimeParseException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.regex.Pattern;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public final class MediaCCCParsingHelper {
// {conference_slug}/{room_slug}
// conference_slug/room_slug
private static final Pattern LIVE_STREAM_ID_PATTERN = Pattern.compile("\\w+/\\w+");
private static JsonArray liveStreams = null;
@ -69,4 +81,98 @@ public final class MediaCCCParsingHelper {
}
return liveStreams;
}
/**
* Get an {@link Image} list from a given image logo URL.
*
* <p>
* If the image URL is null or empty, an empty list is returned; otherwise, a singleton list is
* returned containing an {@link Image} with the image URL with its height, width and
* resolution unknown.
* </p>
*
* @param logoImageUrl a logo image URL, which can be null or empty
* @return an unmodifiable list of {@link Image}s, which is always empty or a singleton
*/
@Nonnull
public static List<Image> getImageListFromLogoImageUrl(@Nullable final String logoImageUrl) {
if (isNullOrEmpty(logoImageUrl)) {
return List.of();
}
return List.of(new Image(logoImageUrl, HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.UNKNOWN));
}
/**
* Get the {@link Image} list of thumbnails from a given stream item.
*
* <p>
* MediaCCC API provides two thumbnails for a stream item: a {@code thumb_url} one, which is
* medium quality and a {@code poster_url} one, which is high quality in most cases.
* </p>
*
* @param streamItem a stream JSON item of MediaCCC's API, which must not be null
* @return an unmodifiable list, which is never null but can be empty.
*/
@Nonnull
public static List<Image> getThumbnailsFromStreamItem(@Nonnull final JsonObject streamItem) {
return getThumbnailsFromObject(streamItem, "thumb_url", "poster_url");
}
/**
* Get the {@link Image} list of thumbnails from a given live stream item.
*
* <p>
* MediaCCC API provides two URL thumbnails for a livestream item: a {@code thumb} one,
* which should be medium quality and a {@code poster_url} one, which should be high quality.
* </p>
*
* @param liveStreamItem a stream JSON item of MediaCCC's API, which must not be null
* @return an unmodifiable list, which is never null but can be empty.
*/
@Nonnull
public static List<Image> getThumbnailsFromLiveStreamItem(
@Nonnull final JsonObject liveStreamItem) {
return getThumbnailsFromObject(liveStreamItem, "thumb", "poster");
}
/**
* Utility method to get an {@link Image} list of thumbnails from a stream or a livestream.
*
* <p>
* MediaCCC's API thumbnails come from two elements: a {@code thumb} element, which links to a
* medium thumbnail and a {@code poster} element, which links to a high thumbnail.
* </p>
* <p>
* Thumbnails are only added if their URLs are not null or empty.
* </p>
*
* @param streamOrLivestreamItem a (live)stream JSON item of MediaCCC's API, which must not be
* null
* @param thumbUrlKey the name of the {@code thumb} URL key
* @param posterUrlKey the name of the {@code poster} URL key
* @return an unmodifiable list, which is never null but can be empty.
*/
@Nonnull
private static List<Image> getThumbnailsFromObject(
@Nonnull final JsonObject streamOrLivestreamItem,
@Nonnull final String thumbUrlKey,
@Nonnull final String posterUrlKey) {
final List<Image> imageList = new ArrayList<>(2);
final String thumbUrl = streamOrLivestreamItem.getString(thumbUrlKey);
if (!isNullOrEmpty(thumbUrl)) {
imageList.add(new Image(thumbUrl, HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.MEDIUM));
}
final String posterUrl = streamOrLivestreamItem.getString(posterUrlKey);
if (!isNullOrEmpty(posterUrl)) {
imageList.add(new Image(posterUrl, HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.HIGH));
}
return Collections.unmodifiableList(imageList);
}
}

View File

@ -23,6 +23,8 @@ import javax.annotation.Nonnull;
public class MediaCCCRecentKiosk extends KioskExtractor<StreamInfoItem> {
public static final String KIOSK_ID = "recent";
private JsonObject doc;
public MediaCCCRecentKiosk(final StreamingService streamingService,
@ -77,6 +79,6 @@ public class MediaCCCRecentKiosk extends KioskExtractor<StreamInfoItem> {
@Nonnull
@Override
public String getName() throws ParsingException {
return "recent";
return KIOSK_ID;
}
}

View File

@ -2,6 +2,7 @@ package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferenceLinkHandlerFactory;
@ -10,9 +11,13 @@ import org.schabi.newpipe.extractor.stream.StreamType;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.List;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getImageListFromLogoImageUrl;
public class MediaCCCRecentKioskExtractor implements StreamInfoItemExtractor {
private final JsonObject event;
@ -31,9 +36,10 @@ public class MediaCCCRecentKioskExtractor implements StreamInfoItemExtractor {
return event.getString("frontend_link");
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return event.getString("thumb_url");
public List<Image> getThumbnails() throws ParsingException {
return getImageListFromLogoImageUrl(event.getString("poster_url"));
}
@Override
@ -65,17 +71,11 @@ public class MediaCCCRecentKioskExtractor implements StreamInfoItemExtractor {
@Override
public String getUploaderUrl() throws ParsingException {
return new MediaCCCConferenceLinkHandlerFactory()
return MediaCCCConferenceLinkHandlerFactory.getInstance()
.fromUrl(event.getString("conference_url")) // API URL
.getUrl(); // web URL
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;

View File

@ -9,8 +9,10 @@ import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelInfoItem;
@ -18,7 +20,6 @@ import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems.MediaCCCStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferencesListLinkHandlerFactory;
@ -38,7 +39,8 @@ public class MediaCCCSearchExtractor extends SearchExtractor {
super(service, linkHandler);
try {
conferenceKiosk = new MediaCCCConferenceKiosk(service,
new MediaCCCConferencesListLinkHandlerFactory().fromId("conferences"),
MediaCCCConferencesListLinkHandlerFactory.getInstance()
.fromId("conferences"),
"conferences");
} catch (final Exception e) {
e.printStackTrace();
@ -156,9 +158,10 @@ public class MediaCCCSearchExtractor extends SearchExtractor {
return item.getUrl();
}
@Nonnull
@Override
public String getThumbnailUrl() {
return item.getThumbnailUrl();
public List<Image> getThumbnails() {
return item.getThumbnails();
}
});
}

View File

@ -1,5 +1,8 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getImageListFromLogoImageUrl;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getThumbnailsFromStreamItem;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.parseDateFrom;
import static org.schabi.newpipe.extractor.stream.AudioStream.UNKNOWN_BITRATE;
import static org.schabi.newpipe.extractor.stream.Stream.ID_UNKNOWN;
@ -8,6 +11,7 @@ import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -51,13 +55,13 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public DateWrapper getUploadDate() throws ParsingException {
return new DateWrapper(MediaCCCParsingHelper.parseDateFrom(getTextualUploadDate()));
return new DateWrapper(parseDateFrom(getTextualUploadDate()));
}
@Nonnull
@Override
public String getThumbnailUrl() {
return data.getString("thumb_url");
public List<Image> getThumbnails() {
return getThumbnailsFromStreamItem(data);
}
@Nonnull
@ -91,8 +95,8 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public String getUploaderAvatarUrl() {
return conferenceData.getString("logo_url");
public List<Image> getUploaderAvatars() {
return getImageListFromLogoImageUrl(conferenceData.getString("logo_url"));
}
@Override
@ -126,7 +130,10 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
// track with multiple languages, so there is no specific language for this stream
// Don't set the audio language in this case
if (language != null && !language.contains("-")) {
builder.setAudioLocale(LocaleCompat.forLanguageTag(language));
builder.setAudioLocale(LocaleCompat.forLanguageTag(language).orElseThrow(() ->
new ParsingException(
"Cannot convert this language to a locale: " + language)
));
}
// Not checking containsSimilarStream here, since MediaCCC does not provide enough

View File

@ -2,10 +2,16 @@ package org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getImageListFromLogoImageUrl;
public class MediaCCCConferenceInfoItemExtractor implements ChannelInfoItemExtractor {
private final JsonObject conference;
@ -43,8 +49,9 @@ public class MediaCCCConferenceInfoItemExtractor implements ChannelInfoItemExtra
return conference.getString("url");
}
@Nonnull
@Override
public String getThumbnailUrl() {
return conference.getString("logo_url");
public List<Image> getThumbnails() {
return getImageListFromLogoImageUrl(conference.getString("logo_url"));
}
}

View File

@ -1,13 +1,18 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getThumbnailsFromStreamItem;
public class MediaCCCStreamInfoItemExtractor implements StreamInfoItemExtractor {
private final JsonObject event;
@ -46,12 +51,6 @@ public class MediaCCCStreamInfoItemExtractor implements StreamInfoItemExtractor
return event.getString("conference_url");
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
@ -84,8 +83,9 @@ public class MediaCCCStreamInfoItemExtractor implements StreamInfoItemExtractor
+ event.getString("guid");
}
@Nonnull
@Override
public String getThumbnailUrl() {
return event.getString("thumb_url");
public List<Image> getThumbnails() {
return getThumbnailsFromStreamItem(event);
}
}

View File

@ -1,12 +1,22 @@
package org.schabi.newpipe.extractor.services.media_ccc.linkHandler;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.utils.Parser;
import java.util.List;
public class MediaCCCConferenceLinkHandlerFactory extends ListLinkHandlerFactory {
/**
* Since MediaCCC does not really have channel tabs (i.e. it only has one single "tab" with videos),
* this link handler acts both as the channel link handler and the channel tab link handler. That's
* why {@link #getAvailableContentFilter()} has been overridden.
*/
public final class MediaCCCConferenceLinkHandlerFactory extends ListLinkHandlerFactory {
private static final MediaCCCConferenceLinkHandlerFactory INSTANCE
= new MediaCCCConferenceLinkHandlerFactory();
public static final String CONFERENCE_API_ENDPOINT
= "https://api.media.ccc.de/public/conferences/";
public static final String CONFERENCE_PATH = "https://media.ccc.de/c/";
@ -14,15 +24,23 @@ public class MediaCCCConferenceLinkHandlerFactory extends ListLinkHandlerFactory
= "(?:(?:(?:api\\.)?media\\.ccc\\.de/public/conferences/)"
+ "|(?:media\\.ccc\\.de/[bc]/))([^/?&#]*)";
private MediaCCCConferenceLinkHandlerFactory() {
}
public static MediaCCCConferenceLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return CONFERENCE_PATH + id;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return Parser.matchGroup1(ID_PATTERN, url);
}
@ -34,4 +52,15 @@ public class MediaCCCConferenceLinkHandlerFactory extends ListLinkHandlerFactory
return false;
}
}
/**
* @see MediaCCCConferenceLinkHandlerFactory
* @return MediaCCC's only channel "tab", i.e. {@link ChannelTabs#VIDEOS}
*/
@Override
public String[] getAvailableContentFilter() {
return new String[]{
ChannelTabs.VIDEOS,
};
}
}

View File

@ -5,15 +5,28 @@ import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import java.util.List;
public class MediaCCCConferencesListLinkHandlerFactory extends ListLinkHandlerFactory {
public final class MediaCCCConferencesListLinkHandlerFactory extends ListLinkHandlerFactory {
private static final MediaCCCConferencesListLinkHandlerFactory INSTANCE =
new MediaCCCConferencesListLinkHandlerFactory();
private MediaCCCConferencesListLinkHandlerFactory() {
}
public static MediaCCCConferencesListLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return "conferences";
}
@Override
public String getUrl(final String id, final List<String> contentFilter,
final String sortFilter) throws ParsingException {
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return "https://media.ccc.de/public/conferences";
}

View File

@ -6,11 +6,22 @@ import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import java.util.List;
import java.util.regex.Pattern;
public class MediaCCCLiveListLinkHandlerFactory extends ListLinkHandlerFactory {
public final class MediaCCCLiveListLinkHandlerFactory extends ListLinkHandlerFactory {
private static final MediaCCCLiveListLinkHandlerFactory INSTANCE =
new MediaCCCLiveListLinkHandlerFactory();
private static final String STREAM_PATTERN = "^(?:https?://)?media\\.ccc\\.de/live$";
private MediaCCCLiveListLinkHandlerFactory() {
}
public static MediaCCCLiveListLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return "live";
}
@ -22,7 +33,8 @@ public class MediaCCCLiveListLinkHandlerFactory extends ListLinkHandlerFactory {
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
// FIXME: wrong URL; should be https://streaming.media.ccc.de/{conference_slug}/{room_slug}
return "https://media.ccc.de/live";
}

View File

@ -1,32 +0,0 @@
package org.schabi.newpipe.extractor.services.media_ccc.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.utils.Parser;
public class MediaCCCLiveStreamLinkHandlerFactory extends LinkHandlerFactory {
public static final String VIDEO_API_ENDPOINT = "https://api.media.ccc.de/public/events/";
private static final String VIDEO_PATH = "https://streaming.media.ccc.de/v/";
private static final String ID_PATTERN
= "(?:(?:(?:api\\.)?media\\.ccc\\.de/public/events/)"
+ "|(?:media\\.ccc\\.de/v/))([^/?&#]*)";
@Override
public String getId(final String url) throws ParsingException {
return Parser.matchGroup1(ID_PATTERN, url);
}
@Override
public String getUrl(final String id) throws ParsingException {
return VIDEO_PATH + id;
}
@Override
public boolean onAcceptUrl(final String url) {
try {
return getId(url) != null;
} catch (final ParsingException e) {
return false;
}
}
}

View File

@ -1,15 +1,27 @@
package org.schabi.newpipe.extractor.services.media_ccc.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import java.util.List;
import java.util.regex.Pattern;
public class MediaCCCRecentListLinkHandlerFactory extends ListLinkHandlerFactory {
public final class MediaCCCRecentListLinkHandlerFactory extends ListLinkHandlerFactory {
private static final MediaCCCRecentListLinkHandlerFactory INSTANCE =
new MediaCCCRecentListLinkHandlerFactory();
private static final String PATTERN = "^(https?://)?media\\.ccc\\.de/recent/?$";
private MediaCCCRecentListLinkHandlerFactory() {
}
public static MediaCCCRecentListLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return "recent";
}
@ -21,7 +33,8 @@ public class MediaCCCRecentListLinkHandlerFactory extends ListLinkHandlerFactory
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return "https://media.ccc.de/recent";
}
}

View File

@ -4,14 +4,24 @@ import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.UnsupportedEncodingException;
import java.util.List;
public class MediaCCCSearchQueryHandlerFactory extends SearchQueryHandlerFactory {
public final class MediaCCCSearchQueryHandlerFactory extends SearchQueryHandlerFactory {
private static final MediaCCCSearchQueryHandlerFactory INSTANCE =
new MediaCCCSearchQueryHandlerFactory();
public static final String ALL = "all";
public static final String CONFERENCES = "conferences";
public static final String EVENTS = "events";
private MediaCCCSearchQueryHandlerFactory() {
}
public static MediaCCCSearchQueryHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String[] getAvailableContentFilter() {
return new String[]{
@ -27,12 +37,10 @@ public class MediaCCCSearchQueryHandlerFactory extends SearchQueryHandlerFactory
}
@Override
public String getUrl(final String query, final List<String> contentFilter,
final String sortFilter) throws ParsingException {
try {
return "https://media.ccc.de/public/events/search?q=" + Utils.encodeUrlUtf8(query);
} catch (final UnsupportedEncodingException e) {
throw new ParsingException("Could not create search string with query: " + query, e);
}
public String getUrl(final String query,
final List<String> contentFilter,
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return "https://media.ccc.de/public/events/search?q=" + Utils.encodeUrlUtf8(query);
}
}

View File

@ -5,7 +5,11 @@ import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper;
import org.schabi.newpipe.extractor.utils.Parser;
public class MediaCCCStreamLinkHandlerFactory extends LinkHandlerFactory {
public final class MediaCCCStreamLinkHandlerFactory extends LinkHandlerFactory {
private static final MediaCCCStreamLinkHandlerFactory INSTANCE =
new MediaCCCStreamLinkHandlerFactory();
public static final String VIDEO_API_ENDPOINT = "https://api.media.ccc.de/public/events/";
private static final String VIDEO_PATH = "https://media.ccc.de/v/";
private static final String RECORDING_ID_PATTERN
@ -15,8 +19,15 @@ public class MediaCCCStreamLinkHandlerFactory extends LinkHandlerFactory {
private static final String LIVE_STREAM_ID_PATTERN
= "streaming\\.media\\.ccc\\.de\\/(\\w+\\/\\w+)";
private MediaCCCStreamLinkHandlerFactory() {
}
public static MediaCCCStreamLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
String streamId = null;
try {
streamId = Parser.matchGroup1(LIVE_STREAM_ID_PATTERN, url);
@ -30,7 +41,7 @@ public class MediaCCCStreamLinkHandlerFactory extends LinkHandlerFactory {
}
@Override
public String getUrl(final String id) throws ParsingException {
public String getUrl(final String id) throws ParsingException, UnsupportedOperationException {
if (MediaCCCParsingHelper.isLiveStreamId(id)) {
return LIVE_STREAM_PATH + id;
}

View File

@ -3,6 +3,8 @@ package org.schabi.newpipe.extractor.services.peertube;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
@ -14,12 +16,21 @@ import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeSepiaSt
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.Parser;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nonnull;
import java.time.Instant;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.time.format.DateTimeParseException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.utils.Utils.isBlank;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public final class PeertubeParsingHelper {
public static final String START_KEY = "start";
@ -32,7 +43,7 @@ public final class PeertubeParsingHelper {
public static void validate(final JsonObject json) throws ContentNotAvailableException {
final String error = json.getString("error");
if (!Utils.isBlank(error)) {
if (!isBlank(error)) {
throw new ContentNotAvailableException(error);
}
}
@ -53,7 +64,7 @@ public final class PeertubeParsingHelper {
} catch (final Parser.RegexException e) {
return null;
}
if (Utils.isBlank(prevStart)) {
if (isBlank(prevStart)) {
return null;
}
@ -72,24 +83,29 @@ public final class PeertubeParsingHelper {
}
}
public static void collectStreamsFrom(final InfoItemsCollector collector,
final JsonObject json,
final String baseUrl) throws ParsingException {
collectStreamsFrom(collector, json, baseUrl, false);
public static void collectItemsFrom(final InfoItemsCollector collector,
final JsonObject json,
final String baseUrl) throws ParsingException {
collectItemsFrom(collector, json, baseUrl, false);
}
/**
* Collect stream from json with collector
* Collect items from the given JSON object with the given collector.
*
* <p>
* Supported info item types are streams with their Sepia variant, channels and playlists.
* </p>
*
* @param collector the collector used to collect information
* @param json the file to retrieve data from
* @param baseUrl the base Url of the instance
* @param sepia if we should use PeertubeSepiaStreamInfoItemExtractor
* @param json the JSOn response to retrieve data from
* @param baseUrl the base URL of the instance
* @param sepia if we should use {@code PeertubeSepiaStreamInfoItemExtractor} to extract
* streams or {@code PeertubeStreamInfoItemExtractor} otherwise
*/
public static void collectStreamsFrom(final InfoItemsCollector collector,
final JsonObject json,
final String baseUrl,
final boolean sepia) throws ParsingException {
public static void collectItemsFrom(final InfoItemsCollector collector,
final JsonObject json,
final String baseUrl,
final boolean sepia) throws ParsingException {
final JsonArray contents;
try {
contents = (JsonArray) JsonUtils.getValue(json, "data");
@ -123,4 +139,179 @@ public final class PeertubeParsingHelper {
}
}
/**
* Get avatars from a {@code ownerAccount} or a {@code videoChannel} {@link JsonObject}.
*
* <p>
* If the {@code avatars} {@link JsonArray} is present and non null or empty, avatars will be
* extracted from this array using {@link #getImagesFromAvatarOrBannerArray(String, JsonArray)}.
* </p>
*
* <p>
* If that's not the case, an avatar will extracted using the {@code avatar} {@link JsonObject}.
* </p>
*
* <p>
* Note that only images for which paths are not null and not empty will be added to the
* unmodifiable {@link Image} list returned.
* </p>
*
* @param baseUrl the base URL of the PeerTube instance
* @param ownerAccountOrVideoChannelObject the {@code ownerAccount} or {@code videoChannel}
* {@link JsonObject}
* @return an unmodifiable list of {@link Image}s, which may be empty but never null
*/
@Nonnull
public static List<Image> getAvatarsFromOwnerAccountOrVideoChannelObject(
@Nonnull final String baseUrl,
@Nonnull final JsonObject ownerAccountOrVideoChannelObject) {
return getImagesFromAvatarsOrBanners(baseUrl, ownerAccountOrVideoChannelObject,
"avatars", "avatar");
}
/**
* Get banners from a {@code ownerAccount} or a {@code videoChannel} {@link JsonObject}.
*
* <p>
* If the {@code banners} {@link JsonArray} is present and non null or empty, banners will be
* extracted from this array using {@link #getImagesFromAvatarOrBannerArray(String, JsonArray)}.
* </p>
*
* <p>
* If that's not the case, a banner will extracted using the {@code banner} {@link JsonObject}.
* </p>
*
* <p>
* Note that only images for which paths are not null and not empty will be added to the
* unmodifiable {@link Image} list returned.
* </p>
*
* @param baseUrl the base URL of the PeerTube instance
* @param ownerAccountOrVideoChannelObject the {@code ownerAccount} or {@code videoChannel}
* {@link JsonObject}
* @return an unmodifiable list of {@link Image}s, which may be empty but never null
*/
@Nonnull
public static List<Image> getBannersFromAccountOrVideoChannelObject(
@Nonnull final String baseUrl,
@Nonnull final JsonObject ownerAccountOrVideoChannelObject) {
return getImagesFromAvatarsOrBanners(baseUrl, ownerAccountOrVideoChannelObject,
"banners", "banner");
}
/**
* Get thumbnails from a playlist or a video item {@link JsonObject}.
*
* <p>
* PeerTube provides two thumbnails in its API: a low one, represented by the value of the
* {@code thumbnailPath} key, and a medium one, represented by the value of the
* {@code previewPath} key.
* </p>
*
* <p>
* If a value is not null or empty, an {@link Image} will be added to the list returned with
* the URL to the thumbnail ({@code baseUrl + value}), a height and a width unknown and the
* corresponding resolution level (see above).
* </p>
*
* @param baseUrl the base URL of the PeerTube instance
* @param playlistOrVideoItemObject the playlist or the video item {@link JsonObject}, which
* must not be null
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty
*/
@Nonnull
public static List<Image> getThumbnailsFromPlaylistOrVideoItem(
@Nonnull final String baseUrl,
@Nonnull final JsonObject playlistOrVideoItemObject) {
final List<Image> imageList = new ArrayList<>(2);
final String thumbnailPath = playlistOrVideoItemObject.getString("thumbnailPath");
if (!isNullOrEmpty(thumbnailPath)) {
imageList.add(new Image(baseUrl + thumbnailPath, HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.LOW));
}
final String previewPath = playlistOrVideoItemObject.getString("previewPath");
if (!isNullOrEmpty(previewPath)) {
imageList.add(new Image(baseUrl + previewPath, HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.MEDIUM));
}
return Collections.unmodifiableList(imageList);
}
/**
* Utility method to get avatars and banners from video channels and accounts from given name
* keys.
*
* <p>
* Only images for which paths are not null and not empty will be added to the unmodifiable
* {@link Image} list returned and only the width of avatars or banners is provided by the API,
* and so is the only image dimension known.
* </p>
*
* @param baseUrl the base URL of the PeerTube instance
* @param ownerAccountOrVideoChannelObject the {@code ownerAccount} or {@code videoChannel}
* {@link JsonObject}
* @param jsonArrayName the key name of the {@link JsonArray} to which
* extract all images available ({@code avatars} or
* {@code banners})
* @param jsonObjectName the key name of the {@link JsonObject} to which
* extract a single image ({@code avatar} or
* {@code banner}), used as a fallback if the images
* {@link JsonArray} is null or empty
* @return an unmodifiable list of {@link Image}s, which may be empty but never null
*/
@Nonnull
private static List<Image> getImagesFromAvatarsOrBanners(
@Nonnull final String baseUrl,
@Nonnull final JsonObject ownerAccountOrVideoChannelObject,
@Nonnull final String jsonArrayName,
@Nonnull final String jsonObjectName) {
final JsonArray images = ownerAccountOrVideoChannelObject.getArray(jsonArrayName);
if (!isNullOrEmpty(images)) {
return getImagesFromAvatarOrBannerArray(baseUrl, images);
}
final JsonObject image = ownerAccountOrVideoChannelObject.getObject(jsonObjectName);
final String path = image.getString("path");
if (!isNullOrEmpty(path)) {
return List.of(new Image(baseUrl + path, HEIGHT_UNKNOWN,
image.getInt("width", WIDTH_UNKNOWN), ResolutionLevel.UNKNOWN));
}
return List.of();
}
/**
* Get {@link Image}s from an {@code avatars} or a {@code banners} {@link JsonArray}.
*
* <p>
* Only images for which paths are not null and not empty will be added to the
* unmodifiable {@link Image} list returned.
* </p>
*
* <p>
* Note that only the width of avatars or banners is provided by the API, and so only is the
* only dimension known of images.
* </p>
*
* @param baseUrl the base URL of the PeerTube instance from which the
* {@code avatarsOrBannersArray} {@link JsonArray} comes from
* @param avatarsOrBannersArray an {@code avatars} or {@code banners} {@link JsonArray}
* @return an unmodifiable list of {@link Image}s, which may be empty but never null
*/
@Nonnull
private static List<Image> getImagesFromAvatarOrBannerArray(
@Nonnull final String baseUrl,
@Nonnull final JsonArray avatarsOrBannersArray) {
return avatarsOrBannersArray.stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.filter(image -> !isNullOrEmpty(image.getString("path")))
.map(image -> new Image(baseUrl + image.getString("path"), HEIGHT_UNKNOWN,
image.getInt("width", WIDTH_UNKNOWN), ResolutionLevel.UNKNOWN))
.collect(Collectors.toUnmodifiableList());
}
}

View File

@ -6,6 +6,7 @@ import static java.util.Arrays.asList;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.kiosk.KioskList;
@ -19,6 +20,7 @@ import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeAccountExtractor;
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeChannelExtractor;
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeChannelTabExtractor;
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeCommentsExtractor;
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubePlaylistExtractor;
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeSearchExtractor;
@ -26,6 +28,7 @@ import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeStreamE
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeSuggestionExtractor;
import org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeTrendingExtractor;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeChannelTabLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeCommentsLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubePlaylistLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeSearchQueryHandlerFactory;
@ -60,6 +63,11 @@ public class PeertubeService extends StreamingService {
return PeertubeChannelLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelTabLHFactory() {
return PeertubeChannelTabLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getPlaylistLHFactory() {
return PeertubePlaylistLinkHandlerFactory.getInstance();
@ -103,6 +111,12 @@ public class PeertubeService extends StreamingService {
}
}
@Override
public ChannelTabExtractor getChannelTabExtractor(final ListLinkHandler linkHandler)
throws ExtractionException {
return new PeertubeChannelTabExtractor(this, linkHandler);
}
@Override
public PlaylistExtractor getPlaylistExtractor(final ListLinkHandler linkHandler)
throws ExtractionException {
@ -136,17 +150,20 @@ public class PeertubeService extends StreamingService {
@Override
public KioskList getKioskList() throws ExtractionException {
final PeertubeTrendingLinkHandlerFactory h =
PeertubeTrendingLinkHandlerFactory.getInstance();
final KioskList.KioskExtractorFactory kioskFactory = (streamingService, url, id) ->
new PeertubeTrendingExtractor(
PeertubeService.this,
new PeertubeTrendingLinkHandlerFactory().fromId(id),
h.fromId(id),
id
);
final KioskList list = new KioskList(this);
// add kiosks here e.g.:
final PeertubeTrendingLinkHandlerFactory h = new PeertubeTrendingLinkHandlerFactory();
try {
list.addKioskEntry(kioskFactory, h, PeertubeTrendingLinkHandlerFactory.KIOSK_TRENDING);
list.addKioskEntry(kioskFactory, h,
@ -160,6 +177,4 @@ public class PeertubeService extends StreamingService {
return list;
}
}

View File

@ -4,31 +4,27 @@ import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.downloader.Response;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeChannelTabLinkHandlerFactory;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.List;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.COUNT_KEY;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.ITEMS_PER_PAGE;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.START_KEY;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.collectStreamsFrom;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.getAvatarsFromOwnerAccountOrVideoChannelObject;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.getBannersFromAccountOrVideoChannelObject;
public class PeertubeAccountExtractor extends ChannelExtractor {
private JsonObject json;
@ -41,20 +37,16 @@ public class PeertubeAccountExtractor extends ChannelExtractor {
this.baseUrl = getBaseUrl();
}
@Nonnull
@Override
public String getAvatarUrl() {
String value;
try {
value = JsonUtils.getString(json, "avatar.path");
} catch (final Exception e) {
value = "/client/assets/images/default-avatar.png";
}
return baseUrl + value;
public List<Image> getAvatars() {
return getAvatarsFromOwnerAccountOrVideoChannelObject(baseUrl, json);
}
@Nonnull
@Override
public String getBannerUrl() {
return null;
public List<Image> getBanners() {
return getBannersFromAccountOrVideoChannelObject(baseUrl, json);
}
@Override
@ -107,9 +99,10 @@ public class PeertubeAccountExtractor extends ChannelExtractor {
return "";
}
@Nonnull
@Override
public String getParentChannelAvatarUrl() {
return "";
public List<Image> getParentChannelAvatars() {
return List.of();
}
@Override
@ -119,54 +112,19 @@ public class PeertubeAccountExtractor extends ChannelExtractor {
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() throws IOException, ExtractionException {
return getPage(new Page(baseUrl + "/api/v1/" + getId() + "/videos?" + START_KEY + "=0&"
+ COUNT_KEY + "=" + ITEMS_PER_PAGE));
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page)
throws IOException, ExtractionException {
if (page == null || isNullOrEmpty(page.getUrl())) {
throw new IllegalArgumentException("Page doesn't contain an URL");
}
final Response response = getDownloader().get(page.getUrl());
JsonObject pageJson = null;
if (response != null && !Utils.isBlank(response.responseBody())) {
try {
pageJson = JsonParser.object().from(response.responseBody());
} catch (final Exception e) {
throw new ParsingException("Could not parse json data for account info", e);
}
}
if (pageJson != null) {
PeertubeParsingHelper.validate(pageJson);
final long total = pageJson.getLong("total");
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
collectStreamsFrom(collector, pageJson, getBaseUrl());
return new InfoItemsPage<>(collector,
PeertubeParsingHelper.getNextPage(page.getUrl(), total));
} else {
throw new ExtractionException("Unable to get PeerTube account info");
}
public List<ListLinkHandler> getTabs() throws ParsingException {
return List.of(
PeertubeChannelTabLinkHandlerFactory.getInstance().fromQuery(getId(),
List.of(ChannelTabs.VIDEOS), "", getBaseUrl()),
PeertubeChannelTabLinkHandlerFactory.getInstance().fromQuery(getId(),
List.of(ChannelTabs.CHANNELS), "", getBaseUrl()));
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
String accountUrl = baseUrl + PeertubeChannelLinkHandlerFactory.API_ENDPOINT;
if (getId().contains(ACCOUNTS)) {
accountUrl += getId();
} else {
accountUrl += ACCOUNTS + getId();
}
final Response response = downloader.get(accountUrl);
final Response response = downloader.get(baseUrl
+ PeertubeChannelLinkHandlerFactory.API_ENDPOINT + getId());
if (response != null) {
setInitialData(response.responseBody());
} else {

View File

@ -3,30 +3,26 @@ package org.schabi.newpipe.extractor.services.peertube.extractors;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.downloader.Response;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeChannelTabLinkHandlerFactory;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.List;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.COUNT_KEY;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.ITEMS_PER_PAGE;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.START_KEY;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.collectStreamsFrom;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.getAvatarsFromOwnerAccountOrVideoChannelObject;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.getBannersFromAccountOrVideoChannelObject;
public class PeertubeChannelExtractor extends ChannelExtractor {
private JsonObject json;
@ -38,20 +34,16 @@ public class PeertubeChannelExtractor extends ChannelExtractor {
this.baseUrl = getBaseUrl();
}
@Nonnull
@Override
public String getAvatarUrl() {
String value;
try {
value = JsonUtils.getString(json, "avatar.path");
} catch (final Exception e) {
value = "/client/assets/images/default-avatar.png";
}
return baseUrl + value;
public List<Image> getAvatars() {
return getAvatarsFromOwnerAccountOrVideoChannelObject(baseUrl, json);
}
@Nonnull
@Override
public String getBannerUrl() {
return null;
public List<Image> getBanners() {
return getBannersFromAccountOrVideoChannelObject(baseUrl, json);
}
@Override
@ -80,15 +72,11 @@ public class PeertubeChannelExtractor extends ChannelExtractor {
return JsonUtils.getString(json, "ownerAccount.url");
}
@Nonnull
@Override
public String getParentChannelAvatarUrl() {
String value;
try {
value = JsonUtils.getString(json, "ownerAccount.avatar.path");
} catch (final Exception e) {
value = "/client/assets/images/default-avatar.png";
}
return baseUrl + value;
public List<Image> getParentChannelAvatars() {
return getAvatarsFromOwnerAccountOrVideoChannelObject(
baseUrl, json.getObject("ownerAccount"));
}
@Override
@ -98,41 +86,12 @@ public class PeertubeChannelExtractor extends ChannelExtractor {
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() throws IOException, ExtractionException {
return getPage(new Page(baseUrl + "/api/v1/" + getId() + "/videos?" + START_KEY + "=0&"
+ COUNT_KEY + "=" + ITEMS_PER_PAGE));
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page)
throws IOException, ExtractionException {
if (page == null || isNullOrEmpty(page.getUrl())) {
throw new IllegalArgumentException("Page doesn't contain an URL");
}
final Response response = getDownloader().get(page.getUrl());
JsonObject pageJson = null;
if (response != null && !Utils.isBlank(response.responseBody())) {
try {
pageJson = JsonParser.object().from(response.responseBody());
} catch (final Exception e) {
throw new ParsingException("Could not parse json data for channel info", e);
}
}
if (pageJson != null) {
PeertubeParsingHelper.validate(pageJson);
final long total = pageJson.getLong("total");
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
collectStreamsFrom(collector, pageJson, getBaseUrl());
return new InfoItemsPage<>(collector,
PeertubeParsingHelper.getNextPage(page.getUrl(), total));
} else {
throw new ExtractionException("Unable to get PeerTube channel info");
}
public List<ListLinkHandler> getTabs() throws ParsingException {
return List.of(
PeertubeChannelTabLinkHandlerFactory.getInstance().fromQuery(getId(),
List.of(ChannelTabs.VIDEOS), "", getBaseUrl()),
PeertubeChannelTabLinkHandlerFactory.getInstance().fromQuery(getId(),
List.of(ChannelTabs.PLAYLISTS), "", getBaseUrl()));
}
@Override

View File

@ -1,22 +1,24 @@
package org.schabi.newpipe.extractor.services.peertube.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.Comparator;
import java.util.List;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.getAvatarsFromOwnerAccountOrVideoChannelObject;
public class PeertubeChannelInfoItemExtractor implements ChannelInfoItemExtractor {
final JsonObject item;
final JsonObject uploader;
final String baseUrl;
private final JsonObject item;
private final String baseUrl;
public PeertubeChannelInfoItemExtractor(@Nonnull final JsonObject item,
@Nonnull final String baseUrl) {
this.item = item;
this.uploader = item.getObject("uploader");
this.baseUrl = baseUrl;
}
@ -30,14 +32,10 @@ public class PeertubeChannelInfoItemExtractor implements ChannelInfoItemExtracto
return item.getString("url");
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return item.getArray("avatars").stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.max(Comparator.comparingInt(avatar -> avatar.getInt("width")))
.map(avatar -> baseUrl + avatar.getString("path"))
.orElse(null);
public List<Image> getThumbnails() throws ParsingException {
return getAvatarsFromOwnerAccountOrVideoChannelObject(baseUrl, item);
}
@Override
@ -52,7 +50,7 @@ public class PeertubeChannelInfoItemExtractor implements ChannelInfoItemExtracto
@Override
public long getStreamCount() throws ParsingException {
return ChannelExtractor.ITEM_COUNT_UNKNOWN;
return ListExtractor.ITEM_COUNT_UNKNOWN;
}
@Override

View File

@ -0,0 +1,80 @@
package org.schabi.newpipe.extractor.services.peertube.extractors;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.downloader.Response;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper;
import org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nonnull;
import java.io.IOException;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.COUNT_KEY;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.ITEMS_PER_PAGE;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.START_KEY;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.collectItemsFrom;
import static org.schabi.newpipe.extractor.services.peertube.linkHandler.PeertubeChannelTabLinkHandlerFactory.getUrlSuffix;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public class PeertubeChannelTabExtractor extends ChannelTabExtractor {
private final String baseUrl;
public PeertubeChannelTabExtractor(final StreamingService service,
final ListLinkHandler linkHandler)
throws ParsingException {
super(service, linkHandler);
baseUrl = getBaseUrl();
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader) {
}
@Nonnull
@Override
public InfoItemsPage<InfoItem> getInitialPage() throws IOException, ExtractionException {
return getPage(new Page(baseUrl + PeertubeChannelLinkHandlerFactory.API_ENDPOINT
+ getId() + getUrlSuffix(getName()) + "?" + START_KEY + "=0&" + COUNT_KEY + "="
+ ITEMS_PER_PAGE));
}
@Override
public InfoItemsPage<InfoItem> getPage(final Page page)
throws IOException, ExtractionException {
if (page == null || isNullOrEmpty(page.getUrl())) {
throw new IllegalArgumentException("Page doesn't contain an URL");
}
final Response response = getDownloader().get(page.getUrl());
JsonObject pageJson = null;
if (response != null && !Utils.isBlank(response.responseBody())) {
try {
pageJson = JsonParser.object().from(response.responseBody());
} catch (final Exception e) {
throw new ParsingException("Could not parse json data for account info", e);
}
}
if (pageJson == null) {
throw new ExtractionException("Unable to get account channel list");
}
PeertubeParsingHelper.validate(pageJson);
final MultiInfoItemsCollector collector = new MultiInfoItemsCollector(getServiceId());
collectItemsFrom(collector, pageJson, getBaseUrl());
return new InfoItemsPage<>(collector,
PeertubeParsingHelper.getNextPage(page.getUrl(), pageJson.getLong("total")));
}
}

View File

@ -6,21 +6,24 @@ import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonWriter;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.ServiceList;
import org.schabi.newpipe.extractor.comments.CommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Objects;
import static org.schabi.newpipe.extractor.services.peertube.extractors.PeertubeCommentsExtractor.CHILDREN;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.getAvatarsFromOwnerAccountOrVideoChannelObject;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.parseDateFrom;
public class PeertubeCommentsInfoItemExtractor implements CommentsInfoItemExtractor {
@Nonnull
@ -52,15 +55,10 @@ public class PeertubeCommentsInfoItemExtractor implements CommentsInfoItemExtrac
return url + "/" + getCommentId();
}
@Nonnull
@Override
public String getThumbnailUrl() {
String value;
try {
value = JsonUtils.getString(item, "account.avatar.path");
} catch (final Exception e) {
value = "/client/assets/images/default-avatar.png";
}
return baseUrl + value;
public List<Image> getThumbnails() {
return getUploaderAvatars();
}
@Override
@ -76,9 +74,10 @@ public class PeertubeCommentsInfoItemExtractor implements CommentsInfoItemExtrac
@Override
public DateWrapper getUploadDate() throws ParsingException {
final String textualUploadDate = getTextualUploadDate();
return new DateWrapper(PeertubeParsingHelper.parseDateFrom(textualUploadDate));
return new DateWrapper(parseDateFrom(textualUploadDate));
}
@Nonnull
@Override
public Description getCommentText() throws ParsingException {
final String htmlText = JsonUtils.getString(item, "text");
@ -97,15 +96,10 @@ public class PeertubeCommentsInfoItemExtractor implements CommentsInfoItemExtrac
return Objects.toString(item.getLong("id"), null);
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
String value;
try {
value = JsonUtils.getString(item, "account.avatar.path");
} catch (final Exception e) {
value = "/client/assets/images/default-avatar.png";
}
return baseUrl + value;
public List<Image> getUploaderAvatars() {
return getAvatarsFromOwnerAccountOrVideoChannelObject(baseUrl, item.getObject("account"));
}
@Override
@ -155,4 +149,10 @@ public class PeertubeCommentsInfoItemExtractor implements CommentsInfoItemExtrac
}
return replyCount;
}
@Override
public boolean hasCreatorReply() {
return item.has("totalRepliesFromVideoAuthor")
&& item.getInt("totalRepliesFromVideoAuthor") > 0;
}
}

View File

@ -3,6 +3,7 @@ package org.schabi.newpipe.extractor.services.peertube.extractors;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -19,11 +20,14 @@ import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.List;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.COUNT_KEY;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.ITEMS_PER_PAGE;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.START_KEY;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.collectStreamsFrom;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.collectItemsFrom;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.getAvatarsFromOwnerAccountOrVideoChannelObject;
import static org.schabi.newpipe.extractor.services.peertube.PeertubeParsingHelper.getThumbnailsFromPlaylistOrVideoItem;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public class PeertubePlaylistExtractor extends PlaylistExtractor {
@ -36,8 +40,8 @@ public class PeertubePlaylistExtractor extends PlaylistExtractor {
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return getBaseUrl() + playlistInfo.getString("thumbnailPath");
public List<Image> getThumbnails() throws ParsingException {
return getThumbnailsFromPlaylistOrVideoItem(getBaseUrl(), playlistInfo);
}
@Override
@ -50,10 +54,11 @@ public class PeertubePlaylistExtractor extends PlaylistExtractor {
return playlistInfo.getObject("ownerAccount").getString("displayName");
}
@Nonnull
@Override
public String getUploaderAvatarUrl() throws ParsingException {
return getBaseUrl()
+ playlistInfo.getObject("ownerAccount").getObject("avatar").getString("path");
public List<Image> getUploaderAvatars() throws ParsingException {
return getAvatarsFromOwnerAccountOrVideoChannelObject(getBaseUrl(),
playlistInfo.getObject("ownerAccount"));
}
@Override
@ -90,9 +95,9 @@ public class PeertubePlaylistExtractor extends PlaylistExtractor {
@Nonnull
@Override
public String getSubChannelAvatarUrl() throws ParsingException {
return getBaseUrl()
+ playlistInfo.getObject("videoChannel").getObject("avatar").getString("path");
public List<Image> getSubChannelAvatars() throws ParsingException {
return getAvatarsFromOwnerAccountOrVideoChannelObject(getBaseUrl(),
playlistInfo.getObject("videoChannel"));
}
@Nonnull
@ -125,7 +130,7 @@ public class PeertubePlaylistExtractor extends PlaylistExtractor {
final long total = json.getLong("total");
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
collectStreamsFrom(collector, json, getBaseUrl());
collectItemsFrom(collector, json, getBaseUrl());
return new InfoItemsPage<>(collector,
PeertubeParsingHelper.getNextPage(page.getUrl(), total));

Some files were not shown because too many files have changed in this diff Show More