Commit Graph

2669 Commits

Author SHA1 Message Date
Kavin f9bd08c649 Address reviews. 2022-10-30 01:25:30 +00:00
Caleb 9282c3c13b Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb c5216f7c12 Update docstring for StreamExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb 04795fe5d2 Use Stream API for ShortFormContent#testShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
chowder 975b38a0b9 Regenerate test mocks 2022-10-30 01:23:15 +00:00
chowder 4cccd33f3d Implement isShortFormContent for StreamExtractor and StreamInfo 2022-10-30 01:23:15 +00:00
chowder 09544ceb23 Fix tests 2022-10-30 01:23:15 +00:00
chowder 644cc6cd76 Rename test function name 2022-10-30 01:23:15 +00:00
chowder daf5674951 Add ability to identify short-form StreamInfoItems 2022-10-30 01:23:12 +00:00
Tobi 3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
AudricV 65125a3eb4
[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist
The video count is now returned for this playlist, so it isn't unknown.

The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
2022-10-29 18:12:10 +02:00
AudricV 7258a53225
[YouTube] Support new playlist layout
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.

The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).

Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
2022-10-29 18:12:10 +02:00
AudricV e37e8b358e
[YouTube] Update mocks that need to be updated 2022-10-29 18:10:18 +02:00
AudricV 60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.

This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
2022-10-29 17:58:33 +02:00
AudricV 4bc90cd9d8
Merge pull request #957 from AudricV/fix-checkstyle-error
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 17:55:33 +02:00
AudricV c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor 2022-10-29 13:24:19 +02:00
Tobi 9ffdd0948b
Merge pull request #945 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.3
Bump spotbugs-annotations from 4.7.2 to 4.7.3
2022-10-18 22:55:24 +02:00
dependabot[bot] 24137bb15f
Bump spotbugs-annotations from 4.7.2 to 4.7.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.2 to 4.7.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.2...4.7.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 09:20:51 +00:00
xz-dev 0ffcb32d9c
[YouTube] Add comment reply count support (#936)
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant

Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
Isira Seneviratne b90a566dd8 Add backport implementation of Locale.forLanguageTag(). 2022-10-12 09:21:39 +05:30
Isira Seneviratne b232c29d22 Use Locale.forLanguageTag(). 2022-10-12 09:21:38 +05:30
TobiGr 4d136599bd [SoundCloud] Fix getting more comments 2022-10-11 15:44:54 +02:00
Tobi a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
TobiGr 02810a7db7 Add a comment 2022-10-10 22:22:12 +02:00
Tobi 2d50369c77
Merge pull request #888 from Isira-Seneviratne/EnumMap
Use EnumMap in PatternsHolder.
2022-10-10 12:00:06 +02:00
Tobi 54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
Tobi d7c23ef118
Merge pull request #924 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.2
Bump spotbugs-annotations from 4.7.1 to 4.7.2
2022-10-10 11:35:40 +02:00
dependabot[bot] 08339d6b0c
Bump spotbugs-annotations from 4.7.1 to 4.7.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.1 to 4.7.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.1...4.7.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:32:17 +00:00
Tobi 5537e84100
Merge pull request #931 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.1
Bump junit-bom from 5.9.0 to 5.9.1
2022-10-10 11:30:53 +02:00
dependabot[bot] 719664593f
Bump junit-bom from 5.9.0 to 5.9.1
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.0 to 5.9.1.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.0...r5.9.1)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:25:42 +00:00
TurtleArmyMc bf70d32eb4 Fix SoundcloudPlaylistExtractor: tracks are in correct order 2022-09-30 15:26:25 -04:00
Stypox 5c710da160
Merge pull request #934 from AudricV/fix-yt-throttling-decryption-function-regex
[YouTube] Fix throttling decryption function regex
2022-09-25 13:26:08 +02:00
AudricV 8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
AudricV abcee87167
[YouTube] Fix throttling parameter decryption function regex
- Quote the function name, as it may contain special regex symbols, such as
  dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
  part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
ThetaDev 7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function (#905) 2022-09-24 19:55:46 +02:00
Isira Seneviratne 0e31c86aee Avoid possible NullPointerException in MediaCCCRecentKiosk. 2022-09-21 05:40:07 +05:30
Stypox a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
Kavin 4e14644c41
Ensure comments are parsed with URLs and timestamps. 2022-09-17 16:03:39 +05:30
Stypox 14ef430546
Merge pull request #920 from AudricV/fix-yt-trending-extraction
[YouTube] Fix extraction of the Trending kiosk with the new data model returned
2022-09-14 11:14:13 +02:00
Stypox 93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
AudricV d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
This new structure allow us to filter easily Trending shorts and Recently
trending sections.

On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.

This makes that the extractor returns now only the real 50 "Now" YouTube
trends.

Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.

The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.

Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
AudricV 884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
AudricV 58b09cf78b
Merge pull request #925 from Theta-Dev/fix-td-regex
[YouTube] Fix throttling parameter decryption on Android
2022-09-10 23:30:50 +02:00
AudricV 119b9129e2
[YouTube] Fix extraction of like count with the new data model
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.

The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.

Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
ThetaDev 4905f74700
[YouTube] Fix throttling parameter decryption on Android
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
Isira Seneviratne e101d58ae7 Use EnumMap in PatternsHolder. 2022-08-26 08:15:21 +05:30
Stypox 6a858368c8
Merge pull request #912 from Stypox/downgrade-rhino
Downgrade Rhino since it uses a class not available on Android
2022-08-25 10:10:27 +02:00
Stypox fa25a7280f
Downgrade Rhino since it uses a class not available on Android
Reverts #774, see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
Stypox 25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
Stypox d8e10dd3ad
Merge pull request #907 from Isira-Seneviratne/Remove_EmptyString
Remove EMPTY_STRING.
2022-08-24 14:22:55 +02:00